SYSLOG Without Server Headaches

I wrote about services without servers a while ago, and today I want to explore an example: our venerable friend SYSLOG, and the uncomfortable frequency with which "out of disk space" and "SYSLOG server" appear together in devops lore.

SYSLOG is as simple as a protocol gets: messages travel in one direction, and there's no message encoding to speak of. You can read about the format of messages and sending SYSLOG over UDP, but the gist of it is that SYSLOG is text datagrams -- plain text inside UDP packets.

Implementing a SYSLOG server **should be** as simple as listening on a UDP port and writing the data to a file. But in reality it's not that simple. Traditional SYSLOG servers have several problems that can cause headaches:

• They're often implemented as a single process, which can become a bottleneck and single point of failure
• They run out of disk space (a disturbingly common issue with Docker containers) and require effective log rotation and archiving
• They may not be able to handle the load of incoming messages when things get bad (the worst time to fail)
• Scaling horizontally requires load balancers and escalating complexity

To be fair, those are all "server" problems and not SYSLOG problems. Why should implementing such a trivial protocol be so hard? Well, it doesn't have to be.

The mistake is combining the protocol handling and the business logic together on a server. It's an architectural pattern we've been following for decades without questioning, but does it still make sense?

SYSLOG is perhaps the purest example of the problem with server-based software architecture:

• The network handling (UDP) is trivial
• The protocol (plain text) is trivial
• The business logic (save logs) is trivial

And yet, it's a headache to make reliable. The server itself (VM or container) becomes the problem. A familiar and well-worn problem, and one without a clear and cross-cutting solution since no cloud provider offered a genuinely serverless way to handle UDP. Until Proxylity UDP Gateway came to be, that is.

Proxylity UDP Gateway is a serverless UDP proxy that handles the network connection separately from the business logic. If you want to configure a syslog service, you can do it in minutes. You don't need to worry about the server, the network connection, or the protocol -- just the content:

• Syslog Communication: Proxylity UDP Gateway handles it
• Scaling: Proxylity UDP Gateway makes it automatic, and you do no capacity planning
• Going Global: It's there from day one with Proxylity UDP Gateway, leveraging AWS's global infrastructure
• Log Storage and Lifecycle: AWS S3, CloudWatch Logs, or maybe DynamoDB handles it -- your choice and configuration

Is it really that easy? Yes. Let me show you.

Here's a CloudFormation excerpt from our examples repository that sets up a complete syslog service:

Resources:
  SyslogLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
    LogGroupName: /proxylity/syslog
    RetentionInDays: 30

  SyslogLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      ServiceToken: !ImportValue ProxylityUDPGatewayServiceToken
      Port: 514
      Protocol: UDP
      Destinations:
      - Type: CloudWatch
      LogGroup: !Ref SyslogLogGroup
      LogStreamPrefix: syslog-
      Format: Text

That's it. With these few lines of CloudFormation, you've created a SYSLOG service that:

• Scales automatically to handle any volume of SYSLOG messages (and traffic spikes)
• Never runs out of disk space (CloudWatch Logs handles storage)
• Maintains logs for exactly as long as you specify (30 days in this example)
• Never needs a security patch or update
• Can easily be deployed globally if needed

You could extend this basic example to filter messages. For instance, you might want to alert on error messages:

Resources:
  MyMetricFilter:
    Type: "AWS::Logs::MetricFilter"
    Properties:
      FilterPattern: "\[\d{2}:\d{2}:\d{2}\].* syslog:.*error.*"
      LogGroupName: !Ref SyslogLogGroup
      MetricTransformations:
        - MetricNamespace: "MyLogNamespace"
          MetricName: "SyslogErrors"
          MetricValue: "1"
          DefaultValue: "0"
  MyAlarm:
    Type: "AWS::CloudWatch::Alarm"
    Properties:
      AlarmDescription: "Alarm when syslog errors occur"
      MetricName: "SyslogErrors"
      Namespace: "MyLogNamespace"
      Statistic: "Sum"
      Period: 60
      EvaluationPeriods: 1
      Threshold: 1
      ComparisonOperator: "GreaterThanThreshold"
      AlarmActions:
        - !Ref MySNSTopic

Or perhaps you want to add processing of log message with a Lambda function for custom handling:

Resources:
  MyLambadaFunction:
    Type: "AWS::Lambda::Function"
    ...

This architectural separation of concerns is what makes serverless UDP so powerful. The network handling is done by a specialized service (UDP Gateway), while the business logic can be implemented using the most appropriate AWS services.

No more worrying about servers running out of disk space. No waking up at 2 AM because syslog crashed. No capacity planning or scaling exercises. Just reliable log collection that costs almost nothing when idle and scales automatically when needed.

We handle the UDP, you decide how to handle the data. That's it. That's all we do. It's our happy niche.