Now Available: Kinesis Data Streams Support

Proxylity UDP Gateway now supports Amazon Kinesis Data Streams for destinations. You can extract partition keys dynamically from packet payloads to control how records distribute across stream shards, enabling ordered processing per device, session, or transaction while maintaining parallel throughput.

Why Kinesis Data Streams?

Kinesis Data Streams provides scalable, real-time data ingestion with millisecond latency. Combined with UDP Gateway, you can stream UDP traffic directly into Kinesis without building custom collection infrastructure.

This integration is useful for gaming telemetry, IoT sensor data, network monitoring, financial market feeds, and syslog aggregation—any UDP-based protocol that benefits from real-time stream processing, analytics, or machine learning pipelines.

Dynamic Partition Keys

Kinesis Data Streams uses partition keys to distribute records across shards. The partition key determines which shard receives each record, directly impacting throughput and ordering guarantees. UDP Gateway allows you to extract partition keys dynamically from packet payloads using binary range expressions, giving you precise control over data distribution.

Default Behavior: Random Distribution

If you don't specify a partition key expression, UDP Gateway generates a random GUID for each packet, distributing records evenly across all shards. This maximizes throughput and parallelism when packet ordering isn't required:

Configuration example (JSON):

{
  "DestinationArn": "arn:aws:kinesis:us-east-1:123456789012:stream/telemetry-stream"
}

This configuration is ideal for high-volume, order-independent workloads like metrics aggregation or statistical sampling.

Custom Partition Keys: Controlled Distribution

For workloads requiring ordered processing per device, session, or transaction, extract partition keys from packet payloads. Records with the same partition key are guaranteed to be processed in order by the same shard consumer:

Configuration example (JSON):

{
  "DestinationArn": "arn:aws:kinesis:us-east-1:123456789012:stream/device-data",
  "Arguments": {
    "PartitionKeyExpression": "[0:4]",
    "PartitionKeyFormatter": "hex"
  }
}

This extracts bytes 0-4 from each packet payload and formats them as hexadecimal. All packets from the same device (assuming the device ID is in those bytes) flow to the same shard, maintaining per-device ordering while enabling parallel processing across different devices.

Formatter Options

The PartitionKeyFormatter argument controls how extracted bytes are converted to partition key strings. Available formatters:

• hex (default) - Hexadecimal encoding (e.g., [0xA1, 0xB2] → A1B2)
• utf8 - UTF-8 string decoding for text-based identifiers
• ascii - ASCII string decoding for 7-bit text
• base64 - Base64 encoding for compact binary representation

Choose the formatter that matches your protocol's identifier format. Hexadecimal is typically best for binary protocols, while UTF-8 works well for text-based protocols with string identifiers.

Record Format

Each UDP packet is transformed into a Kinesis Data Stream record with a JSON envelope containing metadata and the formatted payload:

{
  "Tag": "pxp-batch-12345-00042",
  "Remote": {
    "IpAddress": "198.51.100.42",
    "Port": 51234
  },
  "Local": {
    "Domain": "telemetry.example.com",
    "Port": 8514
  },
  "ReceivedAt": "2025-11-22T14:32:18.123Z",
  "Formatter": "hex",
  "Data": "48656c6c6f20576f726c64"
}

The Data field contains the UDP payload encoded according to the destination's Formatter setting (hex, base64, UTF-8, etc.). This envelope format provides full context about each packet while maintaining compatibility with standard Kinesis consumer libraries.

Batching and Throughput

UDP Gateway batches records before sending to Kinesis Data Streams, optimizing for throughput and cost efficiency. The Kinesis PutRecords API supports up to 500 records per request with a 5MB total size limit.

UDP Gateway automatically chunks large batches into 500-record sub-batches and sends them concurrently, maximizing throughput while respecting API limits. You control batching behavior using the standard batching configuration:

Batching:
  Count: 1000           # Collect up to 1000 packets
  SizeInMB: 4.0         # Or 4MB of data
  TimeoutInSeconds: 5.0 # Or wait 5 seconds

When any threshold is met, UDP Gateway processes the batch by chunking into 500-record groups and sending concurrently to Kinesis. This approach balances latency (small batches = lower latency) with throughput (larger batches = better efficiency).

Error Handling

UDP Gateway monitors Kinesis API responses and logs detailed error information when records fail to be written. The PutRecords API returns individual success/failure status for each record in a batch, and UDP Gateway reports the count of failed records along with HTTP status codes.

To capture these errors in CloudWatch Logs, configure the LogGroupName property on your destination. UDP Gateway will write error and warning messages to this log group in your account, making them available for CloudWatch Insights queries, alarms, and dashboards:

LogGroupName: /proxylity/destinations/kinesis-stream

Your destination's IAM role must include logs:CreateLogStream and logs:PutLogEvents permissions for the specified log group.

Failed records are logged but not automatically retried—UDP is a best-effort protocol, and retrying at the gateway level can cause ordering violations or duplicate processing. For critical workloads requiring guaranteed delivery, consider using SQS FIFO queues as a staging layer with Kinesis as a downstream processor.

Use Cases

Examples of what you can build:

• Gaming Telemetry - Stream player events for matchmaking, anti-cheat detection, or leaderboards. Partition by player ID to maintain event ordering per player.
• IoT Sensor Networks - Process sensor readings for anomaly detection or time-series analysis. Partition by device ID to guarantee ordered processing per sensor.
• Network Monitoring - Ingest NetFlow, sFlow, or IPFIX records for traffic analysis. Use random partitioning for maximum throughput when order doesn't matter.
• Financial Market Data - Stream market data feeds to trading or reporting systems. Partition by instrument ID to maintain quote and trade ordering.
• Syslog Aggregation - Centralize syslog messages from distributed systems. Partition by hostname to maintain log ordering per system.

Downstream Consumers

Kinesis Data Streams serves as a pipeline to numerous AWS services and custom applications:

• Kinesis Data Analytics - Run SQL queries on streaming data for real-time aggregations, filtering, or transformations
• AWS Lambda - Trigger serverless functions to process each record or micro-batch
• Kinesis Data Firehose - Automatically deliver data to S3, Redshift, OpenSearch, or third-party destinations
• Amazon EMR - Process streams with Spark, Flink, or other big data frameworks
• Custom Applications - Build consumers using the Kinesis Client Library (KCL) or AWS SDKs in any language

UDP Gateway sends packets to Kinesis, which streams them to whatever processing pipeline you need.

Getting Started

Kinesis Data Streams destinations are available now in all regions where UDP Gateway is supported.

To use it, specify the stream ARN in your destination configuration. Add partition key extraction arguments if you need custom distribution. UDP Gateway handles batching, API calls, and error logging.

For detailed configuration examples, ARN formats, and IAM permission requirements, see the Kinesis Data Streams destination documentation.

A complete working example demonstrating both random and partition-based distribution is available in the Kinesis example in our GitHub examples repository.