SQS Destinations

Proxylity's integration with SQS writes batches of packets arriving at your listener as messages to your queue. Writing packets to SQS allows delayed handling for moderate to high rate workloads, decoupling packet receipt from processing and enabling horizontal scaling of your backend systems. Standard, FIFO, and Fair queues are all supported with specialized configuration options for ordering and deduplication.

Technical Details

UDP Gateway uses the sqs:SendMessageBatch API to efficiently write packets to your queue. This batch API allows up to 10 messages to be sent in a single API call, reducing costs and improving throughput when handling high packet rates.

Important: While UDP Gateway batches packets for efficient transmission to SQS, each SQS message contains exactly one packet. When your application receives messages from the queue, each message body will be a single JSON packet object, not an array. This design ensures that individual packets can be processed independently, retried separately if processing fails, and distributed across multiple consumers for parallel processing.

The message body contains the complete request packet with all metadata including source/destination addresses, arrival time, and the UDP payload data. Your queue consumers can process these messages using standard SQS polling techniques, Lambda event source mappings, or any other SQS-compatible processing framework.

Message Format Options

The SQS destination supports two configuration options that control how packet data is formatted in queue messages:

SendRaw

SendRaw (boolean, default: false) - When set to true, the raw binary UDP payload is sent as the message body without JSON wrapping. When false (default), the message body contains the full JSON packet object with metadata.

UseMessageAttributes

UseMessageAttributes (boolean, default: false) - When set to true, packet metadata (source IP/port, destination IP/port, timestamp, etc.) is sent as SQS message attributes instead of being included in the JSON body. This option is only meaningful when SendRaw is false.

FIFO and Fair Queue Support

SQS FIFO queues preserve the order of messages and provide exactly-once processing, making them ideal for scenarios where packet ordering matters. AWS Fair queues provide fair processing across message groups with improved throughput compared to FIFO queues. UDP Gateway provides specialized configuration options for working with both queue types.

Message Group ID

Both FIFO and Fair queues require a message group ID. For FIFO queues, the message group ID provides ordering guarantees within each group. For Fair queues, it enables fair processing distribution across groups. UDP Gateway supports two approaches for specifying the message group ID:

• Default Behavior: When no MessageGroupIdExpression is configured, all packets use the same message group ID, ensuring strict ordering across all packets (FIFO) or treating all messages as a single processing group (Fair).
• Dynamic Message Group ID: Set Arguments.MessageGroupIdExpression to a binary range expression (e.g., "[0:4]") that extracts bytes from each packet's payload. Optionally specify Arguments.MessageGroupIdFormatter to control how the extracted bytes are converted to a string—options are "hex" (default), "ascii", "utf8", or "base64". For FIFO queues, this enables ordering within logical groups while allowing parallel processing across different groups. For Fair queues, this ensures fair processing distribution across different groups.

Message Deduplication ID

FIFO queues support content-based deduplication to prevent duplicate message processing. UDP Gateway allows you to dynamically extract deduplication IDs from packet payloads:

• Content-Based Deduplication: The SQS Destination will always set the MessageDeduplicationId which overrides the content-based deduplication setting on queues. If you want to deduplicate messages based on their full content, set the MessageDeduplicationIdExpression to the full binary range expression "[:]".
no additional configuration is required—SQS will automatically deduplicate based on message body content.
• No Deduplication: If the Destination's MessageDeduplicationIdExpression is not set, a random value will be assigned to the MessageDeduplicationId to effectively disable FIFO deduplication while maintaining strict ordering. This can be helpful for protocols that include content-identical messages like heartbeats.
• Dynamic Deduplication ID: Set Arguments.MessageDeduplicationIdExpression to a binary range expression (e.g., "[4:8]") that extracts bytes from each packet's payload. Optionally specify Arguments.MessageDeduplicationIdFormatter to control encoding—options are "hex" (default), "ascii", "utf8", or "base64". This is useful when your UDP protocol includes unique transaction or sequence identifiers.

FIFO Example Configuration (JSON)

{
  "DestinationArn": "arn:aws:sqs:us-east-1:123456789012:my-queue.fifo",
  "Arguments": {
    "MessageGroupIdExpression": "[0:4]",
    "MessageGroupIdFormatter": "hex",
    "MessageDeduplicationIdExpression": "[4:8]",
    "MessageDeduplicationIdFormatter": "hex"
  }
}

This configuration extracts the first 4 bytes as the message group ID and bytes 4-8 as the deduplication ID, both formatted as hexadecimal strings.

Best Practices

• Batching Configuration: Set your destination's batching count to a multiple of 10 (the SQS batch limit) for optimal cost efficiency. For example, values like 10, 20, 30, or 40 packets minimize API calls while maintaining low latency.
• Queue Type Selection: Standard queues provide the highest throughput (unlimited) and are suitable for most workloads where ordering isn't critical. FIFO queues preserve packet ordering but have lower throughput limits (300 messages/second without batching, 3,000 with batching). Fair queues provide balanced processing across message groups with higher throughput than FIFO queues.
• Message Group ID Strategy: When using FIFO or Fair queues, consider whether you need global ordering (single message group ID) or can partition your workload (dynamic message group IDs) to improve parallelism and throughput.
• Deduplication Strategy: For FIFO queues, enable content-based deduplication when your UDP protocol doesn't include unique identifiers. Use dynamic deduplication IDs when your protocol has built-in sequence numbers or transaction IDs.
• Permissions: Ensure your IAM role has the necessary permissions for SQS operations, including sqs:GetQueueUrl and sqs:SendMessage.

Example Use Cases

• Rate Limiting and Throttling: Buffer high-rate UDP traffic spikes and process at a controlled rate using standard queues
• Asynchronous Processing: Decouple packet receipt from expensive operations like database writes or API calls
• Fan-out Processing: Use multiple queue consumers to parallelize packet processing across instances or Lambda functions
• Ordered Transaction Processing: Use FIFO queues with dynamic message group IDs to maintain ordering per device or session while enabling parallel processing across different devices
• Integration with AWS Services: Trigger downstream workflows using SQS as an event source for Lambda, ECS tasks, or Step Functions

Example Code

An example demonstrating both Standard and FIFO queue configurations is available in the SQS example in our GitHub examples repository. The example includes CloudFormation templates showing how to configure both queue types with different delivery options.