AWS Dojo - Free Workshops, Exercises and Tutorials for Amazon Web Services

Important Note: You will create AWS resources during the exercise which will incur cost in your AWS account. It is recommended to clean-up the resources as soon as you finish the exercise to minimize the cost.

Ordered Message Processing using Amazon SNS FIFO

Amazon SQS FIFO (First-In-First-Out) queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can’t be tolerated. FIFO queues also provide exactly-once processing but have a limited number of transactions per second (TPS).

Amazon SNS FIFO Topic provides strict message ordering and de-duplicated message delivery to one or more subscribers.

Amazon SQS FIFO Queue and Amazon SNS FIFO Topic can be used together to design fan out messages along with ordered processing.

In this exercise, you use SQS, SNS and Lambda function to implement ordered processing of the fan out messages.

Step1: Pre-Requisite

You need to have an AWS account with administrative access to complete the exercise. If you don’t have an AWS account, kindly use the link to create free trial account for AWS.

Step2: Create IAM Role for Lambda

You start with creation of the IAM role which AWS Lambda function uses for the authorization to call other AWS Services.

Login to the AWS Console and select Ireland as the region.
Goto the IAM Management console and click on the Roles menu in the left and then click on the Create role button.
On the next screen, select Lambda as the service and click on the Next: Permissions button.
On the next screen, select PowerUserAccess as the policy and click on the Next: Tags button. The exercise is using power user permission but in actual production use it is recommended to use minimum required permission only.
On the next screen, click on the Next: Review button.
On the next screen, type in dojolambdarole for the Role name and click on the Create role button.
The role is created in no time. The next step is to create the Lambda function.

Step3: Create Lambda Function

You create a Lambda function which logs the messages from the SQS Queue into CloudWatch logs. Using logs, you can see the order in which the messages are processed in the Lambda function.

Goto Lambda Management console and click on the Create function button.
On the next screen, select Author from scratch as the option. Type in dojolambda as the Function name. Select Python 3.8 as the Runtime. Under Permissions, select Use an existing role as the option and then select dojolambdarole (you created in the earlier steps) as the role. Finally, click on the Create function button.
The function is created in no time. Let’s update the code of the Lambda function. Goto the Function code setting for the lambda function and replace the code with the following code below.
```
import json

def lambda_handler(event, context):
    print(event['Records'][0]['body'])
```
`
In the code above, when the function is triggered by the SQS Queue, it extracts the body of the event sent by the SQS Queue and writes to Amazon CloudWatch log. It will help us understand the order of message processing in the Lambda function. Click on the Deploy button to update the function.
The lambda function code and configuration is ready. Next step is to configure SNS FIFO topic.

In this step, you create Amazon SNS FIFO Topic. The messages are published to the topic which are then processed by the Lambda function using SQS Queue.

Goto SNS Management console. Type in dojotopic.fifo as the topic name and click on the Next step button.
On the next screen, select FIFO (first-in, first-out) option. Also select Content-based message deduplication checkbox. Keep rest of the configuration to the default and click on the Create topic button.
The topic is created in no time. Make note of the SNS Topic ARN as you need it in the next step.
The next step is to create the FIFO SQS Queue which is used as subscription for the SNS FIFO topic.

Step5: Create SQS FIFO Queue

In this step, you create Amazon SQS FIFO Queue which works as subscription for the SNS topic. The queue also triggers the Lambda function to process the message.

Goto SQS Management console and click on the Create queue button.
On the next screen, select FIFO option. Type in dojoqueue.fifo as the queue name. Select Content-based deduplication checkbox.

On the same screen, in the Access Policy section, select Advanced option. In the policy statement, add the following statement. The statement provides SNS Topic rights to send message to the queue. In this statement, replace 111111111111 with the AWS Account Number of your AWS Account.

Amazon SQS

{
  "Effect": "Allow",
  "Principal": {
    "Service": "sns.amazonaws.com"
  },
  "Action": "SQS:SendMessage",
  "Resource": "arn:aws:sqs:eu-west-1:111111111111:dojoqueue.fifo",
  "Condition": {
    "ArnLike": {
      "aws:SourceArn": "arn:aws:sns:eu-west-1:111111111111:dojotopic.fifo"
    }
  }
}

Keep rest of the configuration to the default and click on the Create queue button. The queue is created in no time. In queue configuration, select Lambda triggers tab and click on the Configure Lambda function trigger button.
On the next screen, select dojolambda and click on the Save button.
The queue is now configured to trigger the Lambda function to process the messages. The next step is to add SQS FIFO Queue as subscription to SNS FIFO Topic.

In this step, you configure SQS Queue as subscription for the SNS Topic so that when the message get published to the topic, it is passed to the queue and finally processed by the Lambda function.

Goto SNS Management console and open dojotopic.fifo topic configuration. Click on the Create subscription button.
On the next screen, select Amazon SQS as the protocol. Select dojoqueue.fifo as the endpoint. Keep rest of the configuration to the default and click on the Create subscription button.
The subscription is created in no time. All configuration are in place now. In the next step, you publish messages in the SNS Topic and see ordered processing in the Lambda function.

Step7: Test Message Processing

In this step, you publish couple of messages to SNS Topic. The messages are send to SQS Queue in the ordered fashion. The Lambda function then processes the messages in the ordered fashion. You will also send duplicate message and see how the duplicate message is not processed.

Goto SNS Management console. Select dojotopic.fifo topic and click on the Publish message button.
On the next screen, type in 1 for the Message group ID field. Type in this is message1 for the message body. Keep rest of the fields as the default and click on the Publish message button.
The message is published in no time. Repeat step 1 and 2 to publish three more messages (as shown below) with Message group ID field value as 1. One of the message is duplicate.

this is message2

this is message1

this is message3
All four messages are published. You now go to Lambda console to check in which order the messages are processed. Goto Lambda Console. Open dojolambda configuration. Click on the Monitoring tab and then click on the View logs in CloudWatch button.
It will open CloudWatch log for the Lambda function. Click to open the log details.
You can see the log details. First this is message1 was processed, then this is message2 and finally this is message3. You can check the timestamp of these messages to see their order of processing. They are processed in FIFO fashion.
You can see that this is message1 was published two times but it got processed only once because the message was duplicate and SNS has been configured to check content based de-duplication of the messages.
This finishes the exercise. Follow the next step to clean-up the resources so that you don’t incur any cost post the exercise.