Implementing Messaging Fanout Exchange for micro-services using AWS Serverless — SNS & SQS
“We have always thought about design as being so much more than just the way something looks. It’s the whole thing: the way something works on so many different levels. Ultimately, of course, design defines so much of our experience.” — Jony Ive
Designing a system that leverages messaging is always a fun thing. The asynchronous nature in design always brings along a baggage of beautiful problems. In the simplest form, designing a message driven system is as easy as throwing a message broker between two or more components and let the communication happen through messages as described in Fig-1 below.
It appears trivial, but these kind of systems are usually challenging to manage and maintain given their operational constraints. Let’s have a quick look at few of the challenges
Imagine a scenario when the message broker is down for few moments and the retires from the producer exhausts. Another scenario when the consumer of message is down and the retries from the broker exhausts. An extreme scenario may be when the message is consumed successfully by the consumer, but the consumer errored while processing the message. What if the messages in broker are piling up? Or when message business duration have timed out in communication before final processing.
These are few of the scenarios that the message driven system needs to ensure messages are processed successfully at their business value. There are many other challenges that pops up on each level of design in an asynchronous systems.
Recently, I have had a chance to develop asynchronous message driven system from ground up for a greenfield project, and the System Architecture threw really beautiful problems at us. Before diving any further lets describe the problem statement.
Problem Statement — Design a system that receive events from some external process, then processes those events, and finally notify to the downstream processes with the following constraints:
Constraint 1 : The downstream processes are designed in a micro service style
Constraint 2 : Every message should be processes only once by downstream services.
Constraint 3: Every message is intended for one and only one of the many different kind of downstream services subscribed to receive notifications.
On a high level this scenario feels like a standard pub-sub problem. But what makes this problem different are the requirements when the published message is intended for only one of the multiple subscribers, and each type of subscribers are working in a micro service situation. A naive approach to solve this problem is to leverage a message publisher that receives the message from external process; processes the message; and finally publish a notification to the topic. Each one of the subscribers are responsible to check if this message is intended for their use and then proceed as described in the Fig-2 below.
This approach will work, but is inefficient as it tends to waste computational resources. Here each subscriber upon receiving the message will have to validate first if the message is intended for them. In some systems this is not a concern. However, if your subscribers have resource constraints like in case of Lambda functions, then the resources consumed to execute single Lambda function just for verifying first if the message belongs to them will play a significant negative role in cost or system performance, especially when operating at scale.
When the publishers are deployed as micro services, it brings additional set of complex challenges, as each one of those service instances have to co-ordinate between themselves to make sure the message is consumed only once. There are tools and strategies readily available to deal with this situation too, but it tends to add unnecessary complexity to business logic.
A better approach in my opinion is to leverage proven techniques such as fanout pattern and use it to develop a message exchange.
Fanning out and Fanning in patterns are very popular patterns of high scale computing. Software that operate at a massive scale heavily employs these patterns in its core systems.
For an instance, Imagine when a celebrity with millions of followers tweets on Twitter, how does their tweet appears on their followers wall in real time given the scale at which thy operate? Companies like Twitter heavily employ fanning out of their tweets to better manage the tweet workloads to perform at scale that probably does not exist elsewhere.
Fanning out or Fan in patterns are not just limited to messaging, but it can be applied to different aspects of software constructions like parallel computations.
In some message oriented middleware such as RabbitMQ , fan out is provided as a standard features in form of fanout exchange.
We did not have RabbitMQ or any other message broker in our stack that can help us directly with message exchange alike features. Kafka was available but it did not fit in the solution here, in the correct way.
Often times engineering teams lean towards Kafka for their message integration needs. But please keep in mind that Kafka is not primarily designed for message integrations. It serves a very different purpose. However, it is still heavily used for message integration now a days, and it will continue as long as it works.
I prefer a more cloud native approach, and during early brainstorming collaboration, I came to learn that AWS SNS provide feature for message filtering or message discriminator in the form of subscription filter policy that can help us operate SNS when combined with SQS as a Fan-Out Exchange. SQS queues are dedicated queues for a particlar type of subscriber.
The idea is simple, messages will be pushed to SNS with a particular attribute in message. This attribute serves as meta data about message. SNS will then look for this attribute and decide the route for this message. The only overhead message publishers have to do is to figure out which downstream service this message is intended for. This was easy in our use-case as the message already contained some meta data to guide the publisher. Now the design described in Fig-2 improves to as described in Fig-3
Adding the message filter subscription and to SNS is very simple. The following terraform script can help create the SNS with message filter subscription.
We need to create and assign a subscription filter policy for the SNS topic . I For more details of using SNS in this context, I would suggest to please follow AWS documentation at link here.
The SQS then needs to assign a subscription filter policy to SNS topic as shown in Fig-4 on left.
Assigning an attributes to the SNS message is shown in code below
Now, SNS topic with use of dedicated SQS will serve as a message exchange while reaping the benefits of Serverless components of AWS cloud.
On a final note, SNS and SQS being managed services aka Serverless really allows us to apply these design techniques with ease. This pattern is serving our current cloud native service architecture with grace and proves to be easy to maintain.
Hope you enjoy this topic & thanks for reading. Please feel free to leave your candid feedback.
