Multi-service data propagation with AWS Lambda Fanout

Multi-service data propagation with AWS Lambda Fanout

A colleague recently posted a question in one of our Slack channels about a customer’s interest in triggering a Lambda function using a Kinesis stream from a separate AWS account and region. I found this to be a provoking question being that Lambda triggers are not actually possible using services that exist in separate AWS regions, let alone separate AWS accounts. This problem made me very curious, and so I took a few minutes to search for ideas and solutions; I ended up finding a couple of interesting options, but none as concise or diverse as what I was really hoping for. I then decided to take a look at a favourite GitHub account, Amazon Web Services - Labs, where I discovered something called AWS Lambda Fanout.

Fanout concept

AWS Lambda Fanout (Fanout) was created to facilitate the need for replicating data from one AWS service to another and/or from one AWS region or account to another using Kinesis Streams and/or DynamoDB Streams as a source. At its core, Fanout is a Node.js Lambda function with supporting resources including a DynamoDB table and an IAM role. To get data to the Fanout Lambda function, you establish a trigger with a Kinesis and/or DynamoDB source. To get data from the Lambda function to desired destination services, you establish targets in the Fanout DynamoDB table for the Lambda function to send data to. Supported AWS service targets include SNS, SQS, IoT, ElastiCache, Elasticsearch Service, Kinesis (Streams and Firehose), or another Lambda function.

The really cool thing about Fanout is that you can simultaneously take data from Kinesis and DynamoDB, and have it sent to a series of targets which you register in the Fanout DynamoDB table. This means that not only can you bring in data from one or multiple sources, but you can also replicate or “fanout” that same data to multiple AWS destinations regardless of region and/or account. This is great for any propagation requirement you might have, whether it be for replicating, messaging, caching, storing, etc.

Below is a slight variation on the architectural diagram provided in the Fanout GitHub repository. The updated diagram below includes the DynamoDB target lookup that the Fanout Lambda function performs prior to propagating data to target services.

aws-lambda-fanout-770

 

Getting started

To get started with Fanout, simply clone the repo @ https://github.com/awslabs/aws-lambda-fanout then have a look at the fanout command to see what high level actions are available; you should see actions such as deploy, register, list, update, activate, etc. The first action to consider is deploy, which will deploy the Fanout Lambda function, Fanout DynamoDB table, and correlating IAM role (non cross-account by default). Once the initial services are setup, you’ll need to register targets to the Fanout DynamoDB table which the Lambda function references for data replication targets. Of course you also need something to trigger the Fanout Lambda function itself, so you’ll want to get setup with a Kinesis stream or a DynamoDB table (in the same account and region as the Fanout function), then add that as a trigger to the Fanout Lambda function.

There are many options available to Fanout and the commands that it supports. I won’t go into detail surrounding these because that would take far too much time, but what I will provide is some information surrounding a test I had run, what the results were, and the caveats that exist for this solution to work effectively cross-region.

Propagation testing

I took some time to test Fanout to ensure it could deliver, and while I have yet to test all supported services, I have been able to render successful tests with the following setup:

Account A (us-east-1)

  1. Created a single sharded Kinesis stream
  2. Created a Lambda function that sends an SMS message via SNS
  3. Set up a Lambda function trigger with the single sharded Kinesis stream
  4. Configured a cross-account IAM role for use with Account B

Account B (us-west-2)

  1. Deployed the Fanout Lambda function, Fanout DynamoDB table and correlating IAM role
  2. Created a single sharded Kinesis stream
  3. Set up a Fanout Lambda function trigger with the single sharded Kinesis stream
  4. Registered a target in the Fanout DynamoDB table that pointed to the Kinesis stream ARN in Account A. Included the cross-account IAM role ARN with the registration CLI command.
  5. Reviewed the Fanout DynamoDB table to ensure all target properties existed, and so that I could see what/how the Lambda function was looking up.

The goal was to use the AWS CLI kinesis put-record command to send a small payload to the Kinesis stream in Account B, then have the stream trigger the Fanout Lambda function, which would then perform a lookup on the Fanout DynamoDB table, find the target, and send the data to the Kinesis stream in Account A. Because the Kinesis stream in Account A was also setup to trigger a Lambda function with SNS logic, it did just that and sent an SMS message to my mobile device.

Kinesis -> Lambda -> DynamoDB -> Cross-Account -> Kinesis -> Lambda -> SNS

Caveats

A few things worth noting with Fanout:

  1. Fanout is a Node.js 2.7 function, so if your requirements are strictly against Node.js, then you might have to continue your search for an alternative solution.
  2. Non cross-account configurations work out of the box.
  3. A few code updates are required to get cross-account access to work.
  4. Additional IAM role configuration is required for cross-account access
  5. ElastiCache does not work with cross-account access.

Despite the additional work to get Fanout going for cross-account access, when that’s all said and done, the function works consistently and as advertised.

Conclusion

If you’re looking for a way to replicate data using services such as Kinesis, DynamoDB, Lambda, etc., whether it be same-region, same-account, cross-region, or cross-account, have a look at AWS Lambda Fanout, it’s specifically designed for this task and serves it well. Also, If you need to get data cross-account and/or cross-region and you are looking for assistance in getting setup with AWS Lambda Fanout, then hit us up at TriNimbus and we’d be happy to help.

We like to thank the OSS contributors at AWS for their hard work in delivering tools such as AWS Lambda Fanout.

No Comments Yet.

Leave a comment