HumanGov: Developing an Event-Driven and Serverless Python Microservice Triggered by DynamoDB Streams Using AWS Lambda Function and AWS SDK Python Boto3.
This hands-on project was a deep dive into the world of serverless architectures and event-driven microservices using AWS technologies. The central task was to develop “HumanGov,” a Python-based serverless microservice, triggered by data modifications in a DynamoDB table and executed by AWS Lambda functions.
The initial phase involved setting up the AWS environment, which included configuring IAM roles for secure access and initializing the Lambda functions with the necessary permissions. The DynamoDB table was designed to hold governmental data, which, upon being updated, would emit streams that trigger our Lambda functions.
With the DynamoDB and Lambda setup, I proceeded to implement the microservice logic in Python using the AWS SDK, Boto3. This SDK facilitated the interaction with AWS services and allowed the Lambda function to process the DynamoDB streams effectively. The microservice was architected to be scalable and responsive to the data flow in real-time, ensuring immediate processing of changes in the database.
To ensure smooth operation, I incorporated logging and monitoring through AWS CloudWatch, which provided insights into the performance and health of the microservice.
The deployment process involved writing the Lambda function, deploying it to AWS, and linking it to the DynamoDB stream. This required a keen understanding of event source mappings and the execution role’s policies.
The end result was a fully operational, event-driven microservice that could efficiently process data changes and scale on demand. The serverless design meant reduced operational costs and a focus on code rather than infrastructure management.
From this project, my key learnings were the intricacies of serverless architectures, the power of AWS Lambda functions in real-time data processing, and the utility of Boto3 in crafting event-driven microservices. It honed my skills in cloud infrastructure, Python programming, and the effective use of AWS services.
Below are few screenshots:
HumanGov Logo
Dynamodb Exports and Streams
Dynamodb Stream details Turn On
Dynamodb Stream details Turned On
Added 2 records in the HumanGov application for the state of California
Dynamodb table shows the records added
AWS S3 bucket shows the pdf added for the two records
Delete a user from the HumanGov application for the state of California
User deleted successfully
Dynamodb table shows only one record now which is expected
But the pdf for the user still exists in the AWS S3 bucket which is not supposed to be there.
Deleted that pdf object manually
Deleted the pdf successfully
Create a AWS Lambda function to fix the above issue
Use Python 3 to write the Lambda funtion
Lambda function created
Updated the Lambda function to use boto3
Add trigger
Error occurred while adding the trigger as the Lambda function execution role doesn’t have the necessary permissions
Adding permissions to the role
Add permissions
Add AmazonS3FullAccess permission
Add AmazonDynamoDBFullAccess permission
Permission policies successfully attached to the Lambda function role
The Lambda function is now attached with required permissions
Create log Group in CloudWatch
Log Group added
Lambda function is now fully functional with log group also added
Added a new record in the HumanGov application for the state of California
Dynamodb table shows the new record added
AWS S3 buckets shows the pdf for the new record added
Deleting the record that was added now to test if the Lambda function works and if it will delete the pdf file in the AWS S3 bucket
Record deleted from the HumanGov application for the state of California
Dynamodb table has only one record and the deleted record is not showing as expected
AWS S3 bucket shows only one record and the pdf for record that was deleted is also deleted by the Lambda function.
Check the CloudWatch Logstreams
The log events shows that the pdf file is deleted for the record that was deleted from the HumanGov application for the state of California
We can see how AWS Lambda Function and AWS SDK Python Boto3 can be used to solve the issue in an automated way. Amazing!!!