What are Web Sockets?

Web Sockets represent a communication protocol that creates a persistent connection between a client (typically a web browser) and a server. Unlike traditional HTTP, which follows a request-response pattern, Web Sockets allow:

• Two-way communication: Both server and client can send messages at any time

• Persistent connection: One connection stays open instead of creating new ones for each interaction.

• Low latency: Messages transmit immediately without the overhead of establishing new connections

Before Web Sockets, developers used techniques like HTTP polling, where browsers would repeatedly ask servers "Do you have anything new for me?". This is inefficient and causes delays. Web Sockets solved this problem by creating a direct communication channel that stays open enabling servers to send data directly to clients as soon as it is available which enables powerful real-time bi-directional communication.

How Web Sockets Work?

WebSocket connections are an instance of an upgraded standard HTTP connection over TCP. When connecting to a web server with the “wss” protocol - the server attempts to upgrade the connection from a standard HTTP web connection to a persistent websocket one.

This process happens in two main phases:

The Handshake

When a client wants to establish a WebSocket connection:

1. Initial Request: The client sends an HTTP request that says "I'd like to upgrade this connection to Web Sockets" and ”I support these sub-protocols”.

2. Server Verification: The server responds with a special code confirming it accepts the WebSocket connection and also confirms the sub-protocol it prefers the client to use.

3. Connection Established: Once the handshake completes successfully, the HTTP connection "upgrades" to a WebSocket connection.

This handshake is designed to be compatible with HTTP infrastructure, so Web Sockets can work on standard web ports (80 and 443).

Data Transfer

After the connection is established:

• Data is sent in units called "frames".

• Messages can be text (UTF-8) or binary data.

• Either side can send messages at any time and messages must conform to the agreed sub-protocol.

• For security, all client-to-server messages are encrypted over the TSL channel created when connecting.

• The connection stays open until either side decides to close it.

It is important to understand that with a Web Socket connection both the client and server are able to send messages to the other at anytime. Any data written to the Web Socket will be transported to the other party. This is where sub-protocols become important and are a critical component of the design process for an application leveraging this technology. The way clients and servers communicate over Web Sockets is defined by the sub-protocol.

What are SubProtocols?

Subprotocols are agreed-upon conventions for how messages should be formatted and interpreted. Think of them as languages that both client and server agree to speak. Sub-protocols are important for:

• Standardization: Provide consistent ways to format messages (like JSON, XML, or custom formats)

• Messaging Syntax: provides the order and meaning of messages and defines any request/response style interactions and how these are tracked. If you are using web-sockets to transfer request/response interactions these need to be carefully thought out and designed so the client and server can correlate related messages.

• Versioning: Allows protocols to evolve while maintaining backward compatibility

• Interoperability: Different implementations can work together if they follow the same subprotocol

During the handshake, the client can say "I can speak these sub-protocols" and the server responds with "Let's use this one." For example:

• A chat application might use a chat-specific sub-protocol.

• A financial application might use a different sub-protocol optimized for market data.

• A game might use yet another sub-protocol designed for real-time player movements.

By eliminating the constant request-response cycle and providing a direct communication channel, Web Sockets make interactive web applications more responsive and efficient, especially when real-time updates are essential.

Where is the Pub-Sub?

The Web Sockets layer itself doesn’t provide any capability or functionality other than handling the client connections and transferring data between the client and server applications that sit at either end of the data pipe connection (Web Socket).

The connections starts as a normal HTTP connection with DNS Lookup and TLS handshake for a secure connection. Once the HTTP connection is complete the client will send a standard HTTP get asking to upgrade the connection to a websocket. It is at this point the client will include a list of sub-protocols it understands. The server provides the client sub-protocols and asks the backend server to authorize the connection. The backend server will respond with the sub-protocol is prefers to use from those provided by the client and whether the connection is authorized to continue.

It is at this point that the connection is now established and either the client or server are able to send data to each other using the selected sub-protocol. The client is able to send a message conforming to the sub-protocol syntax to the server for processing. Periodically the Web Socket server may send keep-alive messages which are documented as part of the Web Socket protocol - these "Ping"-"Pong" messages are a way to ensure the connection is healthy.

Web Sockets, being bi-directional, enables the server to also send notifications to the client. These can occur at anytime and the client decides how to handle these. The sub-protocol determines the structure of data to be sent between the client and server and it also dictates how each actor will respond to messages being sent.

When the client and server communications are complete, either may request to close down the connection and disconnect.

Web Sockets do not provide any functionality other than transporting data between client and server applications, it is the backend server implementation which provides the Publish and Subscribe capability. For this reason it is essential to think about the sub-protocol your applications will use when communicating over web-sockets, without this you will have applications sending garbage to one another and not achieving anything useful.

Considerations for Architects and Developers

Web Sockets provide the core communications capability for realtime data transfer, they handle the establishment of the connection and transfer of data. From a client-server application perspective there are a number of challenges that Architects and Developers need to be aware of when designing and building solutions around Web Sockets.

Connection Management

Unlike stateless HTTP, WebSocket connections create long-lived connections that must be actively managed. Networks are often unreliable with interruptions causing connections to drop. When you build with Web Sockets both the client and server developers must be mindful of the connections and their state - the management required is dependent on the actual features and function of the application sub-protocol being implemented. You will need to consider the following for managing connections:

• Implementing reconnection logic with exponential backoff on both client and server

• Design for graceful handling of unexpected disconnections.

• Consider connection monitoring with Ping/Pong frames to detect "zombie" connections.

• Ensure both client and server can detect "half-open" connections where TCP remains connected but the application level communication has failed.

• Handle abrupt disconnect from clients which can occur through application crashes or browser tab closure or refresh scenarios to ensure you have clean disconnection logic.

Message Ordering and Delivery Guarantees

Web Sockets guarantee in-order delivery on a single connection but lack mechanisms for handling reconnection scenarios. When clients are reconnecting you need to consider the following:

• Implement application-level sequence numbers for all messages to detect missed updates.

• Build acknowledgment protocols for critical messages with timeout-based retransmission.

• Design idempotency into message handling to safely process potential duplicates during reconnection.

• Consider queuing unacknowledged messages for retransmission after re-establishing the Web Socket connection.

State Synchronization after Disruption

Web Sockets do not provide delivery guarantees or a way to rebuild state following a disconnection. When connections drop, there is no built-in way to determine what state was lost. You will need to consider designing a mechanism to deal with these scenarios:

• Design efficient delta synchronization protocols to minimize data transfer after reconnection.

• Consider event sourcing approaches where clients can request specific missed events.

• Build "catchup" mechanisms that determine precisely what updates a reconnecting client needs.

Resource Management and Throttling

Web Sockets lack built-in flow control beyond TCP's mechanisms. This places the burden of resource management and throttling on you as a service provider. You need to consider the following:

• Implement application-level back pressure signals when receivers can't keep up with message volume

• Design rate-limiting systems that account for both message frequency and payload size

• Consider prioritization schemes for different message types during high-load scenarios

• Monitor per-connection memory usage, especially when handling large message fragments

• Implement circuit-breakers that degrade service gracefully under extreme load

Frame Size and Message Fragmentation

Large messages create special challenges with Web Sockets. Web Sockets typically handle a smaller message frame size (e.g. 32kb), which means larger messages are broken up into smaller chunks which need to be reconstituted on the receiving end. You will need to:

• Design protocols that properly handle multi-frame messages, especially during connection instability

• Implement timeout mechanisms for partially received fragmented messages.

• Consider streaming approaches for large data transfers instead of single large messages.

• Test fragmentation edge cases including control frames interleaved with fragmented messages.

Standardized Keep-Alive and Health Monitoring

Since Web Sockets are a bi-directional communication channel, both client and server are responsible for detecting the connection stability and health. Your client and server applications will need to consider some or all of the following:

• Establish consistent Ping/Pong usage patterns across your system with clear timing parameters.

• Implement application-level heartbeats separate from WebSocket protocol mechanisms.

• Design escalating health check procedures that can distinguish between different failure modes.

• Consider how firewalls and proxies may affect long-lived connections with timeout policies.

Request-Response Correlation

Web Sockets provide a message stream with no built-in request-response syntax. These are left to the underlying sub-protocol to design for and consider. In order to ensure your protocol and messaging design is robust you should consider:

• Implement message correlation IDs to match responses with their originating requests.

• Design timeout and retry mechanisms for requests that don't receive timely responses.

• Consider how concurrent operations will be managed without native request multiplexing.

Security Implications

Long-lived connections introduce unique security concerns which must be considered in your design. Think about some or all of the following:

• Implement periodic re-authentication without disrupting connections - tokens expire and you need to refresh them while keeping the connection alive.

• Consider rate limiting to prevent DoS attacks - both connection attempts and message frequency need monitoring.

• Define clear message validation rules in your sub-protocol - validate all incoming messages before processing them.

• Use secure WebSocket connections (wss://) in production to protect data in transit.

• Verify the Origin header to prevent cross-site WebSocket hijacking attempts.

• Implement timeouts for inactive connections to free server resources.

Testing Strategy

Web Sockets require more complex testing approaches due to the challenges already discussed. Think about and consider the following testing strategies to validate your solution:

• Simulate connection failures and recoveries to ensure your designs function as expected, particularly consider the failure scenarios which is where problems and edge cases will occur.

• Test partial message delivery and fragmentation to ensure your application is resilient to network interruptions. Web Socket data occurs in frames which are smaller in size to the overall message - this adds to the testing complexities.

• Verify correct behavior under high message volumes to ensure your sub-protocol design functions correctly.

Protocol Design

The sub-protocol you design will impact on the functioning and reliability of your application. When thinking about the sub-protocol design you need to consider:

• Message acknowledgment mechanisms for critical updates. If you need to guarantee a message delivery - this needs to be a specific design choice in your architecture.

• Define clear message structures with versioning support so that you can support backwards compatibility.

• Consider binary formats (Protocol Buffers, MessagePack) for high-throughput applications.

Designing messaging protocols is not easy, and must be a carefully considered part of your solution design. The sub-protocol should consider many of the elements we have discussed throughout this section, some of them are protocol specific which define the semantics of how the client and server communicate.

Remember Web Sockets don't Pub-Sub!

Web Sockets provide the communication channel, the responsibility for building reliable, scalable systems on top of this communication channel requires significant additional design and development effort. In my opinion success or failure of your application design comes down to the underlying sub-protocol. When implementing Web Sockets as your underlying communication mechanism make sure that you fully consider the impact of HOW your systems will talk - this will determine whether you succeed or fail because Web Sockets don't do pub-sub, your client and server applications do that - Web Sockets is your data pipe between them and everything else you have to build or buy!

To be successful with Serverless, you need to have some understanding of the following:

• Infrastructure as Code

• Security

• Event Driven Architectures

• Available Managed Services

• Code Frameworks

• Software Engineering

• and the List goes on ... !

So I have created a new blog site for all my future writing based around smaller, focused articles that are interlinked and weave together to create a complete picture of each topic. The articles are organised into focused groups and will be linked so people can browse through each topic and learn information in smaller doses which will be easier to read, understand and share.

What is Serverless DNA?

The "DNA" stands for "Digital Native Architecture", which is how I see Serverless fitting into the broad world of cloud technology.

Serverless DNA takes the form of Small Blog articles, "Mini-blogs" to create knowledge about a specific Strand of Serverless Technology. Each Strand of knowledge may be linked to one or more other Strands which are related and should also be looked at to help expand and grow your knowledge of Serverless.

This first release on serverlessdna.com is the MVP of the DNA concept for Serverless knowledge - I want to grow this into more than just my blog site but also enable it as a community resource linking in content from around the globe from other contributors to help categories, link and collate the body of Serverless knowledge out there today. Find out more by browsing the About Strand - which is where I will be sharing my public roadmap for the site.

I hope you find this new site useful, and I really look forward to your thoughts and comments on this new beginning for my writing in 2023.

Accenture's technology boot camps are run throughout the year and bring together University students who are interested in technology careers and put them through a 4-day immersive boot camp experience with mentors from the Accenture MyWizard technology team to guide and support them. Students are put together in small project teams (4-6 students) and are assigned a project. Throughout the 4 days workshops are run on Design thinking, Project management and other technology-related topics to help them with their learning and to enable them to instantly apply what they learn in building their projects. To assist with the short timeframe, boot camps since 2021 have encouraged using AWS Cloud and Serverless technology. This is where I come in at the start of day 2 and do a "Serverless Works!" presentation to show them all how easy and fast it is to get started building with AWS Managed Services.

Most students who attend Accenture boot camps have no exposure to any cloud provider, so this workshop is really important to enable them to make a start. I introduce Serverless as a term and why it's such an important technology when wanting to build solutions quickly to deliver instant business value and provide instant scale meaning anything you build quickly is capable of running live with thousands of users with little to no effort or additional knowledge. Following this, I start demystifying the AWS Console - introducing what the cloud looks like and pointing out important services they will likely want to learn more about over the next four days - CloudFormation, CloudWatch, DynamoDB, API Gateway and Lambda. Next, I talk about the tools I use for my live coding demo to show them how easy it is to get started.

This is my list of getting-started tools and utilities for the boot camps:

1. An AWS Cloud Account

2. Leapp application from Leapp.cloud to secure access to their AWS account without needing to store Access Key and Secret Key on their computer unencrypted.

3. SAM CLI for deploying their application to the cloud

4. Visual Studio Code (or another code editor of their choice for editing files)

5. Python installed and working on their computer

6. AWS Lambda Powertools for Python to make writing AWS Lambda code faster and safer.

The above sounds like a daunting list, and by this time, many students are likely to wonder what it is I am talking about, and that's when I kick into a live coding demo and bare my developer soul to them. The solution I build is a simple REST API with AWS Lambda performing CRUD operations on DynamoDB - this is my target architecture, and I strive to get it finished in my workshop timeframe but do not always get there as I do talk a lot, and I get excited by Serverless!

It is amazing how fast it is to build a REST API these days with SAM CLI using the sample project wizard via sam init. With SAM CLI, I always start with the hello world sample application, which provides a nice project structure.

1. Execute: sam init

Within minutes a demo python project using the "Hello World" python template, and I can build and deploy it to the cloud.

2. Execute: sam build

I do this from an empty folder screen sharing my terminal so they can see first-hand how quick and easy it is to use the tools.

3. Execute: sam deploy -guided

While it deploys, I open the AWS Console and show the Cloudformation console and the stack creation, expanding the details to show the resources. I click the quick links to open the resources in their respective consoles to show how to find what is being built and show them in the console so they can build a mental model of a development workflow and visually start linking the consoles together, so they are less mysterious. Let's be honest - the AWS Console is an abstract, mysterious place that is not easy to get your head around when building a solution with so many different services.

Introducing AWS Lambda Powertools for Python

The last part of my live coding is refactoring the "Hello World" API lambda code using AWS Lambda Powertools for Python. I start by quickly importing AWS Lambda Powertools for Python and work through the following steps to refactor:

1. Import Logger and Tracer

2. create instances of these

3. Import APIGatewayRestResolver - While doing this, I explain how similar the utility is to python fast-API and node express libraries since students may be familiar with these from less serverless projects they have built.

4. Rework the Lambda handler to use the APIGatewatRestResolver class

Pretty quickly, we end up with code like this with the API route changed slightly to enable some dynamic responses based on input with a little fun thrown in.

The refactored code can be easily updated and re-deployed:

1. sam build

2. sam deploy

Once deployed, the demo now changes to show how the dynamic routes work, and I start to talk about how in a lot of cases, boot camp projects often involve Web or mobile app front-ends and dredge up everyone's dreaded API pain - Cross Origin Resource Sharing (CORS). So we add in Lambda Powertools CORS handling option (2 minor changes) and, at the same time, throw in how we can add in some AWS Lambda event logging to make debugging applications easier when they get building.

At this point in my demo, students have enough to get started with building and depending on the time I have left, I will keep going to add in DynamoDB and CRUD API methods or stop and open the floor for questions - the detail of building out a CRUD API is less important since by this time I have achieved the goal of my workshop. It is often far more important to engage with the students at this point and enable them to ask questions so I can address any specific concerns from the group

The final slide before I complete my workshop is a complete list of links for all the tools, demos, AWS online workshops, tutorials and sample code links. I do offer up a complete working boot camp demo repo on my GitHub with the complete CRUD API built and ready to deploy to help accelerate the project teams. I don't feel this is going too far because the students have never used AWS before, and this is the first time they are hearing about Serverless. The sample code provides a nice basis for them to get started in something more real, and they can achieve instant success which is important for a 4-day immersive experience.

Why Choose SAM CLI?

I choose SAM CLI because I find this an easy and approachable tool which has a nice wizard-style approach to getting started and makes building and deploying straightforward to get going. I also provide links to AWS Workshops and the first workshop in the list is Create A Hello World Lambda Function (using SAM CLI) so it ties in well and reinforces my live coding demo!

Why AWS Lambda Powertools?

This is simple - because every Serverless project that writes code for Lambda should use this library. It provides AWS Serverless best practice principles right out of the box and with very little effort. I encourage every serverless team I work with to use AWS Lambda Powertools, and I also strongly push Python as the language they should be building with. This utility library does enable Serverless teams to run faster by providing standard approaches to Logging, Tracing, Efficient Metrics, Resolvers for APIs (all types) and Stream batch handlers which make writing SQS handler functions safer and more robust - so many AWS Well-Architected principles are embodied in AWS Lambda Powertools. It simply makes coding faster.

Seeing project teams embracing Serverless technology with a single workshop presentation is refreshing to experience. I love presenting my Serverless Works! Boot Camp edition for University students attending Accenture's Technology boot camps. I am also addicted to writing code in front of developers to showcase new techniques or ways of building. Sharing my knowledge and love of Serverless building and doing it immediately with instant effect is very satisfying.

Serverless Testing, Part 1: What I forgot at the beginning starts with what I forgot when I started building serverless solutions. It encourages bringing back software architecture principles to your Serverless code and focuses on the Dependency Inversion Principle - something all developers should understand.

Serverless Testing, Part 2: SOLID Architecture for Lambda covers SOLID software architecture principles and how it applies to Serverless. It also introduces how to implement Hexagonal Architecture using software interfaces and adapters as a software abstraction to isolate your business domain code from the details of interacting with the cloud.

In this final piece of the puzzle, I will cover how implementing your business domain logic with Hexagonal SOLID software architecture principles will simplify testing of your code and do away entirely with mocking AWS SDK calls to the cloud. This article focuses on unit testing, integration testing and end-to-end tests for the example architecture presented.

Testing Serverless systems is not a "now we have to simulate the cloud problem".

Testing of code destined for the cloud is a confusing space with many opinions and tools being built to support all manner of local, remote, semi-remote and local-remote testing of your code. This article is born from my continued frustration with developers wanting to replicate the cloud locally so they can run their entire solution end to end on their desktop. This is more a desire to continue developing the way they have always developed rather than looking to change development practices to suit the changing deployment landscape that Serverless brings to us.

A Quick word on "Mocking"

A frustration of mine with Serverless developers is watching them waste time mocking AWS SDK Calls. Mocking AWS SDK calls is never straightforward and usually requires several additional libraries to make it an effective technique. Combined with this you may have junior or less cloud-experienced developers on your team who struggle with mocking AWS Calls, mainly because they don't understand the nuances of the API call responses.

Please do not mistake this to mean that I think mocking is something we should eliminate. Mocking is vital for good unit testing and is something I strongly encourage and is needed. However, not for AWS SDK Calls - we can use better software architecture to remove the need for mocking AWS SDK calls. In the remainder of this article, I will explain how better software architecture will improve and simplify your Serverless testing activities and enable your team to move faster.

The GitHub Example Project

This article is supported by an example python lambda project that applies the testing techniques discussed in this article; You can find it on my GitHub here. This article does not go into every aspect of this project, it is used only as a reference to the testing strategies introduced within this article.

A Few Notes on the Project

• The sample project is focused on general Serverless testing for synchronous API services with some asynchronous components. It does not cover all the possible testing scenarios for serverless and is intended as a general example to get started with Serverless testing more quickly and for a majority of Serverless use cases.

• The README.MD file has notes on the project structure and elaborates on the directory structure and how the project is built.

• AWS CDK is used to deploy the code to your Cloud using the Python language

• The Python folders are structured in a very specific way to ensure the lambda code directory supports the PythonFunction CDK construct which uses docker to package the code to ensure binary packaged dependencies will function correctly when deployed.

• Minimal scripting magic for building and deploying the project working within the existing folder structures at all times. This minimises the need for build magic where some developers build scripts to move source files around for lambda packaging.

• Relies on poetry for packaging and uses a specific convention for installing dependencies in groups for each of the lambda functions present in the project. This will ensure dependencies are packaged only where required and will ensure faster start-up times which is important for AWS Lambda.

• Uses the AWS Lambda Powertools for Python. No other reason for using this library other than all projects should. If you don't know what it is click the link or check out my talk from Melbourne Serverless Meetup: What I love about AWS Lambda PowerTools for Python.

• Pytest is used for unit testing and for integration and end-to-end test modules tear up/down mechanisms are used to create ephemeral stacks for testing. I borrowed this testing technique from the AWS Lambda Powertools for Python project - I do think you should check it out if you haven't already!

• The code within the GitHub project is not a production-ready sample, so use it with caution and modify it to meet your production system requirements.

Better Architecture with Hexagons

I wrote about the Dependency Inversion Principle, SOLID Principles and Hexagonal architecture in the previous parts of this series. Hexagonal architecture is an architectural pattern that splits your system into distinct components (not layers) and introduces a combination of Ports and Adapters which separate your business domain logic from external resources such as databases and cloud services like API gateway, Eventbridge, SQS and DynamoDB. The hexagon shape itself is unimportant and is used to define where the adapters exist within the software. In Alistair Cockburn's original article, the primary motivation for this pattern was to remove the bleeding of functionality between software layers in more traditional, monolithic solutions which countered all the advantages of cleanly layered software with isolation boundaries. By introducing smaller, more specific interfaces used by business logic to achieve a business outcome, the abstraction to external services is more clearly defined and less ambiguous using the Hexagonal architecture idea.

I like the way this architecture is laid out as it helps to quickly understand what are the inputs, where the processing is and the outputs which maps very clearly to how AWS Lambda code works in the cloud.

On the left are the driving adapters for the AWS event sources that drive the event data into our business domain logic through the defined ports (interfaces). On the right are the external cloud resources driven by our business domain logic - where data is stored or pushed for further processing. Our business domain logic interacts with these driven services through interfaces where the adapter implementation makes AWS SDK calls to interact with the cloud resources.

Testing: Unit, Integration and End to End

Combining all these elements can simplify the testing of business domain logic and remove the need to create Mocks for AWS SDK calls. Testing is split into three main areas - Unit, Integration and End to End testing. Unit testing covers the central Business Domain logic, integration testing covers the driven ports side of the hexagonal architecture while the end-to-end tests start with the driving side and continue across the entire architecture.

Your business domain or Service logic is covered by unit tests. With hexagonal architecture, any AWS SDK calls will be isolated by the adapters, behind the ports meaning there will be no AWS SDK calls within your business domain code, so there is no need to introduce AWS SDK mocking libraries. This will simplify your testing and give your developers more time to focus on delivering more value.

The interfaces and adapters are tested using automated integration and end-to-end tests run when your code is deployed to the cloud using CI/CD automation tooling. With tests running in the cloud, we don't need to mock AWS SDK calls since we will be testing with real services. Combining these activities also removes the notion that you need to run the entire solution locally in a mock cloud environment on your desktop so you can set break points in your IDE to help catch errors or see the state of variables.

You can achieve local debugging by running your unit tests in debug mode, fully exercising your business logic, and this is a workflow I encourage all serverless teams to adopt. All you need to do is craft one or more test fixtures for specific test cases and add these as unit tests to your existing test cases. You can execute tests using your IDE in debug mode and set up breakpoints in the exact spot you want to step through. This not only enables local testing with no mocking and no cloud simulation but also ensures you expand your test cases with additional cases that are more real and not just there because we need test coverage. With SOLID Hexagonal architecture in place if you interact with reading data from a database - this will happen in a Port and Adapter. So you can create a fake adapter to return the data your test needs to create the correct outcome.

A Reminder: The Core Principle of Lambda

As we gain more confidence in designing and building serverless systems with more mature managed services on the cloud, we are starting to create complex, distributed systems that scale, run fast, and efficiently. As we travelled to this point, learning about how each of the AWS primitive services work; we have forgotten the principle of how Lambda functions are intended to work.

In a nutshell, Lambda functions exist to add value to data events entering our system and then forward the result to the next component in the solution chain. This next component could be anything and may involve other managed services such as AWS SQS, SNS, EventBridge or API gateway. At the end of the day, to unit test our business logic, we need to pass a test data event and check the data forwarded to the next component in our solution chain after our business domain code has run. Doing this does not involve mocking AWS SDK calls, nor does it require us to run a mock SQS service locally so we can test end to end.

Testing Serverless Architectures

Let's look at an API gateway interface that stores events into an S3 bucket and then pushes a transformed event into an SQS Queue. The data pushed to the SQS queue is processed by another Lambda function and is inserted into a DynamoDB table. A classic serverless pattern that is widely used as a store and forward mechanism.

Taking the above architecture, we can isolate and identify the testing boundaries to simplify the creation of unit tests and eliminate mocking of AWS SDK calls altogether. Let's overlay the hexagonal architecture concepts of ports, adapters and business logic to show how our Service logic is encapsulated.

From the architecture overlay, everything within the Lambda symbol represents the unit test boundary; The outbound arrows from each lambda box represent the integration test boundary (2, 3 & 5) and the entire numbered flow (1 - 5) represents the end-to-end test boundary. The overlay also emphasises the core principle use of lambda: Data Event in (1 & 4) + Lambda Processing Logic = Valuable Business Outcome (2, 3 & 5).

Unit Testing Lambda Code

Using the above boundaries enables you to define where code should live. If we adhere to the hexagonal architecture principles outlined in my second article - Our Service logic will be contained in a service class (or function) which is called by the code in your actual lambda handler. The lambda handler function represents the adapter portion of data event input at step 1 and all it will be doing is validating the input and mapping it to the ports (interfaces) of your service class.

The following code snippet is taken from the GitHub project and represents the first lambda handler function which is triggered by an AWS API REST Gateway event. The project uses the AWS Lambda Powertools for python to assist with handling the AWS Lambda event (I did add a few comments in the code and moved the handler for clarity).

The post_event function which handles the API route for the inbound data maps the API body details to the process_event method of the event_service class. This is a nice way of isolating the different software components that make up your lambda handler and will ensure each of the components are individually testable. This ensures unit tests are small, concise, and easy to understand. This also enables your service logic to be re-used in other AWS Serverless Service contexts without making any changes and means faster development time and less change from a unit test perspective if your solution needs to pivot or change, which is often the case as requirements and volumes change.

Using better software architecture will simplify your Lambda handler function code - don't put everything in your lambda handler unless it is super simple. Any specific business logic should be isolated within a service class or function so that it can be transportable, easier to test and quicker to understand.

The unit tests for this API service are focused on the EventService class which is defined in the adapters folder in the service.py file and represents the central blue section on the hexagonal architecture diagram earlier. Unit tests are about testing our business domain service logic, not testing the Lambda handler function. The EventService is responsible for 2 actions - storing the event in an AWS S3 object and pushing the event data into an SQS queue for processing by another Lambda function. The Event Service uses Hexagonal architectural techniques and uses 2 ports - FileStoragePort and MessagePort. To unit test the EventService we will need to provide fake or mock implementations for these two ports, mocks are still used but we are using specific mocks which do not require AWS knowledge or external mocking libraries.

This diagram shows how our EventService logic calls the FileStoragePort (Fake S3 adapter) and MessagePort (Fake SQS Adapter) which are both replaced by fake implementations to support unit testing. The complete unit test is shown in the test_event_service.py file in the sample project. The constructor of the EventService allows for overriding the adapter implementations that are used which enables fakes to be used for testing purposes where needed. This is a classic Dependency Injection technique where dependencies of a class (or function) are provided during creation so that implementation can be changed when desired.

Some people will be thinking - what about unit testing the Lambda handler function? In this project have opted not to do that, sure it added to unit test coverage but the lambda handler code is very simple and will be tested as a part of the integration tests we go through next. I have some strong opinions when it comes to test coverage and feel that many hours of developer time are wasted on creating tests that deliver no purposeful quality improvements. If the code is critical to your business function you should be writing unit tests. If the code adds no real business value and is acting as an enabler or bridge to your business function then you need to decide as a development team if testing is useful and meaningful.

A Word on Code Coverage Metrics

Code coverage percentage metrics matter for some projects, but not all. In my experience, attaining a goal metric for code coverage with automated testing adds very little to the overall quality of the software you deliver and does not add to an equitable increase in confidence that the code is correct. I believe it wastes valuable developer time and reduces quality because the team are no longer focused on delivering your business value, they are focused on making the code coverage percentage get to the right value or exceed it. In some cases, they will create tests that are invalid just to achieve the metric goal or will focus on the low-hanging fruit and miss out on tests on critical business functions. This is why I do not like mandatory code quality metric targets - they do not add value or increase quality, they just consume development time for no business value.

Automatic tests need to add to your overall product quality and be valuable.

I believe code coverage is vital when creating shared code libraries that are used by more than just your project or that are shared publically or within your organisation, these should strive for as much test coverage as possible since the testing adds value and confidence for all users of your library.

Integration Testing Example

In the sample project on GitHub all of the integration tests can be found in the tests/integration folder with subfolders for each service to be tested. We will remain focused on the event_api service for integration testing.

For integration testing, we are deploying actual AWS Cloud service components which will be tested in the cloud to ensure they integrate correctly. The actual deployment is handled by the pytest module conftest.py which will ensure the components are deployed before executing the integration tests and remove the Cloud Formation stack when the test execution is complete. The Integration test code will use one or more test fixtures representing the real AWS API Gateway event the lambda code will receive when integrated with the API gateway. The test we are writing is focused on the outbound arrows in our architecture (2 & 3) and nothing else. The test infrastructure will deploy the Lambda, S3 bucket and SQS queue and then test the lambda through a synchronous execution using the AWS Lambda execution API call.

The idea is to break down your integration tests into small chunks that can be completed synchronously, which makes writing tests for asynchronously triggered Lambda calls (e.g. triggered by AWS SQS) much simpler and easier to deal with since we no longer need to poll for results, this is why we set up integration tests in this way. The following example shows the complete integration tests for the Event API component triggered by the AWS API gateway. With our lambda and resources deployed to the cloud we are now testing our AWS SDK api calls within our adapters and will be sure that if these tests pass, they will continue working as we expect.

Testing AWS SDK API calls in the cloud means you are testing all the cloud integration including configurations such as IAM Roles and Policies along with your actual Adapter code.

Lines 17 - 30 outline test fixtures which retrieve actual cloud resource details required for the AWS SDK calls we will make to interact with these services. When the test calls the AWS Lambda execution API passing the AWS API Gateway test fixture (line 39) it is a synchronous call so when we receive the response we can be sure all integrated actions have been completed so we will be able to immediately check the S3 bucket for the desired outcome (lines 53 - 56) and check the message pushed to the AWS SQS queue is correctly formatted for the next service to consume (lines 58 - 67). This makes testing simpler and more easily understood and reasoned about by developers of all levels. The idea of running tests in this way is to enable your team to move faster and write smaller, more concise integration tests.

End-to-End Testing Example

Testing Serverless Architectures end-to-end is probably the most difficult part and has the most obstacles - especially when one or more asynchronous services make up your solution. In the sample project and testing, I have tried to keep all testing straightforward using synchronous mechanisms to minimise testing complexity and developer cognitive load. Given we have tested the event API Lambda code in our integration testing using a real AWS Gateway event and we know it correctly interacted with downstream services the end-to-end test shouldn't need to test these components again.

Designing tests to execute synchronously allows your developers to move faster and your testing is less complicated and easier to understand and reduces false positive results for asynchronous polling

So the end-to-end component is all about deploying the API gateway and the API lambda code along with dependent downstream services. This means we can limit the testing to the API boundary and ensure the API gateway integration works correctly by giving us the response we are expecting according to the API contract. You could consider this an API contract test as well and that is really what we are doing in this example and is what the end-to-end component is really about - does the system behave correctly for external users.

Applying the End-to-End as shown in the above architecture diagram means we progressively test each of our components in the cloud in a synchronous fashion. The final part is to test the API gateway triggering Lambda and that we get the correct response for each of our test cases. In our example, we have a fairly simple API and only a single test to illustrate the strategy.

The end-to-end test uses the same fixture interface to get the real API gateway address for the deployed API gateway infrastructure.

I have focused my testing examples around an API gateway-bound service design which is one of the more common serverless system implementations. It is by no means the only type of implementation and trying to go into more detail about every aspect of Serverless system testing would take far more than I have written here. I have focused on a specific example as a means to help you understand how you can achieve automated testing of a Serverless Architecture in a synchronous fashion and without mocking AWS SDK Calls. The idea is to be able to apply testing that makes sense for your projects and enable your developers to move quickly and empower new developers to also get going fast!

Through the use of SOLID Hexagonal software architecture and synchronous testing techniques in the cloud for integration and API Contract testing, you can remove the need for using AWS SDK development mocking libraries and empower new developers to get started and understand your business code faster and easier.

GitHub Code Disclaimer

The examples presented are focused on the Python programming language and the code presented uses object-oriented programming. Python is not the only language for coding with AWS Lambda and the solution samples may not suit your preferred language.

The code presented is not considered production ready and should not be used in production without ensuring it meets your development standards and code quality requirements.

Further Reading

I strongly recommend reading Yan Cui's blogs and also checking out his new course on Serverless testing which is in development (links below).

• Testing Serverless Architectures

• Talking testing with yan Cui - Serverlessfirst podcast

• Yan's blog - theburningmonk.com

Know the Services you are using; really know them. Understanding how they work and fail is essential for constructing a resilient serverless architecture.

This is the number one piece of advice I share with new Serverless Engineers. Knowing the AWS Services you plan to build with is essential. A vital aspect of this is how your Lambda is invoked and what this means. Lambda execution models can be broken down into three groups, Synchronous, Asynchronous and Event Source Mappings.

Synchronous Invocations

This is the simplest execution model - this is where all parties in the lambda execution chain are involved end to end in the transaction and wait for a response. In this model, error handling falls back to your consumers since failures are communicated directly via the lambda or service response. This means you consumers will be responsible for any retry behaviour on failure.

A common pitfall with synchronous invocations is your lambda timeout configuration which can be up to 15 minutes. You need to be careful setting this limit since there is no guarantee your clients will wait this long for the lambda execution to complete. The most common timeout error for synchronous invocations is when lambda is triggered by the API gateway which has a 29-second hard timeout limit. You need to make sure your lambda timeout configuration makes sense within your overall solution.

Since error handling falls back to consumers, the synchronous lambda invoke model is the simplest to code for. Being invoked synchronously means you must carefully consider rate limiting and throttling within your overall solution so that your lambda does not get over-run and cause problems for downstream services.

Asynchronous Invocations

Asynchronous invocations are where the caller of your Lambda does not wait for the outcome. They trigger the Lambda and then go on with whatever else they need to do; all they care about is whether the triggering of the Lambda was successful and nothing else about what your Lambda code is doing. Any errors from your Lambda invocation go nowhere since nobody is waiting for the result. Asynchronous invokes are great for enabling greater scale within your solutions at the expense of adding additional complexity when it comes to handling errors. With this invocation model, it is the responsibility of your Lambda code or configuration to take care of all error handling.

A common mistake serverless developers make with AWS Lambda and the asynchronous invoke model is expecting when their Lambda fails that the service performing the invocation will actually know something about the failure. This will never be the case - the service performing the invoke will only know about the success or failure of the actual invocation action, which returns immediately the Lambda function is triggered successfully. This is a common misunderstanding of how the Eventbridge Dead Letter Queues work - these queues will only capture the Eventbridge events where an asynchronous invoke of the target from the Eventbridge rule failed (after 24 hours of retries) not when the lambda code invoked returns an error response - subtle difference but an essential lesson in knowing how the AWS services you are using work!

AWS Lambda has introduced several nice features around the asynchronous event model to assist with lambda execution failures. By default, an asynchronous Lambda invocation allows for up to two retries on Lambda execution failure with some backoff and delay mechanisms you can configure. This gives you a very simple retry mechanism out of the box. You also have the option to configure a Dead Letter Queue for asynchronous invocation failures when the two retries ultimately fail to execute your code successfully. If this small out of the box retry and dead letter queuing is not enough, the Lambda service also introduced Lambda destinations for the asynchronous invocation model only.

Lambda Destinations

Lambda Destinations is a great feature that allows you to configure several destination types on either success or failure of the asynchronous invocation. You can forward success or failure events to multiple destinations: SNS, SQS, Lambda or Eventbridge Eventbus. This provides you with many options for dealing with failures for your asynchronously invoked lambdas. With Lambda Destinations the event content provided to the destination contains details about the request and response in JSON format, this differs from Dead Letter Queues, where Lambda only sends the content of the event, without details of the response.

Event Source Mappings

The Event Source Mapping is an AWS provided polling mechanism for triggering Lambda from a stream or queue which are not capable of invoking Lambda directly. This mechanism causes Lambda functions to be invoked synchronously with batches of records which can be configurable in size depending on the event source. As we have learned, the caller in the synchronous invoke model is responsible for handling any error response, and the Event Source mapping is no exception. The Event Source Mapping will handle errors in specific ways depending on the AWS Managed service being used - I recommend you read up on how each service behaves with Event Source Mapping failures - another example of knowing your services!

So the funny thing with Event Source mappings is that the lambda execution mode is a synchronous call, even though from a serverless solution perspective the interaction between the AWS service and callers is asynchronous in nature, i.e. my Lambda execution response returns to the AWS managed service and not to the caller or consumer. This to me behaves in an identical fashion to the asynchronous invocation model and I really wish that Lambda destinations were an option for this "asynchronous solution execution model", I get the invocation is synchronous, but from an overall serverless solution perspective, it isn't. I really think that Lambda Destinations for this invocation model being available would open up more flexibility for handling events.

AWS recently introduced filtering to Event Source mappings enabling messages from the source to be specifically filtered for your Lambda function - this is an amazing addition and I encourage you to check it out!

In this article, I have summarised the Lambda invocation models that every Lambda developer must know and understand to be able to successfully develop and deploy to the cloud.

Other Useful References

1. Understanding Different ways to Invoke Lambda Functions

2. Invoking Lambda Functions

Serverless Testing, Part 1: What I forgot at the beginning describes the start of my serverless story where I forget about plain old Software Engineering Principles, and I re-introduced the Dependency Inversion Principle, which is essential when writing software! I want to explore software architecture for serverless functions in this second instalment. You will learn about SOLID principles and we will break down Hexagonal Architectures and how these ideas work together to simplify building Serverless code.

In my experience, many teams who are moving into Serverless development are from a heavy infrastructure background and are less experienced in application development using software architecture principles. So I like to start by going back to basics and talking about SOLID principles of building software.

Single Responsibility Principle

A function should have one reason to change

Functions should have a single responsibility which is the only reason it should change. The more responsibilities your code has, the more reasons you will have to change it, and the more complex your functions will become over time. Keeping functions simple has several key advantages - it lowers code maintenance costs and makes them faster to execute. However, keeping functions simple is not as easy as it sounds and is part of the distributed design that causes friction in some teams who feel that serverless systems are complex to build or have too many moving parts. This aspect of serverless is the hardest to deal with and needs a disciplined approach to keeping your solutions simple.

Open / Closed Principle

Software entities (classes, modules, functions, etc.) should be open for extension but closed for modification.

This software principle means that classes should be open for extension but closed for modification meaning the inner working of a class or function should not need changing when a dependent object or class changes. You can achieve this through interfaces that extend your code's inner working by introducing a new implementation. Using interfaces in this way enhances the loose coupling of dependencies.

I also like to apply this to serverless functions when you need to write code. I always try and make my functions data-driven and configurable so that the processing relies on only the event triggering the function. This principle applied to serverless is about not changing your original service to deal with specific behaviour in a specialised context. Instead, look to make these exceptional cases an option or feature flag and encourage composability to add additional processing.

Liskov's Substitution Principle

Overridden class methods must have the same input parameters as their superclass. So, if it looks like a duck and behaves like a duck, it is a duck, and if you want a new duck, make sure it quacks the same as all the others.

Liskov's principle states that if you use a subclass in place of its parent class anywhere in your software, nothing will break. In simple terms, parent classes define a contract that sub-classes should never alter - violating this principle means clients who try to use this sub-class will no longer work as expected without being modified.

In serverless, a new version of your service should always be able to replace a previous version without breaking anything. This also puts our design focus on service consumers or customers who do not need to change their code when a new version is deployed, which would be considered a breaking change for the Interface.

Interface Segregation Principle

No code should be forced to depend on methods it does not use

The Interface Segregation principle keeps your defined interfaces simple and specific to their purpose. Keeping interfaces clean is essential; otherwise, every implementation is forced to implement functions rarely used, which is a waste of effort and increases software complexity.

This principle also helps in serverless designs by ensuring that we design simple, single-purpose interfaces for our services. A service that does too much will expose too many details and may leak internal implementation details to consumers, which is always a mistake for any microservice architecture. Keeping interfaces simple and to the point is what this principle is about, and we should be strong in applying it to every service boundary we design.

Dependency Inversion Principle

1. High-level modules should not depend on low-level modules. Both should depend on abstractions.
2. Abstractions should not depend on details. Details should depend on abstractions.

Dependency Inversion is all about creating functional abstractions between different layers of our code. This principle enforces the use of interfaces for every dependency, enabling the simple replacement of any component in our code quickly and easily. To embrace this principle means when designing an interaction between a high-level module and a lower level, one should be created with the interactions in mind, which will dictate the overall interface design. In this way, the coupling of our software components is reduced, and we can introduce new implementations at any time, which is ideal when we have embraced all the SOLID principles explained here.

The serverless use-case for this principle is really in building out strong software domain abstractions around cloud services to isolate business logic from the complexity of the cloud services it uses. This was the main point I introduced in Part 1 of this series.

Hexagonal Software Architecture

Hexagonal Architecture is all the rage in the Serverless world and has been written about by many people:

Ready for changes with Hexagonal Architecture on the Netflix TechBlog by Damir Svrtan and Sergii Makagon

Developing evolutionary architecture with AWS Lambda on the AWS Blog by Luca Mezzalira

Service Ports: Finding a Loosely Coupled Utopia with Event-Driven Serverless on Serverless Transformations by Ben Ellerby

These articles describe the benefits and tell you why you should be using hexagonal architecture. They all talk about Ports and Adapters as the magic you need to isolate your business domain logic from the more specific lower-level code that deals with writing data to databases, file systems or API interfaces. I don't like to use "hexagonal" or "ports". Instead, I prefer to talk about software patterns.

Building your software using this concept is nothing more than embracing the Dependency Inversion Principle from the SOLID principles we have already discussed. In hexagonal architectures, Ports are nothing more than well-defined Interfaces clearly defining your business domain interactions with external dependencies. Adapters are specific implementations of these business domain interfaces (Ports) to enable the de-coupling of your business domain logic from the cloud environment you deploy to. Adding dependency injection into the mix at this point is the real magic; Enabling injection of Adapters into your business domain logic allows a lot of freedom in building and testing your code.

I strongly feel there is a lot of effort focused on developers mocking SDK elements of AWS and other cloud providers. A lot of time is wasted trying to work out how to mock SDK responses from SDK API calls. By adopting business domain Interfaces and implementing them with Adapters, you can instantly create mocks using a mock implementation of the Interface for local testing purposes. This means your business domain developers do not need to know about the AWS SDK dependencies. This can be locked away within the Adapter implementations, which can be handled by framework developers who can be specialists for your cloud environment. Combining Adapters with simple dependency injection allows your adapter dependencies to be injected, meaning business domain code does not need to change whether you are unit testing, integration testing, or deploying to the cloud. You will have adapters for each of these environments and can drive the injection of these dependencies when needed so all your business domain logic can be quickly and simply unit tested.

A huge benefit of adopting a Hexagonal architecture is to isolate your business logic from your cloud and serverless components that make up your application. This means you can now have some separation of responsibilities within your team - Experts on your cloud integration and experts on your business domain logic. With this isolation, you can have less cloud experienced programmers focus on business logic and more deeply cloud experienced developers on Cloud integration - this separation of concerns is powerful.

The Road to Hexagonal Applications - Service Oriented Architecture (SOA)

Clean hexagonal architecture comes down to software engineering principles we have discussed throughout this article, and I will throw in one more term - Service Oriented Architecture (SOA). Wrapping our business domain logic into Service classes creates a complete separation between your cloud environment and application logic; it also becomes a natural break between cloud expert developers and business domain developers - something I have been searching for to create layered teams with specific skill-sets across cloud and the business domain.

Using Services as the core base also allows direct and straightforward dependency injection of the various dependent interface adapters (Ports) required for the service to function. The following code snippet shows an example of a base abstract style python class for a Service Interface (Port). It accepts several other adapters for core functionality required, including Logging, Notifications and Storage. Through dependency injection for domain adapters - the business domain logic is now wholly isolated via the Dependency Inversion Principle I mentioned earlier.

Adding Business Functional Adapters (Ports)

In the Service example, several adapters are injected as dependencies. The following code sample shows how you can structure the notification adapter (Port), which has a business-focused method notify_new_user that is functional. In this way, we separate the detail of the notification implementation from our domain programmers. This also means we can pivot the actual implementation without changing business domain code - so from a SOLID perspective, we are now embracing the Open-Closed Principle and the Dependency Inversion Principle - two powerful software engineering techniques.

Bringing it all together

The following example shows how an AWS Lambda handler would apply the techniques discussed in this article. An adapter class (UserEventMapper) translates the Lambda event into a User model, which can be passed into the Service instance to process the new_user. The Service response is also mapped by an adapter class (NewUserResponse) to enable an actual AWS Lambda Service response based on the service used to trigger this function. This creates a very clear Lambda function that anyone can look at and understand what is happening. The adapter classes used can have their implementations changed to alter how this function is being triggered—all without making changes to the business domain logic.

We have explored the core of software engineering - SOLID principles, Dependency Inversion, Dependency Injection and Service-Oriented Architecture (SOA). Combining these techniques can assist in simplifying your development team's journey to Serverless by abstracting away the cloud complexity that Serverless developers are traditionally confronted with. Using these techniques discussed here, we can reduce the cognitive load developers have to contend with in creating serverless solutions. This all does feel like it goes against the simplicity of serverless, but we must acknowledge that we are mostly building software solutions at the end of the day. I say mostly as we should be embracing serverless services where possible, but like Jared Short says - "When you write Serverless code, you need to own it", so engineer it properly and take control.

Own it is what I say. Own it with SOLID principles, Interfaces and Adapters and Service-Oriented Architecture.

This is the first in a series of articles about Serverless software architecture and testing. I am collating a summary of what I have learned on my journey to Serverless and am starting with this introduction, highlighting my thoughts on what I forgot when I started learning about serverless. This series will be made up of three parts:

Part 1: What I forgot at the beginning

Part 2: SOLID Architecture for Lambda

Part 3: Simplify Testing with SOLID Architecture

Serverless Testing has been a hot topic in recent times. Many of the articles focus on general approaches, tooling to automate the creation of temporary stacks, and tell us to stop emulating the cloud locally and test in the cloud! In this article, I won't be covering these tools and approaches. Instead, I want to focus on what I forgot on my Serverless journey, and hopefully, it will help you not make the same mistake.

I started my journey with Serverless over four years ago when Lambda was relatively new, and we were all trying to work out just how this new cloud technology worked, scaled and could be used to create real, working, resilient solutions. There are many things to learn about at first - Lambda, SQS, SNS, DynamoDB, and the list goes on. So many new things that we all get wrapped up in learning how these products work.

When I started building serverless, I started with simple code; they were just functions, after all. Then, I quickly moved to local emulation of SQS, SNS, etc., to do what we have always been encouraged as software engineers - do end-to-end local testing. With mixed success, I moved back to unit testing with event fixtures to test my core business logic and continued to refine my testing knowledge and skills. There are always complexities involved with testing lambda code; There are interactions with cloud managed services such as S3, SNS, SQS or DynamoDB, which have AWS SDK clients to manipulate data being stored or processed. Now I spent a lot of time working out how to mock these AWS dependencies in my unit tests and make sure the SDK calls expected were made. Every new service seemed to introduce a new challenge; I spent more time writing mocks and getting unit tests working in this foreign land of AWS dependencies than building solutions and started to feel less productive. I have spent a lot of time the past few months reflecting on my journey since it began and how I have changed my thinking in many areas, particularly around testing serverless solutions.

I strongly feel I got wrapped up in the details of all the new services I had to deal with and coming to terms with how they all worked. The cognitive load on putting this all together at first is a lot! Combining this additional learning and the understanding that we are just writing functions means we expect to be writing fewer lines and simpler code. I was doing just this when I first started and looking back. I can see what I forgot about! I forgot all about plain old Software Engineering!

As a professional Software Engineer with a lot of experience, this is quite a confronting revelation. However, looking at conversations within the community over the past year, I feel I am not alone here. So what do I mean when I say I forgot about plain old Software Engineering? Three small words come to mind - Dependency Inversion Principle.

Dependency Inversion Principle

This core principle of Object-Oriented design sounds scarier and more confusing than it is. The following describes this design principle defined by Robert C. Martin:

1. High-level modules should not import anything from low-level modules. Both should depend on abstractions (e.g., interfaces).
2. Abstractions should not depend on details. Details (concrete implementations) should depend on abstractions.

Dependency Inversion is all about creating functional abstractions between different layers of our code. In a Serverless function context, this technique allows us to focus on more meaningful, descriptive functions to perform actions for our code rather than getting bogged down in the detail of an AWS SDK client for DynamoDB and its associated API calls. Creating these abstractions also removes the implementation detail and complexity of the cloud service mocking of the past. It allows quick and clean testing of each Interface since we can introduce a test stand-in implementation for any abstraction.

Class Diagram - Dependency inversion applied to Lambda handler

So what we need to look at is creating functional abstractions for any call from our core business logic out to the cloud. For example, in the diagram above, you can see the EventHandler depends on the DomainServiceInterface and the DomainLogicImplementation, which implements the DomainServiceInterface, depends on the StorageInterface. This allows the replacement of any green implementation blocks with Testing Stubs enabling all the components to be tested without creating complex mocks and test assertions. This simple software engineering principle is what I forgot about during the start of my Serverless journey. With the state of confusion and commentary around Serverless testing recently, I felt compelled to share my thoughts and what I forgot about when I first started. Hopefully, this will help you find your way quicker in this often-confusing space.

This article is the first in a new series I am writing on Serverless Software Architecture and Testing. These concepts go hand in hand as good architecture and code organisation naturally lead to better testing. My next article will be about SOLID Architectures for Lambda, taking the Dependency Inversion principle mentioned above further and deeper.

I take pride in writing detailed, technical articles filled with solid code examples and links to Github so you can review and learn from my serverless journey. Today, I think this will be my shortest ever article, and it lacks all these things since starting on this Zero Code adventure.

There are a lot of articles out there on correlating transactions across distributed systems and plenty of technical frameworks and solutions for providing transaction traceability. However, correlating transactions as the data passes through our services is essential to observe our system behaviour. So, as developers, we reach for what we know best our development tools, software frameworks and technical solutions.

Adding Correlation Ids, tracing codes and segment identifiers takes effort and time.

Usually, we place these identifiers in HTTP Headers and other unique Attribute locations for Cloud-Native services so we can hide these away and provide them as transparent travellers across our distributed systems. In an AWS Managed service sense, this means we need to know how to pack and unpack these identifiers at each step through our serverless systems - across SQS boundaries, through EventBridge events and via SNS topics. These cloud-native services each have a different mechanism to save, transport, and unpack these keys to observability.

But what if there was a different way without all this complexity and code?

I want to share with you my thinking on this and what I have arrived at to resolve this technical problem in a way that transcends the AWS service complexities I just mentioned. Nowadays, I believe in using the language of integration, and for tracing transactions through distributed services, I create an internal integration language encapsulating these observability identifiers.

Here is an example of a message structure from a system I am working on right now:

{
    "correlation_id": "b58e7f98-2b58-11ec-8d3d-0242ac130003",
    "message_id": "bde284dc-2b58-11ec-8d3d-0242ac130003",
    "trace_id": "3965a72e-2b59-11ec-8d3d-0242ac130003",
    "data":
    {
        "id": "7e6c7638-218c-4947-8d84-a20a9cf68acd",
        "firstname": "michael",
        "lastname": "walmsley",
        "email": "michael@myemail.com"
    }
}

The data represents the original payload received from the initial caller of the service, usually via an API endpoint. The API entry point is responsible for validating the data to ensure it is correct before transforming it into the internal message format containing the tracing identifiers (correlation_id, message_id, tracing_id). Using a standard messaging structure like this means we no longer worry about changing how we interact with each of the Managed Services we are using since our correlation data is now directly embedded within our message data. All of our messages now have correlation identifiers that pass through data services like SQS, Eventbridge, SNS, Kafka, Kinesis, etc., without the need for any custom code to embed meta-data or other custom markers for correlating our transactions.

This simple approach of defining an internal service messaging format allows your transaction data and correlation data to always be visible within every component of your distributed Serverless Architecture. Furthermore, we achieve Transaction observability within our Serverless data processing systems using nothing more than our already prepared Cloudwatch Logging utilities and Zero additional code!

Nearly every framework at it's core will make use of the aws cli in some fashion. All the beginner documentation around using the AWS CLI or other serverless frameworks document setting up the core AWS Access Key or Secret when interacting with the cloud. Whilst this is mostly fine and okay, I feel its not completely secure. If your computer where you store your access key and secret is compromised in anyway then there is potential for your AWS credentials, the key to your cloud credit card to be compromised and this could be a serious problem. Most of the time we all follow the general tutorials and logon using our root account, go to the IAM console and create an Access Key and Secret for our AWS account, for the root user with the MOST access to your cloud. I actually feel really uncomfortable writing this because this setup for me is wrong, really wrong. Don't take this statemnent out of context, 99.999999999% of the time the setup and tutorials are fine and mostly okay so far as security goes but it could be better.

As an enterprise cloud application architect I feel duty bound to look at the security of everything I do or allow to be done in relation to my cloud account. Step number one for securing any account is to ensure that we are always using more than one mechansim to authenticate every logon, Multi-Factor Authentication (MFA) and this is possible to apply to the AWS cli and most frameworks that interact with AWS to deploy your applications.

Creating a User for Secure Access

It is possible to apply Multi-Factor authentication using the aws-cli through the use of the AWS Security Token Service (STS) and by setting up a new IAM User specifically for all your framework interactions via the cli. I strongly recommend against using your root account logon for doing any aws-cli interactions with your AWS account other than for online console access. So lets get started with setting up a new IAM user and enforce MFA for all operations.

Creating a new IAM Policy - ForceMFA

The first thing I setup is a new IAM Policy that will allow access to everything but which also requires the existence of Multi-Factor Authentication. The following steps assume you are logged on as an admin user to your AWS account.

1. Navigate to the IAM console
2. Under Access Management select Policies
3. Click Create Policy button
4. Click on the JSON TAB to get the JSON editor

5. Copy the following into the policy editor:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Deny",
                "Action": "*",
                "Resource": "*",
                "Condition": {
                    "Bool": {
                        "aws:MultiFactorAuthPresent": "false"
                    }
                }
            }
        ]
    }
   

6. The key to this policy is the Condition which says if MFA is not present "aws:MultiFactorAuthPresent": "false" then the Effect of Deny should be applied to all resources and all actions.

7. Click the Next button
8. Click Next again to review the policy
9. Enter the name for your policy ForceMFA
10. Scroll down and click Create Policy

This is the perfect policy to enforce use of MFA for the aws-cli user we will setup next.

Creating a new IAM Group

This part is not actually neccessary but I like to setup a group that has our new ForceMFA policy attached so that any future Users that I create I can simply assign to the same group.

1. Navigate to the IAM console
2. Under Access Management select Groups
3. Click Create New Group to start the wizard
4. Type the group name and press Next
5. Next attach the ForceMFA policy using the filter box to find it
6. Tick the checkbox next to ForceMFA policy
7. You will also need to provide more permissions so the user can create resources, to make life simple I use AdministratorAccess so I can deploy all services and since we have MFA in place there is further protection.
8. Press Next to review the Group details.
9. Click Create Group to complete group creation

Creating the new User for CLI access

Now lets setup the new user using the new User Group we just created.

1. Navigate to the IAM console
2. Under Access Management select Users
3. Click the Add User button to start the wizard
4. Type the name of the user in the box at the top
5. At the bottom you get to choose one of two options for Access Type
6. Lets tick the first one and force this to be Programmatic access only so we know there is no Console access granted.
7. Click next and we come to Add User to Group
8. At this point we should see the Group you just created, if not hit the refresh button and it should appear.
9. Tick the box next to our new Group
10. Now press Next and we get to Tags, add any Tags if you want to the user.
11. Click Next to review your changes and press Create user to create the new user.
12. The next screen is important it shows the API Access Key and Secret for your new user. Press the Download .csv button to download a csv file with the new credentials which we will need later.

Important Note: This is the only place where the actual Secret access key is displayed or available so make sure you complete the last step and download the .csv or take note of both the Access key ID and the Secret access key.

At this point in time we have a new user with programmatic access using an Access key ID and Secret Access Key. At this point we can try to use the aws-cli using just these and you should not be able to access anything at all - AWS should Deny all access since we have not setup MFA yet.

Enabling MFA for your User

The following steps will walk you through setting up MFA on your new user.

1. Navigate to the IAM console
2. Under Access Management select Users
3. Locate the new user you have just created and click the name to go to the User details.
4. Click the Security credentials TAB
5. On the Assigned MFA Device line click the Manage link to setup MFA
6. The most common option to choose will be Virtual MFA Device which means something like Google Authenticator, this is what I am going to use today.
7. With Virtual MFA Device selected click Continue
8. Follow the steps for installing a compatible application if you do not have one already
9. Click Show QR Code and scan the code using your authenticator application OR click Show secret key and enter this to setup the authenticator.
10. Now lets complete this setup by typing in 2 consecutive MFA codes as they change in the app and ensure we are correctly synced.
11. Click Assign MFA to complete the setup.

At this point we now have a new programmatic access only user with MFA turned on and we can now prepare to use this from the command-line using the AWS Security Token Service (STS) to get a token for use by the cli.

Using MFA with the aws-cli

In order to set this up I have created a shell script to call the STS get-session-token and store the temporary credentials into the shell environment so that the aws-cli will work. The awsenv script can be found here{:target="_blank"}.

The awsenv shell script assumes the following:

• sed is available in your shell environment
• jq is available in your shell environment
• bash shell is available for executing the script
• aws-cli is installed and available in your shell environment
• assumes a bash-like shell environment
• Profile folder exists in ~/.aws/profiles

• A file exists for each user in the profile folder and contains the following details:

    export AWS_ACCESS_KEY_ID=<ACCESS_KEY_ID>
    export AWS_SECRET_ACCESS_KEY=<SECRET_KEY>
    export AWS_DEFAULT_REGION=ap-southeast-2
    export AWS_ARN_MFA=arn:aws:iam::111111111111:mfa/user-device-name
  

AWS_ACCESS_KEY_ID: Is the Access key ID from the IAM console for your cli user.
AWS_SECRET_ACCESS_KEY: Is the Access Secret Key downloaded in the csv file when we created the IAM user in the console.
AWS_DEFAULT_REGION: This is simply a conveninece to set the default region for aws-cli commands.
AWS_ARN_MFA: Is the ARN for the MFA device which is available on the Security tab of the IAM User. The highlighted section of the security TAB shows the MFA ARN you need to have in the profile.

The awsenv is a shell script so we are unable to EXPORT new environment variables by executing the shell-script. Instead you source the script into your environment and pass in the command-line parameters as follows.

    $ source awsenv user-profile <MFA number>

Once you do this the awsenv script will source the user-profile file from $HOME/.aws/profiles folder setting up the core required details for calling aws sts get-session-token.

the get-session-token command will return temporary access token credentials in JSON similar to the following.

    {
          "Credentials": {
           "AccessKeyId": "AKIAIOSFODNN7EXAMPLE",
           "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYzEXAMPLEKEY",
           "SessionToken": "AQoEXAMPLEH4aoAH0gNCAPyJxz4BlCFFxWNE1OPTgk5TthT+FvwqnKwRcOIfrRh3c/LTo6UDdyJwOOvEVPvLXCrrrUtdnniCEXAMPLE/IvU1dYUg2RVAJBanLiHb4IgRmpRV3zrkuWJOgQs8IZZaIv2BXIa2R4OlgkBN9bkUDNCJiBeb/AXlzBBko7b15fjrBs2+cTQtpZ3CYWFXG8C5zqx37wnOE49mRl/+OtkIKGO7fAE",
            "Expiration": "2021-03-09T18:06:10+00:00"
          }
    }

Which is where jq comes in to strip out the credentials ad store them in standard aws-cli environment variables. Now that we have done this we can use the aws-cli or any othe framework command which interacts with the aws-cli to deploy cloud services.