AWS Step Function

AWS Step function is a simple way to coordinate the component of distributed applications or microservices. It simplifies the build and execution of multi step process.

AWS step function helps to split any business problem or computational problem into smaller steps. Steps are then executed in series or in parallel to solve the problem. Some features of step functions are:

  • Simple user interface to arrange and visualize steps.
  • Simple to create a multi step functions.
  • Easy to scale and make changes to the application.
  • Ability to track and retries failure.
  • Step function automatically triggers and tracks each step. For failures it automatically retries.
  • Failure of one component doesn’t bring every thing down.
  • Each component scales independently of other component.
  • Steps can perform work, make decisions, triggers parallel executions, manage timeouts or terminate the workflow.
  • The console shows the status of real-time status of each step and provides a detailed history of execution.
  • It is possible to combine activity tasks and service tasks in same step function.

Activity tasks, Activity worker and Service tasks

Activity tasks: Activity tasks allows us to assign steps in the step function to code running on other machine.

Activity worker: Activity worker is an application running on some other machine. It can make HTTP Connection and is usually hosted somewhere else (mobile devices, other EC2 instance etc). Activity worker polls Step function for work, takes input from step function, perform work as per worker implementation and return results. Since activity worker polls for work, it is easy to use workers that are behind the Firewall.

Service tasks: Service tasks helps to connect workflow to other AWS services. Step functions invokes other services, wait for the execution and then continue to next step based on the response.

Step function can also be used to combine activity worker running in the data center with the activity tasks on AWS. The worker service in data center continue to run as usual, along with any cloud-based service tasks.

Use cases for Step function

  • Data processing: to aggregate data from multiple databases into unified reports, refine and reduce large data sets into useful formats, or coordinate multi-step analytics and machine learning workflows.
  • DevOps and IT automation: to build tools for CICD, or to create event-driven applications that automatically respond to changes in infrastructure.
  • E-commerce: to automate mission-critical business processes, such as order fulfillment and inventory tracking.
  • Web applications: to implement robust user registration processes and sign-on authentication.

AWS Step Function vs Amazon SQS

Amazon SQS is highly reliable, scalable, hosted queue service for handling messages. It is used to decouple components. Components interacts asynchronously. Amazon Step functions is useful for creating highly scalable and auditable applications. SQS requires developer to implement custom application level tracking. The Step Functions Console and visibility APIs provide an application-centric view that lets user search for executions, drill down into an execution’s details, and administer executions. Amazon SQS requires implementing such additional functionality.

Step Functions offers several features that facilitate application development, such as passing data between tasks and flexibility in distributing tasks. Amazon SQS requires implementation of some application-level functionality. While Amazon SQS can be used to build basic workflows to coordinate distributed application, we get this facility out-of-the-box with Step Functions, alongside other application-level capabilities.

AWS Step Function vs Amazon Simple Workflow Service

Developer should consider using AWS Step Functions for all new applications. It provides a more productive and agile approach to coordinate application components using visual workflows. If we require external signals to intervene in our processes, or we would like to launch child processes that return a result to a parent, then we should consider Amazon Simple Workflow Service (Amazon SWF). With Amazon SWF, instead of writing state machines in declarative JSON, we write a decider program to separate activity steps from decision steps. This provides us complete control over orchestration logic, but increases the complexity of developing applications. We may write decider programs in the programming language of our choice, or we may use the Flow framework to use programming constructs that structure asynchronous interactions. 

AWS Step function vs Code Pipeline

We should consider using AWS CodePipeline when we are automating the release of software or infrastructure changes and want to leverage integrations with CI/CD services such as CodeBuild, CodeDeploy, and 3rd party CI/CD solutions as part of our release workflow. Inspired by Amazon’s deployment practices, only one instance of each pipeline exists and each is composed of sequential stages. These stages serve as a locking mechanism to ensure that only a single execution is flowing through a critical section of the pipeline at any given time. CodePipeline’s support for manual approvals holds a release within stage and allows manual testing while prior stages continue their execution. The stage-based model also allows to pause all releases without generating extensive release queue. CodePipeline intelligently-releases the most recent software changes to save time on outdated releases.

By comparison, Step Functions natively supports multiple instances of a state machine which execute concurrently. This capability lends itself to supporting teams of developers performing integrations by building and testing their changes at the same time. However, managing deployments requires custom-solutions to control the order of deployments. We should consider AWS Step Functions when we need to create continuous integration workflows using patterns such as error-handling, parallelization, and branching. Step Functions supports service integration with AWS CodeBuild, Amazon ECS, Amazon EMR, AWS Glue, and other services. If workflow requires us to build software changes or execute data-processing applications, we should consider using Step Functions with AWS CodeBuild, Amazon ECS, Amazon EMR, and AWS Glue service integrations.

Sample projects is available here for learning.