Getting Started with AWS Step Functions

Features of Step Functions

  • Error Handling — AWS has error handling built-in and you can easily specify various catch blocks and tell what action you want to perform if something fails.
  • Retry Handling — Step Functions support retries as well to cope with the inevitable transient error.
  • Multiple ways to Trigger workflow step — there are a few ways you can kick off a workflow such as via API or a CloudWatch event. I will cover them in more detail later.
  • Workflow Progress Visualisation — The AWS Console will show you a visual representation of your workflow as well as showing the progress and whether a step has failed.

Main Benefits of Step Functions

  • Handles all error handling and retry logic — as mentioned in the features, all this work is already built-in.
  • Can handle complex workflows with hundreds of steps — Step Functions can cope with hundreds if not thousands of steps.
  • Copes with manual steps — many business processes require a manual review at some point.
  • Handles long-lived workflows up to 1 year — workflows using the standard type can last for as long as a year. So you don’t have to worry about your workflow timing out if there is a manual approval step.
  • Manage state between executions of stateless functions
  • Decouple the application work from business logic
  • Scales well and can perform parallel executions

Drawbacks & Limitations of Step Function

  • Decoupling all the workflow logic might make your application difficult to understand. Once you start linking together numerous individual Lambda components you need to makes sure that the workflow is well understood and documented.
  • Need to learn the AWS States Language which is based on JSON.
  • Locked into an AWS specific technology — Step Functions are unique to AWS and therefore if you plan on moving to a different cloud provider you will need to come up with a different solution.
  • Execution History limited to 90 days — if you need longer execution history you will need to rely on the CloudWatch logs.
  • Currently, only a few ways to trigger workflow transitions — not all the AWS services support triggering a workflow. The notable ones that are missing are from DynamoDB and Kinesis.
  • Data sent between workflow steps are not encrypted — if you want to pass sensitive data between steps you need to store it encrypted in either DynamoDB or S3 and provide a pointer in your step output.
  • Workflows can last up to 1 year on Standard and 5 mins on Express.
  • Execution History has a 25,000 hard limit — if you think the number of executions (including retries) is likely to go above this you will need to split up your workflow into multiple workflows. Luckily you can kick off other step functions from a step function.
  • Payload size limited to 32KB 256KB* — if you want to store above this amount you will need to use some other storage such as S3 or DynamoDB.
  • Only works natively with a handful of AWS services — if you need other applications to be triggered this will need to be done with a lambda wrapper.

AWS Step Function Workflow Types

Standard

  • Workflows can run for up to one year
  • 2,000 executions per second
  • 4,000 transitions per second per AWS account
  • Priced per state transition
  • Supports Activities & Callbacks — these are for long-running services that you don’t want in a lambda.

Express

  • Workflows can run for up to 5 minutes
  • 100,000 executions per second
  • Nearly unlimited state transitions
  • Priced on the number of executions, duration and memory consumption
  • Doesn’t support Activities or Callbacks

AWS Step Function Pricing

Standard Pricing

Express Pricing

  • $0.00001667 per GB-Second ($0.0600 per GB-hour) for the first 1,000 hours GB-hours
  • $0.00000833 per GB-Second ($0.0300 per GB hour) for the next 4,000 hours GB-hours
  • $0.00000456 per GB-Second ($0.01642 per GB-hour) beyond that

Overview of Step Functions

Triggering a Step Function

  • API Gateway — using StartExecution from an API Gateway and pass in the payload from a POST call to your step function.
  • Cloudwatch/EventBridge Events — using EventBridge or Cloudwatch you can trigger step functions when another event occurs. Such as a file being uploaded to S3.
  • Amazon SDK — You can also trigger StartExecution using the SDK either from a Lambda function or another application.
  • Other Step Functions — it is good practice to split up large workflows into smaller ones that you can trigger using the task state.

Amazon States Language

{
"StartAt": "HelloWorld",
"States": {
"HelloWorld": {
"Type": "Pass",
"Result": "Hello World!",
"End": true
}
}
}
{
"StartAt": "Hello",
"States": {
"Hello": {
"Type": "Pass",
"Result": "Hello",
"Next": "World"
},
"World": {
"Type": "Pass",
"Result": "World!",
"End": true
}
}
}

Step Function States

  • Comment - you can add comments so others know what your workflow is doing (and you in a few weeks)
  • InputPath & OutputPath - these allow you to select a portion of the input to pass on to your state or output to pass on to the next state. I will cover these in more detail in the Inputs and Outputs section below.

Pass

"Result": {
"x-datum": 0.381018,
"y-datum": 622.2269926397355
},
{
"product": {
"name": "T-Shirt",
"details": {
"color": "green",
"size": "large",
"material": "nylon"
},
"availability": "in stock",
"cost": "$23"
}
}
{
"product": "T-Shirt",
"colour": "green",
"size": "large",
"availability": "in stock"
}
"MapValues": {
"Type": "Pass",
"Parameters": {
"product.$": "$.product.name",
"colour.$": "$.product.details.color",
"size.$": "$.product.details.size",
"availability.$": "$.product.availability"
}
"Next": "AnotherState"
},

Task

"LambdaState": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:HelloWorld",
"Next": "NextState"
}

Choice

"ToBeOrNotToBe": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.amount",
"NumericGreaterThan": 1000
"Next": "CEOApproval"
},
{
"Variable": "$.amount",
"NumericGreaterThan": 500
"Next": "ManagerApproval"
}
],
"Default": "Approved"
},

Wait

"wait_ten_seconds": {
"Type": "Wait",
"Seconds": 10,
"Next": "NextState"
}
"wait_until" : {
"Type": "Wait",
"Timestamp": "2021-03-14T01:59:00Z",
"Next": "NextState"
}

Succeed

"SuccessState": {
"Type": "Succeed"
}

Fail

"FailState": {
"Type": "Fail",
"Cause": "Invalid response.",
"Error": "ErrorA"
}

Parallel

"RunTwoThingsInParallel": {
"Type": "Parallel",
"Next": "NextStateAfterParallel",
"Branches": [
{
"StartAt": "APICall1",
"States": {
"APICall1": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:GetInfo1",
"End": true
}
}
},
{
"StartAt": "APICall2",
"States": {
"APICall2": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:GetInfo2",
"End": true
}
}
}
]
}

Map

{
"ship-date": "2016-03-14T01:59:00Z",
"detail": {
"delivery-partner": "UQS",
"shipped": [
{ "prod": "R31", "dest-code": 9511, "quantity": 1344 },
{ "prod": "S39", "dest-code": 9511, "quantity": 40 },
{ "prod": "R31", "dest-code": 9833, "quantity": 12 },
{ "prod": "R40", "dest-code": 9860, "quantity": 887 },
{ "prod": "R40", "dest-code": 9511, "quantity": 1220 }
]
}
}
"Validate-All": {
"Type": "Map",
"InputPath": "$.detail",
"ItemsPath": "$.shipped",
"MaxConcurrency": 0,
"Iterator": {
"StartAt": "Validate",
"States": {
"Validate": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:ship-val",
"End": true
}
}
},
"ResultPath": "$.detail.shipped",
"End": true
}

Inputs and Ouputs

InputPath

{
"result1": {
"val1": 1,
"val2": 2,
"val3": 3
},
"result2": {
"val1": "a",
"val2": "b",
"val3": "c"
}
}
{
"val1": "a",
"val2": "b",
"val3": "c"
}

OutputPath

Parameters

[
{
"product": "T-Shirt",
"color": "green",
"size": "large"
},
{
"product": "T-Shirt",
"stock": 100
}
]
"Parameters": {
"product.$": "$[0].product",
"color.$": "$[0].color",
"size.$": "$[0].size",
"stock.$": "$[1].stock"
}

ResultPath

ResultSelector

{
"product": "T-Shirt",
"color": "green",
"size": "large"
}
{
"product": "T-Shirt",
"stock": 100
}
{
"product": "T-Shirt",
"color": "green",
"size": "large",
"availability": {
"stock": 100
}
}
"ResultSelector": {
"stock.$": "$.stock"
},
"ResultPath": "$.availability"

Handling Errors in Step Functions

Retries

"Retry": [ {
"ErrorEquals": [ "States.ALL" ],
"IntervalSeconds": 3,
"MaxAttempts": 2,
"BackoffRate": 2
} ]
  1. Error occurs
  2. Waits 3 seconds
  3. Retries the function. If it fails again then:
  4. Wait 6 seconds
  5. Retries the function. If it fails again then raise an error.
"Retry": [ {
"ErrorEquals": [ "States.Permissions" ],
"MaxAttempts": 0
}, {
"ErrorEquals": [ "States.ALL" ]
} ]

Catch blocks

"Catch": [ {
"ErrorEquals": [ "States.ALL" ],
"Next": "ErrorState"
} ]

Activities & Callbacks

Activities

Callbacks

  1. Get your step function to send a message to SQS with .waitForTaskToken
  2. Set up an API Gateway that has approve and decline endpoints that send SendTaskSuccess or SendTaskFailure.
  3. Write a lambda that picks up the SQS message and then sends an email via SES with links to your approve and decline endpoints with the taskToken added as a query string parameter.

Best Practices when using AWS Step Functions

States should be idempotent

Pass a unique id to your lambda functions

Final Thoughts

--

--

Software Developer @ https://www.alexhyett.com

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store