Express Workflows

What is Express Workflows?

Express Workflows are a new type of AWS Step Functions workflow type that cost-effectively orchestrate AWS compute, database, and messaging services at event rates greater than 100,000 events per second. Express Workflows automatically start in response to events such as HTTP requests via Amazon API Gateway, AWS Lambda requests, AWS IoT Rules Engine actions, and over 100 other AWS and SaaS event sources from Amazon EventBridge. Express Workflows is suitable for high-volume event processing workloads such as IoT data ingestion, streaming data processing and transformation, and high-volume microservices orchestration.

When to choose Express Workflows Vs Standard Workflows?

You can choose Standard Workflows when you need long-running, durable, and auditable workflows, or Express Workflows for high-volume, event processing workloads.Your state machine executions will behave differently, depending on which Type you select. The Type you choose cannot be changed after your state machine has been created.

# Standard vs Express Workflows #

| Features 					   	   	| Standard Workflows 				 | Express Workflows 	   			|
| -------------------------------- 	| ---------------------------------	 | -------------------------------- |
| Maximum duration 					| 1 year. 							 | 5 minutes. 			   			|
| Supported execution start rate  	| Over 2,000 per second				 | Over 100,000 per second  		|
| Supported state transition rate  	| Over 4,000 per second per account  | Nearly unlimited 				|
| Pricing  							| Priced per state transition 		 | Priced by the number of  		|
|									|									 | executions you run,      		|
|									|									 | their duration, and,     		|
|									|									 | memory consumption.      		|
| Execution semantics			  	| Exactly-once workflow execution.   | At-least-once workflow execution.|

For detailed information on the differences between Standard and Express Workflows, see Standard vs. Express Workflows

Create an AWS Step Functions state machine of type Express

The overall functionality of the workshop remains the same and we will be using the same State Machine Definition and Lambda functions from before.

As a review, we will be creating below step function state machine now with Express Worklow.

initial state machine diagram
  1. Make sure your final rider-photo-state-machine.json file should look like this (the AWS Lambda ARNs are examples):

    (expand to see)
  2. From the AWS Management Console, choose Services then select Step Functions.

  3. You might see the following Get Started page if you have not used AWS Step Functions before. If that’s the case, click on the three horizontal lines in the top-left corner.

    Create Initial State Machine - Step 2
  4. Next, click State Machines.

    Create Initial State Machine - Step 3
  5. Next, click Create State Macine. If you completed the workshop successfully earlier, you would notice the existing RiderPhotoSharing of Type Standard in the list of existing StateMachines.

    Create Initial State Machine - Step 4
  6. Select Author with Code Snippets if it’s not already selected.

  7. Change the default selection from “Standard” to “Express” type.

    Create Initial State Machine - Step 5
  8. Highlight and replace all of the existing sample state machine definition JSON by pasting your custom state machine JSON definition from your rider-photo-state-machine.json file from Cloud9 into the Code editor portion.

  9. You can click on the ↺ sign in the preview panel to visualize the workflow:

    Update state machine with persistence step
  10. Click Next.

  11. Type RiderPhotoProcessingExpress for the state machine name.

  12. For IAM role for executions, pick Create new role.

    Select IAM role
  13. For Logging and Tags keep default selections.

    Select IAM role
  14. Click Create State Machine to create new IAM Role along with the state machine. Note the Type of the state machine is Express.

    Select IAM role
  15. Click the IAM Role link to go to IAM console and click Attach Poilicies to add permissions to execute lambda functions.

    Click Add Policies
  16. Search for AwsLambdaRole and select the checkbox associated to it and click Attach Poilicy.

    Search and Add policy AwsLambdaRole to the state machine
  17. After adding AWSLambdaRole, final policies should looks like this.

    Updated IAM Role for state machine execution
  18. Click the IAM Role link to go to IAM console and click Attach Poilicies to add permissions to send notification to SNS topic.

    Click Add Policies
  19. Search for AmazonSNSFullAccess and select the checkbox associated to it and click Attach Poilicy.

    Search and Add policy AwsLambdaRole to the state machine
  20. After adding AmazonSNSFullAccess, final policies should looks like this.

    Updated IAM Role for state machine execution
  21. Go back to the Express workflows Step Functions in console and Click the Start execution button to start a new execution.

    Updated IAM Role for state machine execution
  22. Here you specify the input data passed into the AWS Step Functions state machine to process.

    Each execution of a Step Functions state machine has an unique ID. You can either specify one when starting the execution, or have the service generate one for you. In the text field that says “enter your execution id here”, you can specify an execution ID, or leave it blank.

    For the input data, type in the follow JSON. Make sure to substitute the s3Bucket field with your own values.

    For s3Bucket field, look in the Outputs section of the wildrydes-step-module-resources stack for RiderPhotoS3Bucket.

    The userId field is needed because in later processing steps, the userId is used to record which user the profile picture is associated with.

    {
    	"userId": "user_a", 
    	"s3Bucket": "REPLACE_WITH_YOUR_BUCKET_NAME",
    	"s3Key": "1_happy_face.jpg"
    }	

    this tells the image processing workflow the userId that uploaded the picture and the Amazon S3 bucket and keys the photo is at.

    Test new execution
  23. Once you Start Execution, Unlike Standard Woflows, you would NOT see execution results in the visual format as below on the Step Functions Page. This is to get improved performance with Express Workflows. Instead, you have to go to CloudWatch to check the execution status and logs for Express Workflows.

    Test new execution
  24. Click View in CloudWatch Logs, to see the execution logs and status of the execution.

    Navigate to CloudWatch logs
  25. Look in the logs for LambdaFunctionSucceeded and click to see more details.

    Successful execution logs
  26. If the last step succeeds, you can use the AWS CLI to check the list of faces indexed in your Rekognition collection (replace the REPLACE_WITH_YOUR_CHOSEN_AWS_REGION portion with the region string of your chosen region):

     aws rekognition list-faces \
       --collection-id rider-photos \
       --region REPLACE_WITH_YOUR_CHOSEN_AWS_REGION
    
  27. You might find the delete-faces command useful when testing:

     aws rekognition delete-faces \
       --collection-id rider-photos \
       --face-ids REPLACE_WITH_ID_OF_FACE_TO_DELETE \
       --region REPLACE_WITH_YOUR_CHOSEN_AWS_REGION
    
  28. You can also use the Amazon S3 Console to check the Amazon S3 bucket created by AWS CloudFormation to store the resized thumbnail images. You should find resized thumbnail images in the bucket.

The name of the S3 bucket can be found in the in AWS CloudFormation output ThumbnailS3Bucket. You can also simply search for it in the S3 Console for wildrydes-step-module-resources-thumbnails3bucket

TroubleShooting Failed Executions

  1. You can now see the state machine execution and status in cloudwatch. You would see the execution failed because of insufficient permission to execute Lambda Functions from a Step Function. This is because by default express workflow state machine does not have aws policy to execute lambda functions. So, we will add required policy in the next step.

    First failed execution logs
  2. If you see error or failure in the logs related to lambda execution, This is because by default express workflow state machine does not have aws policy to execute lambda functions. So, add AWSLambdaRole policy to the new Role created at the time of creation of State Machine above.

    Add policy AwsLambdaRole to the state machine
  3. After adding AWSLambdaRole, final policies should looks like this.

    Updated IAM Role for state machine execution
  4. Re-try another execution by passing same input json as before and confirm from the CloudWatch logs that the execution is succeeded this time.

    Successful execution logs