Step Function

Create Step Function

We will create Step function use Glue job ETL data from Raw to Staging then use Crawler create Glue Table for Athena query

Create Step Function

1. Create work flow:

   • Access the Step Function console.
   • Select Create state machine.

   

   • Select blank.
   • Select Select.

   

   • Drag StartJobRun to work flow.
   • Enter “crawler”
   • Drag StartCrawler to work flow after StartJobRun.
   • Enter “pass”
   • Drag Pass to work flow after StartJobRun.

   

   

   

2. Setup Glue job run

   • Select StartJobRun.
   • Enter Parameters: get param from your input when you execute

{
  "JobName.$": "$.job_name"
}

• • Select Wait for task complete to wait glue job run complete before start next task
  • Select Output

  

• • Select Add original input.
  • Select Discard result and keep original input to forward your input to next task.

  

3. Setup Glue crawler run
   • Select StartCrawler.
   • Enter Parameters: get param from your input when you execute

{
  "Name.$": "$.job_crawler"
}

3. Setup Step function properties

   • Select edit name state machine.
   • Enter Step function’s name
   • Select IAM role of Step function
   • Select Create.