Apache Airflow® is a platform created by the community to programmatically author, schedule and monitor workflows.
A Directed Acyclic Graph is a non-cyclic directional graph that describes all the data processing steps in a process
Workflows are usually defined with the help of Directed Periodic Graphs (DAGs)
Each DAG is defined in a DAG file, which defines a data processing, represented as a non-cyclic directional graph, where nodes are tasks and edges are dependencies between tasks.
Tasks in a DAG are typically processed sequentially or in parallel on a predetermined schedule
When a DAG is executed, it is called a DAG run
Drawing on an overview of the basic components of Apache Airflow.
The Amazon Web Services connection type enables the AWS Integrations.
The Amazon Web Services Connection can be tested in the UI/API or by calling test_connection(), it is important to correctly interpret the result of this test. During this test components of Amazon Provider invoke AWS Security Token Service API GetCallerIdentity. This service can only check if your credentials are valid. Unfortunately it is not possible to validate if credentials have access to specific AWS service or not. If you use the Amazon Provider to communicate with AWS API compatible services (MinIO, LocalStack, etc.) test connection failure doesn’t mean that your connection has wrong credentials. Many compatible services provide only a limited number of AWS API services, and most of them do not implement the AWS STS GetCallerIdentity method.
Authentication may be performed using any of the options described in Boto3 Guide Credentials. Alternatively, one can pass credentials in as a Connection initialisation parameter.
To use IAM instance profile, create an “empty” connection (i.e. one with no AWS Access Key ID or AWS Secret Access Key specified, or aws://)