Astro 10/2/2023, 5:13:53 PM
You can run a dbt model from Apache Airflow by using the BashOperator
or the DbtTaskGroup
class from the Cosmos package.
Here's an example of how to use the BashOperator
to run a dbt model:
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime
with DAG('dbt_dag', start_date=datetime(2022, 1, 1)) as dag:
run_dbt = BashOperator(
task_id='run_dbt',
bash_command='dbt run --profiles-dir ~/.dbt/ --project-dir ~/my_dbt_project/'
)
In this example, dbt run
is the command to run the dbt model. The --profiles-dir
flag indicates the directory where the profiles.yml file is located, and the --project-dir
flag indicates the directory of your dbt project.
Alternatively, you can use the DbtTaskGroup
class from the Cosmos package to create a task group from the models in your dbt project. Dependencies between your dbt models are automatically turned into dependencies between Airflow tasks. Here's an example:
from cosmos import DbtTaskGroup, ProjectConfig, ProfileConfig
from airflow import DAG
from datetime import datetime
with DAG('dbt_dag', start_date=datetime(2022, 1, 1)) as dag:
dbt_task_group = DbtTaskGroup(
project_config=ProjectConfig("/path/to/my_dbt_project"),
profile_config=ProfileConfig(
profile_name="my_profile",
target_name="my_target",
profile_mapping=PostgresUserPasswordProfileMapping(
conn_id="my_postgres_dbt",
profile_args={"schema": "public"},
),
)
)
In this example, the DbtTaskGroup
class uses the ProjectConfig
to specify the directory of your dbt project and the ProfileConfig
to specify the profile and target to use for your dbt project. The PostgresUserPasswordProfileMapping
class is used to map your Airflow connection to a dbt profile.
Human 10/2/2023, 5:13:26 PM
how can I run a dbt model from Airflow ?