This tutorial will guide you through setting up continuous integration (CI) jobs in dbt. CI jobs run when someone opens a new pull request (PR) in your dbt Git repository. By running and testing only modified models, dbt Cloud ensures these jobs are as efficient and resource conscientious as possible on your data platform.
What is a CI job in dbt?
A CI job in dbt is a task that is triggered to run when a new pull request is opened in your dbt Git repository. It tests only the modified models, ensuring efficiency and resource conscientiousness. CI jobs are recommended to be created in a dedicated dbt Cloud deployment environment that's connected to a staging database for better isolation between your temporary CI schema builds and your production data builds.
1. Prerequisites
Before setting up a CI job, ensure you have a dbt Cloud account. For the Concurrent CI checks and Smart cancellation of stale builds features, your dbt Cloud account must be on the Team or Enterprise plan. Also, set up a connection with your Git provider. This integration lets dbt Cloud run jobs on your behalf for job triggering.
2. Creating a CI Job
To create a CI job, navigate to your deployment environment page and click Create job > Continuous integration job. Many options on the CI job page are set to default values that dbt Labs recommends. However, you can change them according to your needs. These options include Job settings and Execution settings.
3. Advanced Settings
Advanced settings are optional and include Environment variables, Target name, Run timeout, dbt version, Threads, and Run source freshness. These settings allow you to customize the behavior of your project when the CI job runs.
Common Challenges and Solutions
One common challenge is the automatic deletion of the temporary schema. This is because automatic deletion relies on incoming webhooks from Git providers, which is only available through the native integrations. To overcome this, you can use the Administrative API to trigger a CI job to run.
- Ensure you have a dbt Cloud account.
- Set up a CI job with the Create Job API endpoint using "job_type": ci or from the dbt Cloud UI.
- Call the Trigger Job Run API endpoint to trigger the CI job.
Best Practices
It's recommended to create your CI job in a dedicated dbt Cloud deployment environment that's connected to a staging database. This provides better isolation between your temporary CI schema builds and your production data builds. Also, it's advisable to use the default settings provided by dbt Labs for creating a CI job.
- Create CI job in a dedicated dbt Cloud deployment environment.
- Use default settings provided by dbt Labs.
- Use the Administrative API to trigger a CI job if not using dbt Cloud’s native Git integration.
Further Learning
You can explore more about CI jobs in dbt by learning about the Create Job API endpoint, the Trigger Job Run API endpoint, and how to set up a connection with your Git provider.
- Learn about the Create Job API endpoint.
- Understand the Trigger Job Run API endpoint.
- Learn how to set up a connection with your Git provider.
Recap of Setting Up CI Jobs in dbt
In this tutorial, we've learned about CI jobs in dbt, how to set them up, and the best practices to follow. We've also discussed the common challenges and their solutions, and provided resources for further learning.
- CI jobs in dbt are tasks that run when a new pull request is opened.
- They should be created in a dedicated dbt Cloud deployment environment.
- Use the default settings provided by dbt Labs for creating a CI job.
Secoda's Integration with dbt for Enhanced Data Analysis
Secoda's integration with dbt provides a robust solution for data analysis and delivery of results. This integration allows users to monitor, debug, and deploy models, and automatically update analytics with new data and insights. It also helps users visualize data flows, detect inconsistencies, and simplify troubleshooting.
- Data Discovery: Secoda is a data discovery tool that helps organizations make sense of their data by providing an interface to explore and analyze data from multiple sources. It offers a unified view of data, visualizations, and search and discovery capabilities to help users identify patterns and trends in their data.
- Increased Transparency: The integration helps data teams increase transparency and truth around company data, providing simple data discovery for every employee.
- Complete View of Data: Users get a complete view of their data, enabling them to quickly identify and resolve issues, make more informed decisions, and ensure accuracy and consistency across data sets.
- Security and Compliance: Secoda is SOC 2 Type 1 and 2 compliant and offers a self-hosted environment, SSH tunneling, auto PII tagging, and data encryption, ensuring the security of your data.
With Secoda's dbt integration, data teams can leverage the power of dbt in a more user-friendly and secure environment, enhancing their data analysis capabilities.
