In our first post on refining testing best practices, we developed a prioritized list of data quality concerns. We also documented first steps for debugging each concern. This post will guide you on where specific tests should go in your data pipeline.

Note that we are constructing this guidance based on how we structure data at dbt Labs. You may use a different modeling approach—that’s okay! Translate our guidance to your data’s shape, and let us know in the comments section what modifications you made.

First, here’s our opinions on where specific tests should go:

• Source tests should be fixable data quality concerns. See the callout box below for what we mean by “fixable”.
• Staging tests should be business-focused anomalies specific to individual tables, such as accepted ranges or ensuring sequential values. In addition to these tests, your staging layer should clean up any nulls, duplicates, or outliers that you can’t fix in your source system. You generally don’t need to test your cleanup efforts.
• Intermediate and marts layer tests should be business-focused anomalies resulting specifically from joins or calculations. You also may consider adding additional primary key and not null tests on columns where it’s especially important to protect the grain.

Where should tests go in your pipeline?​

This diagram above outlines where you might put specific data tests in your pipeline. Let’s expand on it and discuss where each type of data quality issue should be tested.

Sources​

Tests applied to your sources should indicate fixable-at-the-source-system issues. If your source tests flag source system issues that aren’t fixable, remove the test and mitigate the problem in your staging layer instead.

Here’s our recommendation for what tests belong on your sources.

• Source freshness: testing data freshness for sources that are critical to your pipelines.
  • If any sources feed into any of the “top 3” priority categories in our last post, use dbt source freshness in your job execution commands and set the severity to error. That way, if source freshness fails, so does your job.
  • If none of your sources feed into high priority categories, set your source freshness severity to warn and add source freshness to your job execution commands. That way, you still get source freshness information but stale data won't fail your pipeline.
• Data hygiene: tests that are fixable in the source system (see our note above on “fixability”).
  • Examples:
    • Duplicate customer records that can be deleted in the source system
    • Null records, such as a customer name or email address, that can be entered into the source system
    • Primary key testing where duplicates are removable in the source system

Staging​

In the staging layer, your models should be cleaning up or mitigating data issues that can't be fixed at the source. Your tests should be focused on business anomaly detection.

• Data cleanup and issue mitigation: Use our best practices around staging layers to clean things up. Don’t add tests to your cleanup efforts. If you’re filtering out nulls in a column, adding a not_null test is repetitive! 🌶️
• Business-focused anomaly examples: these are data quality issues you should test for in your staging layer, because they fall outside of your business’s defined norms. These might be:
  • Values inside a single column that fall outside of an acceptable range. For example, a store selling a greater quantity of limited-edition items than they received in their stock delivery.
  • Values that should always be positive, are positive. This might look like a negative transaction amount that isn’t classified as a return. This failing test would then spur further investigation into the offending transaction.
  • An unexpected uptick in volume of a quantity column beyond a pre-defined percentage. This might look like a store’s customer volume spiking unexpectedly and outside of expected seasonal norms. This is an anomaly that could indicate a bug or modeling issue.

Intermediate (if applicable)​

In your intermediate layer, focus on data hygiene and anomaly tests for new columns. Don’t re-test passthrough columns from sources or staging. Here are some examples of tests you might put in your intermediate layer based on the use cases of intermediate models we outline in this guide.

• Intermediate models often re-grain models to prepare them for marts.
  • Add a primary key test to any re-grained models.
  • Additionally, consider adding a primary key test to models where the grain has remained the same but has been enriched. This helps future-proof your enriched models against future developers who may not be able to glean your intention from SQL alone.
• Intermediate models may perform a first set of joins or aggregations to reduce complexity in a final mart.
  • Add simple anomaly tests to verify the behavior of your sets of joins and aggregations. This may look like:
    • An accepted_values test on a newly calculated categorical column.
    • A mutually_exclusive_ranges test on two columns whose values behave in relation to one another (ex: asserting age ranges do not overlap).
    • A not_constant test on a column whose value should be continually changing (ex: page view counts on website analytics).
• Intermediate models may isolate complex operations.
  • The anomaly tests we list above may suffice here.
  • You might also consider unit testing any particularly complex pieces of SQL logic.

Marts​

Marts layer testing will follow the same hygiene-or-anomaly pattern as staging and intermediate. Similar to your intermediate layer, you should focus your testing on net-new columns in your marts layer. This might look like:

• Unit tests: validate especially complex transformation logic. For example:
  • Calculating dates in a way that feeds into forecasting.
  • Customer segmentation logic, especially logic that has a lot of CASE-WHEN statements.
• Primary key tests: focus on where where your mart's granularity has changed from its staging/intermediate inputs.
  • Similar to the intermediate models above, you may also want to add primary key tests to models whose grain hasn’t changed, but have been enriched with other data. Primary key tests here communicate your intent.
• Business focused anomaly tests: focus on new calculated fields, such as:
  • Singular tests on high-priority, high-impact tables where you have a specific problem you want forewarning about.
    • This might be something like fuzzy matching logic to detect when the same person is making multiple emails to extend a free trial beyond its acceptable end date.
  • A test for calculated numerical fields that shouldn’t vary by more than certain percentage in a week.
  • A calculated ledger table that follows certain business rules, i.e. today’s running total of spend must always be greater than yesterday’s.

CI/CD​

All of the testing you’ve applied in your different layers is the manual work of constructing your framework. CI/CD is where it gets automated.

You should run a slim CI to optimize your resource consumption.

With CI/CD and your regular production runs, your testing framework can be on autopilot. 😎

If and when you encounter failures, consult your trusty testing framework doc you built in our earlier post.

Advanced CI​

In the early stages of your smarter testing journey, start with dbt Cloud’s built-in flags for advanced CI. In PRs with advanced CI enabled, dbt Cloud will flag what has been modified, added, or removed in the “compare changes” section. These three flags offer confidence and evidence that your changes are what you expect. Then, hand them off for peer review. Advanced CI helps jump start your colleague’s review of your work by bringing all of the implications of the change into one place.

We consider usage of Advanced CI beyond the modified, added, or changed gut checks to be an advanced (heh) testing strategy, and look forward to hearing how you use it.

Wrapping it all up​

Judicious data testing is like training for a marathon. It’s not productive to go run 20 miles a day and hope that you’ll be marathon-ready and uninjured. Similarly, throwing data tests randomly at your data pipeline without careful thought is not going to tell you much about your data quality.

Runners go into marathons with training plans. Analytics engineers who care about data quality approach the issue with a plan, too.

As you try out some of the guidance above here, remember that your testing needs are going to evolve over time. Don’t be afraid to revise your original testing strategy.

Let us know your thoughts on these strategies in the comments section. Try them out, and share your thoughts to help us refine them.

In this blog, we’ll walk through a plan to define data quality. This will look like:

• identifying data hygiene issues
• identifying business-focused anomaly issues
• identifying stats-focused anomaly issues

Once we have defined data quality, we’ll move on to prioritize those concerns. We will:

• think through each concern in terms of the breadth of impact
• decide if each concern should be at error or warning severity

Who are we?​

Let’s start with introductions - we’re Faith and Jerrie, and we work on dbt Labs’s training and services teams, respectively. By working closely with countless companies using dbt, we’ve gained unique perspectives of the landscape.

The training team collates problems organizations think about today and gauge how our solutions fit. These are shorter engagements, which means we see the data world shift and change in real time. Resident Architects spend much more time with teams to craft much more in-depth solutions, figure out where those solutions are helping, and where problems still need to be addressed. Trainers help identify patterns in the problems data teams face, and Resident Architects dive deep on solutions.

Today, we’ll guide you through a particularly thorny problem: testing.

Why testing?​

Mariah Rogers broke early ground on data quality and testing in her Coalesce 2022 talk. We’ve seen similar talks again at Coalesce 2024, like this one from the data team at Aiven and this one from the co-founder at Omni Analytics. These talks share a common theme: testing your dbt project too much can get out of control quickly, leading to alert fatigue.

In our customer engagements, we see wildly different approaches to testing data. We’ve definitely seen what Mariah, the Aiven team, and the Omni team have described, which is so many tests that errors and alerts just become noise. We’ve also seen the opposite end of the spectrum—only primary keys being tested. From our field experiences, we believe there’s room for a middle path. A desire for a better approach to data quality and testing isn’t just anecdotal to Coalesce, or to dbt’s training and services. The dbt community has long called for a more intentional approach to data quality and testing - data quality is on the industry’s mind! In fact, 57% of respondents to dbt’s 2024 State of Analytics Engineering survey said that data quality is a predominant issue facing their day-to-day work.

What does d@tA qUaL1Ty even mean?!​

High-quality data is trusted and used frequently. It doesn’t get argued over or endlessly scrutinized for matching to other data. Data testing should lead to higher data quality and insights, period.

Best practices in data quality are still nascent. That said, a lot of important baseline work has been done here. There are case studies on implementing dbt testing well. dbt Labs also has an Advanced Testing course, emphasizing that testing should spur action and be focused and informative enough to help address failures. You can even enforce testing best practices and dbt Labs’s own best practices using the dbt_meta_testing or dbt_project_evaluator packages and dbt Explorer’s Recommendations page.

The missing piece is still cohesion and guidance for everyday practitioners to help develop their testing framework.

To recap, we’re going to start with:

• identifying data hygiene issues
• identifying business-focused anomaly issues
• identifying stats-focused anomaly issues

Next, we’ll prioritize. We will:

• think through each concern in terms of the breadth of impact
• decide if each concern should be at error or warning severity

Get a pen and paper (or a google doc) and join us in constructing your own testing framework.

Identifying data quality issues in your pipeline​

Let’s start our framework by identifying types of data quality issues.

In our daily work with customers, we find that data quality issues tend to fall into one of three broad buckets: data hygiene, business-focused anomalies, and stats-focused anomalies. Read the bucket descriptions below, and list 2-3 data quality concerns in your own business context that fall into each bucket.

Bucket 1: Data hygiene​

Data hygiene issues are concerns you address in your staging layer. Hygienic data meets your expectations around formatting, completeness, and granularity requirements. Here are a few examples.

• Granularity: primary keys are unique and not null. Duplicates throw off calculations.
• Completeness: columns that should always contain text, do. Incomplete data often has to get excluded, reducing your overall analytical power.
• Formatting: email addresses always have a valid domain. Incorrect emails may affect things like marketing outreach.

Bucket 2: Business-focused anomalies​

Business-focused anomalies catch unexpected behavior. You can flag unexpected behavior by clearly defining expected behavior. Business-focused anomalies are when aspects of the data differ from what you know to be typical in your business. You’ll know what’s typical either through your own analyses, your colleagues’ analyses, or things your stakeholder homies point out to you.

Since business-focused anomaly testing is set by a human, it will be fluid and need to be adjusted periodically. Here’s an example.

Imagine you’re a sales analyst. Generally, you know that if your daily sales amount goes up or down by more than 20% daily, that’s bad. Specifically, it’s usually a warning sign for fraud or the order management system (OMS) dropping orders. You set a test in dbt to fail if any given day’s sales amount is a delta of 20% from the previous day. This works for a while.

Then, you have a stretch of 3 months where your test fails 5 times a week! Every time you investigate, it turns out to be valid consumer behavior. You’re suddenly in hypergrowth, and sales are legitimately increasing that much.

Your 20%-change fraud and OMS failure detector is no longer valid. You need to investigate anew which sales spikes or drops indicate fraud or OMS problems. Once you figure out a new threshold, you’ll go back and adjust your testing criteria.

Although your data’s expected behavior will shift over time, you should still commit to defining business-focused anomalies to grow your understanding of what is normal for your data.

Here’s how to identify potential anomalies.

Start at your business intelligence (BI) layer. Pick 1-3 dashboards or tables that you know are used frequently. List these 1-3 dashboards or tables. For each dashboard or table you have, identify 1-3 “expected” behaviors that your end-users rely on. Here are a few examples to get you thinking:

• Revenue numbers should not change by more than X% in Y amount of time. This could indicate fraud or OMS problems.
• Monthly active users should not decline more than X% after the initial onboarding period. This might indicate user dissatisfaction, usability issues, or that users not finding a feature valuable.
• Exam passing rates should stay above Y%. A decline below that threshold may indicate recent content changes or technical issues are affecting understanding or accessibility.

You should also consider what data issues you have had in the past! Look through recent data incidents and pick out 3 or 4 to guard against next time. These might be in a #data-questions channel or perhaps a DM from a stakeholder.

Bucket 3: Stats-focused anomalies​

Stats-focused anomalies are fluctuations that go against your expected volumes or metrics. Some examples include:

• Volume anomalies. This could be site traffic amounts that may indicate illicit behavior, or perhaps site traffic dropping one day then doubling the next, indicating that a chunk of data were not loaded properly.
• Dimensional anomalies, like too many product types underneath a particular product line that may indicate incorrect barcodes.
• Column anomalies, like sale values more than a certain number of standard deviations from a mean, that may indicate improper discounting.

Overall, stats-focused anomalies can indicate system flaws, illicit site behavior, or fraud, depending on your industry. They also tend to require more advanced testing practices than we are covering in this blog. We feel stats-based anomalies are worth exploring once you have a good handle on your data hygiene and business-focused anomalies. We won’t give recommendations on stats-focused anomalies in this post.

How to prioritize data quality concerns in your pipeline​

Now, you have a written and categorized list of data hygiene concerns and business-focused anomalies to guard against. It’s time to prioritize which quality issues deserve to fail your pipelines.

To prioritize your data quality concerns, think about real-life impact. A couple of guiding questions to consider are:

• Are your numbers customer-facing? For example, maybe you work with temperature-tracking devices. Your customers rely on these devices to show them average temperatures on perishable goods like strawberries in-transit. What happens if the temperature of the strawberries reads as 300C when they know their refrigerated truck was working just fine? How is your brand perception impacted when the numbers are wrong?
• Are your numbers used to make financial decisions? For example, is the marketing team relying on your numbers to choose how to spend campaign funds?
• Are your numbers executive-facing? Will executives use these numbers to reallocate funds or shift priorities?

We think these 3 categories above constitute high-impact, pipeline-failing events, and should be your top priorities. Of course, adjust priority order if your business context calls for it.

Consult your list of data quality issues in the categories we mention above. Decide and mark if any are customer facing, used for financial decisions, or are executive-facing. Mark any data quality issues in those categories as “error”. These are your pipeline-failing events.

If any data quality concerns fall outside of these 3 categories, we classify them as nice-to-knows. Nice-to-know data quality testing can be helpful. But if you don’t have a specific action you can immediately take when a nice-to-know quality test fails, the test should be a warning, not an error.

You could also remove nice-to-know tests altogether. Data testing should drive action. The more alerts you have in your pipeline, the less action you will take. Configure alerts with care!

However, we do think nice-to-know tests are worth keeping if and only if you are gathering evidence for action you plan to take within the next 6 months, like product feature research. In a scenario like that, those tests should still be set to warning.

Start your action plan​

Now, your data quality concerns are listed and prioritized. Next, add 1 or 2 initial debugging steps you will take if/when the issues surface. These steps should get added to your framework document. Additionally, consider adding them to a test’s description.

This step is important. Data quality testing should spur action, not accumulate alerts. Listing initial debugging steps for each concern will refine your list to the most critical elements.

If you can't identify an action step for any quality issue, remove it. Put it on a backlog and research what you can do when it surfaces later.

Here’s a few examples from our list of unexpected behaviors above.

• For calculated field X, a value above Y or below Z is not possible.
  • Debugging initial steps
    • Use dbt test SQL or recent test results in dbt Explorer to find problematic rows
    • Check these rows in staging and first transformed model
    • Pinpoint where unusual values first appear
• Revenue shouldn’t change by more than X% in Y amount of time.
  • Debugging initial steps:
    • Check recent revenue values in staging model
    • Identify transactions near min/max values
    • Discuss outliers with sales ops team

You now have written out a prioritized list of data quality concerns, as well as action steps to take when each concern surfaces. Next, consult hub.getdbt.com and find tests that address each of your highest priority concerns. dbt-expectations and dbt_utils are great places to start.

The data tests you’ve marked as “errors” above should get error-level severity. Any concerns falling into that nice-to-know category should either not get tested or have their tests set to warning.

Your data quality priorities list is a living reference document. We recommend linking it in your project’s README so that you can go back and edit it as your testing needs evolve. Additionally, developers in your project should have easy access to this document. Maintaining good data quality is everyone’s responsibility!

As you try these ideas out, come to the dbt Community Slack and let us know what works and what doesn’t. Data is a community of practice, and we are eager to hear what comes out of yours.

Think of the manufacturing materials needed as our data and the building of the bridge as the building of our ML models. There are thousands of engineers and construction workers taking materials from all over the world, pulling only the specific pieces needed for each part of the project. However, to make this project truly work at this scale, we need the warehousing and logistics to ensure that each load of concrete rebar and steel meets the standards for quality and safety needed and is available to the right people at the right time — as even a single fault can have catastrophic consequences or cause serious delays in project success. This warehouse and the associated logistics play the role of the feature store, ensuring that data is delivered consistently where and when it is needed to train and run ML models.

What is a feature?​

A feature is a transformed or enriched data that serves as an input into a machine learning model to make predictions. In machine learning, a data scientist derives features from various data sources to build a model that makes predictions based on historical data. To capture the value from this model, the enterprise must operationalize the data pipeline, ensuring that the features being used in production at inference time match those being used in training and development.

What role does dbt play in getting data ready for ML models?​

dbt is the standard for data transformation in the enterprise. Organizations leverage dbt at scale to deliver clean and well-governed datasets wherever and whenever they are needed. Using dbt to manage the data transformation processes to cleanse and prepare datasets used in feature development will ensure consistent datasets of guaranteed data quality — meaning that feature development will be consistent and reliable.

Who is going to use this and what benefits will they see?​

Snowflake and dbt are already a well-established and trusted combination for delivering data excellence across the enterprise. The ability to register dbt pipelines in the Snowflake Feature Store further extends this combination for ML and AI workloads, while fitting naturally into the data engineering and feature pipelines already present in dbt.

Some of the key benefits are:

• Feature collaboration — Data scientists, data analysts, data engineers, and machine learning engineers collaborate on features used in machine learning models in both Python and SQL, enabling teams to share and reuse features. As a result, teams can improve the time to value of models while improving the understanding of their components. This is all backed by Snowflake’s role-based access control (RBAC) and governance.
• Feature consistency — Teams are assured that features generated for training sets and those served for model inference are consistent. This can especially be a concern for large organizations where multiple versions of the truth might persist. Much like how dbt and Snowflake help enterprises have a single source of data truth, now they can have a single source of truth for features.
• Feature visibility and use — The Snowflake Feature Store provides an intuitive SDK to work with ML features and their associated metadata. In addition, users can browse and search for features in the Snowflake UI, providing an easy way to identify features
• Point-in-time correctness — Snowflake retrieves point-in-time correct features using ASOF Joins, removing the significant complexity in generating the right feature value for a given time period whether for training or batch prediction retrieval.
• Integration with data pipelines — Teams that have already built data pipelines in dbt can continue to use these with the Snowflake Feature Store. No additional migration or feature re-creation is necessary as teams plug into the same pipelines.

Why did we integrate/build this with Snowflake?​

How does dbt help with ML workloads today? dbt plays a pivotal role in preparing data for ML models by transforming raw data into a format suitable for feature engineering. It helps orchestrate and automate these transformations, ensuring that data is clean, consistent, and ready for ML applications. The combination of Snowflake’s powerful AI Data Cloud and dbt’s transformation prowess makes it an unbeatable pair for organizations aiming to scale their ML operations efficiently.

Making it easier for ML/Data Engineers to both build & deploy ML data & models​

dbt is a perfect tool to promote collaboration between data engineers, ML engineers, and data scientists. dbt is designed to support collaboration and quality of data pipelines through features including version control, environments and development life cycles, as well as built-in data and pipeline testing. Leveraging dbt means that data engineers and data scientists can collaborate and develop new models and features while maintaining the rigorous governance and high quality that's needed.

Additionally, dbt Mesh makes maintaining domain ownership extremely easy by breaking up portions of our data projects and pipelines into connected projects where critical models can be published for consumption by others with strict data contracts enforcing quality and governance. This paradigm supports rapid development as each project can be kept to a maintainable size for its contributors and developers. Contracting on published models used between these projects ensures the consistency of the integration points between them.

Finally, dbt Cloud also provides dbt Explorer — a perfect tool to catalog and share knowledge about organizational data across disparate teams. dbt Explorer provides a central place for information on data pipelines, including lineage information, data freshness, and quality. Best of all, dbt Explorer updates every time dbt jobs run, ensuring this information is always up-to-date and relevant.

What tech is at play?​

Here’s what you need from dbt. dbt should be used to manage data transformation pipelines and generate the datasets needed by ML engineers and data scientists maintaining the Snowflake Feature Store. dbt Cloud Enterprise users should leverage dbt Mesh to create different projects with clear owners for these different domains of data pipelines. This Mesh design will promote easier collaboration by keeping each dbt project smaller and more manageable for the people building and maintaining it. dbt also supports both SQL and Python-based transformations making it an ideal fit for AI/ML workflows, which commonly leverage both languages.

Using dbt for the data transformation pipelines will also ensure the quality and consistency of data products, which is critical for ensuring successful AI/ML efforts.

Snowflake ML overview​

The Feature Store is one component of Snowflake ML’s integrated suite of machine learning features that powers end-to-end machine learning within a single platform. Data scientists and ML engineers leverage ready-to-use ML functions or build custom ML workflows all without any data movement or without sacrificing governance. Snowflake ML includes scalable feature engineering and model training capabilities. Meanwhile, the Feature Store and Model Registry allow teams to store and use features and models in production, providing an end-to-end suite for operating ML workloads at scale.

What do you need to do to make it all work?​

dbt Cloud offers the fastest and easiest way to run dbt. It offers a Cloud-based IDE, Cloud-attached CLI, and even a low-code visual editor option (currently in beta), meaning it’s perfect for connecting users across different teams with different workflows and tooling preferences, which is very common in AI/ML workflows. This is the tool you will use to prepare and manage data for AI/ML, promote collaboration across the different teams needed for a successful AI/ML workflow, and ensure the quality and consistency of the underlying data that will be used to create features and train models.

Organizations interested in AI/ML workflows through Snowflake should also look at the new dbt Snowflake Native App — a Snowflake Native Application that extends the functionality of dbt Cloud into Snowflake. Of particular interest is Ask dbt — a chatbot that integrates directly with Snowflake Cortex and the dbt Semantic Layer to allow natural language questions of Snowflake data.

How to power ML pipelines with dbt and Snowflake’s Feature Store​

Let’s provide a brief example of what this workflow looks like in dbt and Snowflake to build and use the powerful capabilities of a Feature Store. For this example, consider that we have a data pipeline in dbt to process customer transaction data. Various data science teams in the organization need to derive features from these transactions to use in various models, including to predict fraud and perform customer segmentation and personalization. These different use cases all benefit from having related features, such as the count of transactions or purchased amounts over different periods of time (for example, the last day, 7 days, or 30 days) for a given customer.

Instead of the data scientists building out their own workflows to derive these features, let’s look at the flow of using dbt to manage the feature pipeline and Snowflake’s Feature Store to solve this problem. The following subsections describe the workflow step by step.

Create feature tables as dbt models​

The first step consists of building out a feature table as a dbt model. Data scientists and data engineers plug in to existing dbt pipelines and derive a table that includes the underlying entity (for example, customer id, timestamp and feature values). The feature table aggregates the needed features at the appropriate timestamp for a given entity. Note that Snowflake provides various common feature and query patterns available here. So, in our example, we would see a given customer, timestamp, and features representing transaction counts and sums over various periods. Data scientists can use SQL or Python directly in dbt to build this table, which will push down the logic into Snowflake, allowing data scientists to use their existing skill set.

Window aggregations play an important role in the creation of features. Because the logic for these aggregations is often complex, let’s see how Snowflake and dbt make this process easier by leveraging Don’t Repeat Yourself (DRY) principles. We’ll create a macro that will allow us to use Snowflake’s range between syntax in a repeatable way:

{% macro rolling_agg(column, partition_by, order_by, interval='30 days', agg_function='sum') %}
   {{ agg_function }}({{ column }}) over (
       partition by {{ partition_by }}
       order by {{ order_by }}
       range between interval '{{ interval }}' preceding and current row
   )
{% endmacro %}

Now, we use this macro in our feature table to build out various aggregations of customer transactions over the last day, 7 days, and 30 days. Snowflake has just taken significant complexity away in generating appropriate feature values and dbt has just made the code even more readable and repeatable. While the following example is built in SQL, teams can also build these pipelines using Python directly.

select
   tx_datetime,
   customer_id,
   tx_amount,
   {{ rolling_agg("TX_AMOUNT", "CUSTOMER_ID", "TX_DATETIME", "1 days", "sum") }}
   as tx_amount_1d,
   {{ rolling_agg("TX_AMOUNT", "CUSTOMER_ID", "TX_DATETIME", "7 days", "sum") }}
   as tx_amount_7d,
   {{ rolling_agg("TX_AMOUNT", "CUSTOMER_ID", "TX_DATETIME", "30 days", "sum") }}
   as tx_amount_30d,
   {{ rolling_agg("TX_AMOUNT", "CUSTOMER_ID", "TX_DATETIME", "1 days", "avg") }}
   as tx_amount_avg_1d,
   {{ rolling_agg("TX_AMOUNT", "CUSTOMER_ID", "TX_DATETIME", "7 days", "avg") }}
   as tx_amount_avg_7d,
   {{ rolling_agg("TX_AMOUNT", "CUSTOMER_ID", "TX_DATETIME", "30 days", "avg") }}
   as tx_amount_avg_30d,
   {{ rolling_agg("*", "CUSTOMER_ID", "TX_DATETIME", "1 days", "count") }}
   as tx_cnt_1d,
   {{ rolling_agg("*", "CUSTOMER_ID", "TX_DATETIME", "7 days", "count") }}
   as tx_cnt_7d,
   {{ rolling_agg("*", "CUSTOMER_ID", "TX_DATETIME", "30 days", "count") }}
   as tx_cnt_30d
from {{ ref("stg_transactions") }}

Create or connect to a Snowflake Feature Store​

Once a feature table is built in dbt, data scientists use Snowflake’s snowflake-ml-python package to create or connect to an existing Feature Store in Snowflake. Data scientists can do this all in Python, including in Jupyter Notebooks or directly in Snowflake using Snowflake Notebooks.

Let’s go ahead and create the Feature Store in Snowflake:

from snowflake.ml.feature_store import (
    FeatureStore,
    FeatureView,
    Entity,
    CreationMode
)

fs = FeatureStore(
    session=session, 
    database=fs_db, 
    name=fs_schema, 
    default_warehouse='WH_DBT',
    creation_mode=CreationMode.CREATE_IF_NOT_EXIST,
)

Create and register feature entities​

The next step consists of creating and registering entities. These represent the underlying objects that features are associated with, forming the join keys used for feature lookups. In our example, the data scientist can register various entities, including for the customer, a transaction id, or other necessary attributes.

Let’s create some example entities.

customer = Entity(name="CUSTOMER", join_keys=["CUSTOMER_ID"])
transaction = Entity(name="TRANSACTION", join_keys=["TRANSACTION_ID"])
fs.register_entity(customer)
fs.register_entity(transaction)

Register feature tables as feature views​

After registering entities, the next step is to register a feature view. This represents a group of related features that stem from the features tables created in the dbt model. In this case, note that the feature logic, refresh, and consistency is managed by the dbt pipeline. The feature view in Snowflake enables versioning of the features while providing discoverability among teams.

# Create a dataframe from our feature table produced in dbt
customers_transactions_df = session.sql(f"""
    SELECT 
        CUSTOMER_ID,
        TX_DATETIME,
        TX_AMOUNT_1D,
        TX_AMOUNT_7D,
        TX_AMOUNT_30D,
        TX_AMOUNT_AVG_1D,
        TX_AMOUNT_AVG_7D,
        TX_AMOUNT_AVG_30D,
        TX_CNT_1D,
        TX_CNT_7D,
        TX_CNT_30D     
    FROM {fs_db}.{fs_data_schema}.ft_customer_transactions
    """)

# Create a feature view on top of these features
customer_transactions_fv = FeatureView(
    name="customer_transactions_fv", 
    entities=[customer],
    feature_df=customers_transactions_df,
    timestamp_col="TX_DATETIME",
    refresh_freq=None,
    desc="Customer transaction features with window aggregates")

# Register the feature view for use beyond the session
customer_transactions_fv = fs.register_feature_view(
    feature_view=customer_transactions_fv,
    version="1",
    #overwrite=True,
    block=True)

Search and discover features in the Snowflake UI​

Now, with features created, teams can view their features directly in the Snowflake UI, as shown below. This enables teams to easily search and browse features, all governed through Snowflake’s role-based access control (RBAC).

Example of Snowflake UI

Generate training dataset​

Now that the feature view is created, data scientists produce a training dataset that uses the feature view. In our example, whether the data scientist is building a fraud or segmentation model, they will retrieve point-in-time correct features for a customer at a specific point in time using the Feature Store’s generate_training_set method.

To generate the training set, we need to supply a spine dataframe, representing the entities and timestamp values that we will need to retrieve features for. The following example shows this using a few records, although teams can leverage other tables to produce this spine.

spine_df = session.create_dataframe(
    [
        ('1', '3937', "2019-05-01 00:00"), 
        ('2', '2', "2019-05-01 00:00"),
        ('3', '927', "2019-05-01 00:00"),
    ], 
    schema=["INSTANCE_ID", "CUSTOMER_ID", "EVENT_TIMESTAMP"])

train_dataset = fs.generate_dataset(
    name= "customers_fv",
    version= "1_0",
    spine_df=spine_df,
    features=[customer_transactions_fv],
    spine_timestamp_col= "EVENT_TIMESTAMP",
    spine_label_cols = []
)

Now that we have produced the training dataset, let’s see what it looks like.

Example of training dataset

Train and deploy a model​

Now with this training set, data scientists can use Snowflake Snowpark and Snowpark ML Modeling to use familiar Python frameworks for additional preprocessing, feature engineering, and model training all within Snowflake. The model can be registered in the Snowflake Model Registry for secure model management. Note that we will leave the model training for you as part of this exercise.

Retrieve features for predictions​

For inference, data pipelines retrieve feature values using the retrieve_feature_values method. These retrieved values can be fed directly to a model’s predict capability in your Python session using a developed model or by invoking a model’s predict method from Snowflake’s Model Registry. For batch scoring purposes, teams can build this entire pipeline using Snowflake ML. The following code demonstrates how the features are retrieved using this method.

infernce_spine = session.create_dataframe(
    [
        ('1', '3937', "2019-07-01 00:00"), 
        ('2', '2', "2019-07-01 00:00"),
        ('3', '927', "2019-07-01 00:00"),
    ], 
    schema=["INSTANCE_ID", "CUSTOMER_ID", "EVENT_TIMESTAMP"])

inference_dataset = fs.retrieve_feature_values(
    spine_df=infernce_spine,
    features=[customer_transactions_fv],
    spine_timestamp_col="EVENT_TIMESTAMP",
)

inference_dataset.to_pandas()

Here’s an example view of our features produced for model inferencing.

Example of training data set

Conclusion​

We’ve just seen how quickly and easily you can begin to develop features through dbt and leverage the Snowflake Feature Store to deliver predictive modeling as part of your data pipelines. The ability to build and deploy ML models, including integrating feature storage, data transformation, and ML logic within a single platform, simplifies the entire ML life cycle. Combining this new power with the well-established partnership of dbt and Snowflake unlocks even more potential for organizations to safely build and explore new AI/ML use cases and drive further collaboration in the organization.

The code used in the examples above is publicly available on GitHub. Also, you can run a full example yourself in this quickstart guide from the Snowflake docs.

But I have to be honest: I don’t care. But not for the reasons you think.

What is Iceberg?​

To have this conversation, we need to start with the same foundational understanding of Iceberg. Apache Iceberg is a high-performance open table format developed for modern data lakes. It was designed for large-scale datasets, and within the project, there are many ways to interact with it. When people talk about Iceberg, it often means multiple components including but not limited to:

1. Iceberg Table Format - an open-source table format with large-scale data. Tables materialized in iceberg table format are stored on a user’s infrastructure, such as S3 Bucket.
2. Iceberg Data Catalog - an open-source metadata management system that tracks the schema, partition, and versions of Iceberg tables.
3. Iceberg REST Protocol (also called Iceberg REST API) is how engines can support and speak to other Iceberg-compatible catalogs.

If you have been in the industry, you also know that everything I just wrote above about Iceberg could easily be replaced by Hive, Hudi, or Delta. This is because they were all designed to solve essentially the same problem. Ryan Blue (creator of Iceberg) and Michael Armbrust (creator of Delta Lake) recently sat down for this fantastic chat and said two points that resonated with me:

• “We never intended for people to pay attention to this area. It’s something we wanted to fix, but people should be able to not pay attention and just work with their data. Storage systems should just work.”
• “We solve the same challenges with different approaches.”

At the same time, the industry is converging on Apache Iceberg. Iceberg has the highest availability of read and write support.

Credit to Jacques at Sundeck for creating this fantastic chart of all the Iceberg Support

Snowflake launched Iceberg support in 2022. Databricks launched Iceberg support via Uniform last year. Microsoft announced Fabric support for Iceberg in September 2024 at Fabric Con. Customers are demanding interoperability, and vendors are listening.

Why does this matter? Standardization of the industry benefits customers. When the industry standardizes - customers have the gift of flexibility. Everyone has a preferred way of working, and with standardization — they can always bring their preferred tools to their organization’s data.

Just another implementation detail​

I’m not saying open table formats aren't important. The metadata management and performance make them very meaningful and should be paid attention to. Our users are already excited to use it to create data lakes to save on storage costs, create more abstraction from their computing, etc.

But when building data models or focusing on delivering business value through analytics, my primary concern is not how the data is stored—it's how I can leverage it to generate insights and drive decisions. The analytics development lifecycle is hard enough without having to take into every detail. dbt abstracts the underlying platform and lets me focus on writing SQL and orchestrating my transformations. It’s a feature that I don’t need to think about how tables are stored or optimized—I just need to know that when I reference dim_customers or fct_sales, the correct data is there and ready to use. It should just work.

Sometimes the details do matter​

While table formats are an implementation detail for data transformation — Iceberg can impact dbt developers when the implementation details aren’t seamless. Currently, using Iceberg requires a significant amount of upfront configuration and integration work beyond just creating tables to get started.

One of the biggest hurdles is managing Iceberg’s metadata layer. This metadata often needs to be synced with external catalogs, which requires careful setup and ongoing maintenance to prevent inconsistencies. Permissions and access controls add another layer of complexity—because multiple engines can access Iceberg tables, you have to ensure that all systems have the correct access to both the data files and the metadata catalog. Currently, setting up integrations between these engines is also far from seamless; while some engines natively support Iceberg, others require brittle workarounds to ensure the metadata is synced correctly. This fragmented landscape means you could land with a web of interconnected components.

Fixing it​

Today, we announced official support for the Iceberg table format in dbt. By supporting the Iceberg table format, it’s one less thing you have to worry about on your journey to adopting Iceberg.

With support for Iceberg Table Format, it is now easier to convert your dbt models using proprietary table formats to Iceberg by updating your configuration. After you have set up your external storage for Iceberg and connected it to your platforms, you will be able to jump into your dbt model and update the configuration to look something like this:

Iceberg Table Format Support on dbt for Snowflake

It is available on these adapters:

• Athena
• Databricks
• Snowflake
• Spark
• Starburst/Trino
• Dremio

As with the beauty of any open-source project, Iceberg support grew organically, so the implementations vary. However, this will change in the coming months as we converge onto one dbt standard. This way, no matter which adapter you jump into, the configuration will always be the same.

dbt the Abstraction Layer​

dbt is more than about abstracting away the DDL to create and manage objects. It’s also about ensuring an opinionated approach to managing and optimizing your data. That remains true for our strategy around Iceberg Support.

In our dbt-snowflake implementation, we have already started to enforce best practices centered around how to manage the base location to ensure you don’t create technical debt accidentally, ensuring your Iceberg implementation scales over time. And we aren’t done yet.

That said, while we can create the models, there is a lot of initial work to get to that stage. dbt developers must still consider the implementation, like how their external volume has been set up or where dbt can access the metadata. We have to make this better.

Given the friction of getting launched on Iceberg, over the coming months, we will enable more capabilities to empower users to adopt Iceberg. It should be easier to read from foreign Iceberg catalogs. It should be easier to mount your volume. It should be easier to manage refreshes. And you should also trust that permissions and governance are consistently enforced.

And this work doesn’t stop at Iceberg. The framework we are building is also compatible with other table formats, ensuring that whatever table format works for you is supported on dbt. This way — dbt users can also stop caring about table formats. It’s just another implementation detail.

In the past, this primarily looked like a team working in one dbt Project to create a set of transformed objects in their data platform.

As dbt was adopted by larger organizations and began to drive workloads at a global scale, it became clear that we needed mechanisms to allow teams to operate independently from each other, creating and sharing data models across teams — dbt Mesh.

dbt Mesh is powerful because it allows teams to operate independently and collaboratively, each team free to build on their own but contributing to a larger, shared set of data outputs.

The flexibility of dbt Mesh means that it can support a wide variety of patterns and designs. Today, let’s dive into one pattern that is showing promise as a way to enable teams working on very different dbt deployments to work together.

How Hybrid Mesh enables collaboration between dbt Core and dbt Cloud teams​

Scenario — A company with a central data team uses dbt Core. The setup is working well for that team. They want to scale their impact to enable faster decision-making, organization-wide. The current dbt Core setup isn't well suited for onboarding a larger number of less-technical, nontechnical, or less-frequent contributors.

The goal — Enable three domain teams of less-technical users to leverage and extend the central data models, with full ownership over their domain-specific dbt models.

• Central data team: Data engineers comfortable using dbt Core and the command line interface (CLI), building and maintaining foundational data models for the entire organization.

• Domain teams: Data analysts comfortable working in SQL but not using the CLI and prefer to start working right away without managing local dbt Core installations or updates. The team needs to build transformations specific to their business context. Some of these users may have tried dbt in the past, but they were not able to successfully onboard to the central team's setup.

Solution: Hybrid Mesh — Data teams can use dbt Mesh to connect projects across dbt Core and dbt Cloud, creating a workflow where everyone gets to work in their preferred environment while creating a shared lineage that allows for visibility, validation, and ownership across the data pipeline.

Each team will fully own its dbt code, from development through deployment, using the product that is appropriate to their needs and capabilities while sharing data products across teams using both dbt Core and dbt Cloud.

A before and after diagram highlighting how a Hybrid Mesh allows central data teams using dbt Core to work with domain data teams using dbt Cloud.

Creating a Hybrid Mesh is mostly the same as creating any other dbt Mesh workflow — there are a few considerations but mostly it just works. We anticipate it will continue to see adoption as more central data teams look to onboard their downstream domain teams.

A Hybrid Mesh can be adopted as a stable long-term pattern, or as an intermediary while you perform a migration from dbt Core to dbt Cloud.

How to build a Hybrid Mesh​

Enabling a Hybrid Mesh is as simple as a few additional steps to import the metadata from your Core project into dbt Cloud. Once you’ve done this, you should be able to operate your dbt Mesh like normal and all of our standard recommendations still apply.

Step 1: Prepare your Core project for access through dbt Mesh​

Configure public models to serve as stable interfaces for downstream dbt Projects.

• Decide which models from your Core project will be accessible in your Mesh. For more information on how to configure public access for those models, refer to the model access page.
• Optionally set up a model contract for all public models for better governance.
• Keep dbt Core and dbt Cloud projects in separate repositories to allow for a clear separation between upstream models managed by the dbt Core team and the downstream models handled by the dbt Cloud team.

Step 2: Mirror each "producer" Core project in dbt Cloud​

This allows dbt Cloud to know about the contents and metadata of your project, which in turn allows for other projects to access its models.

• Create a dbt Cloud account and a dbt project for each upstream Core project.
  • Note: If you have environment variables in your project, dbt Cloud environment variables must be prefixed with DBT_ (including DBT_ENV_CUSTOM_ENV_ or DBT_ENV_SECRET). Follow the instructions in this guide to convert them for dbt Cloud.
• Each upstream Core project has to have a production environment in dbt Cloud. You need to configure credentials and environment variables in dbt Cloud just so that it will resolve relation names to the same places where your dbt Core workflows are deploying those models.
• Set up a merge job in a production environment to run dbt parse. This will enable connecting downstream projects in dbt Mesh by producing the necessary artifacts for cross-project referencing.
  • Optional: Set up a regular job to run dbt build instead of using a merge job for dbt parse, and centralize your dbt orchestration by moving production runs to dbt Cloud. Check out this guide for more details on converting your production runs to dbt Cloud.
• Optional: Set up a regular job (for example, daily) to run source freshness and docs generate. This will hydrate dbt Cloud with additional metadata and enable features in dbt Explorer that will benefit both teams, including Column-level lineage.

Step 3: Create and connect your downstream projects to your Core project using dbt Mesh​

Now that dbt Cloud has the necessary information about your Core project, you can begin setting up your downstream projects, building on top of the public models from the project you brought into Cloud in Step 2. To do this:

• Initialize each new downstream dbt Cloud project and create a dependencies.yml file.

• In that dependencies.yml file, add the dbt project name from the dbt_project.yml of the upstream project(s). This sets up cross-project references between different dbt projects:

  # dependencies.yml file in dbt Cloud downstream project
  projects:
  - name: upstream_project_name
  

• Use cross-project references for public models in upstream project. Add version to references of versioned models:

  select * from {{ ref('upstream_project_name', 'monthly_revenue') }}
  

And that’s all it takes! From here, the domain teams that own each dbt Project can build out their models to fit their own use cases. You can now build out your Hybrid Mesh however you want, accessing the full suite of dbt Cloud features.

• Orchestrate your Mesh to ensure timely delivery of data products and make them available to downstream consumers.
• Use dbt Explorer to trace the lineage of your data back to its source.
• Onboard more teams and connect them to your Mesh.
• Build semantic models and metrics into your projects to query them with the dbt Semantic Layer.

Conclusion​

In a world where organizations have complex and ever-changing data needs, there is no one-size fits all solution. Instead, data practitioners need flexible tooling that meets them where they are. The Hybrid Mesh presents a model for this approach, where teams that are comfortable and getting value out of dbt Core can collaborate frictionlessly with domain teams on dbt Cloud.

Because of this, we’ve noticed when talking to dbt users that they want to adopt the Semantic Layer, but feel daunted by the idea of migrating their transformations to this new paradigm. The good news is that you do not need to make a huge one-time migration.

We’re here to discuss another way: building a Semantic Layer in pieces. Our goal is to make sure you derive increased leverage and velocity from each step on your journey. If you’re eager to start building but have limited bandwidth (like most busy analytics engineers), this one is especially for you.

System of a noun: deciding what happens where​

When you’re using the dbt Semantic Layer, you want to minimize the modeling that exists outside of dbt. Eliminate it completely if you can. Why?

• It’s duplicative, patchy, and confusing as discussed above.
• It’s less powerful.
• You can’t test it.
• Depending on the tool, oftentimes you can’t version control it.

What you want is a unified development flow that handles normalized transformation in dbt models and dynamic denormalization in the dbt Semantic Layer (meaning it dynamically combines and reshapes normalized data models into different formats whenever you need them).

This enables a more flexible consumption layer, meaning downstream tools (like AI or dashboards) can sit as directly on top of Semantic Layer-generated artifacts and APIs as possible, and focus on what makes them shine instead of being burdened by basic dynamic modeling and aggregation tasks. Any tool-specific constructs should typically operate as close to transparent pass-throughs as you can make them, primarily serving to surface metrics and dimensions from the Semantic Layer in your downstream tool. There may be exceptions of course, but as a general guiding principle this gets you the most dynamic denormalization ability, and thus value, from your Semantic Layer code.

So now we’ve established the system, let’s dig into the plan for how we can get there iteratively.

The plan: towards iterative velocity​

1. Identify a Data Product that is impactful Find something that is in heavy use and high value, but fairly narrow scope. Don’t start with a broad executive dashboard that shows metrics from across the company because you’re looking to optimize for migrating the smallest amount of modeling for the highest amount of impact that you can.

   For example, a good starting place would be a dashboard focused on Customer Acquisition Cost (CAC) that relies on a narrow set of metrics and underlying tables that are nonetheless critical for your company.

2. Catalog the models and their columns that service the Data Product, both in dbt and the BI tool, including rollups, metrics tables, and marts that support those. Pay special attention to aggregations as these will constitute metrics. You can reference this example Google Sheet for one-way you might track this.

3. Melt the frozen rollups in your dbt project, as well as variations modeled in your BI tool, into Semantic Layer code. We’ll go much more in-depth on this process, and we encourage you to read more about this tactical terminology (frozen, rollup, etc) in the link — it will be used throughout this article!

4. Create a parallel version of your data product that points to Semantic Layer artifacts, audit, and then publish. Creating in parallel takes the pressure off, allowing you to fix any issues and publish gracefully. You’ll keep the existing Data Product as-is while swapping the clone to be supplied with data from the Semantic Layer.

Elsa iterates rapidly.

These steps constitute an iterative piece you will ship as you progressively move code into your Semantic Layer. As we dig into how to do this, we’ll discuss the immediate value this provides to your team and stakeholders. Broadly, it enables you to drastically increase iteration velocity.

The process of melting static, frozen tables into more flexible, fluid, dynamic Semantic Layer code is not complex, but it’s helpful to dig into the specific steps in the process. In the next section, we’ll dive into what this looks like in practice so you have a solid understanding of the "what’s required".

This is the most technical, detailed, and specific section of this article, so make sure to bookmark it and reference it as often as you can until the process becomes as intuitive as regular modeling in dbt!

Migrating a chunk: step-by-step​

1. Identify target​

1. Identify a relatively normalized mart that is powering rollups in dbt. If you do your rollups in your BI tool, start there. But we recommend starting with the frozen tables in dbt first and moving through the flow of the DAG progressively, bringing logic in your BI tool into play last. This is because we want to iteratively break up these frozen concepts in such a way that we benefit from earlier parts of the chain being migrated already. Think "moving left-to-right in a big DAG" that spans all your tools.
   • ✅ orders, customers — these are basic concepts powering your business, so should be marts models materialized via dbt.
   • ❌ active_accounts_per_week — this is built on top of the above, and something we want to generate dynamically in the dbt Semantic Layer.
   • Put another way: customers and orders are normalized building blocks, active_accounts_per_week is a rollup and we always want to migrate those to the Semantic Layer.
     A frozen rollup built on normalized marts.

2. Catalog the inputs​

1. Identify normalized columns and ignore any aggregation columns for now. For example, order_id, ordered_at, customer_id, order_total are fields we want to put in our semantic model, a window function that sums customer_cac statically in the dbt model is not a field we want in our semantic model because we want to dynamically codify that calculation as a metric in the Semantic Layer.
   1. If you find in the next step that you can’t express a certain calculation in the Semantic Layer yet, use dbt to model it**.** This is the beauty of having your Semantic Layer code integrated in your dbt codebase, it’s easy to manage the push and pull of the line between the Transformation and Semantic Layers because you’re managing a cohesive set of code and tooling.

3. Write Semantic Layer code​

1. Start with the semantic model going through column by column and putting all identified columns from Step 2 into the 3 semantic buckets:
   1. Entities — these are the spine of your semantic concepts or objects, you can think of them as roughly correlating to IDs or keys that form the grain.
   2. Dimensions — these are ways of grouping and bucketing these objects or concepts, such as time and categories.
   3. Measures — these are numeric values that you want to aggregate such as an order total or number of times a user clicked an ad.
2. Create metrics for the aggregation columns we didn’t encode into the semantic model.
3. Now, identify a rollup you want to melt. Refer to the earlier example to help distinguish these types of models.
4. Repeat these steps for any other concepts that you need to create that rollup e.g. active_accounts_per_week may need both customers and orders.
5. Create metrics for the aggregation columns present in the rollup. If your rollup references multiple models, put metrics in the YAML file that is most closely related to the grain or key aggregation of the table. For example, active_accounts_per_week is aggregated at a weekly time grain, but the key metric counts customer accounts, so we’d want to put that metric in the customers.yml or sem_customers.yml file (depending on the naming system you prefer). If it also contained a metric aggregating total orders in a given week, we’d put that metric into orders.yml or sem_orders.yml.
6. Create saved queries with exports configured to materialize your new Semantic Layer-based artifacts into the warehouse in parallel with the frozen rollup. This will allow us to shift consumption tools and audit results.

4. Connect external tools in parallel​

1. Now, shift your external analysis tool to point at the Semantic Layer exports instead of the rollup. Remember, we only want to shift the pointers for the rollup that we’ve migrated, everything else should stay pointing to frozen rollups. We’re migrating iteratively in pieces!
   1. If your downstream tools have an integration with the Semantic Layer, you’ll want to set that up as well. This will allow not only declarative caching of common query patterns with exports but also easy, totally dynamic on-the-fly queries.
2. Once you’ve replicated the previous state of things, with the Semantic Layer providing the data instead of frozen rollups, now you’re ready to shift the transformations happening in your BI tool into the Semantic Layer, following the same process.
3. Finally, to feel the new speed and power you’ve unlocked, ask a stakeholder for a dimension or metric that’s on their wishlist for the data product you’re working with. Then, bask in the glory of amazing them when you ship it an hour later!

Deep impact​

The first time you turn around a newly sliced, diced, filtered, and rolled up metric table for a stakeholder in under an hour instead of a week, not only you, but the stakeholder will immediately feel the value and power of the Semantic Layer.

dbt Labs’ mission is to create and disseminate organizational knowledge. This process, and building a Semantic Layer generally, is about encoding organizational knowledge in such a way that it creates and disseminates leverage. Enabled by this process, you can start building your Semantic Layer today, without waiting for the magical capacity for a giant overhaul to materialize. Building iterative velocity as you progress, your team can finally make any BI tool deliver the way you need it to.

With dbt 1.8, dbt released support for Snowflake’s external access integrations further enabling the use of dbt + AI to enrich your data. This allows querying of external APIs within dbt Python models, a functionality that was required for dbt Cloud customer, EQT AB. Learn about why they needed it and how they helped build the feature and get it shipped!

Why did EQT require this functionality?​

by Filip Bryén, VP and Software Architect (EQT) and Sebastian Stan, Data Engineer (EQT)

EQT AB is a global investment organization and as a long-term customer of dbt Cloud, presented at dbt’s Coalesce 2020 and 2023.

Motherbrain Labs is EQT’s bespoke AI team, primarily focused on accelerating our portfolio companies' roadmaps through hands-on data and AI work. Due to the high demand for our time, we are constantly exploring mechanisms for simplifying our processes and increasing our own throughput. Integration of workflow components directly in dbt has been a major efficiency gain and helped us rapidly deliver across a global portfolio.

Motherbrain Labs is focused on creating measurable AI impact in our portfolio. We work hand-in-hand with leadership from our deal teams and portfolio company leadership but our starting approach is always the same: identify which data matters.

While we have access to reams of proprietary information, we believe the greatest effect happens when we combine that information with external datasets like geolocation, demographics, or competitor traction.

These valuable datasets often come from third-party vendors who operate on a pay-per-use model; a single charge for every piece of information we want. To avoid overspending, we focus on enriching only the specific subset of data that is relevant to an individual company's strategic question.

In response to this recurring need, we have partnered with Snowflake and dbt to introduce new functionality that facilitates communication with external endpoints and manages secrets within dbt. This new integration enables us to incorporate enrichment processes directly into our DAGs, similar to how current Python models are utilized within dbt environments. We’ve found that this augmented approach allows us to reduce complexity and enable external communications before materialization.

An example with Carbon Intensity: How does it work?​

In this section, we will demonstrate how to integrate an external API to retrieve the current Carbon Intensity of the UK power grid. The goal is to illustrate how the feature works, and perhaps explore how the scheduling of data transformations at different times can potentially reduce their carbon footprint, making them a greener choice. We will be leveraging the API from the UK National Grid ESO to achieve this.

To start, we need to set up a network rule (Snowflake instructions here) to allow access to the external API. Specifically, we'll create an egress rule to permit Snowflake to communicate with api.carbonintensity.org.

Next, to access network locations outside of Snowflake, you need to define an external access integration first and reference it within a dbt Python model. You can find an overview of Snowflake's external network access here.

This API is open and if it requires an API key, handle it similarly to managing secrets. More information on API authentication in Snowflake is available here.

For simplicity’s sake, we will show how to create them using pre-hooks in a model configuration yml file:

models:
  - name: external_access_sample
    config:
      pre_hook: 
        - "create or replace network rule test_network_rule type = host_port mode = egress value_list= ('api.carbonintensity.org.uk:443');"
        - "create or replace external access integration test_external_access_integration allowed_network_rules = (test_network_rule) enabled = true;"

Then we can simply use the new external_access_integrations configuration parameter to use our network rule within a Python model (called external_access_sample.py):

import snowflake.snowpark as snowpark
def model(dbt, session: snowpark.Session):
    dbt.config(
        materialized="table",
        external_access_integrations=["test_external_access_integration"],
        packages=["httpx==0.26.0"]
    )
    import httpx
    return session.create_dataframe(
            [{"carbon_intensity": httpx.get(url="https://api.carbonintensity.org.uk/intensity").text}]
    )

The result is a model with some json I can parse, for example, in a SQL model to extract some information:

{{
    config(
        materialized='incremental',
        unique_key='dbt_invocation_id'
    )
}}

with raw as (
    select parse_json(carbon_intensity) as carbon_intensity_json
    from {{ ref('external_access_demo') }}
)

select
    '{{ invocation_id }}' as dbt_invocation_id,
    value:from::TIMESTAMP_NTZ as start_time,
    value:to::TIMESTAMP_NTZ as end_time,
    value:intensity.actual::NUMBER as actual_intensity,
    value:intensity.forecast::NUMBER as forecast_intensity,
    value:intensity.index::STRING as intensity_index
from raw,
    lateral flatten(input => raw.carbon_intensity_json:data)

The result is a model that will keep track of dbt invocations, and the current UK carbon intensity levels.

Preview in dbt Cloud IDE of output

dbt best practices​

This is a very new area to Snowflake and dbt -- something special about SQL and dbt is that it’s very resistant to external entropy. The second we rely on API calls, Python packages and other external dependencies, we open up to a lot more external entropy. APIs will change, break, and your models could fail.

Traditionally dbt is the T in ELT (dbt overview here), and this functionality unlocks brand new EL capabilities for which best practices do not yet exist. What’s clear is that EL workloads should be separated from T workloads, perhaps in a different modeling layer. Note also that unless using incremental models, your historical data can easily be deleted. dbt has seen a lot of use cases for this, including this AI example as outlined in this external engineering blog post.

A few words about the power of Commercial Open Source Software​

In order to get this functionality shipped quickly, EQT opened a pull request, Snowflake helped with some problems we had with CI and a member of dbt Labs helped write the tests and merge the code in!

dbt now features this functionality in dbt 1.8+ and all Release tracks in dbt Cloud.

dbt Labs staff and community members would love to chat more about it in the #db-snowflake slack channel.

Since the early days of dbt, folks have been interested having MSFT data platforms. Huge shoutout to Mikael Ene and Jacob Mastel for their efforts back in 2019 on the original SQL Server adapters (dbt-sqlserver and dbt-mssql, respectively)

The journey for the Azure Synapse dbt adapter, dbt-synapse, is closely tied to my journey with dbt. I was the one who forked dbt-sqlserver into dbt-synapse in April of 2020. I had first learned of dbt only a month earlier and knew immediately that my team needed the tool. With a great deal of assistance from Jeremy and experts at Microsoft, my team and I got it off the ground and started using it. When I left my team at Avanade in early 2022 to join dbt Labs, I joked that I wasn’t actually leaving the team; I was just temporarily embedding at dbt Labs to expedite dbt Labs getting into Cloud. Two years later, I can tell my team that the mission has been accomplished! Kudos to all the folks who have contributed to the TSQL adapters either directly in GitHub or in the community Slack channels. The integration would not exist if not for you!

Fabric Best Practices​

With the introduction of dbt Cloud support for Microsoft Fabric and Azure Synapse Analytics Dedicated SQL Pools, we're opening up new possibilities for analytics engineers in the Microsoft Ecosystem.

The goal of this blog is to ensure a great experience for both

• end-user data analysts who rely upon the data products built with dbt and
• the analytics engineers, who should predominately spend time creating and maintaining data products instead of maintaining and spinning up infrastructure
• data engineers who focus on data movement and ingestion into Synapse

To achieve this goal, this post will cover four main areas

• Microsoft Fabric: the future of data warehousing in the Microsoft/Azure stack
• strategic recommendations for provisioning Synapse environment
• data modeling in dbt: Synapse style
• Considerations for upstream and downstream of a Synapse-backed dbt project

With that, let’s dive in!

Fabric is the future​

Many data teams currently use Azure Synapse dedicated pools. However, Fabric Synapse Data Warehouse is the future of data warehousing in the Microsoft Ecosystem. Azure Synapse Analytics will remain available for a few more years, but Microsoft’s main focus is on Fabric as we can see in their roadmap and launches.

Because data platform migrations are complex and time-consuming, it’s perfectly reasonable to still be using dbt with Azure Synapse for the next two years while the migration is under way. Thankfully, if your team already is using ASADSP, transitioning to the new Cloud offering will be much more straightforward than the migration from on-premise databases to the Cloud.

In addition, if you're already managing your Synapse warehouse with a dbt project, you'll benefit from an even smoother migration process. Your DDL statements will be automatically handled, reducing the need for manual refactoring.

Bottom line, Fabric is the future of data warehousing for Microsoft customers, and Synapse is will be deprecated at an as-of-yet undeclared End-of-Life.

There’s undeniable potential offered by Fabric with it’s:

• fully-separated storage and compute, and
• pay-per-second compute.

These two things alone greatly simplify the below section on Resource Provisioning.

For more information, see:

• the official guide: Migration: Azure Synapse Analytics dedicated SQL pools to Fabric.
• this blog about the Future of Azure Synapse Analytics

Resource Provisioning​

Here are some considerations if you’re setting up an environment from scratch. If the infrastructure of multiple Synapse dedicated SQL pools and a Git repo already exist, you can skip to the next section, though a review of the below as a refresher wouldn’t hurt.

minimize pools; maximize DWUs​

definitions​

• dedicated SQL pools: effectively one data warehouse
• Data warehouse units (DWUs): the size of the cluster

number of pools​

With Synapse, a warehouse is both storage and compute. That is to say, to access data, the cluster needs to be on and warmed up.

If you only have one team of analytics engineers, you should have two SQL pools: one for development and one for production. If you have multiple distinct teams that will be modeling data in Synapse using dbt, consider using dbt Cloud’s Mesh paradigm to enable cross team collaboration.

Each should be at the highest tier that you can afford. You should also consider purchasing “year-long reservations” for a steep discount.

Some folks will recommend looking into scaling up and down pools based on demand. However, I’ve learned from personal experience that this optimization is not a free lunch and will require significant investment to not only build out but maintain. A large enough instance that is on whenever needed, keeps at least half an engineers time free to work on actual data modeling rather than platform maintenance.

DWUs​

The starting tier is DW100c, which costs $1.20/hour, has limitations such as only allowing 4 concurrent queries. To add 4 concurrent queries, you must increase the DWH tier. For every increase in 100 c's, you gain an additional 4 concurrent queries.

If this warehouse is intended to be the single source of truth for data analysts, you should design it to perform for that use case. In all likelihood, that means paying for a higher tier. Just like the above discussed potential for saving money by turning the cluster on and off as needed, paying for a lower tier, introduces another host of problems. If the limitation of 4 concurrent queries becomes a bottleneck, your choice is to either

• design infrastructure to push the data out of Synapse and into a Azure SQL db or elsewhere
• increase the tier of service paid (i.e. increase the DWUs)

I’m of the opinion that minimizing Cloud spend should not come at the expense of developer productivity — both sides of the equation need to be considered. As such, I advocate predominately for the latter of the above two choices.

Deployment Resources​

In the Microsoft ecosystem, data warehouse deployments are more commonly conducted with Azure Data Factory instead of Azure DevOps pipelines or GitHub Actions. We recommend separating dbt project deployments from any ingestion pipeline defined in ADF.

However, if you must use ADF as the deployment pipeline, it is possible to use dbt Cloud APIs. Running dbt Core within Azure Data Factory can be challenging as there’s no easy way to install and invoke dbt Core, because there’s no easy way to install and run Python. The workarounds aren’t great, for example: Setting up dbt calls via Azure Serverless Functions and invoking them from ADF.

access control​

permissions for analytics engineers​

In the development warehouse, each user should have the following privileges: EXECUTE, SELECT, INSERT, UPDATE, and DELETE.

service principal permissions​

In addition, a service principal is required for dbt Cloud to directly interact with both the warehouse and your Git service provider (e.g. GitHub or Azure DevOps).

Only the Service Principal in charge of deployment has the above permissions in production. End users have only SELECT access to this environment.

Model Considerations​

The magic begins when the environments are provisioned and dbt Cloud is connected.

With dbt on Synapse, you can own the entire data transformation workflow from raw data to modeled data that data analysts and end users rely upon. The end product of which will be documented and tested.

With dbt Cloud, things are even more streamlined. The dbt Cloud CLI allows developers to build only the models they need for a PR, deferring to the production environment for dependencies. There’s also dbt Explorer, which now has column-level lineage.

While there are already platform-agnostic best practice guides that still apply for Synapse, there are some additional factors related to data distribution and indexing.

distributions & indices​

Working in ASADSP, it is important to remember that you’re working in a Massively-Parallel Processing (MPP) architecture.

What this means for an analytics engineer working using dedicated SQL pools is that for every table model, it must have an index and distribution configured. In dbt-synapse the defaults are:

• index: CLUSTERED COLUMNSTORE INDEX
• distribution ROUND_ROBIN

If you want something different, you can define it like below. For more information, see dbt docs: configurations for Azure Synapse DWH: Indices and distributions.

{{
    config(
        index='HEAP',
        dist='ROUND_ROBIN'
        )
}}
SELECT * FROM {{ ref('some_model') }}

A distribution specifies how the table rows should be stored across the 60 nodes of the cluster. The goal is to provide a configuration that both:

1. ensures data is split evenly across the nodes of the cluster, and
2. minimizes inter-node movement of data.

For example, imagine querying a 100-row seed table in a downstream model. Using distribution=ROUND_ROBIN instructs the pool to evenly distribute the rows between the 60 node, which equates to having only one or two rows in each node. This SELECT-ing all these an operation that touches all 60 nodes. The end result is that the query will run much slower than you might expect.

The optimal distribution is REPLICATE which will load a full copy of the table to every node. In this scenario, any node can return the 100 rows without coordination from the others. This is ideal for a lookup table which could limit the result set within each node before aggregating each nodes results.

more information​

• Guidance for designing distributed tables using dedicated SQL pool in Azure Synapse Analytics
• source code for synapse__create_table_as() macro

Deployments & Ecosystem​

With the infrastructure in place and the analytics engineers enabled with best practices, the final piece is to think through how a dbt project sits in the larger data stack of your organization both upstream and downstream.

Upstream​

In dbt, we assume the data has already been ingested into the warehouse raw. This follows a broader paradigm known as Extract-Load-Transform (ELT). The same goes for dbt with Azure Synapse. The goal should be to have the data ingested into Synapse that is as “untouched” as possible from when it came from the upstream source system. It’s common for data teams using Azure Data Factory to continue to imploy an ETL-paradigm where data is transformed before it even lands in the warehouse. We do not recommend this, as it results in critical data transformation living outside of the dbt project, and therefore undocumented.

If you have not already, engage the central/upstream data engineering team to devise a plan to integrate data extraction and movement in tools such as SSIS and Azure Data Factory with the transformation performed via dbt Cloud.

Downstream Consumers (Power BI)​

It is extremely common in MSFT data ecosystem to have significant amounts of data modeling live within Power BI reports and/or datasets. This is ok up to a certain point.

The correct approach is not to mandate that all data modeling should be done in dbt with SQL. Instead seek out the most business critical Power BI datasets and reports. Any modeling done in those reports should be upstreamed into the dbt project where it can be properly tested and documented.

There should be a continuous effort to take and Power Query code written in PBI as transformation code and to upstream it into the data warehouse where the modeling can be tested, documented, reused by others and deployed with confidence.

Conclusion​

There’s great opportunity in dbt Cloud today for data teams using Azure Synapse. While Fabric is the future, there’s meaningful considerations when it comes to resource provisioning, model design, and deployments within the larger ecosystem.

As we look ahead, we're excited about the possibilities that Microsoft Fabric holds for the future of data analytics. With dbt Cloud and Azure Synapse, analytics engineers can be disseminate knowledge with confidence to the rest of their organization.

Here's the one I had last night:

It began with a midnight bug hunt. A menacing insect creature has locked my colleagues in a dungeon, and they are pleading for my help to escape . Finding the key is elusive and always seems just beyond my grasp. The stress is palpable, a physical weight on my chest, as I raced against time to unlock them.

Of course I wake up without actually having saved them, but I am relieved nonetheless. And I've had similar nightmares involving a heroic code refactor or the launch of a new model or feature.

Good news: beginning in dbt v1.8, we're introducing a first-class unit testing framework that can handle each of the scenarios from my data nightmares.

Before we dive into the details, let's take a quick look at how we got here.

Story of data quality in dbt​

The underlying reason behind my bad dreams is worry about unfortunate data quality that affects shared outcomes.

One of the things I loved right away when I first started using dbt was that it had a first-class mechanism for asserting data quality on our full production data in the form of data tests.

I no longer had to worry about whether or not my primary key was actually unique, I could just add a dbt data test to assert that expectation!

dbt test quickly became a beloved command, allowing me to run our full suite of data quality tests in production each day. And these same tests would run in CI and development.

But while this mechanism is tremendously useful at a holistic level, it doesn't lend itself as well at the granular level. It was not designed to handle minimal test cases for a model with with fixed inputs and the expected output from those inputs. Nor was it designed to handle isolated test cases that can run simultaneously for the same model.

So it doesn't meet the standard software engineering use-case of setting up and running individual test cases and other desireable properties.

Introducing unit testing in dbt​

dbt version 1.8 marks the introduction of a built-in unit testing framework to extend the capabilities of software engineering best practices for analytics engineers. It allows for crafting isolated and repeatable unit tests that are well-suited to execute during development and CI. They are useful in a variety of scenarios like responding to bug reports, confident code refactoring, and using test-driven development when adding new features.

Let's dive into the details...

Hello, unit testing world​

Hello unit testing world

A key way that I build self-confidence is starting out with the simplest example possible. Once I've gotten the initial thing to work, then I can tweak it to take on more complicated use-cases (scroll down to the "real world example" section below for something more realistic!). So here's a super simple example that you can use to get your feet wet. Afterwards, I'll explain more about each of the main components and how you can apply them to your own test cases.

First, create this trivial model:

-- models/hello_world.sql

select 'world' as hello

Then, add a simple unit test for that model:

# models/_properties.yml

unit_tests:
  - name: test_hello_world

    # Always only one transformation to test
    model: hello_world

    # No inputs needed this time!
    # Most unit tests will have inputs -- see the "real world example" section below
    given: []

    # Expected output can have zero to many rows
    expect:
      rows:
        - {hello: world}

Finally, run the model and all its tests in a single command like this:

dbt build --select hello_world

Terminal output of hello world unit test

Voilà! We can see that a single unit test ran and it passed.

Crafting Unit Tests​

After you've run your first "hello, world" unit test, you'll want to get started writing your own. There's two things that will help you be successful:

1. How to think about a unit test conceptually
2. How to actually craft your unit tests in YAML

Here's a step-by-step guide for you to follow:

Organizing your thoughts​

1. Identify your scenarios: Which scenarios do you want to be more confident about? For each scenario, what is the relevant model? Consider edge cases: which inputs might be tricky for that model to handle correctly? This will identify your model and given inputs.
2. Define the success criteria: What is the expected output for each scenario? Be specific. This will identify your expected output.

Writing your unit tests​

1. Start with a "model-inputs-output" structure: When running this model, given these test inputs, then expect this output.
2. Use meaningful descriptions: They should clearly explain what the test is doing so collaborators and future developers can understand the purpose.
3. Test one behavior per test case: This keeps tests focused and easier to debug.

Additional tips:

• Think about maintainability: Write tests that are easy to understand and update.
• Refactor tests as needed: Keep them up-to-date with code changes.
• Practice test-driven development (TDD): Write tests before writing code to guide your development process.
• Remember, unit testing is just one part of quality assurance. Combine it with other testing methods like data tests and model contracts for a comprehensive approach.

Next, I'll show you a brief example from the "real" world.

Real world example​

When we were trying out the developer experience and ergonomics of unit testing in dbt, we went to our trusty Jaffle Shop repo. We began to follow the framework above to identify scenarios and then define the success criteria.

The first scenario we considered was counting the number of food items and drink items within an order. One natural edge case is an order without any drinks. Our success criteria in this case is for count_drink_items to be 0 in the order_items_summary model.

To implement the unit test, we started by starting with a "model-inputs-output" (MIO) structure above. The relevant model was orders with given inputs were from order_items and stg_orders. In this case, we expect our output for order_id 2 to be count_drink_items: 0.

Here's what the unit test YAML looked like:

Unit test YAML​

unit_tests:

  - name: test_order_items_count_drink_items_with_zero_drinks
    description: >
      Scenario: Order without any drinks
        When the `order_items_summary` table is built
        Given an order with nothing but 1 food item
        Then the count of drink items is 0

    # Model
    model: order_items_summary

    # Inputs
    given:
      - input: ref('order_items')
        rows:
          - {
              order_id: 76,
              order_item_id: 3,
              is_drink_item: false,
            }
      - input: ref('stg_orders')
        rows:
          - { order_id: 76 }

    # Output
    expect:
      rows:
        - {
            order_id: 76,
            count_drink_items: 0,
          }

Suffice it to say that when we ran the unit test for the first time, it failed! 💥

But it wasn't because we defined the unit test incorrectly – it was because we found a bug that we didn't know about previously. To get things back on the right path, we opened a PR that added the relevant unit test to confirm the bug as well as the bug fix. The good news is that by implementing the unit test, we were able to find a bug before someone else did. 😎

If you're curious about what the model looked like before and the code changes for the fix, here you go:

Original SQL code​

with

order_items as (

    select * from {{ ref('order_items') }}

)

select
    order_id,

    sum(supply_cost) as order_cost,
    sum(product_price) as order_items_subtotal,
    count(order_item_id) as count_order_items,
    count(
        case
            when is_food_item then 1
            else 0
        end
    ) as count_food_items,
    count(
        case
            when is_drink_item then 1
            else 0
        end
    ) as count_drink_items

from order_items

group by 1

SQL Code fix​

17c17
<     count(
---
>     sum(
23c23
<     count(
---
>     sum(

Caveats and pro-tips​

See the docs for helpful information before you begin, including unit testing incremental models, models that depend on ephemeral model(s), and platform-specific considerations like STRUCTs in BigQuery. In many cases, the sql format can help solve tricky edge cases that come up.

Another advanced topic is overcoming issues when non-deterministic factors are involved, such as a current timestamp. To ensure that the output remains consistent regardless of when the test is run, you can set a fixed, predetermined value by using the overrides configuration.

Before we wrap up, let's do a brief comparison of the different data quality capabilties in dbt and identify the situations where each would be most effective.

Unit tests vs. model contracts vs. data tests​

dbt has multiple complementary features that support data quality including unit tests, model contracts, and data tests. Here's a table of how they compare and when you might use each:

Summary​

You're now ready to build your first unit tests with this new feature coming to dbt in v1.8! We're eager for you to try this out – let us know how it works for you by commenting in this discussion or opening an issue.

There's more details about the syntax which you can access in our documentation. We hope this gives you the tools to boost your confidence in your data pipelines and sleep easier at night 😴

As a solutions architect at dbt Labs, my role is to help our customers and prospects understand how to best utilize the dbt Cloud platform to solve their unique data challenges. That uniqueness presents itself in different ways - organizational maturity, data stack, team size and composition, technical capability, use case, or some combination of those. With all those differences though, there has been one common thread throughout most of my engagements: Generative AI and Large Language Models (LLMs). Data teams are either 1) proactively thinking about applications for it in the context of their work or 2) being pushed to think about it by their stakeholders. It has become the elephant in every single (zoom) room I find myself in.

Gen AI Everywhere!

Clearly, this technology is not going away. There are already countless number of use cases and applications already showing very real improvements to efficiency, productivity, and creativity. Inspired by the common problem faced by data teams I'm talking to, I built a Streamlit app which uses Snowflake Cortex and the dbt Semantic Layer to answer free-text questions accurately and consistently. You can preview examples of the questions it's able to answer below:

Why Build This​

So, why build this and what makes it different?

• The outcome of an application like this aligns incredibly well with the mandate of most data teams - empower stakeholders by providing them with the data they need, in a medium they can consume, all while considering aspects of trust, governance, and accuracy
• The accuracy component is the very unique value proposition of an application like this relative to any other solution out there that purports to write SQL from a text prompt (check out some early benchmarks here). The reason for that is we’re not asking the LLM to write a SQL query, which is prone to hallucinating tables, columns, or just SQL that’s not valid. Instead, it generates a highly structured MetricFlow request. MetricFlow is the underlying piece of technology in the semantic layer that will translate that request to SQL based on the semantics you’ve defined in your dbt project.
• If I’m being honest, it’s also an incredibly valuable tool to show our customers and prospects!

The Tech​

The application uses dbt Cloud’s Semantic Layer alongside Snowflake Cortex and Streamlit to power a natural language interface that enables users to retrieve data from their Snowflake platforms by simply asking questions like “What is total revenue by month in 2024?”. Before we go too deep, let’s review what these tools are:

Prerequisites​

Snowflake​

Within Snowflake, you’ll need the following:

The required privileges for Snowflake Cortex are laid out in detail here, but at a high level you’ll need to grant the SNOWFLAKE.CORTEX_USER database role to the user(s) accessing any of the functions available via Cortex. For example:

use role accountadmin;

create role cortex_user_role;
grant database role snowflake.cortex_user to role cortex_user_role;

grant role cortex_user_role to user some_user;

To create streamlit apps within Snowflake, you need to grant the CREATE STREAMLIT privilege. An example script is below:

-- If you want all roles to create Streamlit apps, run
grant usage on database <database_name> to role public;
grant usage on schema <database_name>.<schema_name> to role public;
grant create streamlit on schema <database_name>.<schema_name> to role public;
grant create stage on schema <database_name>.<schema_name> to role public;

-- Don't forget to grant USAGE on a warehouse (if you can).
grant usage on warehouse <warehouse_name> to role public;

-- If you only want certain roles to create Streamlit apps, 
-- change the role name in the above commands.

Additionally, you’ll need to set up a network rule, an external access integration, and a UDF that makes a request to the dbt Cloud Semantic Layer. Be mindful of the values you have in your network rule and UDF - they'll need to correspond to the host where your dbt Cloud account is deployed.

grant create network rule on schema <database_name>.<schema_name> to role public;
grant create secret on schema <database_name>.<schema_name> to role public;

use database <database_name>;
use schema <schema_name>;

create or replace network rule dbt_cloud_semantic_layer_rule
    mode = egress
    type = host_port
    value_list = (
        'semantic-layer.cloud.getdbt.com',
        'semantic-layer.emea.dbt.com',
        'semantic-layer.au.dbt.com'
    );

create or replace secret dbt_cloud_service_token
    type = generic_string
    secret_string = '<service_token>';

create or replace external access integration dbt_cloud_semantic_layer_integration
    allowed_network_rules = (dbt_cloud_semantic_layer_rule)
    allowed_authentication_secrets = (dbt_cloud_service_token)
    enabled = true;

grant usage on integration dbt_cloud_semantic_layer_integration to role public;
grant ownership on secret dbt_cloud_service_token to role public;
grant usage on secret dbt_cloud_service_token to role public;

The UDFs are called out individually in further sections below.

dbt Cloud​

Within dbt Cloud, you’ll need the following (more detail can be found here):

• Have a dbt Cloud Team or Enterprise account. Suitable for both Multi-tenant and Single-tenant deployment.
• Have both your production and development environments running dbt version 1.6 or higher.
• Create a successful job run in the environment where you configure the Semantic Layer.

The Code​

There are several components to the application that are worth calling out here individually: retrieving your project’s semantics (specifically metrics and dimensions) when the application loads, examples that guide the LLM to what valid and invalid output looks like, parsing the output to a structured object, and then using that output as an argument in the UDF we built earlier that makes a request to the Semantic Layer.

Retrieving Semantics​

When we create our prompt for the LLM, we’ll need to pass in the relevant metrics and dimensions that have been defined in our dbt project. Without this, the LLM wouldn’t have the relevant information to parse when a user asks their particular question. Additionally, this is an external request to dbt Cloud’s Semantic Layer API, so we’ll need to use an existing UDF. Again, make sure you update the url to match your deployment type. Also, note that we're using the external access integration and secret that we created earlier.

create or replace function retrieve_sl_metadata()
    returns object
    language python
    runtime_version = 3.9
    handler = 'main'
    external_access_integrations = (dbt_cloud_semantic_layer_integration)
    packages = ('requests')
    secrets = ('cred' = dbt_cloud_service_token)
as
$$
from typing import Dict
import _snowflake
import requests

query = """
query GetMetrics($environmentId: BigInt!) {
  metrics(environmentId: $environmentId) {
    description
    name
    queryableGranularities
    type
    dimensions {
      description
      name
      type
    }
  }
}
"""

def main():
    session = requests.Session()
    token = _snowflake.get_generic_secret_string('cred')
    session.headers = {'Authorization': f'Bearer {token}'}

    # TODO: Update for your environment ID
    payload = {"query": query, "variables": {"environmentId": 1}}

    # TODO: Update for your deployment type
    response = session.post("https://semantic-layer.cloud.getdbt.com/api/graphql", json=payload)
    response.raise_for_status()
    return response.json()

$$;

grant usage on function retrieve_sl_metadata() to role public;

Couple of things to note about the code above:

• Make sure you update the code to include your environment ID and your URL that’s specific to your deployment type. You could modify the function to accept arguments for payload, variables, query, etc. to make it more dynamic and satisfy other use cases outside of this one.
• Once the data has been returned, we’re going to use streamlit’s session state feature to store the dbt project’s defined metrics and dimensions. This feature will allow us to make multiple calls without having to continually retrieve this metadata.

Creating Examples​

Aside from using the metrics and dimensions that we retrieved in the step above, we’re also going to use in the prompt, examples of questions a user would ask and what the corresponding output should look like. This allows us to “train” the LLM and ensure we can accommodate the various ways people ask questions. An example of this is guiding the LLM in how it can structure SQL used in a where clause when a question is time-based (e.g. “Give me year-to-date revenue by department”). An example of this might look like:

{
    "metrics": "revenue, costs, profit",
    "dimensions": "department, salesperson, cost_center, metric_time, product__product_category",
    "question": "Give me YTD revenue by department?",
    "result": Query.model_validate(
        {
            "metrics": [{"name": "revenue"}],
            "groupBy": [{"name": "department"}],
            "where": [
                {
                    "sql": "{{ TimeDimension('metric_time', 'DAY') }} >= date_trunc('year', current_date)"
                }
            ],
        }
    )
    .model_dump_json()
    .replace("{", "{{")
    .replace("}", "}}"),
}

There is a tradeoff with this approach though that is worth mentioning - the examples we use to guide the LLM will be used in the prompt and thus increase the number of tokens processed, which is how Snowflake’s Cortex functions measure compute cost. For some context, the LLM used in this application is mistral-8x7b, which charges .22 Credits / 1M Tokens and has a context window of 32,000 tokens.

Structured Object Parsing​

Another important piece to this application is parsing the output from the LLM into a structured object, specifically a Pydantic model. As I was building out this application, I continually ran into problems with the LLM. The problem was not providing correct responses, which it did, but responses that had the same structure and continuity from question to question. Even trying very explicit instructions in the prompt like “Only return relevant metrics and dimensions” or “Do not explain your output in any way”, I continued to receive output that made it hard to parse and then extract the relevant information to form an appropriate request to the semantic layer. This led me to LangChain and the PydanticOutputParser, which allowed me to specify an arbitrary Pydantic Model and make the output from the LLM conform to that schema.

class Query(BaseModel):
    metrics: List[MetricInput]
    groupBy: Optional[List[GroupByInput]] = None
    where: Optional[List[WhereInput]] = None
    orderBy: Optional[List[OrderByInput]] = None
    limit: Optional[int] = None

The beauty of this approach is that I can create the individual attributes that form a query, like metrics or groupBy, and create individual Pydantic models for each of those that map to the schema that the GraphQL API expects. Once I get it into this format, it becomes very easy to create the API request to finally return data from my snowflake warehouse that answers the question the user asked.

Retrieving Data​

Once my Query object has been created, I can use that output to both form the GraphQL query and the relevant variables to be used in the payload. This payload will be the argument we pass to the UDF that we created earlier to 1) create a query via the Semantic Layer and 2) using that query ID, poll until a completion event and return the data back to the Streamlit application. This is again an external request to the dbt Cloud Semantic Layer so a UDF will be used.

create or replace function submit_sl_request(payload string)
    returns object
    language python
    runtime_version = 3.9
    handler = 'main'
    external_access_integrations = (dbt_cloud_semantic_layer_integration)
    packages = ('requests')
    secrets = ('cred' = dbt_cloud_service_token )
as
$$
from typing import Dict
import _snowflake
import json
import requests


def main(payload: str):
    session = requests.Session()
    token = _snowflake.get_generic_secret_string('cred')
    session.headers = {'Authorization': f'Bearer {token}'}
    payload = json.loads(payload)
    results = submit_request(session, payload)
    try:
        query_id = results["data"]["createQuery"]["queryId"]
    except TypeError as e:
        raise e
    
    data = None
    while True:
        graphql_query = """
            query GetResults($environmentId: BigInt!, $queryId: String!) {
                query(environmentId: $environmentId, queryId: $queryId) {
                    arrowResult
                    error
                    queryId
                    sql
                    status
                }
            }
        """
        results_payload = {"variables": {"queryId": query_id}, "query": graphql_query}
        results = submit_request(session, results_payload)
        try:
            data = results["data"]["query"]
        except TypeError as e:
            break
        else:
            status = data["status"].lower()
            if status in ["successful", "failed"]:
                break

    return data

def submit_request(session: requests.Session, payload: Dict):
    if not "variables" in payload:
        payload["variables"] = {}
    payload["variables"].update({"environmentId": 1})
    response = session.post(
        "https://semantic-layer.cloud.getdbt.com/api/graphql", json=payload
    )
    response.raise_for_status()
    return response.json()
$$;

grant usage on function submit_sl_request(string) to role public;

Wrapping Up​

Building this application has been an absolute blast for multiple reasons. First, we’ve been able to use it internally within the SA org to demonstrate how the semantic layer works. It provides yet another integration point that further drives home the fundamental value prop of using the Semantic Layer. Secondly, and more importantly, it has served as an example to those customers thinking about (or being pushed to think about) how they can best utilize these technologies to further their goals. Finally, I’ve been able to be heads down, hands on keyboard learning about all of these interesting technologies and stepping back into the role of builder is something I will never turn down!

Finally, to see the entire code, from Snowflake to Streamlit, check out the repo here.

As long as dbt Cloud has existed, it has required users to select a version of dbt Core to use under the hood in their jobs and environments. This made sense in the earliest days, when dbt Core minor versions often included breaking changes. It provided a clear way for everyone to know which version of the underlying runtime they were getting.

However, this came at a cost. While bumping a project's dbt version appeared as simple as selecting from a dropdown, there was real effort required to test the compatibility of the new version against existing projects, package dependencies, and adapters. On the other hand, putting this off meant foregoing access to new features and bug fixes in dbt.

But no more. Today, we're ready to announce the general availability of a new option in dbt Cloud: the "Latest" release track.

For customers, this means less maintenance overhead, faster access to bug fixes and features, and more time to focus on what matters most: building trusted data products. This will be our stable foundation for improvement and innovation in dbt Cloud.

But we wanted to go a step beyond just making this option available to you. In this blog post, we aim to shed a little light on the extensive work we've done to ensure that using the "Latest" release track is a stable and reliable experience for the thousands of customers who rely daily on dbt Cloud.

How we safely deploy dbt upgrades to Cloud​

We've put in place a rigorous, best-in-class suite of tests and control mechanisms to ensure that all changes to dbt under the hood are fully vetted before they're deployed to customers of dbt Cloud.

This pipeline has in fact been in place since January! It's how we've already been shipping continuous changes to the hundreds of customers who've selected the "Latest" release track while it's been in Beta and Preview. In that time, this process has enabled us to prevent multiple regressions before they were rolled out to any customers.

We're very confident in the robustness of this process**. We also know that we'll need to continue building trust with time.** We're sharing details about this work in the spirit of transparency and to build that trust.

Any new change to dbt-core and adapters goes through the following steps before it's available to customers in dbt Cloud:

Testing and deploy pipeline

Step 1: Unit & functional tests in dbt Core + adapters​

First up is a battery of thousands of tests that we run dozens of times per day. No change, in either dbt-core or in the data platform adapters supported by dbt Cloud, is merged until it has passed this full suite of tests.

Here, unit tests test internal components in isolation from one another, and functional tests represent edge cases in expected behavior under known conditions.

For adapters, tests also ensure that the full matrix of data platform features continue to work as expected: BigQuery partitioning + incremental strategies, Snowflake data types + model contracts, Redshift sort keys — so on and so forth.

Step 2: Smoke testing​

Next, we create a Docker image with the latest dbt changes installed alongside each adapter supported in dbt Cloud. We run an additional suite of end-to-end tests on this image across a matrix of supported adapters, test projects that represent real-world complexity, popular third-party packages, and typical dbt user workflows. In doing so, this phase of testing also ensures that the latest version of dbt does not break compatibility with frequently relied-upon dbt packages.

This breadth of testing provides early detection of any regressions that might have been introduced by our changes to dbt-core, changes by adapter maintainers, or any of their dependencies and drivers — using the exact installed versions that would be deployed to dbt Cloud. Crucially, this helps safeguard us from breaking changes in third-party software.

Step 3: dbt Cloud service tests​

Before the new image version goes live, we ensure that all dbt changes are cross-compatible with every dbt Cloud service that depends on Core functionality, including areas such as the Cloud IDE, the Cloud CLI, scheduled job runs, CI, and connection testing.

For each dbt Cloud service, we run a testing suite that consists of:

• Unit and integration specific tests to behaviour of each dbt Cloud service
• End-to-end headless browser testing for our UI-heavier applications
• Compatibility for each adapter with that service

This step provides further depth in testing the interplay between dbt Core and dbt Cloud application-specific functionality, covering cases such as linting SQL that has an ephemeral reference, or resolving cross-project refs across multi-project "dbt Mesh" deployments.

Step 4: Canary deployment​

Once all the aforementioned tests have passed, we roll out the latest deployment to a small subset (5%) of accounts, including our own Internal Analytics project.

These "canary" deployments are continually monitored against a set of precise observability metrics. Metrics we monitor include overall job error and cancellation rates to ensure they don't deviate from our expectations relative to a stable baseline. Any anomaly immediately alerts us, and we can shut off the canary in a matter of seconds, keeping all accounts on the last stable version.

Step 5: Phased rollout​

Once the canary deployment has been proven to run stably for at least 24 hours, we mark it as eligible for all accounts to upgrade in their next scheduled deployment of dbt Cloud.

The outcome of this process is that, when you select the "Latest" release track in dbt Cloud, the time between an improvement being made to dbt Core and you safely getting access to it in your projects is a matter of days — rather than months of waiting for the next dbt Core release, on top of any additional time it may have taken to actually carry out the upgrade.

We’re pleased to say that, at the time of writing (May 2, 2024), since the beta launch of the "Latest" release track in dbt Cloud in March, we have not had any functional regressions reach customers, while we’ve also been shipping multiple improvements to dbt functionality every day. This is a foundation that we aim to build on for the foreseeable future.

Stability as a feature​

A rigorous testing pipeline in dbt Cloud is crucial, but real ongoing stability required some deeper changes in the dbt framework itself. We take our responsibility as the maintainers of dbt Core seriously, as well the open-source ecosystem around it.

We've taken a longer release cycle for the upcoming release of dbt Core v1.8 to revisit some of the "do later" design choices we made in the past — specifically around adapter compatibility, behaviour change management, and metadata artifacts.

Decoupling the adapter interface​

The adapter interface — i.e. how dbt Core actually connects to a third-party data platform — has historically been somewhat of a pain point. Adapter maintainers have often been required to make reactive changes when there's been an update to dbt Core.

To solve that, we've released a new set of interfaces that are entirely independent of the dbt-core library: dbt-adapters==1.0.0. From now on, any changes to dbt-adapters will be backward and forward-compatible. This also decouples adapter maintenance from the regular release cadence of dbt Core — meaning maintainers get full control over when they ship implementations of new adapter-powered features.

Note that adapters running in dbt Cloud must be migrated to the new decoupled architecture as a baseline in order to support the new "Latest" release track.

Managing behavior changes: stability as a feature​

We all want the benefits of a stable, actively maintained product. Occasionally the dbt Labs team sees the opportunity for a change to default behaviour that we believe is more sensible, more secure, more helpful — just better in some way — but which would come as a change to users who have grown accustomed to the existing behaviour.

To accommodate both groups in these scenarios, we've extended dbt to support project-level behavior flags. These can be used to opt into or opt out of changes to default behavior. From now on, backward-incompatible changes to dbt functionality will be implemented behind a flag with a default value that preserves the legacy behavior. After a few months, the new behavior will become the default — but only after some proactive communication with customers and external package maintainers.

The same behavior change flags will naturally extend to dbt packages, which are fundamentally just dbt projects. This allows package maintainers to ensure that behavior doesn't change unexpectedly as a result of changes to dbt Core. For more details, check out our user documentation on legacy behaviors, as well as our contributor documentation for introducing behavior changes safely.

Stability for metadata artifacts​

Lastly, we’ve revisited our process around artifact interfaces. These are the workhorses of many integrations in the dbt ecosystem: those maintained by dbt Labs, by third-party vendors, or just homegrown at a particular organization. While these schemas have been versioned and well-defined since dbt Core v1.0, they have changed in many of the minor releases since.

We’ve now formalized our development best practices to strongly prefer minor schema evolutions over major breaking changes. We’ve also put checks in place to ensure we’re not unintentionally introducing breaking changes to artifacts, thus avoiding disruption to integrations across the ecosystem.

Our commitment​

In conclusion, we’re putting a lot of new muscle behind our commitments to dbt Cloud customers, the dbt Community, and the broader ecosystem:

• Continuous updates: The "Latest" release track in dbt Cloud simplifies the update process, ensuring you always have the latest features and bug fixes without the maintenance overhead.
• A rigorous new testing and deployment process: Our new testing pipeline ensures that every update is carefully vetted against documented interfaces, Cloud-supported adapters, and popular packages before it reaches you. This process minimizes the risk of regressions — and has now been successful at entirely preventing them for hundreds of customers over multiple months.
• A commitment to stability: We’ve reworked our approaches to adapter interfaces, behaviour change management, and metadata artifacts to give you more stability and control.

As we continue to enhance dbt Cloud, our commitment remains firm: to provide a stable, dependable platform that allows our users to spend less time on maintenance overhead and focus on creating value.

The first new feature we’re going to look at is called extended attributes.

Profile pick​

Extended attributes is a feature that brings the flexibility of dbt Core’s profiles.yml configuration to dbt Cloud. Before the release of the extended attributes feature, you configured a project-level connection and were mostly stuck with it. You could develop and orchestrate into different schemas to keep development work away from production or configure a staging layer with manual workarounds but, beyond that, things got more challenging. By borrowing the flexibility of profiles.yml, which allows configuring as many unique connections as you need, you can now do the same with the security and orchestration tools in dbt Cloud.

How extended attributes work​

The Extended attributes option is available as a textbox on the Environment settings page, where you can input profiles.yml type configurations. When developing in the dbt Cloud IDE, dbt Cloud CLI, or orchestrating job runs, dbt Cloud will parse the provided YAML for extended attributes and merge it with your base project connection settings. If the attribute exists in another source (typically, this would be your project connection settings or the job's configurations), it will replace its value, including overriding any custom environment variables. If the attribute doesn't exist, it will add the attribute to the connection config. You can check out the documentation for more specific details, but now that you’ve got the basic idea, let’s dive into some examples to see why this is so cool.

Multiple accounts for development and production environments​

The most pressing use case for dbt Cloud users is the ability to use different account connections for different teams or development stages in their pipelines. Let’s consider a team that has a typical dev, staging, production setup (known as a WAP workflow): development for active work with small datasets, staging to promote and vet changes against cloned production data, and production for the final deployed code that feeds BI tools. For this hypothetical team though, these are separate accounts in their data platform with their own sets of RBAC and other settings. This is a perfect use case for extended attributes. Let’s take a look at how this team might set this up for a company that uses multiple BigQuery accounts, projects, and datasets (projects and datasets are analogous to databases and schemas on other platforms like Snowflake) to separate dev, staging, and prod:

The extended attributes textbox at the bottom of the environment settings.

Development​

account: 123dev
project: dev
dataset: dbt_winnie
method: oauth
threads: 1

Staging​

account: 123dev
project: staging
dataset: main
method: service-account-json
threads: 16

Production​

account: 456prod
project: analytics
dataset: main
method: service-account-json
threads: 16

With this setup, we have a separate account for development work, using individual development datasets for each developer (with a single thread so that the development build logs are easier to read) connected via OAuth; and a shared staging project with a default main dataset for the staging environment that's only built via a GCP Service Account through dbt Cloud. In that project, we can then configure IAM permissions to only allow building into the staging schema from jobs that use the staging environment as well.

Production is then pointed to a completely separate account that's only writable from production environment builds and readable from the BI tool.

It’s really that simple. This works with PrivateLink connections handling the authentication as well! Again, while we have one project connection that's the default, you can now configure unique connections securely per environment.

All the world a Stage​

Earlier, we touched on staging environments in discussing extended attributes but let's dig deeper into how dbt Cloud now supports those in a first-class way. You now have the option when configuring an environment to choose Development, Production, or Staging. When you configure an environment as a staging type, you’ll unlock new abilities, most importantly the ability to defer to that environment for development work. This fully enables a proper Write-Audit-Publish flow, where development work is built against and promoted to staging before being merged into a production branch when releases have been tested.

All you need to do is configure an environment as staging and enable the Defer to staging/production option in the dbt Cloud IDE. Doing this will favor a staging environment over prod if you have one set up.

Setting an environment to staging type.

Toggle to turn on deferral to staging or production environment.

Upgrading on a curve​

Lastly, let’s consider a more specialized use case. Imagine we have a "tiger team" (consisting of a lone analytics engineer named Dave) tasked with upgrading from dbt version 1.6 to the new Latest release track, to take advantage of new features and performance improvements. We want to keep the rest of the data team being productive in dbt 1.6 for the time being, while enabling Dave to upgrade and do his work with Latest (and greatest) dbt.

Development environment​

By default, the development environment is configured to be version 1.6:

Development environments configured to v1.6 by default.

Development connection settings​

Dave's development connection settings are:

Dave's development environment override.

Launch special​

Each connection you make from every environment is now unique. You can deploy, develop, and test your data with a setup that molds to your organization, not to what’s available in dbt Cloud. Whether you’re looking to create advanced, layered environments to launch new models safely or enable greater independence between developers, dbt Cloud extends to support what you need. The best part is, we're just getting started: the upcoming global connections feature set will take this even further, allowing you to set multiple connections globally and reuse them wherever needed.

I encourage you to take these new features for a spin by creating a staging environment, configuring the unique connections you need to enable it at your org, and seeing how it can make your data team more efficient and secure. As always, if you need help or have questions, the dbt Community Forum and Slack are here to support you. Happy modeling!

The original paradigm shift that enabled dbt to exist and be useful was databases going to the cloud.

All of a sudden it was possible for more people to do better data work as huge blockers became huge opportunities:

• We could now dynamically scale compute on-demand, without upgrading to a larger on-prem database.
• We could now store and query enormous datasets like clickstream data, without pre-aggregating and transforming it.

Today, the next wave of innovation is happening in AI and LLMs, and it's coming to the cloud data platforms dbt practitioners are already using every day. For one example, Snowflake have just released their Cortex functions to access LLM-powered tools tuned for running common tasks against your existing datasets. In doing so, there are a new set of opportunities available to us:

• We can now derive meaning from large unstructured blocks of text, without painstakingly building complex regexes
• We can now summarize or translate content without having to call out to external third-party APIs.
• Most significantly, we can now bake reasoning capabilities into our dbt models by describing what we want to happen.

Analytics Engineers have always existed at the intersection of business context and data - LLMs on the warehouse make it possible to embed more business context and unlock more data, increasing our leverage in both directions at once.

Anatomy of an LLM-powered workflow​

My colleagues and I did some experiments last year using GPT-4 to enhance the Semantic Layer, but this is the first time it's been possible to use AI directly inside of our dbt project, without any additional tooling.

When we were looking for a first AI-powered use case in our analytics stack, we wanted to find something that:

• Solves a real business problem for us today
• Makes use of the unique capabilities of LLMs
• Was cognisant of their current uncertainties and limitations
• Anticipated future improvements to the models available to us, so things that don't work today might soon work very well indeed.

Once we selected our use case, the analytics engineering work of building and orchestrating the new dbt models felt very familiar; in fact it was exactly the same as any other model I've built.

• I still built a DAG in layers, with existing staging models as the foundation and building new modular segments on top
• I still followed the same best practices and conventions around writing, styling and versioning controlling my code
• I still ensured my models behaved as I expected by going through a code review and automated testing process, before deploying my LLM workloads to production with the dbt Cloud orchestrator.

In short, the same dbt I know and love, but augmented by the new power that Cortex exposes.

Developing our first LLM-powered analytics workflow in dbt Cloud​

When thinking about a project that would only be possible if we could make sense of a large volume of unstructured text, I pretty quickly realised this could help me keep up to date with the dbt Community Slack. Even though we spend a lot of time in Slack, there's hundreds of threads taking place across dozens of channels every day, so we often miss important or interesting conversations.

We already pull Slack data into Snowflake for basic analytics, but having a triage agent that could keep a watchful eye over the Slack – and let us know about things we'd otherwise have missed – would help the Developer Experience team do a better job of keeping our finger on the pulse of dbt developers' needs.

Once it was finished, it looked like this:

An example of some summarized threads for review (lightly edited for anonymity).

Up to once a day, we'll get a post in our internal Slack with links to a handful of interesting threads for each person's focus areas and a brief summary of the discussion so far. From there, we can go deeper by diving into the thread ourselves, wherever it happens to take place. While developing this I found multiple threads that I wouldn't have found any other way (which was itself a problem, since my model filters out threads once a dbt Labs employee is participating in it, so I kept losing all my testing data).

You probably don't have the exact same use case as I do, but you can imagine a wide set of use case for LLM powered analytics engineering:

• A SaaS company could pull information from sales calls or support tickets to gain insight into conversations
• A mobile app developer might pull in app store reviews for sentiment analysis
• By calculating the vector embeddings for text, deduplicating similar but nonidentical text becomes more tractable.

Here's an extract of some of the code, using the cortex.complete() function - notice that the whole thing feels just like normal SQL, because it is!

        select trim(
            snowflake.cortex.complete(
                'llama2-70b-chat',
                concat(
                    'Write a short, two sentence summary of this Slack thread. Focus on issues raised. Be brief. <thread>',
                    text_to_summarize,
                    '</thread>. The users involved are: <users>',
                    participant_metadata.participant_users::text,
                    '</users>'
                )
            )
        ) as thread_summary,

Tips for building LLM-powered dbt models​

• Always build incrementally. Anyone who's interacted with any LLM-powered tool knows that it can take some time to get results back from a request, and that the results can vary from one invocation to another. For speed, cost and consistency reasons, I implemented both models incrementally even though in terms of row count the tables are tiny. I also added the full_refresh: false config to protect against other full refreshes we run to capture late-arriving facts.
• Beware of token limits. Requests that contain too many tokens are truncated, which can lead to unexpected results if the cutoff point is halfway through a message. In future I would first try to use the llama-70b model (~4k token limit), and for unsuccessful rows make a second pass using the mistral-7b model (~32k token limit). Like many aspects of LLM powered workflows, we expect token length constraints to increase substantially in the near term.
• Orchestrate defensively, for now. Because of the above considerations, I've got these steps running in their own dbt Cloud job, triggered by the successful completion of our main project job. I don't want the data team to be freaked out by a failing production run due to my experiments. We use YAML selectors to define what gets run in our default job; I created a new selector for these models and then added that selector to the default job's exclusion list. Once this becomes more stable, I'll fold it into our normal job.
• Iterate on your prompt. In the same way as you gradually iterate on a SQL query, you have to tweak your prompt frequently in development to ensure you're getting the expected results. In general, I started with the shortest command I thought could work and tweaked it based on the results I was seeing. One slightly disappointing part of prompt engineering: I can spend an afternoon working on a problem, and at the end of it only have a single line of code to check into a commit.
• Remember that your results are non-deterministic. For someone who loves to talk about idempotency, having a model whose results vary based on the vibes of some rocks we tricked into dreaming is a bit weird, and requires a bit more defensive coding than you may be used to. For example, one of the prompts I use is classification-focused (identifying the discussion's product area), and normally the result is just the name of that product. But sometimes it will return a little spiel explaining its thinking, so I need to explicitly extract that value from the response instead of unthinkingly accepting whatever I get back. Defining the valid options in a Jinja variable has helped keep them in sync: I can pass them into the prompt and then reuse the same list when extracting the correct answer.

-- a cut down list of segments for the sake of readability
{% set segments = ['Warehouse configuration', 'dbt Cloud IDE', 'dbt Core', 'SQL', 'dbt Orchestration', 'dbt Explorer', 'Unknown'] %}

    select trim(
        snowflake.cortex.complete(
            'llama2-70b-chat',
            concat(
                'Identify the dbt product segment that this message relates to, out of [{{ segments | join ("|") }}]. Your response should be only the segment with no explanation. <message>',
                text,
                '</message>'
            )
        )
    ) as product_segment_raw,

    -- reusing the segments Jinja variable here
    coalesce(regexp_substr(product_segment_raw, '{{ segments | join ("|") }}'), 'Unknown') as product_segment

Share your experiences​

If you're doing anything like this in your work or side project, I'd love to hear about it in the comment section on Discourse or in machine-learning-general in Slack.

Appendix: An example complete model​

Here's the full model that I'm running to create the overall rollup messages that get posted to Slack, built on top of the row-by-row summary in an earlier model:

{{
    config(
        materialized='incremental',
        unique_key='unique_key',
        full_refresh=false
    )
-}}


{# 
    This partition_by dict is to dry up the columns that are used in different parts of the query. 
    The SQL is used in the partition by components of the window function aggregates, and the column 
    names are used (in conjunction with the SQL) to select the relevant columns out in the final model.
    They could be written out manually, but it creates a lot of places to update when changing from 
    day to week truncation for example. 

    Side note: I am still not thrilled with this approach, and would be happy to hear about alternatives!
#}
{%- set partition_by = [
    {'column': 'summary_period', 'sql': 'date_trunc(day, sent_at)'},
    {'column': 'product_segment', 'sql': 'lower(product_segment)'},
    {'column': 'is_further_attention_needed', 'sql': 'is_further_attention_needed'},
] -%}

{% set partition_by_sqls = [] -%}
{% set partition_by_columns = [] -%}

{% for p in partition_by -%}
    {% do partition_by_sqls.append(p.sql) -%}
    {% do partition_by_columns.append(p.column) -%}
{% endfor -%}


with

summaries as (

    select * from {{ ref('fct_slack_thread_llm_summaries') }}
    where not has_townie_participant

),

aggregated as (
    select distinct
        {# Using the columns defined above #}
        {% for p in partition_by -%}
            {{ p.sql }} as {{ p.column }},
        {% endfor -%}

        -- This creates a JSON array, where each element is one thread + its permalink. 
        -- Each array is broken down by the partition_by columns defined above, so there's
        -- one summary per time period and product etc.
        array_agg(
            object_construct(
                'permalink', thread_permalink,
                'thread', thread_summary
            )
        ) over (partition by {{ partition_by_sqls | join(', ') }}) as agg_threads,
        count(*) over (partition by {{ partition_by_sqls | join(', ') }}) as num_records,
        
        -- The partition columns are the grain of the table, and can be used to create
        -- a unique key for incremental purposes
        {{ dbt_utils.generate_surrogate_key(partition_by_columns) }} as unique_key
    from summaries
    {% if is_incremental() %}
        where unique_key not in (select this.unique_key from {{ this }} as this) 
    {% endif %}

),

summarised as (

    select
        *,
        trim(snowflake.cortex.complete(
            'llama2-70b-chat',
            concat(
                'In a few bullets, describe the key takeaways from these threads. For each object in the array, summarise the `thread` field, then provide the Slack permalink URL from the `permalink` field for that element in markdown format at the end of each summary. Do not repeat my request back to me in your response.',
                agg_threads::text
            )
        )) as overall_summary
    from aggregated

),

final as (
    select 
        * exclude overall_summary,
        -- The LLM loves to say something like "Sure, here's your summary:" despite my best efforts. So this strips that line out
        regexp_replace(
            overall_summary, '(^Sure.+:\n*)', ''
        ) as overall_summary

    from summarised
)

select * from final

Raising a dbt project is hard work. We, as data professionals, have poured ourselves into raising happy healthy data products, and we should be proud of the insights they’ve driven. It certainly wasn’t without its challenges though — we remember the terrible twos, where we worked hard to just get the platform to walk straight. We remember the angsty teenage years where tests kept failing, seemingly just to spite us. A lot of blood, sweat, and tears are shed in the service of clean data!

Once the project could dress and feed itself, we also worked hard to get buy-in from our colleagues who put their trust in our little project. Without deep trust and understanding of what we built, our colleagues who depend on your data (or even those involved in developing it with you — it takes a village after all!) are more likely to be in your DMs with questions than in their BI tools, generating insights.

When our teammates ask about where the data in their reports come from, how fresh it is, or about the right calculation for a metric, what a joy! This means they want to put what we’ve built to good use — the challenge is that, historically, it hasn’t been all that easy to answer these questions well. That has often meant a manual, painstaking process of cross checking run logs and your dbt documentation site to get the stakeholder the information they need.

Enter dbt Explorer! dbt Explorer centralizes documentation, lineage, and execution metadata to reduce the work required to ship trusted data products faster.

dbt Explorer: an upgrade to data discovery​

In the days of yore, answering a question about your data platform may have required a bit of cryptography, sifting through possibly-up-to-date documentation in your internal wiki, run logs to figure out when your models were executed, and slacking the data team member with the most tenure. In the past several years, dbt Docs helped centralize the documentation workflow and dramatically improved the documentation process. While useful, dbt Docs only ever provides a single point in time snapshot, and lacks any sense of your platform’s deployment and execution information. dbt Explorer supercharges the docs experience by providing stateful awareness of your data platform, making support and triage of your platform easier than ever — it even proactively lets you know what to focus on to build even higher quality data products!

Where’s this data coming from?​

Your stakeholders and fellow developers both need a way to orient themselves within your dbt project, and a way to know the full provenance of the number staring at them in their spreadsheet. Where did this info come from? Does it include XYZ data source, or just ABC?

It’s the classic stakeholder question for a reason! Knowing data lineage inherently increases your level of trust in the reporting you use to make the right decisions. The dbt DAG has long served as the map of your data flows, tracing the flow from raw data to ready-to-query data mart.

Look at that lineage!

dbt Explorer builds on this experience in three key ways:

• Lineage 🤝 Docs - dbt Explorer’s lineage is embedded into the documentation page for each resource, meaning there’s no need to toggle between your DAG and your docs, and lose valuable context. Similarly, when you’re navigating the DAG in full screen mode, clicking on a resource in your project loads a summary panel of the most critical info about the resource you’re interested in (including execution status, data contract info, you name it). Understanding the lineage via the DAG and the context from your written documentation is one workflow in Explorer, not two.
• Cross project lineage - if you’re using the new dbt Mesh architecture, you may trace your data back to the end of the DAG and find its source is not raw data, but in fact the output of another team’s dbt project! Luckily, dbt Explorer provides first class support for visualizing and understanding cross project lineage when using the dbt Mesh:
  • Account View + Project DAG: dbt Explorer provides a higher level view of the relationships between all your projects in your dbt Cloud Account — you can trace the lineage across the projects, and easily drill down into each project. When you click on a project in this view, the side panel includes a list of all the public models available for use. Double clicking opens up the lineage for that specific project, making it easy to traverse across your organization’s knowledge graph!
  • Cross Project Icons: When you’re in a project’s lineage, dbt Explorer marks cross-project relationships to make it clear when there are dependencies that span multiple projects. Stakeholders can quickly understand which project owners they may need to contact if they need more information about a dataset.
• Column level lineage - long time listeners of the pod know that column level lineage is a frequently requested feature within dbt. It’s one thing to know how data flows between models, but the column level relationships help you understand precisely how data is used in models — this makes debugging data issues a lot simpler! We’re stoked to announce that dbt Explorer offers this feature embedded alongside your model lineage as well.

You can trace the data in a column from the source to the end of your DAG!

With dbt Explorer, you can answer any question about your data’s lineage at any grain, whether its project to project, model to model, or column to column.

Ok but is it fresh? Is it right?​

Once the data’s journey to your BI tool is clear, there’s a natural second question one would ask before using it — is it, uh, good data? Just knowing where it came from is not enough to build trust in the data product — you need to know if it’s timely and accurate.

dbt Explorer marries the execution metadata to the documentation experience — it reflects the latest state of your project across all your job runs in your production environment, and embeds the execution information throughout the product. For each model, seed, or snapshot, Explorer displays its latest execution status, as well as statuses for any tests run against those resources. Sources show the latest source freshness info, and exposures embed the aggregate test and freshness info right into the details page! No more leaving the docs site to check the most recent logs to see what’s fresh and what’s not — Explorer centralizes everything so you don’t have to!

passing model! passing tests!

have you ever seen a fresher source?

Is the project healthy? Are we managing it properly?​

Beyond building solid data products and making sure they are trusted and used, developers need to know how they may improve their projects’ quality, or what areas may need some focus for refactoring and optimization in the next quarter. There’s always a balance between maintaining a data platform and adding new features to it. Historically, it’s been hard to know exactly where to invest time and effort to improve the health of your project — dbt Explorer provides two features that shine a light on possible areas for improvement within your project.

Recommendations​

One of dbt’s more popular open source packages is dbt_project_evaluator , which tests your project against a set of well established dbt best practices. dbt Explorer now surfaces many of the same recommendations directly within the explorer UI using the metadata from the Discovery API, without any need to download and run the package!

Each model and source has a Recommendations tab on their resource details page, with specific recommendations on how to improve the quality of that resource. Explorer also offers a global view, showing all the recommendations across the project, and includes some top level metrics measuring the test and documentation coverage of the models in your project. These recommendations provide insight into how you can build a more well documented, well tested, and well built project, leading to less confusion and more trust.

The recommendations summary — I’ve got some work to do!

Model Performance Trends​

A huge pain point for analytics engineers is trying to understand if their dbt models are taking longer or are running less efficiently over time. A model that worked great when your data was small may not work so great when your platform matures! Unless things start to actively break, it can be hard to know where to focus your refactoring work.

dbt Explorer now surfaces model execution metadata to take the guesswork out of fine tuning your dbt runs. There’s a new high level overview page to highlight models that are taking the longest to run, erroring the most, and that have the highest rate of test failures. Each model details page also has a new Performance tab, which shows that particular model’s execution history for up to three months of job runs. Spotting an ominous slow increase in runtimes may indicate it’s time for some refactoring — no need to comb through countless run_results.json files yourself! dbt Explorer gets you the data you need where you need it.

maybe I should check on that one long run!

Bon voyage!​

They say the best time to invest plant a tree document your dbt project is yesterday, and the second best time is today. With all the bells and whistles that supercharge your documentation experience in dbt Explorer, there’s no time like the present! Leaning into your documentation and taking advantage of your metadata in dbt Explorer will lead to better data products shipped faster — get out there and explore!

The problem: My partner and I are considering buying a property in Portugal. There is no reference data for the real estate market here - how many houses are being sold, for what price? Nobody knows except the property office and maybe the banks, and they don’t readily divulge this information. The only data source we have is Idealista, which is a portal where real estate agencies post ads.

Unfortunately, there are significantly fewer properties than ads - it seems many real estate companies re-post the same ad that others do, with intentionally different data and often misleading bits of info. The real estate agencies do this so the interested parties reach out to them for clarification, and from there they can start a sales process. At the same time, the website with the ads is incentivised to allow this to continue as they get paid per ad, not per property.

The builder: I’m a data freelancer who deploys end to end solutions, so when I have a data problem, I cannot just let it go.

The tools: I want to be able to run my project on Google Cloud Functions due to the generous free tier. dlt is a new Python library for declarative data ingestion which I have wanted to test for some time. Finally, I will use dbt Core for transformation.

The starting point​

If I want to have reliable information on the state of the market I will need to:

• Grab the messy data from Idealista and historize it.
• Deduplicate existing listings.
• Try to infer what listings sold for how much.

Once I have deduplicated listings with some online history, I can get an idea:

• How expensive which properties are.
• How fast they get sold, hopefully a signal of whether they are “worth it” or not.

Towards a solution​

The solution has pretty standard components:

• An EtL pipeline. The little t stands for normalisation, such as transforming strings to dates or unpacking nested structures. This is handled by dlt functions written in Python.
• A transformation layer taking the source data loaded by my dlt functions and creating the tables necessary, handled by dbt.
• Due to the complexity of deduplication, I needed to add a human element to confirm the deduplication in Google Sheets.

These elements are reflected in the diagram below and further clarified in greater detail later in the article:

Project architecture

Ingesting the data​

For ingestion, I use a couple of sources:

First, I ingest home listings from the Idealista API, accessed through API Dojo's freemium wrapper. The dlt pipeline I created for ingestion is in this repo.

After an initial round of transformation (described in the next section), the deduplicated data is loaded into BigQuery where I can query it from the Google Sheets client and manually review the deduplication.

When I'm happy with the results, I use the ready-made dlt Sheets source connector to pull the data back into BigQuery, as defined here.

Transforming the data​

For transforming I use my favorite solution, dbt Core. For running and orchestrating dbt on Cloud Functions, I am using dlt’s dbt Core runner. The benefit of the runner in this context is that I can re-use the same credential setup, instead of creating a separate profiles.yml file.

This is the package I created: https://github.com/euanjohnston-dev/idealista_dbt_pipeline

Production-readying the pipeline​

To make the pipeline more “production ready”, I made some improvements:

• Using a credential store instead of hard-coding passwords, in this case Google Secret Manager.
• Be notified when the pipeline runs and what the outcome is. For this I sent data to Slack via a dlt decorator that posts the error on failure and the metadata on success.

from dlt.common.runtime.slack import send_slack_message

def notify_on_completion(hook):
    def decorator(func):
        def wrapper(*args, **kwargs):
            try:
                load_info = func(*args, **kwargs)
                message = f"Function {func.__name__} completed successfully. Load info: {load_info}"
                send_slack_message(hook, message)
                return load_info
            except Exception as e:
                message = f"Function {func.__name__} failed. Error: {str(e)}"
                send_slack_message(hook, message)
                raise
        return wrapper
    return decorator

The outcome​

The outcome was first and foremost a visualisation highlighting the unique properties available in my specific area of search. The map shown on the left of the page gives a live overview of location, number of duplicates (bubble size) and price (bubble colour) which can amongst other features be filtered using the sliders on the right. This represents a much better decluttered solution from which to observe the actual inventory available.

Dashboard mapping overview

Further charts highlight additional metrics which – now that deduplication is complete – can be accurately measured including most importantly, the development over time of “average price/square metre” and those properties which have been inferred to have been sold.

Next steps​

This version was very much about getting a base from which to analyze the properties for my own personal use case.

In terms of further development which could take place, I have had interest from people to run the solution on their own specific target area.

For this to work at scale I would need a more robust method to deal with duplicate attribution, which is a difficult problem as real estate agencies intentionally change details like number of rooms or surface area.

Perhaps this is a problem ML or GPT could solve equally well as a human, given the limited options available.

Learnings and conclusion​

The data problem itself was an eye opener into the real-estate market. It’s a messy market full of unknowns and noise, which adds a significant purchase risk to first time buyers.

Tooling wise, it was surprising how quick it was to set everything up. dlt integrates well with dbt and enables fast and simple data ingestion, making this project simpler than I thought it would be.

dlt​

Good:

• As a big fan of dbt I love how seamlessly the two solutions complement one another. dlt handles the data cleaning and normalisation automatically so I can focus on curating and modelling it in dbt. While the automatic unpacking leaves some small adjustments for the analytics engineer, it’s much better than cleaning and typing json in the database or in custom python code.
• When creating my first dummy pipeline I used duckdb. It felt like a great introduction into how simple it is to get started and provided a solid starting block before developing something for the cloud.

Bad:

• I did have a small hiccup with the google sheets connector assuming an oauth authentication over my desired sdk but this was relatively easy to rectify. (explicitly stating GcpServiceAccountCredentials in the init.py file for the source).
• Using both a verified source in the gsheets connector and building my own from Rapid API endpoints seemed equally intuitive. However I would have wanted more documentation on how to run these 2 pipelines in the same script with the dbt pipeline.

dbt​

No surprises there. I developed the project locally, and to deploy to cloud functions I injected credentials to dbt via the dlt runner. This meant I could re-use the setup I did for the other dlt pipelines.

def dbt_run():
  # make an authenticated connection with dlt to the dwh
    pipeline = dlt.pipeline(
        pipeline_name='dbt_pipeline',
        destination='bigquery', # credentials read from env
        dataset_name='dbt'
    )
  # make a venv in case we have lib conflicts between dlt and current env
    venv = dlt.dbt.get_venv(pipeline)
  # package the pipeline, dbt package and env
    dbt = dlt.dbt.package(pipeline, "dbt/property_analytics", venv=venv)
  # and run it
    models = dbt.run_all()
    # show outcome
    for m in models:
        print(f"Model {m.model_name} materialized in {m.time} with status {m.status} and message {m.message}"

Cloud functions​

While I had used cloud functions before, I had never previously set them up for dbt and I was able to easily follow dlt’s docs to run the pipelines there. Cloud functions is a great solution to cheaply run small scale pipelines and my running cost of the project is a few cents a month. If the insights drawn from the project help us save even 1% of a house price, the project will have been a success.

To sum up​

dlt feels like the perfect solution for anyone who has scratched the surface of python development. To be able to have schemas ready for transformation in such a short space of time is truly… transformational. As a freelancer, being able to accelerate the development of pipelines is a huge benefit within companies who are often frustrated with the amount of time it takes to start ‘showing value’.

I’d welcome the chance to discuss what’s been built to date or collaborate on any potential further development in the comments below.

We’re announcing that dbt Server is officially deprecated and will no longer be maintained by dbt Labs going forward. You can continue to use the repository and fork it for your needs. We’re also looking for a maintainer of the repository from our community! If you’re interested, please reach out by opening an issue in the repository.

Why are we deprecating dbt Server?​

At dbt Labs, we are continually working to build rich experiences that help our users scale collaboration around data. To achieve this vision, we need to take moments to think about which products are contributing to this goal, and sometimes make hard decisions about the ones that are not, so that we can focus on enhancing the most impactful ones.

dbt Server previously supported our legacy Semantic Layer, which was fully deprecated in December 2023. In October 2023, we introduced the GA of the revamped dbt Semantic Layer with significant improvements, made possible by the acquisition of Transform and the integration of MetricFlow into dbt.

The dbt Semantic Layer is now fully independent of dbt Server and operates on MetricFlow Server, a powerful new proprietary technology designed for enhanced scalability. We’re incredibly excited about the new updates and encourage you to check out our documentation, as well as this blog on how the product works.

The deprecation of dbt Server and updates to the Semantic Layer signify the evolution of the dbt ecosystem towards more focus on in product and out-of-the-box experiences around connectivity, scale, and flexibility. We are excited that you are along with us on this journey.

“OK,” you think further, “I’ll just run a dbt build -s +my_changed_model to make sure I have everything I need built into my dev schema and I can test my changes”. You run the command. You wait. You wait some more. You get some coffee, and completely take yourself out of your dbt development flow state. A lot of time and money down the drain to get to a point where you can start your work. That’s no good!

Luckily, dbt’s defer functionality allow you to only build what you care about when you need it, and nothing more. This feature helps developers spend less time and money in development, helping ship trusted data products faster. dbt Cloud offers native support for this workflow in development, so you can start deferring without any additional overhead!

Defer to prod or prefer to slog​

A lot of dbt’s magic relies on the elegance and simplicity of the {{ ref() }} function, which is how you can build your lineage graph, and how dbt can be run in different environments — the {{ ref() }} functions dynamically compile depending on your environment settings, so that you can run your project in development and production without changing any code.

Here's how a simple {{ ref() }} would compile in different environments:

-- in models/my_model.sql
select * from {{ ref('model_a') }}

-- in target/compiled/models/my_model.sql
select * from analytics.dbt_dconnors.model_a

-- in target/compiled/models/my_model.sql
select * from analytics.analytics.model_a

All of that is made possible by the dbt manifest.json, the artifact that is produced each time you run a dbt command, containing the comprehensive and encyclopedic compendium of all things in your project. Each node is assigned a unique_id (like model.my_project.my_model ) and the manifest stores all the metadata about that model in a dictionary associated to that id. This includes the data warehouse location that gets returned when you write {{ ref('my_model') }} in SQL. Different runs of your project in different environments result in different metadata written to the manifest.

Let’s think back to the hypothetical above — what if we made use of the production metadata to read in data from production, so that I don’t have to rebuild everything upstream of the model I’m changing? That’s exactly what defer does! When you supply dbt with a production version of the manifest.json artifact, and pass the --defer flag to your dbt command, dbt will resolve the {{ ref() }} functions for any resource upstream of your selected models with the production metadata — no need to rebuild anything you don’t have to!

Let’s take a look at a simplified example — let’s say your project looks like this in production:

A simplified dbt project running in production.

And you’re tasked with making changes to model_f. Without defer, you would need to make sure to at minimum execute a dbt run -s +model_f to ensure all the upstream dependencies of model_f are present in your development schema so that you can start to run model_f.* You just spent a whole bunch of time and money duplicating your models, and now your warehouse looks like this:

The whole project has been rebuilt into the dev schema, which can be time consuming and expensive!

With defer, we should not build anything other than the models that have changed, and are now different from their production counterparts! Let’s tell dbt to use production metadata to resolve our refs, and only build the model I have changed — that command would be dbt run -s model_f --defer .**

Using defer, we can only build one single model

This results in a much slimmer build — we read data in directly from the production version of model_b and model_c, and don’t have to worry about building anything other than what we selected!

* Another option is to run dbt clone -s +model_f , which will make clones of your production models into your development schema, making use of zero copy cloning where available. Check out this great dev blog from Doug and Kshitij on when to use clone vs defer!

** in dbt Core, you also have to tell dbt where to find the production artifacts! Otherwise it doesn’t know what to defer to. You can either use the --state path/to/artifact/folder option, or set a DBT_STATE environment variable.

Batteries included deferral in dbt Cloud​

dbt Cloud offers a seamless deferral experience in both the dbt Cloud IDE and the dbt Cloud CLI — dbt Cloud always has the latest run artifacts from your production environment. Rather than having to go through the painful process of somehow getting a copy of your latest production manifest.json into your local filesystem to defer to, and building a pipeline to always keep it fresh, dbt Cloud does all that work for you. When developing in dbt Cloud, the latest artifact is automatically provided to you under the hood, and dbt Cloud handles the --defer flag for you when you run commands in “defer mode”. dbt Cloud will use the artifacts from the deployment environment in your project marked as Production in the environments settings in both the IDE and the Cloud CLI. Be sure to configure a production environment to unlock this feature!

In the dbt Cloud IDE, there’s as simple toggle switch labeled Defer to production. Simply enabling this toggle will defer your command to the production environment when you run any dbt command in the IDE!

The defer to prod toggle in the IDE

The cloud CLI has this setting on by default — there’s nothing else you need to do to set this up! If you prefer not to defer, you can pass the --no-defer flag to override this behavior. You can also set an environment other than your production environment as the deferred to environment in your dbt-cloud settings in your dbt_project.yml :

dbt-cloud:
  project-id: <Your project id>
  defer-env-id: <An environment id>

When you’re developing with dbt Cloud, you can defer right away, and completely avoid unnecessary model builds in development!

Other things to to know about defer​

Favoring state

One of the major gotchas in the defer workflow is that when you’re in defer mode, dbt assumes that all the objects in your development schema are part of your current work stream, and will prioritize those objects over the production objects when possible.

Let’s take a look at that example above again, and pretend that some time before we went to make this edit, we did some work on model_c, and we have a local copy of model_c hanging out in our development schema:

Hypothetical starting point, with a development copy of model_c in the development schema at the start of the development cycle.

When you run dbt run -s model_f --defer , dbt will detect the development copy of model_c and say “Hey, y’know, I bet Dave is working on that model too, and he probably wants to make sure his changes to model_c work together with his changes to model_f . Because I am a kind and benevolent data transformation tool, i’ll make sure his {{ ref('model_c') }] function compiles to his development changes!” Thanks dbt!

As a result, we’ll effectively see this behavior when we run our command:

With a development version of model_a in our dev schema, dbt will preferentially use that version instead of deferring

Where our code would compile from

# in models/model_f.sql
with 

model_b as (
 select * from {{ ref('model_b') }}
),

model_c as (
 select * from {{ ref('model_c') }}
),

...

to

# in target/compiled/models/model_f.sql
with 

model_b as (
 select * from analytics.analytics.model_b
),

model_c as (
 select * from analytics.dbt_dconnors.model_b
),

...

A mix of prod and dev models may not be what we want! To avoid this, we have a couple options:

1. Start fresh every time: The simplest way to avoid this issue is to make sure you are always drop your development schema at the start of a new development session. That way, the only things that show up in your development schema are the things you intentionally selected with your commands!
2. Favor state: Passing the --favor-state flag to your command tells dbt “Hey benevolent tool, go ahead and use what you find in the production manifest no matter what you find in my development schema” so that both {{ ref() }} functions in the example above point to the production schema, even if model_c was hanging around in there.

In this example, model_c is a relic of a previous development cycle, but I should be clear here that defaulting to using dev relations is usually the right course of action — generally, a dbt PR spans a few models, and you want to coordinate your changes across those models together. This behavior can just get a bit confusing if you’re encountering it for the first time!

When should I not defer to prod

While defer is a faster and cheaper option for most folks in most situations, defer to prod does not support all projects. The most common reason you should not use defer is regulatory — defer to prod makes the assumption that data is shared between your production and development environments, so reading between these environments is not an issue. For some organizations, like healthcare companies, have restrictions around the data access and sharing that precludes the basic defer structure presented here.

Call me Willem Defer​

Willem Dafoe after using the `-—defer` flag

Defer to prod is a powerful way to improve your development velocity with dbt, and dbt Cloud makes it easier than ever to make use of this feature! You too could look this cool while you’re saving time and money developing on your dbt projects!

Over the course of my three years running the Partner Engineering team at dbt Labs, the most common question I've been asked is, How do we integrate with dbt? Because those conversations often start out at the same place, I decided to create this guide so I’m no longer the blocker to fundamental information. This also allows us to skip the intro and get to the fun conversations so much faster, like what a joint solution for our customers would look like.

This guide doesn't include how to integrate with dbt Core. If you’re interested in creating a dbt adapter, please check out the adapter development guide instead.

Instead, we're going to focus on integrating with dbt Cloud. Integrating with dbt Cloud is a key requirement to become a dbt Labs technology partner, opening the door to a variety of collaborative commercial opportunities.

Here I'll cover how to get started, potential use cases you want to solve for, and points of integrations to do so.

New to dbt Cloud?​

If you're new to dbt and dbt Cloud, we recommend you and your software developers try our Getting Started Quickstarts after reading What is dbt. The documentation will help you familiarize yourself with how our users interact with dbt. By going through this, you will also create a sample dbt project to test your integration.

If you require a partner dbt Cloud account to test on, we can upgrade an existing account or a trial account. This account may only be used for development, training, and demonstration purposes. Please contact your partner manager if you're interested and provide the account ID (provided in the URL). Our partner account includes all of the enterprise level functionality and can be provided with a signed partnerships agreement.

Integration points​

• Discovery API (formerly referred to as Metadata API)
  • Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. We have two schemas available (environment and job level). By default, we always recommend that you integrate with the environment level schema because it contains the latest state and historical run results of all the jobs run on the dbt Cloud project. The job level will only provide you the metadata of one job, giving you only a small snapshot of part of the project.
• Administrative (Admin) API
  • Overview — This REST API allows you to orchestrate dbt Cloud jobs runs and help you administer a dbt Cloud account. For metadata retrieval, we recommend integrating with the Discovery API instead.
• Webhooks
  • Overview — Outbound webhooks can send notifications about your dbt Cloud jobs to other systems. These webhooks allow you to get the latest information about your dbt jobs in real time.
• Semantic Layers/Metrics
  • Overview — Our Semantic Layer is made up of two parts: metrics definitions and the ability to interactively query the dbt metrics. For more details, here is a basic overview and our best practices.
  • Metrics definitions can be pulled from the Discovery API (linked above) or the Semantic Layer Driver/GraphQL API. The key difference is that the Discovery API isn't able to pull the semantic graph, which provides the list of available dimensions that one can query per metric. That is only available with the SL Driver/APIs. The trade-off is that the SL Driver/APIs doesn't have access to the lineage of the entire dbt project (that is, how the dbt metrics dependencies on dbt models).
  • Three integration points are available for the Semantic Layer API.

dbt Cloud hosting and authentication​

To use the dbt Cloud APIs, you'll need access to the customer’s access urls. Depending on their dbt Cloud setup, they'll have a different access URL. To find out more, refer to Regions & IP addresses to understand all the possible configurations. My recommendation is to allow the customer to provide their own URL to simplify support.

If the customer is on an Azure single tenant instance, they don't currently have access to the Discovery API or the Semantic Layer APIs.

For authentication, we highly recommend that your integration uses account service tokens. You can read more about how to create a service token and what permission sets to provide it. Please note that depending on their plan type, they'll have access to different permission sets. We do not recommend that users supply their user bearer tokens for authentication. This can cause issues if the user leaves the organization and provides you access to all the dbt Cloud accounts associated to the user rather than just the account (and related projects) that they want to integrate with.

Potential use cases​

• Event-based orchestration
  • Desired action — You want to receive information that a scheduled dbt Cloud job has been completed or has kicked off a dbt Cloud job. You can align your product schedule to the dbt Cloud run schedule.
  • Examples — Kicking off a dbt job after the ETL job of extracting and loading the data is completed. Or receiving a webhook after the job has been completed to kick off your reverse ETL job.
  • Integration points — Webhooks and/or Admin API
• dbt lineage
  • Desired action — You want to interpolate the dbt lineage metadata into your tool.
  • Example — In your tool, you want to pull in the dbt DAG into your lineage diagram. For details on what you could pull and how to do this, refer to Use cases and examples for the Discovery API.
  • Integration points — Discovery API
• dbt environment/job metadata
  • Desired action — You want to interpolate the dbt Cloud job information into your tool, including the status of the jobs, the status of the tables executed in the run, what tests passed, etc.
  • Example — In your Business Intelligence tool, stakeholders select from tables that a dbt model created. You show the last time the model passed its tests/last run to show that the tables are current and can be trusted. For details on what you could pull and how to do this, refer to What's the latest state of each model.
  • Integration points — Discovery API
• dbt model documentation
  • Desired action — You want to interpolate the dbt project Information, including model descriptions, column descriptions, etc.
  • Example — You want to extract the dbt model description so you can display and help the stakeholder understand what they are selecting from. This way, the creators can easily pass on the information without updating another system. For details on what you could pull and how to do this, refer to What does this dataset and its columns mean.
  • Integration points — Discovery API

dbt Core only users will have no access to the above integration points. For dbt metadata, oftentimes our partners will create a dbt Core integration by using the dbt artifact files generated by each run and provided by the user. With the Discovery API, we are providing a dynamic way to get the latest information parsed out for you.

dbt Cloud plans & permissions​

The dbt Cloud plan type will change what the user has access to. There are four different types of plans:

• Developer — This is free and available to one user with a limited amount of successful models built. This plan can't access the APIs, Webhooks, or Semantic Layer and is limited to just one project.
• Team — This plan provides access to the APIs, webhooks, and Semantic Layer. You can have up to eight users on the account and one dbt Cloud Project. This is limited to 15,000 successful models built.
• Enterprise (multi-tenant/multi-cell) — This plan provides access to the APIs, webhooks, and Semantic Layer. You can have more than one dbt Cloud project based on how many dbt projects/domains they have using dbt. The majority of our enterprise customers are on multi-tenant dbt Cloud instances.
• Enterprise (single tenant): This plan might have access to the APIs, webhooks, and Semantic Layer. If you're working with a specific customer, let us know and we can confirm if their instance has access.

FAQs​

• What is a dbt Cloud project?
  • A dbt Cloud project is made up of two connections: one to the Git repository and one to the data warehouse/platform. Most customers will have only one dbt Cloud project in their account but there are enterprise clients who might have more depending on their use cases. The project also encapsulates two types of environments at minimal: a development environment and deployment environment.
  • Folks commonly refer to the dbt project as the code hosted in their Git repository.
• What is a dbt Cloud environment?
  • For an overview, check out About environments. At a minimum, a project will have one deployment type environment that they will be executing jobs on. The development environment powers the dbt Cloud IDE and Cloud CLI.
• Can we write back to the dbt project?
  • At this moment, we don't have a Write API. A dbt project is hosted in a Git repository, so if you have a Git provider integration, you can manually open a pull request (PR) on the project to maintain the version control process.
• Can you provide column-level information in the lineage?
  • Column-level lineage is currently in beta release with more information to come.
• How do I get a Partner Account?
  • Contact your Partner Manager with your account ID (in your URL).
• Why shouldn't I use the Admin API to pull out the dbt artifacts for metadata?
  • We recommend not integrating with the Admin API to extract the dbt artifacts documentation. This is because the Discovery API provides more extensive information, a user-friendly structure, and a more reliable integration point.
• How do I get access to the dbt brand assets?
  • Check out our Brand guidelines page. Please make sure you’re not using our old logo (hint: there should only be one hole in the logo). Please also note that the name dbt and the dbt logo are trademarked by dbt Labs, and that use is governed by our brand guidelines, which are fairly specific for commercial uses. If you have any questions about proper use of our marks, please ask your partner manager.
• How do I engage with the partnerships team?
  • Email partnerships@dbtlabs.com.

At dbt we recently made the Semantic Layer Generally Available. The Semantic Layer lets teams define business metrics centrally, in dbt, and access them in multiple analytics tools through our semantic layer APIs. I’m a Product Manager on the Semantic Layer team, and the launch of the Semantic Layer put our team in an interesting, somewhat “meta,” position: we need to understand how a product launch is doing, and the product we just launched is designed to make defining and consuming metrics much more efficient. It’s the perfect opportunity to put the semantic layer through its paces for product analytics. This blog post walks through the end-to-end process we used to set up product analytics for the dbt Semantic Layer using the dbt Semantic Layer.

Getting your data ready for metrics​

The first steps to building a product analytics pipeline with the Semantic Layer look the same as just using dbt - it’s all about data transformation. The steps we followed were broadly:

1. Work with engineering to understand the data sources. In our case, it’s db exports from Semantic Layer Server.
2. Load the data into our warehouse. We use Fivetran and Snowflake.
3. Transform the data into normalized tables with dbt. This step is a classic. dbt’s bread and butter. You probably know the drill by now.

There are plenty of other great resources on how to accomplish the above steps, I’m going to skip that in this post and focus on how we built business metrics using the Semantic Layer. Once the data is loaded and modeling is complete, our DAG for the Semantic Layer data looks like the following:

Semantic Layer DAG in dbt Explorer

Let’s walk through the DAG from left to right: First, we have raw tables from the Semantic Layer Server loaded into our warehouse, next we have staging models where we apply business logic and finally a clean, normalized fct_semantic_layer_queries model. Finally, we built a semantic model named semantic_layer_queries on top of our normalized fact model. This is a typical DAG for a dbt project that contains semantic objects. Now let’s zoom in to the section of the DAG that contains our semantic layer objects and look in more detail at how we defined our semantic layer product metrics.

How we build semantic models and metrics​

What is a semantic model? Put simply, semantic models contain the components we need to build metrics. Semantic models are YAML files that live in your dbt project. They contain metadata about your dbt models in a format that MetricFlow, the query builder that powers the semantic layer, can understand. The DAG below in dbt Explorer shows the metrics we’ve built off of semantic_layer_queries.

Semantic Layer DAG in dbt Explorer

Let’s dig into semantic models and metrics a bit more, and explain some of the data modeling decisions we made. First, we needed to decide what model to use as a base for our semantic model. We decide to usefct_semantic_layerqueries as our base model because defining a semantic model on top of a normalized fact table gives us maximum flexibility to join to other tables. This increased the number of dimensions available, which means we can answer more questions.

You may wonder: why not just build our metrics on top of raw tables and let MetricFlow figure out the rest? The reality is, that you will almost almost always need to do some form of data modeling to create the data set you want to build your metrics off of. MetricFlow’s job isn’t to do data modeling. The transformation step is done with dbt.

Next, we had to decide what we wanted to put into our semantic models. Semantic models contain dimensions, measures, and entities. We took the following approach to add each of these components:

• Dimensions: We included all the relevant dimensions in our semantic model that stakeholders might ask for, like the time a query was created, the query status, and booleans showing if a query contained certain elements like a where filter or multiple metrics.
• Entities: We added entities to our semantic model, like dbt cloud environment id. Entities function as join keys in semantic models, which means any other semantic models that have a joinable entity can be used when querying metrics.
• Measures: Next we added Measures. Measures define the aggregation you want to run on your data. I think of measures as a metric primitive, we’ll use them to build metrics and can reuse them to keep our code DRY.

Finally, we reference the measures defined in our semantic model to create metrics. Our initial set of usage metrics are all relatively simple aggregations. For example, the total number of queries run.

## Example of a metric definition 
metrics:
  - name: queries
    description: The total number of queries run
    type: simple
    label: Semantic Layer Queries
    type_params:
      measure: queries

Having our metrics in the semantic layer is powerful in a few ways. Firstly, metric definitions and the generated SQL are centralized, and live in our dbt project, instead of being scattered across BI tools or sql clients. Secondly, the types of queries I can run are dynamic and flexible. Traditionally, I would materialize a cube or rollup table which needs to contain all the different dimensional slices my users might be curious about. Now, users can join tables and add dimensionality to their metrics queries on the fly at query time, saving our data team cycles of updating and adding new fields to rollup tables. Thirdly, we can expose these metrics to a variety of downstream BI tools so stakeholders in product, finance, or GTM can understand product performance regardless of their technical skills.

Now that we’ve done the pipeline work to set up our metrics for the semantic layer launch we’re ready to analyze how the launch went!

Our Finance, Operations and GTM teams are all looking at the same metrics 😊​

To query to Semantic Layer you have two paths: you can query metrics directly through the Semantic Layer APIs or use one of our first-class integrations. Our analytics team and product teams are big Hex users, while our operations and finance teams live and breathe Google Sheets, so it’s important for us to have the same metric definitions available in both tools.

The leg work of building our pipeline and defining metrics is all done, which makes last-mile consumption much easier. First, we set up a launch dashboard in Hex as the source of truth for semantic layer product metrics. This tool is used by cross-functional partners like marketing, sales, and the executive team to easily check product and usage metrics like total semantic layer queries, or weekly active semantic layer users. To set up our Hex connection, we simply enter a few details from our dbt Cloud environment and then we can work with metrics directly in Hex notebooks. We can use the JDBC interface, or use Hex’s GUI metric builder to build reports. We run all our WBRs off this dashboard, which allows us to spot trends in consumption and react quickly to changes in our business.

Semantic Layer query builder in Hex

On the finance and operations side, product usage data is crucial to making informed pricing decisions. All our pricing models are created in spreadsheets, so we leverage the Google Sheets integration to give those teams access to consistent data sets without the need to download CSVs from the Hex dashboard. This lets the Pricing team add dimensional slices, like tier and company size, to the data in a self-serve manner without having to request data team resources to generate those insights. This allows our finance team to iteratively build financial models and be more self-sufficient in pulling data, instead of relying on data team resources.

Semantic Layer query builder in Google Sheets

As a former data scientist and data engineer, I personally think this is a huge improvement over the approach I would have used without the semantic layer. My old approach would have been to materialize One Big Table with all the numeric and categorical columns I needed for analysis. Then write a ton of SQL in Hex or various notebooks to create reports for stakeholders. Inevitably I’m signing up for more development cycles to update the pipeline whenever a new dimension needs to be added or the data needs to be aggregated in a slightly different way. From a data team management perspective, using a central semantic layer saves data analysts cycles since users can more easily self-serve. At every company I’ve ever worked at, data analysts are always in high demand, with more requests than they can reasonably accomplish. This means any time a stakeholder can self-serve their data without pulling us in is a huge win.

The Result: Consistent Governed Metrics​

And just like that, we have an end-to-end pipeline for product analytics on the dbt Semantic Layer using the dbt Semantic Layer 🤯. Part of the foundational work to build this pipeline will be familiar to you, like building out a normalized fact table using dbt. Hopefully walking through the next step of adding semantic models and metrics on top of those dbt models helped give you some ideas about how you can use the semantic layer for your team. Having launch metrics defined in dbt made keeping the entire organization up to date on product adoption and performance much easier. Instead of a rollup table or static materialized cubes, we added flexible metrics without rewriting logic in SQL, or adding additional tables to the end of our DAG.

The result is access to consistent and governed metrics in the tool our stakeholders are already using to do their jobs. We are able to keep the entire organization aligned and give them access to consistent, accurate data they need to do their part to make the semantic layer product successful. Thanks for reading! If you’re thinking of using the semantic layer, or have questions we’re always happy to keep the conversation going in the dbt community slack. Drop us a note in #dbt-cloud-semantic-layer. We’d love to hear from you!

Environmental segmentation has always been an important part of the analytics engineering workflow:

• When developing new models you can process a smaller subset of your data by using target.name or an environment variable.
• By building your production-grade models into a different schema and database, you can experiment in peace without being worried that your changes will accidentally impact downstream users.
• Using dedicated credentials for production runs, instead of an analytics engineer's individual dev credentials, ensures that things don't break when that long-tenured employee finally hangs up their IDE.

Historically, dbt Cloud required a separate environment for Development, but was otherwise unopinionated in how you configured your account. This mostly just worked – as long as you didn't have anything more complex than a CI job mixed in with a couple of production jobs – because important constructs like deferral in CI and documentation were only ever tied to a single job.

But as companies' dbt deployments have grown more complex, it doesn't make sense to assume that a single job is enough anymore. We need to exchange a job-oriented strategy for a more mature and scalable environment-centric view of the world. To support this, a recent change in dbt Cloud enables project administrators to mark one of their environments as the Production environment, just as has long been possible for the Development environment.

Explicitly separating your Production workloads lets dbt Cloud be smarter with the metadata it creates, and is particularly important for two new features: dbt Explorer and the revised CI workflows.

Make sure dbt Explorer always has the freshest information available​

The old way: Your dbt docs site was based on a single job's run.

The new way: dbt Explorer uses metadata from across every invocation in a defined Production environment to build the richest and most up-to-date understanding of your project.

Because dbt docs could only be updated by a single predetermined job, users who needed their documentation to immediately reflect changes deployed throughout the day (regardless of which job executed them) would find themselves forced to run a dedicated job which did nothing other than run dbt docs generate on a regular schedule.

The Discovery API that powers dbt Explorer ingests all metadata generated by any dbt invocation, which means that it can always be up to date with the applied state of your project. However it doesn't make sense for dbt Explorer to show docs based on a PR that hasn't been merged yet.

To avoid this conflation, you need to mark an environment as the Production environment. All runs completed in that environment will contribute to dbt Explorer's, while others will be excluded. (Future versions of Explorer will support environment selection, so that you can preview your documentation changes as well.)

Run Slimmer CI than ever with environment-level deferral​

The old way: Slim CI deferred to a single job, and would only detect changes as of that job's last build time.

The new way: Changes are detected regardless of the job they were deployed in, removing false positives and overbuilding of models in CI.

Just like dbt docs, relying on a single job to define your state for comparison purposes leads to a choice between unnecessarily rebuilding models which were deployed by another job, or creating a dedicated job that runs dbt compile on repeat to keep on top of all changes.

By using the environment as the arbiter of state, any time a change is made to your Production deployment it will immediately be taken into consideration by subsequent Slim CI runs.

The easiest way to break apart your jobs​

Your organization's data landscape should separate Dev, CI and Prod environments. To achieve this, configure your data warehouse, git repo and dbt Cloud account as shown above.

For most projects, changing from a job-centric to environment-centric approach to metadata is straightforward and immediately pays dividends as described above. Assuming that your Staging/CI and Production jobs are currently intermingled, you can extricate them as follows:

1. Create a new dbt Cloud environment called Staging
2. For each job that belongs to the Staging environment, edit the job and update its environment
3. Tick the "Mark as Production environment" box in your original environment's settings

Conclusion​

Until very recently, I only thought of Environments in dbt Cloud as a way to use different authentication credentials in different contexts. And until very recently, I was mostly right.

Not anymore. The metadata dbt creates is critical for effective data teams – whether you're concerned about cost savings, discoverability, increased development speed or reliable results across your organization – but is only fully effective if it's segmented by the environment that created it.

Take a few minutes to clean up your environments - it'll make all the difference.