Metabase | Business Intelligence, Dashboards, and Data Visualization

Maps data visualizations: best practices

Thu, 19 Dec 2024 00:05:05 +0000

Maps are a powerful way to visualize data, but they work best when used thoughtfully. In this article, we share the basic philosophy behind putting your geospatial data on the grid, like summarizing information to reflect how people naturally think about geography—neighborhoods, states, or countries—rather than scattering individual data points across a pin map. We’ll explore why region maps often tell a clearer, more actionable story and how considering the shape of your data can help you choose the right visualization to highlight patterns and drive better decisions.

We also covered this in a dedicated webinar, check out the recording here.

To help you get started, here’s a quick cheat sheet with everything we’ve discussed about visualizing maps in Metabase, including a philosophy behind picking which map to use. Bookmark it and use it as a reference for your next maps visualization.

If you’re interested in maps, check out this guide — it covers everything you need to know about map visualization in Metabase.

What are map data visualizations?

Map data visualizations display data linked to specific geographic locations. Rather than showing individual data points, they group data by regions—such as countries, states, or neighborhoods—to reveal patterns and trends. This approach makes it easier to analyze how data varies across different areas. Common use cases include visualizing sales by region or population density by city. Map visualizations help transform complex data into clear, actionable insights by providing a geographic context.

Common types of maps data visualizations

Pin map

A pin map displays a single discrete marker for each point of interest. Great for pinpointing exact locations using latitude and longitude. While they look nice, keep in mind that too many points can clutter the map and make trends hard to spot.

Grid map

A grid map is a map with values overlaid graphically in a regular grid, e.g., as squared 10km by 10km colored to show values. These break down data into a grid, similar to a 2D histogram, and work well when you don’t have specific categories. They’re especially useful for data on natural events, like wildfire locations.

Region map

A region map is a map that shows information for geographic regions (such as countries or states), e.g., by coloring each region to show a value. If you want to show data by country, state, or neighborhood, region maps give a clear snapshot of distribution by area. They’re a great choice for highlighting patterns or differences across regions.

Using custom maps in Metabase

Metabase supports custom maps, allowing you to upload a JSON file to create region-based visualizations.

While this offers flexibility, there are a few best practices and limitations to consider:

Avoid adding too much detail. Do you need the exact location of every rack, or would an aggregated view—such as grouping data by administrative regions—be more useful? Summarized data often provides clearer insights.
Focus on actionable insights. Visualizing areas with poor coverage is helpful, but think about the next step. Who needs this information, and what actions should they take?

Technical limitations to keep in mind:

Your dataset must include a region column. Metabase can’t infer regions from latitude and longitude for region maps.
Metabase doesn’t link custom maps to your data automatically or allow drill-through functionality for regions.
When setting up filters, use “Category” instead of “Location” for region-based filtering.

Which map data visualization type to choose?

The choice of map will depend on what you’re trying to communicate. Metabase has 3 different map types for 3 different types of data:

Data type	Map visualization type
Individual data points	Pin map
Summary by coordinates	Grid map
Summary by regions	Region map

Do you really need a pin map?

Before you drop data points onto a pin map, ask yourself: is it necessary? Just like we don’t create bar charts with raw data, the goal of visualizations is to summarize information and highlight the bigger picture—not overwhelm with every individual detail. Pin maps work well when exact locations matter, like tracking deliveries or identifying store locations. But in most cases, summary maps are more effective.

Smarter map visualizations

Use GIS maps for unfiltered data. If you’re plotting all data points without filtering, embed a GIS map instead. With Metabase v51, you can use iframe cards to embed GIS maps, improving performance and clarity. Metabase dashboard cards are best when connected to filters; if filtering isn’t part of your setup, a GIS map might be the better option.
Choose region maps when possible.
- Grid maps work with any geo data but can feel too generic to provide clear insights.
- Region maps require upfront work, like adding region data to your dataset (e.g., states, districts, neighborhoods). However, they align better with how your organization views geographic data, making it easier to identify patterns and take action.
Do you even need a map?

Not all data benefits from a geographical representation. Ask yourself: does the geographic relationship between areas matter? For instance, do you care that Texas is next to New Mexico, or are you only focused on the metrics? If geography isn’t relevant, a bar chart might convey your data more effectively and reduce cognitive load for viewers.

Maps are just one way to tell a story with your data. Choosing the right map (or deciding if you need a map at all) helps to make your visuals clear, actionable, and aligned with your goals. By focusing on how people naturally interpret geographic data, you can help drive better decisions.

Start simple, experiment, and remember: the best visualizations make it easier for your audience to see the bigger picture.

More data visualization resources

How to visualize time-series data: best practices

Wed, 20 Nov 2024 00:05:05 +0000

Here’s a guide to best practices in time-series visualization, covering chart selection, data structure, and advanced techniques. We’ll skip the mechanics of creating charts, and focus on the principles behind building clear and impactful time-based visualizations,

This article includes stuff we covered in our webinar on visualizing time-series data.

We also made a time-series cheat sheet dashboard with everything we’ve discussed about time-series visualizations, from selecting the right chart type to structuring your data for comparison across time periods. Bookmark it and use it as a reference for your next time-series visualization.

What is time-series data?

Time-series data refers to a sequence of data points that include datetimes. Unlike other types of data, time series data is ordered chronologically, where each data point represents a specific moment in time, such as hourly, daily, or monthly measurements.

Examples of time-series data include stock prices, weather temperatures, sales figures, or economic indicators like GDP. These data points are typically grouped by a particular time granularity, like daily sales or weekly temperature readings.

Types of time-series visualizations

Charts for visualizing time-series data include:

Line Charts track trends or data progression over time.
Bar Charts compare values across time intervals, especially for discrete data points.
Area Charts illustrate cumulative totals over time.
Trend Charts analyze performance changes by comparing values to previous periods.
Waterfall Charts display sequential changes, though they’re more niche in application.

Best practices for time-series charts

Focus on a single message: Each chart should communicate one key insight. For multiple insights, use separate visualizations. Different charts highlight different parts of your data, so start by structuring the results table to match your goal.

Consider:

“How should the results look for this chart?” For comparisons, include values for current, last week, and last month in your table.
“How do I control which month shows in a trend?” Make sure the target month is at the top of the table.

Consider data shape: Data structure informs the best chart type.

One metric over time

Multiple metrics over time

One metric with a breakout

Different charts emphasize different aspects of data, so pick the right chart for the aspect that you want to communicate.

Current state

Evolution

Magnitude

Proportion

Area charts (and bar charts) work better for showing size instead of trend.
Stacked area charts can be used to show how the shares of a metric’s breakouts change over time.
Don’t make conclusions about change in the metric itself from stacked area charts - they can be misleading! Stacked bar charts are only meant to show the change in proportions of the metric’s breakouts.

Accumulation

Use Offset for comparisons: The offset function in Metabase allows you to compare a metric’s value in the current row to its value in a previous row. For example, if you’re grouping by month, setting the offset to -12 will show the metric’s value from 12 months prior. You can use Offset to add columns to a row that include values from previous rows, which makes it easier to compare metrics like year-over-year changes in a single chart.

Now that you’ve got the basics down, it’s time to get hands-on and start building your own time-series charts with Metabase. Go ahead, create something awesome, and share it with the community!

More data visualization resources

How to build better line and bar charts

Fri, 18 Oct 2024 00:05:05 +0000

Bar and line charts make up about 80% of what you’ll see in data visualizations. While building these charts in Metabase is straightforward and takes just 30 seconds, in this article, we’ll focus on the thought process behind building effective charts rather than how to create them.

We’ll go beyond the basics and explore how to improve your charts, the best practices for using different types of visualizations.

We’ve also had an entire webinar on this - check out the recording here (no registration required!).

To start with, we’ve put together this handy cheat sheet which could refer to when visualizing your next thing. It covers 5 commonly used charts: line chart, bar chart, area chart, combo chart, and row chart.

How to choose the right chart type?

Choosing the right chart type is key to telling the right story with your data. These guidelines will help you choose between line, bar, and area charts.

Use line and area charts for continuous data

Line and area charts work best when there’s a clear, continuous flow between data points—like tracking data over time. If your data points are just categories with no inherent connection between them, opt for something like a bar chart instead.

Leave trends for line charts, totals for bars

Line charts are perfect for showing how a metric changes over time, focusing on the trends. On the other hand, bar and area charts are better for emphasizing the size or total value of a metric at specific points. Use bars when you want to compare totals clearly.

Use line charts for small changes

Bar and area charts suggest accumulation and work best when your metric relates to zero. But if your data shows only small fluctuations — like bouncing between 31 and 33 — using bars can be misleading. A line chart is more accurate for highlighting minor changes without overstating them.

How to improve your bar charts

Let’s take a bar chart as an example and try to improve it step by step.

Order your data

For bar charts, there’s often no natural order to the data. Sorting your bars from largest to smallest makes it easier to spot trends and understand the story your data is telling. Organized charts let people quickly see what’s important.

Show information where it’s needed

When reading bar or line charts, it’s frustrating to constantly check the Y-axis to figure out values. Instead, put the information directly on the chart — like displaying the value on top of each bar. This way, people can interpret your chart at a glance without unnecessary back-and-forth.

Keep bar charts simple

Focus on what’s necessary. Do viewers need to know there were exactly 1,494 homework assignments, or is it enough to show there were roughly twice as many assignments as quizzes? If exact numbers aren’t critical, simplify your chart. Reduce clutter by leaving out unnecessary labels or gridlines so your main message stands out.

Use natural reading patterns

People read charts the same way they read text—left to right, top to bottom. For English readers, a “Z” pattern is natural. A row chart might work better than a bar chart in some cases, as it aligns with this flow and makes your data easier to absorb.

Highlight what matters

What’s the goal of your chart? Should people focus on a specific number or trend? Use visual emphasis—like bolding key data points or changing color—to direct attention to the insights that matter most.

Use meaningful colors

Colors like green, red, and orange often come with implied meanings—green for positive, red for negative. If your chart reflects positive or negative outcomes, use these colors to help communicate those sentiments. But if your chart isn’t about sentiment, it’s best to avoid these colors to prevent unintended associations. You can also use color saturation to highlight important areas of your chart without overwhelming viewers.

Best practices for stacked charts

Stacked charts can add complexity to your data visualization, so it’s important to know when to use them and when to avoid them. Let’s break down the key considerations for using stacked charts effectively.

Stacking charts increases cognitive load

When you stack data, you’re adding an extra layer of complexity for viewers to process. Before choosing a stacked chart, ask yourself: are you trying to compare data within categories or across categories? Often, creating separate charts provides a clearer view of your data.

Stacking implies accumulation

Only stack metrics that make sense together—like totals or counts. Don’t stack metrics like averages or maximums that don’t naturally add up. For example, you can’t have an “average rating” of 12 out of 5. Use stacking only when the data logically accumulates.

Use line charts to show trends over time

Line charts are ideal for showing how metrics evolve over time, especially when tracking small changes. If you’re visualizing something like average ratings, a line chart will provide a clearer picture than a bar chart. To zoom in on small variations, you can unpin the Y-axis from zero—but never do this with bar or area charts, as it can distort the scale.

To make your line charts even clearer, reduce visual clutter by removing unnecessary data points. Use line thickness and color to emphasize the most important data series.

Dealing with categories of different sizes

When categories in a chart have vastly different sizes, it’s tough to compare them. If your goal is to compare the proportion of categories within a bar, use a 100% stacked chart. But if you’re more interested in comparing raw sizes, rearrange the order of breakouts or use an unstacked chart.

Consider using row charts for a more natural reading order, especially if a custom order aligns better with the business context. You can also use color saturation and visual emphasis to highlight key benchmarks.

Even better:

Row chart instead of a bar chart for natural reading order
Custom order of bars that makes more business sense
Visual emphasis on an important benchmark
Color shade and saturation that communicate context

Challenges of comparing breakouts in stacked charts

Comparing breakouts in stacked charts can be difficult because comparing heights is easier than comparing areas. In a stacked chart, only the first (bottom) segment is easy to compare by height—anything above has to be judged by area, which is less precise.

Solutions:

Use unstacked charts, but be mindful that they can get visually overwhelming.
Consider stacked area charts, but keep in mind they can mislead depending on the data and context. Choose the right visualization based on the insight you’re trying to convey.

Be cautious with stacked area charts

Stacked area charts can easily distort the data because the order of the areas impacts how changes are perceived. It’s easy to overestimate or underestimate changes when the layers overlap. Remember, area charts are best for showing size, not changes or trends.

Conclusion

Now that you’ve got the basics down, head over to your Metabase account and try out these techniques to improve your data visualizations. See how they can make your charts more clear and useful. Happy charting!

More data visualization resources

Communicate Data Effectively

Tue, 08 Oct 2024 00:05:05 +0000

This post was originally published in the Data Analysis Journal and written by our analytics engineer, Thomas Schmidt.

TL;DR: Bridge the gap between data and decision-making with clear, actionable communication

Understand the core problem and tailor your message to your audience’s needs.
Start with a TL;DR: Summarize key insights upfront to respect executives’ time.
Keep it concise. Remove unnecessary details and clutter from presentations.
Use colors intentionally. Highlight key points and create visual connections.
Make bold statements. Provide confident recommendations, using appendices for details.

→ Implement these strategies in your next presentation to ensure your data insights drive informed decisions.

Have you ever tried getting directions in a foreign country from someone who doesn’t speak your language? That’s how I felt the first time I presented a complex data analysis to an executive team. While the data made sense to me, it felt like I was speaking a foreign language to them. Just like a translator bridges gaps, we need to translate our analyses into something executives can act on. Effective communication isn’t just about sharing insights—it’s about ensuring those insights are understood and drive decisions. In this post, I’ll share lessons I’ve learned on turning data into clear, actionable messages for executives.

Identify the real problem and tailor your message to the audience’s needs

In our data jobs, we are often driven by curiosity and the joy of solving problems using data. However, it’s crucial to understand two key aspects before diving into analytics projects: the core problem and the audience.

Understand the Core Problem

Stakeholders often approach us with a proposed solution in mind (“I need data X”). By digging into the ‘why’ and understanding the core problem behind the request, we can choose a better approach that serves their needs, leading to better project outcomes. Knowing the problem you are trying to solve will inform your decisions and influence what next steps you recommend.

Know your audience

We don’t want to find ourselves in a situation where our audience feels like we are speaking a foreign language to them. Executives haven’t been along for the entire analytical journey that we went through. Thus, we need to translate our findings for them. Before crafting a communication, consider the following

Identify what’s important to them and their current priorities. Usually, this means strategic insights and actionable recommendations are preferred over details of the fancy machine learning algorithm you used for the project.
What do we need to explain and what do they already know? Executives usually operate on a more abstract level and don’t necessarily know (and care) about each tiny detail of a process. Speak your audiences’ (domain) language! Explain acronyms or don’t use them at all. Not everyone knows what an “ICP” (Ideal Customer Profile) is and depending on the domain it might even mean different things.
How to start our narrative to get folks excited. Talking to the head of sales? If you start by telling them that this will give them insights on how to beat the competitor, you will have their attention.
What is the best presentation format. Each format has its pros and cons: A written document might be good for async communication but folks will be “alone” while reading it. You cannot clarify things with your words compared to a live presentation where they can directly ask questions (but it might be hard to find an overlapping time slot with the executives). Be sure to be thoughtful about that.
Assess your relationship and level of trust. At Shopify we used the concept of a trust battery. Make sure you know “how charged it is” - this will influence how folks perceive and hear you, and how you need to adjust your communication.

Don’t try to communicate to too many people at once: In case your audience is too broad, try to narrow it down to the decision-maker and tailor your communication to their needs.

Provide a concise summary (TL;DR) upfront and keep messages clear and direct

Executives need quick access to essential insights to make informed decisions promptly. In the fast-paced world of business, they juggle multiple priorities and have limited time for detailed reports. This is where the concept of TL;DR (Too Long; Didn’t Read) becomes invaluable. I first learned about this concept while working at Shopify where we used a similar approach for our communication (be it for presentations, reports or even longer Slack messages). Let’s give it a try:

Executives will love you when you start using TL;DRs:

Quick insights for busy executives. TL;DR provides essential information upfront for prompt decision-making.
Effective crafting tips. Summarize core findings and recommendations, avoiding jargon and using clear language.
Position for impact. Place TL;DR at the beginning of presentations or reports to immediately capture attention.
End with a recommendation. Always have a call to action in the form of a recommendation or provocative question.

→ Adopt the TL;DR approach to enhance clarity and efficiency in executive communications.

When crafting a powerful TL;DR, the focus should be on what is most relevant and important to the audience:

Summarize core findings and recommendations in a few sentences.
Use clear, straightforward language, avoiding jargon.
Use formatting to highlight key points with bullet points and bold text.
End with an actionable recommendation or provocative question.
Always position the TL;DR at the very beginning to capture the audience’s attention immediately.

Pro Tip: Add links to specific slides or sections with detailed data for those who want to dive deeper.

Example of a Well-Crafted TL;DR:

Example: Improving Customer Service Response Times

TL;DR: Faster response times lead to higher customer satisfaction.

Implementing the new ticketing system reduced average response time by 50%.
Customer satisfaction scores increased by 20% after the change. → Recommendation: Expand the system to all customer service teams.

A side note: We also want to make sure to use KPI precision that makes sense for executives. They are usually not interested in the decimal precision number. Thus, rounding is mostly fine (except you know your audience will care about it). If you want to read more about TL;DRs, find some additional instructions in this article.

Time is Everything and Less is More

As we learned in the TL;DR section, attention is a limited resource in our day to day world and we need to fight for it. Here are some tips that have worked well for me in the past, along with good and bad examples.

Respect your Colleagues’ Time

Apart from using TL;DRs, make sure to use concise and direct communication. Avoid “bullshit” and unnecessary information (here is a good book on that topic) and aim for placing your main message in the headline of your messages, slides or report sections.

Use concise and direct communication. Avoid unnecessary information and place your main message in the headline.

This message could use some work:

Better version:

The 2nd message lets you parse the information quicker and decide whether it is relevant and you want to dig deeper (i.e. continue reading in the thread). There are a lot of great additional examples in the book “writing without bullshit” in case you want to read further.

Simplify slides and reports by reducing text and eliminating non-essential elements

The above tips connect well with the next topic we are going to cover: Clutter.

In the Slack message example above you saw a lot of additional information that did not support the message e.g.

“I listened […] on my Podcast Addict app […] I made 40km that day” → Your target audience will not care! Make a post in the #biking channel or tell your friends instead if you need to.
“It is like an hour long but I can totally recommend it to you” → If someone decides to check out the link, they will figure that out.

These are extreme examples but we tend to add such “clutter” to our presentations or reports as well. When was the last time you listened to a presentation and started thinking about something else since you could not follow all the information on that single slide? In those cases, the presenter lost your attention because of cognitive overload. Decluttering can help to reduce the cognitive load for your audience and make it less effort to parse your presentation or report.

Things you can start doing today:

Reduce the amount of text. When you give a presentation, your slides don’t necessarily need a lot of text. You will talk the audience through anyways. Note, this can be different for slides folks consume on their own - here it can be helpful to give them context with few annotations.
Challenge each element in your chart. Look at each of your axes, labels, data points, grid lines, and ask yourself: Would eliminating this element change anything? If the answer is no, get rid of it.
Avoid gifs or animations. If not helpful for your message, avoid them. Our brains get distracted by such content very quickly. Only use if there is a very good reason.

If you are using a BI tool, you will usually also find options to reduce clutter there. Here is an example that uses Metabase, an open source BI tool (which, as you know, is behind the blog you’re reading right now.). It mostly comes with reasonable defaults but we can still tweak it a bit:

This chart could use some work:

Better version:

Apply colors consistently to create visual connections and highlight key points

Colors are a very powerful tool in our communication tool box - if used wisely and consistent. Make sure to be intentional about your use of color. One of my favorite examples is to use it for visual connections between slide titles and my charts where I try to color a category I mention in the same way as it would appear in the chart. Additionally, you can use black vs. gray color and bold vs. normal text to create a visual hierarchy. This helps guide the consumer through the content. Here is an example:

This chart could use some work:

Better version:

Adding a clear indication of when the Shop redesign happened that led to the increased AOV can help us make our story stand out. This helps to reduce the cognitive load and allows your colleagues to process the content in a shorter time while increasing the chance that they perceive your message. This can also help across slides: If you stay with your color palette, your audience will quickly realize and it can be powerful to connect the dots with your previous story.

Give clear, confident recommendations and use appendices for details

“Don’t caveat everything”

Once, an executive at Shopify told me that Data Scientists tend to caveat everything and he often wished that folks were bolder about their recommendations. While it is natural as data professionals to want to highlight the nuances and limitations of our analysis, it is important to remember that executives are looking for clear, actionable insights. They prefer confident recommendations to make strategic decisions quickly over being walked through the details of your analysis.

Some examples:

Caveated Statement: “If the proposed customer retention strategy is implemented and the market response is favorable, we might see a potential decrease in customer churn by up to 5%.”
Bold Statement: “Implementing the proposed customer retention strategy will decrease customer churn by 5%.”
Caveated Statement: “While there are multiple factors to consider, and it’s possible that some variables might change, we believe that the new feature could potentially improve user engagement.”
Bold Statement: “The new feature will improve user engagement.”

You might argue that there are cases where it is important to add caveats. A good approach in those cases is to have an appendix and refer to it. This might give you more confidence and executives have the chance to dig deeper if needed. However, they mostly will ask you directly if questions arise.

In conclusion, effective communication with executives requires thoughtful consideration of their time and needs. By understanding the problem and your audience, starting with a TL;DR, removing clutter, using colors wisely, and making bold statements, you can ensure your insights are understood and lead to actionable decisions. Implement these strategies in your next presentation or report to enhance clarity and effectiveness in executive communications.

How we enriched customer contact and organization data

Wed, 25 Sep 2024 00:05:05 +0000

We wanted to get more context for the data we collect, so we shopped around for data enrichment services. These companies can contextualize your data with more details. For example, given a domain name, a third-party enrichment service can give you the domain’s associated company name, its size, industry, and so on.

Data enrichment is a typical practice for growing companies, and certainly for large companies. But before you go shopping for enrichment services, you should have a known set of problems to solve (or goals to achieve), that enrichment can help with. For Metabase, we wanted to enrich our customer contacts for two reasons:

To keep in touch with our customers. Enrichment can alert us to job changes for contacts that may impact our relationship. For example, if a contact transitions to another job within or outside the company, we may want to reach out to congratulate them and make sure we’re in touch with whomever is taking over the relationship so that important product comms don’t get lost.
To gain a better understanding of organization size and industry, which can help us tailor our marketing and product efforts to make sure we’re building Metabase to solve the kinds of problems these segments face.

Evaluation of service providers

There are surprisingly many service providers for data enrichment, each with its own pros and cons. While we considered various LinkedIn data dump providers, we decided against using them, as we shouldn’t store that kind of data in our data warehouse. We also didn’t evaluate Clearbit, since it’s no longer available as a standalone service.

TL;DR: Apollo.io offered the best coverage, pricing, and features for our use cases.

Criteria / Provider	LinkedIn Sales Navigator	Crunchbase	Lusha	Apollo	CommonRoom
Job History / LinkedIn Profile	Best available, but UI access only; API access is limited	Not available	Not available	Good / Looks recent	Mostly good, but some missing recent job changes
Firmographic, such as industry size, industry, etc	Good. Search is UI only; API access is limited	Good, but about 70% (high/med) to 80% (low confidence) coverage for 100 recent contacts	Ok. Coverage is about 26% for 100 recent contacts	Great at 80% coverage using domain match for 100 recent contacts	Seems to be available mostly for company size, but industry is spotty and coverage is about ~60% for recent 100 contacts
Demographic, such as job title, etc	Great / Everything on LinkedIn Profile. Search is UI only; API access is limited	Limited to select key people, like execs	Ok. Coverage is about 26% for recent 100 contacts	Good at 60% coverage for name, 50% for title / history for 100 recent contacts	Yes, but: 1. Coverage is limited to ~60% for org and ~30% for job title based on 100 recent contacts
Export to data warehouse	Only integration with CRM, such as Salesforce or HubSpot with limited capabilities (differs per CRM)	Enterprise plan supports dataset download. API can do exact domain/name and fuzzy search. 200 calls per minute, 1000 limit	CSV upload / download via UI or API	API or CSV (UI) Contact enrichment is slow at 0.5 secs per call — that’s 1.4 hours per 10k records. API offers bulk download, 10 at a time	Recurring/custom export for Enterprise plan only, otherwise manually via UI. Any field visible on the filter/browse screen can be exported via UI
Cost	Core: $960 per person/year Advanced with CRM integration: $1600 per person/year	API: $10k per year with 30% buy-now discount. CSV Export of all 3m+ companies: $50k with 50% buy-now discount	About $20k to $25k per year for 100k contacts	$400 per month for 10k enrichments via API. 4¢ per record. $3k per month for 100k enrichments. 3¢ per record. More plan options	Many plans from free to Enterprise based on # of contacts and features: 1. Free up to 500 contacts / 50 orgs 2. Starter at $625/mo up to 35k contacts. 3. Team at $1250/mo up to 100k contacts 4. Enterprise at custom pricing with export to data warehouse feature. $50k+ per year

Continuous enrichment

With the service provider selected, we used dlt and Apollo.io’s API to enrich new contacts hourly based on priority—prioritizing new contacts before updating existing ones, and so on.

Here’s a code snippet in Python that:

Gets a list of emails from a prioritized list of contacts from our data warehouse
Calls Apollo’s API to enrich those contacts.
Then saves the enriched information back in another table in our data warehouse.

def enrich_contact(self, postgres_connect_string, to_schema):
    pipeline = dlt.pipeline(pipeline_name='enrich_contact', destination='postgres', dataset_name=to_schema,
                            credentials=postgres_connect_string)

    @dlt.resource(write_disposition='merge', primary_key='email')
    def enriched_contact():
        with pipeline.sql_client() as psql:
            with psql.execute_query("select email from prioritized_contact") as cursor:
                emails = cursor.fetchall()

        for (email,) in emails:
            enriched = self.people_match(email)  # Call Apollo's "people/match" API
            yield enriched['person']

    print(pipeline.run(enriched_contact))

Modeling for self-service analytics

We added the enriched data, such as organization size and industry, to our customer and contact models. For organization size, we grouped the data into a few categories to simplify analysis for our teams.

, case
    when estimated_employees <= 50 then 'Micro'
    when estimated_employees <= 200 then 'Small'
    when estimated_employees <= 1000 then 'Medium'
    when estimated_employees <= 10000 then 'Enterprise'
    when estimated_employees > 10000 then 'Mega Enterprise'
  end as organization_size

In our contact model, we also added left_company_at to indicate when a contact left the customer organization. This field makes it easy to find out which organizations we should reach out to so that they can stay up to date on important product communications.

Final thoughts on data enrichment

Enriching our data has already proved valuable. One interesting insight from the enriched data is that our customers come from a mix of micro to mega organizations. Our teams have already used this data to better understand our customers and monitor job changes.

We’re now discussing how to use enrichment for additional purposes, such creating ideal customer profiles to make the most of our marketing efforts.

But enrichment isn’t free, so we’ll continue to evaluate the value we gain relative to the cost and iterate accordingly. And we hope this post can help you shop for an enrichment service that fits your needs.

Set up a basic pipeline for log analysis

Mon, 04 Mar 2024 00:10:22 +0000

Choosing tools to clean, parse, and structure log data can be overwhelming (and expensive). But when you’re just getting started, you can get away with a simple setup for ad hoc analysis with a BI tool like Metabase. Here are a few best practices to follow for setting up a basic pipeline for analyzing logs.

Use a data connector tool as a shortcut for ingestion

Tools like Airbyte can quickly connect to your database and structure logs for you. Choose your logging source, like AWS CloudTrail, and connect it to a database, like Snowflake (a relatively easy, scalable, reasonable cost solution), or AWS Aurora Serverless Postgres(an easy, somewhat scalable, low-cost solution).

Other ETL tools, like Fivetran or Stitch, work in a similar vein. They use a connector to move log data from a source, like CloudTrail, to your database. You can also use an ETL tool and perform data modeling in tandem to do some of the heavier lifting for you.

Use a single cloud provider to keep everything under one roof

Google Cloud Logging connects with BigQuery so you can automatically ingest logs right into your data warehouse. AWS has multiple logging options, like CloudTrail or CloudWatch, that you can connect to one of their database options, like Postgres for RDS. Azure Monitor also has logging and storage capabilities.

Advanced use case: dump logs from multiple AWS services into an S3 bucket and query them with Athena

If you have a bit of experience with cloud services, like AWS, you can use an entire suite of cloud services to take logs from several different services and push them into one central location to prepare for analysis.

For example, push web server or application logs from your EC2 instances into an S3 bucket, along with your CloudTrail logs. Connect your S3 bucket to a querying tool, like Athena, so you can create a few tables to use for analysis. Once you have tables, you can connect to your analytics tool and create a troubleshooting dashboard, like one that maps EC2 events to CloudTrail incidents for root cause analysis.

Here’s some other AWS logging options that you can use with S3 and Athena:

CloudWatch: store application, system, or custom logs
RDS: store error, slow query, or transaction logs
Lambda: store lambda logs that contain execution details, error messages, and custom log statements
Elastic Load Balancer (ELB): store ELB logs that contain client IP address, request time, and response status code

If want to go a step further, you can connect Athena to dbt and learn how to write your own data transformations in SQL. dbt streamlines version control, deployment, and testing without having to run individual tools. However, we only recommend this setup if you’re familiar with data modeling and developer tooling.

Batch load logs for efficiency

You should batch load your logs into your data warehouse directly, or a storage option like S3, to avoid latency and resource consumption. Most cloud services offer a batch service where you can schedule and queue jobs. Note if you’re paying for a database or log storage, double-check the price of batch loading first as some cloud services charge per batch upload.

Use a database client library or connector for ingestion

Not having access to a connector tool is not an issue, but it may take more development work. Using an existing database client library or driver can help you ingest logs directly into storage / data warehouse at log time.

For example, Postgres has drivers, and MySQL has connectors. Use one of these to hook into your database without having to reinvent the wheel.

Make sure your logs include a timestamp, source, message, and log level

There are four areas we recommend to add to your log files to make log analysis smoother:

Timestamps: It’ll be easier to establish a sequence of events. Timestamps are especially important if you’re using an analytics tool to create dashboards for performance monitoring, or auditing and compliance.
Source: Like the service that created the log, but also which location/file/sub-service the logs are coming from. You can use the service field for troubleshooting, or just to gauge which resources are allocated to each service.
Log message: Keep messages clear and concise so you can understand each event. Reuse keywords so it’s easier to filter and find what you need during textual analysis.
Log level: You can filter on levels like ALERT and CRITICAL to get to the bottom of which log requires immediate investigation and response.

Additional best practices and ideas for log analysis

If you’re aiming for real-time log analysis, or advanced or frequent log analysis, log-specific tooling tends to be a better fit. If your team is already using an ELK stack, a tool like Grafana could fit the bill.

Here are a few additional resources you can use to make decisions when building out a small scale logging pipeline:

Embed a Metabase dashboard in Zendesk

Mon, 26 Feb 2024 00:05:05 +0000

One cool thing about Metabase is that you can embed a dashboard in the apps you use every day.

We recently embedded our customer dashboard right into Zendesk. Having this dashboard side-by-side with support tickets allows us to see customer information without having to switch between Metabase and Zendesk. We even passed a few filter parameters to automatically filter on the customer and organization in the dashboard, significantly speeding up how quickly we can troubleshoot issues.

The dashboard includes:

Account information: Customer name, when their account was created, if it was cancelled and when, subscription status, plan name
Support tier information: Support tier, whether their account is active
Deployment and Metabase version information: Deployment type, cloud provider, Metabase version number and last time they updated
Account details: Annual value, LTV, number of users, country
Cloud details: How many questions and dashboards the customer has, and link to their instance log
Contact information: Email, name, level of technical knowledge
Customer happiness or sentiment: Survey responses, CSAT responses, etc.
List of associated GitHub issues and tickets

Technically, you can embed a Metabase dashboard into any app that allows iframe embedding or allows third-party apps that support the use of embedding URLs. A few other platforms that you can test this out with are Salesforce, Jira, Stripe, and Shopify.

Here’s a quick walkthrough of how we embedded our customer dashboard in Zendesk.

Our setup: an interactive dashboard embedded in an iframe using a third-party app

We used interactive embedding in this example. You can use public embedding to embed a dashboard, but for this instance, we needed to protect customer data as public embedding enables public links. It’s best to stick with interactive embedding if you’re in a similar position.
We decided to use an iFrame app rather than build our own app. By going with a third-party app, we saved engineering resources and got to a working implementation faster. One downside: the logo for the app we went with, Customer360, is always visible. Not ideal, but not really an issue for our internal-facing use.

The iframe apps we considered

To get started, we narrowed down apps and made a list of their pros and cons:

Iframe Plus: $7 per instance, so it’s low price, but the con is there isn’t any support.
Zendesk Iframe: Free, but there was no ticket sidebar option, which is where we wanted the dashboard to live. E.g., the dashboard will only show up in organization view, not ticket view.
Customer360: $4 per agent, so it’s low price and the UI is easy-to-use. This is the app we went with.

Enable interactive embedding

Next, we enabled interactive embedding by going to Settings > Admin settings > Embedding. Click on Enable, and click on Interactive embedding.

We then grabbed the URL for our dashboard from Metabase and properly formatted it to use in the Customer360 app.

You’ll need to set the source attribute to your site URL. For example, http://metabase.yourcompany.com/dashboard/1.

Authorize Zendesk and Customer 360 URLs in Metabase

We also needed to authorize the following Zendesk URLs in Metabase. To do this, we navigated to Admin Settings > Settings > Embedding > Interactive > Authorized Origins and added the following URLs:

Zendesk URLs

https://*.zdusercontent.com
https://*.zndsk.com
https://*.zendesk.com

Customer 360 URL

https://*.myplaylist.io

Install the Customer 360 app and add your dashboard URL

Next, we installed the Customer 360 app and input our dashboard URL (with the source attribute set to our Metabase instance).

Show only the customer data you need by passing parameters in the URL to filters in your dashboard

You can pass values to your dashboard’s filters via parameterized URLs. For example, we pass both Organization and Ticket requester information, so now our dashboard automatically filters to show only the information we need about the customer and their organization.

The Customer 360 app lets you use the following parameters:

{{ticket.requester.email}}
{{ticket.requester.emails}} (comma-separated list of requester’s emails)
{{ticket.requester.external_id}}
{{ticket.requester.id}}
{{ticket.requester.custom_fields.<field_key>}}
{{ticket.organization.id}}
{{ticket.organization.external_id}}
{{ticket.organization.custom_fields.<field_key>}}
{{ticket.ticket_field_<field ID number>}}

See the setting a SQL variable docs for an example.

A small caveat about SSO

Metabase will ask you to sign in again if you refresh the Zendesk ticket page or add a new ticket. Also, you will need an active session in your Metabase to not be kicked out every time you enter the ticket sidebar. We got around this by setting our environment’s MB_SESSION_COOKIE_SAMESITE to none, as mentioned in the interactive embedding setup docs and environment variables docs.

Tidy up your dashboard view in Zendesk

Now, when we click on a ticket in Zendesk and the ticket sidebar pops up, our customer dashboard appears and is filtered down to the customer and their organization!

We hid a few Metabase UI components, like headers or breadcrumbs, by setting header to false in the embedding URL. We did this to clean up the way our dashboard looks in-app.

There’s a list of examples of what else you can do in the showing or hiding Metabase UI components docs. Note that some of these feature may not work for certain dashboards as it depends on the components in the dashboard.

Check out other embedding projects

If you need inspiration, check out projects and feedback in our Github tracker. If you enter the repo manually in the future, just filter by label:Embedding/Interactive to get a full list of current embedding-related issues (click on closed to see resolved issues.)

Keeping tabs on embedded analytics

Mon, 19 Feb 2024 00:03:50 +0000

Metabase offers a few different types of embedding: interactive, static, and public embedding. With usage analytics, you can track how your customers interact with your customer-facing analytics.

The level of detail available in usage analytics depends on the embedding type. For example, interactive embedding gives you an additional capability to track usage tied to a log in, whereas usage stats, like views and downloads, with static and public embedding are anonymous.

However, this guide can give you a roundabout idea of how to start track usage analytics for embedded analytics.

See how often people use your customer-facing analytics

Justify your investment in embedding by keeping track of how often it’s being used. The Metabase metrics dashboard tracks how many active users you have, how many questions people are viewing and creating per week, and trends over time as usage changes week over week.

Understand what your customers are most interested in

In that same Metabase metrics dashboard, you’ll find a list of who your most engaged data consumers are. See what they’re looking at so you can create more relevant content geared towards their interests. Knowing what your customers actually care about can also lead to more meaningful conversations about how you can support them.

You can also learn when customers start to lose interest. For example, for interactive embedding, you can spot if a particularly active customer suddenly stops visiting your analytics. This could be a sign to reach out and see if they need your support.

For static and public embedding, you can look at usage in aggregate by filtering on anonymous to find what your customers are most interested in.

Keep questions and dashboards quick and minimize friction so your customers can self-serve what they need

There are two different dashboards you can use to figure out which dashboards and questions are running the slowest or consuming the most resources.

Use the Performance Overview dashboard to find out what your slowest dashboards and questions are, and refactor, cache, or archive them.
Look in the Content with Cobwebs dashboard to see what’s not being viewed. It might be time to archive, or if you think people are really missing out, put this content somewhere more prominent.

Keep your data secure with multi-tenant embedded analytics

Usage analytics gives you a comprehensive log for auditing and security purposes.

You can monitor the people or activity logs for anomalies, like suspicious logins or changed settings, and set up subscriptions and alerts on questions created from these logs to keep a constant watch on what’s going on in your instance.

You can use these logs in combination with other security features, like SSO, sandboxing, and more, for secure, multi-tenant analytics.

Metabase usage analytics resources

Usage analytics spans beyond just tracking embedded analytics. You can also use usage analytics for internal things, like cleaning up your Metabase, giving your data team an idea of how often your data is being used, and more. Check out a few of the other ways you can use the Metabase analytics collection:

Comparing the most popular open-source charting libraries

Tue, 13 Feb 2024 00:12:18 +0000

Here’s a breakdown of some of the most popular charting libraries to help you figure out which library could be best for your embedding project, as well as a discussion about when you should consider a more batteries-included solution.

Chart.js

Chart.js is an open-source JavaScript visualization library that started in 2013 and, over time, has ranked as the most popular charting library according to GitHub stars and npm downloads.

Chart.js might be right for you if you:

Have large data sets. Data is ingested using their internal format, so you can just set parsing: true and it’ll parse your data for you.
Need an active dev community to lean on. Check out Chart.js’ Discord channel and resource list.
Require plugin support. There are plenty of Chart.js-related repos available to test and use. For example, Chart.js has a well-maintained, popular React components library available.

Visx

The brainchild of Airbnb’s Engineering and Data Science team, visx is a collection of expressive, low-level visualization primitives for React.

While visx isn’t technically a visualization library, it helps you build your own charting using visualization primitives. visx bundles D3 and React to give you the bare bones of charting while keeping the animations, theming, and styling you’re already using for your app.

visx might be right for you if you need to:

Keep bundle size small. visx offers multiple packages, but you can pick and choose the packages you need.
Use your own animation library. Other visualization libraries can sometimes impose animations and style that can impede on existing design work.
Play around and get inspiration first before going all-in. Their gallery has plenty of cool, lightweight examples.

Nivo

Nivo is built on top of D3 as a set of React components to offer server side rendering and fully declarative charts. Nivo’s Storybook offers interactive examples, so you can get a sense of what to build, and their documentation lets you testing out individual components before implementing.

Nivo might be right for you if you need:

Highly customizable, fully declarative, responsive charts.
A library fully written in React (Nivo isn’t the typical wrapper library).
An active developer community to work with. Nivo’s Github repo is pretty active and they have a Discourse channel you can join.

ECharts

ECharts is an open-source Javascript visualization library maintained by the Apache Software Foundation. Companies like Amazon, Gitlab, and Intel use ECharts to include data visualizations in their products, reports, research papers, etc.

We’re working on incorporating ECharts for Metabase data visualizations, so you can say we’re fans of this library, too.

You may want to use ECharts if you need:

“Smart charts”. ECharts automatically uses the appropriate animation to represent data changes.
A multi-rendering solution with cross-platform support. You can render charts in Canvas, SVG, and VML. ECharts has support for PC and mobile browsers, and a node-canvas on Node for server-side rendering.
Access to many pre-built series beyond basic line, bar, and pie charts. There are 20+ options like box plots, heat maps, funnels, and more. You can plot multiple series on a chart, too.
To render fully-customized maps. The ECharts map series type supports geoJSON.
Accessibility features. It’s one of the few visualization libraries that has automatically generated chart descriptions and decal patterns for users with disabilities.

Plotly

Looking for a bundle of open source libraries? With Plotly, you can create interactive charts and maps for Python, R, Julia, and multiple other languages.

You may want to use Plotly if you need:

A library for statistical programming languages. Plotly even has a MATLAB® library. Plotly’s fundamentals docs have plenty of code examples, so they’re a great place to start if you’re still learning the ropes.
To integrate with Jupyter notebooks or IDEs like PyCharm, QtConsole, or Spyder.
Standalone HTML files for your application. You can also use Dash to add graphs to your existing Python web applications.

When you might need more than a simple viz library

Visualization libraries are a good option if you’re already deep into component-driven development or have a skilled dev team on hand that can quickly implement and iterate on charts.

However, as you scale, maintaining a visualization library can mean taking on a lot of overhead. For example:

If you outgrow your visualization library in any way, you’ll either need to work with maintainers to add new features or fixes, or pivot to an entirely new visualization library.
Your dev team will need to handle ad-hoc requests for embedding and charting, or you’ll have to find a middle ground, like adding new interfaces and platforms to support teams that aren’t as familiar with a visualization library.

In these scenarios, using a Business Intelligence (BI) tool that has features to support both technical and non-technical teams can save you down the line.

Open-source Business Intelligence tools

Metabase OSS is a free option if you need a quick solution for embedding. You can create, embed, and iterate on charts at scale, in a matter of minutes. Other BI tools, like Apache Superset or Redash, are also open source and offer similar functionality.

Metabase offers some additional free, out-of-the-box features that can cover the bases for most, if not all of your teams:

A SQL editor for your data and dev teams, so they can quickly query against your database without hassle.
A data browser that can act as one centralized place for everyone to explore tables and fields; learn more about your data.
A data modeling tool, so your Data team can view schema, edit definition and metadata, and more.
An intuitive query builder and one-click visualization options for teams that aren’t as familiar with dev or data tooling. Everyone can build and maintain their own charts and visualizations without Eng or Data team support.

To test it out, check out the OSS installation instructions.

More charts and Embedding resources

Understand how your team uses Metabase

Thu, 11 Jan 2024 09:10:18 +0000

Metabase usage analytics is the new version of auditing tools on Pro and Enterprise plans, available as a collection of interactive dashboards, questions, and models built on your usage data.

With Metabase usage analytics you can see:

Which dashboards and questions are the most popular
How many weekly active Metabase users you have
Which groups are most actively using dashboards and questions
Who’s creating and viewing what content

This guide gives a few real-world examples of how you can take advantage of these usage analytics.

There are many factors that may determine how you use usage analytics, like industry standards and regulations, team and org size, structure, etc. This guide isn’t a specific set of rules, so feel free to take these ideas and build on them for your use case.

Track and monitor model and data performance as a data team

Our data team uses Metabase usage analytics to track and monitor question and dashboard usage and performance to ensure things run smoothly. Usage analytics allows the team to track which questions people view the most (e.g. Seeing if their latest activation models are being used in questions).

The prebuilt Most viewed content dashboard lets you filter by group name. Once you know what people are most interested in viewing, you can structure your Metabase so that the most used stuff is more discoverable, like by putting certain metrics on your Metabase home page.

You can also set up as a dashboard subscription to a Slack #general channel to send relevant data to your teammates.

Find out which questions and dashboards you could archive

We use usage analytics to find out which dashboards or questions are no longer being used and retire them. To see this, take a look in the Content with cobwebs prebuilt dashboard and archive unused content.

We also use usage analytics to find dashboard questions that take a long time to load and refactor or cache results. To see this yourself, go to the Performance overview prebuilt dashboard for a full overview of your slowest content.

Quickly catch changes to configurations and settings

We track changes to configurations and settings, like that time someone changed the language of our instance from English to Brazilian Portuguese, or disabled Google single sign-on.

To see changes to settings, you can go to the activity log model, and filter on setting-update. You can continue to keep tabs on them without living in Metabase by setting up subscriptions or alerts if something changes.

Monitor who has access to, and is downloading, your data

We use Metabase for compliance, auditing, and monitoring data security. It’s important to note that the definition of what we do can look very different compared to other organizations. How you use usage analytics in this situation can vary greatly depending on scale, industry, and the needs of your organization.

For Metabase, we’re mainly focused on SOC2 Type 2 compliance. We want to know who has access to what, whether that access is really needed. If not, we change the account’s permissions, or deactivate the account entirely.

For example, your IT team could use usage analytics when offboarding team members or restricting access by teams. Having an overview of data access available in Metabase helps us ensure that each person is mapped to the correct user group and has the right access levels and permissions.

To view which groups a person belongs to, go to Person Overview dashboard. You can filter on a person’s name, email, or user ID, looking specifically at date range, activity type, query source, and more. The dashboard includes details about the person, groups they belong to, how many questions and dashboard they’re viewing month over month, and what they’re looking at.

Audit logs

The Metabase analytics collection also comes with audit logs to help you understand who is downloading and accessing your data. For example, the UI only allows you to see up to 10k rows of data, but in audit logs you can see if someone downloaded 1M rows of data.

Audit logs are a feature we’ve used as part of our incident recovery plan. For example, if someone sees or does something on accident, audit logs lets us know what happened so we can prevent it from happening again. We report on any kind of data breaches as they occur, so any data we have to investigate and provide context is helpful.

To view audit logs, you can go to the Metabase analytics collection and click on any of the log models under Useful data. For example, you can view the Activity Log and click on any ID to see the audit log details.

Play around with Metabase usage analytics

You can get to Metabase usage analytics by clicking on “Metabase analytics” under your collections. Try out any of these use cases, or come up with your own workflow.