Over the past few years, the data world has convinced itself that it needs many different tools to extract insights:

• one tool to ingest data

• another one to transform it

• another one to check the quality

• another one to orchestrate them all

• another one for cataloging

• another one for governance

The result? A very fragile, expensive, and rigid infrastructure with a terrible experience. Teams have to build a lot of glue between these systems, and try to get different parts of these systems to talk to each other while trying to onboard analytical teams on them.

Does it work? No.

Are we ready to have that conversation? I hope so.

Obsessing over impact

The engineering work behind building a ticking machine is very satisfying: small pieces that each do their part, and they work like a clock. It feels like an engineering marvel:

It is simple: you push code to the main branch, the backend automatically pulls the branch, pulls the DAGs and uploads them to S3. The sync sidecars on Airflow containers automatically pull from S3, which will then update the DAGs. When the DAG runs, for data ingestion jobs it will connect to our Airbyte deployment and trigger the ingestion from Airflow, then we create a sensor that waits until the ingestion is done. Then we connect to dbt Cloud to initiate some parts of the transformation jobs from the analytical team, if anything fails Airflow connects to our notification system to find the right team if they are defined on the catalog, if not we check our AD users and try to find a matching org to send a notification. Once the transformation is done then we execute our custom Python operators that do X and Y, then we provision a pod in our Kubernetes cluster to run quality checks. Our Kafka sinks are ingesting CDC data from the internal Postgres with Debezium in the meantime, then load them to the data lake in Parquet format, then we register them as Glue tables so that they can be queried, then the sensors in our Airflow clusters keep track of these states to run SQL transformations with the internal framework, and…

Sounds ridiculous, doesn’t it? It certainly does to me, whereas this is a very common response when we ask engineering teams what their data infrastructure looks like. The joy they get out of building a house of cards is far more important than the business impact that is being delivered. In the meantime, the analytics teams, data analysts, data scientists, and the business teams are waiting for their questions to be answered, trying to understand why it takes 6 weeks to get a new chart on their sales dashboard.

I am not sure if this is due to ZIRP or not, but it is pretty easy to spot organizations where highly inefficient engineering teams, coupled with engineering leaders who don’t know what their teams are doing, are ruling the game, where the people that create real value with data is left alone. They have to jump through a billion different tools, trying to figure out why their dashboard didn’t update, and waiting for a response to their ticket from the central data team.

These are data analysts in business teams, a growth hacker running marketing campaigns over 5 different platforms, or an all-rounder data scientist who is trying to predict LTV. They are trying to create a real impact, but their progress is being heavily hindered by the internal toys.

We are building Bruin for these people: simpler data tooling for impact-obsessed teams.

Bruin CLI & VS Code Extension

Bruin CLI is an end-to-end data pipeline tool that brings together data ingestion, data transformation with SQL and Python, and data quality in a single framework.

https://github.com/bruin-data/bruin

Bruin is batteries included:

• 📥 ingest data with ingestr / Python

• ✨ run SQL & Python transformations on many platforms

• 📐 table/view materializations, incremental tables

• 🐍 run Python in isolated environments using uv

• 💅 built-in data quality checks

• 🚀 Jinja templating to avoid repetition

• ✅ validate pipelines end-to-end via dry-run

• 👷 run on your local machine, an EC2 instance, or GitHub Actions

• 🔒 secrets injection via environment variables

• VS Code extension for a better developer experience

• 📦 easy to install and use

This means using Bruin, teams can build end-to-end workflows without having to resort to a bunch of different tools. It is extensible enough with the usage of SQL and Python, while also guiding the users through its opinionated approach to build maintainable data pipelines.

Code-first & UI supported

One of the things that accompany Bruin CLI is our open-source Visual Studio Code extension:

The extension does a few stuff that makes it pretty unique:

• While everything in Bruin is driven with code, the extension adds a UI layer on top of it, which means you get:

  • visual documentation

  • rendered queries

  • column & quality checks

  • lineage

  • the ability to validate code & run backfills

  • syntax highlighting

• everything happens locally, which means there are no external servers or systems that can access any of your data

• extension visualizes a lot of the configuration options, which makes it trivial to run backfills, validations, and more.

This is a good example of our design principles: everything is version-controlled, while also giving a better experience through a thoughtful UI.

The extension is a first-class citizen of the Bruin ecosystem, and we intend to expand its functionality further to make it the easiest platform to build data workloads out there.

Supported Platforms

Bruin supports many of the cloud data platforms out of the box at the launch:

• AWS Athena

• Databricks

• DuckDB

• Google BigQuery

• Microsoft SQL Server

• Postgres

• Redshift

• Snowflake

• Synapse

The list of platforms we support will grow more and more over time. We are always looking forward to hearing community feedback on these, so feel free to share your thoughts with us in our Slack community.

Bruin Cloud

We are building Bruin for those obsessed with impact. You can go from zero to full data pipelines in minutes, and we are dedicated to making this experience even bigger. Using all of our open-source tooling you can build and run all of your data workloads locally, on GitHub Actions, in Airflow, or anywhere else.

While we do believe there are many useful deployment options of Bruin CLI across different infrastructures, we are also obsessed with building the best managed-experience for building and running Bruin workloads on production. That’s why we are building Bruin Cloud:

It has quite a few niceties:

• managed environment for ingestion, transformation, and ML workloads

• column-level lineage

• governance & cost reporting

• team management

• cross-pipeline dependencies

• multi-repo “mesh”

and quite a few more. Feel free to drop your email to get a demo.

Share your thoughts

We are very excited to share Bruin CLI & VSCode Extension with the world, and we would love to hear from the community. We’d appreciate if you shared your thoughts on what would make Bruin more useful for your needs.

https://github.com/bruin-data/bruin

We often see data teams considering no-code solutions as a potential option in the early days of the journey. The whole thing sounds pretty nice on paper:

• a drag-and-drop UI to build workflows

• visual representation of the pipelines

• no need to install anything

• even business users can build data products

These are very exciting at their core, and quite understandably many teams take these promises for granted. My intention with this article is not to say they are not the right solution for certain problems, but to offer another perspective for growing teams.

To be locked in or not to be locked in

It all starts the same: you have a new platform, it has a beautiful UI, you can do the first steps without writing a single line of code, and you feel like you are making progress. In fact, you are making progress, but with some costs associated with it.

The UI-driven workflow tools will have certain ways of working, primarily driven through their UI, duh, and their own workflows. While clicking around in the UI has its benefits, especially when getting started, there’s a reason these tools ask you to do that.

• The workflows you build there will stay with them forever unless you are ready to do the same steps in another tool, which is often not feasible after a certain stage.

• The ways of working cannot be extended other than the abilities of the platform, which means as you grow, you start hitting more and more of the rough edges and the lack of flexibility the business requires.

• Moving away from these platforms require a significant effort and investment, which means in the best case you are left with a subpar solution, in the worst case the platform can decide to charge you 10x more.

The inability to move away is the one that scared me the most always, let’s dive into that a bit more.

The inability to move away

Businesses change. Team structures change. Individuals change. Problems change. The tools, also, need to change.

Let’s think of the following scenario where we used a no-code data pipeline tool:

• Acme Corp, where Laura (Data Architect), Michael (Chief Data Officer), and Nina (Marketing Analyst) work, has a rather simple setup:

  • They ingest data from Facebook Ads, Google Ads, and Shopify.

  • They do some data cleaning.

  • They put them in a report on PowerBI.

Simple enough, nice.

Step 1: Initial implementation

Laura, the Data Architect, was tasked with integrating a new data source, Amazon Marketplace, into their existing no-code data pipeline tool. The tool was already ingesting data from Facebook Ads, Google Ads, and Shopify, with basic data cleaning and reporting set up in PowerBI. Laura began by connecting the Amazon Marketplace data source to the no-code tool. The initial setup seemed straightforward, and the data started flowing into the pipeline.

Step 2: First struggles

As the marketing team ramped up their campaigns, the volume of data from all sources increased significantly. The no-code tool began to struggle with the increased load, leading to slower data processing times and delays in report generation. What used to take 4 minutes to build now takes 3.5 hours to process data every morning, increasing every day.

Step 3: Michael’s business questions

Michael, the Chief Data Officer, requested more complex data transformations to better analyze customer behavior across all platforms. He needed to understand cross-channel attribution, customer lifetime value, and segment performance more accurately to optimize marketing spend and improve customer targeting. These transformations required advanced calculations and custom logic that the no-code tool could not handle efficiently.

To address this, Nina decided to run the complex logic in Python. However, the no-code platform did not support running custom scripts natively. Nina had to collaborate with a software engineer to deploy the Python scripts on AWS Lambda. This solution allowed the custom logic to be executed, but it added an extra layer of complexity and dependency outside the no-code tool.

Step 4: Expanding Data Sources and Integrations

As the business continued to grow, Michael saw the need to integrate additional data sources, including third-party customer data platforms (CDPs) and external market intelligence services. These sources were crucial for gaining deeper insights into customer behavior and staying competitive in the market.

However, the no-code tool struggled to handle the growing number of integrations. It lacked the flexibility to connect seamlessly with the wide variety of APIs and data formats required by these new sources. Laura had to write custom scripts to bridge the gaps, often relying on external services like AWS Lambda and AWS Glue for data transformation and ETL processes.

To complicate matters further, the no-code tool's interface became increasingly cluttered and difficult to manage as more data sources were added. This led to frequent errors and delays in the data pipeline, requiring constant troubleshooting and maintenance.

Step 5: Data Governance

As the company expanded into new markets, Michael needed to ensure that their data processing adhered to strict data governance and compliance requirements, such as GDPR and CCPA. The no-code tool offered limited features for managing data lineage, auditing, and encryption, making it difficult to maintain compliance across multiple jurisdictions.

Laura had to manually implement additional security measures and auditing processes outside the no-code platform, using a combination of AWS services and custom scripts. This made the pipeline even more complex and introduced new points of failure.

Outcome

As the data requirements became more complex, and the need for robust integrations, accurate segmentation, and compliance grew, the limitations of the no-code tool became a significant bottleneck. The team, including Nina, spent more time managing and troubleshooting the pipeline than deriving insights from the data.

The story is far more common than you would think. We have spoken with hundreds of companies, and all of those that adopted a no-code solution for their data infrastructure had also gone through -or were going through- an offboarding process.

Flexibility: yes-code

We build software because we can shape it as we want, hence the “soft” in software. The things we dream of can turn into reality in skillful hands, and building data workloads is not an exception.

Well, except, if you are using a no-code tool.

The no-code platforms are optimized for 80% of the workloads: they take care of the common scenarios in a, supposedly, easy way, and you can build the remaining 20% yourself.

This would work really well if the effort for that 20% scaled in a way where it complements the no-code part. However, in reality, the additional effort to build that custom part requires a significant investment that is pretty much rebuilding the whole thing in itself.

Why do we have to rebuild things again if I have my no-code platform already? Glad you asked.

There are quite a few questions that need to be answered for that custom part of your data workloads:

• Where do I run these workloads?

• How do I deploy them?

• How do I schedule them?

• How do I get them to work with the existing parts of the no-code pipelines?

• How do I get notified of failures?

• How do I retry failures?

So on and so forth…

Without having a clear strategy on how to build and run custom data workloads, i.e. your bespoke Python scripts, building and deploying the first custom stuff outside of the no-code platform means having to figure out all of these questions. This means setting up orchestration from scratch, figuring out a way to ingest data to another location, building a framework around all of these, deploying them somewhere, building notifications, etc.

Governance? What is that?

Data governance, even though it sounds too-enterprisey these days, is a concern that hits every company eventually. It doesn’t have to be a complicated process, data governance concerns start even in very early stages:

• What data do I store?

• Where do I store that data?

• Who can access that data?

• Do I comply with the regulations?

• How much do I pay for this data?

• How often is it being used?

• What’s the ROI of a specific piece of data?

The list grows from here on. Some of these questions might be easy to answer, whereas some others are more challenging. In order to be able to answer these questions in an accurate and timely way, you will often need to access the raw source of the data workloads, a.k.a “the code”, and guess who doesn’t have “the code”? The no-code platform.

This simply means the governance questions are answered based on what’s available on the platform, rather than giving you access to the underlying code. Even if you had the engineering ability to analyze the underlying SQL queries yourself, you won’t be able to, since you don’t have the code available to you.

Go for the code, always

In the end, there’s a fundamental disagreement between myself and no-code tools: I think teams should not be deprived of their freedom when it comes to their own data.

The code is the representation fo the real world. It gives you the flexibility to represent all the intricate complexities of your business. There’s never going to be a piece of off-the-shelf software that can cover every complexity of your business, which means you might as well start the right way: go for the code.

This is even more significant when it comes to your data workloads: data grows faster than the software stack, which means the complexities and the processing of the data will grow faster as well, which means you need that freedom more with your data workloads than ever.

That’s why we are building Bruin, it is the platform that I wished always existed: just let me write the code for my business logic and take care of the rest. In practice, this means you write your data transformation code, you version-control it, you add quality checks, everything is in code, and the platform takes care of the generic problems such as running them at the right time and in the right order, reporting executions, notifying you on failures, and more.

Get Started with Bruin

There are already a lot of teams that have been bitten by the pains of the no-code tools, and I am very excited for a future where this doesn’t happen anymore.

It’s the third blog post of Modeling Google Analytics & Firebase Data. You can find the previous posts:

1. Exporting Firebase Data to BigQuery

2. Firebase Events Table

A healthy Data Warehouse (DWH) consists of a balanced combination of aggregated data models and raw event data. Aggregated models help in several ways:

• Lower compute costs by reducing the volume of data processed.

• Higher performance and faster queries due to the smaller dataset size.

• Simplified querying with embedded business metrics and facts.

• Consistent analysis by standardizing metrics and data cleaning.

• Easy feature calculation for ML pipelines

Probably the most common data model in clickstream events is daily user aggregates regardless of business/industry since most of the dashboards are built on daily granularity. I like naming tables by the primary keys since it makes it more semantic. So I’ll call this table `users_daily`.

This table will contain 1 row for each active user on a specific day. At the end, it will include all the user metrics and attributes to help us understand the business. It’s very easy to create dashboards on top of this dashboard, so you’ll love it.

In the previous article we created an `events.events` table, so `users_daily` will be built on top of this table. Let’s start by having the most simple metrics of Google Analytics Sample Gaming Project.

The starter template is:

SELECT
    user_pseudo_id,
    dt,
    --- attributes
    --- metrics
FROM events.events
WHERE -- data cleaning
GROUP BY 1,2

In this query, you can use user_pseudo_id instead of user_id, but the main idea is we should never add any other columns to the group by. This way, we’ll ensure the primary key (PK) is consistent.

Where

This table contains only user activity data. So I recommend excluding the data not relevant to user activity. For example, Firebase triggers app_remove, os_update, app_clear_data, app_update and app_exception events even the app is not open.

Also, if a row doesn’t have a user_id, it doesn’t make sense to aggregate them either. Let’s exclude them, too.

So now, our query looks like:

SELECT
    user_id,
    dt,
    --- attributes
    --- metrics
FROM events.events
WHERE 
    user_id is not null
    and event_name not in ("app_remove", "os_update", "app_clear_data", "app_update", "app_exception")
GROUP BY 1,2

Metrics

Metrics are the aggregates that shows what users did within the day. The simplest way to obtain the main metrics is to count the main events.

When I query the table, I see the events as metric candidates:

SELECT event_name, count(*) as cnt
FROM `bruin-blog-firebase-demo.events.events_json` 
WHERE event_name not in ("app_remove", "os_update", "app_clear_data", "app_update", "app_exception")
group by 1

SELECT
    user_id,
    dt,
    --- attributes
    --- metrics
    count(*) as event_cnt,

    -- app monetization metrics
    countif(event_name="ad_reward") as ad_reward_cnt,
    countif(event_name="in_app_purchase") as iap_cnt,

    -- app engagement metrics
    countif(event_name = "level_start") as level_start_cnt,
    countif(event_name = "level_end") as level_end_cnt,
    countif(event_name = "level_complete") as level_complete_cnt,
    countif(event_name = "level_fail") as level_fail_cnt,
    countif(event_name = "level_retry") as level_retry_cnt,
    countif(event_name = "level_start_quickplay") as level_start_quickplay_cnt,
    countif(event_name = "level_end_quickplay") as level_end_quickplay_cnt,
    countif(event_name = "level_complete_quickplay") as level_complete_quickplay_cnt,
    countif(event_name = "level_fail_quickplay") as level_fail_quickplay_cnt,
    countif(event_name = "level_retry_quickplay") as level_retry_quickplay_cnt,
    countif(event_name = "level_up") as level_up_cnt,
    countif(event_name = "challenge_a_friend") as challenge_a_friend_cnt,
    countif(event_name = "challenge_accepted") as challenge_accepted_cnt,
    countif(event_name = "no_more_extra_steps") as no_more_extra_steps_cnt,
    countif(event_name = "use_extra_steps") as use_extra_steps_cnt,
    countif(event_name = "select_content") as select_content_cnt,
    
FROM events.events
WHERE 
    user_id is not null
    and event_name not in ("app_remove", "os_update", "app_clear_data", "app_update", "app_exception")
GROUP BY 1,2

As you can see, I tried to add all the relevant events to the summary table. This will help us to monitor the user behavior in the dashboards very quickly.

In the second iteration of the metrics, we will use parameters. This process requires a good understanding the events and their parameters. For example:

level events have these parameters:

SELECT event_name, ep as event_params
FROM `bruin-blog-firebase-demo.events.events_json` 
WHERE event_name like "level%"

By using these events, we can obtain the min&max level played on that day. To do it, we can just add these to the query:

SELECT
    user_id,
    dt,

    min(level) as min_level,
    max(level) as max_level,

FROM events.events
WHERE ...
GROUP BY 1,2

Here, you should be very careful that the level parameter is not used in other levels in another context. If it’s used, you can filter your query for specific metrics:

    min(if(event_name like 'level_%', level, null)) as min_level,
    max(if(event_name like 'level_%', level, null)) as max_level,
    min(if(event_name = 'level_complete', level, null)) as min_level_completed,
    max(if(event_name = 'level_complete', level, null)) as max_level_completed,

Let’s inspect the in_app_purchase event to add the revenue to our table:

SELECT event_name, event_params, event_value_in_usd
FROM `bruin-blog-firebase-demo.events.events_json` 
WHERE event_name = "in_app_purchase"

As you can see, there are currency and price parameters. However, since they have different currencies, using it is impossible. To solve this problem, Google Analytics provides the `event_value_in_usd` field. Let’s add it to our metrics as well:

    sum(if(event_name="in_app_purchase", event_value_in_usd, 0)) as iap_revenue

After that point, I recommend learning the product very well, deciding on the metrics with product managers, and adding all of them to the users_daily table. The only exception is, for calculated metrics, I recommend you to do it on the BI layer.

For example, if you want to calculate win_rate, it’s better to add level_start_cnt and level_complete_cnt to this table. By using level_complete_cnt / level_start_cnt, you can easily calculate the win rate.

Attributes

Attributes are non-aggregatable information that provides additional details about the user, such as the user’s device brand, country, OS version, and platform. To summarize these attributes, I recommend using two different methods based on their types:

1. Dynamic Values

2. Static Values

Let’s start with the country attribute. A user can use your app in more than one country during a single day. For such dynamic values, it makes sense to capture the first and last values of the day. Country, os_version, app_version are examples of this group.

To get them, you can use min_by and max_by functions.

SELECT
    user_id,
    dt,

    MIN_BY(country, ts) as first_country,
    MAX_BY(country, ts) as last_country,

FROM events.events
WHERE ...
GROUP BY 1,2

Some of the values don’t change for a specific user id. For example, user_pseudo_id is assigned right after installing the app. If you are using user_pseudo_id as the primary key, we expect device platform, brand, device model values to be stationary. For that kind of attributes, we can use any_value since it doesn’t change for a user.

SELECT
    user_pseudo_id,
    dt,

    ANY_VALUE(platform) as platform,
    ANY_VALUE(device_brand) as device_brand,
    ANY_VALUE(device_model) as device_model,

FROM events.events
WHERE ...
GROUP BY 1,2

If you are controlling your user ids from a backend system, I don’t recommend using ANY_VALUE, but FIRST/LAST since all the attributes become dynamic.

Lastly, I want to mention user behaviour attributes. They show user behaviour like metrics, but still they’re mostly strings, so summing, counting doesn’t make sense on them.

One example is screen or event_name. If you want to measure where the users leave your app, you might want to calculate the first/last screen and events.

SELECT
    user_pseudo_id,
    dt,

    MIN_BY(event_name, ts) as first_event,
    MAX_BY(event_name, ts) as last_event,
    MIN_BY(screen, ts) as first_screen,
    MAX_BY(screen, ts) as last_screen,

FROM events.events
WHERE ...
GROUP BY 1,2

When we bring everything together, you’ll have a query like:

SELECT
    user_id,
    dt,

    MIN_BY(country, ts) as first_country,
    MAX_BY(country, ts) as last_country,
    MIN_BY(device_brand, ts) as first_device_brand,
    MAX_BY(device_brand, ts) as last_device_brand,
    MIN_BY(os_version, ts) as first_os_version,
    MAX_BY(os_version, ts) as last_os_version,
    MIN_BY(device_language, ts) as first_device_language,
    MAX_BY(device_language, ts) as last_device_language,

    count(*) as event_cnt,
    min(ts) as min_ts,
    max(ts) as max_ts,

    MIN_BY(event_name, ts) as first_event,
    MAX_BY(event_name, ts) as last_event,
    MIN_BY(screen, ts) as first_screen,
    MAX_BY(screen, ts) as last_screen,

    min(level) as min_level,
    max(level) as max_level,
    count(distinct timestamp_trunc(ts, minute)) as session_duration, 

    -- app monetization metrics
    countif(event_name="ad_reward") as ad_reward_cnt,
    countif(event_name="in_app_purchase") as iap_cnt,
    sum(if(event_name="in_app_purchase", event_value_in_usd, 0)) as iap_revenue,

    -- app engagement metrics
    countif(event_name = "level_start") as level_start_cnt,
    countif(event_name = "level_end") as level_end_cnt,
    countif(event_name = "level_complete") as level_complete_cnt,
    countif(event_name = "level_fail") as level_fail_cnt,
    countif(event_name = "level_retry") as level_retry_cnt,
    countif(event_name = "level_start_quickplay") as level_start_quickplay_cnt,
    countif(event_name = "level_end_quickplay") as level_end_quickplay_cnt,
    countif(event_name = "level_complete_quickplay") as level_complete_quickplay_cnt,
    countif(event_name = "level_fail_quickplay") as level_fail_quickplay_cnt,
    countif(event_name = "level_retry_quickplay") as level_retry_quickplay_cnt,
    countif(event_name = "level_up") as level_up_cnt,
    countif(event_name = "challenge_a_friend") as challenge_a_friend_cnt,
    countif(event_name = "challenge_accepted") as challenge_accepted_cnt,
    countif(event_name = "no_more_extra_steps") as no_more_extra_steps_cnt,
    countif(event_name = "use_extra_steps") as use_extra_steps_cnt,
    countif(event_name = "select_content") as select_content_cnt,
    
FROM events.events
WHERE 
    user_id is not null
    and event_name not in ("app_remove", "os_update", "app_clear_data", "app_update", "app_exception")
GROUP BY 1,2

Running on Production

Since the previous days’ data never changes, I recommend using an incremental flow for this table. You can find the full version of this pipeline here.

Also, you can use `bruin init firebase` in your terminal to bring all Firebase Analytics pipeline to your computer.

If you have questions about this article or running Bruin pipelines, don’t forget to join our Slack community.

If you need help to run your data pipelines, visit Bruin website. Bruin is an end-to-end data product platform that empowers your data analysts/scientists by 10x!

Even though these functions make our lives easier, we can make it even better. get_param and get_prop parameters require you to type a lot of characters, and also you have to remember the data type every time. Also, since event_params contains all the other parameters, querying it becomes very expensive.

To avoid this problem, I'll recommend a new data model to simplify querying.

Problem 1 - Too many characters

To be able to get a single parameter, let's say level, I have to type these all the time:

fn.get_param_int(event_params, 'level') as level

The problems here are:

1. I have to repeat fn.get_param_int for every parameter

2. I have to repeat event_params every time

3. I have to know the data type of the level parameter (in this example int)

Problem 2- Query costs

BigQuery charges per data scanned, and it costs a lot every time we access event_params.

To understand why, let's check an example event_params row:

Here, even though the client sends only 1 parameter, board, Firebase automatically adds 3 other parameters.

Let's calculate the data size BigQuery scans only to access the parameter “board”.

Since event_params is an array, we have to scan all the details in the array (there are very small optimizations behind, but for now I ignore them)

key column:

firebase_screen_class: 21 characters
firebase_event_origin: 21 characters
firebase_screen_id: 18 characters
board: 5 characters
total: 65 characters

values.string_value column

game_board: 10 characters
app+gtm: 7 characters
S: 1 character
total: 18 characters

So just to get a single character, `S`, BigQuery had to scan 65 + 18 = 83 characters. So this query could be 83 times cheaper!

To make this calculation, I already ignored the int and float value columns.

Also, this event has only 4 parameters, but most have many more parameters.

So imagine there are 15 parameters with an average of 15 characters. It would mean scanning 225 characters just for a single character! It’s a crazy amount of savings. (1/225 = Only 0.4% of the total scans)

Solution 1 - Unnested Events Table

So how can we solve this problem?

If we had a column named board, it would be great, right?

Imagine all the parameters are already a column in the events table, and BigQuery autocompletes your query, and also it costs much much less...

In this approach, we run an unnesting query periodically, so this query only runs once and then the rest of the queries are done on the new table.

An example unnested events query would be like:

CREATE TABLE events.events 
PARTITION BY dt CLUSTER BY event_name
AS
SELECT
	app_info.id AS app,
	platform,
	PARSE_DATE('%Y%m%d', event_date) AS dt,
	TIMESTAMP_MICROS(event_timestamp) AS ts,
	TIMESTAMP_MICROS(user_first_touch_timestamp) AS install_ts,
	user_pseudo_id,
	user_id,
	lower(event_name) as event_name,
	app_info.version AS app_version,
	device.advertising_id,
	device.vendor_id,
	geo.country AS geo_country,
	device.category AS device_type,
	device.mobile_brand_name AS device_brand,
	device.mobile_marketing_name AS device_marketing_name,
	device.mobile_model_name AS device_model,
	device.mobile_os_hardware_model AS device_hardware_model,
	device.language AS device_language,
	CASE lower(device.is_limited_ad_tracking)
		WHEN 'yes' THEN True
		WHEN 'no' THEN False
	END AS device_limited_ad_tracking,
	device.operating_system AS device_os,
	device.operating_system_version AS device_os_version,
	event_server_timestamp_offset / 1000 AS event_server_timestamp_offset,
	device.time_zone_offset_seconds / 3600 AS device_time_zone_offset_seconds,
	func.get_param_str(event_params, 'firebase_screen') AS screen,
	func.get_param_str(event_params, 'firebase_screen_class') AS screen_class,
	func.get_param_int(event_params, 'ga_session_id') AS session_id,
	func.get_param_int(event_params, 'ga_session_number') AS session_number,
	func.get_param_int(event_params, 'board') AS board,
FROM `firebase-public-project.analytics_153293282.events_*`

In this query, you can add all your event parameters and user properties, then schedule it to run once a day. You will have a very clean events table and it will be much cheaper to query.

However, this approach has a problem. As your events evolve and you send more and more parameters, it will be hard to maintain this table.

For every new parameter, you have to create a new column like this, first:

ALTER TABLE events
ADD COLUMN new_column INT64;

If you are late adding the new parameters to the table, you have to backfill it for the previous days.

Solution 2- Json Events Table

In previous years, BigQuery introduced a relatively new datatype: JSON.

JSON datatype gives flexibility in terms of new keys/parameters, and also it's stored as columnar in the storage layer. So, if you access a parameter in a JSON field, it would still be very cheap compared to the event_params column.

Here, I come up with another function:

CREATE OR REPLACE FUNCTION `fn.event_params_to_json`(
event_params ARRAY<STRUCT<
key STRING,
value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64>
>>
)
AS ((
with t1 as
(
	select
		array_agg(p.key) as keys,
		array_agg(coalesce(
			p.value.string_value,
			cast(p.value.int_value as string),
			cast(coalesce(p.value.double_value, p.value.float_value) as string)
		)) as vals
	from unnest(event_params) as p
	where key not in ("firebase_screen", "firebase_screen_class", "ga_session_id", "ga_session_number", "engaged_session_event", "engagement_time_msec", "firebase_event_origin", "firebase_previous_id", "firebase_screen_id")
)
select case when array_length(keys) > 0 then json_object(keys, vals) end
from t1
));

This function converts all the parameters to string and then creates a JSON object from them.

The reason I converted all of them to string is that BigQuery doesn't allow creating a JSON object with different data types.

At the end, you'll have a dictionary like:

{
"firebase_screen_class": "game_board",
"firebase_event_origin": "app+gtm",
"firebase_screen_id": "-8894705......",
"board": "S"
}

To create a JSON event_params column, you can use this query:

CREATE TABLE events.events_json
PARTITION BY dt CLUSTER BY event_name AS
SELECT
	app_info.id as app,
	platform,
	PARSE_DATE('%Y%m%d', event_date) as dt,
	TIMESTAMP_MICROS(event_timestamp) as ts,
	TIMESTAMP_MICROS(user_first_touch_timestamp) as user_first_touch_ts,
	user_pseudo_id,
	user_id,
	lower(event_name) as event_name,
	fn.parse_version(app_info.version) as app_version,
	fn.event_params_to_json(event_params) as ep,
FROM `firebase-public-project.analytics_153293282.events_*`

And then, you can use this query to access the board parameter:

SELECT lax_string(ep.board) as board
FROM events.events_json

With this query, we gained flexibility but lost query comfort. Now, we have to type more characters. To fix it, I recommend creating a view on top of this table, so it will be very similar to solution 1, but also flexible. Views don't occupy space in the storage but use the underlying tables to retrieve data.

To create the view, you can use this query:

CREATE OR REPLACE VIEW events.events AS
SELECT
	* except(up, ep),
	lax_string(ep.board) as board,
FROM events.events_json

So every time you query the table events.events, it actually goes through events.events_json table and retrieves the recent data. In the case there is a new parameter, the only thing you need to do is add it to events.events view.

Running this pipeline on production

Event tables are incremental tables and older days' data is not updated. If you're using intraday, it doesn't backfill older data, if you're using batch export, every day it updates the last 3 days of data. Based on my personal experience, added data to previous days tends to be ~1-2% of the total events, so you might choose to ignore it.

Since data is only added incrementally, you don't have to create the whole events table every day, and instead run incremental pipelines. While incremental strategies can go a long way, it is pretty simple for our article: you can just read yesterday’s data and append it to the current table. This way, your ETL costs will decrease significantly, and your ETL process will be completed much faster.

I only shared the CREATE queries above to keep it short, but in this repo, you can find how to make them incremental using Bruin.

Running Bruin

To run the pipeline steps in the right order, you need what’s called an “orchestrator”. Bruin is an open-source orchestrator that allows you to define your data pipelines as a collection of assets, and run them in your data warehouse.

You can install Bruin CLI by following the instructions here.

If you already have Bruin CLI, you can use the command below in the terminal to get a fresh start and it will automatically create a starter template for you. You can update this pipeline according to your needs and use it in the production.

brew tap bruin-data/tap
bruin init firebase

As a result of this command, you will have a fully-prepared production-ready data pipeline based on our articles, applying the best practices of managing Firebase data.

If you have questions about this article or running Bruin pipelines, join our Slack community.

In the next article, I'll explain how to create a daily summary table and we will create our first dashboard on Looker Studio. You can subscribe to this newsletter to get updated about the next article.

One of the first issues companies run into when it comes to analyzing their data is having to move the data off of their production databases:

• The production instances have different abilities that are not suitable for analytical workloads, such as row-oriented vs column-oriented databases.

• Analytical use-cases have very different SLAs and requirements than production applications, which means their reliability needs are very different. An analytical query may take 1.2s instead of 400ms and that would be fine, whereas that latency could be a huge disruption to the user experience if it happened on the production database.

• Online Transaction Processing (OLTP) databases are focused on transactional use-cases, which means they lack quite a few features around data analysis.

Due to these contributing factors, companies usually move their data off to an analytical database such as Google BigQuery or Snowflake after a certain size and scale for analytical purposes. 

While moving the data to a database that is fit for purpose sounds good, it has its own challenges:

• The data needs to be copied over at a regular cadence via some tool/custom code. This means extra effort and cost to build and maintain.

• The data is now duplicated across multiple databases, meaning that changes in the source need to be accurately reflected in the analytical database.

• The assumptions around the atomicity/reliability of the data change since there are multiple places where the data resides now.

• After a certain data size, transferring the data becomes expensive and slow, requiring further engineering investment to make the process more efficient.

All of these reasons build up the problems around data ingestion, and there are already a bunch of tools in the market that aim to solve this.

Building everything from scratch 

The moment the data ingestion/copy problem is acknowledged, the first reaction across many teams is to build a tool that does the ingestion for them, and then schedule it via cronjobs or more advanced solutions. The problem sounds simple on the surface:

• Download a copy of the original data

• Upload it to the destination database either via SQL insert statements, or some other platform-specific way to load the data into the database

However, this on-the-surface analysis forgets quite a few crucial questions:

• How do you ensure the data is copied over accurately?

• What happens when there’s a bug that requires copying the data again?

• What if the data does not fit into the memory at once? How do you paginate?

• What happens as the data grows and copying/overwriting everything becomes too slow/expensive?

• What happens when the schema of the data changes?

• How does the team know about failures?

• Where do you deploy the logic?

• and quite a few more…

As you can see, there are many open points and they all require a solid understanding of the problem at hand, along with the investment to make the overall initiative success. Otherwise, the engineering team builds quick hacks to get them up and running, and these “hacks” start to become the backbone of the analytical use-cases, making it very hard, if not impossible, to evolve the architecture as the business evolves.

Some smart people saw the problem at hand and came up with various solutions to make this process easier.

No-code solutions

Over the years some teams have decided that data ingestion can be performed simply via UI-driven solutions that have pre-built connectors across various platforms, which means non-technical people can also ingest data. Two major players that come to mind are Fivetran and Airbyte, both giant companies trying to tackle the long-tail of the data ingestion problem.

Even though there are a few differences between these no-code platforms, their primary approach is that you use their UI to set up connectors, and you forget about the problem without needing any technical person, e.g. a marketing person can set up a data ingestion task from Postgres to BigQuery.

While these tools do have a great deal of convenience, they still pose some challenges:

• The movement of the data is a fairly technical work that has quite a few questions about the ways of copying the data, and the data work rarely ends after just copying the data, therefore technical people such as data analysts, scientists, or engineers still need to be involved in the process, which means that the actual audience is the data people rather than non-technical folks.

• The UI-driven workflow causes lock-in, which means that the company is not going to be able to move away from these platforms until they build a replacement, which usually means further investment later on without disrupting the current ways of working + migrating the existing usecases.

• For the open-source solutions such as Airbyte, they still need to be hosted and maintained internally, which means engineering time and effort & infrastructure costs.

All in all, while the UI-driven data ingestion tools like Fivteran or Airbyte allow teams to get going from zero, there are still some issues that causes teams to stay away from them and resort to writing code due to the flexibility it provides.

Yes-code solutions: dlt

There has been an emerging open-source Python library from the company dltHub called dlt, which focuses more on the use cases where there’ll still be code written to ingest the data, but the code could actually be a lot smaller and maintainable. dlt has built-in open-source connectors, but it also allows custom sources & destinations to be built by teams for their specific needs. It is flexible but allows quick iteration when it comes to ingestion.

There are a couple of things dlt takes care of very nicely:

• dlt supports schema evolution, meaning that when the schema of an upstream dataset changes, dbt will make sure the destination tables are updated accordingly based on the changes.

• dlt supports custom sources and destinations, meaning that their prebuilt sources & destinations can be combined with custom ones.

• dlt has support for incremental updates and deduplication, which means that the data can be incrementally updated with only the changed data while ensuring it still matches the source data.

dlt is quite a powerful library and has a very vibrant, growing community. It might be the perfect companion when it comes to engineers wanting to write code for certain custom requirements.

However, we felt that there might be a middle ground for simpler use-cases that doesn’t require coding, but also doesn’t lock us into a UI-driven workflow.

While we like dlt a lot at Bruin, we felt that there were quite a few simpler scenarios that we couldn’t justify writing, maintaining, and deploying code for:

• Some of our customers wanted to be able to simply copy a source table to a destination, and override the data in the destination because the data was small enough.

• Some others required fixed incremental strategies, such as “just get the latest data based on the `updated_at` column.

• Some others needed to be able to merge the new records with the old ones incrementally.

While all of these are possible with dlt, it requires these people to write code and figure out a way to deploy them and monitor them. It is not incredibly hard, but it is also not trivial. Seeing all these patterns, we have decided to take a stab at the problem in an open-source fashion.

Introducing: ingestr

ingestr is a command-line application that allows you to ingest data from any source into any destination using simple command-line flags, without having to write any code and still having it as part of your tech stack.

• ✨ copy data from your Postgres / MySQL / SQL Server or any other source into any destination, such as BigQuery or Snowflake

• ➕ incremental loading

• 🐍 single-command installation: pip install ingestr

ingestr takes away the complexity of managing any backend or writing any code for ingesting data, simply run the command and watch the magic

.

ingestr makes a couple of opinionated decisions about how the ingestion should work:

• it is built as an open-source solution, and anyone can use it for any purpose based on its permissive license

• ingestr treats everything as a URI: every source and every destination has its own URI based on SQLAlchemy connection URIs.

• ingestr has a few built-in incremental loading strategies: replace, append, merge and delete+insert

While there will be quite a few scenarios where the teams would benefit from the flexibility dlt provides, we believe that 80% of the real-life scenarios out there would fall into these presets, and for those ingestr could simplify things quite a bit.

🌟 Give it a look and give us a star on GitHub! We’d love to hear your feedback and feel free to join our Slack community here.

While a comprehensive explanation of each column is available here, this post will focus on making the table more accessible.

Accessing Firebase Demo Project

Let's kick things off with the Firebase demo project. If you already have a project, you don’t need to follow this step, you can just jump to the next topic below.

The demo dataset resides in the firebase-public-project, and you can add it to your workspace with a simple click on the + ADD button near above the data catalog.

Select the Star a project by name option, and type firebase-public project in the window that appears.

Upon clicking 'Star,' you'll find this dataset conveniently located in the left pane.

Understanding Events Table

Once you start exporting your data to BigQuery, you'll encounter a very wide table in BigQuery which is hard to understand and contains too much information.

Depending on your export settings, Firebase exports events in 2 names:

• events_

• events_intraday_

These tables are sharded tables, and generates a table for each day. For example, for 30 September 2023, the table name will be events_20230930.

Let's write our first query to see the event counts on a specific day, "2018-10-03".

SELECT
	event_name,
	count(*) as event_cnt,
	count(distinct user_pseudo_id) as user_cnt,
	count(*) / count(distinct user_pseudo_id) as event_per_user
FROM `firebase-public-project.analytics_153293282.events_20181003`
GROUP BY 1
ORDER BY 2 desc

Firebase triggers some events like screen_view, user_engagement automatically, so if you haven't triggered any of them, you don't need to get worried.

Working with Event Timestamp Column

If you preview the events table, the first thing you’ll see is, event_timestamp is an integer, and not easy to understand.

To convert it to a timestamp, you can use the TIMESTAMP_MICROS function.

Let's find hourly level starts in our game. To understand how timestamp_trunc function works, you can check the documentation.

SELECT 
    timestamp_trunc(timestamp_micros(event_timestamp), hour) as hour, 
    count(*) as events
FROM `firebase-public-project.analytics_153293282.events_20181003`
GROUP BY 1
ORDER BY 1

As you can see here, even though we queried the data for 2018-10-03, we got data from 7:00 AM to the next day until 7:00 AM. It happens due to the reporting time zone in Google Analytics settings. There always will be a difference between event_date and event_timestamp, but to minimize it you can choose UTC as time zone.

Querying Events, Event Params and User Properties

Let's focus on level_start_quickplay event and see some of the events and parameters.

There is a `board` parameter in event_params, and the value is `S`. It keeps the board size value for the played level.

Let's count level starts per board, but event_params is an array. How do we access it?

For both event_params and user_properties, we can use this code to access the parameters:

(select value.{PARAMETER_TYPE}_value from unnest(event_params) where key = "{PARAMETER_NAME}")

So to access the parameter, let's use it in the query and count it:

SELECT
	(select value.string_value from unnest(event_params) where key = "board") as board,
	count(*) as cnt
FROM `firebase-public-project.analytics_153293282.events_20181003`
WHERE event_name = "level_start_quickplay"
GROUP BY 1

Get Param Functions

As you want to include more and more parameters, writing this query will be very annoying.

For that purpose, I created some functions for you. First, I recommend you to create a dataset called fn and store all the functions in that dataset. Don't forget to choose the same region as the events tables region for this dataset.

CREATE OR REPLACE FUNCTION fn.get_param_str(event_params ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64>>>, param_key STRING) AS 
((
	SELECT
		COALESCE(
			value.string_value,
			CAST(value.int_value as string),
			CAST(value.float_value as string),
			CAST(value.double_value as string)
		)
	FROM UNNEST(event_params) WHERE key = param_key	
));

CREATE OR REPLACE FUNCTION fn.get_param_int(event_params ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64>>>, param_key STRING) AS 
((
	SELECT
		COALESCE(
			value.int_value,
			SAFE_CAST(value.float_value as int64),
			SAFE_CAST(value.double_value as int64),
			SAFE_CAST(value.string_value as int64)
		)
	FROM UNNEST(event_params) WHERE key = param_key
));

CREATE OR REPLACE FUNCTION fn.get_param_float(event_params ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64>>>, param_key STRING) AS 
((
	SELECT
		COALESCE(
			value.float_value,
			value.double_value,
			CAST(value.int_value as FLOAT64),
			SAFE_CAST(value.string_value as FLOAT64)
		)
	FROM UNNEST(event_params) WHERE key = param_key	
));

CREATE OR REPLACE FUNCTION `fn.get_param_bool`(event_params ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64>>>, param_key STRING) AS (
	SAFE_CAST(fn.get_param_int(event_params, param_key) as bool)
);

After creating these functions, you can rewrite your query as:

SELECT
	fn.get_param_str(event_params, "board") as board,
	count(*) as cnt
FROM `firebase-public-project.analytics_153293282.events_20181003`
WHERE event_name = "level_start_quickplay"
GROUP BY 1

Let’s query post_score event with parameters:

First, to find the parameters, let’s check the table without manipulation:

SELECT 
    user_pseudo_id,
    timestamp_micros(event_timestamp) as ts,
    event_params
FROM `firebase-public-project.analytics_153293282.events_20181003`
WHERE event_name = "post_score"

Here, we can see that there are 4 parameters relevant to the event.

• level_name (string)

• level (double)

• time (double)

• score (double)

If we use our functions to unnest the parameters, the query looks like that. Since BigQuery doesn’t have a double datatype, doubles are queried as floats:

SELECT 
    user_pseudo_id,
    timestamp_micros(event_timestamp) as ts,
    fn.get_param_str(event_params, "level_name") as level_name,
    fn.get_param_float(event_params, "level") as level,
    fn.get_param_float(event_params, "time") as time,
    fn.get_param_float(event_params, "score") as score,
FROM `firebase-public-project.analytics_153293282.events_20181003`
WHERE event_name = "post_score"

Even though parameters are sent as double/float, it looks like they indeed are integers. Let’s verify it by checking all the values:

SELECT 
    fn.get_param_float(event_params, "score") as score,
    count(*)    
FROM `firebase-public-project.analytics_153293282.events_20181003`
WHERE event_name = "post_score"
group by 1
order by 1

As suspected, all the rows are actually integers. If we do the same check for level and time, we see that they are also integers.

So let’s rewrite our query:

SELECT 
    user_pseudo_id,
    timestamp_micros(event_timestamp) as ts,
    fn.get_param_str(event_params, "level_name") as level_name,
    fn.get_param_int(event_params, "level") as level,
    fn.get_param_int(event_params, "time") as time,
    fn.get_param_int(event_params, "score") as score,
FROM `firebase-public-project.analytics_153293282.events_20181003`
WHERE event_name = "post_score"

We get the same table, but with correct datatypes this time.

get_param_* and get_prop_* functions are automatically casting the data to the right data types, so you don’t have to cast them again.

Here is the code for get_prop functions:

CREATE OR REPLACE FUNCTION `fn.get_prop_str`(user_properties ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64, set_timestamp_micros INT64>>>, param_key STRING) AS (
(
    SELECT 
        COALESCE(
            value.string_value,
            CAST(value.int_value as string),
            CAST(value.float_value as string),
            CAST(value.double_value as string)
        )
    FROM UNNEST(user_properties) WHERE key = param_key
)
);

CREATE OR REPLACE FUNCTION `fn.get_prop_int`(user_properties ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64, set_timestamp_micros INT64>>>, param_key STRING) AS (
(
    SELECT 
        COALESCE(
            value.int_value,
            SAFE_CAST(value.float_value as int64),
            SAFE_CAST(value.double_value as int64),
            SAFE_CAST(value.string_value as int64)
        )
    FROM UNNEST(user_properties) WHERE key = param_key
)
);

CREATE OR REPLACE FUNCTION `fn.get_prop_bool`(user_properties ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64, set_timestamp_micros INT64>>>, param_key STRING) AS 
(
    COALESCE(
        SAFE_CAST(fn.get_prop_int(user_properties, param_key) as bool),
        SAFE_CAST(fn.get_prop_str(user_properties, param_key) as bool)
    )
);

CREATE OR REPLACE FUNCTION `fn.get_prop_double`(user_properties ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64, set_timestamp_micros INT64>>>, param_key STRING) AS (
(
    SELECT 
        COALESCE(
            value.double_value,
            value.float_value,
            CAST(value.int_value as FLOAT64),
            SAFE_CAST(value.string_value as FLOAT64)
        )
    FROM UNNEST(user_properties) WHERE key = param_key
)
);

Querying multiple days

Until here, we only queried 1 day of data, specifically 2018-10-03.

What if we want to query multiple days?

BigQuery gives you a flexible way of querying sharded tables. You can just use `*` instead of the date in the table name, and you will be able to query all the dates:

SELECT
    event_date,
    count(*) as cnt,
    avg(fn.get_param_int(event_params, "score")) as avg_score,
    avg(fn.get_param_int(event_params, "level")) as avg_level
FROM `firebase-public-project.analytics_153293282.events_*`
WHERE event_name = "post_score"
GROUP BY 1
ORDER BY 1

This query returns 114 rows, for days between 12 June and 3 October.

However, you might need only the days between instead of full data. It will be both cheaper since we scan less data, and also might be more useful for our analysis. In that case, we’re using _TABLE_SUFFIX to filter the dates:

SELECT 
    PARSE_DATE('%Y%m%d', event_date) as dt,
    count(*) as cnt,
    avg(fn.get_param_int(event_params, "score")) as avg_score,
    avg(fn.get_param_int(event_params, "level")) as avg_level
FROM `firebase-public-project.analytics_153293282.events_*`
WHERE event_name = "post_score"
    AND _TABLE_SUFFIX between '20180701' and '20180731'
GROUP BY 1
ORDER BY 1

This query returns only July’s data which I need, also it costs much less compared to querying all data.

I made another change in that query as well, instead of using event_date, I used the PARSE_DATE function because event_date is stored as a string, and it’s both hard to read and also impossible to apply date operations. So by using parse_date, we converted it to date datatype.

Even though it’s much easier to query now, we still need to know of the parameters name, datatype, etc. In the next article we’ll transform this table to make it easier and cheaper to query.

The mythical "data teams" have been living their prime time over the past few years. The first thing every company that is moving towards a data-driven culture does is to hire engineers for the things they wished they did well, without thinking why.

It's like the "DevOps" of 2010s: everyone claims they do it, but everyone understands the role –or the philosophy, for that matter– differently.

The whole story revolves around this idea that the lifecycle of data is too complex –it is!– for analysts/scientists to manage –not just for them, all of us!–, therefore we drop these engineers from outside into a highly complex business environment, expecting them to deliver value using data right away.

Of course, this never works: engineering teams spend months building infrastructure, restructuring the data, and building tooling for things that might not even be used by the rest of the business. In the end, the company is out millions of dollars, and likely worse than where they were before.

Delivering without impact

The idea that data would be dealt with outside the rest of the business by throwing more engineers on it is a very old idea. It often plays out as follows:

• The company builds software and becomes successful in some way.

• They generate a lot of data; although, not as a first-class citizen, always as an afterthought.

• There'll be a Jane in marketing, and a Joe in sales who knows how to pull some data to Excel and get some numbers out. The data and the story it tells are still an afterthought.

• The company grows further, but the "Jane"s and "Joe"s are not enough to serve the rest of the business. They decide to hire an analyst or two.

  • Guess what? Data is still an afterthought.

• The analysts are trying to help the rest of the business, but are unable to keep up with all the demand and feel abandoned while trying to navigate the hot mess.

• At some point, some exec will get mad at not being able to get some numbers and call the shots to build a data team.

This is where things get nasty because an issue that is primarily a cultural matter is being tackled by the means of throwing people on it. The budgets are secured, the openings are posted, and the applications start flowing in, with no change in how to think about data.

The engineers that are hired jump straight into throwing solutions on problems that are not real problems, simply because that's what they can influence rather than what should be done. The ability to deliver results is euphoric, regardless of their impact. A complex infrastructure is built to move a CSV from Google Drive to S3, and the leaders feel accomplished: "look at all this cloud bill we have! we are definitely data-driven."

Data, my friends, is still an afterthought.

Data as a core value

The reason that companies struggle with shifting away from the data being an afterthought mindset is the top-down approach towards getting value out of data: data is not like cash that a leader can decide how to use the best at the moment; on the contrary, data is like oil. Not in the sense of "data is the new oil yay", but in the sense that it needs a lengthy and expensive process to bring out its value, and that requires deep investment. You know people, they don't love deep investments.

The first step in this process is to treat the data as an asset, just as any other asset the company has, not something nice to have. The good use of data can propel the business way further than any accounting process could, or any lawyer that can protect the company, therefore it needs to be treated with the same importance. Do you leave your bookkeeping unattended? Then you should not leave your data as well. This needs to propagate the ranks: the data is incredibly valuable, no data should be wasted, and it should be treated with utmost care.

Once the importance of the data is clear to the rest of the business, then comes the first tangible action to take: data is not a separate entity from the software organization, make them own it.

Own the damn data

I have seen it repeated countless times that data is being treated as if it is a separate thing from the rest of the software –and the team that builds it–, which means that there's a data analyst on one side trying to duct-tape a bunch of tables together in some weird drag-and-drop tool, while the software team just drops a full table from production the next day.

The organizations that have the healthiest data landscape are those that have a very clear understanding of data ownership: every bit of data that is produced/ingested/transformed/used must have an owner, no discussion. Do you own the service that writes to this database? You own the data. Do you own the internal events being generated on Kafka? You own the data. Do you join these billion different tables into a new table? You own the data.

The most important point here is that the software teams are aware that the data they produce is owned by them. This means that they will be responsible for a few core questions to be answered properly:

• How will this data be made available to the rest of the organization?

• How will the quality of this data be ensured?

• How would anyone notice if the data went corrupt? How quickly?

• What is the change management process around this data?

This means that the software teams will start treating their data just as they are treating their services. Not sure about your experience, but the quality of software services has been treated way higher than the quality of the data they produced in my experience, which means this is a win in my book.

Data as a product

There has been a large shift in the software world with the spread of concepts such as Domain-Driven Design, and the result ended up being domain-oriented services, owned by domain-focused teams, managing individual "products" of the larger product the company produces. This enabled ownership, independence, reliability, and more importantly the ability to deliver high-quality software quickly to become a competitive advantage. The data world is in for a similar transition.

The data teams are in for a similar transition: the days of having a team called "data teams" are over. Any sufficiently large software team has noticed the drawbacks of having isolated functional teams, and instead transitioning towards having cross-functional, agile teams, and the same principle applies to data teams. Instead of having a siloed team that takes in the data everyone else in the company produces and tries to make sense of it, the data team should be distributed among the business teams.

The mindset of the business teams started shifting towards treating data as an end-to-end product, and the associated rise in quality that comes with that. The organization treats data as part of its core product, and applies the appropriate measures to building, changing, governing, and protecting it.

This requires a rethinking of the structure of the data team:

• The team is transparent and spread across the whole organization.

• The data people within the organization work very closely with the business and product teams, no more siloes.

• The data team needs fewer engineers, more analysts & scientists who understand the business context.

In the end, you do not have a central data team, you have an organization that speaks data on all levels, across all teams.

This will take time, and it's fine

Shifting the whole data-as-an-afterthought mindset to making it a central beat of the company's heart is kind of like finally deciding to clean up that one junk drawer in your kitchen. You know it’s going to be a mess, and it’s way easier to just keep shoving stuff in there, but once you get it sorted, finding batteries or that one specific takeout menu becomes a breeze. It’s all about making data not just something you do, but a part of who you are as a company.

The real shift in terms of value generated from the data will come once the organizations internalize this change in the mindset.

1. Backed by Google: Firebase Analytics isn't your average analytics tool. It's built on the foundation of Google's internal solutions for app challenges, refined and available for everyone.

2. Seamless Google Analytics Integration: Firebase and Google Analytics are buddies. Whatever you track in Firebase, you can seamlessly analyze in Google Analytics. The best part? It's user-friendly, no need for a tech whiz; your product manager can handle it.

3. Comprehensive Toolbox: Firebase is more than just an analytics tool. It handles errors, secures user data, allows for remote config adjustments, and even offers A/B testing. It's a holistic solution for refining and perfecting your app's operation.

4. Industry Standard with Community Support: Many other apps, similar to yours, use it. That means a supportive community and ample resources to tap into when challenges arise.

5. Cost-Effective: In a landscape where similar tools come with a hefty price tag, Firebase stands out. Oh, and did I mention it's free? Google's offering it without a bill attached, making it an attractive choice for budget-conscious startups.

Now, let's dive a bit deeper. As a bonus, we'll explore how to export your Firebase Analytics data to BigQuery, taking your insights to the next level. Ready to get hands-on? Let's jump into the how-to.

Exporting Firebase Analytics Data to BigQuery: Recommended Settings

Moving forward, I'll walk you through the recommended settings for exporting your Firebase Analytics data to BigQuery. To avoid redundancy, we won't replicate the instructions available in the official documentation here, which covers the initial setup:

• Go to the Integrations page in the Firebase console.

• In the BigQuery card, click Link.

• Follow the on-screen instructions to enable BigQuery.

With those initial steps covered, let's now explore additional settings to optimize your data export process:

1. Region: Consider regulations like GDPR in Europe, CCPA in the US, or KVKK in Turkey when choosing a region to store your data.

2. Export Frequency: Firebase has its quirks with batch processing, like a daily limit of 1 million events. Instead, I recommend the more real-time Streaming option. Yes, it's a bit pricier, but the pros outweigh the cons, especially if you're aiming for almost real-time data.

3. Advertising Identifiers: Device tracking can be tricky, making it smart to have IDFA/IDFV and GPS-Adid for debug purposes. Enabling advertising IDs also empowers you for future data integration, particularly when dealing with various data sources such as marketing (MMPs like Adjust/Appsflyer, ad networks like Facebook/Google/TikTok), or revenue data.

With data in BigQuery, our journey progresses. In the forthcoming article, we'll shift our focus to understanding Firebase data in BigQuery. Get ready for a detailed exploration of how data looks in BigQuery and practical strategies for leveraging its power for a deeper understanding of your app's performance.