It’s the third blog post of Modeling Google Analytics & Firebase Data. You can find the previous posts:

1. Exporting Firebase Data to BigQuery

2. Firebase Events Table

A healthy Data Warehouse (DWH) consists of a balanced combination of aggregated data models and raw event data. Aggregated models help in several ways:

• Lower compute costs by reducing the volume of data processed.

• Higher performance and faster queries due to the smaller dataset size.

• Simplified querying with embedded business metrics and facts.

• Consistent analysis by standardizing metrics and data cleaning.

• Easy feature calculation for ML pipelines

Probably the most common data model in clickstream events is daily user aggregates regardless of business/industry since most of the dashboards are built on daily granularity. I like naming tables by the primary keys since it makes it more semantic. So I’ll call this table `users_daily`.

This table will contain 1 row for each active user on a specific day. At the end, it will include all the user metrics and attributes to help us understand the business. It’s very easy to create dashboards on top of this dashboard, so you’ll love it.

In the previous article we created an `events.events` table, so `users_daily` will be built on top of this table. Let’s start by having the most simple metrics of Google Analytics Sample Gaming Project.

The starter template is:

SELECT
    user_pseudo_id,
    dt,
    --- attributes
    --- metrics
FROM events.events
WHERE -- data cleaning
GROUP BY 1,2

In this query, you can use user_pseudo_id instead of user_id, but the main idea is we should never add any other columns to the group by. This way, we’ll ensure the primary key (PK) is consistent.

Where

This table contains only user activity data. So I recommend excluding the data not relevant to user activity. For example, Firebase triggers app_remove, os_update, app_clear_data, app_update and app_exception events even the app is not open.

Also, if a row doesn’t have a user_id, it doesn’t make sense to aggregate them either. Let’s exclude them, too.

So now, our query looks like:

SELECT
    user_id,
    dt,
    --- attributes
    --- metrics
FROM events.events
WHERE 
    user_id is not null
    and event_name not in ("app_remove", "os_update", "app_clear_data", "app_update", "app_exception")
GROUP BY 1,2

Metrics

Metrics are the aggregates that shows what users did within the day. The simplest way to obtain the main metrics is to count the main events.

When I query the table, I see the events as metric candidates:

SELECT event_name, count(*) as cnt
FROM `bruin-blog-firebase-demo.events.events_json` 
WHERE event_name not in ("app_remove", "os_update", "app_clear_data", "app_update", "app_exception")
group by 1

SELECT
    user_id,
    dt,
    --- attributes
    --- metrics
    count(*) as event_cnt,

    -- app monetization metrics
    countif(event_name="ad_reward") as ad_reward_cnt,
    countif(event_name="in_app_purchase") as iap_cnt,

    -- app engagement metrics
    countif(event_name = "level_start") as level_start_cnt,
    countif(event_name = "level_end") as level_end_cnt,
    countif(event_name = "level_complete") as level_complete_cnt,
    countif(event_name = "level_fail") as level_fail_cnt,
    countif(event_name = "level_retry") as level_retry_cnt,
    countif(event_name = "level_start_quickplay") as level_start_quickplay_cnt,
    countif(event_name = "level_end_quickplay") as level_end_quickplay_cnt,
    countif(event_name = "level_complete_quickplay") as level_complete_quickplay_cnt,
    countif(event_name = "level_fail_quickplay") as level_fail_quickplay_cnt,
    countif(event_name = "level_retry_quickplay") as level_retry_quickplay_cnt,
    countif(event_name = "level_up") as level_up_cnt,
    countif(event_name = "challenge_a_friend") as challenge_a_friend_cnt,
    countif(event_name = "challenge_accepted") as challenge_accepted_cnt,
    countif(event_name = "no_more_extra_steps") as no_more_extra_steps_cnt,
    countif(event_name = "use_extra_steps") as use_extra_steps_cnt,
    countif(event_name = "select_content") as select_content_cnt,
    
FROM events.events
WHERE 
    user_id is not null
    and event_name not in ("app_remove", "os_update", "app_clear_data", "app_update", "app_exception")
GROUP BY 1,2

As you can see, I tried to add all the relevant events to the summary table. This will help us to monitor the user behavior in the dashboards very quickly.

In the second iteration of the metrics, we will use parameters. This process requires a good understanding the events and their parameters. For example:

level events have these parameters:

SELECT event_name, ep as event_params
FROM `bruin-blog-firebase-demo.events.events_json` 
WHERE event_name like "level%"

By using these events, we can obtain the min&max level played on that day. To do it, we can just add these to the query:

SELECT
    user_id,
    dt,

    min(level) as min_level,
    max(level) as max_level,

FROM events.events
WHERE ...
GROUP BY 1,2

Here, you should be very careful that the level parameter is not used in other levels in another context. If it’s used, you can filter your query for specific metrics:

    min(if(event_name like 'level_%', level, null)) as min_level,
    max(if(event_name like 'level_%', level, null)) as max_level,
    min(if(event_name = 'level_complete', level, null)) as min_level_completed,
    max(if(event_name = 'level_complete', level, null)) as max_level_completed,

Let’s inspect the in_app_purchase event to add the revenue to our table:

SELECT event_name, event_params, event_value_in_usd
FROM `bruin-blog-firebase-demo.events.events_json` 
WHERE event_name = "in_app_purchase"

As you can see, there are currency and price parameters. However, since they have different currencies, using it is impossible. To solve this problem, Google Analytics provides the `event_value_in_usd` field. Let’s add it to our metrics as well:

    sum(if(event_name="in_app_purchase", event_value_in_usd, 0)) as iap_revenue

After that point, I recommend learning the product very well, deciding on the metrics with product managers, and adding all of them to the users_daily table. The only exception is, for calculated metrics, I recommend you to do it on the BI layer.

For example, if you want to calculate win_rate, it’s better to add level_start_cnt and level_complete_cnt to this table. By using level_complete_cnt / level_start_cnt, you can easily calculate the win rate.

Attributes

Attributes are non-aggregatable information that provides additional details about the user, such as the user’s device brand, country, OS version, and platform. To summarize these attributes, I recommend using two different methods based on their types:

1. Dynamic Values

2. Static Values

Let’s start with the country attribute. A user can use your app in more than one country during a single day. For such dynamic values, it makes sense to capture the first and last values of the day. Country, os_version, app_version are examples of this group.

To get them, you can use min_by and max_by functions.

SELECT
    user_id,
    dt,

    MIN_BY(country, ts) as first_country,
    MAX_BY(country, ts) as last_country,

FROM events.events
WHERE ...
GROUP BY 1,2

Some of the values don’t change for a specific user id. For example, user_pseudo_id is assigned right after installing the app. If you are using user_pseudo_id as the primary key, we expect device platform, brand, device model values to be stationary. For that kind of attributes, we can use any_value since it doesn’t change for a user.

SELECT
    user_pseudo_id,
    dt,

    ANY_VALUE(platform) as platform,
    ANY_VALUE(device_brand) as device_brand,
    ANY_VALUE(device_model) as device_model,

FROM events.events
WHERE ...
GROUP BY 1,2

If you are controlling your user ids from a backend system, I don’t recommend using ANY_VALUE, but FIRST/LAST since all the attributes become dynamic.

Lastly, I want to mention user behaviour attributes. They show user behaviour like metrics, but still they’re mostly strings, so summing, counting doesn’t make sense on them.

One example is screen or event_name. If you want to measure where the users leave your app, you might want to calculate the first/last screen and events.

SELECT
    user_pseudo_id,
    dt,

    MIN_BY(event_name, ts) as first_event,
    MAX_BY(event_name, ts) as last_event,
    MIN_BY(screen, ts) as first_screen,
    MAX_BY(screen, ts) as last_screen,

FROM events.events
WHERE ...
GROUP BY 1,2

When we bring everything together, you’ll have a query like:

SELECT
    user_id,
    dt,

    MIN_BY(country, ts) as first_country,
    MAX_BY(country, ts) as last_country,
    MIN_BY(device_brand, ts) as first_device_brand,
    MAX_BY(device_brand, ts) as last_device_brand,
    MIN_BY(os_version, ts) as first_os_version,
    MAX_BY(os_version, ts) as last_os_version,
    MIN_BY(device_language, ts) as first_device_language,
    MAX_BY(device_language, ts) as last_device_language,

    count(*) as event_cnt,
    min(ts) as min_ts,
    max(ts) as max_ts,

    MIN_BY(event_name, ts) as first_event,
    MAX_BY(event_name, ts) as last_event,
    MIN_BY(screen, ts) as first_screen,
    MAX_BY(screen, ts) as last_screen,

    min(level) as min_level,
    max(level) as max_level,
    count(distinct timestamp_trunc(ts, minute)) as session_duration, 

    -- app monetization metrics
    countif(event_name="ad_reward") as ad_reward_cnt,
    countif(event_name="in_app_purchase") as iap_cnt,
    sum(if(event_name="in_app_purchase", event_value_in_usd, 0)) as iap_revenue,

    -- app engagement metrics
    countif(event_name = "level_start") as level_start_cnt,
    countif(event_name = "level_end") as level_end_cnt,
    countif(event_name = "level_complete") as level_complete_cnt,
    countif(event_name = "level_fail") as level_fail_cnt,
    countif(event_name = "level_retry") as level_retry_cnt,
    countif(event_name = "level_start_quickplay") as level_start_quickplay_cnt,
    countif(event_name = "level_end_quickplay") as level_end_quickplay_cnt,
    countif(event_name = "level_complete_quickplay") as level_complete_quickplay_cnt,
    countif(event_name = "level_fail_quickplay") as level_fail_quickplay_cnt,
    countif(event_name = "level_retry_quickplay") as level_retry_quickplay_cnt,
    countif(event_name = "level_up") as level_up_cnt,
    countif(event_name = "challenge_a_friend") as challenge_a_friend_cnt,
    countif(event_name = "challenge_accepted") as challenge_accepted_cnt,
    countif(event_name = "no_more_extra_steps") as no_more_extra_steps_cnt,
    countif(event_name = "use_extra_steps") as use_extra_steps_cnt,
    countif(event_name = "select_content") as select_content_cnt,
    
FROM events.events
WHERE 
    user_id is not null
    and event_name not in ("app_remove", "os_update", "app_clear_data", "app_update", "app_exception")
GROUP BY 1,2

Running on Production

Since the previous days’ data never changes, I recommend using an incremental flow for this table. You can find the full version of this pipeline here.

Also, you can use `bruin init firebase` in your terminal to bring all Firebase Analytics pipeline to your computer.

If you have questions about this article or running Bruin pipelines, don’t forget to join our Slack community.

If you need help to run your data pipelines, visit Bruin website. Bruin is an end-to-end data product platform that empowers your data analysts/scientists by 10x!

Even though these functions make our lives easier, we can make it even better. get_param and get_prop parameters require you to type a lot of characters, and also you have to remember the data type every time. Also, since event_params contains all the other parameters, querying it becomes very expensive.

To avoid this problem, I'll recommend a new data model to simplify querying.

Problem 1 - Too many characters

To be able to get a single parameter, let's say level, I have to type these all the time:

fn.get_param_int(event_params, 'level') as level

The problems here are:

1. I have to repeat fn.get_param_int for every parameter

2. I have to repeat event_params every time

3. I have to know the data type of the level parameter (in this example int)

Problem 2- Query costs

BigQuery charges per data scanned, and it costs a lot every time we access event_params.

To understand why, let's check an example event_params row:

Here, even though the client sends only 1 parameter, board, Firebase automatically adds 3 other parameters.

Let's calculate the data size BigQuery scans only to access the parameter “board”.

Since event_params is an array, we have to scan all the details in the array (there are very small optimizations behind, but for now I ignore them)

key column:

firebase_screen_class: 21 characters
firebase_event_origin: 21 characters
firebase_screen_id: 18 characters
board: 5 characters
total: 65 characters

values.string_value column

game_board: 10 characters
app+gtm: 7 characters
S: 1 character
total: 18 characters

So just to get a single character, `S`, BigQuery had to scan 65 + 18 = 83 characters. So this query could be 83 times cheaper!

To make this calculation, I already ignored the int and float value columns.

Also, this event has only 4 parameters, but most have many more parameters.

So imagine there are 15 parameters with an average of 15 characters. It would mean scanning 225 characters just for a single character! It’s a crazy amount of savings. (1/225 = Only 0.4% of the total scans)

Solution 1 - Unnested Events Table

So how can we solve this problem?

If we had a column named board, it would be great, right?

Imagine all the parameters are already a column in the events table, and BigQuery autocompletes your query, and also it costs much much less...

In this approach, we run an unnesting query periodically, so this query only runs once and then the rest of the queries are done on the new table.

An example unnested events query would be like:

CREATE TABLE events.events 
PARTITION BY dt CLUSTER BY event_name
AS
SELECT
	app_info.id AS app,
	platform,
	PARSE_DATE('%Y%m%d', event_date) AS dt,
	TIMESTAMP_MICROS(event_timestamp) AS ts,
	TIMESTAMP_MICROS(user_first_touch_timestamp) AS install_ts,
	user_pseudo_id,
	user_id,
	lower(event_name) as event_name,
	app_info.version AS app_version,
	device.advertising_id,
	device.vendor_id,
	geo.country AS geo_country,
	device.category AS device_type,
	device.mobile_brand_name AS device_brand,
	device.mobile_marketing_name AS device_marketing_name,
	device.mobile_model_name AS device_model,
	device.mobile_os_hardware_model AS device_hardware_model,
	device.language AS device_language,
	CASE lower(device.is_limited_ad_tracking)
		WHEN 'yes' THEN True
		WHEN 'no' THEN False
	END AS device_limited_ad_tracking,
	device.operating_system AS device_os,
	device.operating_system_version AS device_os_version,
	event_server_timestamp_offset / 1000 AS event_server_timestamp_offset,
	device.time_zone_offset_seconds / 3600 AS device_time_zone_offset_seconds,
	func.get_param_str(event_params, 'firebase_screen') AS screen,
	func.get_param_str(event_params, 'firebase_screen_class') AS screen_class,
	func.get_param_int(event_params, 'ga_session_id') AS session_id,
	func.get_param_int(event_params, 'ga_session_number') AS session_number,
	func.get_param_int(event_params, 'board') AS board,
FROM `firebase-public-project.analytics_153293282.events_*`

In this query, you can add all your event parameters and user properties, then schedule it to run once a day. You will have a very clean events table and it will be much cheaper to query.

However, this approach has a problem. As your events evolve and you send more and more parameters, it will be hard to maintain this table.

For every new parameter, you have to create a new column like this, first:

ALTER TABLE events
ADD COLUMN new_column INT64;

If you are late adding the new parameters to the table, you have to backfill it for the previous days.

Solution 2- Json Events Table

In previous years, BigQuery introduced a relatively new datatype: JSON.

JSON datatype gives flexibility in terms of new keys/parameters, and also it's stored as columnar in the storage layer. So, if you access a parameter in a JSON field, it would still be very cheap compared to the event_params column.

Here, I come up with another function:

CREATE OR REPLACE FUNCTION `fn.event_params_to_json`(
event_params ARRAY<STRUCT<
key STRING,
value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64>
>>
)
AS ((
with t1 as
(
	select
		array_agg(p.key) as keys,
		array_agg(coalesce(
			p.value.string_value,
			cast(p.value.int_value as string),
			cast(coalesce(p.value.double_value, p.value.float_value) as string)
		)) as vals
	from unnest(event_params) as p
	where key not in ("firebase_screen", "firebase_screen_class", "ga_session_id", "ga_session_number", "engaged_session_event", "engagement_time_msec", "firebase_event_origin", "firebase_previous_id", "firebase_screen_id")
)
select case when array_length(keys) > 0 then json_object(keys, vals) end
from t1
));

This function converts all the parameters to string and then creates a JSON object from them.

The reason I converted all of them to string is that BigQuery doesn't allow creating a JSON object with different data types.

At the end, you'll have a dictionary like:

{
"firebase_screen_class": "game_board",
"firebase_event_origin": "app+gtm",
"firebase_screen_id": "-8894705......",
"board": "S"
}

To create a JSON event_params column, you can use this query:

CREATE TABLE events.events_json
PARTITION BY dt CLUSTER BY event_name AS
SELECT
	app_info.id as app,
	platform,
	PARSE_DATE('%Y%m%d', event_date) as dt,
	TIMESTAMP_MICROS(event_timestamp) as ts,
	TIMESTAMP_MICROS(user_first_touch_timestamp) as user_first_touch_ts,
	user_pseudo_id,
	user_id,
	lower(event_name) as event_name,
	fn.parse_version(app_info.version) as app_version,
	fn.event_params_to_json(event_params) as ep,
FROM `firebase-public-project.analytics_153293282.events_*`

And then, you can use this query to access the board parameter:

SELECT lax_string(ep.board) as board
FROM events.events_json

With this query, we gained flexibility but lost query comfort. Now, we have to type more characters. To fix it, I recommend creating a view on top of this table, so it will be very similar to solution 1, but also flexible. Views don't occupy space in the storage but use the underlying tables to retrieve data.

To create the view, you can use this query:

CREATE OR REPLACE VIEW events.events AS
SELECT
	* except(up, ep),
	lax_string(ep.board) as board,
FROM events.events_json

So every time you query the table events.events, it actually goes through events.events_json table and retrieves the recent data. In the case there is a new parameter, the only thing you need to do is add it to events.events view.

Running this pipeline on production

Event tables are incremental tables and older days' data is not updated. If you're using intraday, it doesn't backfill older data, if you're using batch export, every day it updates the last 3 days of data. Based on my personal experience, added data to previous days tends to be ~1-2% of the total events, so you might choose to ignore it.

Since data is only added incrementally, you don't have to create the whole events table every day, and instead run incremental pipelines. While incremental strategies can go a long way, it is pretty simple for our article: you can just read yesterday’s data and append it to the current table. This way, your ETL costs will decrease significantly, and your ETL process will be completed much faster.

I only shared the CREATE queries above to keep it short, but in this repo, you can find how to make them incremental using Bruin.

Running Bruin

To run the pipeline steps in the right order, you need what’s called an “orchestrator”. Bruin is an open-source orchestrator that allows you to define your data pipelines as a collection of assets, and run them in your data warehouse.

You can install Bruin CLI by following the instructions here.

If you already have Bruin CLI, you can use the command below in the terminal to get a fresh start and it will automatically create a starter template for you. You can update this pipeline according to your needs and use it in the production.

brew tap bruin-data/tap
bruin init firebase

As a result of this command, you will have a fully-prepared production-ready data pipeline based on our articles, applying the best practices of managing Firebase data.

If you have questions about this article or running Bruin pipelines, join our Slack community.

In the next article, I'll explain how to create a daily summary table and we will create our first dashboard on Looker Studio. You can subscribe to this newsletter to get updated about the next article.

While a comprehensive explanation of each column is available here, this post will focus on making the table more accessible.

Accessing Firebase Demo Project

Let's kick things off with the Firebase demo project. If you already have a project, you don’t need to follow this step, you can just jump to the next topic below.

The demo dataset resides in the firebase-public-project, and you can add it to your workspace with a simple click on the + ADD button near above the data catalog.

Select the Star a project by name option, and type firebase-public project in the window that appears.

Upon clicking 'Star,' you'll find this dataset conveniently located in the left pane.

Understanding Events Table

Once you start exporting your data to BigQuery, you'll encounter a very wide table in BigQuery which is hard to understand and contains too much information.

Depending on your export settings, Firebase exports events in 2 names:

• events_

• events_intraday_

These tables are sharded tables, and generates a table for each day. For example, for 30 September 2023, the table name will be events_20230930.

Let's write our first query to see the event counts on a specific day, "2018-10-03".

SELECT
	event_name,
	count(*) as event_cnt,
	count(distinct user_pseudo_id) as user_cnt,
	count(*) / count(distinct user_pseudo_id) as event_per_user
FROM `firebase-public-project.analytics_153293282.events_20181003`
GROUP BY 1
ORDER BY 2 desc

Firebase triggers some events like screen_view, user_engagement automatically, so if you haven't triggered any of them, you don't need to get worried.

Working with Event Timestamp Column

If you preview the events table, the first thing you’ll see is, event_timestamp is an integer, and not easy to understand.

To convert it to a timestamp, you can use the TIMESTAMP_MICROS function.

Let's find hourly level starts in our game. To understand how timestamp_trunc function works, you can check the documentation.

SELECT 
    timestamp_trunc(timestamp_micros(event_timestamp), hour) as hour, 
    count(*) as events
FROM `firebase-public-project.analytics_153293282.events_20181003`
GROUP BY 1
ORDER BY 1

As you can see here, even though we queried the data for 2018-10-03, we got data from 7:00 AM to the next day until 7:00 AM. It happens due to the reporting time zone in Google Analytics settings. There always will be a difference between event_date and event_timestamp, but to minimize it you can choose UTC as time zone.

Querying Events, Event Params and User Properties

Let's focus on level_start_quickplay event and see some of the events and parameters.

There is a `board` parameter in event_params, and the value is `S`. It keeps the board size value for the played level.

Let's count level starts per board, but event_params is an array. How do we access it?

For both event_params and user_properties, we can use this code to access the parameters:

(select value.{PARAMETER_TYPE}_value from unnest(event_params) where key = "{PARAMETER_NAME}")

So to access the parameter, let's use it in the query and count it:

SELECT
	(select value.string_value from unnest(event_params) where key = "board") as board,
	count(*) as cnt
FROM `firebase-public-project.analytics_153293282.events_20181003`
WHERE event_name = "level_start_quickplay"
GROUP BY 1

Get Param Functions

As you want to include more and more parameters, writing this query will be very annoying.

For that purpose, I created some functions for you. First, I recommend you to create a dataset called fn and store all the functions in that dataset. Don't forget to choose the same region as the events tables region for this dataset.

CREATE OR REPLACE FUNCTION fn.get_param_str(event_params ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64>>>, param_key STRING) AS 
((
	SELECT
		COALESCE(
			value.string_value,
			CAST(value.int_value as string),
			CAST(value.float_value as string),
			CAST(value.double_value as string)
		)
	FROM UNNEST(event_params) WHERE key = param_key	
));

CREATE OR REPLACE FUNCTION fn.get_param_int(event_params ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64>>>, param_key STRING) AS 
((
	SELECT
		COALESCE(
			value.int_value,
			SAFE_CAST(value.float_value as int64),
			SAFE_CAST(value.double_value as int64),
			SAFE_CAST(value.string_value as int64)
		)
	FROM UNNEST(event_params) WHERE key = param_key
));

CREATE OR REPLACE FUNCTION fn.get_param_float(event_params ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64>>>, param_key STRING) AS 
((
	SELECT
		COALESCE(
			value.float_value,
			value.double_value,
			CAST(value.int_value as FLOAT64),
			SAFE_CAST(value.string_value as FLOAT64)
		)
	FROM UNNEST(event_params) WHERE key = param_key	
));

CREATE OR REPLACE FUNCTION `fn.get_param_bool`(event_params ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64>>>, param_key STRING) AS (
	SAFE_CAST(fn.get_param_int(event_params, param_key) as bool)
);

After creating these functions, you can rewrite your query as:

SELECT
	fn.get_param_str(event_params, "board") as board,
	count(*) as cnt
FROM `firebase-public-project.analytics_153293282.events_20181003`
WHERE event_name = "level_start_quickplay"
GROUP BY 1

Let’s query post_score event with parameters:

First, to find the parameters, let’s check the table without manipulation:

SELECT 
    user_pseudo_id,
    timestamp_micros(event_timestamp) as ts,
    event_params
FROM `firebase-public-project.analytics_153293282.events_20181003`
WHERE event_name = "post_score"

Here, we can see that there are 4 parameters relevant to the event.

• level_name (string)

• level (double)

• time (double)

• score (double)

If we use our functions to unnest the parameters, the query looks like that. Since BigQuery doesn’t have a double datatype, doubles are queried as floats:

SELECT 
    user_pseudo_id,
    timestamp_micros(event_timestamp) as ts,
    fn.get_param_str(event_params, "level_name") as level_name,
    fn.get_param_float(event_params, "level") as level,
    fn.get_param_float(event_params, "time") as time,
    fn.get_param_float(event_params, "score") as score,
FROM `firebase-public-project.analytics_153293282.events_20181003`
WHERE event_name = "post_score"

Even though parameters are sent as double/float, it looks like they indeed are integers. Let’s verify it by checking all the values:

SELECT 
    fn.get_param_float(event_params, "score") as score,
    count(*)    
FROM `firebase-public-project.analytics_153293282.events_20181003`
WHERE event_name = "post_score"
group by 1
order by 1

As suspected, all the rows are actually integers. If we do the same check for level and time, we see that they are also integers.

So let’s rewrite our query:

SELECT 
    user_pseudo_id,
    timestamp_micros(event_timestamp) as ts,
    fn.get_param_str(event_params, "level_name") as level_name,
    fn.get_param_int(event_params, "level") as level,
    fn.get_param_int(event_params, "time") as time,
    fn.get_param_int(event_params, "score") as score,
FROM `firebase-public-project.analytics_153293282.events_20181003`
WHERE event_name = "post_score"

We get the same table, but with correct datatypes this time.

get_param_* and get_prop_* functions are automatically casting the data to the right data types, so you don’t have to cast them again.

Here is the code for get_prop functions:

CREATE OR REPLACE FUNCTION `fn.get_prop_str`(user_properties ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64, set_timestamp_micros INT64>>>, param_key STRING) AS (
(
    SELECT 
        COALESCE(
            value.string_value,
            CAST(value.int_value as string),
            CAST(value.float_value as string),
            CAST(value.double_value as string)
        )
    FROM UNNEST(user_properties) WHERE key = param_key
)
);

CREATE OR REPLACE FUNCTION `fn.get_prop_int`(user_properties ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64, set_timestamp_micros INT64>>>, param_key STRING) AS (
(
    SELECT 
        COALESCE(
            value.int_value,
            SAFE_CAST(value.float_value as int64),
            SAFE_CAST(value.double_value as int64),
            SAFE_CAST(value.string_value as int64)
        )
    FROM UNNEST(user_properties) WHERE key = param_key
)
);

CREATE OR REPLACE FUNCTION `fn.get_prop_bool`(user_properties ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64, set_timestamp_micros INT64>>>, param_key STRING) AS 
(
    COALESCE(
        SAFE_CAST(fn.get_prop_int(user_properties, param_key) as bool),
        SAFE_CAST(fn.get_prop_str(user_properties, param_key) as bool)
    )
);

CREATE OR REPLACE FUNCTION `fn.get_prop_double`(user_properties ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64, set_timestamp_micros INT64>>>, param_key STRING) AS (
(
    SELECT 
        COALESCE(
            value.double_value,
            value.float_value,
            CAST(value.int_value as FLOAT64),
            SAFE_CAST(value.string_value as FLOAT64)
        )
    FROM UNNEST(user_properties) WHERE key = param_key
)
);

Querying multiple days

Until here, we only queried 1 day of data, specifically 2018-10-03.

What if we want to query multiple days?

BigQuery gives you a flexible way of querying sharded tables. You can just use `*` instead of the date in the table name, and you will be able to query all the dates:

SELECT
    event_date,
    count(*) as cnt,
    avg(fn.get_param_int(event_params, "score")) as avg_score,
    avg(fn.get_param_int(event_params, "level")) as avg_level
FROM `firebase-public-project.analytics_153293282.events_*`
WHERE event_name = "post_score"
GROUP BY 1
ORDER BY 1

This query returns 114 rows, for days between 12 June and 3 October.

However, you might need only the days between instead of full data. It will be both cheaper since we scan less data, and also might be more useful for our analysis. In that case, we’re using _TABLE_SUFFIX to filter the dates:

SELECT 
    PARSE_DATE('%Y%m%d', event_date) as dt,
    count(*) as cnt,
    avg(fn.get_param_int(event_params, "score")) as avg_score,
    avg(fn.get_param_int(event_params, "level")) as avg_level
FROM `firebase-public-project.analytics_153293282.events_*`
WHERE event_name = "post_score"
    AND _TABLE_SUFFIX between '20180701' and '20180731'
GROUP BY 1
ORDER BY 1

This query returns only July’s data which I need, also it costs much less compared to querying all data.

I made another change in that query as well, instead of using event_date, I used the PARSE_DATE function because event_date is stored as a string, and it’s both hard to read and also impossible to apply date operations. So by using parse_date, we converted it to date datatype.

Even though it’s much easier to query now, we still need to know of the parameters name, datatype, etc. In the next article we’ll transform this table to make it easier and cheaper to query.

1. Backed by Google: Firebase Analytics isn't your average analytics tool. It's built on the foundation of Google's internal solutions for app challenges, refined and available for everyone.

2. Seamless Google Analytics Integration: Firebase and Google Analytics are buddies. Whatever you track in Firebase, you can seamlessly analyze in Google Analytics. The best part? It's user-friendly, no need for a tech whiz; your product manager can handle it.

3. Comprehensive Toolbox: Firebase is more than just an analytics tool. It handles errors, secures user data, allows for remote config adjustments, and even offers A/B testing. It's a holistic solution for refining and perfecting your app's operation.

4. Industry Standard with Community Support: Many other apps, similar to yours, use it. That means a supportive community and ample resources to tap into when challenges arise.

5. Cost-Effective: In a landscape where similar tools come with a hefty price tag, Firebase stands out. Oh, and did I mention it's free? Google's offering it without a bill attached, making it an attractive choice for budget-conscious startups.

Now, let's dive a bit deeper. As a bonus, we'll explore how to export your Firebase Analytics data to BigQuery, taking your insights to the next level. Ready to get hands-on? Let's jump into the how-to.

Exporting Firebase Analytics Data to BigQuery: Recommended Settings

Moving forward, I'll walk you through the recommended settings for exporting your Firebase Analytics data to BigQuery. To avoid redundancy, we won't replicate the instructions available in the official documentation here, which covers the initial setup:

• Go to the Integrations page in the Firebase console.

• In the BigQuery card, click Link.

• Follow the on-screen instructions to enable BigQuery.

With those initial steps covered, let's now explore additional settings to optimize your data export process:

1. Region: Consider regulations like GDPR in Europe, CCPA in the US, or KVKK in Turkey when choosing a region to store your data.

2. Export Frequency: Firebase has its quirks with batch processing, like a daily limit of 1 million events. Instead, I recommend the more real-time Streaming option. Yes, it's a bit pricier, but the pros outweigh the cons, especially if you're aiming for almost real-time data.

3. Advertising Identifiers: Device tracking can be tricky, making it smart to have IDFA/IDFV and GPS-Adid for debug purposes. Enabling advertising IDs also empowers you for future data integration, particularly when dealing with various data sources such as marketing (MMPs like Adjust/Appsflyer, ad networks like Facebook/Google/TikTok), or revenue data.

With data in BigQuery, our journey progresses. In the forthcoming article, we'll shift our focus to understanding Firebase data in BigQuery. Get ready for a detailed exploration of how data looks in BigQuery and practical strategies for leveraging its power for a deeper understanding of your app's performance.