Hi, I'm Anthony.

Telemetry Demystified

August 4, 2015

Mozilla Telemetry is a measurement and analytics platform used at Mozilla, for products such as Firefox and Fennec.

This post is intended to be a high level overview of the systems we have in place. It is targeted toward people who haven’t worked with Telemetry systems much before, but want to use it to answer questions.

Glossary

Unified Telemetry/FHR, Pipeline v4, New pipeline

A rewrite of the v2 Telemetry pipeline to support changing requirements and additional functionality/flexibility. This project covers the same requirements as the v2 pipeline, but in a more clean, elegant, and flexible way. This is the officially supported pipeline for Telemetry-related tasks.

Unified telemetry also includes Firefox Health Report (FHR) pings, so it’s really a one-stop shop for all your data collection needs. A detailed description of this pipeline can be found here.

See also:

Telemetry v2, Pipeline v2, Old pipeline

The legacy backend for data collection and processing. This covers everything from the servers that collect data submitted by Mozilla products, to the Javascript libraries that expose a high level API to manipulate the aggregated data. Note that as of September 2015, this is no longer updated - projects should be using Unified Telemetry/FHR.

A more detailed description of this pipeline can be found here.

See also:

Telemetry-Wrapper.js

Telemetry-Wrapper.js uses Telemetry.js (v2) to generate nifty charts, much like the histograms/plots shown by the v4 pipeline dashboards. This is the library powering the Dashboard Generator on Mozilla Telemetry.

A more detailed overview of this library can be found here, with examples and usage information.

See also:

Telemetry.js (v2)

Telemetry.js v2 is a Javascript library for accessing data from the v4 pipeline. As of this writing, this is still under active development, but is quite stable already. This library powers the v4 pipeline dashboards on Mozilla Telemetry.

This library is the easiest way of accessing v4 pipeline data from Javascript, but the v4 pipeline also provides a more general HTTP API. The library is intended for use in the browser, but can also work with Node.js using telemetry-next-node, which takes care of downloading the latest version of the library and making sure things work properly in the Node.js environment.

This library is recommended over Telemetry.js v1 for new projects.

Telemetry.js v2 is not backwards compatible with Telemetry.js v1, since the v4 pipeline differs significantly in structure from the v2 pipeline. However, if you have a Telemetry.js v1 application and really don’t want to convert the code over, there is now a v1 shim available that allows you to use Telemetry.js v2 with the v1 API (with certain limitations).

See also:

Telemetry.js (v1)

Telemetry.js is a Javascript library for accessing data from the v2 pipeline. As of this writing, when people say “Telemetry.js”, they usually mean v1. This library powers the v2 pipeline dashboards on Mozilla Telemetry (at the moment, v2 pipeline dashboards are the default, while v4 pipeline dashboards are explicitly marked as such).

This library is the only officially supported way of accessing v2 pipeline data - the storage format on the backend is subject to change without warning. This can be used with Node.js with telemetry-js-node, which takes care of downloading the latest version of the library and making sure things work properly outside the browser.

See also:

How to answer questions with Telemetry

Dashboards

Mozilla Telemetry has a whole bunch of dashboards available for answering questions.

On that page, the left column has the Histogram and Evolution dashboards, which are used for everything from finding the median startup time to comparing error rates across OSs.

Using the Histogram and Evolution dashboards are hopefully pretty self-explanatory, but there is also a detailed tutorial for using these dashboards to answer questions. For example, to find the average startup time on the latest nightly, one would go to the histogram dashboard, select “SIMPLE_MEASURES_FIRSTPAINT distribution for nightly 42, on any OS for any architecture”, and read the result off either the histogram, or the summary statistics on the right.

The Dashboard Generator has an overview and tutorial here. This tool lets you generate plots much like the histogram/evolution dashboards, but customizable and embeddable for your own purposes.

There are also various other dashboards for more specialized purposes, listed on the right pane of the main landing page. For example, the Background Hang Reporting dashboard shows common sources of hangs, and Graphics Telemetry breaks down submissions by OS, product version, and more.

Not all of these dashboards are documented, but you can probably find a few tips and tricks from the #perf and #telemetry channels on the Mozilla IRC.

Custom Dashboards

If none of the existing dashboards can answer your questions, a custom dashboard might. Dashboards using Telemetry.js actually do significant amount of processing on the data before it’s shown to the user, and along the way some of the information gets filtered out.

The telemetry-wrapper.js library builds on top of Telemetry.js to generate histogram/evolution plots. This is useful for embedding customizable Telemetry data visualizations into other webpages, which can then be further tweaked to better display the relevant information.

For information about writing other kinds of dashboards, see the Telemetry.js v2 documentation.

For help with creating custom dashboards, make sure to check out the #perf and #telemetry channels on the Mozilla IRC.

Custom Analyses

To do custom analyses, you must have an @mozilla.com email address. Custom analyses are the most flexible option, but also require more familiarity with the underlying systems.

For each custom analysis, a new AWS instance is provisioned specifically for it containing our analysis environment, accessible via SSH. Custom analyses have access to individual pings, as well as the full power of Apache Spark and IPython. They also require the most resources to run.

A step-by-step guide to running custom analyses can be found here. More information about what’s going on under the hood can be found here.

Again, for help with performing custom analyses, the #perf and #telemetry channels on the Mozilla IRC are quite helpful.

Questions or comments? Drop me a line at azhang9@gmail.com.
Creative Commons License This work by Anthony Zhang is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. Copyright 2013-2017 Anthony Zhang.