fnordig - posts tagged with 'mozilla'

Eight-year Moziversary

Mon, 02 Mar 2026 15:00:00 +0100

In March 2018 I started a job as a Telemetry engineer at Mozilla. Eight years later I'm still here on my Moziversary, as I was in 2019, 2020, 2021, 2022, 2023, 2024 and 2025.

In the past year we had 1 CEO change, 14 reorgs, 12 Firefox releases¹ and 31 Glean releases².

My team is still part of the Infrastructure Org, now under a bigger organization called Core Services. Other Data Engineers however have been moved over to the neighboring Data org. I was told nothing else changes and priorities stay the same. My ~~goals~~ individual expectations will be adjusted accordingly. I haven't seen my team in person since Dublin, due to policy changes around work travel in and out of the US.

Despite all the company changes, the work I'm directly involved in did remain largely the same. I'm still focused on Glean, as expected. We (and by that I mean mostly chutten) moved legacy telemetry to Glean fully (we are Glean now). In the last half of 2025 I finally found dedicated time for performance work. We now have the first benchmarks on Glean and I was able to close out some performance gaps previously identified. More work remains. Early this year then I was able to revive my prototype for moving Glean to a new storage backend based on SQLite and early benchmarks are very promising. This year I will focus on finalizing that work, deploying a new Glean version with a more reliable (and faster!) client-side storage, accompanied by benchmarks we can run and rely on. Along with that I can do some refactoring and cleanups that will make our codebase easier to work with. Other work includes a better integration with other Rust components that want to use Glean directly.

The rest of Mozilla is still working on many big and small things. Leadership is still chasing the AI hype, with questionable outcomes³. I don't know where that will take the company and I disagree with some of the approaches. So far I've been spared from the worst and no one is forcing AI tools on me just yet.

Thank you

Eight years in the same job is a long time. I get to make decisions. I have to live with them. I have to live with decisions of others and maintain them. None of that however would be possible without a team, which makes it a joy to work alongside. So a big thank you to my team members Chris, Travis, Charlie and Abhishek for being the Data Collection Tools team. And also a thank you to Alessio, our team manager. There's many others across Mozilla I get to work with and chat with. Thank you!

Footnotes:

Contrary to popular belief there is not a fixed reorg schedule (that I know of), there is a Firefox release schedule though. ↩
Now with a release song playlist ↩
You can disable the things you don't want in Firefox. ↩

Incident Report: A compiler bug and JSON

Tue, 09 Dec 2025 16:01:00 +0100

This article is cross-posted to the Data@Mozilla blog.

It all started rather inconspicuous: The Data Engineering team filed a bug report about a sudden increase in schema errors at ingestion of telemetry data from Firefox for Android. At that point in time about 0.9% of all incoming pings were not passing our schema validation checks.

The data we were seeing was surprising. Our ingestion endpoint received valid JSON that contained snippets like this:

{
    "metrics": {
        "schema: counter": {
            "glean.validation.pings_submitted": {
                "events": 1
            }
        },
        ...
    },
    ...
}

What we would expect and would pass our schema validation is this:

{
    "metrics": {
        "labeled_counter": {
            "glean.validation.pings_submitted": {
                "events": 1
            }
        },
        ...
    },
    ...
}

The difference? 8 characters:

-        "schema: counter": {
+        "labeled_counter": {

8 different characters that still make up valid JSON, but break validation.

A week later the number of errors kept increasing, affecting up to 2% of all ingested pings from Firefox for Android Beta. That's worryingly high. That's enough to drop other work and call an incident.

Aside: Telemetry ingestion

In Firefox the data is collected using the Glean SDK. Data is stored in a local database and eventually assembled into what we call a ping: A bundle of related metrics, gathered in a JSON payload to be transmitted. This JSON document is then POSTed to the Telemetry edge server. From there the decoder eventually picks it up and processes it further. One of the early things it does is verify the received data against one of the pre-defined schemas. When data is coming from the Glean SDK it must pass the pre-defined glean.1.schema.json. This essentially describes which fields to expect in the nested JSON object. One thing it is expecting is a labeled_counter A thing it is NOT expecting is schema: counter. In fact keys other than the listed ones are forbidden.

The missing schema:_

The data we were receiving from a growing number of clients contained 8 bytes that we didn't expect in that place: schema: . That 8-character string didn't even show up in the Glean SDK source code. Where does it come from? Why was it showing up now?

We did receive entirely valid JSON, so it's unlikely to be simple memory corruption¹. More like memory confusion, if that's a thing.

We know where the payload is constructed. The nested object for labeled metrics is constructed in its own function. It starts with string formatting:

let ping_section = format!("labeled_{}", metric.ping_section());

There's our 8-character string labeled_ that gets swapped. The Glean SDK is embedded into Firefox inside mozilla-central and compiled with all the other code together. A single candidate for the schema: string exists in that codebase. That's another clue it could be memory confusion.

My schema? Confused.

I don't know much about how string formatting in Rust works under the hood, but luckily Mara blogged about it 2 years ago: Behind the Scenes of Rust String Formatting: format_args!() (and then recently improved the implementation²).

So the format! from above expands into something like this:

std::io::_format(
    // Simplified expansion of format_args!():
    std::fmt::Arguments {
        template: &[Str("labeled_ "), Arg(0)],
        arguments: &[&metric.ping_section() as &dyn Display],
    }
);

Another clue that the labeled_ string is referenced all by itself and swapping out the pointer to it would be enough to lead to the corrupted data we were seeing.

Architecturing more clues

Whenever we're faced with data anomalies we start by dissecting the data to figure out if the anomalies are from a particular subset of clients. The hope is that identifying the subset of clients where it happens gives us more clues about the bug itself.

After initially focusing too much on actual devices colleagues helpfully pointed out that the actual split was the device's architecture³:

ARMv8, the 64-bit architecture, did not run into this issue⁴. ARMv7, purely 32-bit, was the sole driver of this data anomaly. Another clue that something in the code specifically for this architecture was causing this.

Logically unchanged

With a hypothesis what was happening, but no definite answer why, we went to speculative engineering: Let's avoid the code path that we think is problematic.

By explicitly listing out the different strings we want to have in the JSON payload we avoid the formatting and thus hopefully any memory confusion.

let ping_section = match metric.ping_section() {
    "boolean" => "labeled_boolean".to_string(),
    "counter" => "labeled_counter".to_string(),
    // <snip>
    _ => format!("labeled_{}", metric.ping_section()),
};

This was implemented in 912fc80 and shipped in Glean v66.1.2. It landed in Firefox the same day of the SDK release and made it to Firefox for Android Beta the Friday after. The data shows: It's working, no more memory confusion!

A bug gone but still there

The immediate incident-causing data anomaly was mitigated, the bug is not making it to the Firefox 146 release.

But we still didn't know why this was happening in the first place. My colleagues Yannis and Serge kept working and searching and were finally able to track down what exactly is happening in the code. The bug contains more information on the investigation.

While I was trying to read and understand the disassembly of the broken builds, they went ahead and wrote a tiny emulator (based on the Unicorn engine) that runs just enough of the code to find the offending code path⁵.

> python ./emulator.py libxul.so
Path: libxul.so
GNU build id: 1b9e9c8f439b649244c7b3acf649d1f33200f441
Symbol server ID: 8F9C9E1B9B43926444C7B3ACF649D1F30
Please wait, downloading symbols from: https://symbols.mozilla.org/try/libxul.so/8F9C9E1B9B43926444C7B3ACF649D1F30/libxul.so.sym
Please wait, uncompressing symbols...
Please wait, processing symbols...
Proceeding to emulation.
Result of emulation: bytearray(b'schema: ')
This is a BAD build.

The relevant section of the code boils down to this:

ldr   r3, [pc, #0x20c]
add   r3, pc
strd  r3, r0, [sp, #0xd0]
add   r1, sp, #0xd0
bl    alloc::fmt::format_inner

The first two instructions build the pointer to the slice in r3, by using a pc-relative offset found in a nearby constant. Then we store that pointer at sp+0xd0, and we put the address sp+0xd0 into r1. So before we reach alloc::fmt::format_inner, r1 points to a stack location that contains a pointer to the slice of interest. The slice lives in .data.rel.ro and contains a pointer to the string, and the length of the string (8). The string itself lives in .rodata.

In good builds the .rodata r3 points to looks like this:

0x06f0c3d4: 0x005dac18  -->  "labeled_"
0x06f0c3d8:        0x8
0x06f0c3dc: 0x0185d707  -->  "/builds/<snip>/rust/glean-core/src/storage/mod.rs"
0x06f0c3e0:       0x4d

In bad builds however it points to something that has our dreaded schema: string:

0x06d651c8: 0x010aa2e8  -->  "schema: "
0x06d651cc:        0x8
0x06d651d0: 0x01a869a7  -->  "maintenance: "
0x06d651d4:        0xd
0x06d651d8: 0x01a869b4  -->  "storage dir: "
0x06d651dc:        0xd
0x06d651e0: 0x01a869c8  -->  "from variant of type "
0x06d651e4:       0x15
0x06d651e8: 0x017f793c  -->  ": "
0x06d651ec:        0x2

This confirms the suspicion that it's a compiler/linker bug. Now the question was how to fix that.

Firefox builds with a variety of Clang/LLVM versions. Mozilla uses its own build of LLVM and Clang to build the final applications, the exact version used is updated as soon as possible, but never on release. Sometimes additional patches are applied on top of the Clang release, like some backports fixing other compiler bugs.

After identifying that this is indeed a bug in the linker and that it has already been patched in later LLVM versions, Serge did all the work to bisect the LLVM release to find which patches to apply to Mozilla's own Clang build. Ultimately he tracked it down to these two patches for LLVM:

With those patches applied, the old code, without our small code rearrangement, does not lead to broken builds anymore.

With the Glean code patched, the ingestion errors dropping and the certainty that we have identified and patched the compiler bug, we can safely ship the next release of Firefox (for Android).

Collaboration

Incidents are stressful situations, but a great place for collaboration across the whole company. The number of people involved in resolving this is long.

Thanks to Eduardo & Ben from Data Engineering for raising the issue.
Thanks to Alessio (my manager) for managing the incident.
Thanks to chutten and Travis (from my team) for brainstorming what caused this and suggesting solutions/workarounds.
Thanks to Donal (Release Management) for fast-tracking the mitigation into a Beta release.
Thanks to Alex (Release Engineering) for some initial investigation into the linker bug.
Thanks to Brad (Data Science) for handling the data analysis side.
Thanks to Yannis and Serge (OS integration) for identifying, finding and patching the linker bug.

Footnotes:

Memory corruption is never "simple". But if it were memory corruption I would expect data to be broken worse or in other places too. Not just a string swap in a single place. ↩
That improvement is not yet available to us. The application experiencing the issue was compiled using Rust 1.86.0. ↩
Our checklist initially omitted architecture. A mistake we since fixed. ↩
Apparently we do see some errors, but they are so infrequent that we can ignore them for now. ↩
Later Yannis wrote a script that can identify broken builds purely much quicker, just by searching for the right string patterns. ↩

Glean Memory Usage Reporting

Wed, 28 May 2025 14:00:00 +0200

(This article is cross-posted on the Data@Mozilla blog.)

Since Bug 1896609 landed we now have Glean & Firefox on Glean (FOG) memory reporting built into the Firefox Memory Reporter. This allows us to measure the allocated memory in use by Glean and FOG. It currently covers memory allocated by the C++ module of FOG and all instantiated Glean metrics. It does not yet measure the memory used by Glean and its database.

How it works

Firefox has a built-in memory usage reporter, available as about:memory. Components of Firefox can expose their own memory usage by implementing the nsIMemoryReporter interface. FOG implements this interface and delegates the measurement to the firefox-on-glean Rust component.

firefox-on-glean then collects the memory usage of objects under its own control: all user-defined and runtime-instantiated metrics, additional hashmaps used to track metrics & all user-defined and runtime-instantiated pings. It will soon also collect the memory size of the global Glean object, and thus the memory used for built-in metrics as well as the in-memory database.

Memory measurement works by following all heap-allocated pointers, asking the allocator for the memory size of each and summing everything up. Because we do most of this measurement in Rust we use the existing wr_malloc_size_of crate, which already implements the correct measurement for most Rust libstd types as well as some additional library-provided types. Our own types implement the required trait using malloc_size_of_derive for automatically deriving the trait, or manual implementations.

How it looks

The memory measurement is built into Firefox and works in every shipped build. Open up about:memory in a running Firefox, click the “Measure” button and wait for the measurement. Once all data is collected it will show a long tree of measured allocations across all processes. Type fog into the filter box on the right to trim it down to only allocations from the fog component. The exact numbers differ between runs and operating systems.

You will see a view similar to this:

about:memory on a freshly launched developer build of Firefox. fog reports 0.35 MB of allocated memory in the main process.

After opening a few tabs and browsing the web a new measurement on about:memory will show a different number, as Glean is instantiating more metrics and therefore allocating more memory. This number will grow as more metrics are instantiated and kept in memory.

This currently does not show the allocations from the global Glean object and its in-memory database. In the future we will be able to measure those allocations as well. In a prototype locally this already works as expected: As more data is recorded and stored the allocated memory grows. Once a ping is assembled, submitted and sent the allocations will be freed and about:memory will report less memory allocated again.

Seven-year Moziversary

Mon, 03 Mar 2025 10:50:00 +0100

In March 2018 I started a job as a Telemetry engineer at Mozilla. Seven years later I'm still here on my Moziversary, as I was in 2019, 2020, 2021, 2022, 2023 and 2024.

Mozilla is not the same company it was when I joined. No one on the C-level is here for as long as I am. There have been 5 larger reorgs in the past year and two layoffs in different departments, plus the big round of layoffs at the Mozilla Foundation. My part of the organization was moved around again and for the moment Data Engineering is nested under the Infrastructure Org. Good colleagues left or were laid off. Just recently my team shrunk again. Notably I haven't posted anything under the mozilla tag since 2022, other than my Moziversary blog posts.

~~All-Hands~~ MozWeek¹ Dublin happened. The next one will be in Washington, D.C. Probably one of the worst choices given the state of the USA right now, and I might just skip it. I do hope for a team work week, but that's not decided yet.

The one constant over the past years is my work on Glean, which hasn't stopped. We're finally putting dedicated effort to move legacy telemetry in Firefox Desktop over to Glean (Are we Glean yet?). But still Glean isn't where I'd like it to be. Some of the early decisions in its design are coming back to bite us, the codebase grew significantly and could use a bit of cleanup, we never got the time to work on performance and memory improvements and our chosen local data storage desperately needs an update. Some of that was on my list last year already. If only there were less distractions that pull me of this work again and again. It's on my list of goals again this year, so maybe it works out better this time.

But other than that I don't know where Mozilla will be a year from now. It's going to be a challenging year.

Thank you

I'm in this job for seven years not only because I like the tech I get to work on, but also because my colleagues make it a delight to come back to work every day. Thanks to Alessio, Chris, Travis and Abhishek for being the Data Collection Tools team. Also thanks to Bruno, who was part of the team until recently. There's countless other people at Mozilla, inside Data Engineering but also across other parts of the organization, that I got to connect, chat and work with. Thank you!

Footnotes:

Change everywhere. Those Mozilla work weeks were renamed by Marketing. ↩

Six-year Moziversary

Tue, 12 Mar 2024 11:30:00 +0100

Another year went by, so that it's now been 6 years since I joined Mozilla as a Telemetry engineer, I blogged every year since then: 2019, 2020, 2021, 2022, 2023

Looking back at the past year it sure was different than the years before, again. Obviously we left most of the pandemic isolation behind us and I got to meet more of my coworkers in person: At the Mozilla All-Hands in Montreal, Canada, though that was cut short for me due to ... of course: Covid. At PyCon DE and PyData here in Berlin. And at another workweek with my extended team also here in Berlin.

My work also changed. As predicted a year ago I branched out beyond the Glean SDK, took a look at our data pipeline, worked on features across the stack and wrote a ton of stuff that is not code. Most of that work spanned months and months and some is still not 100% finished.

For this year I'm focusing a bit more on the SDK and client-side world again. With Glean used just about everywhere it's time we look into some optimizations. In the past we made it correct first and paid less attention to optimize resource usage (CPU & memory for example). Now that we have more and more usage in Firefox Desktop (Use Counters!) we need to look into making data collection more efficient. The first step is to get better insights where and how much memory we use. Then we can optimize. Firefox comes with some of its own tooling for that with which we need to integrate, see about:memory for example.

I'm also noticing some parts in our codebase where in hindsight I wish we had made different implementation decisions. At the time though we did make the right choices, now we need to deal with that (memory usage might be one of these, storage sure is another one).

And Mozilla more broadly? It's changing. All the time. We just had layoffs and reprioritization of projects. That certainly dampens the mood. Focus shifts and work changes. But underneath there's still the need to use data to drive our decisions and so I'm rather confident that there's work for me to do.

Thank you

None of my work would happen if it weren't for my manager Alessio and team mates Chris, Travis, Perry, Bruno and Abhishek. They make it fun to work here, always have some interesting things to share and they still endure my bad jokes all the time. Thank you!
Thanks also goes out to the bigger data engineering team within Mozilla, and all the other people at Mozilla I work or chat with.

This post was planned to be published more than a week ago. It's still perfectly in time. I wasn't able to focus on it earlier.

Five-year Moziversary

Wed, 01 Mar 2023 08:00:00 +0200

I can't believe it's already my fifth Moziversary. It's been 5 years now since I joined Mozilla as a Telemetry engineer, I blogged every year since then: 2019, 2020, 2021, 2022. As I'm writing this I'm actually off on vacation (and will be for another week or so) and also it's super early here. Nonetheless it's time to look back and forward.

So what have I been up to in the past year? My team changed again. We onboarded Perry and Bruno and when Mike left we got Alessio as the manager of us all. In September we finally met again at the Mozilla All Hands in Hawaii. Not everyone was there, but it was great to meet those that were. I also went to the Berlin office more often. It's still good to have that other place to work from.

I didn't add any new posts to the "This Week in Glean" series and it was effectively retired. I still believe that openly communicating about our work is very useful and would like to see more of that again. Maybe I'll find some topics to write about this year (and I have some drafts lying around I should finish). One major piece of work in the past year was migrating Glean to use UniFFI. That shipped and I'm proud we rolled it out. Beyond that I spent large parts of my time supporting our users (other Mozilla applications, mostly mobile), fixing bugs and slowly tackling some feature improvements.

And what is for the next year? I'm in the process of handing over my Glean SDK tech lead role to Travis. After over 2 years I feel it's the right time to give up some responsibility and decision power over the project. I believe that sharing responsibilities and empowering others to fill tech lead positions is an overwhelmingly good and important thing.

This shift will free me up to expand my work into other places. I'm staying with the same team and of course a major part of my work will be on the SDK regardless, but I also hope to expand my knowledge about our data systems end-to-end and have a high-level view and opinion about it. For the most part Glean feels "complete", but of course there's always feature requests, use cases we want to support better and improvements to make. My list of little things I would like to improve keeps growing, but now also gets new items beyond just the SDK.

To the next year and beyond!

Thank you

Thanks to my team mates Alessio, Chris, Travis, Perry and Bruno and also thanks to the bigger data engineering team within Mozilla. And thanks to all the other people at Mozilla I work with.

Four-year Moziversary

Fri, 04 Mar 2022 15:20:00 +0100

It's my fourth Moziversary. It's been 4 years (and three days) now since I joined Mozilla as a Telemetry engineer. I joined Mozilla as a Firefox Telemetry Engineer in March 2018, I blogged three times already: 2019, 2020, 2021.

The past year continued to be challenging. Except for a brief 3-week period the Berlin office stayed close, so we all continue to work from home. I haven't met (most of) my team mates in person since 2020. I hope that in 2022 I will have the chance to meet some of them again, maybe even all at once.

I already spent some time on looking back on most of the work that happened on the Glean project last year in a This Week in Glean post, so no need to reiterate that.

For 2022 Glean will be about stabilizing, some new features and more widespread adoption across our products. I'm still excited to continue that work. We will see what else I pick up along the way.

Thank you

Thanks to my team mates Alessio, Bea, Chris, Travis, and Mike, and also thanks to the bigger data engineering team within Mozilla. And thanks to all the other people at Mozilla I work with.

This Week in Glean: Your personal Glean data pipeline

Fri, 25 Feb 2022 15:00:00 +0100

(“This Week in Glean” is a series of blog posts that the Glean Team at Mozilla is using to try to communicate better about our work. They could be release notes, documentation, hopes, dreams, or whatever: so long as it is inspired by Glean.) All "This Week in Glean" blog posts are listed in the TWiG index (and on the Mozilla Data blog). This article is cross-posted on the Mozilla Data blog.

On February 11th, 2022 we hosted a Data Club Lightning Talk session. There I presented my small side project of setting up a minimal data pipeline & data storage for Glean.

The premise:

Can I build and run a small pipeline & data server to collect telemetry data from my own usage of tools?

To which I can now answer: Yes, it's possible. The complete ingestion server is a couple hundred lines of Rust code. It's able to receive pings conforming to the Glean ping schema, transform them and store it into an SQLite database. It's very robust, not crashing once on me (except when I created an infinite loop within it).

You can watch the lightning talk here:

Instead of creating some slides for the talk I created an interactive report. The full report can be read online.

Besides actually writing a small pipeline server this was also an experiment in trying out Irydium and Datasette to produce an interactive & live-updated data report.

Irydium is a set of tooling designed to allow people to create interactive documents using web technologies, started by wlach a while back. Datasette is an open source multi-tool for exploring and publishing data, created and maintained by simonw. Combining both makes for a nice experience, even though there's still some things that could be simplified.

My pipeline server is currently not open source. I might publish it as an example at a later point.

This Week in Glean: Building and Deploying a Rust library on iOS

Mon, 31 Jan 2022 12:40:00 +0100

We ship the Glean SDK for multiple platforms, one of them being iOS applications. Previously I talked about how we got it to build on the Apple ARM machines. Today we will take a closer look at how to bundle a Rust static library into an iOS application.

The Glean SDK project was set up in 2019 and we have evolved its project configuration over time. A lot has changed in Xcode since then, so for this article we're starting with a fresh Xcode project, a fresh Rust library and put it all together step by step.
This is essentially an update to the Building and Deploying a Rust library on iOS article from 2017.

For future readers: This was done using Xcode 13.2.1 and rustc 1.58.1.
One note: I learned iOS development to the extent required to ship Glean iOS. I've never written a full iOS application and lack a lot of experience with Xcode.

The application

The premise of our application is easy:

Show a non-interactive message to the user with data from a Rust library.

Let's get started on that.

The project

We start with a fresh iOS project. Go to File -> New -> Project, then choose the iOS App template, give it a name such as ShippingRust, select where to store it and finally create it. You're greeted with ContentView.swift and the following code:

import SwiftUI

struct ContentView: View {
    var body: some View {
        Text("Hello, world!")
            .padding()
    }
}

You can build and run it now. This will open the Simulator and display "Hello, world!". We'll get back to the Swift application later.

The Rust parts

First we set up the Rust library.

In a terminal navigate to your ShippingRust project directory. In there create a new Rust crate:

cargo new --lib shipping-rust-ffi

We will need a static library, so we change the crate type in the generated shipping-rust-ffi/Cargo.toml. Add the following below the package configuration:

[lib]
crate-type = ["staticlib"]

Let's also turn the project into a Cargo workspace. Create a new top-level Cargo.toml with the content:

[workspace]
members = [
  "shipping-rust-ffi"
]

cargo build in the project directory should work now and create a new static library.

$ ls -l target/debug/libshipping_rust_ffi.a
-rw-r--r-- 2 jer staff 16061952 Jan 28 13:09 target/debug/libshipping_rust_ffi.a

Let's add some code to shipping-rust-ffi/src/lib.rs next. Nothing fancy, a simple function taking some arguments and returning the sum:

use std::os::raw::c_int;

#[no_mangle]
pub extern "C" fn shipping_rust_addition(a: c_int, b: c_int) -> c_int {
    a + b
}

The no_mangle ensures the name lands in the compiled library as-is and the extern "C" makes sure it uses the right ABI.

We now have a Rust library exporting a C-ABI compatible interface. We can now consume this in our iOS application.

The Xcode parts

Before we can use the code we need a bit more setup. Strap in, there's a lot of fiddly manual steps now¹.

We start by linking against the libshipping_rust_ffi.a library. In your Xcode project open your target configuration², go to "Build Phases", then look for "Link Binary with Libraries". Add a new one, in the popup select "Add files" on the bottom left and look for the target/debug/libshipping_rust_ffi.a file. Yes, that's actually for the wrong target. This is just for the name, we'll fix up the path next. Go to "Build Settings" and search for "Library Search Paths". It probably has the path to file in there right now for both Debug and Release builds. Remove that one for Debug, then add a new row by clicking the small + symbol. Select the Any Driverkit matcher. It doesn't matter which matcher you choose or what value you give it, but when we overwrite this manually in the next step I'll assume you chose Any Driverkit. Do the same for the Release configuration.

Once that's done, save your project and go back to your project directory. We will modify the project configuration to have Xcode look for the library based on the target it is building for³. Open up ShippingRust.xcodeproj/project.pbxproj in a text editor, then search for the first line with "LIBRARY_SEARCH_PATHS[sdk=driverkit*]". It should be in a section saying /* Debug */. Remove the LIBRARY_SEARCH_PATHS line and add 3 new ones:

"LIBRARY_SEARCH_PATHS[sdk=iphoneos*][arch=arm64]" = "$(PROJECT_DIR)/target/aarch64-apple-ios/debug";
"LIBRARY_SEARCH_PATHS[sdk=iphonesimulator*][arch=arm64]" = "$(PROJECT_DIR)/target/aarch64-apple-ios-sim/debug";
"LIBRARY_SEARCH_PATHS[sdk=iphonesimulator*][arch=x86_64]" = "$(PROJECT_DIR)/target/x86_64-apple-ios/debug";

Look for the next line with "LIBRARY_SEARCH_PATHS[sdk=driverkit*]", now in a /* Release */ section and replace it with:

"LIBRARY_SEARCH_PATHS[sdk=iphoneos*][arch=arm64]" = "$(PROJECT_DIR)/target/aarch64-apple-ios/release";
"LIBRARY_SEARCH_PATHS[sdk=iphonesimulator*][arch=arm64]" = "$(PROJECT_DIR)/target/aarch64-apple-ios-sim/release";
"LIBRARY_SEARCH_PATHS[sdk=iphonesimulator*][arch=x86_64]" = "$(PROJECT_DIR)/target/x86_64-apple-ios/release";

Save the file and return focus back to Xcode. If you didn't make any typos Xcode should still have your project open. In the settings you will find the library search paths as we've just defined them. If you messed something up Xcode will complain that it cannot read the project file if you try to go to the settings.

Next we need to teach Xcode how to compile Rust code. Once again go to your target settings, selecting the "Build Phases" tab again.

There add a new "Run Script" phase, give it the name "Build Rust library" (double-click the "Run Script" section header), and set the command to:

bash ${PROJECT_DIR}/bin/compile-library.sh shipping-rust-ffi $buildvariant

The compile-library.sh script is going to do the heavy lifting. The first argument is the crate name we want to compile, the second is the build variant to select. This is not yet defined, so let's do it first.

Go to the "Build Settings" tab and click the + button to add a new "User-Defined Setting". Give it the name buildvariant and choose a value based on the build variant: debug for Debug and release for Release.

Now we need the actual script to build the Rust library for the right targets. It's a bit long to write out, but the logic is not too complex: First we select the Cargo profile to use based on our buildvariant (that is whether to pass --release or not), then we set up LIBRARY_PATH if necessary and finally compile the Rust library for the selected target. Xcode passes the architectures to build in ARCHS. It's either x86_64 for simulator builds on Intel Mac hardware or arm64. If it's arm64 it can be either the simulator or an actual hardware target. Those differ, but we can know which is which from what's in LLVM_TARGET_TRIPLE_SUFFIX and select the right Rust target.

Let's put all of that into a compile-library.sh script. Create a new directory bin in your project directory. In there create the file with the following content:

#!/usr/bin/env bash

if [ "$#" -ne 2 ]
then
    echo "Usage (note: only call inside xcode!):"
    echo "compile-library.sh <FFI_TARGET> <buildvariant>"
    exit 1
fi

# what to pass to cargo build -p, e.g. your_lib_ffi
FFI_TARGET=$1
# buildvariant from our xcconfigs
BUILDVARIANT=$2

RELFLAG=
if [[ "$BUILDVARIANT" != "debug" ]]; then
  RELFLAG=--release
fi

set -euvx

if [[ -n "${DEVELOPER_SDK_DIR:-}" ]]; then
  # Assume we're in Xcode, which means we're probably cross-compiling.
  # In this case, we need to add an extra library search path for build scripts and proc-macros,
  # which run on the host instead of the target.
  # (macOS Big Sur does not have linkable libraries in /usr/lib/.)
  export LIBRARY_PATH="${DEVELOPER_SDK_DIR}/MacOSX.sdk/usr/lib:${LIBRARY_PATH:-}"
fi

IS_SIMULATOR=0
if [ "${LLVM_TARGET_TRIPLE_SUFFIX-}" = "-simulator" ]; then
  IS_SIMULATOR=1
fi

for arch in $ARCHS; do
  case "$arch" in
    x86_64)
      if [ $IS_SIMULATOR -eq 0 ]; then
        echo "Building for x86_64, but not a simulator build. What's going on?" >&2
        exit 2
      fi

      # Intel iOS simulator
      export CFLAGS_x86_64_apple_ios="-target x86_64-apple-ios"
      $HOME/.cargo/bin/cargo build -p $FFI_TARGET --lib $RELFLAG --target x86_64-apple-ios
      ;;

    arm64)
      if [ $IS_SIMULATOR -eq 0 ]; then
        # Hardware iOS targets
        $HOME/.cargo/bin/cargo build -p $FFI_TARGET --lib $RELFLAG --target aarch64-apple-ios
      else
        $HOME/.cargo/bin/cargo build -p $FFI_TARGET --lib $RELFLAG --target aarch64-apple-ios-sim
      fi
  esac
done

And now we're done with the setup for compiling the Rust library automatically as part of the Xcode project build.

The code parts

We now have an Xcode project that builds our Rust library and links against it. We now need to use this library!

Swift speaks Objective-C, which is an extension to C, but we need to tell it about the things available. In C land that's done with a header. Let's create a new file, select the "Header File" template and name it FfiBridge.h. This will create a new file with this content:

#ifndef FfiBridge_h
#define FfiBridge_h


#endif /* FfiBridge_h */

Here we need to add the definition of our function. As a reminder this is its definition in Rust:

extern "C" fn shipping_rust_addition(a: c_int, b: c_int) -> c_int;

This translates to the following in C:

int shipping_rust_addition(int a, int b);

Add that line between the #define and #endif lines. Xcode doesn't know about that file yet, so once more into the Build Settings of the target. Search for Objective-C Bridging Header and set it to $(PROJECT_DIR)/ShippingRust/FfiBridge.h. In Build Phases add a new Header Phase. There you add the FfiBridge.h as well.

If it now all compiles we're finally ready to use our Rust library.

Open up ContentView.swift and change the code to call your Rust library:

struct ContentView: View {
    var body: some View {
        Text("Hello, world! \(shipping_rust_addition(30, 1))")
            .padding()
    }
}

We simply interpolate the result of shipping_rust_addition(30, 1) into the string displayed.

Once we compile and run it in the simulator we see we've succeeded at satisfying our premise:

Show a non-interactive message to the user with data from a Rust library.

Compiling for any iOS device should work just as well.

The next steps

This was a lot of setup for calling one simple function. Luckily this is a one-time setup. From here on you can extend your Rust library, define them in the header file and call them from Swift. If you go that route you should really start using cbindgen to generate that header file automatically for you.

This time we looked at building an iOS application directly calling a Rust library. That's not actually how Glean works. The Glean Swift SDK itself wraps the Rust library and exposes a Swift library. In a next blog post I'll showcase how we ship that as a Swift package.

For Glean we're stepping away from manually writing our FFI functions. We're instead migrating our code base to use UniFFI. UniFFI will generate the C API from an API definitions file and also comes with a bit of runtime code to handle conversion between Rust, C and Swift data types for us. We're not there yet for Glean, but you can try it on your own. Read the UniFFI documentation and integrate it into your project. It should be possible to extent the setup we done her to also run the necessary steps for UniFFI. Eventually I'll document how we did it as well.

Footnotes:

And most of these steps are user-interface-dependent and might be different in future Xcode version. :( ↩
Click your project name in the tree view on the left. This gets you to the project configuration (backed by the ShippingRust.xcodeproj/project.pbxproj file). You should then see the Targets, including your ShippingRust target and probably ShippingRustTests as well. We need the former. ↩
Previously we would have built a universal library containing the library of multiple targets. That doesn't work anymore now that arm64 can stand for both the simulator and hardware targets. Thus linking to the individual libraries is the way to go, as the now-deprecated cargo-lipo also points out. ↩

This Week in Glean: Looking back at Glean in 2021

Fri, 17 Dec 2021 14:00:00 +0100

A year ago I posted Glean in 2021 as a way to look into the future and set out a vision and plan for the project. Today I'm looking back at 2021, if we were able to follow up on my plans back then and look at all the other things we did for Glean.

Let's start easy: According to the index we wrote 21 This Week in Glean blog posts (including this one). Close enough to one every other week. Communicating about our work is important and TWiGs are one way to put our ideas and thoughts about the project out there.

Let's first look at the topics I identified as important in last year's blog post.

The vision

The Glean SDK is a fully self-servable telemetry SDK, usable across different platforms. It enables product owners and engineers to instrument their products and rely on their data collection, while following Mozilla policies & privacy standards.

I'd say we're in pretty good shape here. The docs on how to add Glean to a project need some updates, but we also added a post for a slightly different audience: Integrating Glean for product managers, so it's not only for engineers anymore. We assisted others when they integrated Glean, but more for general data and instrumentation knowledge.

The ideas

Metric types

We specified & implemented 3 new metric types this year. The rate metric has been implemented in the Rust and JavaScript SDKs and is also available on Firefox Desktop. The url metric is available in all SDKs, but not yet exposed on Firefox Desktop. The text metric is currently JavaScript-only.

However the url and text metric were not usable until recently because of bugs in our schema generation tool. We had to change how we generate the table schema for these and other metric types in the future a bit, but the pipeline now correctly ingests data for these metrics and users can query the data in the columns where they expect them.

Revamped testing APIs

We implemented some of the new testing APIs, namely metric coverage support and Ping.testBeforeNextSubmit. Glean and FOG already use these new testing capabilities, but to my knowledge other Glean users are not yet relying on that. Certainly an area we should make more progress on in the coming year.

UniFFI - generate all the code

Migration to rely on UniFFI for code generation has started! Right now adding new features to Glean requires changes across all SDKs. With UniFFI we can define the API of metric types once, implement it in Rust and get the code for the SDKs generated automatically. Less manual code means less potential for bugs and easier maintenance.

The uniffi branch is used for slowly migrating Glean piece by piece. Most of the metric types are defined in UDL now and the Kotlin code base has been migrated to it already, with all tests passing again. Work on this will continue next year, migrating the Swift, Python and Rust SDKs and then eventually merging it back into the main branch, so we can ship it to users. From there adding new metrics will require less implementation work and thus will allow us to follow up with some of the existing but currently paused requests.

All the other things

Glean

We released 9 major versions of Glean and did 33 releases in total. We're very liberal with breaking changes & major releases, though we make sure that upgrades require minimal changes where possible. The Kotlin, Swift, Python & Rust SDKs are versioned all the same, so a breaking change in only one SDK will still cause a major release for other SDKs, even without breaking changes.

More products and projects are now using Glean, including Rally studies, the Mozilla VPN client, and the new version of Focus for Android. A full list of products is available in the Glean Dictionary.

To ease integration between telemetry collected in Gecko and telemetry collected in the Android parts of the browser we're now shipping Glean through GeckoView. (After crashing it, fixing it and shipping it again without new bugs cropping up.) There's some work to be done next year to make local development easier again and to automate some of the tasks of upgrading Glean across applications.

Firefox on Glean (FOG)

We invested a lot of time in making the Firefox integration better, fixing bugs and convincing people to migrate their instrumentation to the new Glean-based system. It's going slow, but we now have the first few important metrics in Glean pings coming from Firefox Desktop. Additionally Glean was used to provide telemetry for the new Desktop Background Updater. Next year we hope to convince more people to start using Glean in Desktop, with the added benefit that this data will also be available in Firefox for Android.

Glean.js

A year ago Glean.js was just started. Thanks to Bea's tireless work it's now nearly feature-complete compared to the other Glean SDKs and used in production in applications & web extensions and soon even used to instrument websites. Next year Bea aims for a proper stable v1.0 release and hopefully more use across different products. It's a real alternative for those platforms where the initial set of SDKs don't cut it.

Glean surely is becoming the one telemetry solution across Mozilla products.

Tooling: GLAM, Dictionary, Looker

These are the tools I'm much less directly involved in, but they make up an important part of the Glean ecosystem. After all these are where users actually get to analyze at the data they collect.

GLAM currently only serves data for Firefox Desktop using legacy telemetry data as well as Firefox for Android using Glean data. Things are in place to enable more Glean-powered products soon, once we iron out some details around data expectations from applications.

The Glean Dictionary saw near-weekly releases and went from "promising prototype" into "valuable production tool" this year, thanks to a lot of work from wlach and linh. It was the basis of several initiatives such as metric annotations and tagging support. These things will be helpful to make data at Mozilla more discoverable and better documented.

The primary tool for analyzing data at Mozilla is now Looker. Folks have been very busy making more and more data accessible to both data experts and non-experts alike¹. Glean data is (more or less) automatically available in Looker explores, for example event counts. Instead of bringing up ad hoc dashboards for initiatives when needed we can usually recommend that people just build things in Looker. That has the major advantage that exploration beyond what the dashboard summarizes is just one or two clicks away. I'm excited to shift my own use away from fiddling around with SQL to more approachable explores & dashboards, even for one-time analysis.

2022

I'm saving an outlook on the next year of Glean for some other time. I called out some areas of work above that will see more work next year, and then there's also new projects & ideas coming up. I still plan to continue work on Glean for the vast majority of my time.

Thanks

The This Week in Glean series is going into winter hiatus until January 2022. Thanks to everyone who contributed to This Week in Glean blog posts in 2021:

Alessio, Travis, Chutten, Bea, Raphael, Anthony, Will.

And of course thanks to everyone who contributed to Glean and the wider ecosystem around it.

Footnotes:

We have some pretty cool documentation to get you started with Looker! ↩

This Week in Glean: Crashes & a buggy Glean

Mon, 01 Nov 2021 15:00:00 +0100

In September I finally landed work to ship Glean through GeckoView. Contrary to what that post said Fenix did not actually use Glean coming from GeckoView immediately due to another bug that took us another few days to land. Shortly after that was shipped in a Fenix Nightly release we received a crash report (bug 1733757) pointing to code that we haven't touched in a long time. And yet the change of switching from a standalone Glean library to shipping Glean in GeckoView uncovered a crashing bug, that quickly rose to be the top crasher for Fenix for more than a week.

When I picked up that bug after the weekend I was still thinking that this would be just a bug, which we can identify & fix and then get into the next Fenix release. But in data land nothing is ever just a bug.

As I don't own an Android device myself I was initially restricted to the Android emulator, but I was unsuccessful in triggering that bug in my simple use of Fenix or in any tests I wrote. At some point I went as far as leveraging physical devices in the Firebase test lab, still with no success of hitting the bug. Later that week I picked up an Android test device, hoping to find easy steps to reproduce the crash. Thankfully we also have quality assurance people running tests and simply using the browser and reporting back steps to reproduce bugs and crashes. And so that's what one of them, Delia, did: Providing simple steps to reproduce the crash. Simple here meant: open a website and leave the phone untouched for roughly 8-9 minutes. With those steps I was able to reproduce the crash in both the emulator and my test device as well.

As I was waiting for the crash to occur I started reading the code again. From the crash report I already knew the exact place where the code panics and thus crashes the application, I just didn't know how it got there. While reading the code around the panic and the knowledge that it takes several minutes to get to the panic I finally stumbled upon a clue: The map implementation that was used internally has a maximum capacity of elements it can store. It even documents that it will panic¹!

/// The maximum capacity of a [`HandleMap`]. Attempting to instantiate one with
/// a larger capacity will cause a panic.
///
/// [...]
pub const MAX_CAPACITY: usize = (1 << 15) - 1;

I now knew that at some point Glean exhausts the map capacity, slowly but eventually leading to a panic². Most of Glean is set up to not require dynamically allocating new things other than the data itself, but we never store the data itself within such a map. So where are we using a map and dynamically add new entries to it and potentially forget to remove them after use?

Luckily the stack trace from those crash reports gave me another clue: All crashes were caused by labeled counters. Some instrument-the-code-paths-and-recompile cycles later I was able to pinpoint it to the exact metric that was causing the crash.

With all this information collected I was able to determine that while the panic was happening in a library we use, the problem was that Glean had the wrong assumptions about how the use of labeled counters in Kotlin maps to objects in a Rust map. On every subscript access to a labeled counter (myCounter["label"]) Glean would create a new object, store it in the map and return a handle to this object. The solution was to avoid creating new entries in that map on every access from a foreign language and instead cache created objects by a unique identifier and hand out already-created objects when re-accessed. This fix was implemented in a few lines, accompanied by an extensive commit message as well as tests in all of the 3 major foreign-language Glean SDKs.

That fix was released in Glean v42.0.1, and shortly after in Fenix Nightly and Beta.

The bug was fixed and the crash prevented, but that wasn't the end of it. The code leading to the bug has been in Glean since at least March 2020 and yet we only started seeing crashes with the GeckoView change. What was happening before? Did we miss these crashes for more than a year or were there simply no crashes?

To get a satisfying answer to these questions I had to retrace the issue in older Glean versions. I took older versions of Glean & Fenix from just before the GeckoView-Glean changes, added logging and built myself a Fenix to run in the emulator. And as quickly as that³ logcat spit out the following lines repeatedly:

E  ffi_support::error: Caught a panic calling rust code: "called `Result::unwrap()` on an `Err` value: \"PoisonError { inner: .. }\""
E  glean_ffi::handlemap_ext: Glean failed (ErrorCode(-1)): called `Result::unwrap()` on an `Err` value: "PoisonError { inner: .. }"

No crashes! It panics, but it doesn't crash in old Fenix versions. So why does it crash in new versions? It's a combination of things.

First Glean data recording happens in a thread to avoid blocking the main thread. Right now that thread is handled on the Kotlin side. A panic in the Rust code base will bubble up the stack and crash the thread, thus poisoning the internal lock of the map, but it will not abort the whole application by itself. Subsequent calls will run on a new thread, handled by the Kotlin side. When using a labeled counter again it will try to acquire the lock, detect that it is poisoned and panic once more. That panic is caught by the FFI layer of Glean and turned into a log message.

Second the standalone Glean library is compiled with panic=unwind, the default in Rust, which unwinds the stack on panics. If not caught the runtime will abort the current thread, writing a panic message to the error output. ffi-support however catches it, logs it and returns without crashing or aborting. Gecko on the other hand sets panic=abort. In this mode a panic will immediately terminate the current process (after writing the crash message to the error output), without ever trying to unwind the stack, giving no chance for the support library to catch it. The Gecko crash reporter is able to catch those hard aborts and send them as crash reports. As Glean is now part of the overall Gecko build all of Gecko's build flags will transitively apply to Glean and its dependencies, too. So when Glean is shipped as part of GeckoView it runs in panic=abort mode, leading to internal panics aborting the whole application.

That behavior by itself is fine: Glean should only panic in exceptional cases and we'd like to know about them. It's good to know that an application could continue running without Glean working correctly; we won't be able to record and send telemetry data, but at least we're not interrupting someone using the application. However unless we engineers run into those bugs and see the log we will not notice them and thus can't fix them. So ultimately this change in (crashing) behavior is acceptable (and wanted) going forward.

After fixing the initial bug and being able to answer why it only started crashing recently my work was still not done. We were likely not recording data in exceptional cases for quite some time, which is not acceptable for a telemetry library. I had to explore our data, estimate how many metrics for how many clients were affected, inform relevant stakeholders and plan further mitigations. But that part of the investigation is a story for another time.

This bug investigation had help from a lot of people. Thanks to Mike Droettboom for coordination, Marissa Gorlick for pushing me to evaluate the impact on data & reaching the right people, Travis Long for help with the Glean code & speedy reviews, Alessio Placitelli for reviews on the Gecko side, Delia Pop for the initial steps to reproduce the bug & Christian Sadilek for help on the Android Components & Fenix side.

Footnotes:

Except that it panics at a slightly different place than the explicit check in the code base would suggest. ↩
That meant I could also tune how quickly it crashed. A smaller maximum capacity means its reached more quickly, reducing my bug reproduction time significantly. ↩ ↩2
No, not after 9 minutes, but in just under 3 minutes after tuning the maximum map capacity, see ². ↩

Fenix Physical Device Testing

Thu, 14 Oct 2021 17:00:00 +0200

The Firefox for Android (Fenix) project runs extensive tests on every pull request and when merging code back into the main branch.

While many tests run within an isolated Java environment, Fenix also contains a multitude of UI tests. They allow testing the full application, interaction with the UI and other events. Running these requires the Android emulator running or a physical Android device connected. To run these tests in the CI environment the Fenix team relies on the Firebase test lab, a cloud-based testing service offering access to a range of physical and virtual devices to run Android applications on.

To speed up development, the automatically scheduled tests associated with a pull request are only run on virtual devices. These are quick to spin up, there is basically no upper limit of devices that can spawn on the cloud infrastructure and they usually produce the same result as running the test on a physical device.

But once in a while you encounter a bug that can only be reproduced reliably on a physical device. If you don't have access to such a device, what do you do? Or you know the bug happens on that one specific device type you don’t have?

You remember that the Firebase Test Lab offers physical devices as well and the Fenix repository is very well set up to run your test on these too if needed!

Here's how you change the CI configuration to do this.

NOTE: Do not land a Pull Request that switches CI from virtual to physical devices! Add the pr:do-not-land label and call out that the PR is only there for testing!

By default the Fenix CI runs tests using virtual devices on x86. That's faster when the host is also a x86(_64) system, but most physical devices use the Arm platform. So first we need to instruct it to run tests on Arm.

Which platform to test on is defined in taskcluster/ci/ui-test/kind.yml. Find the line where it downloads the target.apk produced in a previous step and change it from x86 to arm64-v8a:

  run:
      commands:
-         - [wget, {artifact-reference: '<signing/public/build/x86/target.apk>'}, '-O', app.apk]
+         - [wget, {artifact-reference: '<signing/public/build/arm64-v8a/target.apk>'}, '-O', app.apk]

Then look for the line where it invokes the ui-test.sh and tell it to use arm64-v8a again:

  run:
      commands:
-         - [automation/taskcluster/androidTest/ui-test.sh, x86, app.apk, android-test.apk, '-1']
+         - [automation/taskcluster/androidTest/ui-test.sh, arm64-v8a, app.apk, android-test.apk, '-1']

With the old CI configuration it will look for Firebase parameters in automation/taskcluster/androidTest/flank-x86.yml. Now that we switched the architecture it will pick up automation/taskcluster/androidTest/flank-arm64-v8a.yml instead.

In that file we can now pick the device we want to run on:

   device:
-   - model: Pixel2
+   - model: dreamlte
      version: 28

You can get a list of available devices by running gcloud locally:

gcloud firebase test android models list

The value from the MODEL_ID column is what you use for the model parameter in flank-arm64-v8a.yml. dreamlte translates to a Samsung Galaxy S8, which is available on Android API version 28.

If you only want to run a subset of tests define the test-targets:

test-targets:
 - class org.mozilla.fenix.glean.BaselinePingTest

Specify an exact test class as above to run tests from just that class.

And that's all the configuration necessary. Save your changes, commit them, then push up your code and create a pull request. Once the decision task on your PR finishes you will find a ui-test-x86-debug job (yes, x86, we didn't rename the job). Its log file will have details on the test run and contain links to the test run summary. Follow them to get more details, including the logcat output and a video of the test run.

This explanation will eventually move into documentation for Mozilla's Android projects.
Thanks to Richard Pappalardo & Aaron Train for the help figuring out how to run tests on physical devices and early feedback on the post. Thanks to Will Lachance for feedback and corrections. Any further errors are mine alone.

This Week in Glean: Glean & GeckoView

Fri, 17 Sep 2021 13:00:00 +0200

This is a followup post to Shipping Glean with GeckoView.

It landed!

It took us several more weeks to put everything into place, but we're finally shipping the Rust parts of the Glean Android SDK with GeckoView and consume that in Android Components and Fenix. And it still all works, collects data and is sending pings! Additionally this results in a slightly smaller APK as well.

This unblocks further work now. Currently Gecko simply stubs out all calls to Glean when compiled for Android, but we will enable recording Glean metrics within Gecko and exposing them in pings sent from Fenix. We will also start work on moving other Rust components into mozilla-central in order for them to use the Rust API of Glean directly. Changing how we deliver the Rust code also made testing Glean changes across these different components a bit more challenging, so I want to invest some time to make that easier again.

This Week in Glean: Shipping Glean with GeckoView

Mon, 26 Jul 2021 12:00:00 +0200

Glean SDK

The Glean SDK is Mozilla's telemetry library, used in most mobile products and now for Firefox Desktop as well. By now it has grown to a sizable code base with a lot of functionality beyond just storing some metric data. Since its first release as a Rust crate in 2019 we managed to move more and more logic from the language SDKs (previously also known as "language bindings") into the core Rust crate. This allows us to maintain business logic only once and can easily share that across different implementations and platforms. The Rust core is shipped precompiled for multiple target platforms, with the language SDK distributed through the respective package manager.

I talked about how this all works in more detail last year, this year and blogged about it in a previous TWiG.

GeckoView

GeckoView is Mozilla's alternative implementation for WebViews on Android, based on Gecko, the web engine that also powers Firefox Desktop. It is used as the engine behind Firefox for Android (also called Fenix). The visible parts of what makes up Firefox for Android is written in Kotlin, but it all delegates to the underlying Gecko engine, written in a combination of C++, Rust & JavaScript.

The GeckoView code resides in the mozilla-central repository, next to all the other Gecko code. From there releases are pushed to Mozilla's own Maven repository.

One Glean too many

Initially Firefox for Android was the only user of the Glean SDK. Up until today it consumes Glean through its release as part of Android Components, a collection of libraries to build browser-like applications.

But the Glean SDK is also available outside of Android Components, as its own package. And additionally it's available for other languages and platforms too, including a Rust crate. Over the past year we've been busy getting Gecko to use Glean through the Rust crate to build its own telemetry on top.

With the Glean SDK used in all these applications we're in a difficult position: There's a Glean in Firefox for Android that's reporting data. Firefox for Android is using Gecko to render the web. And Gecko is starting to use Glean to report data.

That's one Glean too many if we want coherent data from the full application.

Shipping it all together, take one

Of course we knew about this scenario for a long time. It's been one of the goals of Project FOG to transparently collect data from Gecko and the embedding application!

We set out to find a solution so that we can connect both sides and have only one Glean be responsible for the data collection & sending.

We started with more detailed planning all the way back in August of last year and agreed on a design in October. Due to changed priorities & availability of people we didn't get into the implementation phase until earlier this year.

By February I had a first rough prototype in place. When Gecko was shipped as part of GeckoView it would automatically look up the Glean library that is shipped as a dynamic library with the Android application. All function calls to record data from within Gecko would thus ultimately land in the Glean instance that is controlled by Fenix. Glean and the abstraction layer within Gecko would do the heavy work, but users of the Glean API would notice no difference, except their data would now show up in pings sent from Fenix.

This integration was brittle. It required finding the right dynamic library, looking up symbols at runtime as well as reimplementing all metric types to switch to the FFI API in a GeckoView build. We abandoned this approach and started looking for a better one.

Shipping it all together, take two

After the first failed approach the issue was acknowledged by other teams, including the GeckoView and Android teams.

Glean is not the only Rust project shipped for mobile, the application-services team is also shipping components written in Rust. They bundle all components into a single library, dubbed the megazord. This reduces its size (dependencies & the Rust standard library are only linked once) and simplifies shipping, because there's only one library to ship. We always talked about pulling in Glean as well into such a megazord, but ultimately didn't do it (except for iOS builds).

With that in mind we decided it's now the time to design a solution, so that eventually we can bundle multiple Rust components in a single build. We came up with the following plan:

The Glean Kotlin SDK will be split into 2 packages: a glean-native package, that only exists to ship the compiled Rust library, and a glean package, that contains the Kotlin code and has a dependency on glean-native.
The GeckoView-provided libxul library (that's "Gecko") will bundle the Glean Rust library and export the C-compatible FFI symbols, that are used by the Glean Kotlin SDK to call into Glean core.
The GeckoView Kotlin package will then use Gradle capabilities to replace the glean-native package with itself (this is actually handle by the Glean Gradle plugin).

Consumers such as Fenix will depend on both GeckoView and Glean. At build time the Glean Gradle plugin will detect this and will ensure the glean-native package, and thus the Glean library, is not part of the build. Instead it assumes libxul from GeckoView will take that role.

This has some advantages. First off everything is compiled together into one big library. Rust code gets linked together and even Rust consumers within Gecko can directly use the Glean Rust API. Next up we can ensure that the version of the Glean core library matches the Glean Kotlin package used by the final application. It is important that the code matches, otherwise calling native functions could lead to memory or safety issues.

Glean is running ahead here, paving the way for more components to be shipped the same way. Eventually the experimentation SDK called Nimbus and other application-services components will start using the Rust API of Glean. This will require compiling Glean alongside them and that's the exact case that is handled in mozilla-central for GeckoView then.

Now the unfortunate truth is: these changes have not landed yet. It's been implemented for both the Glean SDK and mozilla-central, but also requires changes for the build system of mozilla-central. Initially that looked like simple changes to adopt the new bundling, but it turned into bigger changes across the board. Some of the infrastructure used to build and test Android code from mozilla-central was untouched for years and thus is very outdated and not easy to change. With everything else going on for Firefox it's been a slow process to update the infrastructure, prepare the remaining changes and finally getting this landed.

But we're close now!

Big thanks to Agi for connecting the right people, driving the initial design and helping me with the GeckoView changes. He also took on the challenge of changing the build system. And also thanks to chutten for his reviews and input. He's driving the FOG work forward and thus really really needs us to ship GeckoView support.

This Week in Glean: rustc, iOS and an M1

Fri, 16 Apr 2021 15:00:00 +0100

Back in February I got an M1 MacBook. That's Apple's new ARM-based hardware.

I got it with the explicit task to ensure that we are able to develop and build Glean on it. We maintain a Swift language binding, targeting iOS, and that one is used in Firefox iOS. Eventually these iOS developers will also have M1-based machines and want to test their code, thus Glean needs to work.

Here's what we need to get to work:

Compile the Rust portions of Glean natively on an M1 machine
Build & test the Kotlin & Swift language bindings on an M1 machine, even if non-native (e.g. Rosetta 2 emulation for x86_64)
Build & test the Swift language bindings natively and in the iPhone simulator on an M1 machine
Stretch goal: Get iOS projects using Glean running as well

Rust on an M1

Work on getting Rust compiled on M1 hardware started last year in June already, with the availability of the first developer kits. See Rust issue 73908 for all the work and details. First and foremost this required a new target: aarch64-apple-darwin. This landed in August and was promoted to Tier 2¹ with the December release of Rust 1.49.0.

By the time I got my MacBook compiling Rust code on it was as easy as on an Intel MacBook. Developers on Intel MacBooks can cross-compile just as easily:

rustup target add aarch64-apple-darwin
cargo build --target aarch64-apple-darwin

Glean Python & Kotlin on an M1

Glean Python just ... worked. We use cffi to load the native library into Python. It gained aarch64² macOS support in v14.4.1. My colleague glandium later contributed support code so we build release wheels for that target too. So it's both possible to develop & test Glean Python, as well as use it as a dependency without having a full Rust development environment around.

Glean Android is not that straight forward. Some of our transitive dependencies are based on years-old pre-built binaries of SQLite and of course there's not much support behind updating those Java libraries. It's possible. A friend managed to compile and run that library on an M1. But for Glean development we simply recommend relying on Rosetta 2 (the x86_64 compatibility layer) for now. It's as easy as:

arch -x86_64 $SHELL
make build-kotlin

At least if you have Java set up correctly... The default Android emulator isn't usable on M1 hardware yet, but Google is working on a compatible one: Android M1 emulator preview. It's usable enough for some testing, but for that part I most often switch back to my Linux Desktop (that has the additional CPU power on top).

Glean iOS on an M1

Now we're getting to the interesting part: Native iOS development on an M1. Obviously for Apple this is a priority: Their new machines should become the main machine people do iOS development on. Thus Xcode gained aarch64 support in version 12 long before the hardware was available. That caused quite some issues with existing tooling, such as the dependency manager Carthage. Here's the issue:

When compiling for iOS hardware you would pick a target named aarch64-apple-ios, because ... iPhones and iPads are ARM-based since forever.
When compiling for the iOS simulator you would pick a target named x86_64-apple-ios, because conveniently the simulator uses the host's CPU (that's what makes it fast)

So when the compiler saw x86_64 and iOS it knew "Ah, simulator target" and when it saw aarch64 and ios it knew "Ah, hardware". And everyone went with this, Xcode happily built both targets and, if asked to, was able to bundle them into one package.

With the introduction of Apple Silicion³ the iOS simulator run on these machines would also be aarch64⁴, and also contain ios, but not be for the iOS hardware.

Now Xcode and the compiler will get confused what to put where when building on M1 hardware for both iOS hardware and the host architecture.

So the compiler toolchain gained knowledge of a new thing: arm64-apple-ios14.0-simulator, explicitly marking the simulator target. The compiler knows from where to pick the libraries and other SDK files when using that target. You still can't put code compiled for arm64-apple-ios and arm64-apple-ios14.0-simulator into the same universal binary⁵, because you can have each architecture only once (the arm64 part in there). That's what Carthage and others stumbled over.

Again Apple prepared for that and for a long time they have wanted you to use XCFramework bundles⁶. Carthage just didn't used to support that. The 0.37.0 release fixed that.

That still leaves Rust behind, as it doesn't know the new -simulator target. But as always the Rust community is ahead of the game and deg4uss3r started adding a new target in Rust PR #81966. He got half way there when I jumped in to push it over the finish line. How these targets work and how LLVM picks the right things to put into the compiled artifacts is severly underdocumented, so I had to go the trial-and-error route in combination with looking at LLVM source code to find the missing pieces. Turns out: the 14.0 in arm64-apple-ios14.0-simulator is actually important.

With the last missing piece in place, the new Rust target landed in February and is available in Nightly. Contrary to the main aarch64-apple-darwin or aarch64-apple-ios target, the simulator target is not Tier 2 yet and thus no prebuilt support is available. rustup target add aarch64-apple-darwin does not work right now. I am now in discussions to promote it to Tier 2, but it's currently blocked by the RFC: Target Tier Policy.

It works on nightly however and in combination with another cargo capability I'm able to build libraries for the M1 iOS simulator:

cargo +nightly build -Z build-std --target aarch64-apple-ios-sim

For now Glean iOS development on an M1 is possible, but requires Nightly. Goal achieved, I can actually work with this!

In a future blog post I want to explain in more detail how to teach Xcode about all the different targets it should build native code for.

All The Other Projects

This was marked a stretch goal for a reason. This involves all the other teams with Rust code and the iOS teams too. We're not there yet and there's currently no explicit priority to make development of Firefox iOS on M1 hardware possible. But when it comes to it, Glean will be ready for it and the team can assist others to get it over the finish line.

Want to hear more about Glean and our cross-platform Rust development? Come to next week's Rust Linz meetup, where I will be talking about this.

Footnotes:

See Platform Support for what the Tiers means. ↩
The other name for that target. ↩
"Apple Silicon" is yet another name for what is essentially the same as "M1" or "macOS aarch64" ↩
Or arm64 for that matter. Yes, yet another name for the same thing. ↩
"Universal Binaries" have existed for a long time now and allow for one binary to include the compiled artifacts for multiple targets. It's how there's only one Firefox for Mac download which runs natively on either Mac platform. ↩
Yup, the main documentation they link to is a WWDC 2019 talk recording video. ↩

Three-year Moziversary

Mon, 01 Mar 2021 11:00:00 +0100

Has it really been 3 years? I guess it has. I joined Mozilla as a Firefox Telemetry Engineer in March 2018, I blogged twice already: 2019, 2020.

And now it's 2021. 2020 was nothing like I thought it would and still been a lot like I said last year at this point. It's been Glean all over the year, but instead of working from the office and occasionally meeting my team in person, it's been working from home for 12 months now.

In September of last year I officially became the Glean SDK tech lead and thus I'm now responsible for the technical direction and organisation of the whole project, but really this is a team effort. Throughout the year we made Project FOG happen. At least the code is there and we can start migrating now. It's far from finished of course.

I blogged about Glean in 2021 and we're already seeing the first pieces of that plan being implemented. After some reorganization the team I'm in grew again so now we have Daniel, Anthony and Frank there as well (we're still Team "Browser Measurement III"). This brings two parts of the Glean project closer together (not that it was far apart).

There's more work ahead, interesting challenges to tackle and projects to support in migrating to Glean. To the next year and beyond!

Thank you

Thanks to my team: Bea, Chris, Travis, Alessio, Mike, Daniel, Anthony and Frank and also thanks to the bigger data engineering team within Mozilla. And thanks to all the other people at Mozilla I work with.

This Week in Glean: Boring Monitoring

Wed, 24 Feb 2021 15:00:00 +0100

All "This Week in Glean" blog posts are listed in the TWiG index (and on the Mozilla Data blog). This article is cross-posted on the Mozilla Data blog.

Every Monday the Glean has its weekly Glean SDK meeting. This meeting is used for 2 main parts: First discussing the features and bugs the team is currently investigating or that were requested by outside stakeholders. And second bug triage & monitoring of data that Glean reports in the wild.

Most of the time looking at our monitoring is boring and that's a good thing.

From the beginning the Glean SDK supported extensive error reporting on data collected by the framework inside end-user applications. Errors are produced when the application tries to record invalid values. That could be a negative value for a counter that should only ever go up or stopping a timer that was never started. Sometimes this comes down to a simple bug in the code logic and should be fixed in the implementation. But often this is due to unexpected and surprising behavior of the application the developers definitely didn't think about. Do you know all the ways that your Android application can be started? There's a whole lot of events that can launch it, even in the background, and you might miss instrumenting all the right parts sometimes. Of course this should then also be fixed in the implementation.

Monitoring Firefox for Android

For our weekly monitoring we look at one application in particular: Firefox for Android. Because errors are reported in the same way as other metrics we are able to query our database, aggregate the data by specific metrics and errors, generate graphs from it and create dashboards on our instance of Redash.

The above graph displays error counts for different metrics. Each line is a specific metric and error (such as Invalid Value or Invalid State). The exact numbers are not important. What we're interested in is the general trend. Are the errors per metrics stable or are there sudden jumps? Upward jumps indicate a problem, downward jumps probably means the underlying bug got fixed and is finally rolled out in an update to users.

We have another graph that doesn't take the raw number of errors, but averages it across the entire population. A sharp increase in error counts sometimes comes from a small number of clients, whereas the errors for others stay at the same low-level. That's still a concern for us, but knowing that a potential bug is limited to a small number of clients may help with finding and fixing it. And sometimes it's really just bogus client data we get and can dismiss fully.

Most of the time these graphs stay rather flat and boring and we can quickly continue with other work. Sometimes though we can catch potential issues in the first days after a rollout.

In this graph from the nightly release of Firefox for Android two metrics started reporting a number of errors that's far above any other error we see. We can then quickly find the implementation of these metrics and report that to the responsible team (Filed bug, and the remediation PR).

But can't that be automated?

It probably can! But it requires more work than throwing together a dashboard with graphs. It's also not as easy to define thresholds on these changes and when to report them. There's work underway that hopefully enables us to more quickly build up these dashboards for any product using the Glean SDK, which we can then also extend to do more reporting automated. The final goal should be that the product teams themselves are responsible for monitoring their data.

This Week in Glean: Glean in 2021

Fri, 18 Dec 2020 15:00:00 +0100

All "This Week in Glean" blog posts are listed in the TWiG index (and on the Mozilla Data blog). This article is cross-posted on the Mozilla Data blog.

A year ago the Glean project was different. We had just released Glean v22.1.0 and Fenix (aka Firefox for Android aka Firefox Daylight) was not released yet, Project FOG was just an idea for the year to come.

2020 changed that, but 2020 changed a lot. What didn't change was my main assignment: I kept working all throughout the year on the Glean SDK, fixing bugs, expanding its capabilities, enabling more platforms and integrating it into more products. Of course this was only possible because the whole team did that as well.

In September I took over the tech lead role for the SDK from Alessio. One part of this role includes thinking bigger and sketching out the future for the Glean SDK.

Let's look at this future and what ideas we have for 2021.

(One note: right now these are ideas more than a plan. It neither includes a timeline nor does it include all the other things maintenance includes.)

The vision

The Glean SDK is a fully self-servable telemetry SDK, usable across different platforms. It enables product owners and engineers to instrument their products and rely on their data collection, while following Mozilla policies & privacy standards.

The ideas

In the past weeks I started a list of things I want the Glean SDK to do next. This is a short and incomplete list of not-yet-proposed or accepted ideas. For 2021 we will need to fit this in with the larger Glean project, including plans and ideas for the pipeline and tooling. Oh, and I need to talk with my team to actually decide on the plan and allocate who does what when.

Metric types

When we set out to revamp our telemetry system we build it with the idea of offering higher-level metric types, that give more meaning to individual data points, allowing more correct data collection and better aggregation, analysis and visualisation of this data. We're not there yet. We currently support more than a dozen metric types across all platforms equally, with the same user-friendly APIs in all supported languages. We also know that this still does not cover all intended use cases that, for example, Firefox Desktop wants. In 2021 we will probably need to work on a few more types, better ergonomics and especially documentation on when which metric type is appropriate.

Revamped testing APIs

Engineers should instrument their code where appropriate and use the collected data to analysis behavior and performance in the wild. The Glean SDK ensures their data is reliably collected and sent to our pipeline. But we cannot ensure that the way the data is collected is correct or whether the metric even makes sense. That's why we encourage that each metric recording is accompanied by tests to ensure data is collected under the right circumstances and that its the right data under the given test case. Only that way folks will be able to verify and analyse the incoming data later and rely on their results.

The available testing APIs in the Glean SDK provide a bare minimum. For each metric type one can check which data it currently holds. That's probably easy to validate for simpler metrics such as strings, booleans and counters, but as soon as you have more complex one like any of the distributions or a timespan it gets a bit more complex. Additionally we offer a way to check if any errors occured during recording, which are usually also reported in the data. But here again developers need to know which errors can happen under what circumstances for which metric type.

And at last we want developers to reach for custom pings when the default pings are not sufficient. Testing these is currently near impossible.

Testing these different usages of the SDK can and should be improved. We already have the first proposals and bugs for this:

Custom Ping Unit Tests proposes a simple callback-like API to validate data in custom pings
Testing parameter in Glean metric definitions proposes to make existing tests/testing documentation for metrics more visible
Improvements to Glinter, the Glean linter (e.g. opinions on naming of things)

(Note: as of writing these document might be inaccessible to the wider public, but they will be made public once we gather wider feedback)

For sure we will have more ideas on this in 2021.

UniFFI - generate all the code

The Glean SDK is cross-platform by shipping a core library written in Rust with language bindings on top that connect it with the target platform. This has the advantage that most of the logic is written once and works across all the targets we have. But this also has the downside that each language binding sort of duplicates the user-visible API in their target language again. Currently all metric type implementations need to happen in Rust first, followed by implementations in the language bindings.

All of the implementation work today is mostly manual, but usually follows the same approach. The language binding implementations should not hold additional state, but currently some still do.

Implementation of new metric types and fixing bugs or improving the recording APIs of existing ones results in a lot of busy work replicating the same code patterns in all the languages (of which we now have have 7: Kotlin, Swift, Python, C#, Rust, JavaScript and C++). If we also come up with new metric types next year this becomes worse.

A while ago some of my colleagues started the UniFFI project, a multi-language bindings generator for Rust. For a couple of reasons the Glean SDK cannot rely on that yet.

In 2021 I want us to work towards the goal of using UniFFI (or something like it) to reduce the manual work required on each language binding. This should reduce our general workload supporting several languages and help us avoid accidental bugs on implementing (and maintaining) metric types.

This is a rather short list of things that only focus on the SDK. Let's see what we get tackled in 2021.

End of 2020

The This Week in Glean series is going into winter hiatus until the second week of January 2021. Thanks to everyone who contributed to 25 This Week in Glean blog posts in 2020:

Alessio, Travis, Mike, Chutten, Bea, Raphael, Anthony, Will & Daosheng

This Week in Glean: FOG progress report

Tue, 06 Oct 2020 16:00:00 +0200

All "This Week in Glean" blog posts are listed in the TWiG index (and on the Mozilla Data blog). This article is cross-posted on the Mozilla Data blog.

About a year ago chutten started the "This Week in Glean" series with an initial blog post about Glean on Desktop. Back then we were just getting started to bring Glean to Firefox Desktop. No code had been written for Firefox Desktop, no proposals had been written to discuss how we even do it.

Now, 12 months later, after four completed milestones, a dozen or so proposal and quite a bit of code, the Project Firefox on Glean (FOG) is finally getting into a stage where we can actually use and test it. It's not ready for prime time yet, but FOG is enabled in Firefox Nightly already.

Over the past 4 weeks I've been on and off working on building out our support for a C++ and a JavaScript API. Soon Firefox engineers will be able to instrument their code using Glean. In C++ this will look like:

mozilla::glean::test_only::count_something.Add(42);

JavaScript developers will use it like this:

Glean.test_only.count_something.add(32);

My work is done in bug 1646165. It's still under review and needs some fine-tuning before that, but I hope to land it by next week. We will then gather some data from Nightly and validate that it works as expected (bug 1651110), before we let it ride the trains to land in stable (bug 1651111).

Our work won't end there. With my initial API work landing we can start supporting all metric types, we still need to figure out some details of how FOG will handle IPC, and then there's the whole process of convincing other people to use the new system.

2020 will be the year of Glean on the Desktop.

This Week in Glean: Leveraging Rust to build cross-platform mobile libraries

Tue, 01 Sep 2020 15:00:00 +0200

All "This Week in Glean" blog posts are listed in the TWiG index (and on the Mozilla Data blog). This article is cross-posted on the Mozilla Data blog.

A couple of weeks ago I gave a talk titled "Leveraging Rust to build cross-platform mobile libraries". You can find my slides as a PDF. It was part of the Rusty Days Webference, an online conference that was initially planned to happen in Poland, but had to move online. Definitely check out the other talks.

One thing I wanted to achieve with that talk is putting that knowledge out there. While multiple teams at Mozilla are already building cross-platform libraries, with a focus on mobile integration, the available material and documentation is lacking. I'd like to see better guides online, and I probably have to start with what we have done. But that should also be encouragement for those out there doing similar things to blog, tweet & speak about it.

Who else is using Rust to build cross-platform libraries, targetting mobile?

I'd like to hear about it. Find me on Twitter (@badboy_) or drop me an email.

The Glean SDK

I won't reiterate the full talk (go watch it, really!), so this is just a brief overview of the Glean SDK itself.

The Glean SDK is our approach to build a modern Telemetry library, used in Mozilla's mobile products and soon in Firefox on Desktop as well.

The SDK consists of multiple components, spanning multiple programming languages for different implementations. All of the Glean SDK lives in the GitHub repository at mozilla/glean. This is a rough diagram of the Glean SDK tech stack:

On the very bottom we have glean-core, a pure Rust library that is the heart of the SDK. It's responsible for controlling the database, storing data and handling additional logic (e.g. assembling pings, clearing data, ..). As it is pure Rust we can rely on all Rust tooling for its development. We can write tests that cargo test picks up. We can generate the full API documentation thanks to rustdoc and we rely on clippy to tell us when our code is suboptimal. Working on glean-core should be possible for everyone that knows some Rust.

On top of that sits glean-ffi. This is the FFI layer connecting glean-core with everything else. While glean-core is pure Rust, it doesn't actually provide the nice API we intend for users of Glean. That one is later implemented on top of it all. glean-ffi doesn't contain much logic. It's a translation between the proper Rust API of glean-core and C-compatible functions exposed into the dynamic library. In it we rely on the excellent ffi-support crate. ffi-support knows how to translate between Rust and C types, offers a nice (and safer) abstraction for C strings. glean-ffi holds some state: the instantiated global Glean object and metric objects. We don't need to pass pointers back and forth. Instead we use opaque handles that index into a map held inside the FFI crate.

The top layer of the Glean SDK are the different language implementations. Language implementations expose a nice ergonomic API to initialize Glean and record metrics in the respective language. Additionally each implementation handles some special cases for the platform they are running on, like gathering application and platform data or hooking into system events. The nice API calls into the Glean SDK using the exposed FFI functions of glean-ffi. Unfortunately at the moment different language implementations carry different amounts of actual logic in them. Sometimes metric implementations require this (e.g. we rely on the clock source of Kotlin for timing metrics), in other parts we just didn't move the logic out of the implementations yet. We're actively working on moving logic into the Rust part where we can and might eventually use some code generation to unify the other parts. uniffi is a current experiment for a multi-language bindings generator for Rust we might end up using.

This Week in Glean: Bytes in Memory (on Android)

Mon, 04 May 2020 15:30:00 +0200

Last week's blog post: This Week in Glean: Glean for Python on Windows by Mike Droettboom. All "This Week in Glean" blog posts are listed in the TWiG index (and on the Mozilla Data blog). This article is cross-posted on the Mozilla Data blog.

With the Glean SDK we follow in the footsteps of other teams to build a cross-platform library to be used in both mobile and desktop applications alike. In this blog post we're taking a look at how we transport some rich data across the FFI boundary to be reused on the Kotlin side of things. We're using a recent example of a new API in Glean that will drive the HTTP upload of pings, but the concepts I'm explaining here apply more generally.

Note: This blog post is not a good introduction on doing Rust on Android, but I do plan to write about that in the future as well.

Most of this implementation was done by Bea and I have been merely a reviewer, around for questions and currently responsible for running final tests on this feature.

The problem

The Glean SDK provides an FFI API that can be consumed by what we call language bindings. For the most part we pass POD over that API boundary: integers of various sizes (a problem in and of itself if the sides disagree about certain integer sizes and signedness), bools (but actually encoded as an 8-bit integer¹), but also strings as pointers to null-terminated UTF-8 strings in memory (for test APIs we encode data into JSON and pass that over as strings).

However for some internal mechanisms we needed to communicate a bit more data back and forth. We wanted to have different tasks, where each task variant could have additional data. Luckily this additional data is either some integers or a bunch of strings only and not further nested data.

So this is the data we have on the Rust side:

enum Task {
    Upload(Request),
    Wait,
    Done,
}

struct Request {
    id: String,
    url: String,
}

(This code is simplified for the sake of this blog post. You can find the full code online in the Glean repository.)

And this is the API a user would call:

fn get_next_task() -> Task

The solution

Before we can expose a task through FFI we need to transform it into something C-compatible:

use std::os::raw::c_char;

#[repr(u8)]
pub enum FfiTask {
    Upload {
        id: *mut c_char,
        url: *mut c_char,
    },
    Wait,
    Done,
}

We define a new enum that's going to be represent its variant as an 8-bit integer plus the additional data for Upload. The other variants stay data-less.

We also provide conversion from the proper Rust type to the FFI-compatible type:

impl From<Task> for FfiTask {
    fn from(task: Task) -> Self {
        match task {
            Task::Upload(request) => {
                let id = CString::new(request.id).unwrap();
                let url = CString::new(request.url).unwrap();
                FfiTask::Upload {
                    id: document_id.into_raw(),
                    url: path.into_raw(),
                }
            }
            Task::Wait => FfiTask::Wait,
            Task::Done => FfiTask::Done,
        }
    }
}

The FFI API becomes:

#[no_mangle]
extern "C" fn glean_get_next_task() -> FfiTask

With this all set we can throw cbindgen at our code to generate the C header, which will produce this snippet of code:

enum FfiTask_Tag {
  FfiTask_Upload,
  FfiTask_Wait,
  FfiTask_Done,
};
typedef uint8_t FfiTask_Tag;

typedef struct {
  FfiTask_Tag tag;
  char *id;
  char *url;
} FfiTask_Upload_Body;

typedef union {
  FfiTask_Tag tag;
  FfiTask_Upload_Body upload;
} FfiTask;

This is the C representation of a tagged union. The layout of these tagged unions has been formally defined in Rust RFC 2195.

Each variant's first element is the tag, allowing us to identify which variant we have. cbindgen automatically inlined the Wait and Done variants: they are nothing more than a tag. The Upload variant however gets its own struct.

On the Kotlin side of things we use JNA (Java Native Access) to call C-like functions and interact with C types. After some research by Bea we found that it already provides abstractions over C unions and structs and we could implement the equivalent parts for our Task in Kotlin.

First some imports and replicating the variants our tag takes.

import com.sun.jna.Structure
import com.sun.jna.Pointer
import com.sun.jna.Union

enum class TaskTag {
    Upload,
    Wait,
    Done
}

Next is the body of our Upload variant. It's a structure with two pointers to strings. Kotlin requires some annotations to specify the order of fields in memory. We also inherit from Structure, a class provided by JNA, that will take care of reading from memory and making the data accessible in Kotlin.

@Structure.FieldOrder("tag", "id", "url")
class UploadBody(
    @JvmField val tag: Byte = TaskTag.Done.ordinal.toByte(),
    @JvmField val id: Pointer? = null,
    @JvmField val url: Pointer? = null,
) : Structure() { }

And at last we define our union. We don't need a field order, it's a union afterall, only one of the fields is valid at a time.

open class FfiTask(
    @JvmField var tag: Byte = TaskTag.Done.ordinal.toByte(),
    @JvmField var upload: UploadBody = UploadBody()
) : Union() {
    class ByValue : FfiTask(), Structure.ByValue

    fun toTask(): Task {
        this.readField("tag")
        return when (this.tag.toInt()) {
            TaskTag.Upload.ordinal -> {
                this.readField("upload")
                val request = Request(
                    this.upload.id.getRustString(),
                    this.upload.url.getRustString()
                )
                Task.Upload(request)
            }
            TaskTag.Wait.ordinal -> Task.Wait
            else -> Task.Done
        }
    }
}

This also defines the conversion to a new Kotlin type that eases usage on the Kotlin side. getRustString is a small helper to copy the null-terminated C-like string to a Kotlin string.

The FFI function on the Kotlin side is defined as:

fun glean_get_next_task(): FfiTask.ByValue

The types we convert to and then work with in Kotlin are small classes around the data. If there's no attached data it's an object.

class Request(
    val id: String,
    val url: String,
) { }

sealed class Task {
    class Upload(val request: Request) : Task()

    object Wait : Task()

    object Done : Task()
}

As the final piece of this code on the Kotlin side we can now fetch new tasks, convert it to more convenient Kotlin objects and work with them:

val incomingTask = LibGleanFFI.INSTANCE.glean_get_next_task()
when (val action = incomingTask.toTask()) {
    is Task.Upload -> upload(action.request.id, action.request.url)
    Task.Wait -> return Result.retry()
    Task.Done -> return Result.success()
}

What's next?

Currently our new upload mechanism is under testing. We're reasonably sure that our approach of passing rich data across the FFI boundary is sound and not causing memory safety issues for now.

The advantage of going the way of encoding Rust enums into tagged unions for us is that we can use this API in all our current API consumers. C structs and unions are supported in Kotlin (for Android), Swift (for our iOS users) and Python (e.g. Desktop apps such as mozregression). It didn't require new tooling or dependencies to get it working and it's reasonably cheap in terms of processing cost (it's copying around a few bytes of data in memory).

The disadvantage however is that it requires quite a bit of coordination of the different pieces of code. As you've seen above for Kotlin we need to be careful to replicate the exact layout of data and all of this is (currently) hand-written. Any change on the Rust side might break this easily. Swift is a bit easier on this front as it has direct C translation.

The application-services team faced the same problem of how to transport rich data across the FFI boundary. They decided to go with protocol buffers and generate code for both sides of the FFI. They wrote about it in Crossing the Rust FFI frontier with Protocol Buffers. We decided against this way (for now), as it requires a bit more of a heavy-handed setup initially. We might reconsider this if we need to expand this API further.

My dream solution is still a *-bindgen crate akin to wasm-bindgen that creates all this code.

Footnotes:

Don't use a bool with JNA. JNA doesn't handle it well and has its own conception of the size of a bool that differs from what C and Rust think. See When to use what method of passing data between Rust and Java/Swift in the Glean SDK book. ↩

Review Feedback: a response to the Feedback Ladder

Wed, 18 Mar 2020 12:30:00 +0100

Last week I read Feedback Ladders: How We Encode Code Reviews at Netlify and also shared that with my team at Mozilla. In this post I want to summarize how we organize our reviews and compare that to Netlify's Feedback Ladder.

My team is mainly responsible for all work on Firefox Telemetry and our other projects. (Nearly) everything we do is first tracked in Bugs on Bugzilla. No code change (nor doc change) will land without review. For changes to land in Firefox the developer is responsible for picking the right reviewer, though right now that's mostly shared work between chutten and me. Sometimes we need to involve experts from other components of Firefox.

On Glean we rely on an auto-assign bot to pick a reviewer after opening a pull request. Sometimes the submitter also actively picks one from the team as a reviewer, e.g. if it's a followup to previous work or if some niche expertise is needed.

When reviewing we use a system not too dissimilar to the Feedback ladder. However it is much more informal.

Let's compare the different steps:

⛰ Mountain / Blocking and requires immediate action

In Mozilla speak that would be an "r-" - "Rejected". In my team this rarely (never?) happens on code changes.

On any bigger changes or features we usually start with a design proposal, that goes through feedback iterations with stakeholders (the direct team or colleagues from other teams, depending on scope). This would be the point to shut down ideas or turn them around to fit our and our user's needs. Design proposals vary in depth, but may already include implementation details where required.

🧗‍♀️ Boulder / Blocking

For us this is "Changes requested". Both review tools we use (Phabricator and plain GitHub PRs) have this as explicit review states.

The code change can't land until problems are fixed. Once the developer pushed new changes the pull request will need another round of review.

All problems should be clearly pointed out during the review and comments attached to where the problem is. However, unlike the Feedback ladder, our individual comments don't follow a strict wording, so the developer who submitted the change can't differentiate between them easily.

⚪️ Pebble / Non-blocking but requires future action

This is famously known here as "r+wc" - "review accepted, with comments".

Now for us this is actually two different parts:

First, the reviewer is fine with the overall change, but found some smaller things that definitely need to be changed, such as documentation wording, code comments or naming. However, it is considered the developer's task to ensure these changes get made before the PR is landed and no additional round of review needs to follow. GitHub luckily allows reviewers to submit the exact change required and the developer can apply it with a button click, so there's not always the need to go back to the code editor, commit code, push it, ...

Second, some things require a follow-up, such as some possible code refactor, additional features or tracking the bug fix through the release process for later validation (we deal with data, so for some fixes we need to see real-world data coming in to determine if the fix worked). Reviewers should ask for a bug to be filed and the developer usually posts the filed bug as a comment. In that state the pull request is then ready to get merged.

Again, there's no formal concept or wording (other than "this needs a follow-up bug" and GitHub's "Apply suggestion") we use to make the review comments stick out.

⏳ Sand / Non-blocking but requires future consideration

This is very similar to the "⚪️ Pebble" step, but the follow-up bugs filed are more likely going to be "Investigate ..." or "Proposal for ...".

🌫 Dust / Non-blocking, “take it or leave it”

This is all the other little comments and will usually end in a "r+" - "Review done & accepted". We enforce code formatting via tools, so there's rarely need to discuss this. Therefore this comes down to naming or slightly different code patterns.

More often than not I label these in my comments as "small nit: ...". More often than not these are still taken in and applied.

So ... ?

In my team we already use an informal but working method to express the different kinds of review feedback. We certainly lack in clarity and immediate visibility of the different patterns and that's certainly a thing we can improve. Not only might that help us right now, it would also help in later onboarding new folks.

I'm not fully sold on the metaphors used in the Feedback Ladder, but I do like using emojis in combination with plaintext to signal things.

Two-year Moziversary

Mon, 02 Mar 2020 15:00:00 +0100

Woops, looks like I missed my Two Year Moziversary! 2 years ago, in March 2018, I joined Mozilla as a Firefox Telemetry Engineer. Last year I blogged about what happened in my first year.

One year later I am still in the same team, but other things changed. Our team grew (hello Bea & Mike!), and shrank (bye Georg!), lots of other changes at Mozilla happened as well.

However one thing stayed during the whole year: Glean.
Culminating in the very first Glean-written-in-Rust release, but not slowing down after, the Glean project is changing how we do telemetry in Mozilla products.

Glean is now being used in multiple products across mobile and desktop operating systems (Firefox Preview, Lockwise for Android, Lockwise for iOS, Project FOG). We've seen over 2000 commits on that project, published 16 releases and posted 14 This Week in Glean blogposts.

This year will be focused on maintaining Glean, building new capabailities and bringing more of Glean back into Firefox on Desktop. I am also looking forward to speak more about Rust and how we use it to build and maintain cross-platform libraries.

To the next year and beyond!

Thank you

Thanks to my team: :bea, :chutten, :Dexter, :mdboom & :travis!
Lots of my work involves collaborating with people outside my direct team, gathering feedback, taking in feature requests and triaging bug reports. So thanks also to all the other people at Mozilla I work with.

"Edit this file on GitHub"

Thu, 06 Feb 2020 16:38:00 +0100

At work I help with maintaining two large documentation books:

Firefox Data Docs aka docs.telemetry.mozilla.org aka dtmo
The Glean SDK book

Back in 2018 I migrated dtmo from gitbook to mdbook (see the pull request). mdbook is maintained by the Rust project and hosts the Rust book as well as a multitude of other community projects. It provided all we need, plus a way to extend it with some small things, I blogged about ToC and mermaid before.

During the Mozilla All Hands last week my colleague Mike casually asked why we don't have links to quickly edit the documentation. When someone discovers a mistake or inaccuracy in the book the current process involves finding the repository of the book, then finding the right file, then edit that file (through the GitHub UI or by cloning the repository), then push changes, open a pull request, wait for review and finally get it merged and deployed.

I immediately set out to build this feature.

I present to you: mdbook-open-on-gh

It's another preprocessor for mdbook, that adds a link to the edit dialog on GitHub (if your book is actually hosted on GitHub). And that's how it looks:

It's already deployed on dtmo and the Glean SDK book and simplifies the workflow to: click the link, edit the file on GitHub, commit and open a PR, get a review and merge it to deploy.

If you want to use this preprocessor, install it:

cargo install mdbook-open-on-gh

Add it as a preprocessor to your book.toml:

[preprocessor.open-on-gh]
command = "mdbook-open-on-gh"
renderer = ["html"]

Add a repository URL to use as a base in your book.toml:

[output.html]
git-repository-url = "https://github.com/mozilla/glean"

To style the footer add a custom CSS file for your HTML output:

[output.html]
additional-css = ["open-in.css"]

And in open-in.css style the <footer> element or directly the CSS element id open-on-gh:

footer {
  font-size: 0.8em;
  text-align: center;
  border-top: 1px solid black;
  padding: 5px 0;
}

This code block shrinks the text size, center-aligns it under the rest of the content and adds a small horizontal bar above the text to separate it from the page content.

Finally, build your book as normal:

mdbook path/to/book

This Week in Glean: Cargo features - an investigation

Mon, 03 Feb 2020 15:00:00 +0100

The last blog post: This Week in Glean: Glossary by Bea.

As :chutten outlined in the first TWiG blog post we're currently prototyping Glean on Desktop. After a couple rounds of review, some adjustements and some learnings from doing Rust on mozilla-central, we were ready to land the first working prototype code earlier this year (Bug 1591564).

Unfortunately the patch set was backed out nearly immediately ¹ for 2 failures. The first one was a "leak" (we missed cleaning up memory in a way to satisfy the rigorous Firefox test suite, that was fixed in another patch). The second one was a build failure on a Windows platform.

This is what the log had to say about it (shortened to the relevant parts here, see the full log output):

lld-link: error: undefined symbol: __rbt_backtrace_pcinfo
>>> referenced by gkrust_gtest.lib(backtrace-5286ea09b9822175.backtrace.3kzojw1m-cgu.3.rcgu.o)
lld-link: error: undefined symbol: __rbt_backtrace_create_state
>>> referenced by gkrust_gtest.lib(backtrace-5286ea09b9822175.backtrace.3kzojw1m-cgu.3.rcgu.o)
lld-link: error: undefined symbol: __rbt_backtrace_syminfo
>>> referenced by gkrust_gtest.lib(backtrace-5286ea09b9822175.backtrace.3kzojw1m-cgu.3.rcgu.o)
clang-9: error: linker command failed with exit code 1 (use -v to see invocation)
/builds/worker/workspace/build/src/config/rules.mk:608: recipe for target 'xul.dll' failed]

I set out to investigate this error. While I had not seen that particular error before, I knew about the backtrace crate. It caused me some trouble before (it depends on a C library, and won't work on all targets easily). I knew that the Glean SDK doesn't really depend on its functionality² and thus removing it from our dependency graph would probably solve the issue. But first I had to find out why we depend on it somewhere and why it is causing these linker errors to begin with.

The first thing I noticed is that we didn't include anything new in the patch set that was now rejected. Through some experimentation and use cargo-tree I could tell that backtrace was included in the build before our Glean patch³, as a transitive dependency of another crate: failure.

So why didn't it fail the build before? As per the errors above, the build failed only during linking, not compilation, which makes me believe those functions were never linked in previously, because no one passed around any errors that would cause these functions to be used.

As said before, the Glean SDK doesn't really need failure's backtrace feature, so I tried disabling its default features. Due to how cargo currently works, this needs to be done across all transitive dependencies (the final feature set a crate is compiled with is the union across everything).

I then changed mozilla-central to use the crates from git directly for testing.

Turns out that still fails with the same issue on the Windows target. Something was re-enabling the "std" feature of failure in tree.

cargo-feature-set was able to show me all enabled features for all dependencies I tracked it down further⁴.

Turns out the quantum_render feature enables the webrender_bindings crate, which then somehow pulls in failure through transitive dependencies again. More trial-and-error revealed its a dependency of the dirs crate, only used on Windows. Except dirs doesn't need failure for the target we're building for (x86_64-pc-windows-gnu or Mac or Linux). It's again a transitive dependency for a crate called redox_users, which is only pulled in when compiled for Redox⁵.

Except that's not how Cargo works. Cargo always pulls in all dependencies, merges all features and only later ignores the crates it doesn't actually need. That's a long standing issue:

So now we identified who's pulling in the backtrace crate and maybe even identified why it was not a problem before. How do we fix this?

As shown before, just disabling the backtrace feature in crates we use directly doesn't solve it, so one quick workaround was to force failure itself to not have that feature ever. Easily done:

--- Cargo.toml
+++ Cargo.toml
@@ -23,5 +23,5 @@ members = [".", "failure_derive"]
 [features]
 default = ["std", "derive"]
 #small-error = ["std"]
-std = ["backtrace"]
+std = []
 derive = ["failure_derive"]

I forked failure and commited that patch, then made mozilla-central use my forked version instead. Later I also removed failure from both Glean and application-services' ffi-support, as the small functionality we got from it was easily reimplemented manually.

Both approaches are short-term fixes for getting Glean into Firefox and it's clear that this issue might easily come up in some form soon again for either us or another team. It's also a major hassle for lots of people outside of Mozilla, for example people working on embedded Rust frequently run into problems with no_std libraries suddenly linking in libstd again.

Initially I also planned to figure out a way forward for Cargo and come up with a fix for it, but as it turns out: Someone is already doing that!

@ehuss started working on adding a new feature resolver. While not yet final, it will bring a new -Zfeatures flag initially:

itarget — Ignores features for target-specific dependencies for targets that don't match the current compile target.
build_dep — Prevents features enabled on build dependencies from being enabled for normal dependencies.
dev_dep — Prevents features enabled on dev dependencies from being enabled for normal dependencies.

I'm excited to see this being worked on and can't wait to try it out. It will take longer for mozilla-central to rely on this of course, but I hope this will eventually solve one of the long standing issues with cargo.

cargo tree --features bitsdownload,cranelift_x86,cubeb-remoting,fogotype,gecko_debug,gecko_profiler,gecko_refcount_logging,moz_memory,moz_places,new_cert_storage,new_xulstore,quantum_render

Footnotes:

When patches are merged, a full set of tests are run on the Mozilla CI servers. If these tests fail the patch is reverted ("backed out") and the initial committer informed. ↩
Glean's error handling is pretty simplistic. We mostly only log errors on the FFI boundary and don't propagate them over this boundary. So far we didn't need a backtrace for errors there. ↩
The exact command enabling all the same features as the build I run in toolkit/library/rust/shared was (on the parent commit of my later patch to fix it: 7a3be2bbc): ↩
Same feature set as above, just running cargo feature-set this time. ↩
"Redox is a Unix-like Operating System written in Rust, aiming to bring the innovations of Rust to a modern microkernel and full set of applications". Firefox doesn't build on Redox. ↩

This Week in Glean: Differences

Fri, 29 Nov 2019 11:37:00 +0100

Last week's blog post: This Week in Glean: Glean in Private by chutten.

Currently my team is responsible for the Telemetry framework inside Firefox on Desktop and also the Glean SDK, targeting our mobile products. We're working on bringing the Glean experience to Firefox on Desktop, but in the meantime Telemetry is what we have, need to support and sometimes implement new features on.

One of these features is a new ping (or, better, a change in a ping), that we now want to support across all our products. I'm speaking of the deletion-request ping here. When a user opts out of Telemetry we take this as a signal to also delete associated data from our pipeline.

Implementation in Firefox Desktop was merely renaming an existing ping that is triggered when the user disables "Data Collection and Use" (about:preferences -> Privacy & Security). It contains no additional data. Implementation in Glean was not much harder either. Glean already supports custom pings: Pings that can be defined and send by the application using Glean. Glean's internal pings follow the same pattern, they are just pre-defined. The biggest difference?

It's called deletion_request ping instead.

On ingestion data from a ping is decoded from its JSON form and put into tables on BigQuery (in our documentation you can find an overview of the data pipeline if you are interested). BigQuery table names can only contain alphanumeric characters and underscores (see "Table naming" in the BigQuery documentation). We avoid any translation in the pipeline by just enforcing this directly on ping names.

Glean also enforces the payload schema of pings. Glean itself controls portion of the data, including a sequence number, date field and a bit of metadata about the application its running in (see the ping sections). The rest of the payload consists of metrics as defined by users of Glean. While implementing the new ping I stumbled upon another small detail of Glean: Pings won't be sent out if they would not contain any metrics. And our new ping, by design, should not contain any metrics!

We don't want to change this for other pings, so I had to introduce a new flag now: sendIfEmpty (PR #139).

That way we can allow the deletion_request ping to be sent without any metrics in there, only containing the basic information.

The implementation of the new ping is now done and currently waiting for data review (PR #526). I hope to land this early next week.

This Week in Glean: A Release

Thu, 24 Oct 2019 17:30:00 +0200

Last week's blog post: This Week in Glean: Glean on Desktop (Project FOG) by chutten.

Back in June when Firefox Preview shipped, it also shipped with Glean, our new Telemetry library, initially targeting mobile platforms. Georg recently blogged about the design principles of Glean in Introducing Glean — Telemetry for humans.

Plans for improving mobile telemetry for Mozilla go back as as far as December 2017. The first implementation of the Glean SDK was started around August 2018, all written in Kotlin (though back then it was mostly ideas in a bunch of text documents). This implementation shipped in Firefox Preview and was used up until now.

On March 18th I created an initial Rust workspace. This kicked of a rewrite of Glean using Rust to become a cross-platform telemetry SDK to be used on Android, iOS and eventually coming back to desktop platforms again.

1382 commits later¹ I tagged v19.0.0².

Obviously that doesn't make people use it right away, but given all consumers of Glean right now are Mozilla products, it's up on us to get them to use it. So Alessio did just that by upgrading Android Components, a collection of Android libraries to build browsers or browser-like applications, to this new version.

This will soon roll out to nightly releases of Firefox Preview and, given we don't hit any larger bugs, hit the release channel in about 2 weeks. Additionally, that finally unblocks the Glean team to work on new features, ironing out some sharp edges and bringing Glean to Firefox on Desktop. Oh, and of course we still need to actually release it for iOS.

Thanks

Glean in Rust is the project I've been constantly working on since March. But getting it to a release was a team effort with help from a multitude of people and teams.

Thanks to everyone on the Glean SDK team:

Thanks to Frank Bertsch for a ton of backend work as well as the larger Glean pipeline team led by Mark Reid, to ensure we can handle the incoming telemetry data and also reliably analyse it. Thanks to the Data Engineering team led by Katie Parlante. Thanks to Mihai Tabara, the Release Engineering team and the Cloud Operations team, to help us with the release on short notice. Thanks to the Application Services team for paving the way of developing mobile libraries with Rust and to the Android Components team for constant help with Android development.

Footnotes:

Not all of which are just code for the Android version. There's a lot of documentation too. ↩
This is the first released version. This is just the version number that follows after the Kotlin implementation. Version numbers are cheap. ↩

One-year Moziversary

Fri, 01 Mar 2019 12:00:00 +0100

At this day last year I walked into the Mozilla Berlin office to start my first day of work. 365 days later, today is my very first #moziversary.

I joined the Firefox Telemetry Team, now consisting of :chutten, :Dexter, :gfritzsche, :travis_ (he joined us in November) and me.

Since I joined I:

visited 5 of the Mozilla Offices (Berlin, Paris, London, Mountain View, San Francisco)
hosted 2 Rust All-Hands in the Berlin office
attended 2 Mozilla All-Hands
had a work week with my team
hosted regular Rust Hack'n'Learns in the office's Community Space (every second Wednesday!)
commented on over 250 Bugzilla bugs (about 800 comments apparently)
were assigned to and closed over 100 bugs
became a Telemetry peer
broke Firefox and Firefox for Android once or twice
read a dozen or so internal project proposals
had an uncountable number of meetings (yes, meetings can be fun & useful)
delivered crucial new features for Telemetry in Firefox
wrote far more SQL than I thought this job would involve
move twice inside Berlin

... and probably lots of other stuff I forgot.

The next projects for Firefox Telemetry are already in progress or about to start, mobile Telemetry is progressing quickly and, while I haven't written code for the current implementation, I hope to talk more about that soon.

To the next year and beyond!

Thank you

First and foremost of course a big THANK YOU to my team. It's been fun working along you.
Another big thank you to everyone in the Mozilla Berlin office, it's always a pleasure to come to the office. And of course thank you for everyone I worked with so far at Mozilla.

multi-store - Custom Telemetry with shared data

Tue, 22 Jan 2019 10:00:00 +0100

Last year I implemented a new feature for Firefox Telemetry that changes how we can collect and analyze data with different requirements in regard to user privacy & frequency. This post will shine some light on the (rather simple) implementation and usage.

Intro: What is Firefox Telemetry?

In order to understand how Firefox performs in the wild, it can collect a bunch of performance metrics and other information. How and why we do this and what data we collect is explained in more detail in a blog post by Rebecca Weiss, Director of Data Science here at Mozilla: It’s your data, we’re just living in it.

I work on the Telemetry component inside Firefox. It provides APIs that are used by the various other parts of the browser to gather data and is responsible for storing, collecting and sending this data in what we call "pings", a periodic collection of measurements. Telemetry data is only ever sent out if the user agreed to it (see "Data Collection and Use" in the "Privacy & Security" preferences of your Firefox).

Most data is collected in one of 3 different formats: histograms, scalars & events (see our collection overview). Firefox sends this data in the "main" ping once in a while (usually roughly daily) and clears out the stored data locally. A "main" ping always corresponds to a subsession, which itself is part of a session. This is further explained in our Session concepts.

People working with the data can therefore make certain assumptions on how to interpret the data from multiple pings across sessions and subsessions.

The problem

Some data should not be correlated with other data due to privacy concerns. So far we had to push Telemetry users to create custom pings and keep track of their own data. If they rely on scalars or histograms as recorded by Telemetry, but send a custom ping in different intervals, they can't make valid assumptions about the metrics, as they might have been reset in between. This leads to weird hacks or unnecessary code duplication. Additionally we can't provide any help or support for custom data and our tools can't handle it automatically (e.g. to generate dashboards).

A solution

Multi-Store.

Every metric was always tied to the schedule of the "main" ping (read: subsession/session). Our multi-store solution now enables metrics to be associated with multiple stores at once, defaulting to be in the "main" ping only. Any user with custom requirements can now select metrics to be included in their custom store (which is still subject to Data Collection Review). Telemetry is still responsible for actual data storage and the APIs, but now the custom ping is responsible for collecting, clearing and periodically sending this data.

An example

To demonstrate how this is used, let's create a custom ping, which will include one metric of its own and one that's also available in the "main" store (and thus the "main" ping).

We start by adding a new metric to Scalars.yaml:

  tick_times_rand:
    bug_numbers:
      - 0
    description: "A random value at every tick"
    expires: '71'
    kind: uint
    notification_emails:
      - janerik@fnordig.de
    release_channel_collection: opt-out
    record_in_processes:
      - "main"
    record_into_store:
      - "custom-store"

This defines a scalar named tick_times_rand in the browser.engagement category. The exact details are not important as we will record random values anyway. The important bit is setting the store using record_into_store.

We also include browser.engagement.tab_open_event_count into our store by just adding our custom store name to record_into_store (don't forget to also include "main" there).

--- c/toolkit/components/telemetry/Scalars.yaml
+++ i/toolkit/components/telemetry/Scalars.yaml
@@ -66,6 +66,9 @@ browser.engagement:
     release_channel_collection: opt-out
     record_in_processes:
       - 'main'
+    record_into_store:
+      - "main"
+      - "custom-store"

Next, we define our new custom ping in a new file in toolkit/components/telemetry/pings/CustomPing.jsm. We only define a very simple interface: A way to start the custom ping, which itself sets up a (persistent) timer (firing every 24 hours). When fired, notify() will be called, which then collects the payload from custom-store and schedule the ping for sending. Telemetry will take care of actually sending the ping (and retrying, storing it in the archive, etc.). For test reasons only we also send this ping when start() is called (so we don't actually have to wait 24 hours to see something).

var EXPORTED_SYMBOLS = ["TelemetryCustomPing"];

var TelemetryCustomPing = Object.freeze({
  start() {
    UpdateTimerManager.registerTimer(
      "telemetry_custom_ping",
      this,
      24 * 60 * 60; /* 1 day */
    );

    this.sendPing();
  },

  notify() {
    this.sendPing();
  },

  sendPing() {
    // Let's record just one more value, so _something_ is included.
    let randValue = Math.floor(Math.random() * Math.floor(100));
    Services.telemetry.scalarSet("browser.engagement.tick_times_rand", randValue);

    // We only include the scalars, as that's all we are recording for this ping.
    let payload = {
      custom: "This is custom data",
      scalars: Services.telemetry.getSnapshotForScalars("custom-store", /* clear */ true),
    };

    TelemetryController.submitExternalPing("custom", payload,
      {
        addClientId: false,
        addEnvironment: true,
      }
    );
  }
});

Now we need to actually start the ping timer. This should be done in the initialization phase of TelemetryController:

--- c/toolkit/components/telemetry/app/TelemetryController.jsm
+++ i/toolkit/components/telemetry/app/TelemetryController.jsm
@@ -62,6 +62,7 @@ XPCOMUtils.defineLazyModuleGetters(this, {
   TelemetryReportingPolicy: "resource://gre/modules/TelemetryReportingPolicy.jsm",
   TelemetryModules: "resource://gre/modules/ModulesPing.jsm",
   TelemetryUntrustedModulesPing: "resource://gre/modules/UntrustedModulesPing.jsm",
+  TelemetryCustomPing: "resource://gre/modules/CustomPing.jsm",
   UpdatePing: "resource://gre/modules/UpdatePing.jsm",
   TelemetryHealthPing: "resource://gre/modules/HealthPing.jsm",
   TelemetryEventPing: "resource://gre/modules/EventPing.jsm",
@@ -720,6 +721,9 @@ var Impl = {
           if (AppConstants.NIGHTLY_BUILD && AppConstants.platform == "win") {
             TelemetryUntrustedModulesPing.start();
           }
+
+          // Start the custom ping, which reports minimal information.
+          TelemetryCustomPing.start();
         }

         TelemetryEventPing.startup();

But it's not measuring anything!

That is right, but for now we only record one value when sending the ping.

And that's it! Our custom ping, with data sourced from a custom store and a release-shipped metric is ready¹. Let's build Firefox (this is the part where you have time to grab a cup of coffee if you haven't compiled Firefox before):

./mach build

...

2 - 40 minutes later. Still with me?

Now run the freshly built Firefox:

./mach run

Once Telemetry is initialized (should take 60 seconds), you should be able to see the freshly generated custom ping (in a development build this ping is never actually send out). To see it, go to about:telemetry, click on Current Ping in the upper-left corner, select Archived ping data and then the custom ping type. There you have it!

The full changeset is available in this commit on the gecko-dev mirror (don't worry, this is not landing in Firefox).

What's next?

Currently there's no user of the multi-store feature, mainly because it was only finished in early December and is currently lacking some documentation (which this post should change a bit). We expect some usage of this soon though.

Footnotes:

Not entirely true. This needs some small changes to the build system. ↩

What Rust is it?

Wed, 28 Nov 2018 20:09:00 +0100

A while ago (sometime in August) I built a small service called What Rust is it?.

Based on what What Train is it now? is for Firefox, this small service lists the three trains of Rust: release, beta & nightly as well as their respective versions. It's updated every day and actually checks what rustup installed. Additionally it shows the (planned) date of the next release.

Similar information is available on the Rust Lang Forge.

See it live:

## [www.whatrustisit.com](http://www.whatrustisit.com)

MozFest 2018: An inside perspective

Wed, 07 Nov 2018 14:34:00 +0100

It was the first time for me to attend MozFest, after I heard only good things about it from last year. When I walked into the venue on Saturday morning, I immediately knew that this conference was a bit different than others I attended: Spread over 9 floors, separated into different topic areas people from all sorts of backgrounds presented their work, project & ideas. Fully unprepared and a bit overwhelmed I jumped into sessions discussing interplanetary communication tools, data journalism using satellite imagery or how and if we should preserve all digital content as historic artifacts in an ever-changing world.

The one thing I enjoyed in all these sessions: Technology by itself was not considered as the solution to problems. It might inform what solutions we built, what features we add or what data we track, but we also need to be critical of its implementations. I definitely want to come back next year.

MozFest 2018: An inside perspective on the Mozilla Berlin Blog (English article)
So war's beim MozFest 2018 (German version of the article)

How I made a useless tool and the reactions made my sunday better

Sun, 30 Sep 2018 23:00:00 +0200

Last night I had an idea, which made me stay up two hours longer than I wanted. I built a useless tool: lysbilder, the slide system no one needed.

Today I tweeted about it and then went out to enjoy the day a bit:

@badboy_: I might have built a very useless slideshow tool on top of rustdoc, generated from code documentation, linked by trait implementations.

Please don’t use it¹.

https://badboy.github.io/lysbilder/

¹: however, I will use it once.

Quickly I got some responses:

@myriamjessier: I laughed. It was good enough to retweet just because of that.

and

@QuietMisdreavus: i love this so much

Those made me smile and made my sunday a bit better. :)

A new job

Sun, 18 Feb 2018 16:00:00 +0100

In September 2012 I was hired by rrbone to work as a software developer alongside my studies. Over the last 5 years I worked on multiple projects, some internal tools, some external work with and for clients and was also allowed to consult customers on-site or give Rust trainings. In addition, rrbone covered some of my conference visits, became the main sponsor of otsconf, the first conference I organized, and allowed me to attend other events under sponsorships. Together we built networks for large festivals, small unconferences and other community conferences. I was allowed to write Rust code for an opera play and open-source it, attended IETF 91 in Honolulu, Hawaii, and learned a ton about networks and how the internet works (sometimes it doesn't). All of that while I was still studying.

For all of this I am very grateful.

Thanks Dominik, for being my boss and friend for such a long time and for all the things I could learn.

But now after five years it is time for new challenges.

I will join Mozilla as a Firefox Telemetry Engineer, starting next Month.

Back in 2015 I met Flaki, who later pushed me to give my very first conference talk. Shortly after that he introduced me to the Mozilla Tech Speaker program. Not soon after I became a Tech Speaker, attended conferences and meetups as a speaker and worked closer with Mozilla. Since then I met a lot of people working at or with Mozilla, some of who I consider friends by now.

When I finished university last year some of the same people pushed me to apply at Mozilla, especially Ola. I did and today I can announce that it paid off and they hired me. Taking this new job also means I will move to Berlin soon.

As a Telemetry Engineer I will work on improving the quality of Firefox and help different teams to instrument new features in Firefox. I'm incredibly excited to work on this project together with folks from whom I can learn a thing or two (probably a lot more).