fnordig - posts tagged with 'rust'

Build your project Zig-style

Tue, 16 Jun 2026 11:00:00 +0200

Over the past few nights I've been tinkering with the Zig build system, after it got a bit of a rework lately. I had a plan: Learn a bit more about Zig, learn a bit more about the Zig build system and get a deeper understanding of what Cargo does to build a Rust project.

Building Rust projects with something that is not Rust is something I have experience with. With Zig I tinkered around a bit before and I'm keeping an eye on its development. The Zig build system was completely new to me.

So just like that I present: bygge-zig.
It builds Rust code:

src/mozilla/glean$ ls -l Cargo.toml
-rw-r--r-- 1 jer staff 616 Jun 15 16:10 Cargo.toml
src/mozilla/glean$ zig build --summary line
Build Summary: 194/194 steps succeeded
src/mozilla/glean$ du -sh .zig-cache
952M    .zig-cache
src/mozilla/glean$ tree zig-out
zig-out
├── bin
│   └── uniffi-bindgen
└── lib
    ├── glean
    ├── glean_core
    └── uniffi_bindgen

This is a build of the Glean SDK, my main project at work.

In 374 lines of Zig (and 79 lines of Ruby) I replicated what Cargo does in 80.000 lines of Rust. That includes building Rust code, building proc-macros and building and running build scripts (those build.rs files in the top-level directory of some crates).
Of course that is an oversimplification. Turns out a complete Rust build system is really tricky.

bygge-zig does nothing about resolving and fetching dependencies. It does not figure out what depends on what. It does not decide which features should be active. It translates what the build should look like into something the Zig build system can handle.

At one point Cargo had build plans, which allowed to gather information about how to run the build, mostly. That was removed and now there is a new way to get most of that information: the unit graph (on Rust Nightly):

$ cargo +nightly build -Z unstable-options --unit-graph | jq .
{
  "version": 1,
  "units": [
    {
      "pkg_id": "path+file:///home/jer/src/bygge-zig/crates/hello-world#0.1.0",
      "target": {
        "kind": [
          "bin"
        ],
        "crate_types": [
          "bin"
        ],
        "name": "hello-world",
        "src_path": "/home/jer/src/bygge-zig/crates/hello-world/src/main.rs",
        "edition": "2024",
        "doc": true,
        "doctest": false,
        "test": true
      },
      "profile": {
        "name": "dev",
        "opt_level": "0",
        "lto": "false",
        "codegen_backend": null,
        "codegen_units": null,
        "debuginfo": 2,
        "split_debuginfo": "unpacked",
        "debug_assertions": true,
        "overflow_checks": true,
        "rpath": false,
        "incremental": true,
        "panic": "unwind",
        "strip": {
          "deferred": "None"
        }
      },
      "platform": null,
      "mode": "build",
      "features": [],
      "dependencies": [
        {
          "index": 1,
          "extern_crate_name": "world",
          "public": false,
          "noprelude": false,
          "nounused": false
        }
      ]
    },
<snip>

This emits the units of work that Cargo will execute for the build. It still requires a lot more effort to turn that into something actually executable. For example the generated shell command for the first unit in the example graph is this:

CARGO_CRATE_NAME=hello_world \
CARGO_PKG_NAME=hello-world \
CARGO_PKG_VERSION=1.0.0 \
<snip>
rustc /home/jer/src/bygge-zig/crates/hello-world/src/main.rs \
  --edition 2024 \
  --extern world=target/debug/deps/libworld.rlib \
  -L target/debug/deps

This assumes the world library is already built and placed in the target/debug/deps/ directory. There's many more more environment variables that are set for a build.

The Glean SDK (and the expanded sample project) does make use of both proc-macros and build scripts across its dependencies. The way they need to be built differs from library code. proc-macros are built into dynamic libraries and loaded by the Rust compiler at compile time. Build scripts are always built for the host target and run before the crate's code gets compiled. Build scripts print out additional configuration to stdout, which is parsed by Cargo and injected into the rest of the build.

$ RUSTC=rustc OUT_DIR=.zig-cache/tmp .zig-cache/o/98128f/rust/build_script_build
cargo::rustc-check-cfg=cfg(build_ran)
cargo::rustc-cfg=build_ran
$ cat .zig-cache/tmp/code.rs
pub const NAME: &'static str = "builder";

This will result in the parameter --cfg=build_ran being appended to the rustc invocation for hello-world/src/lib.rs.

Not every environment variable is read by every project. Not every build.rs output line is needed for every build. To build the Glean SDK I'm getting away with the bare minimum. Nonetheless bygge-zig is able to drive quite a complicated build. And it does that on both macOS and Linux just fine.

What I learned about Zig

In this project I didn't really use much of what makes Zig Zig. No comptime, no error handling, no arena allocators. It is definitely an improvement over writing similar code in C, with a larger standard library and some generic data types to use. Some of the more Zig things I barely scratched. String handling requires an allocator, so it needs to be passed around, it's embedded in the std.Build structure available in a build.zig. Built-in deserialization works by using comptime magic behind the scenes. JSON parsing makes use of that (but comes with terrible error messages).

I really got used to Rust's ownership system, where I don't have to think about who's responsible for freeing some data. Zig doesn't have that. Sometimes you pass things by value, other times by reference. But in the latter case are you now responsible to free it? That's communicated by documentation, not by types. In bygge-zig I simply do not deallocate. It's a short-running process, so all memory is gone by the end of it anyway.

Zig wants all code to be warning-free. And so do I — when the code is done. Unused variables however are a part of development and I would prefer if the build still runs through in that case. Different philosophy. At the same time Zig is lazy and unused bits of the code, like functions that never get called, don't even get looked at and don't trigger compile errors.

The biggest plus though is: Compiling Zig code is fast.

What I learned about the Zig build system

It's badly documented, if at all.
The Zig Build System article tells you a bit of how to use it for your Zig project and maybe some dependencies. The std.Build page has a bit of API documentation, but no usage examples. Some parts don't even get API documentation and so you have to guess from the name how it's intended to be used. Because of the recent build system changes a lot of the existing usage doesn't apply anymore. After the recent big changes to the build system it's also incomplete. Some sharp edges are expected.

My current use of the Zig build system in bygge-zig is inefficient, guaranteed to be wrong and very very hacky. After all I'm not using it for what it's mainly build for (building Zig projects). But I got it to work.

I still don't know how the caching system is supposed to work. Right now bygge-zig rebuilds far too much stuff. Every run of zig build grows .zig-cache by some 200 MB. This makes bygge-zig far worse than Cargo. There's still more to learn about this.

What I learned about the Cargo build

It's hella complicated. I learned some of what it does with bygge back then, but I cheated on some things and hardcoded a lot of options. Cargo does so much behind the scenes and getting that replicated in an outside tool is hard. Every crate in your dependency tree relies on a different Cargo feature. Implementing a replacement is thus a lot of trial and error until you get it right. There's a lot of legacy ways to do things and Cargo supports them all. On top of that rustc does a lot of things by itself as well. rustc knows how to traverse the module tree of a crate if you give it the top-level file. Built Rust libraries are identified by their metadata and metadata gets placed into .rmeta files. Metadata (and filenames in case of Cargo) contain hashes of all the input, including the compiler version and other external bits of information. rustc refuses to use built libraries if the metadata of the library and its transitive dependencies is wrong or missing. Sometimes the error message doesn't tell you that.

I feel like I now understand 20% of the build process.

makerust

Thu, 23 Apr 2026 12:29:00 -0400

In 2023 I learned about (GNU) make's .ONESHELL. Put this in your Makefile:

SHELL = python3

.ONESHELL: date
date:
	from datetime import datetime
	print(datetime.now())

and make date works.

; make date
from datetime import datetime
print(datetime.now())
2026-04-23 12:32:27.193732

So for obvious reasons I put together an example so you can write Rust in your Makefile:

.SHELLFLAGS = -c 'echo $$1 > tmp.rs && rustc tmp.rs -o out.bin && ./out.bin' --
.ONESHELL: all
all:
	@fn main() {
		println!("hello make!")
	}

; make all
hello make!

But what if you want to add dependencies? That made me write makerust. I never wrote it up as a blog post, so here it is, now that I updated it to use cargo script (needs Rust nightly, ).

So now you can write:

include makerust.mk

info: dep.xshell dep.anyhow
	@
	use anyhow::Result;
	use xshell::{cmd, Shell};
	
	fn main() -> Result<()> {
		let sh = Shell::new()?;
		let branch = "main";
		let commit_hash = cmd!(sh, "git rev-parse {branch}").read()?;
		println!("makerust running on commit {commit_hash}");
		Ok(())
	}

And then it works:

; make info
makerust running on commit 438da52bae8e0afbcb46d19852cb0def6f21d619

Why?

make is good. Rust is good. In combination it can only get better.

But really ... why?

We don't ask this question around here.

Should I really?

Check with your team first, but I can't see a reason why not.

This is a joke, right?

I don't make jokes on the internet.

THE SOFTWARE IS PROVIDED "AS IS", BUT ALSO I TAKE FULL RESPONSIBILITY FOR THIS UNUTTERABLE HORROR AND YOU SHALL RECITE MY NAME FOR FOREVER ALONGSIDE IT.

Rust reflection and a multi-array list

Wed, 25 Mar 2026 11:00:00 +0100

Rust is experimenting with compile-time reflection in the form of type_info (Tracking issue #146922). It's far from finished, but ~~the other day~~ more than a month ago¹ the pull request for Support ADT types in type info reflection landed. With that PR the types expose enough information about structs to be useful to play around with.

This is what the type info looks like for some types:

u32
Type {
    kind: Int(
        Int {
            bits: 32,
            signed: false,
        },
    ),
    size: Some(
        4,
    ),
}

bool
Type {
    kind: Bool(
        Bool,
    ),
    size: Some(
        1,
    ),
}

struct Foo {
    a: u32,
}
Type {
    kind: Struct(
        Struct {
            generics: [],
            fields: [
                Field {
                    name: "a",
                    ty: TypeId(0x1378bb1c0a0202683eb65e7c11f2e4d7),
                    offset: 0,
                },
            ],
            non_exhaustive: false,
        },
    ),
    size: Some(
        4,
    ),
}

This code snippet shows the structure of several more types.

The full type information is available at compile-time, so in const fn and const { } blocks. The const functions need to be called in a const context though. TypeId::info panics if called at runtime.

Whereas currently anything that wants to act on the shape of types usually works as a proc-macro, the new reflection allows const code to do it without external crates nor external types to cooperate, e.g. by also annotating types using the proc-macro.

Building a multi-array list

Inspired by Zig's MultiArrayList I set out to implement a similar type in Rust.

To understand what a multi-array list even is I'll cite the Zig documentation:

A MultiArrayList stores a list of a struct or tagged union type. Instead of storing a single list of items, MultiArrayList stores separate lists for each field of the struct or lists of tags and bare unions. This allows for memory savings if the struct or union has padding, and also improves cache usage if only some fields or just tags are needed for a computation.

To translate that into Rust code, imagine we have a struct like this:

struct Point {
    x: i32,
    y: i32,
}

And we create a list of points:

let mut list = Vec::new();
list.push(Point { x: 47, y: 23 });
list.push(Point { x: 0, y: 5 });

In memory this will be laid out roughly like:

┌─ list ───────────────────────┐
│    2 │    4 │ 0x5c2c3ef8cae0 │
└───────────────│──────────────┘
                │
                └─┌─ 0x5c2c3ef8cae0 ──────────┐
                  │   47 │   23 │    0 │    5 │
                  └───────────────────────────┘

The vector holds the length (2), the capacity (4²) and a pointer to the actual data allocated on the heap. On the heap all values are laid out flat one after the other. To Rust it's known that the Point struct consists of 2 integers, of which it knows the size (32 bit = 4 bytes), and thus a Point is 8 bytes. Thus we can expect a new Point every 8 bytes³.

For a multi-array list we turn it into this construct instead:

┌─ list ────────────────────────┐
│  0x6000028bc010 │    2 │    2 │
└──│────────────────────────────┘
   │
   └─┌─ 0x6000028bc010 ────────────────┐
     │ 0x6000028bc020 │ 0x6000028bc030 │
     └──│───────────────│──────────────┘
        │               └───────┐
        └─┌─ 0x6000028bc020 ─┐  └─┌─ 0x6000028bc030 ─┐
          │   47 │    0      │    │   23 │    5      │
          └──────────────────┘    └──────────────────┘

Our list has a pointer to a list of pointers (0x6000028bc010), a length (2) and a capacity (also 2⁴). The pointers in that list of pointers point to more lists. Each of these lists contain the values of just one of the fields of Point. Or in Rust code⁵:

struct MultiArrayList {
    elems: *mut *mut u8,
    cap: usize,
    len: usize,
}

This ends up being usable in Rust thanks to reflection⁶:

let mut points = MultiArrayList::<Point>::new();

let point = Point { x: 47, y: 23 };
points.push(point);

let point: Box<Point> = points.pop().unrwap();

It allows to efficiently iterate over just one of the fields:

for x in points.items::<"x", i32>() {
    dbg!(x);
}

Or iterate over the whole list and grab the pieces as we need them:

for point in points.iter() {
  let x = point.get::<"x", i32>();
  let y = point.get::<"y", i32>();
  dbg!(x, y);
}

Now this example doesn't actually show any of the advantages a multi-array list might have. There's no padding we could save.

But I can prove that it does save reduce memory allocation that with the right ~~toppings~~ types:

; cargo run -q --example comparison
Allocating 500 pizzas.
Vec allocated:  16000 bytes
List allocated: 14016 bytes (-12.40%)

The code is available online: https://git.fnordig.de/jer/multi-array-list.
You can also grab it from crates.io:

cargo add multi-array-list

Note though that it's nightly-only, uses several experimental features⁷. It is tested and Miri doesn't complain (yet), but I give absolutely no guarantees about it. I think it might be easy to misuse this to transmute data without resorting to unsafe and there be dragons. I do not plan to actively maintain it, but maybe I use this as a playground to explore reflection a bit more.

Footnotes:

I wrote the code shortly after the pull request landed and started the blog post the same day, but only finished it last night.
Rust allocates a bit more up front to avoid many small allocations when it expects to grow later anyway. ↩
I'm hand-waving things here a little bit. In theory Rust could choose a different layout here and swap around x and y in memory. ↩
I did not implement over-allocation in my code. ↩
Hand-waving even more and skipping the generic part here. ↩
Initially I wrote "safe Rust", but the API currently exposed is unsafe to use. I should mark this unsafe. ↩
Notably adt_const_params, which gives me the wonderful possibility of using "x" in the items API, but breaks my blog's code highlighting. ↩

oelf - Mach-O support for sqlelf

Tue, 02 Jan 2024 13:00:00 +0100

sqlelf lets you explore ELF objects through the power of SQL. It turns any executable into a queryable database. Any? No, just those in ELF format, the standard binary file format on most Unix and Linux systems. But not on macOS. macOS relies on the Mach object file format, short Mach-O and sqlelf doesn't support that. The library it uses for parsing in theory does, but that failed on my machine. It depends on a heavy C++ library and I didn't want to bother figuring out how to build and change that.

I still wanted sqlelf for mach-o binaries. Luckily the hardest part was settled early on: naming.

<@fnordfish@mastodon.social> @jer sollte auf jeden Fall „Ölf“ heißen!
(translation: it should definitely be called „Ölf“)

So oelf exists now. The source code is on GitHub and my fork of sqlelf adds it to sqlelf for easy use.

Install

I have released pre-built versions of oelf, but nothing is upstreamed to sqlelf yet. You can install it from git:

pip install git+https://github.com/badboy/sqlelf@with-macho-support#egg=sqlelf

On my M1 MacBook sqlelf doesn't work out of the box. sqlelf depends on Capstone and the installed library coming with the Python wrapper is x86_64 only, so it won't load

That's fixable. Assuming you installed into a Python venv .venv:
Install capstone from Homebrew, remove the bundled library and link to the global one instead:

brew install capstone
rm .venv/lib/python3.11/site-packages/capstone/lib/libcapstone.dylib
ln -s $(brew --cellar capstone)/5.0.1/lib/libcapstone.5.dylib .venv/lib/python3.11/site-packages/capstone/lib/libcapstone.dylib

Usage

Invoke sqlelf and pass any number of Mach-O binaries as arguments. This gives you an SQLite REPL, or you specify SQL commands with --sql.

For example sqlelf knows about libraries references by the binary:

$ sqlelf /usr/bin/grep --sql 'select * from macho_libs'
┌───────────────┬────────────────────────────┐
│     path      │            lib             │
│ /usr/bin/grep │ self                       │
│ /usr/bin/grep │ /usr/lib/libbz2.1.0.dylib  │
│ /usr/bin/grep │ /usr/lib/liblzma.5.dylib   │
│ /usr/bin/grep │ /usr/lib/libz.1.dylib      │
│ /usr/bin/grep │ /usr/lib/libSystem.B.dylib │
└───────────────┴────────────────────────────┘

None of those /usr/lib/*.dylib actually exists in the filesystem though, because Apple now ships those as a big bundled cache file instead.

I have not yet documented the schema nor is it anywhere near complete. Use .schema to get an overview.

$ sqlelf /usr/bin/grep --sql '.schema macho_headers'
CREATE TABLE macho_headers(
  path,
  magic,
  cputype,
  cpusubtype,
  filetype,
  ncmds,
  sizeofcmds,
  flags,
  reserved
);

Tables are persisted views over the data, so everything is in memory. Most values are the raw values read from the file, so you will have to look up what those values mean.

For example the headers include all sorts of magic numbers and file types as integers:

$ sqlelf /usr/bin/grep --sql 'select * from macho_headers'
┌───────────────┬────────────┬──────────┬────────────┬──────────┬───────┬────────────┬─────────┬──────────┐
│     path      │   magic    │ cputype  │ cpusubtype │ filetype │ ncmds │ sizeofcmds │  flags  │ reserved │
│ /usr/bin/grep │ 4277009103 │ 16777223 │ 3          │ 2        │ 21    │ 1688       │ 2097285 │ 0        │
└───────────────┴────────────┴──────────┴────────────┴──────────┴───────┴────────────┴─────────┴──────────┘

You can slice and dice the data as you wish¹.

$ sqlelf /usr/bin/grep --sql "select name, type, global, n_value from macho_symbols where path = '/usr/bin/grep' limit 3"
┌─────────────────────┬────────┬────────┬────────────┐
│        name         │  type  │ global │  n_value   │
│ radr://5614542      │ N_PBUD │ 0      │ 90260802   │
│ __mh_execute_header │ N_SECT │ 1      │ 4294967296 │
│ _BZ2_bzRead         │ N_UNDF │ 1      │ 0          │
└─────────────────────┴────────┴────────┴────────────┘

Status

I hacked together oelf in a matter of days. I'm using the excellent pyo3 to wrap goblin's functionality into a Python package, built with maturin. It works reliably (yey for great tooling written in and for Rust!), but so far I haven't documented much. oelf itself is a bit inconsistent in how it exposes different data. The sqlelf integration is really simple, thanks to a nice extensible code structure of the project. Now every newly exposed functionality in oelf needs only defining the schema of a table and mapping the retrieved data to its columns.

I have yet to actually use sqlelf myself more to explore binaries and all the data in there. I also have only a bare understanding of the ELF format and much much less of the Mach-O format, I'm just barely good at plugging together existing things.

Some things that might be good to do:

Documentation (of course!)
Add more sections and tables and "translate" magic values
- e.g. load commands (macho_load_commands) pretty-print the Rust object right now, this should be proper data in the column, maybe just JSON to begin with?
Can we extract and parse system libraries from the shared dyld cache?
- Others have built stuff
Can we more lazily fetch data instead of copying into a persisted table once?
Upstream changes or fork it so sqlelf actually works out of the box on any non-Linux machine

Footnotes:

adr://5614542: this is a workaround for a workaround because of a bug in the old classic static linker. ↩

This Week in Glean: Building and Deploying a Rust library on iOS

Mon, 31 Jan 2022 12:40:00 +0100

(“This Week in Glean” is a series of blog posts that the Glean Team at Mozilla is using to try to communicate better about our work. They could be release notes, documentation, hopes, dreams, or whatever: so long as it is inspired by Glean.) All "This Week in Glean" blog posts are listed in the TWiG index (and on the Mozilla Data blog). This article is cross-posted on the Mozilla Data blog.

We ship the Glean SDK for multiple platforms, one of them being iOS applications. Previously I talked about how we got it to build on the Apple ARM machines. Today we will take a closer look at how to bundle a Rust static library into an iOS application.

The Glean SDK project was set up in 2019 and we have evolved its project configuration over time. A lot has changed in Xcode since then, so for this article we're starting with a fresh Xcode project, a fresh Rust library and put it all together step by step.
This is essentially an update to the Building and Deploying a Rust library on iOS article from 2017.

For future readers: This was done using Xcode 13.2.1 and rustc 1.58.1.
One note: I learned iOS development to the extent required to ship Glean iOS. I've never written a full iOS application and lack a lot of experience with Xcode.

The application

The premise of our application is easy:

Show a non-interactive message to the user with data from a Rust library.

Let's get started on that.

The project

We start with a fresh iOS project. Go to File -> New -> Project, then choose the iOS App template, give it a name such as ShippingRust, select where to store it and finally create it. You're greeted with ContentView.swift and the following code:

import SwiftUI

struct ContentView: View {
    var body: some View {
        Text("Hello, world!")
            .padding()
    }
}

You can build and run it now. This will open the Simulator and display "Hello, world!". We'll get back to the Swift application later.

The Rust parts

First we set up the Rust library.

In a terminal navigate to your ShippingRust project directory. In there create a new Rust crate:

cargo new --lib shipping-rust-ffi

We will need a static library, so we change the crate type in the generated shipping-rust-ffi/Cargo.toml. Add the following below the package configuration:

[lib]
crate-type = ["staticlib"]

Let's also turn the project into a Cargo workspace. Create a new top-level Cargo.toml with the content:

[workspace]
members = [
  "shipping-rust-ffi"
]

cargo build in the project directory should work now and create a new static library.

$ ls -l target/debug/libshipping_rust_ffi.a
-rw-r--r-- 2 jer staff 16061952 Jan 28 13:09 target/debug/libshipping_rust_ffi.a

Let's add some code to shipping-rust-ffi/src/lib.rs next. Nothing fancy, a simple function taking some arguments and returning the sum:

use std::os::raw::c_int;

#[no_mangle]
pub extern "C" fn shipping_rust_addition(a: c_int, b: c_int) -> c_int {
    a + b
}

The no_mangle ensures the name lands in the compiled library as-is and the extern "C" makes sure it uses the right ABI.

We now have a Rust library exporting a C-ABI compatible interface. We can now consume this in our iOS application.

The Xcode parts

Before we can use the code we need a bit more setup. Strap in, there's a lot of fiddly manual steps now¹.

We start by linking against the libshipping_rust_ffi.a library. In your Xcode project open your target configuration², go to "Build Phases", then look for "Link Binary with Libraries". Add a new one, in the popup select "Add files" on the bottom left and look for the target/debug/libshipping_rust_ffi.a file. Yes, that's actually for the wrong target. This is just for the name, we'll fix up the path next. Go to "Build Settings" and search for "Library Search Paths". It probably has the path to file in there right now for both Debug and Release builds. Remove that one for Debug, then add a new row by clicking the small + symbol. Select the Any Driverkit matcher. It doesn't matter which matcher you choose or what value you give it, but when we overwrite this manually in the next step I'll assume you chose Any Driverkit. Do the same for the Release configuration.

Once that's done, save your project and go back to your project directory. We will modify the project configuration to have Xcode look for the library based on the target it is building for³. Open up ShippingRust.xcodeproj/project.pbxproj in a text editor, then search for the first line with "LIBRARY_SEARCH_PATHS[sdk=driverkit*]". It should be in a section saying /* Debug */. Remove the LIBRARY_SEARCH_PATHS line and add 3 new ones:

"LIBRARY_SEARCH_PATHS[sdk=iphoneos*][arch=arm64]" = "$(PROJECT_DIR)/target/aarch64-apple-ios/debug";
"LIBRARY_SEARCH_PATHS[sdk=iphonesimulator*][arch=arm64]" = "$(PROJECT_DIR)/target/aarch64-apple-ios-sim/debug";
"LIBRARY_SEARCH_PATHS[sdk=iphonesimulator*][arch=x86_64]" = "$(PROJECT_DIR)/target/x86_64-apple-ios/debug";

Look for the next line with "LIBRARY_SEARCH_PATHS[sdk=driverkit*]", now in a /* Release */ section and replace it with:

"LIBRARY_SEARCH_PATHS[sdk=iphoneos*][arch=arm64]" = "$(PROJECT_DIR)/target/aarch64-apple-ios/release";
"LIBRARY_SEARCH_PATHS[sdk=iphonesimulator*][arch=arm64]" = "$(PROJECT_DIR)/target/aarch64-apple-ios-sim/release";
"LIBRARY_SEARCH_PATHS[sdk=iphonesimulator*][arch=x86_64]" = "$(PROJECT_DIR)/target/x86_64-apple-ios/release";

Save the file and return focus back to Xcode. If you didn't make any typos Xcode should still have your project open. In the settings you will find the library search paths as we've just defined them. If you messed something up Xcode will complain that it cannot read the project file if you try to go to the settings.

Next we need to teach Xcode how to compile Rust code. Once again go to your target settings, selecting the "Build Phases" tab again.

There add a new "Run Script" phase, give it the name "Build Rust library" (double-click the "Run Script" section header), and set the command to:

bash ${PROJECT_DIR}/bin/compile-library.sh shipping-rust-ffi $buildvariant

The compile-library.sh script is going to do the heavy lifting. The first argument is the crate name we want to compile, the second is the build variant to select. This is not yet defined, so let's do it first.

Go to the "Build Settings" tab and click the + button to add a new "User-Defined Setting". Give it the name buildvariant and choose a value based on the build variant: debug for Debug and release for Release.

Now we need the actual script to build the Rust library for the right targets. It's a bit long to write out, but the logic is not too complex: First we select the Cargo profile to use based on our buildvariant (that is whether to pass --release or not), then we set up LIBRARY_PATH if necessary and finally compile the Rust library for the selected target. Xcode passes the architectures to build in ARCHS. It's either x86_64 for simulator builds on Intel Mac hardware or arm64. If it's arm64 it can be either the simulator or an actual hardware target. Those differ, but we can know which is which from what's in LLVM_TARGET_TRIPLE_SUFFIX and select the right Rust target.

Let's put all of that into a compile-library.sh script. Create a new directory bin in your project directory. In there create the file with the following content:

#!/usr/bin/env bash

if [ "$#" -ne 2 ]
then
    echo "Usage (note: only call inside xcode!):"
    echo "compile-library.sh <FFI_TARGET> <buildvariant>"
    exit 1
fi

# what to pass to cargo build -p, e.g. your_lib_ffi
FFI_TARGET=$1
# buildvariant from our xcconfigs
BUILDVARIANT=$2

RELFLAG=
if [[ "$BUILDVARIANT" != "debug" ]]; then
  RELFLAG=--release
fi

set -euvx

if [[ -n "${DEVELOPER_SDK_DIR:-}" ]]; then
  # Assume we're in Xcode, which means we're probably cross-compiling.
  # In this case, we need to add an extra library search path for build scripts and proc-macros,
  # which run on the host instead of the target.
  # (macOS Big Sur does not have linkable libraries in /usr/lib/.)
  export LIBRARY_PATH="${DEVELOPER_SDK_DIR}/MacOSX.sdk/usr/lib:${LIBRARY_PATH:-}"
fi

IS_SIMULATOR=0
if [ "${LLVM_TARGET_TRIPLE_SUFFIX-}" = "-simulator" ]; then
  IS_SIMULATOR=1
fi

for arch in $ARCHS; do
  case "$arch" in
    x86_64)
      if [ $IS_SIMULATOR -eq 0 ]; then
        echo "Building for x86_64, but not a simulator build. What's going on?" >&2
        exit 2
      fi

      # Intel iOS simulator
      export CFLAGS_x86_64_apple_ios="-target x86_64-apple-ios"
      $HOME/.cargo/bin/cargo build -p $FFI_TARGET --lib $RELFLAG --target x86_64-apple-ios
      ;;

    arm64)
      if [ $IS_SIMULATOR -eq 0 ]; then
        # Hardware iOS targets
        $HOME/.cargo/bin/cargo build -p $FFI_TARGET --lib $RELFLAG --target aarch64-apple-ios
      else
        $HOME/.cargo/bin/cargo build -p $FFI_TARGET --lib $RELFLAG --target aarch64-apple-ios-sim
      fi
  esac
done

And now we're done with the setup for compiling the Rust library automatically as part of the Xcode project build.

The code parts

We now have an Xcode project that builds our Rust library and links against it. We now need to use this library!

Swift speaks Objective-C, which is an extension to C, but we need to tell it about the things available. In C land that's done with a header. Let's create a new file, select the "Header File" template and name it FfiBridge.h. This will create a new file with this content:

#ifndef FfiBridge_h
#define FfiBridge_h


#endif /* FfiBridge_h */

Here we need to add the definition of our function. As a reminder this is its definition in Rust:

extern "C" fn shipping_rust_addition(a: c_int, b: c_int) -> c_int;

This translates to the following in C:

int shipping_rust_addition(int a, int b);

Add that line between the #define and #endif lines. Xcode doesn't know about that file yet, so once more into the Build Settings of the target. Search for Objective-C Bridging Header and set it to $(PROJECT_DIR)/ShippingRust/FfiBridge.h. In Build Phases add a new Header Phase. There you add the FfiBridge.h as well.

If it now all compiles we're finally ready to use our Rust library.

Open up ContentView.swift and change the code to call your Rust library:

struct ContentView: View {
    var body: some View {
        Text("Hello, world! \(shipping_rust_addition(30, 1))")
            .padding()
    }
}

We simply interpolate the result of shipping_rust_addition(30, 1) into the string displayed.

Once we compile and run it in the simulator we see we've succeeded at satisfying our premise:

Show a non-interactive message to the user with data from a Rust library.

Compiling for any iOS device should work just as well.

The next steps

This was a lot of setup for calling one simple function. Luckily this is a one-time setup. From here on you can extend your Rust library, define them in the header file and call them from Swift. If you go that route you should really start using cbindgen to generate that header file automatically for you.

This time we looked at building an iOS application directly calling a Rust library. That's not actually how Glean works. The Glean Swift SDK itself wraps the Rust library and exposes a Swift library. In a next blog post I'll showcase how we ship that as a Swift package.

For Glean we're stepping away from manually writing our FFI functions. We're instead migrating our code base to use UniFFI. UniFFI will generate the C API from an API definitions file and also comes with a bit of runtime code to handle conversion between Rust, C and Swift data types for us. We're not there yet for Glean, but you can try it on your own. Read the UniFFI documentation and integrate it into your project. It should be possible to extent the setup we done her to also run the necessary steps for UniFFI. Eventually I'll document how we did it as well.

Footnotes:

And most of these steps are user-interface-dependent and might be different in future Xcode version. :( ↩
Click your project name in the tree view on the left. This gets you to the project configuration (backed by the ShippingRust.xcodeproj/project.pbxproj file). You should then see the Targets, including your ShippingRust target and probably ShippingRustTests as well. We need the former. ↩
Previously we would have built a universal library containing the library of multiple targets. That doesn't work anymore now that arm64 can stand for both the simulator and hardware targets. Thus linking to the individual libraries is the way to go, as the now-deprecated cargo-lipo also points out. ↩

This Week in Glean: Crashes & a buggy Glean

Mon, 01 Nov 2021 15:00:00 +0100

In September I finally landed work to ship Glean through GeckoView. Contrary to what that post said Fenix did not actually use Glean coming from GeckoView immediately due to another bug that took us another few days to land. Shortly after that was shipped in a Fenix Nightly release we received a crash report (bug 1733757) pointing to code that we haven't touched in a long time. And yet the change of switching from a standalone Glean library to shipping Glean in GeckoView uncovered a crashing bug, that quickly rose to be the top crasher for Fenix for more than a week.

When I picked up that bug after the weekend I was still thinking that this would be just a bug, which we can identify & fix and then get into the next Fenix release. But in data land nothing is ever just a bug.

As I don't own an Android device myself I was initially restricted to the Android emulator, but I was unsuccessful in triggering that bug in my simple use of Fenix or in any tests I wrote. At some point I went as far as leveraging physical devices in the Firebase test lab, still with no success of hitting the bug. Later that week I picked up an Android test device, hoping to find easy steps to reproduce the crash. Thankfully we also have quality assurance people running tests and simply using the browser and reporting back steps to reproduce bugs and crashes. And so that's what one of them, Delia, did: Providing simple steps to reproduce the crash. Simple here meant: open a website and leave the phone untouched for roughly 8-9 minutes. With those steps I was able to reproduce the crash in both the emulator and my test device as well.

As I was waiting for the crash to occur I started reading the code again. From the crash report I already knew the exact place where the code panics and thus crashes the application, I just didn't know how it got there. While reading the code around the panic and the knowledge that it takes several minutes to get to the panic I finally stumbled upon a clue: The map implementation that was used internally has a maximum capacity of elements it can store. It even documents that it will panic¹!

/// The maximum capacity of a [`HandleMap`]. Attempting to instantiate one with
/// a larger capacity will cause a panic.
///
/// [...]
pub const MAX_CAPACITY: usize = (1 << 15) - 1;

I now knew that at some point Glean exhausts the map capacity, slowly but eventually leading to a panic². Most of Glean is set up to not require dynamically allocating new things other than the data itself, but we never store the data itself within such a map. So where are we using a map and dynamically add new entries to it and potentially forget to remove them after use?

Luckily the stack trace from those crash reports gave me another clue: All crashes were caused by labeled counters. Some instrument-the-code-paths-and-recompile cycles later I was able to pinpoint it to the exact metric that was causing the crash.

With all this information collected I was able to determine that while the panic was happening in a library we use, the problem was that Glean had the wrong assumptions about how the use of labeled counters in Kotlin maps to objects in a Rust map. On every subscript access to a labeled counter (myCounter["label"]) Glean would create a new object, store it in the map and return a handle to this object. The solution was to avoid creating new entries in that map on every access from a foreign language and instead cache created objects by a unique identifier and hand out already-created objects when re-accessed. This fix was implemented in a few lines, accompanied by an extensive commit message as well as tests in all of the 3 major foreign-language Glean SDKs.

That fix was released in Glean v42.0.1, and shortly after in Fenix Nightly and Beta.

The bug was fixed and the crash prevented, but that wasn't the end of it. The code leading to the bug has been in Glean since at least March 2020 and yet we only started seeing crashes with the GeckoView change. What was happening before? Did we miss these crashes for more than a year or were there simply no crashes?

To get a satisfying answer to these questions I had to retrace the issue in older Glean versions. I took older versions of Glean & Fenix from just before the GeckoView-Glean changes, added logging and built myself a Fenix to run in the emulator. And as quickly as that³ logcat spit out the following lines repeatedly:

E  ffi_support::error: Caught a panic calling rust code: "called `Result::unwrap()` on an `Err` value: \"PoisonError { inner: .. }\""
E  glean_ffi::handlemap_ext: Glean failed (ErrorCode(-1)): called `Result::unwrap()` on an `Err` value: "PoisonError { inner: .. }"

No crashes! It panics, but it doesn't crash in old Fenix versions. So why does it crash in new versions? It's a combination of things.

First Glean data recording happens in a thread to avoid blocking the main thread. Right now that thread is handled on the Kotlin side. A panic in the Rust code base will bubble up the stack and crash the thread, thus poisoning the internal lock of the map, but it will not abort the whole application by itself. Subsequent calls will run on a new thread, handled by the Kotlin side. When using a labeled counter again it will try to acquire the lock, detect that it is poisoned and panic once more. That panic is caught by the FFI layer of Glean and turned into a log message.

Second the standalone Glean library is compiled with panic=unwind, the default in Rust, which unwinds the stack on panics. If not caught the runtime will abort the current thread, writing a panic message to the error output. ffi-support however catches it, logs it and returns without crashing or aborting. Gecko on the other hand sets panic=abort. In this mode a panic will immediately terminate the current process (after writing the crash message to the error output), without ever trying to unwind the stack, giving no chance for the support library to catch it. The Gecko crash reporter is able to catch those hard aborts and send them as crash reports. As Glean is now part of the overall Gecko build all of Gecko's build flags will transitively apply to Glean and its dependencies, too. So when Glean is shipped as part of GeckoView it runs in panic=abort mode, leading to internal panics aborting the whole application.

That behavior by itself is fine: Glean should only panic in exceptional cases and we'd like to know about them. It's good to know that an application could continue running without Glean working correctly; we won't be able to record and send telemetry data, but at least we're not interrupting someone using the application. However unless we engineers run into those bugs and see the log we will not notice them and thus can't fix them. So ultimately this change in (crashing) behavior is acceptable (and wanted) going forward.

After fixing the initial bug and being able to answer why it only started crashing recently my work was still not done. We were likely not recording data in exceptional cases for quite some time, which is not acceptable for a telemetry library. I had to explore our data, estimate how many metrics for how many clients were affected, inform relevant stakeholders and plan further mitigations. But that part of the investigation is a story for another time.

This bug investigation had help from a lot of people. Thanks to Mike Droettboom for coordination, Marissa Gorlick for pushing me to evaluate the impact on data & reaching the right people, Travis Long for help with the Glean code & speedy reviews, Alessio Placitelli for reviews on the Gecko side, Delia Pop for the initial steps to reproduce the bug & Christian Sadilek for help on the Android Components & Fenix side.

Footnotes:

Except that it panics at a slightly different place than the explicit check in the code base would suggest. ↩
That meant I could also tune how quickly it crashed. A smaller maximum capacity means its reached more quickly, reducing my bug reproduction time significantly. ↩ ↩2
No, not after 9 minutes, but in just under 3 minutes after tuning the maximum map capacity, see ². ↩

This Week in Glean: Glean & GeckoView

Fri, 17 Sep 2021 13:00:00 +0200

This is a followup post to Shipping Glean with GeckoView.

It landed!

It took us several more weeks to put everything into place, but we're finally shipping the Rust parts of the Glean Android SDK with GeckoView and consume that in Android Components and Fenix. And it still all works, collects data and is sending pings! Additionally this results in a slightly smaller APK as well.

This unblocks further work now. Currently Gecko simply stubs out all calls to Glean when compiled for Android, but we will enable recording Glean metrics within Gecko and exposing them in pings sent from Fenix. We will also start work on moving other Rust components into mozilla-central in order for them to use the Rust API of Glean directly. Changing how we deliver the Rust code also made testing Glean changes across these different components a bit more challenging, so I want to invest some time to make that easier again.

This Week in Glean: Shipping Glean with GeckoView

Mon, 26 Jul 2021 12:00:00 +0200

Glean SDK

The Glean SDK is Mozilla's telemetry library, used in most mobile products and now for Firefox Desktop as well. By now it has grown to a sizable code base with a lot of functionality beyond just storing some metric data. Since its first release as a Rust crate in 2019 we managed to move more and more logic from the language SDKs (previously also known as "language bindings") into the core Rust crate. This allows us to maintain business logic only once and can easily share that across different implementations and platforms. The Rust core is shipped precompiled for multiple target platforms, with the language SDK distributed through the respective package manager.

I talked about how this all works in more detail last year, this year and blogged about it in a previous TWiG.

GeckoView

GeckoView is Mozilla's alternative implementation for WebViews on Android, based on Gecko, the web engine that also powers Firefox Desktop. It is used as the engine behind Firefox for Android (also called Fenix). The visible parts of what makes up Firefox for Android is written in Kotlin, but it all delegates to the underlying Gecko engine, written in a combination of C++, Rust & JavaScript.

The GeckoView code resides in the mozilla-central repository, next to all the other Gecko code. From there releases are pushed to Mozilla's own Maven repository.

One Glean too many

Initially Firefox for Android was the only user of the Glean SDK. Up until today it consumes Glean through its release as part of Android Components, a collection of libraries to build browser-like applications.

But the Glean SDK is also available outside of Android Components, as its own package. And additionally it's available for other languages and platforms too, including a Rust crate. Over the past year we've been busy getting Gecko to use Glean through the Rust crate to build its own telemetry on top.

With the Glean SDK used in all these applications we're in a difficult position: There's a Glean in Firefox for Android that's reporting data. Firefox for Android is using Gecko to render the web. And Gecko is starting to use Glean to report data.

That's one Glean too many if we want coherent data from the full application.

Shipping it all together, take one

Of course we knew about this scenario for a long time. It's been one of the goals of Project FOG to transparently collect data from Gecko and the embedding application!

We set out to find a solution so that we can connect both sides and have only one Glean be responsible for the data collection & sending.

We started with more detailed planning all the way back in August of last year and agreed on a design in October. Due to changed priorities & availability of people we didn't get into the implementation phase until earlier this year.

By February I had a first rough prototype in place. When Gecko was shipped as part of GeckoView it would automatically look up the Glean library that is shipped as a dynamic library with the Android application. All function calls to record data from within Gecko would thus ultimately land in the Glean instance that is controlled by Fenix. Glean and the abstraction layer within Gecko would do the heavy work, but users of the Glean API would notice no difference, except their data would now show up in pings sent from Fenix.

This integration was brittle. It required finding the right dynamic library, looking up symbols at runtime as well as reimplementing all metric types to switch to the FFI API in a GeckoView build. We abandoned this approach and started looking for a better one.

Shipping it all together, take two

After the first failed approach the issue was acknowledged by other teams, including the GeckoView and Android teams.

Glean is not the only Rust project shipped for mobile, the application-services team is also shipping components written in Rust. They bundle all components into a single library, dubbed the megazord. This reduces its size (dependencies & the Rust standard library are only linked once) and simplifies shipping, because there's only one library to ship. We always talked about pulling in Glean as well into such a megazord, but ultimately didn't do it (except for iOS builds).

With that in mind we decided it's now the time to design a solution, so that eventually we can bundle multiple Rust components in a single build. We came up with the following plan:

The Glean Kotlin SDK will be split into 2 packages: a glean-native package, that only exists to ship the compiled Rust library, and a glean package, that contains the Kotlin code and has a dependency on glean-native.
The GeckoView-provided libxul library (that's "Gecko") will bundle the Glean Rust library and export the C-compatible FFI symbols, that are used by the Glean Kotlin SDK to call into Glean core.
The GeckoView Kotlin package will then use Gradle capabilities to replace the glean-native package with itself (this is actually handle by the Glean Gradle plugin).

Consumers such as Fenix will depend on both GeckoView and Glean. At build time the Glean Gradle plugin will detect this and will ensure the glean-native package, and thus the Glean library, is not part of the build. Instead it assumes libxul from GeckoView will take that role.

This has some advantages. First off everything is compiled together into one big library. Rust code gets linked together and even Rust consumers within Gecko can directly use the Glean Rust API. Next up we can ensure that the version of the Glean core library matches the Glean Kotlin package used by the final application. It is important that the code matches, otherwise calling native functions could lead to memory or safety issues.

Glean is running ahead here, paving the way for more components to be shipped the same way. Eventually the experimentation SDK called Nimbus and other application-services components will start using the Rust API of Glean. This will require compiling Glean alongside them and that's the exact case that is handled in mozilla-central for GeckoView then.

Now the unfortunate truth is: these changes have not landed yet. It's been implemented for both the Glean SDK and mozilla-central, but also requires changes for the build system of mozilla-central. Initially that looked like simple changes to adopt the new bundling, but it turned into bigger changes across the board. Some of the infrastructure used to build and test Android code from mozilla-central was untouched for years and thus is very outdated and not easy to change. With everything else going on for Firefox it's been a slow process to update the infrastructure, prepare the remaining changes and finally getting this landed.

But we're close now!

Big thanks to Agi for connecting the right people, driving the initial design and helping me with the GeckoView changes. He also took on the challenge of changing the build system. And also thanks to chutten for his reviews and input. He's driving the FOG work forward and thus really really needs us to ship GeckoView support.

This Week in Glean: rustc, iOS and an M1

Fri, 16 Apr 2021 15:00:00 +0100

Back in February I got an M1 MacBook. That's Apple's new ARM-based hardware.

I got it with the explicit task to ensure that we are able to develop and build Glean on it. We maintain a Swift language binding, targeting iOS, and that one is used in Firefox iOS. Eventually these iOS developers will also have M1-based machines and want to test their code, thus Glean needs to work.

Here's what we need to get to work:

Compile the Rust portions of Glean natively on an M1 machine
Build & test the Kotlin & Swift language bindings on an M1 machine, even if non-native (e.g. Rosetta 2 emulation for x86_64)
Build & test the Swift language bindings natively and in the iPhone simulator on an M1 machine
Stretch goal: Get iOS projects using Glean running as well

Rust on an M1

Work on getting Rust compiled on M1 hardware started last year in June already, with the availability of the first developer kits. See Rust issue 73908 for all the work and details. First and foremost this required a new target: aarch64-apple-darwin. This landed in August and was promoted to Tier 2¹ with the December release of Rust 1.49.0.

By the time I got my MacBook compiling Rust code on it was as easy as on an Intel MacBook. Developers on Intel MacBooks can cross-compile just as easily:

rustup target add aarch64-apple-darwin
cargo build --target aarch64-apple-darwin

Glean Python & Kotlin on an M1

Glean Python just ... worked. We use cffi to load the native library into Python. It gained aarch64² macOS support in v14.4.1. My colleague glandium later contributed support code so we build release wheels for that target too. So it's both possible to develop & test Glean Python, as well as use it as a dependency without having a full Rust development environment around.

Glean Android is not that straight forward. Some of our transitive dependencies are based on years-old pre-built binaries of SQLite and of course there's not much support behind updating those Java libraries. It's possible. A friend managed to compile and run that library on an M1. But for Glean development we simply recommend relying on Rosetta 2 (the x86_64 compatibility layer) for now. It's as easy as:

arch -x86_64 $SHELL
make build-kotlin

At least if you have Java set up correctly... The default Android emulator isn't usable on M1 hardware yet, but Google is working on a compatible one: Android M1 emulator preview. It's usable enough for some testing, but for that part I most often switch back to my Linux Desktop (that has the additional CPU power on top).

Glean iOS on an M1

Now we're getting to the interesting part: Native iOS development on an M1. Obviously for Apple this is a priority: Their new machines should become the main machine people do iOS development on. Thus Xcode gained aarch64 support in version 12 long before the hardware was available. That caused quite some issues with existing tooling, such as the dependency manager Carthage. Here's the issue:

When compiling for iOS hardware you would pick a target named aarch64-apple-ios, because ... iPhones and iPads are ARM-based since forever.
When compiling for the iOS simulator you would pick a target named x86_64-apple-ios, because conveniently the simulator uses the host's CPU (that's what makes it fast)

So when the compiler saw x86_64 and iOS it knew "Ah, simulator target" and when it saw aarch64 and ios it knew "Ah, hardware". And everyone went with this, Xcode happily built both targets and, if asked to, was able to bundle them into one package.

With the introduction of Apple Silicion³ the iOS simulator run on these machines would also be aarch64⁴, and also contain ios, but not be for the iOS hardware.

Now Xcode and the compiler will get confused what to put where when building on M1 hardware for both iOS hardware and the host architecture.

So the compiler toolchain gained knowledge of a new thing: arm64-apple-ios14.0-simulator, explicitly marking the simulator target. The compiler knows from where to pick the libraries and other SDK files when using that target. You still can't put code compiled for arm64-apple-ios and arm64-apple-ios14.0-simulator into the same universal binary⁵, because you can have each architecture only once (the arm64 part in there). That's what Carthage and others stumbled over.

Again Apple prepared for that and for a long time they have wanted you to use XCFramework bundles⁶. Carthage just didn't used to support that. The 0.37.0 release fixed that.

That still leaves Rust behind, as it doesn't know the new -simulator target. But as always the Rust community is ahead of the game and deg4uss3r started adding a new target in Rust PR #81966. He got half way there when I jumped in to push it over the finish line. How these targets work and how LLVM picks the right things to put into the compiled artifacts is severly underdocumented, so I had to go the trial-and-error route in combination with looking at LLVM source code to find the missing pieces. Turns out: the 14.0 in arm64-apple-ios14.0-simulator is actually important.

With the last missing piece in place, the new Rust target landed in February and is available in Nightly. Contrary to the main aarch64-apple-darwin or aarch64-apple-ios target, the simulator target is not Tier 2 yet and thus no prebuilt support is available. rustup target add aarch64-apple-darwin does not work right now. I am now in discussions to promote it to Tier 2, but it's currently blocked by the RFC: Target Tier Policy.

It works on nightly however and in combination with another cargo capability I'm able to build libraries for the M1 iOS simulator:

cargo +nightly build -Z build-std --target aarch64-apple-ios-sim

For now Glean iOS development on an M1 is possible, but requires Nightly. Goal achieved, I can actually work with this!

In a future blog post I want to explain in more detail how to teach Xcode about all the different targets it should build native code for.

All The Other Projects

This was marked a stretch goal for a reason. This involves all the other teams with Rust code and the iOS teams too. We're not there yet and there's currently no explicit priority to make development of Firefox iOS on M1 hardware possible. But when it comes to it, Glean will be ready for it and the team can assist others to get it over the finish line.

Want to hear more about Glean and our cross-platform Rust development? Come to next week's Rust Linz meetup, where I will be talking about this.

Footnotes:

See Platform Support for what the Tiers means. ↩
The other name for that target. ↩
"Apple Silicon" is yet another name for what is essentially the same as "M1" or "macOS aarch64" ↩
Or arm64 for that matter. Yes, yet another name for the same thing. ↩
"Universal Binaries" have existed for a long time now and allow for one binary to include the compiled artifacts for multiple targets. It's how there's only one Firefox for Mac download which runs natively on either Mac platform. ↩
Yup, the main documentation they link to is a WWDC 2019 talk recording video. ↩

This Week in Glean: Glean in 2021

Fri, 18 Dec 2020 15:00:00 +0100

All "This Week in Glean" blog posts are listed in the TWiG index (and on the Mozilla Data blog). This article is cross-posted on the Mozilla Data blog.

A year ago the Glean project was different. We had just released Glean v22.1.0 and Fenix (aka Firefox for Android aka Firefox Daylight) was not released yet, Project FOG was just an idea for the year to come.

2020 changed that, but 2020 changed a lot. What didn't change was my main assignment: I kept working all throughout the year on the Glean SDK, fixing bugs, expanding its capabilities, enabling more platforms and integrating it into more products. Of course this was only possible because the whole team did that as well.

In September I took over the tech lead role for the SDK from Alessio. One part of this role includes thinking bigger and sketching out the future for the Glean SDK.

Let's look at this future and what ideas we have for 2021.

(One note: right now these are ideas more than a plan. It neither includes a timeline nor does it include all the other things maintenance includes.)

The vision

The Glean SDK is a fully self-servable telemetry SDK, usable across different platforms. It enables product owners and engineers to instrument their products and rely on their data collection, while following Mozilla policies & privacy standards.

The ideas

In the past weeks I started a list of things I want the Glean SDK to do next. This is a short and incomplete list of not-yet-proposed or accepted ideas. For 2021 we will need to fit this in with the larger Glean project, including plans and ideas for the pipeline and tooling. Oh, and I need to talk with my team to actually decide on the plan and allocate who does what when.

Metric types

When we set out to revamp our telemetry system we build it with the idea of offering higher-level metric types, that give more meaning to individual data points, allowing more correct data collection and better aggregation, analysis and visualisation of this data. We're not there yet. We currently support more than a dozen metric types across all platforms equally, with the same user-friendly APIs in all supported languages. We also know that this still does not cover all intended use cases that, for example, Firefox Desktop wants. In 2021 we will probably need to work on a few more types, better ergonomics and especially documentation on when which metric type is appropriate.

Revamped testing APIs

Engineers should instrument their code where appropriate and use the collected data to analysis behavior and performance in the wild. The Glean SDK ensures their data is reliably collected and sent to our pipeline. But we cannot ensure that the way the data is collected is correct or whether the metric even makes sense. That's why we encourage that each metric recording is accompanied by tests to ensure data is collected under the right circumstances and that its the right data under the given test case. Only that way folks will be able to verify and analyse the incoming data later and rely on their results.

The available testing APIs in the Glean SDK provide a bare minimum. For each metric type one can check which data it currently holds. That's probably easy to validate for simpler metrics such as strings, booleans and counters, but as soon as you have more complex one like any of the distributions or a timespan it gets a bit more complex. Additionally we offer a way to check if any errors occured during recording, which are usually also reported in the data. But here again developers need to know which errors can happen under what circumstances for which metric type.

And at last we want developers to reach for custom pings when the default pings are not sufficient. Testing these is currently near impossible.

Testing these different usages of the SDK can and should be improved. We already have the first proposals and bugs for this:

Custom Ping Unit Tests proposes a simple callback-like API to validate data in custom pings
Testing parameter in Glean metric definitions proposes to make existing tests/testing documentation for metrics more visible
Improvements to Glinter, the Glean linter (e.g. opinions on naming of things)

(Note: as of writing these document might be inaccessible to the wider public, but they will be made public once we gather wider feedback)

For sure we will have more ideas on this in 2021.

UniFFI - generate all the code

The Glean SDK is cross-platform by shipping a core library written in Rust with language bindings on top that connect it with the target platform. This has the advantage that most of the logic is written once and works across all the targets we have. But this also has the downside that each language binding sort of duplicates the user-visible API in their target language again. Currently all metric type implementations need to happen in Rust first, followed by implementations in the language bindings.

All of the implementation work today is mostly manual, but usually follows the same approach. The language binding implementations should not hold additional state, but currently some still do.

Implementation of new metric types and fixing bugs or improving the recording APIs of existing ones results in a lot of busy work replicating the same code patterns in all the languages (of which we now have have 7: Kotlin, Swift, Python, C#, Rust, JavaScript and C++). If we also come up with new metric types next year this becomes worse.

A while ago some of my colleagues started the UniFFI project, a multi-language bindings generator for Rust. For a couple of reasons the Glean SDK cannot rely on that yet.

In 2021 I want us to work towards the goal of using UniFFI (or something like it) to reduce the manual work required on each language binding. This should reduce our general workload supporting several languages and help us avoid accidental bugs on implementing (and maintaining) metric types.

This is a rather short list of things that only focus on the SDK. Let's see what we get tackled in 2021.

End of 2020

The This Week in Glean series is going into winter hiatus until the second week of January 2021. Thanks to everyone who contributed to 25 This Week in Glean blog posts in 2020:

Alessio, Travis, Mike, Chutten, Bea, Raphael, Anthony, Will & Daosheng

This Week in Glean: FOG progress report

Tue, 06 Oct 2020 16:00:00 +0200

All "This Week in Glean" blog posts are listed in the TWiG index (and on the Mozilla Data blog). This article is cross-posted on the Mozilla Data blog.

About a year ago chutten started the "This Week in Glean" series with an initial blog post about Glean on Desktop. Back then we were just getting started to bring Glean to Firefox Desktop. No code had been written for Firefox Desktop, no proposals had been written to discuss how we even do it.

Now, 12 months later, after four completed milestones, a dozen or so proposal and quite a bit of code, the Project Firefox on Glean (FOG) is finally getting into a stage where we can actually use and test it. It's not ready for prime time yet, but FOG is enabled in Firefox Nightly already.

Over the past 4 weeks I've been on and off working on building out our support for a C++ and a JavaScript API. Soon Firefox engineers will be able to instrument their code using Glean. In C++ this will look like:

mozilla::glean::test_only::count_something.Add(42);

JavaScript developers will use it like this:

Glean.test_only.count_something.add(32);

My work is done in bug 1646165. It's still under review and needs some fine-tuning before that, but I hope to land it by next week. We will then gather some data from Nightly and validate that it works as expected (bug 1651110), before we let it ride the trains to land in stable (bug 1651111).

Our work won't end there. With my initial API work landing we can start supporting all metric types, we still need to figure out some details of how FOG will handle IPC, and then there's the whole process of convincing other people to use the new system.

2020 will be the year of Glean on the Desktop.

This Week in Glean: Leveraging Rust to build cross-platform mobile libraries

Tue, 01 Sep 2020 15:00:00 +0200

All "This Week in Glean" blog posts are listed in the TWiG index (and on the Mozilla Data blog). This article is cross-posted on the Mozilla Data blog.

A couple of weeks ago I gave a talk titled "Leveraging Rust to build cross-platform mobile libraries". You can find my slides as a PDF. It was part of the Rusty Days Webference, an online conference that was initially planned to happen in Poland, but had to move online. Definitely check out the other talks.

One thing I wanted to achieve with that talk is putting that knowledge out there. While multiple teams at Mozilla are already building cross-platform libraries, with a focus on mobile integration, the available material and documentation is lacking. I'd like to see better guides online, and I probably have to start with what we have done. But that should also be encouragement for those out there doing similar things to blog, tweet & speak about it.

Who else is using Rust to build cross-platform libraries, targetting mobile?

I'd like to hear about it. Find me on Twitter (@badboy_) or drop me an email.

The Glean SDK

I won't reiterate the full talk (go watch it, really!), so this is just a brief overview of the Glean SDK itself.

The Glean SDK is our approach to build a modern Telemetry library, used in Mozilla's mobile products and soon in Firefox on Desktop as well.

The SDK consists of multiple components, spanning multiple programming languages for different implementations. All of the Glean SDK lives in the GitHub repository at mozilla/glean. This is a rough diagram of the Glean SDK tech stack:

On the very bottom we have glean-core, a pure Rust library that is the heart of the SDK. It's responsible for controlling the database, storing data and handling additional logic (e.g. assembling pings, clearing data, ..). As it is pure Rust we can rely on all Rust tooling for its development. We can write tests that cargo test picks up. We can generate the full API documentation thanks to rustdoc and we rely on clippy to tell us when our code is suboptimal. Working on glean-core should be possible for everyone that knows some Rust.

On top of that sits glean-ffi. This is the FFI layer connecting glean-core with everything else. While glean-core is pure Rust, it doesn't actually provide the nice API we intend for users of Glean. That one is later implemented on top of it all. glean-ffi doesn't contain much logic. It's a translation between the proper Rust API of glean-core and C-compatible functions exposed into the dynamic library. In it we rely on the excellent ffi-support crate. ffi-support knows how to translate between Rust and C types, offers a nice (and safer) abstraction for C strings. glean-ffi holds some state: the instantiated global Glean object and metric objects. We don't need to pass pointers back and forth. Instead we use opaque handles that index into a map held inside the FFI crate.

The top layer of the Glean SDK are the different language implementations. Language implementations expose a nice ergonomic API to initialize Glean and record metrics in the respective language. Additionally each implementation handles some special cases for the platform they are running on, like gathering application and platform data or hooking into system events. The nice API calls into the Glean SDK using the exposed FFI functions of glean-ffi. Unfortunately at the moment different language implementations carry different amounts of actual logic in them. Sometimes metric implementations require this (e.g. we rely on the clock source of Kotlin for timing metrics), in other parts we just didn't move the logic out of the implementations yet. We're actively working on moving logic into the Rust part where we can and might eventually use some code generation to unify the other parts. uniffi is a current experiment for a multi-language bindings generator for Rust we might end up using.

Build your project

Fri, 19 Jun 2020 13:46:00 +0200

What if you could build a Rust project without Cargo? It's not an actual problem I encountered. It's still a problem I looked into.

Cargo is the Rust package manager. Cargo downloads your Rust package’s dependencies, compiles your packages, makes distributable packages, and uploads them to crates.io, the Rust community’s package registry.

(from the Cargo Book)

That's a lot of work this tool does and it's very good at doing it.

I wanted to understand just the bit where it takes your Rust code and builds that into a binary that you can execute and run on your machine. I didn't really want to read all of Cargo, so instead I turned to what Cargo itself tells me. So I took the output of cargo build --verbose and turned that into simple build instructions.

What does it look like?

$ cargo build --verbose
[...]
   Compiling smallvec v1.4.0
   Compiling matches v0.1.8
   Compiling libc v0.2.71
     Running `rustc --crate-name smallvec --edition=2018 /Users/jer/.cargo/registry/src/github.com-1ecc6299db9ec823/smallvec-1.4.0/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C debuginfo=2 -C metadata=d85954c5ff803feb -C extra-filename=-d85954c5ff803feb --out-dir /Users/jer/projects/rust/bygge/target/debug/deps -L dependency=/Users/jer/projects/rust/bygge/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name libc /Users/jer/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.71/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=bbaef975a82234c8 -C extra-filename=-bbaef975a82234c8 --out-dir /Users/jer/projects/rust/bygge/target/debug/deps -L dependency=/Users/jer/projects/rust/bygge/target/debug/deps --cap-lints allow --cfg freebsd11 --cfg libc_priv_mod_use --cfg libc_union --cfg libc_const_size_of --cfg libc_align --cfg libc_core_cvoid --cfg libc_packedN`
[...]
     Running `rustc --crate-name bygge --edition=2018 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=c1c12e33a54c5b24 -C extra-filename=-c1c12e33a54c5b24 --out-dir /Users/jer/projects/rust/bygge/target/debug/deps -C incremental=/Users/jer/projects/rust/bygge/target/debug/incremental -L dependency=/Users/jer/projects/rust/bygge/target/debug/deps --extern cargo_lock=/Users/jer/projects/rust/bygge/target/debug/deps/libcargo_lock-43ce509885ad1aff.rlib --extern cargo_toml=/Users/jer/projects/rust/bygge/target/debug/deps/libcargo_toml-5b6fff58e6c9a546.rlib --extern dirs=/Users/jer/projects/rust/bygge/target/debug/deps/libdirs-4eb870f2ccf6ed43.rlib --extern petgraph=/Users/jer/projects/rust/bygge/target/debug/deps/libpetgraph-7cb995dcfae4c1f7.rlib --extern pico_args=/Users/jer/projects/rust/bygge/target/debug/deps/libpico_args-03badf8072a773d9.rlib`
    Finished dev [unoptimized + debuginfo] target(s) in 30.82s

I removed a bunch of the output, but it all boils down to the same. You can re-run these commands and you should end up with the same binary as the one Cargo puts into target/debug/.

You could put them all in a Makefile and use that to re-run them, but you would have gained nothing. Changing your own code or adding, removing or updating dependencies won't be picked up unless you carefully specify all these dependencies in the Makefile and update them when something changes.

We can automate that part.

Lately (again) I've been also reading about other build systems, such as Ninja and was intrigued to rebuild that. I haven't done that yet, but I decided to generate Ninja configuration to see how it behaves.

Let's combine these two things.

The basic manual version

Let's work from a basic binary crate, with no dependencies and a single file.

$ cargo new --bin hello-world
     Created binary (application) `hello-world` package
$ cat
fn main() {
    println!("Hello, world!");
}

Building just that is easy:

$ rustc src/main.rs -o hello-world
$ ./hello-world
Hello, world!

If we rerun rustc it will re-build the code and generate a new file hello-world (that should be the same as the one from the first invocation). That's a bit of extra work; if nothing changed we shouldn't need to rebuild. Let's create a Ninja configuration file build.ninja:

# The build rule to invoke rustc.
# $in and $out are variables provided by ninja.
rule rustc
  command = rustc $in -o $out
  description = RUSTC $out

# Specifying dependencies:
# For `hello-world` to be build we invoke the above `rustc` rule with `src/main.rs` as $in
build hello-world: rustc src/main.rs

With Ninja installed we can build and run this:

$ ninja
[1/1] RUSTC hello-world
$ ./hello-world
Hello, world!

Re-running Ninja won't build it again:

$ ninja
ninja: no work to do.

Ninja knows that nothing changed and thus skips extra work. If we modify the source it rebuilds:

$ echo >> src/main.rs
$ ninja
[1/1] RUSTC hello-world
$ ./hello-world
Hello, world!

We don't gain anything if we keep continuing writing a Ninja build configuration from hand. Especially not once we add more files and some external crates.

Bygge - build your project

Instead of writing out the whole Ninja file, or coming up with some rules for Make that expand to the right files, we can generate the build configuration once and then not touch it again until something changes. That's exactly what Ninja was designed for:

it is designed to have its input files generated by a higher-level build system, [...]

(from the Ninja Website)

After some more exploration with Cargo and the command lines it produces and the files it creates I managed to more-or-less auto-generate a Ninja configuration for a crate.

I've published this experiment as bygge, the crate to build your crates (and itself).

What it does

bygge create generates a Ninja build configuration (in build.ninja by default), listing all the targets a binary crate depends on, including all crate dependencies. ninja can then take this configuration and assemble the final binary. The result should be about the same as an invocation of cargo build.

What it doesn't

bygge is and never will be an alternative to Cargo. Cargo is a full-fledged build system, aware of different build targets, allowing to enable features per dependency, easily cross-compile to different targets and run the built programs as well as tests and generate documentation.

bygge ... builds.

Features

Builds cargo dependencies as listed in a project's Cargo.toml
Can build only crates with a single binary target
Runs on (at least) macOS and Linux
No support for build.rs files
No support for linking non-Rust libraries
Hard-coded Cargo features for its dependencies
Uses cargo fetch to download your project's dependencies

Building bygge

Bygge is able to create a Ninja build configuration to build itself. But first you need a compiled bygge. It comes with a pre-generated configuration for that, that only works on my machine unless you change the paths to the local checkouts of the dependencies.

$ ninja -f manual.ninja
[29/29] RUSTC build/bygge

It builds a build/bygge file that is able to create the build configuration and run Ninja:

$ build/bygge create
==> Creating build file: build.ninja
==> Package: bygge
$ build/bygge build
[29/29] RUSTC build/bygge
$ build/bygge -V
bygge v0.1.0

You can also build it using cargo build and use cargo run create (or target/debug/bygge create) to create the build.ninja file.

And there you have it: a 300-line Ninja build configuration to build bygge.

This Week in Glean: Bytes in Memory (on Android)

Mon, 04 May 2020 15:30:00 +0200

Last week's blog post: This Week in Glean: Glean for Python on Windows by Mike Droettboom. All "This Week in Glean" blog posts are listed in the TWiG index (and on the Mozilla Data blog). This article is cross-posted on the Mozilla Data blog.

With the Glean SDK we follow in the footsteps of other teams to build a cross-platform library to be used in both mobile and desktop applications alike. In this blog post we're taking a look at how we transport some rich data across the FFI boundary to be reused on the Kotlin side of things. We're using a recent example of a new API in Glean that will drive the HTTP upload of pings, but the concepts I'm explaining here apply more generally.

Note: This blog post is not a good introduction on doing Rust on Android, but I do plan to write about that in the future as well.

Most of this implementation was done by Bea and I have been merely a reviewer, around for questions and currently responsible for running final tests on this feature.

The problem

The Glean SDK provides an FFI API that can be consumed by what we call language bindings. For the most part we pass POD over that API boundary: integers of various sizes (a problem in and of itself if the sides disagree about certain integer sizes and signedness), bools (but actually encoded as an 8-bit integer¹), but also strings as pointers to null-terminated UTF-8 strings in memory (for test APIs we encode data into JSON and pass that over as strings).

However for some internal mechanisms we needed to communicate a bit more data back and forth. We wanted to have different tasks, where each task variant could have additional data. Luckily this additional data is either some integers or a bunch of strings only and not further nested data.

So this is the data we have on the Rust side:

enum Task {
    Upload(Request),
    Wait,
    Done,
}

struct Request {
    id: String,
    url: String,
}

(This code is simplified for the sake of this blog post. You can find the full code online in the Glean repository.)

And this is the API a user would call:

fn get_next_task() -> Task

The solution

Before we can expose a task through FFI we need to transform it into something C-compatible:

use std::os::raw::c_char;

#[repr(u8)]
pub enum FfiTask {
    Upload {
        id: *mut c_char,
        url: *mut c_char,
    },
    Wait,
    Done,
}

We define a new enum that's going to be represent its variant as an 8-bit integer plus the additional data for Upload. The other variants stay data-less.

We also provide conversion from the proper Rust type to the FFI-compatible type:

impl From<Task> for FfiTask {
    fn from(task: Task) -> Self {
        match task {
            Task::Upload(request) => {
                let id = CString::new(request.id).unwrap();
                let url = CString::new(request.url).unwrap();
                FfiTask::Upload {
                    id: document_id.into_raw(),
                    url: path.into_raw(),
                }
            }
            Task::Wait => FfiTask::Wait,
            Task::Done => FfiTask::Done,
        }
    }
}

The FFI API becomes:

#[no_mangle]
extern "C" fn glean_get_next_task() -> FfiTask

With this all set we can throw cbindgen at our code to generate the C header, which will produce this snippet of code:

enum FfiTask_Tag {
  FfiTask_Upload,
  FfiTask_Wait,
  FfiTask_Done,
};
typedef uint8_t FfiTask_Tag;

typedef struct {
  FfiTask_Tag tag;
  char *id;
  char *url;
} FfiTask_Upload_Body;

typedef union {
  FfiTask_Tag tag;
  FfiTask_Upload_Body upload;
} FfiTask;

This is the C representation of a tagged union. The layout of these tagged unions has been formally defined in Rust RFC 2195.

Each variant's first element is the tag, allowing us to identify which variant we have. cbindgen automatically inlined the Wait and Done variants: they are nothing more than a tag. The Upload variant however gets its own struct.

On the Kotlin side of things we use JNA (Java Native Access) to call C-like functions and interact with C types. After some research by Bea we found that it already provides abstractions over C unions and structs and we could implement the equivalent parts for our Task in Kotlin.

First some imports and replicating the variants our tag takes.

import com.sun.jna.Structure
import com.sun.jna.Pointer
import com.sun.jna.Union

enum class TaskTag {
    Upload,
    Wait,
    Done
}

Next is the body of our Upload variant. It's a structure with two pointers to strings. Kotlin requires some annotations to specify the order of fields in memory. We also inherit from Structure, a class provided by JNA, that will take care of reading from memory and making the data accessible in Kotlin.

@Structure.FieldOrder("tag", "id", "url")
class UploadBody(
    @JvmField val tag: Byte = TaskTag.Done.ordinal.toByte(),
    @JvmField val id: Pointer? = null,
    @JvmField val url: Pointer? = null,
) : Structure() { }

And at last we define our union. We don't need a field order, it's a union afterall, only one of the fields is valid at a time.

open class FfiTask(
    @JvmField var tag: Byte = TaskTag.Done.ordinal.toByte(),
    @JvmField var upload: UploadBody = UploadBody()
) : Union() {
    class ByValue : FfiTask(), Structure.ByValue

    fun toTask(): Task {
        this.readField("tag")
        return when (this.tag.toInt()) {
            TaskTag.Upload.ordinal -> {
                this.readField("upload")
                val request = Request(
                    this.upload.id.getRustString(),
                    this.upload.url.getRustString()
                )
                Task.Upload(request)
            }
            TaskTag.Wait.ordinal -> Task.Wait
            else -> Task.Done
        }
    }
}

This also defines the conversion to a new Kotlin type that eases usage on the Kotlin side. getRustString is a small helper to copy the null-terminated C-like string to a Kotlin string.

The FFI function on the Kotlin side is defined as:

fun glean_get_next_task(): FfiTask.ByValue

The types we convert to and then work with in Kotlin are small classes around the data. If there's no attached data it's an object.

class Request(
    val id: String,
    val url: String,
) { }

sealed class Task {
    class Upload(val request: Request) : Task()

    object Wait : Task()

    object Done : Task()
}

As the final piece of this code on the Kotlin side we can now fetch new tasks, convert it to more convenient Kotlin objects and work with them:

val incomingTask = LibGleanFFI.INSTANCE.glean_get_next_task()
when (val action = incomingTask.toTask()) {
    is Task.Upload -> upload(action.request.id, action.request.url)
    Task.Wait -> return Result.retry()
    Task.Done -> return Result.success()
}

What's next?

Currently our new upload mechanism is under testing. We're reasonably sure that our approach of passing rich data across the FFI boundary is sound and not causing memory safety issues for now.

The advantage of going the way of encoding Rust enums into tagged unions for us is that we can use this API in all our current API consumers. C structs and unions are supported in Kotlin (for Android), Swift (for our iOS users) and Python (e.g. Desktop apps such as mozregression). It didn't require new tooling or dependencies to get it working and it's reasonably cheap in terms of processing cost (it's copying around a few bytes of data in memory).

The disadvantage however is that it requires quite a bit of coordination of the different pieces of code. As you've seen above for Kotlin we need to be careful to replicate the exact layout of data and all of this is (currently) hand-written. Any change on the Rust side might break this easily. Swift is a bit easier on this front as it has direct C translation.

The application-services team faced the same problem of how to transport rich data across the FFI boundary. They decided to go with protocol buffers and generate code for both sides of the FFI. They wrote about it in Crossing the Rust FFI frontier with Protocol Buffers. We decided against this way (for now), as it requires a bit more of a heavy-handed setup initially. We might reconsider this if we need to expand this API further.

My dream solution is still a *-bindgen crate akin to wasm-bindgen that creates all this code.

Footnotes:

Don't use a bool with JNA. JNA doesn't handle it well and has its own conception of the size of a bool that differs from what C and Rust think. See When to use what method of passing data between Rust and Java/Swift in the Glean SDK book. ↩

Rust in an instant

Sat, 02 May 2020 14:43:00 +0200

So I tweeted:

This happened in an instant. I am not even sorry.

It included this snippet of Rust code:

use std::os::raw::{c_int, c_void};
use std::sync::atomic::{AtomicU64, Ordering::SeqCst};
use std::time::Instant;

#[repr(C)]
struct timespec {
    tv_sec: u64,
    tv_nsec: u64,
}

static TIME_COUNTER: AtomicU64 = AtomicU64::new(0);

#[no_mangle]
extern "C" fn clock_gettime(clk_id: *mut c_void, tp: *mut timespec) -> c_int {
    unsafe {
        let next_tick = TIME_COUNTER.fetch_add(1, SeqCst);
        (*tp).tv_sec = next_tick;
        (*tp).tv_nsec = 0;
        0
    }
}

fn main() {
    let a = Instant::now();
    let b = Instant::now();

    let diff = b.duration_since(a);
    println!("{:?}", diff);
}

(Playground link)

And I was then asked what that piece of code does:

@varjmes: what am i seeing here? started reading the rust book yesterday :)

(Tweet)

So let me explain.

First let's take this from a tweet-worthy piece of code to be more production ready with cross-platform support. I'm on macOS most of the day. Let's add support for it:

#[cfg(all(unix, not(target_os = "macos")))]
use std::os::raw::{c_int, c_void};
use std::sync::atomic::{AtomicU64, Ordering::SeqCst};
use std::time::Instant;

#[cfg(all(unix, not(target_os = "macos")))]
#[repr(C)]
struct timespec {
    tv_sec: u64,
    tv_nsec: u64,
}

static TIME_COUNTER: AtomicU64 = AtomicU64::new(0);

#[cfg(all(unix, not(target_os = "macos")))]
#[no_mangle]
extern "C" fn clock_gettime(_clk_id: *mut c_void, tp: *mut timespec) -> c_int {
    unsafe {
        let next_tick = TIME_COUNTER.fetch_add(1, SeqCst);
        (*tp).tv_sec = next_tick;
        (*tp).tv_nsec = 0;
        0
    }
}

#[cfg(target_os = "macos")]
#[no_mangle]
extern "C" fn mach_absolute_time() -> u64 {
    const NANOSECONDS_IN_A_SECOND: u64 = 1_000_000_000;
    TIME_COUNTER.fetch_add(1, SeqCst) * NANOSECONDS_IN_A_SECOND
}

fn main() {
    let a = Instant::now();
    let b = Instant::now();

    let diff = b.duration_since(a);
    println!("{:?}", diff);
}

We can compile that:

rustc instant.rs

And then run it:

> ./instant
1s

What is happening here?

Time is an illusion

Computers provide clock sources that allow a program to make assumptions about time passed. If all you need is measure the time passed between some instant a and instant b the Rust libstd provides you with Instant, a measurement of a monotonically nondecreasing clock (I'm sorry to inform you that guarantees about monotonically increasing clocks are lacking). Instants are an opaque thing and only good for getting you the difference between two of them, resulting in a Duration. Good enough if you can rely on the operating system to not lie to you.

How is Instant implemented? That depends on the operating system your running your code on. Luckily the documentation lists the (currently) used system calls to get time information.

Platform[^1]	System Call
UNIX	`clock_gettime` (Monotonic Clock)
Darwin	`mach_absolute_time`

([^1]: Trimmed down to the platforms I work on.)

So that's where Rust gets its time from.

The operating system is an illusion

But how do we replace operating system functionality? Turns out functions are merely a name and the linker tries to figure out where the code behind them is and then points to that.

If we look at a simpler version of our code without any #[no_mangle] extern "C" functions we can see where these come from:

(macOS) ❯ nm -a ./instant | rg mach_absolute_time
                 U _mach_absolute_time

(linux) ❯ nm -a instant | rg clock_gettime
                 U clock_gettime@@GLIBC_2.17

nm lists symbols from object files. That U right there stands for undefined: The binary doesn't have knowledge where it comes from (though in the case of Linux it gave it a slightly different name letting us guess what to expect). Details about this symbol are later filled in, once your program is loaded and symbols to dynamic libaries are resolved (and I'm sure the "Making our own executable packer" series by @fasterthanlime will have lots more details on it, I should go read it).

What if we do that work before even running the program?

Strings are strings, no matter what order

So now let's try to expose a function under the same name from our code.
Note: I'm only doing the macOS part here, it works the same for Linux.

If we start with the plain function like this:

fn mach_absolute_time() -> u64 {
    1
}

it will not turn up in the final binary at all and rustc right fully complains:

warning: function is never used: `mach_absolute_time`

Even making it pub won't work. By default symbols such as function names get mangled; some additional information is encoded into the name as well and at the same time this ensures uniqueness of names (see also Name Mangling). One can disable that in Rust using the #[no_mangle] attribute. So we apply that.

We're trying to override a function that is defined in terms of C. C has slightly different calling conventions than Rust. It's part of their respective ABIs: how data is passed in and out of functions. In Rust one can define the used ABI with the extern "name" tag. So we add that.

And thus we end up with

#[no_mangle]
extern "C" fn mach_absolute_time() -> u64 {
    1
}

If we compile our code again and look at the symbols we get this:

(macOS) ❯ nm -a ./instant | rg mach_absolute_time
0000000100000c20 T _mach_absolute_time
(linux) ❯ nm -a instant | rg clock_gettime
0000000000005250 T clock_gettime

Now we know that the symbol is defined somewhere in the binary ("T - The symbol is in the text (code) section.") and also the location (0000000100000c20 / 0000000000005250).

If this program is run it will still need to resolve undefined symbols ... but this time our functions are not undefined anymore. As the stdlib just calls whatever is behind that respective name it now calls our code instead!

And yes, this works for pretty much all the functions defined by other libraries or libc:

use std::fs::File;
use std::os::raw::{c_char, c_int};

#[no_mangle]
extern "C" fn open(_path: *mut c_char, _flag: c_int) -> c_int {
    -1
}

fn main() {
    File::open("/etc/passwd").unwrap();
}

❯ rustc file.rs
❯ ./file
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 0, kind: Other, message: "Undefined error: 0" }', file.rs:10:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Please don't do this.

If you liked this you might also like STDSHOUT!!!1!, I even gave a talk about it.

"Edit this file on GitHub"

Thu, 06 Feb 2020 16:38:00 +0100

At work I help with maintaining two large documentation books:

Firefox Data Docs aka docs.telemetry.mozilla.org aka dtmo
The Glean SDK book

Back in 2018 I migrated dtmo from gitbook to mdbook (see the pull request). mdbook is maintained by the Rust project and hosts the Rust book as well as a multitude of other community projects. It provided all we need, plus a way to extend it with some small things, I blogged about ToC and mermaid before.

During the Mozilla All Hands last week my colleague Mike casually asked why we don't have links to quickly edit the documentation. When someone discovers a mistake or inaccuracy in the book the current process involves finding the repository of the book, then finding the right file, then edit that file (through the GitHub UI or by cloning the repository), then push changes, open a pull request, wait for review and finally get it merged and deployed.

I immediately set out to build this feature.

I present to you: mdbook-open-on-gh

It's another preprocessor for mdbook, that adds a link to the edit dialog on GitHub (if your book is actually hosted on GitHub). And that's how it looks:

It's already deployed on dtmo and the Glean SDK book and simplifies the workflow to: click the link, edit the file on GitHub, commit and open a PR, get a review and merge it to deploy.

If you want to use this preprocessor, install it:

cargo install mdbook-open-on-gh

Add it as a preprocessor to your book.toml:

[preprocessor.open-on-gh]
command = "mdbook-open-on-gh"
renderer = ["html"]

Add a repository URL to use as a base in your book.toml:

[output.html]
git-repository-url = "https://github.com/mozilla/glean"

To style the footer add a custom CSS file for your HTML output:

[output.html]
additional-css = ["open-in.css"]

And in open-in.css style the <footer> element or directly the CSS element id open-on-gh:

footer {
  font-size: 0.8em;
  text-align: center;
  border-top: 1px solid black;
  padding: 5px 0;
}

This code block shrinks the text size, center-aligns it under the rest of the content and adds a small horizontal bar above the text to separate it from the page content.

Finally, build your book as normal:

mdbook path/to/book

This Week in Glean: Cargo features - an investigation

Mon, 03 Feb 2020 15:00:00 +0100

The last blog post: This Week in Glean: Glossary by Bea.

As :chutten outlined in the first TWiG blog post we're currently prototyping Glean on Desktop. After a couple rounds of review, some adjustements and some learnings from doing Rust on mozilla-central, we were ready to land the first working prototype code earlier this year (Bug 1591564).

Unfortunately the patch set was backed out nearly immediately ¹ for 2 failures. The first one was a "leak" (we missed cleaning up memory in a way to satisfy the rigorous Firefox test suite, that was fixed in another patch). The second one was a build failure on a Windows platform.

This is what the log had to say about it (shortened to the relevant parts here, see the full log output):

lld-link: error: undefined symbol: __rbt_backtrace_pcinfo
>>> referenced by gkrust_gtest.lib(backtrace-5286ea09b9822175.backtrace.3kzojw1m-cgu.3.rcgu.o)
lld-link: error: undefined symbol: __rbt_backtrace_create_state
>>> referenced by gkrust_gtest.lib(backtrace-5286ea09b9822175.backtrace.3kzojw1m-cgu.3.rcgu.o)
lld-link: error: undefined symbol: __rbt_backtrace_syminfo
>>> referenced by gkrust_gtest.lib(backtrace-5286ea09b9822175.backtrace.3kzojw1m-cgu.3.rcgu.o)
clang-9: error: linker command failed with exit code 1 (use -v to see invocation)
/builds/worker/workspace/build/src/config/rules.mk:608: recipe for target 'xul.dll' failed]

I set out to investigate this error. While I had not seen that particular error before, I knew about the backtrace crate. It caused me some trouble before (it depends on a C library, and won't work on all targets easily). I knew that the Glean SDK doesn't really depend on its functionality² and thus removing it from our dependency graph would probably solve the issue. But first I had to find out why we depend on it somewhere and why it is causing these linker errors to begin with.

The first thing I noticed is that we didn't include anything new in the patch set that was now rejected. Through some experimentation and use cargo-tree I could tell that backtrace was included in the build before our Glean patch³, as a transitive dependency of another crate: failure.

So why didn't it fail the build before? As per the errors above, the build failed only during linking, not compilation, which makes me believe those functions were never linked in previously, because no one passed around any errors that would cause these functions to be used.

As said before, the Glean SDK doesn't really need failure's backtrace feature, so I tried disabling its default features. Due to how cargo currently works, this needs to be done across all transitive dependencies (the final feature set a crate is compiled with is the union across everything).

I then changed mozilla-central to use the crates from git directly for testing.

Turns out that still fails with the same issue on the Windows target. Something was re-enabling the "std" feature of failure in tree.

cargo-feature-set was able to show me all enabled features for all dependencies I tracked it down further⁴.

Turns out the quantum_render feature enables the webrender_bindings crate, which then somehow pulls in failure through transitive dependencies again. More trial-and-error revealed its a dependency of the dirs crate, only used on Windows. Except dirs doesn't need failure for the target we're building for (x86_64-pc-windows-gnu or Mac or Linux). It's again a transitive dependency for a crate called redox_users, which is only pulled in when compiled for Redox⁵.

Except that's not how Cargo works. Cargo always pulls in all dependencies, merges all features and only later ignores the crates it doesn't actually need. That's a long standing issue:

So now we identified who's pulling in the backtrace crate and maybe even identified why it was not a problem before. How do we fix this?

As shown before, just disabling the backtrace feature in crates we use directly doesn't solve it, so one quick workaround was to force failure itself to not have that feature ever. Easily done:

--- Cargo.toml
+++ Cargo.toml
@@ -23,5 +23,5 @@ members = [".", "failure_derive"]
 [features]
 default = ["std", "derive"]
 #small-error = ["std"]
-std = ["backtrace"]
+std = []
 derive = ["failure_derive"]

I forked failure and commited that patch, then made mozilla-central use my forked version instead. Later I also removed failure from both Glean and application-services' ffi-support, as the small functionality we got from it was easily reimplemented manually.

Both approaches are short-term fixes for getting Glean into Firefox and it's clear that this issue might easily come up in some form soon again for either us or another team. It's also a major hassle for lots of people outside of Mozilla, for example people working on embedded Rust frequently run into problems with no_std libraries suddenly linking in libstd again.

Initially I also planned to figure out a way forward for Cargo and come up with a fix for it, but as it turns out: Someone is already doing that!

@ehuss started working on adding a new feature resolver. While not yet final, it will bring a new -Zfeatures flag initially:

itarget — Ignores features for target-specific dependencies for targets that don't match the current compile target.
build_dep — Prevents features enabled on build dependencies from being enabled for normal dependencies.
dev_dep — Prevents features enabled on dev dependencies from being enabled for normal dependencies.

I'm excited to see this being worked on and can't wait to try it out. It will take longer for mozilla-central to rely on this of course, but I hope this will eventually solve one of the long standing issues with cargo.

cargo tree --features bitsdownload,cranelift_x86,cubeb-remoting,fogotype,gecko_debug,gecko_profiler,gecko_refcount_logging,moz_memory,moz_places,new_cert_storage,new_xulstore,quantum_render

Footnotes:

When patches are merged, a full set of tests are run on the Mozilla CI servers. If these tests fail the patch is reverted ("backed out") and the initial committer informed. ↩
Glean's error handling is pretty simplistic. We mostly only log errors on the FFI boundary and don't propagate them over this boundary. So far we didn't need a backtrace for errors there. ↩
The exact command enabling all the same features as the build I run in toolkit/library/rust/shared was (on the parent commit of my later patch to fix it: 7a3be2bbc): ↩
Same feature set as above, just running cargo feature-set this time. ↩
"Redox is a Unix-like Operating System written in Rust, aiming to bring the innovations of Rust to a modern microkernel and full set of applications". Firefox doesn't build on Redox. ↩

This Week in Glean: A Release

Thu, 24 Oct 2019 17:30:00 +0200

Last week's blog post: This Week in Glean: Glean on Desktop (Project FOG) by chutten.

Back in June when Firefox Preview shipped, it also shipped with Glean, our new Telemetry library, initially targeting mobile platforms. Georg recently blogged about the design principles of Glean in Introducing Glean — Telemetry for humans.

Plans for improving mobile telemetry for Mozilla go back as as far as December 2017. The first implementation of the Glean SDK was started around August 2018, all written in Kotlin (though back then it was mostly ideas in a bunch of text documents). This implementation shipped in Firefox Preview and was used up until now.

On March 18th I created an initial Rust workspace. This kicked of a rewrite of Glean using Rust to become a cross-platform telemetry SDK to be used on Android, iOS and eventually coming back to desktop platforms again.

1382 commits later¹ I tagged v19.0.0².

Obviously that doesn't make people use it right away, but given all consumers of Glean right now are Mozilla products, it's up on us to get them to use it. So Alessio did just that by upgrading Android Components, a collection of Android libraries to build browsers or browser-like applications, to this new version.

This will soon roll out to nightly releases of Firefox Preview and, given we don't hit any larger bugs, hit the release channel in about 2 weeks. Additionally, that finally unblocks the Glean team to work on new features, ironing out some sharp edges and bringing Glean to Firefox on Desktop. Oh, and of course we still need to actually release it for iOS.

Thanks

Glean in Rust is the project I've been constantly working on since March. But getting it to a release was a team effort with help from a multitude of people and teams.

Thanks to everyone on the Glean SDK team:

Thanks to Frank Bertsch for a ton of backend work as well as the larger Glean pipeline team led by Mark Reid, to ensure we can handle the incoming telemetry data and also reliably analyse it. Thanks to the Data Engineering team led by Katie Parlante. Thanks to Mihai Tabara, the Release Engineering team and the Cloud Operations team, to help us with the release on short notice. Thanks to the Application Services team for paving the way of developing mobile libraries with Rust and to the Android Components team for constant help with Android development.

Footnotes:

Not all of which are just code for the Android version. There's a lot of documentation too. ↩
This is the first released version. This is just the version number that follows after the Kotlin implementation. Version numbers are cheap. ↩

Rust - 4+ years later

Wed, 15 May 2019 14:00:00 +0200

Exactly 4 years ago, on May 15th 2015, the Rust Core Team announced the first stable version, Version 1.0, of the Rust Programming language.

I started my Rust journey even earlier than that, somewhere around August 2014, in parallel to my Bachelor thesis. Shortly after my first blog post, I gave a last minute talk titled "Rust for Rubyists" ¹ at RedFrogConf, a (Ruby-focused) subconf of FrOSCon. Back then Rust had "lightweight green tasks"!

Two months before the 1.0 release I spoke at CCCAC about Rust (those slides were built on minimal JavaScript so it could run in Servo). I remember that I spent a lot of time on explaining the (safe) thread API. Not only did I make a mistake in the explanation (and the code), but also shortly after the scoped thread API was found to be unsound and subsequently removed (use crossbeam's thread::scope API if you need something similar but safe and sound now).

Then 1.0 came, more stuff was ripped out just before that and we got the first edition of The Rust Programming Language book. Work obviously didn't stop there.

In 2016 we created the first ever RustFest, a community conference dedicated to Rust. Due to luck and enthusiasm of many more people we managed to pull of another RustFest just half a year later and then four more editions after that (I missed only the Zurich one). The sixth edition, RustFest Barcelona, is already announced.

In 2014 Florian and Johann founded the Rust Berlin Hack and Learn. Last year, when I joined Mozilla and moved to Berlin, I became a co-organizer of that².
Today we're celebrating, but not just Rust's birthday, but also over 100 events under the Rust Berlin label (that includes our bi-weekly Hack and Learn, infrequent Rust Berlin talk meetups and the Rust & Tell).

A month after I joined Mozilla, the first ever Rust All-Hands happened in the Berlin office and, as so much about my involvement with Rust, I stumbled into being the responsible on-site person during that week. It went pretty well. We repeated the Rust All-Hands this year (German article), now with me being officially the on-site organizer, but also attendee.

In 2017 I shipped my first piece of Rust code used in production (an actual opera production): midioscar. A couple of weeks ago I started my first Mozilla project using Rust (more about that in a future blog post).

Not only did Rust become a community I invest my time and resources in, it is now also officially part of my job.

We're at Rust 1.34.2 now (that's over 34 36 releases in 4 years!), with even more improvements, bug fixes, community conferences, hack fests and All Hands ahead of us. Rust is here to stay.

🎉 To the next 4+ years! 🎉

Footnotes:

Based on @steveklabnik's "Rust for Rubyists" tutorial. ↩
Fun fact: by accident I kinda was the one responsible for a Hack'n'Learn one night when I was in Berlin for other work, but none of the dedicated organizers was able to make it to the meetup. ↩

What Rust is it?

Wed, 28 Nov 2018 20:09:00 +0100

A while ago (sometime in August) I built a small service called What Rust is it?.

Based on what What Train is it now? is for Firefox, this small service lists the three trains of Rust: release, beta & nightly as well as their respective versions. It's updated every day and actually checks what rustup installed. Additionally it shows the (planned) date of the next release.

Similar information is available on the Rust Lang Forge.

See it live:

## [www.whatrustisit.com](http://www.whatrustisit.com)

How I made a useless tool and the reactions made my sunday better

Sun, 30 Sep 2018 23:00:00 +0200

Last night I had an idea, which made me stay up two hours longer than I wanted. I built a useless tool: lysbilder, the slide system no one needed.

Today I tweeted about it and then went out to enjoy the day a bit:

@badboy_: I might have built a very useless slideshow tool on top of rustdoc, generated from code documentation, linked by trait implementations.

Please don’t use it¹.

https://badboy.github.io/lysbilder/

¹: however, I will use it once.

Quickly I got some responses:

@myriamjessier: I laughed. It was good enough to retweet just because of that.

and

@QuietMisdreavus: i love this so much

Those made me smile and made my sunday a bit better. :)

Ideas for Rust Meetups

Wed, 08 Aug 2018 12:45:00 +0200

Not every meetup follows the same structure. Not every meetup needs to have 75+ attendees. If you find a room and 3 people, it's a meetup.

LLogiq, organizer of the Rhein-Main Rust meetup, published a list with some Ideas for Rust Meetups.

Here are 2 more:

RFC Lightning talks - Attendees sign up before the meetup to present one open RFC. Allocate some time to discuss it in more detail afterwards. Rust Cologne had such a meetup.
Hack'n'Learn - Open the door, have some desks and let people work on their projects. This works very well for the Berlin Hack'n'Learn. We usually start with a quick introduction round, where everyone also states what they plan to work on. Later that evening people get a couple more minutes to present their results if they want.

Update:

Another that @killercup brought up:

Open Space Meetups - Imagine a bazaar, where you can suggest topics and spontaneously join any discussion you find interesting. Collect topics at the beginning, assign 30 minute slots. Start discussing in small groups. Hop between discussions when you think it's finished. Based on 4 simple principles.

DOH everything

Fri, 09 Feb 2018 14:52:00 +0100

Two days ago I wrote about a new small utility crate for doing DNS over HTTPS. When I started with the code on Monday I had an actual use case in mind.

Due to some combination of SmartOS, Ubuntu 17.04 and glibc my server currently has problems resolving DNS. It turns out a server without a working DNS resolver is kind of a pain, adding hostnames and their IPs to /etc/hosts gets tiring pretty fast.

Luckily, glibc's gethostbyname makes us of external plugins to do the actual resolving. That's how /etc/hosts and an external resolver work in the first place. The documentation on the exact plugin mechanism is quite sparse, but luckily systemd comes with its own resolve plugin: nss-resolve.c.

All it takes to build a resolver plugin is to implement two functions¹:

enum nss_status _nss_$plugin_gethostbyname4_r(...)
enum nss_status _nss_$plugin_gethostbyname3_r(...)

Both take a bunch of arguments, but it boils down to this:

Get the hostname (and optionally the address family to look for)
Resolve this hostname to its IP addresses²
Copy the data into the provided buffer
Let the result point somewhere into this buffer
Clear the error values and return a success value

If at any point an error is encountered, several error values are set and the function returns a failure.

If all goes right, the requesting application will receive a data structure to read the different IP addresses from, which it can then use to establish a connection.

Equipped with dnsoverhttps the resolving part is suprisingly easy. First turn the passed name into a string, then call into the library and collect the results.

unsafe {
    let slice = CStr::from_ptr(orig_name);
    let name = slice.to_string_lossy();
    let addrs = match dnsoverhttps::resolve_host(&name) {
        Ok(a) => a,
        Err(_) => {
            *errnop = EINVAL;
            *h_errnop = NO_RECOVERY;
            return NSS_STATUS_UNAVAIL;
        }
    };

    // ...
}

Filling the appropiate data structures and memory buffer is a bit more complex and involves some pointer trickery. Instead of trying to find the right (unsafe) approach in Rust to do it correctly I opted to copy the systemd-code for now.

In the Rust part we can forward the received arguments and add an array of results³:

let addrs : Vec<_> = addrs.into_iter().map(ip_addr_to_tuple).collect();

write_addresses4(orig_name, pat, buffer, buflen, errnop, h_errnop, ttlp,
  addrs.as_ptr(), addrs.len())

This calls into a function written in C and compiled just before the Rust code is compiled. It's also statically linked into the resulting shared object, ready to be deployed.

In order to use it, the library has to be copied into a directory where the system can find it:

cp target/release/libnss_dnsoverhttps.so /usr/lib/libnss_dnsoverhttps.so.2

And then it can finally be used by adding it to /etc/nsswitch.conf like this:

hosts: dnsoverhttps files mymachines dns myhostname

curl and other processes calling gethostbyname will from now on get addresses resolved by this small module.

Footnotes:

And forward _nss_$plugin_gethostbyname2_r and _nss_$plugin_gethostbyname_r to _nss_$plugin_gethostbyname3_r. ↩
Remember that you can't use gethostbyname here, because you are gethostbyname. ↩
In between turning our vector of addresses into a simpler structure. ↩

D'oh! - DNS over HTTPS in Rust

Wed, 07 Feb 2018 09:05:00 +0100

Last weekend I met Daniel Stenberg, author of curl, at FOSDEM and we talked a bit about curl, Firefox and also Rust. One recent project he was working on was DNS over HTTPS support for Firefox and eventually for curl as well.

DNS over HTTPS, short DOH, is a recent idea to do DNS queries over HTTPS for privacy, performance and security reasons. There's an RFC draft in version 3¹ at the IETF describing it in more detail.
The tl;dr: Send the DNS protocol in an HTTP POST request or base64-encoded in a GET request, get back the DNS protocol in the response body and parse it.

Given how simple that sounds, I decided to implement a minimal DOH client in Rust. I present to you:

dnsoverhttps

It exports one function to resolve a hostname to its IPv6 and IPv4 addresses:

extern crate dnsoverhttps;

fn main() {
    let name = "example.com";
    let addr = dnsoverhttps::resolve_host(name);

    for a in addr {
        println!("{} has address {}", name, a);
    }
}

It currently uses dns.google.com (or to be exact one of its IPs: 172.217.21.110) and skips TLS certificate checks (because the underlying HTTP request library has no option to pass in the hostname yet). It will also always query for both IPv6 and IPv4 addresses (AAAA and A records respectively). CNAME records will work as well by the mere fact that the server recursively resolves it and dnsoverhttps simply takes the found IP address. Other records are not supported at the moment.

For a quick try you can install the bundled CLI tool to replace the host tool to resolve hostnames.

cargo install dnsoverhttps

And then execute it:

$ ~/.cargo/bin/host example.com
example.com has address 2606:2800:220:1:248:1893:25c8:1946
example.com has address 93.184.216.34

Documentation is available online.

Footnotes:

dnsoverhttps currently implements Version 2, because that's what dns.google.com supports. Version 3 changes the query parameter from body to dns. ↩

Retrospective: Rust Cologne in January

Wed, 10 Jan 2018 12:25:00 +0100

Last monday, 8th of January, we had the first Rust Cologne meetup this year. We opted for a Open Space-style meetup, allowing for broad discussion rounds in small groups, which worked reasonably well this time (it's the second time we did it). More on that later.

News

We usually start off the meeting with a short round of news since the last meetup (usually 4-6 weeks). This time we did it a bit adhoc:

Open Space

Open Space follows 4 simple principles:

Whoever comes are the right people.
Whatever happens is the only thing that could have.
When it starts is the right time.
When it's over, it's over.

This time around we had a couple of new attendees, which is very encouraging for everyone involved. With a dozen people around, we managed to have between two and four concurrently running small discussion rounds.

We collected some topics upfront:

Learning Rust
Favourite not-yet-stable feature
Embedded Rust
Realtime programming
Tokio / async programming
Rust for the Web
Miri

Obviously, I was not able to take part in all discussions, but I collected some rough notes from the things I was in. In the following are some rough notes regarding each topic. There are not necessarily clear outcomes of the discussions or steps to take.

Learning Rust

With a couple beginners and some more seasoned Rustaceans around, we discussed what it takes to learn Rust, (missing) material and projects.

High-level docs on Cargo were requested: what does it do, why are dependencies not that bad, what to look out for in dependencies, drawbacks of too many dependencies
A book on patterns in Rust: not everything in Rust is as easy to write as in other languages, sometimes a whole other pattern is necessary. These should be collected (also a "X in C++ will become Y in Rust").
Edit: The Rust Design Patterns repository exists.
Confusion about split between 1st and 2nd version of The Rust Programming Language (the book). Old links are obviously sometimes broken, not all content is in both books. Some links in official documentation are broken due to the split (we didn't find a example that night, but it's worth triaging this)
Better crate caching: A global per-system/per-user cache of compiled crates could go a long way to improve compile times (in theory). Why is it not done? Drawbacks? We should look up the current status and discussions.
A Rust Cookbook might be a good idea, but can easily become overwhelming. In the end it needs to be easily findable through a quick Google search.

In the end this discussion spun off into the embedded discussion and I left.

Tokio / async programming

Tokio is undocumented, which definitely hinders adoption
We discussed/explained what an event reactor (like Tokio) even is and how it ties in with futures
We discussed why futures in Tokio are tied to the Tokio event reactor and can't easily be re-used for another event reactor
Briefly discussed the coming async/await/Generator features
Someone requested a larger guide on how to structure a large server with asynchronous network I/O. How to handle state? How to deal with disconnects/reconnects?
We discussed trade-offs between async I/O and just relying on threads for server applications and when the latter makes more sense

Rust for the Web

I talked a bit about the story of Rust for web servers and also about the frontend part.
Lots of interest in doing Rust in the frontend and not need to deal with JavaScript for most of the time
The state of current web frameworks: there's a lot of work to do; Rocket might be promising, but it's bound to nightly and a lot of code rewriting, which I don't necessarily like; Iron and Nickel, the earliest frameworks, are on their way out

After 3 hours of lively discussions we brought the evening to an end. This was (most likely) my last meetup as an official part of the organizer team. Plans for the February meetup are started and I might join that one again if I'm around. Please jump into the conversation and propose ideas, talks or projects!

Thanks to Colin (who also left the organizer team, but took care of invitations for the January meetup one last time), Pascal & Florian.

Hello, Rust! Hello, WebAssembly!

Sat, 18 Nov 2017 16:26:57 +0100

hellorust.com is live!

This site will collect resources, guides, articles and links around Rust & WebAssembly. If you want to help or have more resources we should add, open an issue or send a Pull Request to github.com/badboy/hellorust.

To get started, head over to the Setup section or take a look at some example code.

novemb.rs Code Sprint Weekend 2016 - Retrospective

Tue, 29 Nov 2016 11:55:00 +0100

This post is a tiny bit late, but better late than never.

So on 19th and 20th of November, just over a week ago, we had the very first novemb.rs Code Sprint. In 10 locations in Europe and the US as well as online, people gathered to hack on projects, start new ones or just to learn Rust. Bringing people together is one goal of the Rust community and coding, learning and having fun together is a lot of fun as well.

We opened doors at Chaos Computer Club Cologne

I was part of the novemb.rs Event in Cologne. On both days about a dozen people showed up, from noon to late in the evening.

In Cologne, we had several different projects and ideas being worked on:

Some audio/midi processing. I don't have much to show here, but I hope we will see (and hear) more of this soon, maybe even a talk at our meetup next year.
@panicbit and @killercup tried to port FLIF, a "Free Lossless Image Format" to Rust. This kept everyone on-site entertained for quite a bit.
@Florob worked on a problem with buffering, when mixing rust-openssl with a BufReader. No code to show sadly either.
@colin-kiegel followed up on my Pull Request on rust-derive-builder, diving deep into Macros 1.1. We decided to move forward with some of the ideas to make that crate use the new way of doing things, once they hit stable.
As part of my "job" as a community team member I took a look at rustaceans.org, a website is for finding Rustaceans. We will soon extend it to make it easier to find potential speakers for your meetup or event.
@sangyye dropped by to learn some more Rust. He's one of the organisators behind Cologne.rb and *hint* *hint* there might be a joint meetup next year.
Personally, I wanted to work on semantic-rs. but didn't get to write a single line of code for that project. Though I fixed a few other projects and wrote another few hundred lines for a (not yet published) project.

Thanks to our attendees and thanks to my co-organizers Colin, Flo & Pascal. And another big thanks to Mozilla for sponsoring the pizza and the C4 for offering the space.

Things that worked well

We got people together! About a dozen of people per day in Cologne and another handful at other locations as well.
We got some things coded. See above.
We learned and discussed Rust and had fun the whole weekend.

Things that we should improve

Organization in other locations was a bit ad-hoc and on short-term, which is why we might have missed out on a few people.
With a bit more time ahead it might be easier to get people involved. We also lacked in communication and (in the beginning) didn't make it clear what the event was supposed to be in.
novemb.rs tried to focus on people in UTC-1 to UTC+4, which covers most of Europe, but also the African continent. People from the US jumped in and opened their offices or other locations as well, but we were unable to get anything going on the African continent. We should try harder to get people from everywhere involved.
Online Communication between locations wasn't used that much. The whole idea was to have a distributed code sprint, so working together on projects across locations would have been nice. Maybe we should offer to have ad-hoc video calls as well?

Coding on the whiteboard (only half of the attendees in the picture)

Try again?

Definitely! Given that the overall organization effort is low I'd like to make this a more regular thing. With the experience from this year we have enough points to improve and we can plan this in advance. Plus, we already have the domain anyway ;)

Until then, there are of course other opportunities to get in contact with fellow Rustaceans. We have a monthly meetup in Cologne, with a Christmas-themed impl Glühwein for RustCologne happening next week. Feel free to join us on the Weihnachtsmarkt or in January at another more regular meetup. If you are not from the Rhein area, take a look if there's a Rust User Group near you.

Signs showed the way

Signify - Ed25519 signatures for your files (in Rust)

Wed, 28 Sep 2016 13:16:00 +0200

From time to time I try to write a piece of code or port some existing library or application just for fun. So a while back in June I had some free time again and I came across signify. I ported it to rust: signify-rs

signify is a small command line utility to create Ed25519 signatures of files. It was developed to cryptographically sign and verify OpenBSD releases. Read "Securing OpenBSD From Us To You" for more details.

Now all you need to create signatures and verify them is a private & a public key. Both parts are super short, so it is no problem to embed or print them. This is what a public key would look like:

RWR/JMX+u3pyzGSyTSnOFINVcHJycZE/o6UTUshxvpcp6S4annmJK4DB

The corresponding private key is this:

RWRCSwAAAAASfh0v1XJS4FYg59ntf4d7zYS9GR848h1/JMX+u3pyzIFNjvCMxLUgurelzAWTCSd9y6ghUcbcPVMuwnlDJy4WZLJNKc4Ug1VwcnJxkT+jpRNSyHG+lynpLhqeeYkrgME=

(Warning: Do NOT use above keys for anything! I included both for demonstration only)

signify will add an additional comment to the key files, but they are not used for any verification (besides making sure they are actually in the file).

Whereas pure Ed25519 keys are just 32 bytes, above keys already include some additional information used by signify to create and verify signatures. The private key can also be protected by a passphrase.

When I started porting this small application I knew nothing about Ed25519 or signify. I more or less translated the existing C code into Rust. Back then, I used rust-crypto, a pure Rust implementation various common cryptographic algorithms. It provided all I needed: Ed25519 key generation, signing & verification and bcrypt for the passphrase handling. In just one day I had a working application and less than 2 weeks later I also implemented proper passphrase-protection.

In August then the most promising Rust crypto library, *ring*, was released on crates.io as well (before that it could only be used as a git dependency). I used *ring* before for nobsign and Brian Smith, the author of *ring*, already helped out with some code review & helping me use the API properly.
So I was not too surprised when Brian reached out to me asking if I would be willing to port signify-rs over to *ring* as well.

I was (however, it was shortly before RustFest, so I couldn't dedicate much time to it).

I took a look at the *ring* documentation and immediately realised I had a completely wrong understanding of Ed25519. Whereas in signify the public key is stored in 32 byte and the private key is stored in 64 byte, *ring* had both keys as only 32 byte long. With feedback from Brian I realised that the longer private key as used in signify is actually just the 32-byte public key concatenated to the (real) 32-byte private key, resulting in 64 byte total.

Equipped with this information (and after the first RustFest day was over), it was easy to port over signify-rs to *ring*.

signify-rs still depends on rust-crypto though, as it provides the necessary bcrypt_pbkdf for encrypting the private key with a user-chosen passphrase. *ring* does not provide bcrypt and probably won't do so anytime soon. If anyone wants to implement a really good bcrypt crate, please contact me or Brian for feedback.

Use `signify-rs`

First you need to install it, do so with cargo:

cargo install signify

Generate a key pair:

signify -G -p public-key -s secret-key

This will ask you for a passphrase to protect the secret key. Remember that. Now you can sign a file:

signify -S -s secret-key -m README.md

This will create README.md.sig containing the signature. To verify it:

signify -V -p public-key -m README.md

If it prints Signature Verified it went well. Otherwise it will show an error.

A signature

This is the signature using above private key on the README.md:

untrusted comment: signature from signify secret key
RWR/JMX+u3pyzIDv+Gt4JwMbWsb+dt0R/9tYDEjVw7zmBQoQR06Pcd2yr03XqvTSqaJBTbUhm74iUxB98BQVAZemq692g5Xv0gs=

If you put that signature in a file, put the public key from above into another, you can verify it!

Future plans

I still have todos left for this project. First I want this to be fully compatbile with the original implemenation. The original implementation can embed signatures into the signed file and also verify an embedded signature. I want to add that. I also want to distribute pre-compiled binaries for various platforms (hello, rust-everywhere) and provide proper Ed25519 signatures on all those releases.

If you want to help with any of that, jump over to the GitHub repository and start hacking.

Rust from the Back to the Front (Rust Sthlm)

Wed, 31 Aug 2016 16:30:00 +0200

Last monday I attended the Rust Sthlm Meetup and gave a talk about using Rust for web development. About 60 people attended, had pizza and listened to the two talks of the evening.

I started off with my talk Rust from the Back to the Front, giving an overview of the ecosystem around all things related to web programming in Rust. This was an updated talk of the one I gave in Budapest last year (video online). I had some technical difficulties this time in the beginning (yeah, computers…), but otherwise the talk went well. People showed interest in the presented topics. Sadly I only briefly touched the new way of doing asynchronous I/O using futures and tokio. I definitely need to look deeper into this topic, as I think it can bring huge improvements to existing web frameworks and libraries as well. This has to wait a bit though, as I will first dive deeper into Emscripten (and present that next week in Cologne and in Pittsburgh in October). My slides are online and I will try to collect more resources in a Gist.

The second talk that evening was by Kristoffer Grönlund, giving us a quick introduction to some of Rust's features, followed by an overview of his work trying to get Rust into the openSUSE package repositories. Turns out it is not that easy, especially if everything has to be built from source and offline, but at least there are some improvements that might help make this easier. His slides are online as well.

Even after the talks some of the people stuck around and we discussed several more things around Rust, how it is still evolving, fast moving and a bit unstable from time to time.

All in all I had a great time, talked to a number of people about differen topics and I hope I could convince some to actually try Rust. With such a large and interested tech community in Stockholm, I'm sure the Meetup will live on.

Quick check: does your code work?

Thu, 12 May 2016 23:00:00 +0200

… because mine didn't. At least not correctly in all cases. I'm talking about my Rust library lzf-rs, a port of the small compression library LibLZF. It started as a wrapper around the C library, but I rewrote it in Rust for v0.3. I now found three major bugs and I want to tell you how (tl;dr: Bug fixes and tests: PR #1).

For a university paper I'm currently looking into different methods for automatic test generation, such as symbolic execution, fuzzing and random test generation. One of the popular methods is property-based testing, with QuickCheck being the best known application of this method. QuickCheck started as a Haskell library (see the original paper), but is ported to several other languages, including C (see theft) and of course Rust: QuickCheck.

I knew this library for some time now, but never used it. So today I decided to use it for my lzf crate. Let me walk you through the process on how to use it.

First, you need to add the dependency and load it in your code. Add it to your Cargo.toml:

[dev-dependencies]
quickcheck = "0.2"

Add this to your src/lib.rs:

#[cfg(test)]
extern crate quickcheck;

Next, you need to decide what property to test. As the compression library needs data to compress and valid data to decompress, I decided the easiest way to go through everything would be to test the round trip: Compress some random input, then decompress the compressed data and check that it maches the initial input. This should hold for all inputs, that can be compressed. Everything that cannot be compressed can be ignored at this point (a first test allowing all input turned up too many false-positives).

The property function looks like this:

use quickcheck::{quickcheck, TestResult};

fn compress_decompress_round(data: Vec<u8>) -> TestResult {
    let compr = match compress(&data) {
        Ok(compr) => compr,
        Err(LzfError::NoCompressionPossible) => return TestResult::discard(),
        Err(LzfError::DataCorrupted) => return TestResult::discard(),
        e @ _ => panic!(e),
    };

    let decompr = decompress(&compr, data.len()).unwrap();
    TestResult::from_bool(data == decompr)
}

Of course we need to test this. QuickCheck handles the heavy part for us:

#[test]
fn qc_roundtrip() {
    quickcheck(compress_decompress_round as fn(_) -> _);
}

Running the tests immediately turned up a bug:

$ cargo test
running 13 tests
...
thread 'safe' panicked at 'index out of bounds: the len is 67 but the index is 67', ../src/libcollections/vec.rs:1187
test quickcheck_test::qc_roundtrip ... FAILED

failures:

---- quickcheck_test::qc_roundtrip stdout ----
    thread 'quickcheck_test::qc_roundtrip' panicked at '[quickcheck] TEST FAILED (runtime error). Arguments: ([0, 0, 0, 0, 1, 0, 0, 2, 0, 0, 3, 0, 0, 4, 0, 1, 1, 0, 1, 2, 0, 1, 3, 0, 1, 4, 0, 0, 5, 0, 0, 6, 0, 0, 7, 0, 0, 8, 0, 0, 9, 0, 0, 10, 0, 0, 11, 0, 1, 5, 0, 1, 6, 0, 1, 7, 0, 1, 8, 0, 1, 9, 0, 1, 10, 0, 0])
Error: "index out of bounds: the len is 67 but the index is 67"', /home/jer/.cargo/registry/src/github.com-88ac128001ac3a9a/quickcheck-0.2.27/src/tester.rs:116

It would be okay to return an error, but out-of-bounds indexing (and thus panicing) is a clear bug in the library. Luckily, QuickCheck automatically collects the input the test failed on, tries to shrink it down to a minimal example and then displays it. I figured this bug is happening in the compress step, so I added an explicit test case for that:

#[test]
fn quickcheck_found_bug() {
    let inp = vec![0, 0, 0, 0, 1, 0, 0, 2, 0, 0, 3, 0, 0, 4, 0, 1, 1, 0, 1, 2, 0, 1, 3, 0, 1, 4, 0, 0, 5, 0, 0, 6, 0, 0, 7, 0, 0, 8, 0, 0, 9, 0, 0, 10, 0, 0, 11, 0, 1, 5, 0, 1, 6, 0, 1, 7, 0, 1, 8, 0, 1, 9, 0, 1, 10, 0, 0];

    assert_eq!(LzfError::NoCompressionPossible, compress(&inp).unwrap_err());
}

Taking a look at the full stack trace (run RUST_BACKTRACE=1 cargo test) lead to the exact location of the bug. Turns out I was checking the bounds on the wrong variable. I fixed it in 88242ffe. After this fix, I re-run the QuickCheck tests and it discovered a second bug ([0] lead to another out-of-bounds access) and I fixed it in 5b2e8150. I found a third bug, which I (hopefully) fixed, but I don't fully understand how it's happening yet.

Additionally to the above I added QuickCheck tests comparing the Rust functions to the output of the C library. The full changeset is in PR #1 (currently failing tests, because of a broken Clippy on newest nightly).

Now quick, check your own code!

Update 2016-05-13: QuickCheck can be added as a dev dependency, instead of making it optional and activating it with a feature. Additionally it's necessary to use names from the crate (or specify the full path). Thanks to RustMeUp and burntsushi in the reddit thread.

Releasing Rust projects, the automatic way

Tue, 29 Mar 2016 20:47:00 +0200

One of the strength of the Rust ecosystem is its package manager Cargo and the package system crates.io. Pulling in some dependencies is as easy as adding it to your projects' Cargo.toml and running cargo build.

Releasing your own project is nearly as easy. Make sure you got everything working, add a version number in your Cargo.toml and run cargo publish. It will package the code and upload it.

Of course that's not the whole story. For a proper release that people will like to use you want to follow some good practices:

Have tests and make sure they are green. Most people already use Travis CI. The travis-cargo project makes it easy to test all channels (stable, beta, nightly, maybe a specific version), run documentation tests and upload coverage info and documentation.
Keep a changelog. Your software is not done with the first release. It changes, bugs get fixed, new features get introduced. Keeping a changelog helps users to understand what changed from version to version.
Pick a version number. This is not nearly as easy as it sounds. Your project's version number carries a lot of information. Often more than we'd like. The Rust ecosystem recommends to strictly follow semver, but even that has ambiguities and requires a lot of thinking to do the right thing.
Release on the right platforms. Even though crates.io is the package system you want your project in, having a GitHub release is a nice to have. Maybe your project is an application and you want to distributed pre-compiled binaries.

At the moment a lot of people process each of these steps manually. Maybe they have a few scripts lying around that help in reducing the number of errors that can happen. All in all there's still to much manual work required. It does not have to be that way.

Stephan Bönnemann build semantic-release for the npm eco system a while ago. It allows for fully automated package publishing by relying on a few conventions and a lot of automatisation.

I wanted to have a similar thing for the Rust eco system. That's why Jan aka @neinasaservice and I sat down at last year's 32c3 and started hacking on a tool to achieve that.

It took us a while to get something working, but now I can present to you:

🚀 semantic-rs 🚀

What is it?

semantic-rs gives you fully automatic crate publishing. It runs after your tests are finished, analyzes the latest commits, picks out a version number, creates a commit and git tag, creates a release on GitHub and publishes your crate on crates.io.

All you have to do is follow the Angular.js commit message conventions, which are really easy. Your commit message consists of a type, an optional scope, a subject and an optional body.

<type>(<scope>): <subject>
<BLANK LINE>
<body>

The type should be one of the following:

feat: A new feature
fix: A bug fix
docs: Documentation only changes
style: White-space, formatting, missing semi-colons, etc
refactor: A code change that neither fixes a bug nor adds a feature
perf: A code change that improves performance
test: Adding missing tests
chore: Changes to the build process or auxiliary tools/libraries/documentation

The next version number is decided depending on type of commits since the last release. A feat will trigger a minor version bump, a fix a patch version bump. The other types don't cause a release.

However, should you make a breaking change, you need to document this in the commit message as well. Include BREAKING CHANGE in the body of the commit message and add information what changed and how to change existing code to make it work again (if possible). This will then trigger a major version bump.

What works?

The Happy Path.

If everything is configured properly and the tests succeed, semantic-rs will correctly pick a version, add changes to a Changelog.md, create a release commit, tag it, create a GitHub release and publish on crates.io.

The test-project crate is published completely automatically now.

semantic-rs already has some safety features integrated. It will only run when the build is on the master branch (or the branch you configure), and it will make sure that it only runs once on the build leader (which is always the first job in your build matrix). It also waits for the other jobs to finish and succeed before trying to do a release.

What's missing?

In case of problems, semantic-rs will just bail out. That might leave you with changes pushed to GitHub, but not published on crates.io (at worst), or with no visible changes but no new release (at best). We're working hard on making this safer to use with better error reporting.

Installing semantic-rs from source each time your tests run adds significant overhead to the build time, as it must be compiled again and again. In the future we will provide binary releases that you can simpy drop into Travis and it will work.

It's not released on crates.io yet, because we're using a dependency from GitHub. That one should soon be fixed once they push out a release as well.

Now that we got that out of the way, let's see how to actually use it.

How to use it

Right now usage of semantic-rs is not as straight-forward as it can be, we're working on that. To run it on Travis you have to follow these manual steps.

The first job of your build matrix will be used to do the publish, so make sure it is a full build. Make it your stable build to be on the safe side. Your .travis.yml should contain this:

rust:
  - stable
  - beta
  - nightly

Next, install semantic-rs on Travis by adding this to your .travis.yml:

before_script:
  - |
      cargo install --git https://github.com/semantic-rs/semantic-rs --debug &&
      export PATH=$HOME/.cargo/bin:$PATH &&
      git config --global user.name semantic-rs &&
      git config --global user.email semantic@rs

(This installs semantic-rs in debug mode, which is quite a lot faster to compile without significant runtime impact at the moment)

This will also set a git user and mail address, which will be used to create the git tag. You can change this to your own name and email address.

Now add a personal access token from GitHub. It only needs the public_repo permission (unless of course your repository is private).

Add it to your .travis.yml encrypted:

$ travis encrypt GH_TOKEN=<your token here> --add env.global

To release on crates.io you need a token as well. Get it from your account settings and add it to your .travis.yml:

$ travis encrypt CARGO_TOKEN=<your token here> --add env.global

At last make sure semantic-rs runs after the tests succeeds. Add this to the .travis.yml:

after_success:
  - semantic-rs

Make sure to follow the AngularJS Git Commit Message Conventions. semantic-rs will use this convention to decide which should be the next release version.

See the full .travis.yml of our test project.

What's next?

We still have some plans for semantic-rs.

First we need to make it more safe and easy to integrate into a project's workflow.

We also want to look into how we can determine more information about a project to assist the developers. Ideas we have include running integration tests from the previous version to detect breaking changes and statically analyzing code changes to determine their impact. Rust's RFC 1105 already defines the impact certain changes should have. Maybe it is possible to automatically check some of these things.

We would be happy to hear from you. If semantic-rs breaks or otherwise does not fit into your workflow, let us know. Open an issue to discuss this. If you want to use it and have more ideas what is necessary or could be improved, talk to us!

Load your config into your environment

Fri, 04 Mar 2016 12:30:00 +0100

It became quite popular to store certain configuration variables in your environment, to be later loaded by your aplication. This way of having all configuration available is part of the twelve-factor app definition.

The idea is to place your variables in a .env file and load this as environment variables to be accessed by your application. Most of the time you can just plug in one of the dozens of libraries that load this config from a file and your application can fetch the values as normal from the environment.

But sometimes you might want to have this config loaded into your shell or some other interactive tool. That's where you can use dotenv-shell, a small tool, written in Rust. It wraps around rust-dotenv and allows to load the config and then execute a program (your shell by default).

First install the tool:

cargo install dotenv-shell

Create a .env file with your config:

echo "REDIS_URL=redis://localhost:6379" > .env

Then start a shell and you can access the configuration as environment variables:

$ dotenv-shell
$ echo $REDIS_URL
redis://localhost:6379

Of course you can launch whatever tool you want:

$ dotenv-shell /usr/bin/irb
irb(main):001:0> ENV['REDIS_URL']
=> "redis://localhost:6379"

Available on GitHub and as a Crate.

Update:

Only now I learn about another application doing just the same: benv by @timonvonk.

Create GitHub releases with Rust using Hubcaps

Tue, 23 Feb 2016 20:32:00 +0100

For one of my projects I need to access the GitHub API to create releases. Luckily, through reading This Week in Rust #119, I discovered Hubcaps, a library for interfacing with GitHub.

Though it lacks some documentation and is not yet fully finished, it already provides APIs for the relevant parts regarding releases.

On GitHub a release is always associated with a Git tag, but need to be specifially created to be shown on the site with the full description and optional assets attached to them. It is also possible to mark releases as a draft (then it is only visible to repo contributors) or as a pre-release, useful for alpha releases of a library or application.

Once you have a Git tag in your repository the API can be used to create an associated release using the following Rust code:

extern crate hyper;
extern crate hubcaps;

use std::{env, process};
use hyper::Client;
use hubcaps::{Github, ReleaseOptions};

fn main() {
    let token = match env::var("GITHUB_TOKEN").ok() {
        Some(token) => token,
        _ => {
            println!("example missing GITHUB_TOKEN");
            process::exit(1);
        }
    };

    let client = Client::new();
    let github = Github::new("hubcaps/0.1.1", &client, Some(token));

    let user = "username";
    let repo = "my-library";
    let name = "ONE DOT OH";
    let body = "This is a long long body";
    let tag = "v1.0.0";

    let opts = ReleaseOptions::builder(tag)
        .name(name)
        .body(body)
        .commitish("master")
        .draft(false)
        .prerelease(false)
        .build();

    let repo = github.repo(user, repo);
    let release = repo.releases();
    match release.create(&opts) {
        Ok(_) => println!("Release created"),
        Err(e) => println!("Failed to create release: {:?}", e),
    };
}

If you clone Hubcaps and put the above code in a file named releases.rs into the examples/ folder, you can run it with cargo run --example releases. You need to get a personal access token first and set it in your environment (export GITHUB_TOKEN=<your token here>)

Of course it has the repository and tag hard-coded, but this is easy to adapt.

The code was tested with Rust 1.6 and hubcaps 0.1.1

Updated version to work with hubcaps 0.2.0 online.

omnomnom - Parsing ISO8601 dates using nom

Thu, 16 Jul 2015 11:00:00 +0200

Update 2018-07-01: This article was updated in 2018 and is now available as documentation in the iso8601 repository. Thanks to Chris Couzens for the updates.

There are thousands of ways to note down a date and time. The international date format is standardized as ISO8601, though it still allows a widespread of different formats.

The basic format looks like this:

2015-07-02T19:45:00+0100

And that's what we will parse today using nom, a parser combinator library created by Geoffroy Couprie.

The idea is that you write small self-contained parsers, which all do only one simple thing, like parsing the year in our string, and then combine these small parsers to a bigger one to parse the full format. nom comes with a wide variety of small parsers: handling different integers, reading simple byte arrays, optional fields, mapping parsed data over a function, ... Most of them are provided as combinable macros. It's very easy to implement your own small parsers, either by providing a method that handles a short byte buffer or by combining existing parsers.

So let's dive right in and see how to use nom in real code.

Analysis

This is what we want to parse:

2015-07-02T19:45:00+0100

It has several parts we need to parse:

YYYY-MM-DDTHH:MM:SS+OOOO

with the following meaning:

Characters	Meaning
YYYY	The year, can be negative or null and can be extended if necessary
MM	Month from 1 to 12 (0-prefixed)
DD	Day from 1 to 31 (0-prefixed)
T	Separator between date and time
HH	Hour, 0-23 (0-prefixed)
MM	Minutes, 0-59 (0-prefixed)
SS	Seconds, 0-59 (0-prefixed)
OOOO	Timezone offset, separated by a `+` or `-` sign or `Z` for UTC

Parts like the seconds and the timezone offset are optional. Datetime strings without them will default to a zero value for that field. The date parts are separated by a dash (-) and the time parts by a colon (:).

We will built a small parser for each of these parts and at the end combine them to parse a full date time string.

Parsing the date: 2015-07-16

Let's start with the sign. As we need it several times, we create its own parser for that. Parsers are created by giving them a name, stating the return value (or defaulting to a byte slice) and the parser combinators to handle the input.

named!(sign <&[u8], i32>, alt!(
        tag!("-") => { |_| -1 } |
        tag!("+") => { |_| 1 }
        )
    );

First, we parse either a plus or a minus sign. This combines two already existing parsers: tag!, which will match the given byte array (in our case a single character) and alt!, which will try a list of parsers, returning on the first successful one. We can directly map the result of the sub-parsers to either -1 or 1, so we don't need to deal with the byte slice later.

Next we parse the year, which consists of an optional sign and 4 digits (I know, I know, it is possible to extend this to more digits, but let's keep it simple for now).

named!(positive_year  <&[u8], i32>, map!(call!(take_4_digits), buf_to_i32));
named!(pub year <&[u8], i32>, chain!(
        pref: opt!(sign) ~
        y:    positive_year
        ,
        || {
            pref.unwrap_or(1) * y
        }));

A lot of additional stuff here. So let's separate it.

named!(positive_year  <&[u8], i32>, map!(call!(take_4_digits), buf_to_i32));

This creates a new named parser, that again returns the remaining input and an 32-bit integer. To work, it first calls take_4_digits and then maps that result to the corresponding integer (using a small helper function).

take_4_digits is another small helper parser. We also got one for 2 digits:

named!(pub take_4_digits, flat_map!(take!(4), check!(is_digit)));
named!(pub take_2_digits, flat_map!(take!(2), check!(is_digit)));

This takes 4 (or 2) characters from the input and checks that each character is a digit. flat_map! and check! are quite generic, so they are useful for a lot of cases.

named!(pub year <&[u8], i32>, chain!(

The year is also returned as a 32-bit integer (there's a pattern!). Using the chain! macro, we can chain together multiple parsers and work with the sub-results.

        pref: opt!(sign) ~
        y:    positive_year

Our sign is directly followed by 4 digits. It's optional though, that's why we use opt!. ~ is the concatenation operator in the chain! macro. We save the sub-results to variables (pref and y).

        ,
        || {
            pref.unwrap_or(1) * y
        }));

To get the final result, we multiply the prefix (which comes back as either 1 or -1) with the year. Don't forget the , (comma) right before the closure. This is a small syntactic hint for the chain! macro that the mapping function will follow and no more parsers.

We can now successfully parse a year:

assert_eq!(Done(&[][..], 2015), year(b"2015"));
assert_eq!(Done(&[][..], -0333), year(b"-0333"));

Our nom parser will return an IResult. If all went well, we get Done(I,O) with I and O being the appropriate types. For our case I is the same as the input, a buffer slice (&[u8]), and O is the output of the parser itself, an integer (i32). The return value could also be an Error(Err), if something went completely wrong, or Incomplete(u32), requesting more data to be able to satisfy the parser (you can't parse a 4-digit year with only 3 characters input).

Parsing the month and day is a bit easier now: we simply take the digits and map them to an integer:

named!(pub month <&[u8], u32>, map!(call!(take_2_digits), buf_to_u32));
named!(pub day   <&[u8], u32>, map!(call!(take_2_digits), buf_to_u32));

All that's left is combining these 3 parts to parse a full date. Again we can chain the different parsers and map it to some useful value:

named!(pub date <&[u8], Date>, chain!(
        y: year      ~
           tag!("-") ~
        m: month     ~
           tag!("-") ~
        d: day
        ,
        || { Date{ year: y, month: m, day: d }
        }
        ));

Date is a small struct, that can hold the necessary information, just as you would expect.

And it already works:

assert_eq!(Done(&[][..], Date{ year: 2015, month: 7, day: 16  }), date(b"2015-07-16"));
assert_eq!(Done(&[][..], Date{ year: -333, month: 6, day: 11  }), date(b"-0333-06-11"));

Parsing the time: 16:43:52

Next, we parse the time. The individual parts are really simple, just some digits:

named!(pub hour   <&[u8], u32>, map!(call!(take_2_digits), buf_to_u32));
named!(pub minute <&[u8], u32>, map!(call!(take_2_digits), buf_to_u32));
named!(pub second <&[u8], u32>, map!(call!(take_2_digits), buf_to_u32));

Putting them together becomes a bit more complex, as the second part is optional:

named!(pub time <&[u8], Time>, chain!(
        h: hour      ~
           tag!(":") ~
        m: minute    ~
        s: empty_or!(chain!(tag!(":") ~ s:second , || { s }))
        ,
        || { Time{ hour: h,
                   minute: m,
                   second: s.unwrap_or(0),
                   tz_offset: 0 }
           }
        ));

As you can see, even chain! parsers can be nested. The sub-parts then must be mapped once for the inner parser and once into the final value of the outer parser. empty_or! returns an Option. Either None if there is no input left or it applies the nested parser. If this parser doesn't fail, Some(value) is returned.

Our parser now works for simple time information:

assert_eq!(Done(&[][..], Time{ hour: 16, minute: 43, second: 52, tz_offset: 0}), time(b"16:43:52"));
assert_eq!(Done(&[][..], Time{ hour: 16, minute: 43, second:  0, tz_offset: 0}), time(b"16:43"));

But it leaves out one important bit: the timezone.

Parsing the timezone: +0100

2015-07-02T19:45:00-0500
2015-07-02T19:45:00Z
2015-07-02T19:45:00+01

Above are three variants of valid dates with timezones. The timezone in an ISO8601 string is either an appended Z, indicating UTC, or it's separated using a sign (+ or -) and appends the offset from UTC in hours and minutes (with the minutes being optional).

Let's cover the UTC special case first:

named!(timezone_utc <&[u8], i32>, map!(tag!("Z"), |_| 0));

This should look familiar by now. It's a simple Z character, which we map to 0.

The other case is the sign-separated hour and minute offset.

named!(timezone_hour <&[u8], i32>, chain!(
        s: sign ~
        h: hour ~
        m: empty_or!(chain!(tag!(":")? ~ m: minute , || { m }))
        ,
        || { (s * (h as i32) * 3600) + (m.unwrap_or(0) * 60) as i32 }
        ));

We can re-use our already existing parsers and once again chain them to get what we want. The minutes are optional (and might be separated using a colon).

Instead of keeping this as is, we're mapping it to the offset in seconds. We will see why later. We could also just map it to a tuple like
(s, h, m.unwrap_or(0)) and handle conversion at a later point.

Combined we get

named!(timezone <&[u8], i32>, alt!(timezone_utc | timezone_hour));

Putting it all together

We now got individual parsers for the date, the time and the timezone offset.

Putting it all together, our final datetime parser looks quite small and easy to understand:

named!(pub datetime <&[u8], DateTime>, chain!(
        d:   date      ~
             tag!("T") ~
        t:   time      ~
        tzo: empty_or!(call!(timezone))
        ,
        || {
            DateTime{
                date: d,
                time: t.set_tz(tzo.unwrap_or(0)),
            }
        }
        ));

Nothing special anymore. We can now parse all kinds of date strings:

datetime("2007-08-31T16:47+00:00");
datetime("2007-12-24T18:21Z");
datetime("2008-02-01T09:00:22+05");

But it will also parse invalid dates and times:

datetime("2234-13-42T25:70Z");

But this is fine for now. We can handle the actual validation in a later step. For example, we could use chrono, a time library, to handle this for us. Using chrono it's obvious why we already multiplied our timezone offset to be in seconds: this time we can just hand it off to chrono as is.

The full code for this ISO8601 parser is available in easy.rs. The repository also includes a more complex parser, that does some validation while parsing (it checks that the time and date are reasonable values, but it does not check that it is a valid date for example)

What's left?

These simple parsers or even some more complex ones are already usable. At least if you already got all the data at hand and if a simple return value satisfies your needs. But especially for larger and more complex formats like media files reading everything into memory and spitting out a single large value isn't sufficient at all.

nom is prepared for that. Soon it will become as easy as using an object from which nom can Read. For most things you shouldn't worry about that, as a simple BufReader will work.

For the other end of the chain, nom has Consumers. A Consumer handles the complex part of actually requesting data, calling the right sub-parsers and holding the necessary state. This is what you need to build yourself. Internally it's best abstracted using some kind of state machine, so you always know which part of the format to expect next, how to parse it, what to return to the user and so on. Take a look at the MP4 parser, which has an MP4Consumer handling the different parts of the format. Soon my own library, rdb-rs, will have this as well.

Small thing aside: Geoffroy created machine to define a state machine and I got microstate for this.

Why am I doing this?

I'm currently developing rdb-rs, a library to parse and analyze Redis dump files. It's currently limited to parsing and reformatting into several formats and can be mainly used as a CLI utility. But there are projects that could benefit from a nicer API to integrate it into another tool. The current parser is hand-made. It's fast, it's working, but it provides a limited, not very extensible API. I hope to get a proper parser done with nom, that I can build on to provide all necessary methods, while still being super-fast and memory-safe. Work already started, but I'm far from done for now

Thanks to Geoffroy for the discussions, the help and for reading a draft of this post.

The difference of Rust's thread::spawn and thread::scoped

Thu, 05 Mar 2015 14:41:00 +0100

So yesterday I gave a Rust introduction talk at the local hackerspace, CCCAC. The slides are already online. The talk went pretty well and I think I could convince a few people why the ideas in Rust are actually useful. Though I made one mistake in explaining a concurrency feature (see slide 30). As it turns out, the example as I explained it was different from the presented code and one of the attendees actually asked me about it.

// Careful, this example is not quite right.
use std::thread;
use std::sync::{Arc, Mutex};

fn main() {
    let numbers = Arc::new(Mutex::new(vec![1, 2, 3]));

    for i in 0..3 {
        let number = numbers.clone();

        let _ = thread::scoped(|| {
            let mut array = number.lock().unwrap();

            array[i] += 1;

            println!("numbers[{}] is {}", i, array[i]);
        });
    }
}

I used this example to explain why it is necessary to wrap the vector in a mutex and the mutex in an Arc to make it possible to write to it from several threads. The problem lies within the used thread abstraction: thread::scoped.

Spawn a new scoped thread, returning a JoinGuard for it. The join guard can be used to explicitly join the child thread (via join), returning Result, or it will implicitly join the child upon being dropped.

So in the case of the above code each thread is joined right after it was created and thus the threads don't even run concurrently, making the Arc and Mutex unnecessary. The following shortened example will still work, though not show casing what I intended to:

use std::thread;

fn main() {
    let mut numbers = vec![1, 2, 3];

    for i in 0..3 {
        let number = &mut numbers;

        let _ = thread::scoped(|| {
            number[i] += 1;

            println!("numbers[{}] is {}", i, number[i]);
        });
    }
}

There is another in-built threading method: thread::spawn. Its documentation reads:

Spawn a new thread, returning a JoinHandle for it. The join handle will implicitly detach the child thread upon being dropped.

And this is actually what I need to correctly demonstrate what I wanted to: the use of Arc and Mutex to safely share writable access to shared memory through mutual exclusion. The following example works and has all necessary parts:

use std::thread;
use std::sync::{Arc, Mutex};

fn main() {
    let numbers = Arc::new(Mutex::new(vec![1, 2, 3]));

    let mut threads = vec![];
    for i in 0..3 {
        let number = numbers.clone();

        let cur = thread::spawn(move|| {
            let mut array = number.lock().unwrap();

            array[i] += 1;

            println!("numbers[{}] is {}", i, array[i]);
        });
        threads.push(cur);
    }

    for i in threads {
        let _ = i.join();
    }
}

Running it gives the expected output (your output might differ, the order is non-deterministic):

$ rustc concurrency.rs
$ ./concurrency
numbers[1] is 3
numbers[2] is 4
numbers[0] is 2

The Rust book contains a complete chapter on this topic: Concurrency, covering a bit more of the background and also the Channel concept.

Again, thanks to the CCCAC and for all people listening to me and quite some questions afterwards. For all who could not attend: the video should be up soon.

rdb-rs - fast and efficient RDB parsing utility

Thu, 15 Jan 2015 23:50:00 +0100

Ever since I started looking into Rust I knew I needed a bigger project for which I could use it. I released a few small libraries, all based on Redis code/tools, so I figured: Why not a bigger project focused on Redis as well? We already have a nice client library, so I did not need to write one myself. I then thought about writing a Cluster-aware library on top of that, but before I got any real code written I faced some difficulties in how I want the API to look like, so I abandonded the idea for now as well. The next idea was to have another implementation of the Cluster configuration utility redis-trib. Problem though: I did not even finish my attempt in Go. I looked around for more ideas and then I came across the redis-rdb-tools again. This small Python utility can parse, format and analyze Redis' dump files. Its author Sripathi Krishnan also documented the file format and version history of RDB.

Sripathi didn't pay much attention to his project anymore, and so some bugs and feature requests remained unsolved. With the current changes for Redis 3.0 like the Quicklist and a new RDB version, the redis-rdb-tools can't be used anymore for new dump files without patches.

So on last year's 31c3 I took some time and started reading and implementing a RDB parser in Rust based on the documentation. While reading the documentation, the Python code as well as Redis' own rdb.c I took notes and later rewrote and reformatted Sripathi's documentation to be up to date and to also include the latest changes.

Today I release this updated documentation. It's available online at rdb.fnordig.de:

I will keep it updated, should there be need and of course I will improve it if necessary.

At the same time I also open source my port of the redis-rdb-tools to Rust:

rdb-rs

rdb-rs is a library and tool to parse RDB and dump it into another format like JSON or the Redis protocol.

rdb-rs is offered both as a library and as a stand-alone command line tool.

The command line tool can be used to dump an existing RDB file in one of the provided formats:

$ rdb --format json dump.rdb
[{"key":"value"}]
$ rdb --format protocol dump.rdb
*2
$6
SELECT
$1
0
*3
$3
SET
$3
key
$5
value

For now it is nothing too fancy, but it gets the job done. Over the next days and weeks I will improve it, add missing features such as filter options and hopefully also a memory reporter.

Library

Using the library is as easy as calling the rdb::parse function and pass it a stream to read from and a formatter to use.

use std::io::{BufferedReader, File};

let file = File::open(&Path::new("dump.rdb"));
let reader = BufferedReader::new(file);
rdb::parse(reader, rdb::JSONFormatter::new());

rdb-rs brings 4 pre-defined formatters, which can be used:

PlainFormatter: Just plain output for testing
JSONFormatter: JSON-encoded output
NilFormatter: Surpresses all output
ProtocolFormatter: Formats the data in RESP, the Redis Serialization Protocol

Adding your own formatter is as easy as implementing the RdbParseFormatter trait.

Over the next weeks I will rework parts of the library. Currently I'm not too happy with the API offered, especially proper return values and error handling. The code also needs to be refactored to allow for filtering and memory reporting. For the redis-rdb-tools Sripathi also reverse-engineered and documented the memory usage of key-value pairs, which needs updates as well. This will take me some time to bring into the same format.

Finally, I want to say thanks to Sripathi Krishnan for building the rdb-tools and for the proper and very well written documentation. It helped a lot getting this done.

Also thanks to Matt, Andy and Itamar for some input and comments on the small parts of the project I showed them.

The story of my childhood or: Rust on the PSP

Wed, 03 Dec 2014 20:03:00 +0100

Don't care for all the stuff of my childhood? ↓ Head down to see the Rust part

Let's do a time travel.

10 years back, it's 2004. Sony releases the first version of the Playstation Portable. First in Japan, 4 months later in the US and yet another half year later in the EU (including Germany).

I read about this new little device in the autumn of 2004 and I knew immediately: I want this. I also knew: I can't wait a whole year for it to get to the German market. So 13-year old me sat down in front of his parent's PC and started searching through the depths of the internets. Well, mostly Ebay, because that seemed to be the best place to get brand new hardware.

It must have been around January or February that I finally found a more or less trustworthy looking offer that was not too expensive. Just 4 days after clicking Buy on Ebay my PSP finally arrived (yes, just 4 days shipping straight from Japan) and I could unpack it.

But I couldn't do much more. D'oh! The first firmware version on the PSP was quite limited. It only supported WEP, luckily we still had WEP set up at home. The next problem: it didn't even include a web browser yet, so even though I had a Wifi connection, I couldn't use it. On to the next problem: Music and videos. Well, Music worked sort of (this was becoming my most expensive MP3 player at that time). Videos were another story. It only supported MP4 in some weird encoding formats and the only way to get it on the PSP was by using a Memory Stick Pro, one of these tiny little memory cards that only Sony used. They were really expensive, the PSP came with a 32 MB card and I scraped together all my left pocket money to get a 128 MB card. Even back then that was quite limited space. So videos were off the hook as well.

Afterall the PSP was a gaming device, so I should be able to play right? Yeah, actually for me that was a small afterthought, I didn't buy a game together with the PSP. Off to Ebay again, luckily I found the original Ridge Racer, of course the Japanese version. Even now I would say: This was the best game ever released for the PSP. The graphics were astonishing, the music, the gameplay was fantastic, the rising dificulty of tracks and opponents promised hours of fun (and frustration).

While I played this game for hours, other people used their valuable time for far more important things: Hacking the PSP. Successfully.

Firmware 1.0 and 1.5 were exploited just 6 months after the first release. What followed was the big uprise of the Homebrew scene. And I was part of it. Younger me already knew some HTML, some (frontend) Javascript and I just learned C and C++. I wanted to use that knowledge for my first own and publicly released apps! The first step was getting the development toolchain set up. I had an old 600 MHz computer, running Windows XP (I guess, or was it still Windows 98?) The toolchain was based on GCC and was optimized for POSIX environments, which Windows clearly isn't, but someone invested enough time to get it working on cygwin, including a nice one-click installer that did most of the work. This was the time before Google Code and long before GitHub, so you had to download pre-compiled EXE files from one-click hoster sites. The links could be found in one of the dozen forums that existed back then. The whole process to setup Cygwin and the PSP toolchain on my slow computer took nearly 24 hours, so I had to leave the computer on over night (a novelty back then for me). And it failed quite often. So repeat and repeat until it finishes without any failure once.

In the end I got it working and went on developing for the PSP. I learned C, I battled with C, I switched over to C++ and back to C, I threw in some Lua (Lua was one of the first bigger things fully ported to the PSP and thus the best option to quickly develop for the PSP).

The PSP scene was also the first online community I was part of. I became a moderator and blogger of one of the bigger German PSP news sites, portable-news.de. I became friends with people that later offered me my first job as a developer (thanks @g33konaut), that I still see once or twice a year (thanks @atsutane), and others that I never heard anything of again (I'm bad with names, I can't even remember much more from back then).

The biggest released application from my time as a PSP dev is probably the PSP Customizer. A release post on some french forum is still online and of course v1.0 on the good old qj.net. The PSP Customizer could be used to change the boot screen, the menu backgrounds, change menu texts or dump and restore your flash registry (this contained some internal config I guess). Nothing fancy, it was not even originally written by me (though I ended up rewritting most of it). As the text on QJ suggests it was quite known around the community and people actually used it. Sadly I don't have any code of it anymore. It wasn't open source (before GitHub publishing code was really a big hassle), it wasn't even version-controlled. I remember a day where I deleted the whole project by mistake. It must have been just before another release. I spent the whole night trying to recover as much as possible. In the end I had to rewrite large parts off my head (luckily it wasn't the most complicated code).

I also wrote some other small applications, most of which I can't even remember. What I can remember is a PoC of a round-based tactics game, written as a web app (HTML and some crude Javascript magic to work in the limited web browser on the PSP) and inspired by Field Commander. I never finished it or showed it to anyone.

The only application I still have the source code of is the first-ever Twitter client for the PSP: p-twit!. I published the code 5 years ago, probably one of the more horrible pieces of code I've got online. The original p-twit! announcement:

@badboy_: tada! twitter via PSP! release soon (tweet)

That was also the last thing I wrote for the PSP. I think 2008/2009 was also the time the homebrew scene slowed down. My community place, portable-news.de, was dead, more and more devs lost interest, there was nothing entirely new to be done. Smartphones became the new go-to hardware for small applications and fun games developed by people at home.

(More about the homebrew history)

Jumping back to 2014. I learned quite a lot more since my PSP homebrew days. I'm now studying computer science and working part-time as a developer. I write C, C++, Ruby, Javascript. And most recently I started learning Rust.

Then, on the 12th of November I was chilling on the balcony of the hotel on Hawaii, browsing the Internet, when I came across the following post on Reddit:

"Hello World" on a PSP via Rust!

Actually it was nothing more than a picture showing a PSP executing some "Hello World" app. Then some comments came in, including a link to the code and instructions.

As soon as I was back home, I grabbed my PSP, booted up a new virtual machine and installed the psptoolchain (mostly the same hassle as in the old days, it just finishes a lot quicker).

First step: Re-compile and run p-twit. Yip, it works (ok, it starts. Of course the Twitter API is so closed these days, it wouldn't be able to do anything).

Second step: Follow the instructions from the Gist.

jer@psp-dev:~/rust-for-psp$ rustc hello.rs --target psp -lpspdebug -lpspdisplay -lpspge -lm -lc -lpspuser -lgcc
jer@psp-dev:~/rust-for-psp$ psp-fixup-imports hello
Error, no sceModuleInfo section found

D'oh! That didn't work. I played around, tried all kinds of things, then gave up a little frustrated.

A few days later I talked to luqmana in the Rust IRC channel. He turned his Gist into a real repository, rust-psp-hello, including scripts and a Cargofile. Still, no luck for me. Even after re-installing the toolchain several times on different machines I still faced the awful Error, no sceModuleInfo section found after every compile.

After some more tweaking, re-installing, compiling and chatting with luqmana it turns out: He's not using the same toolchain build as I do. He simply used the pre-compiled minpspw package (Update 04.12.2014: Luqman informed me he does in fact built it himself from the minpspw svn, which in turn uses the psptoolchain I tried with first, so absolutely no idea why directly using it failed). Once I grabbed that, the "Hello World" application compiled and ran! Success!

The journey doesn't end here. Only partly satisfied with what I had, I took some free time on a train ride to write a wrapper for the input handling.

Now it's possible to react to user input with this simple piece of code:

let mut pad_data = ctrl::Input::new();

if pad_data.read_changed() {
  match pad_data.pressed_key() {
      Button::Cross => utils::debug_print("X pressed\n"),
      Button::Circle => utils::debug_print("O pressed\n"),
      Button::Triangle => utils::debug_print("TRIANGLE pressed\n"),
      Button::Square => utils::debug_print("SQUARE pressed\n"),
      Button::None => (),
      _ => utils::debug_print("unhandled keypress\n"),
  }
}

The full implementation is up in my fork of rust-psp-hello.

There's still a lot I want to do now that I'm started:

Implement wrappers for the missing libraries (display, wifi)
Figure out a good (Rusty) API for these wrappers
Get my own PSP to connect to a WiFi again (I'm not sure why but it won't connect to my home network)
Figure out why the freshly compiled psptoolchain fails to link in the sceModuleInfo section
Build something cool to present

I'd be happy if anyone wants to help or wants to know more. Just work on the repo (Pull Requests welcome) or drop me a message on Twitter or via email. I'd also be happy to hear storys from others that developed for the PSP 5 or 7 or 9 years ago.

A Rust library for LibLZF

Wed, 15 Oct 2014 11:10:00 +0200

In the last four months I did not produce much open-source code. I was busy writing my Bachelor Thesis. But I was active in the community, I attended several conferences, I read a lot of stuff and I wrote down a lot more things to do.

One of these things was "learn Rust", as you might figure (and writing more). Well, I submitted my thesis on Monday and went on to write some actual Rust code.

So I created lzf-rs, a wrapper around LibLZF, a super small and fast compression library, originally written by Marc Lehmann. LibLZF consists of just 4 files and is super-easy to compile as a small library. LibLZF itself is used by Redis for compressing strings in the RDB, that's how I got to know it. I still don't know how it works, but I want to learn it now (and maybe port the whole code to Rust instead of using a wrapper around the C library?)

I used Rust's Foreign Function Interface to write the wrapper. Accessing C code from within Rust is quite simple. The hard part is choosing the right data types and writing the API in a safe and usable way. I'm not 100% happy with the current API of lzf-rs, so expect it to change. I'm still in the process of figuring out what to expose and what not.

And now, enough of the talking. This is how to use the library:

use lzf;

fn main() {
  let data = "foobar";

  let compressed = lzf::compress(data.as_bytes()).unwrap();

  let decompressed = lzf::decompress(compressed.as_slice(), data.len()).unwrap();
}

I'm happy to get some feedback. It's my first serious Rust code, so prone to bugs and maybe not to Rusty afterall. Ping me on Twitter, drop me an email or open an issue.

first experience with Rust

Tue, 12 Aug 2014 13:25:00 +0200

tl;dr: Rust code for the SIMPLE language available in a git-repo.

I had some spare time yesterday after I finished my last-ever bachelor exam (yey!) so I took my chances and started to write some Rust. Just two days ago I also started to read Tom Stuart's book "Understanding Computation". In Chapter 1, "The Meaning of Programs", he introduces a small and simple programming language called SIMPLE. By using a small-step approach and Ruby as the language of implementation he builds up the virtual machine to execute an already built AST. Go read the first chapter, it's in the sample (and thus free) to get a better idea. The original code is some simple to understand Ruby, just under 200 lines.

It has:

Numbers
Booleans
Variables
Assignments
Addition and Multiplication
less-than comparison
if-else clause
while loop

That's nearly all it takes for a programming language. Usage of the Ruby code looks like this:

>> Machine.new(
    Sequence.new(
      Assign.new(:x, Number.new(3)),
      Assign.new(:res, Add.new(Add.new(Number.new(38), Variable.new(:x)), Variable.new(:y)))
    )
    { :y => Number.new(1) }
  ).run
x = 3; res = 38 + x + y, {:y=>«1»}
do-nothing; res = 38 + x + y, {:y=>«1», :x=>«3»}
res = 38 + x + y, {:y=>«1», :x=>«3»}
res = 38 + 3 + y, {:y=>«1», :x=>«3»}
res = 41 + y, {:y=>«1», :x=>«3»}
res = 41 + 1, {:y=>«1», :x=>«3»}
res = 42, {:y=>«1», :x=>«3»}
do-nothing, {:y=>«1», :x=>«3», :res=>«42»}
=> nil

Because we build the AST by ourself instead of taking a lexer/parser to do the job, the example takes a bit more code than you would think. But it still is simple enough to grasp.

Now the same small sample "programm" in Rust:

let mut env = HashMap::new();
env.insert("y".to_string(), number!(1));

let mut m = Machine::new(
    sequence!(
        assign!("x", number!(3)),
        assign!("res", add!(add!(number!(38), variable!("x")), variable!("y")))
        
      ),
    env);

m.run();
// Add this point `res` in the HashMap will be `Number(42)`

Yep, not much of a difference. It will even output the same steps as above (but without the environment hash). The full code is available in a git repository.

I don't want to explain each and every peace of that code, but instead focus on what I learned about Rust.

Use the docs

There are API docs as well as a Reference Manual and even more general documentation (all links are the nightly versions as I use that). But Rust is in active development. Things are changing fast. Documentation on features might be outdated (especially older blog posts or StackOverflow answers) or even missing (but it gets better with Steven Klabnik working full-time at it). Get familiar with the Standard Library and its docs. Often you need to read that to fully understand how to use a certain data structure or how to implement a specific trait for your code.

Document your code. Rust makes that real easy. Comments are parsed as markdown (you know that already, don't you?) and rustdoc generates a static HTML site from your code (all the API docs are generated this way).

Don't fear the errors

The error and warning messages of the Rust compiler are great. Most often they actually find simple mistakes (wrong variable name, missing self) and actually tell you how to fix it. Other times, even though the message is quite clear, it's not clear how to properly fix your code to compile. But when it compiles, it actually did what I wanted it to do. Most of my errors or "design mistakes" were caught by the explicit type system and safeness checks. Everytime I googled an error message I ended up in the GitHub issue tracker of Rust. Sometimes this helped as it clearly discussed the problem and gave useful answers.

Test your stuff

Rust comes with in-built test support. Simply tag code with #[test] and it will only be compiled and run if you pass the --test flag to the compiler (or use cargo test). It was the first time that I actually ended up doing TDD: First a small test, see it fail, fix the code, see the test pass, refactor if needed. Extend tests. Just super convenient, but make sure to extract your tests in larger code bases (huge files won't help anyone).

Use language features

Pattern matching is nice. It often saves you one or two more if statements. Just deconstruct your object right in the pattern match. Macros are a nice addition, too. The pure Rust syntax might be a little bit verbose sometimes, so just save some keystrokes for your convenience and use a short macro (please, don't over-do it, the docs explicitely say that macros are harder to debug than pure Rust code). Make sure you're editor has proper language support, rust.vim worked quite good for me and did everything right from the start.

I still need to learn about a few Rust features.

I still don't know how to properly seperate your stuff in modules and seperate source files
I have no idea about proper pointer borrowing and other safety features
I've got no idea how lifetimes work and the syntax is not the best in my opinion (' as the marker to indicate a lifetime name).

Summing it all up

Yes, Rust is more verbose than Ruby. Yes, it's much more strict than C or Ruby. Yes, it's not yet ready and changing fast.

But its concepts are nice. More safe and robust code just by more explicit type checks and immutable by default make you think more about the actual application design. They catch errors before you even run the code and make it easy to explicitely state the intent of your code ("code should be self-documenting" ;)).

I don't see me writing lots of Rust code in the near future, but it certainly helps to once in a while have a look at different systems to learn.