rdb-rs - fast and efficient RDB parsing utility

(by )

Ever since I started looking into Rust I knew I needed a bigger project for which I could use it. I released a few small libraries, all based on Redis code/tools, so I figured: Why not a bigger project focused on Redis as well? We already have a nice client library, so I did not need to write one myself. I then thought about writing a Cluster-aware library on top of that, but before I got any real code written I faced some difficulties in how I want the API to look like, so I abandonded the idea for now as well. The next idea was to have another implementation of the Cluster configuration utility redis-trib. Problem though: I did not even finish my attempt in Go. I looked around for more ideas and then I came across the redis-rdb-tools again. This small Python utility can parse, format and analyze Redis’ dump files. Its author Sripathi Krishnan also documented the file format and version history of RDB.

Sripathi didn’t pay much attention to his project anymore, and so some bugs and feature requests remained unsolved. With the current changes for Redis 3.0 like the Quicklist and a new RDB version, the redis-rdb-tools can’t be used anymore for new dump files without patches.

So on last year’s 31c3 I took some time and started reading and implementing a RDB parser in Rust based on the documentation. While reading the documentation, the Python code as well as Redis’ own rdb.c I took notes and later rewrote and reformatted Sripathi’s documentation to be up to date and to also include the latest changes.

Today I release this updated documentation. It’s available online at rdb.fnordig.de:

I will keep it updated, should there be need and of course I will improve it if necessary.

At the same time I also open source my port of the redis-rdb-tools to Rust:

rdb-rs


rdb-rs is a library and tool to parse RDB and dump it into another format like JSON or the Redis protocol.

rdb-rs is offered both as a library and as a stand-alone command line tool.

The command line tool can be used to dump an existing RDB file in one of the provided formats:

$ rdb --format json dump.rdb
[{"key":"value"}]
$ rdb --format protocol dump.rdb
*2
$6
SELECT
$1
0
*3
$3
SET
$3
key
$5
value

For now it is nothing too fancy, but it gets the job done. Over the next days and weeks I will improve it, add missing features such as filter options and hopefully also a memory reporter.

Library

Using the library is as easy as calling the rdb::parse function and pass it a stream to read from and a formatter to use.

use std::io::{BufferedReader, File};

let file = File::open(&Path::new("dump.rdb"));
let reader = BufferedReader::new(file);
rdb::parse(reader, rdb::JSONFormatter::new());

rdb-rs brings 4 pre-defined formatters, which can be used:

Adding your own formatter is as easy as implementing the RdbParseFormatter trait.

Over the next weeks I will rework parts of the library. Currently I’m not too happy with the API offered, especially proper return values and error handling. The code also needs to be refactored to allow for filtering and memory reporting. For the redis-rdb-tools Sripathi also reverse-engineered and documented the memory usage of key-value pairs, which needs updates as well. This will take me some time to bring into the same format.


Finally, I want to say thanks to Sripathi Krishnan for building the rdb-tools and for the proper and very well written documentation. It helped a lot getting this done.

Also thanks to Matt, Andy and Itamar for some input and comments on the small parts of the project I showed them.

2014 in many words

(by )

My year in numbers

Important things first: I got about 12 new T-Shirts for my collection (I didn’t count them).

I had my very first real job interview (I didn’t get the job, but had a nice time in Dublin).
I traveled about 35000 km in a plane, spread across 8 flights, the longest from San Francisco to Frankfurt.
I probably traveled another 5000 km in trains, to and from home to Dortmund, Kamen, Berlin, Hamburg, …
I visited 9 different conferences in 7 different cities across 4 different countries:

  1. FOSDEM in Brussel, Belgium (February)
  2. eurucamp in Berlin (August)
  3. FrOSCon/RedFrogConf in Sankt Augustin (August)
  4. reject.js in Berlin (September)
  5. jsconf EU in Berlin (September)
  6. RailsCamp Germany in Cologne (September)
  7. The next Web Barcamp in Salzburg (October)
  8. IETF 91 in Honolulu, USA (November)
  9. 31c3 in Hamburg (December)

I held 3 talks this year, twice the same Rust talk, both times spontaneous to fill time, and once a Redis talk.
I held 3 different workshops, twice for the OTS DO coaching HTML & CSS and Javascript and once as part of the Barcamp Salzburg, coaching Hoodie.

I published 20 blog posts covering a wide range of topics.
I wrote more than 8200 tweets this year, most of them in reply to someone or as part of a longer rage about computers. I posted 36 photos on Instagram and my ~/photos/2014/ directory now contains about 3000 other photos.

In total I made 380 countable contributions on GitHub (or 879 if you count private repositories as well) and took part in hundreds of issues.
We saw 18 releases of Redis, I was mentioned 4 times in the release notes and became #5 in the list of committers (though only 19 commits so far).
I sent 120 mails to the Redis mailing list (if I can trust my local mail copy).
I learned a new language, Rust, and immediately released 4 libraries.

For the university, I wrote a 15-page paper, containing about 7500 words.
I wrote 6 exams and passed each one.
I wrote a 50-page thesis, containing about 21000 words and made 233 code commits across 4 different repositories for the thesis.
I finally got my Bachelor in Computer Science after 3 years of studying.

I attended 3 different sport tournaments with the Handball team of my university, winning one with my team by accident. For the first time in my life I actually played Beach-Handball, the whole summer long once a week.

2014 was an amazing year for me.

Thanks

All of the above was only possible because I had so many people supporting me, working with me, discussing with me, hosting me, sponsoring me, partying with me or traveling with me. That’s why I want to say thanks to all these people.

Thanks to the great Hoodie community, which I am proud to be a part of. Special thanks to Jan, Gregor, Ola and Lena for the discussions and a few funny hangout sessions. Very special thanks to Lena for hosting the Hoodie Workshop at the Barcamp in Salzburg with me.

Thanks to Hannes and Stephan for inviting me to Salzburg.

Thanks to the Open Tech School Dortmund team, Ola, Carsten, Leif and Hendrik. These are the nicest people, investing their free time to provide awesome talks and workshops for free. They encouraged me to coach at workshops and to finally hold a talk. I know they will keep this going in 2015!

Thanks to my employer rrbone and my boss Dominik for making it possible for me to learn, to work on what I love, and to sponsor me trips to some of the conferences.

Thanks to Lotte for joining Dominik and me on our trip to the US, for telling me quite a bit about beer and, even better, for drinking with me.

And thanks to all the people I forgot to mention explicitely, thanks to my friends and fellow students for making lectures less boring, thanks to my family for welcoming me back everytime I head back home.

There are already a lot of things planned for 2015, so it will be just as busy as this year.

<3

The story of my childhood or: Rust on the PSP

(by )


Don’t care for all the stuff of my childhood? ↓ Head down to see the Rust part


Let’s do a time travel.

10 years back, it’s 2004. Sony releases the first version of the Playstation Portable. First in Japan, 4 months later in the US and yet another half year later in the EU (including Germany).

I read about this new little device in the autumn of 2004 and I knew immediately: I want this. I also knew: I can’t wait a whole year for it to get to the German market. So 13-year old me sat down in front of his parent’s PC and started searching through the depths of the internets. Well, mostly Ebay, because that seemed to be the best place to get brand new hardware.

It must have been around January or February that I finally found a more or less trustworthy looking offer that was not too expensive. Just 4 days after clicking Buy on Ebay my PSP finally arrived (yes, just 4 days shipping straight from Japan) and I could unpack it.

But I couldn’t do much more. D’oh! The first firmware version on the PSP was quite limited. It only supported WEP, luckily we still had WEP set up at home. The next problem: it didn’t even include a web browser yet, so even though I had a Wifi connection, I couldn’t use it. On to the next problem: Music and videos. Well, Music worked sort of (this was becoming my most expensive MP3 player at that time). Videos were another story. It only supported MP4 in some weird encoding formats and the only way to get it on the PSP was by using a Memory Stick Pro, one of these tiny little memory cards that only Sony used. They were really expensive, the PSP came with a 32 MB card and I scraped together all my left pocket money to get a 128 MB card. Even back then that was quite limited space. So videos were off the hook as well.

Afterall the PSP was a gaming device, so I should be able to play right? Yeah, actually for me that was a small afterthought, I didn’t buy a game together with the PSP. Off to Ebay again, luckily I found the original Ridge Racer, of course the Japanese version. Even now I would say: This was the best game ever released for the PSP. The graphics were astonishing, the music, the gameplay was fantastic, the rising dificulty of tracks and opponents promised hours of fun (and frustration).

While I played this game for hours, other people used their valuable time for far more important things: Hacking the PSP. Successfully.

Firmware 1.0 and 1.5 were exploited just 6 months after the first release. What followed was the big uprise of the Homebrew scene. And I was part of it. Younger me already knew some HTML, some (frontend) Javascript and I just learned C and C++. I wanted to use that knowledge for my first own and publicly released apps! The first step was getting the development toolchain set up. I had an old 600 MHz computer, running Windows XP (I guess, or was it still Windows 98?) The toolchain was based on GCC and was optimized for POSIX environments, which Windows clearly isn’t, but someone invested enough time to get it working on cygwin, including a nice one-click installer that did most of the work. This was the time before Google Code and long before GitHub, so you had to download pre-compiled EXE files from one-click hoster sites. The links could be found in one of the dozen forums that existed back then. The whole process to setup Cygwin and the PSP toolchain on my slow computer took nearly 24 hours, so I had to leave the computer on over night (a novelty back then for me). And it failed quite often. So repeat and repeat until it finishes without any failure once.

In the end I got it working and went on developing for the PSP. I learned C, I battled with C, I switched over to C++ and back to C, I threw in some Lua (Lua was one of the first bigger things fully ported to the PSP and thus the best option to quickly develop for the PSP).

The PSP scene was also the first online community I was part of. I became a moderator and blogger of one of the bigger German PSP news sites, portable-news.de. I became friends with people that later offered me my first job as a developer (thanks @g33konaut), that I still see once or twice a year (thanks @atsutane), and others that I never heard anything of again (I’m bad with names, I can’t even remember much more from back then).

The biggest released application from my time as a PSP dev is probably the PSP Customizer. A release post on some french forum is still online and of course v1.0 on the good old qj.net. The PSP Customizer could be used to change the boot screen, the menu backgrounds, change menu texts or dump and restore your flash registry (this contained some internal config I guess). Nothing fancy, it was not even originally written by me (though I ended up rewritting most of it). As the text on QJ suggests it was quite known around the community and people actually used it. Sadly I don’t have any code of it anymore. It wasn’t open source (before GitHub publishing code was really a big hassle), it wasn’t even version-controlled. I remember a day where I deleted the whole project by mistake. It must have been just before another release. I spent the whole night trying to recover as much as possible. In the end I had to rewrite large parts off my head (luckily it wasn’t the most complicated code).

I also wrote some other small applications, most of which I can’t even remember. What I can remember is a PoC of a round-based tactics game, written as a web app (HTML and some crude Javascript magic to work in the limited web browser on the PSP) and inspired by Field Commander. I never finished it or showed it to anyone.

The only application I still have the source code of is the first-ever Twitter client for the PSP: p-twit!. I published the code 5 years ago, probably one of the more horrible pieces of code I’ve got online. The original p-twit! announcement:

@badboy_: tada! twitter via PSP! release soon (tweet)

That was also the last thing I wrote for the PSP. I think 2008/2009 was also the time the homebrew scene slowed down. My community place, portable-news.de, was dead, more and more devs lost interest, there was nothing entirely new to be done. Smartphones became the new go-to hardware for small applications and fun games developed by people at home.

(More about the homebrew history)


Jumping back to 2014. I learned quite a lot more since my PSP homebrew days. I’m now studying computer science and working part-time as a developer. I write C, C++, Ruby, Javascript. And most recently I started learning Rust.

Then, on the 12th of November I was chilling on the balcony of the hotel on Hawaii, browsing the Internet, when I came across the following post on Reddit:

“Hello World” on a PSP via Rust!

Actually it was nothing more than a picture showing a PSP executing some “Hello World” app. Then some comments came in, including a link to the code and instructions.

As soon as I was back home, I grabbed my PSP, booted up a new virtual machine and installed the psptoolchain (mostly the same hassle as in the old days, it just finishes a lot quicker).

First step: Re-compile and run p-twit. Yip, it works (ok, it starts. Of course the Twitter API is so closed these days, it wouldn’t be able to do anything).

p-twit runs again

Second step: Follow the instructions from the Gist.

jer@psp-dev:~/rust-for-psp$ rustc hello.rs --target psp -lpspdebug -lpspdisplay -lpspge -lm -lc -lpspuser -lgcc
jer@psp-dev:~/rust-for-psp$ psp-fixup-imports hello
Error, no sceModuleInfo section found

D’oh! That didn’t work. I played around, tried all kinds of things, then gave up a little frustrated.

A few days later I talked to luqmana in the Rust IRC channel. He turned his Gist into a real repository, rust-psp-hello, including scripts and a Cargofile. Still, no luck for me. Even after re-installing the toolchain several times on different machines I still faced the awful Error, no sceModuleInfo section found after every compile.

After some more tweaking, re-installing, compiling and chatting with luqmana it turns out: He’s not using the same toolchain build as I do. He simply used the pre-compiled minpspw package (Update 04.12.2014: Luqman informed me he does in fact built it himself from the minpspw svn, which in turn uses the psptoolchain I tried with first, so absolutely no idea why directly using it failed). Once I grabbed that, the “Hello World” application compiled and ran! Success!

From Rust with love

The journey doesn’t end here. Only partly satisfied with what I had, I took some free time on a train ride to write a wrapper for the input handling.

Press X, press O

Now it’s possible to react to user input with this simple piece of code:

let mut pad_data = ctrl::Input::new();

if pad_data.read_changed() {
  match pad_data.pressed_key() {
    Button::Cross => utils::debug_print("X pressed\n"),
      Button::Circle => utils::debug_print("O pressed\n"),
      Button::Triangle => utils::debug_print("TRIANGLE pressed\n"),
      Button::Square => utils::debug_print("SQUARE pressed\n"),
      Button::None => (),
      _ => utils::debug_print("unhandled keypress\n"),
  }
}

The full implementation is up in my fork of rust-psp-hello.

There’s still a lot I want to do now that I’m started:

I’d be happy if anyone wants to help or wants to know more. Just work on the repo (Pull Requests welcome) or drop me a message on Twitter or via email. I’d also be happy to hear storys from others that developed for the PSP 5 or 7 or 9 years ago.

How to not write a Bachelor Thesis

(by )

Iteration 1

… and how I got it finished anyway.

From May to October I was busy working on my Bachelor Thesis. I finally handed it in right on time on the 13th of October. But especially the last 4 weeks were the busiest and hardest of 3 years of studying.

As with everything you do, you have a lot of plans when you start; you make a time schedule, you promise to take notes, you promise to work on it every day. I broke each and every promise I made before. Now that I’m done I know that I could have done better and so in the following I will expand on all the things I encountered in these 5 months of work.

The first and most important advice I can give: Start writing from day 1. Whatever you write will probably not end up in the final thesis, your writing doesn’t even need to be on topic, but getting 50 pages written is not an easy task if you don’t train for it. Writing technical texts is different from a novel or a short blog post. Work on that.

The next thing: buy a paper notebook dedicated for your notes on the thesis topic. I kept mine with me and actually used it. Keeping meeting minutes, tasks to do for the week, simulation results or general ideas (though most of the stuff was added in the last 4 weeks).

Everything done

Once you know your topic, start the research. That includes finding documentation, papers and whatever seems necessary for your topic. Make sure to note down your sources immediately. Read papers and summarize them for yourself and discuss them with your supervisor or other students. Get familiar with scholar.google.com and other paper search sites. Keep bookmarks to sites you visited and might need. Consider saving them offline just to be safe.

What helped me a lot in understanding the topic and getting ideas how to write about it was explaining it to other people, especially non-tech people. People without any background in your topic will be absolutely clueless, they can point out missing pieces or inaccurities in your explanations and they tend to ask the most basic question, which you would otherwise skip (because you know the answer right away). Of course this won’t help when it comes to the details of your actual topic. For this, find students working on similar topics. Discuss your approach with them. My supervisor forced us to have a meeting once a week. We were 4 thesis students with the same broader topic. Even though these meetings tend to be quite long, we often could help each other in nailing down a good solution to an implementation problem.

At the beginning of my thesis I had to hold a talk at the chair introducing my topic and laying out a basic time schedule. I planned 2 months for the implementation and another 6-8 weeks for the actual writing. In the end I had about 4 weeks for writing, while at the same time I was still chasing bugs in my implementation and kicking of the evaluation runs. That resulted in really long working days. My advice: keep track of your time. Note down how long you worked each day and what you did. 12 or more hours working a day don’t help. You get much more done in 8 productive hours, with breaks and some free time in between.

I guess no matter what you do, the last weeks will still be busy. What got me most focussed on the task at hand is the Pomodoro technique. 25 minutes of concentrated work, no distractions, no drinking, no eating, no running around. 5 minutes break. Check Twitter, Facebook, Mails, whatever. Then repeat. I use this technique for 3 years of studying now, and it also works for coding and writing.

The printed thesis

I had to write my thesis in English, but I’m far from a perfect English speaker or writer. I now know that I should practice much more often (and I try to do that by getting more blog posts out). Together with the first books on the topic I got from the university library I also got a book on proper technical writing. This small book had a lot of tips on how to phrase things, on typical mistakes, be it grammar or spelling. And don’t forget to use spell checking tools. They can catch simple typos, british/american differences and just wrong words.

If you write at a technical university it is not unlikely you will use LaTeX for your thesis. Make sure to set up your build chain early. You don’t want to deal with weird errors, broken packages and all sorts of layout fuckups 2 days before you need to submit your thesis.

One last important thing: Conferences are awesome. You meet nice people, you learn about a lot of interesting stuff and you most often get great motivation. But 5 conferences in 2 months, when you should be writing your thesis, might not be the best idea.
I did this. I went to Berlin twice, I was in St. Augustin for a whole weekend and in Cologne for another. I got near to nothing done for the thesis (besides having a great summer :D). So keep this in mind when planning ahead.

In the end I got my thesis finished and I am now a Bachelor of Science. The last 3 years were a great time and I plan to extend it with atleast another 2 years reaching for a Master’s degree.

Review: Redis Applied Design Patterns

(by )

In the mid of October I was approached by Packt Publishing asking if I would be willing to review a newly released eBook called Redis Applied Design Patterns by Arun Chinnachamy. I agreed to review the 100 page eBook given that I’m free to criticize the book as I see fit.

It took me some time to finish the book, but flights to Salzburg, New York and Honolulu meant enough (offline) time to read everything in detail and write the review. So here it is.

What is it about?

The book gives an overview of different use cases of Redis. It is definitely not for beginners, but for people that are already familiar how to setup Redis and how to use it. The books starts with the basics of a key-value store and how it differs from relational databases. It then goes on to discuss a wide variety of use cases. From using Redis as a cache store, using it as the main data store for e-commerce inventories or a comment system to using it for autosuggestion, real-time analysis and game leaderboards. Each chapter of the book discusses one of the named use cases, explaining the used Redis commands and showing code and ideas on how to implement the given system.

The good things

Having different use cases documented and accessible in a concise way is really useful. The book shows a wide range of these use cases and not only discusses the simplest implementation, but also tries to give an outlook on further iterations for the given requirements, building a increasingly complex system step by step.

It doesn’t focus on a single aspect of Redis and shows off a lot of the available commands and data types. Included are explanations of the used commands as they are described on redis.io, as well as links to the documentation for people that want a deeper understanding. The included code samples are kept short and are a mix of PHP code, implementing the discussed use case, and single Redis commands, showing how they work and what values they return. Even for programmers without knowledge of PHP, it should be possible to understand the implementation and adopt it for the language of choice.

The chapters are short and self-contained and the book is short enough to be read in just one afternoon (it took me longer because I took a lot of notes to come up with this review).

The not so good things

Quickly after I started reading the book I was a bit baffled about the overall text and code quality. The very first thing I noticed was that some of the code examples used slightly different key names as the describing text itself. This didn’t block the understanding of the overall concept, but is certainly a thing that needs to be improved.

Slightly off-putting are the code samples itself, especially the ones that are larger than just a couple of lines. They use an inconsistent code style (CamelCase vs. snake_case in one single line) and could use just a little more explanations here and there (or the text should refer to lines more directly). Some code examples are just plain wrong. The more advanced problem solutions are only briefly mentioned in the text and no code examples are shown.

It is nice to have the used Redis commands explained once, but the book mostly copies the official documentation for this. It also mentions the time complexity of each command, but nowhere in the book it is explained what O(n) actually means (Big O notation explained in the Wikipedia).

As mentioned, the chapters are mostly independent of each other, which makes it easy to just jump in and read them in whatever order you want. But as the first 3 chapters explain how things work in Redis, other chapters could easily refer back to them. Instead latter chapters discuss things again or link only to the online documentation. One chapter is actually re-implementing the same stuff as in the chapter before. Plain references between chapters could save this duplication and help the reader to jump between chapters.

I won’t discuss each chapter in detail, but there are two things that I need to mention here. First, the chapter about an auto-suggest system uses the old approach of using n-grams in Sorted Sets, but since version 2.8.9 (released in April 2014) Redis has powerful lexicographic commands like ZRANGEBYLEX, which are not mentioned in the book at all (though I don’t know when the book was written, maybe a future version can include this). Second, the summary of the last chapter talks about scaling Redis across multiple machines and mentions Sentinel as under development and usable for this, which is just incorrect (for the ones that don’t know: Sentinel is considered stable since 2.8 and is used for monitoring and failover, if you want to shard your data use Redis Cluster). There are a few other minor incorrect things or explanations that leave out some important details.

If you are already familiar with Redis, these things are obvious, but if you only know the basics and want to read up on possible implementations, the errors and mistakes will misguide you. This makes it hard to recommend the book as a useful resource on Redis design patterns.

Conclusion

Redis Applied Design Patterns is a short little book explaining different use cases of Redis. It gives insight into different techniques and implementations of high-level features. But don’t expect to be a super Redis pro after reading it. The given examples are only touching the surface and show the most basic implementations. The book should definitely be revised to fix several issues in quality and correctness. A more in-depth discussion on more complex implementations and revised examples would make it recommendable, but at this stage it is not.

In case you want to buy the book anyway, use this link.