hiredis is up to date

(by )

Back in December 2014 antirez reached out to the community, to find a new maintainer of hiredis. In a joined effort Michael, Matt and me took on the effort and just 2 weeks ago Matt released Version 0.12.1, after just two years without a proper release.

This weekend I got in contact with Pieter who handed over access to all major packages based on hiredis to me. Sunday I pushed out new versions for all of them:

All of them include only non-breaking changed and an upgrade of the underlying hiredis code, so all additional libraries and applications depending on them should just work after an upgrade. If not, please open a ticket.

I again want to say thank you to Pieter who maintained all of these packages in the past. Also thank you to Matt and Michael, who helped push hiredis forward.

rdb-rs - fast and efficient RDB parsing utility

(by )

Ever since I started looking into Rust I knew I needed a bigger project for which I could use it. I released a few small libraries, all based on Redis code/tools, so I figured: Why not a bigger project focused on Redis as well? We already have a nice client library, so I did not need to write one myself. I then thought about writing a Cluster-aware library on top of that, but before I got any real code written I faced some difficulties in how I want the API to look like, so I abandonded the idea for now as well. The next idea was to have another implementation of the Cluster configuration utility redis-trib. Problem though: I did not even finish my attempt in Go. I looked around for more ideas and then I came across the redis-rdb-tools again. This small Python utility can parse, format and analyze Redis’ dump files. Its author Sripathi Krishnan also documented the file format and version history of RDB.

Sripathi didn’t pay much attention to his project anymore, and so some bugs and feature requests remained unsolved. With the current changes for Redis 3.0 like the Quicklist and a new RDB version, the redis-rdb-tools can’t be used anymore for new dump files without patches.

So on last year’s 31c3 I took some time and started reading and implementing a RDB parser in Rust based on the documentation. While reading the documentation, the Python code as well as Redis’ own rdb.c I took notes and later rewrote and reformatted Sripathi’s documentation to be up to date and to also include the latest changes.

Today I release this updated documentation. It’s available online at rdb.fnordig.de:

I will keep it updated, should there be need and of course I will improve it if necessary.

At the same time I also open source my port of the redis-rdb-tools to Rust:

rdb-rs


rdb-rs is a library and tool to parse RDB and dump it into another format like JSON or the Redis protocol.

rdb-rs is offered both as a library and as a stand-alone command line tool.

The command line tool can be used to dump an existing RDB file in one of the provided formats:

$ rdb --format json dump.rdb
[{"key":"value"}]
$ rdb --format protocol dump.rdb
*2
$6
SELECT
$1
0
*3
$3
SET
$3
key
$5
value

For now it is nothing too fancy, but it gets the job done. Over the next days and weeks I will improve it, add missing features such as filter options and hopefully also a memory reporter.

Library

Using the library is as easy as calling the rdb::parse function and pass it a stream to read from and a formatter to use.

use std::io::{BufferedReader, File};

let file = File::open(&Path::new("dump.rdb"));
let reader = BufferedReader::new(file);
rdb::parse(reader, rdb::JSONFormatter::new());

rdb-rs brings 4 pre-defined formatters, which can be used:

Adding your own formatter is as easy as implementing the RdbParseFormatter trait.

Over the next weeks I will rework parts of the library. Currently I’m not too happy with the API offered, especially proper return values and error handling. The code also needs to be refactored to allow for filtering and memory reporting. For the redis-rdb-tools Sripathi also reverse-engineered and documented the memory usage of key-value pairs, which needs updates as well. This will take me some time to bring into the same format.


Finally, I want to say thanks to Sripathi Krishnan for building the rdb-tools and for the proper and very well written documentation. It helped a lot getting this done.

Also thanks to Matt, Andy and Itamar for some input and comments on the small parts of the project I showed them.

2014 in many words

(by )

My year in numbers

Important things first: I got about 12 new T-Shirts for my collection (I didn’t count them).

I had my very first real job interview (I didn’t get the job, but had a nice time in Dublin).
I traveled about 35000 km in a plane, spread across 8 flights, the longest from San Francisco to Frankfurt.
I probably traveled another 5000 km in trains, to and from home to Dortmund, Kamen, Berlin, Hamburg, …
I visited 9 different conferences in 7 different cities across 4 different countries:

  1. FOSDEM in Brussel, Belgium (February)
  2. eurucamp in Berlin (August)
  3. FrOSCon/RedFrogConf in Sankt Augustin (August)
  4. reject.js in Berlin (September)
  5. jsconf EU in Berlin (September)
  6. RailsCamp Germany in Cologne (September)
  7. The next Web Barcamp in Salzburg (October)
  8. IETF 91 in Honolulu, USA (November)
  9. 31c3 in Hamburg (December)

I held 3 talks this year, twice the same Rust talk, both times spontaneous to fill time, and once a Redis talk.
I held 3 different workshops, twice for the OTS DO coaching HTML & CSS and Javascript and once as part of the Barcamp Salzburg, coaching Hoodie.

I published 20 blog posts covering a wide range of topics.
I wrote more than 8200 tweets this year, most of them in reply to someone or as part of a longer rage about computers. I posted 36 photos on Instagram and my ~/photos/2014/ directory now contains about 3000 other photos.

In total I made 380 countable contributions on GitHub (or 879 if you count private repositories as well) and took part in hundreds of issues.
We saw 18 releases of Redis, I was mentioned 4 times in the release notes and became #5 in the list of committers (though only 19 commits so far).
I sent 120 mails to the Redis mailing list (if I can trust my local mail copy).
I learned a new language, Rust, and immediately released 4 libraries.

For the university, I wrote a 15-page paper, containing about 7500 words.
I wrote 6 exams and passed each one.
I wrote a 50-page thesis, containing about 21000 words and made 233 code commits across 4 different repositories for the thesis.
I finally got my Bachelor in Computer Science after 3 years of studying.

I attended 3 different sport tournaments with the Handball team of my university, winning one with my team by accident. For the first time in my life I actually played Beach-Handball, the whole summer long once a week.

2014 was an amazing year for me.

Thanks

All of the above was only possible because I had so many people supporting me, working with me, discussing with me, hosting me, sponsoring me, partying with me or traveling with me. That’s why I want to say thanks to all these people.

Thanks to the great Hoodie community, which I am proud to be a part of. Special thanks to Jan, Gregor, Ola and Lena for the discussions and a few funny hangout sessions. Very special thanks to Lena for hosting the Hoodie Workshop at the Barcamp in Salzburg with me.

Thanks to Hannes and Stephan for inviting me to Salzburg.

Thanks to the Open Tech School Dortmund team, Ola, Carsten, Leif and Hendrik. These are the nicest people, investing their free time to provide awesome talks and workshops for free. They encouraged me to coach at workshops and to finally hold a talk. I know they will keep this going in 2015!

Thanks to my employer rrbone and my boss Dominik for making it possible for me to learn, to work on what I love, and to sponsor me trips to some of the conferences.

Thanks to Lotte for joining Dominik and me on our trip to the US, for telling me quite a bit about beer and, even better, for drinking with me.

And thanks to all the people I forgot to mention explicitely, thanks to my friends and fellow students for making lectures less boring, thanks to my family for welcoming me back everytime I head back home.

There are already a lot of things planned for 2015, so it will be just as busy as this year.

<3

The story of my childhood or: Rust on the PSP

(by )


Don’t care for all the stuff of my childhood? ↓ Head down to see the Rust part


Let’s do a time travel.

10 years back, it’s 2004. Sony releases the first version of the Playstation Portable. First in Japan, 4 months later in the US and yet another half year later in the EU (including Germany).

I read about this new little device in the autumn of 2004 and I knew immediately: I want this. I also knew: I can’t wait a whole year for it to get to the German market. So 13-year old me sat down in front of his parent’s PC and started searching through the depths of the internets. Well, mostly Ebay, because that seemed to be the best place to get brand new hardware.

It must have been around January or February that I finally found a more or less trustworthy looking offer that was not too expensive. Just 4 days after clicking Buy on Ebay my PSP finally arrived (yes, just 4 days shipping straight from Japan) and I could unpack it.

But I couldn’t do much more. D’oh! The first firmware version on the PSP was quite limited. It only supported WEP, luckily we still had WEP set up at home. The next problem: it didn’t even include a web browser yet, so even though I had a Wifi connection, I couldn’t use it. On to the next problem: Music and videos. Well, Music worked sort of (this was becoming my most expensive MP3 player at that time). Videos were another story. It only supported MP4 in some weird encoding formats and the only way to get it on the PSP was by using a Memory Stick Pro, one of these tiny little memory cards that only Sony used. They were really expensive, the PSP came with a 32 MB card and I scraped together all my left pocket money to get a 128 MB card. Even back then that was quite limited space. So videos were off the hook as well.

Afterall the PSP was a gaming device, so I should be able to play right? Yeah, actually for me that was a small afterthought, I didn’t buy a game together with the PSP. Off to Ebay again, luckily I found the original Ridge Racer, of course the Japanese version. Even now I would say: This was the best game ever released for the PSP. The graphics were astonishing, the music, the gameplay was fantastic, the rising dificulty of tracks and opponents promised hours of fun (and frustration).

While I played this game for hours, other people used their valuable time for far more important things: Hacking the PSP. Successfully.

Firmware 1.0 and 1.5 were exploited just 6 months after the first release. What followed was the big uprise of the Homebrew scene. And I was part of it. Younger me already knew some HTML, some (frontend) Javascript and I just learned C and C++. I wanted to use that knowledge for my first own and publicly released apps! The first step was getting the development toolchain set up. I had an old 600 MHz computer, running Windows XP (I guess, or was it still Windows 98?) The toolchain was based on GCC and was optimized for POSIX environments, which Windows clearly isn’t, but someone invested enough time to get it working on cygwin, including a nice one-click installer that did most of the work. This was the time before Google Code and long before GitHub, so you had to download pre-compiled EXE files from one-click hoster sites. The links could be found in one of the dozen forums that existed back then. The whole process to setup Cygwin and the PSP toolchain on my slow computer took nearly 24 hours, so I had to leave the computer on over night (a novelty back then for me). And it failed quite often. So repeat and repeat until it finishes without any failure once.

In the end I got it working and went on developing for the PSP. I learned C, I battled with C, I switched over to C++ and back to C, I threw in some Lua (Lua was one of the first bigger things fully ported to the PSP and thus the best option to quickly develop for the PSP).

The PSP scene was also the first online community I was part of. I became a moderator and blogger of one of the bigger German PSP news sites, portable-news.de. I became friends with people that later offered me my first job as a developer (thanks @g33konaut), that I still see once or twice a year (thanks @atsutane), and others that I never heard anything of again (I’m bad with names, I can’t even remember much more from back then).

The biggest released application from my time as a PSP dev is probably the PSP Customizer. A release post on some french forum is still online and of course v1.0 on the good old qj.net. The PSP Customizer could be used to change the boot screen, the menu backgrounds, change menu texts or dump and restore your flash registry (this contained some internal config I guess). Nothing fancy, it was not even originally written by me (though I ended up rewritting most of it). As the text on QJ suggests it was quite known around the community and people actually used it. Sadly I don’t have any code of it anymore. It wasn’t open source (before GitHub publishing code was really a big hassle), it wasn’t even version-controlled. I remember a day where I deleted the whole project by mistake. It must have been just before another release. I spent the whole night trying to recover as much as possible. In the end I had to rewrite large parts off my head (luckily it wasn’t the most complicated code).

I also wrote some other small applications, most of which I can’t even remember. What I can remember is a PoC of a round-based tactics game, written as a web app (HTML and some crude Javascript magic to work in the limited web browser on the PSP) and inspired by Field Commander. I never finished it or showed it to anyone.

The only application I still have the source code of is the first-ever Twitter client for the PSP: p-twit!. I published the code 5 years ago, probably one of the more horrible pieces of code I’ve got online. The original p-twit! announcement:

@badboy_: tada! twitter via PSP! release soon (tweet)

That was also the last thing I wrote for the PSP. I think 2008/2009 was also the time the homebrew scene slowed down. My community place, portable-news.de, was dead, more and more devs lost interest, there was nothing entirely new to be done. Smartphones became the new go-to hardware for small applications and fun games developed by people at home.

(More about the homebrew history)


Jumping back to 2014. I learned quite a lot more since my PSP homebrew days. I’m now studying computer science and working part-time as a developer. I write C, C++, Ruby, Javascript. And most recently I started learning Rust.

Then, on the 12th of November I was chilling on the balcony of the hotel on Hawaii, browsing the Internet, when I came across the following post on Reddit:

“Hello World” on a PSP via Rust!

Actually it was nothing more than a picture showing a PSP executing some “Hello World” app. Then some comments came in, including a link to the code and instructions.

As soon as I was back home, I grabbed my PSP, booted up a new virtual machine and installed the psptoolchain (mostly the same hassle as in the old days, it just finishes a lot quicker).

First step: Re-compile and run p-twit. Yip, it works (ok, it starts. Of course the Twitter API is so closed these days, it wouldn’t be able to do anything).

p-twit runs again

Second step: Follow the instructions from the Gist.

jer@psp-dev:~/rust-for-psp$ rustc hello.rs --target psp -lpspdebug -lpspdisplay -lpspge -lm -lc -lpspuser -lgcc
jer@psp-dev:~/rust-for-psp$ psp-fixup-imports hello
Error, no sceModuleInfo section found

D’oh! That didn’t work. I played around, tried all kinds of things, then gave up a little frustrated.

A few days later I talked to luqmana in the Rust IRC channel. He turned his Gist into a real repository, rust-psp-hello, including scripts and a Cargofile. Still, no luck for me. Even after re-installing the toolchain several times on different machines I still faced the awful Error, no sceModuleInfo section found after every compile.

After some more tweaking, re-installing, compiling and chatting with luqmana it turns out: He’s not using the same toolchain build as I do. He simply used the pre-compiled minpspw package (Update 04.12.2014: Luqman informed me he does in fact built it himself from the minpspw svn, which in turn uses the psptoolchain I tried with first, so absolutely no idea why directly using it failed). Once I grabbed that, the “Hello World” application compiled and ran! Success!

From Rust with love

The journey doesn’t end here. Only partly satisfied with what I had, I took some free time on a train ride to write a wrapper for the input handling.

Press X, press O

Now it’s possible to react to user input with this simple piece of code:

let mut pad_data = ctrl::Input::new();

if pad_data.read_changed() {
  match pad_data.pressed_key() {
    Button::Cross => utils::debug_print("X pressed\n"),
      Button::Circle => utils::debug_print("O pressed\n"),
      Button::Triangle => utils::debug_print("TRIANGLE pressed\n"),
      Button::Square => utils::debug_print("SQUARE pressed\n"),
      Button::None => (),
      _ => utils::debug_print("unhandled keypress\n"),
  }
}

The full implementation is up in my fork of rust-psp-hello.

There’s still a lot I want to do now that I’m started:

I’d be happy if anyone wants to help or wants to know more. Just work on the repo (Pull Requests welcome) or drop me a message on Twitter or via email. I’d also be happy to hear storys from others that developed for the PSP 5 or 7 or 9 years ago.

How to not write a Bachelor Thesis

(by )

Iteration 1

… and how I got it finished anyway.

From May to October I was busy working on my Bachelor Thesis. I finally handed it in right on time on the 13th of October. But especially the last 4 weeks were the busiest and hardest of 3 years of studying.

As with everything you do, you have a lot of plans when you start; you make a time schedule, you promise to take notes, you promise to work on it every day. I broke each and every promise I made before. Now that I’m done I know that I could have done better and so in the following I will expand on all the things I encountered in these 5 months of work.

The first and most important advice I can give: Start writing from day 1. Whatever you write will probably not end up in the final thesis, your writing doesn’t even need to be on topic, but getting 50 pages written is not an easy task if you don’t train for it. Writing technical texts is different from a novel or a short blog post. Work on that.

The next thing: buy a paper notebook dedicated for your notes on the thesis topic. I kept mine with me and actually used it. Keeping meeting minutes, tasks to do for the week, simulation results or general ideas (though most of the stuff was added in the last 4 weeks).

Everything done

Once you know your topic, start the research. That includes finding documentation, papers and whatever seems necessary for your topic. Make sure to note down your sources immediately. Read papers and summarize them for yourself and discuss them with your supervisor or other students. Get familiar with scholar.google.com and other paper search sites. Keep bookmarks to sites you visited and might need. Consider saving them offline just to be safe.

What helped me a lot in understanding the topic and getting ideas how to write about it was explaining it to other people, especially non-tech people. People without any background in your topic will be absolutely clueless, they can point out missing pieces or inaccurities in your explanations and they tend to ask the most basic question, which you would otherwise skip (because you know the answer right away). Of course this won’t help when it comes to the details of your actual topic. For this, find students working on similar topics. Discuss your approach with them. My supervisor forced us to have a meeting once a week. We were 4 thesis students with the same broader topic. Even though these meetings tend to be quite long, we often could help each other in nailing down a good solution to an implementation problem.

At the beginning of my thesis I had to hold a talk at the chair introducing my topic and laying out a basic time schedule. I planned 2 months for the implementation and another 6-8 weeks for the actual writing. In the end I had about 4 weeks for writing, while at the same time I was still chasing bugs in my implementation and kicking of the evaluation runs. That resulted in really long working days. My advice: keep track of your time. Note down how long you worked each day and what you did. 12 or more hours working a day don’t help. You get much more done in 8 productive hours, with breaks and some free time in between.

I guess no matter what you do, the last weeks will still be busy. What got me most focussed on the task at hand is the Pomodoro technique. 25 minutes of concentrated work, no distractions, no drinking, no eating, no running around. 5 minutes break. Check Twitter, Facebook, Mails, whatever. Then repeat. I use this technique for 3 years of studying now, and it also works for coding and writing.

The printed thesis

I had to write my thesis in English, but I’m far from a perfect English speaker or writer. I now know that I should practice much more often (and I try to do that by getting more blog posts out). Together with the first books on the topic I got from the university library I also got a book on proper technical writing. This small book had a lot of tips on how to phrase things, on typical mistakes, be it grammar or spelling. And don’t forget to use spell checking tools. They can catch simple typos, british/american differences and just wrong words.

If you write at a technical university it is not unlikely you will use LaTeX for your thesis. Make sure to set up your build chain early. You don’t want to deal with weird errors, broken packages and all sorts of layout fuckups 2 days before you need to submit your thesis.

One last important thing: Conferences are awesome. You meet nice people, you learn about a lot of interesting stuff and you most often get great motivation. But 5 conferences in 2 months, when you should be writing your thesis, might not be the best idea.
I did this. I went to Berlin twice, I was in St. Augustin for a whole weekend and in Cologne for another. I got near to nothing done for the thesis (besides having a great summer :D). So keep this in mind when planning ahead.

In the end I got my thesis finished and I am now a Bachelor of Science. The last 3 years were a great time and I plan to extend it with atleast another 2 years reaching for a Master’s degree.