The story of my childhood or: Rust on the PSP

(by )


Don’t care for all the stuff of my childhood? ↓ Head down to see the Rust part


Let’s do a time travel.

10 years back, it’s 2004. Sony releases the first version of the Playstation Portable. First in Japan, 4 months later in the US and yet another half year later in the EU (including Germany).

I read about this new little device in the autumn of 2004 and I knew immediately: I want this. I also knew: I can’t wait a whole year for it to get to the German market. So 13-year old me sat down in front of his parent’s PC and started searching through the depths of the internets. Well, mostly Ebay, because that seemed to be the best place to get brand new hardware.

It must have been around January or February that I finally found a more or less trustworthy looking offer that was not too expensive. Just 4 days after clicking Buy on Ebay my PSP finally arrived (yes, just 4 days shipping straight from Japan) and I could unpack it.

But I couldn’t do much more. D’oh! The first firmware version on the PSP was quite limited. It only supported WEP, luckily we still had WEP set up at home. The next problem: it didn’t even include a web browser yet, so even though I had a Wifi connection, I couldn’t use it. On to the next problem: Music and videos. Well, Music worked sort of (this was becoming my most expensive MP3 player at that time). Videos were another story. It only supported MP4 in some weird encoding formats and the only way to get it on the PSP was by using a Memory Stick Pro, one of these tiny little memory cards that only Sony used. They were really expensive, the PSP came with a 32 MB card and I scraped together all my left pocket money to get a 128 MB card. Even back then that was quite limited space. So videos were off the hook as well.

Afterall the PSP was a gaming device, so I should be able to play right? Yeah, actually for me that was a small afterthought, I didn’t buy a game together with the PSP. Off to Ebay again, luckily I found the original Ridge Racer, of course the Japanese version. Even now I would say: This was the best game ever released for the PSP. The graphics were astonishing, the music, the gameplay was fantastic, the rising dificulty of tracks and opponents promised hours of fun (and frustration).

While I played this game for hours, other people used their valuable time for far more important things: Hacking the PSP. Successfully.

Firmware 1.0 and 1.5 were exploited just 6 months after the first release. What followed was the big uprise of the Homebrew scene. And I was part of it. Younger me already knew some HTML, some (frontend) Javascript and I just learned C and C++. I wanted to use that knowledge for my first own and publicly released apps! The first step was getting the development toolchain set up. I had an old 600 MHz computer, running Windows XP (I guess, or was it still Windows 98?) The toolchain was based on GCC and was optimized for POSIX environments, which Windows clearly isn’t, but someone invested enough time to get it working on cygwin, including a nice one-click installer that did most of the work. This was the time before Google Code and long before GitHub, so you had to download pre-compiled EXE files from one-click hoster sites. The links could be found in one of the dozen forums that existed back then. The whole process to setup Cygwin and the PSP toolchain on my slow computer took nearly 24 hours, so I had to leave the computer on over night (a novelty back then for me). And it failed quite often. So repeat and repeat until it finishes without any failure once.

In the end I got it working and went on developing for the PSP. I learned C, I battled with C, I switched over to C++ and back to C, I threw in some Lua (Lua was one of the first bigger things fully ported to the PSP and thus the best option to quickly develop for the PSP).

The PSP scene was also the first online community I was part of. I became a moderator and blogger of one of the bigger German PSP news sites, portable-news.de. I became friends with people that later offered me my first job as a developer (thanks @g33konaut), that I still see once or twice a year (thanks @atsutane), and others that I never heard anything of again (I’m bad with names, I can’t even remember much more from back then).

The biggest released application from my time as a PSP dev is probably the PSP Customizer. A release post on some french forum is still online and of course v1.0 on the good old qj.net. The PSP Customizer could be used to change the boot screen, the menu backgrounds, change menu texts or dump and restore your flash registry (this contained some internal config I guess). Nothing fancy, it was not even originally written by me (though I ended up rewritting most of it). As the text on QJ suggests it was quite known around the community and people actually used it. Sadly I don’t have any code of it anymore. It wasn’t open source (before GitHub publishing code was really a big hassle), it wasn’t even version-controlled. I remember a day where I deleted the whole project by mistake. It must have been just before another release. I spent the whole night trying to recover as much as possible. In the end I had to rewrite large parts off my head (luckily it wasn’t the most complicated code).

I also wrote some other small applications, most of which I can’t even remember. What I can remember is a PoC of a round-based tactics game, written as a web app (HTML and some crude Javascript magic to work in the limited web browser on the PSP) and inspired by Field Commander. I never finished it or showed it to anyone.

The only application I still have the source code of is the first-ever Twitter client for the PSP: p-twit!. I published the code 5 years ago, probably one of the more horrible pieces of code I’ve got online. The original p-twit! announcement:

@badboy_: tada! twitter via PSP! release soon (tweet)

That was also the last thing I wrote for the PSP. I think 2008/2009 was also the time the homebrew scene slowed down. My community place, portable-news.de, was dead, more and more devs lost interest, there was nothing entirely new to be done. Smartphones became the new go-to hardware for small applications and fun games developed by people at home.

(More about the homebrew history)


Jumping back to 2014. I learned quite a lot more since my PSP homebrew days. I’m now studying computer science and working part-time as a developer. I write C, C++, Ruby, Javascript. And most recently I started learning Rust.

Then, on the 12th of November I was chilling on the balcony of the hotel on Hawaii, browsing the Internet, when I came across the following post on Reddit:

“Hello World” on a PSP via Rust!

Actually it was nothing more than a picture showing a PSP executing some “Hello World” app. Then some comments came in, including a link to the code and instructions.

As soon as I was back home, I grabbed my PSP, booted up a new virtual machine and installed the psptoolchain (mostly the same hassle as in the old days, it just finishes a lot quicker).

First step: Re-compile and run p-twit. Yip, it works (ok, it starts. Of course the Twitter API is so closed these days, it wouldn’t be able to do anything).

p-twit runs again

Second step: Follow the instructions from the Gist.

jer@psp-dev:~/rust-for-psp$ rustc hello.rs --target psp -lpspdebug -lpspdisplay -lpspge -lm -lc -lpspuser -lgcc
jer@psp-dev:~/rust-for-psp$ psp-fixup-imports hello
Error, no sceModuleInfo section found

D’oh! That didn’t work. I played around, tried all kinds of things, then gave up a little frustrated.

A few days later I talked to luqmana in the Rust IRC channel. He turned his Gist into a real repository, rust-psp-hello, including scripts and a Cargofile. Still, no luck for me. Even after re-installing the toolchain several times on different machines I still faced the awful Error, no sceModuleInfo section found after every compile.

After some more tweaking, re-installing, compiling and chatting with luqmana it turns out: He’s not using the same toolchain build as I do. He simply used the pre-compiled minpspw package (Update 04.12.2014: Luqman informed me he does in fact built it himself from the minpspw svn, which in turn uses the psptoolchain I tried with first, so absolutely no idea why directly using it failed). Once I grabbed that, the “Hello World” application compiled and ran! Success!

From Rust with love

The journey doesn’t end here. Only partly satisfied with what I had, I took some free time on a train ride to write a wrapper for the input handling.

Press X, press O

Now it’s possible to react to user input with this simple piece of code:

let mut pad_data = ctrl::Input::new();

if pad_data.read_changed() {
  match pad_data.pressed_key() {
    Button::Cross => utils::debug_print("X pressed\n"),
      Button::Circle => utils::debug_print("O pressed\n"),
      Button::Triangle => utils::debug_print("TRIANGLE pressed\n"),
      Button::Square => utils::debug_print("SQUARE pressed\n"),
      Button::None => (),
      _ => utils::debug_print("unhandled keypress\n"),
  }
}

The full implementation is up in my fork of rust-psp-hello.

There’s still a lot I want to do now that I’m started:

I’d be happy if anyone wants to help or wants to know more. Just work on the repo (Pull Requests welcome) or drop me a message on Twitter or via email. I’d also be happy to hear storys from others that developed for the PSP 5 or 7 or 9 years ago.

How to not write a Bachelor Thesis

(by )

Iteration 1

… and how I got it finished anyway.

From May to October I was busy working on my Bachelor Thesis. I finally handed it in right on time on the 13th of October. But especially the last 4 weeks were the busiest and hardest of 3 years of studying.

As with everything you do, you have a lot of plans when you start; you make a time schedule, you promise to take notes, you promise to work on it every day. I broke each and every promise I made before. Now that I’m done I know that I could have done better and so in the following I will expand on all the things I encountered in these 5 months of work.

The first and most important advice I can give: Start writing from day 1. Whatever you write will probably not end up in the final thesis, your writing doesn’t even need to be on topic, but getting 50 pages written is not an easy task if you don’t train for it. Writing technical texts is different from a novel or a short blog post. Work on that.

The next thing: buy a paper notebook dedicated for your notes on the thesis topic. I kept mine with me and actually used it. Keeping meeting minutes, tasks to do for the week, simulation results or general ideas (though most of the stuff was added in the last 4 weeks).

Everything done

Once you know your topic, start the research. That includes finding documentation, papers and whatever seems necessary for your topic. Make sure to note down your sources immediately. Read papers and summarize them for yourself and discuss them with your supervisor or other students. Get familiar with scholar.google.com and other paper search sites. Keep bookmarks to sites you visited and might need. Consider saving them offline just to be safe.

What helped me a lot in understanding the topic and getting ideas how to write about it was explaining it to other people, especially non-tech people. People without any background in your topic will be absolutely clueless, they can point out missing pieces or inaccurities in your explanations and they tend to ask the most basic question, which you would otherwise skip (because you know the answer right away). Of course this won’t help when it comes to the details of your actual topic. For this, find students working on similar topics. Discuss your approach with them. My supervisor forced us to have a meeting once a week. We were 4 thesis students with the same broader topic. Even though these meetings tend to be quite long, we often could help each other in nailing down a good solution to an implementation problem.

At the beginning of my thesis I had to hold a talk at the chair introducing my topic and laying out a basic time schedule. I planned 2 months for the implementation and another 6-8 weeks for the actual writing. In the end I had about 4 weeks for writing, while at the same time I was still chasing bugs in my implementation and kicking of the evaluation runs. That resulted in really long working days. My advice: keep track of your time. Note down how long you worked each day and what you did. 12 or more hours working a day don’t help. You get much more done in 8 productive hours, with breaks and some free time in between.

I guess no matter what you do, the last weeks will still be busy. What got me most focussed on the task at hand is the Pomodoro technique. 25 minutes of concentrated work, no distractions, no drinking, no eating, no running around. 5 minutes break. Check Twitter, Facebook, Mails, whatever. Then repeat. I use this technique for 3 years of studying now, and it also works for coding and writing.

The printed thesis

I had to write my thesis in English, but I’m far from a perfect English speaker or writer. I now know that I should practice much more often (and I try to do that by getting more blog posts out). Together with the first books on the topic I got from the university library I also got a book on proper technical writing. This small book had a lot of tips on how to phrase things, on typical mistakes, be it grammar or spelling. And don’t forget to use spell checking tools. They can catch simple typos, british/american differences and just wrong words.

If you write at a technical university it is not unlikely you will use LaTeX for your thesis. Make sure to set up your build chain early. You don’t want to deal with weird errors, broken packages and all sorts of layout fuckups 2 days before you need to submit your thesis.

One last important thing: Conferences are awesome. You meet nice people, you learn about a lot of interesting stuff and you most often get great motivation. But 5 conferences in 2 months, when you should be writing your thesis, might not be the best idea.
I did this. I went to Berlin twice, I was in St. Augustin for a whole weekend and in Cologne for another. I got near to nothing done for the thesis (besides having a great summer :D). So keep this in mind when planning ahead.

In the end I got my thesis finished and I am now a Bachelor of Science. The last 3 years were a great time and I plan to extend it with atleast another 2 years reaching for a Master’s degree.

Review: Redis Applied Design Patterns

(by )

In the mid of October I was approached by Packt Publishing asking if I would be willing to review a newly released eBook called Redis Applied Design Patterns by Arun Chinnachamy. I agreed to review the 100 page eBook given that I’m free to criticize the book as I see fit.

It took me some time to finish the book, but flights to Salzburg, New York and Honolulu meant enough (offline) time to read everything in detail and write the review. So here it is.

What is it about?

The book gives an overview of different use cases of Redis. It is definitely not for beginners, but for people that are already familiar how to setup Redis and how to use it. The books starts with the basics of a key-value store and how it differs from relational databases. It then goes on to discuss a wide variety of use cases. From using Redis as a cache store, using it as the main data store for e-commerce inventories or a comment system to using it for autosuggestion, real-time analysis and game leaderboards. Each chapter of the book discusses one of the named use cases, explaining the used Redis commands and showing code and ideas on how to implement the given system.

The good things

Having different use cases documented and accessible in a concise way is really useful. The book shows a wide range of these use cases and not only discusses the simplest implementation, but also tries to give an outlook on further iterations for the given requirements, building a increasingly complex system step by step.

It doesn’t focus on a single aspect of Redis and shows off a lot of the available commands and data types. Included are explanations of the used commands as they are described on redis.io, as well as links to the documentation for people that want a deeper understanding. The included code samples are kept short and are a mix of PHP code, implementing the discussed use case, and single Redis commands, showing how they work and what values they return. Even for programmers without knowledge of PHP, it should be possible to understand the implementation and adopt it for the language of choice.

The chapters are short and self-contained and the book is short enough to be read in just one afternoon (it took me longer because I took a lot of notes to come up with this review).

The not so good things

Quickly after I started reading the book I was a bit baffled about the overall text and code quality. The very first thing I noticed was that some of the code examples used slightly different key names as the describing text itself. This didn’t block the understanding of the overall concept, but is certainly a thing that needs to be improved.

Slightly off-putting are the code samples itself, especially the ones that are larger than just a couple of lines. They use an inconsistent code style (CamelCase vs. snake_case in one single line) and could use just a little more explanations here and there (or the text should refer to lines more directly). Some code examples are just plain wrong. The more advanced problem solutions are only briefly mentioned in the text and no code examples are shown.

It is nice to have the used Redis commands explained once, but the book mostly copies the official documentation for this. It also mentions the time complexity of each command, but nowhere in the book it is explained what O(n) actually means (Big O notation explained in the Wikipedia).

As mentioned, the chapters are mostly independent of each other, which makes it easy to just jump in and read them in whatever order you want. But as the first 3 chapters explain how things work in Redis, other chapters could easily refer back to them. Instead latter chapters discuss things again or link only to the online documentation. One chapter is actually re-implementing the same stuff as in the chapter before. Plain references between chapters could save this duplication and help the reader to jump between chapters.

I won’t discuss each chapter in detail, but there are two things that I need to mention here. First, the chapter about an auto-suggest system uses the old approach of using n-grams in Sorted Sets, but since version 2.8.9 (released in April 2014) Redis has powerful lexicographic commands like ZRANGEBYLEX, which are not mentioned in the book at all (though I don’t know when the book was written, maybe a future version can include this). Second, the summary of the last chapter talks about scaling Redis across multiple machines and mentions Sentinel as under development and usable for this, which is just incorrect (for the ones that don’t know: Sentinel is considered stable since 2.8 and is used for monitoring and failover, if you want to shard your data use Redis Cluster). There are a few other minor incorrect things or explanations that leave out some important details.

If you are already familiar with Redis, these things are obvious, but if you only know the basics and want to read up on possible implementations, the errors and mistakes will misguide you. This makes it hard to recommend the book as a useful resource on Redis design patterns.

Conclusion

Redis Applied Design Patterns is a short little book explaining different use cases of Redis. It gives insight into different techniques and implementations of high-level features. But don’t expect to be a super Redis pro after reading it. The given examples are only touching the surface and show the most basic implementations. The book should definitely be revised to fix several issues in quality and correctness. A more in-depth discussion on more complex implementations and revised examples would make it recommendable, but at this stage it is not.

In case you want to buy the book anyway, use this link.

A Rust library for LibLZF

(by )

In the last four months I did not produce much open-source code. I was busy writing my Bachelor Thesis. But I was active in the community, I attended several conferences, I read a lot of stuff and I wrote down a lot more things to do.

One of these things was “learn Rust”, as you might figure (and writing more). Well, I submitted my thesis on Monday and went on to write some actual Rust code.

So I created lzf-rs, a wrapper around LibLZF, a super small and fast compression library, originally written by Marc Lehmann. LibLZF consists of just 4 files and is super-easy to compile as a small library. LibLZF itself is used by Redis for compressing strings in the RDB, that’s how I got to know it. I still don’t know how it works, but I want to learn it now (and maybe port the whole code to Rust instead of using a wrapper around the C library?)

I used Rust’s Foreign Function Interface to write the wrapper. Accessing C code from within Rust is quite simple. The hard part is choosing the right data types and writing the API in a safe and usable way. I’m not 100% happy with the current API of lzf-rs, so expect it to change. I’m still in the process of figuring out what to expose and what not.

And now, enough of the talking. This is how to use the library:

use lzf;

fn main() {
  let data = "foobar";

  let compressed = lzf::compress(data.as_bytes()).unwrap();

  let decompressed = lzf::decompress(compressed.as_slice(), data.len()).unwrap();
}

I’m happy to get some feedback. It’s my first serious Rust code, so prone to bugs and maybe not to Rusty afterall. Ping me on Twitter, drop me an email or open an issue.

jsfest week 2014

(by )

rejectjs banner

Last week I had the opportunity to spend a long weekend in the beautiful capital of Germany for jsfest.berlin. What should I say? It was amazing.

After eurucamp set quite a basis for “great conference”, reject.js and jsconfeu had no easy start, but the people, the organization, the venue and the talks totally made up for the less good weather.

The 3 conference days were packed with interesting talks, quite a few parties and great food. JavaScript seems to be quite a nice language (sometimes) and people do awesome stuff with it. From sleep-hacking over making music and animations to making gifs even more awesome. Oh, and some stuff others would consider more useful in work life too.

rejectjs badge

The only downside was that I in fact didn’t really have time for a whole 4 days off, so I wasn’t too focused on the reject.js talks and I might have missed good things from it :(

But the two talks I did attend and where I was attentive where Hacking Sleep with JavaScript by Nick Rowe and Location-based Pokémon with Node.js by Stefanie Taepke.

Nick built himself a nice smart home powered by node.js, with automatic light dimming for an even better sleep. Definitely an interesting topic, though his setup isn’t the cheapest one. Stefanie built a prototype of a Location-based Pokémon game. There’s an online Pokédex you can use if you want to do something similar: Pokéapi - The Pokémon RESTful API.

For jsconfeu I was more focused and I tried to take at least a few notes per talk to remember all the stuff I heard. I did manage to tweet a photo of each talk I attended, so if you’re into that check my twitter stream from Saturday and Sunday.

Following are a few short notes per talk.

Day 1

After a nice intro (wait for the videos), Jake Archibald started with The ServiceWorker is coming; look busy. With this new addition to the browser stack web app can finally easily compete with native apps when it comes to notifications, background sync and offline-first. Definitely worth a watch.

The talk was followed by We Will All Be Game Programmers from Hunter Loftis. Today’s apps use a lot of the ideas that were used in games years ago, maybe we should have a look at game developing techniques and tooling a bit more to see what’s relevant for modern web app programming.

beanbag.js

JavaScript Level 9000 by Christoph Martens explained a lot of low-level stuff in JavaScript. I’m not that into JavaScript yet, so I didn’t take too much from it with me.

Next I watched Mathias Bynens talk about why JavaScript ♥ Unicode. There are quite a few changes how to properly handle Unicode in ECMAScript 6 and also a few libraries to make Unicode usage a lot easier today. Simply look through his notes and slides. For some 💩-humor watch the talk once it is out.

This was then followed by Parallel JavaScript from Jaswanth Sreeram. I’m currently writing my bachelor thesis in the field of high-performance computing and parallel computing, so this seemed quite relevant to me. Turns out: It might actually be relevant. JavaScript can be executed on GPUs to allow for multi-threaded computation. New APIs to make use of this are already in the making and the first implementation landed in Firefox Nightly. Once I’ve got some more time again I will check it out.

Sitting nice and comfy at beanbag.js I watched Supercharging Node with Kraken from Tim. PayPal transitioned most of their apps from a C++/Java-stack over to node.js, replacing apps one by one. They now have smaller teams, are able to more quickly adopt the apps as needed. They also make heavy use of npm together with a company-internal package registry to distribute common modules for security, logging, monitoring and other things.

After skipping two talks, James Coglan entered the stage with Practical functional programming: pick two. I followed his twitter stream in the past weeks where he already shared lots of his research and learning. Functional programming is quite an interesting topic with lots of good ideas (ooooh, what would I give to have immutability in my thesis project right now). James showed how even just a few functional ideas can make code a lot easier to reason about. Recommended watch! The slides are online.

Raul Fraile followed with How GZIP compression works. He explained the basic principles behind compression and GZIP. It’s not really my topic though.

Michael Donohoe talked about how to Deep-link to Anything on the Web. Working as a developer for journalistic sites he implemented multiple solutions to help linking to sections of an article, annotating it as needed and how to combine that with the backend.

party with nerds

After another coffee break (there were a lot coffee breaks that weekend) Mathieu Henri showed Monster Audio-Visual demos in a TCP packet, a short introduction into the demo scene and the techniques he uses to build demos in as few bytes as possible. I wish I was more creative with visuals and audio.

Just before the party really began, Tim Pietrusky showed his Nerd Disco. Nice what you can do with a small chip, some LEDs, an Ikea lamp and JavaScript. In fact I still have an LED strip lying around, I should definitely try this.

Day 2

Day 2 began early, far too early as the party the evening before was long into the night (well, and maybe I had one or two gin tonic).

I missed the first talk, but was right on time for Javascript… Torrents… and Mad Science! with Mathias Buus Madsen. I saw his projects before, but I was still amazed at what he did. peerflix, peerwiki and torrent-mount are 3 really cool projects, now if only more legal things would be available via torrent…

Still a little hangover I sat down at beanbag.js and watched Got make? by Rob Ashton. His live-demo talk was a little bit confusing.

What followed next (after one round of skipped talks) was Lena Reinhard’s talk This is bigger than us: Building a future for Open Source. WATCH IT! It’s worth it! Slides and notes are on speakerdeck.

Sebastien Cevey followed up with Server-less applications powered by Web Components. He combined web components to build up a dashboard showing stats from his servers at work.

In == ? === ??? …#@^% Shirmung Bielefeld explained the small details one needs to know to not get confused when JavaScript once again decides to do type coercion.

Sebastian Markbage from Facebook talked about Minimal API Surface Area - Learning patterns instead of frameworks. We should focus less on complete framework solutions and instead rely on reusable patterns. That’s exactly what they try to focus on in React.

Even though I thought I knew a bit about JavaScript What the heck is the event loop anyway? by Philip Roberts explained a bit more to me. He used his tool Loupe (a hack built on hacks and more hacks) to visualize how certain things happen when run in the JavaScript environment. Apart from the topic we also got a good stretch from this talk.

Amy Palamountain showed us the Enemy of the State. Don’t build servers on the client. Slides are online.

What followed was The Linguistic Relativity of Programming Languages by Jenna Zeigen. We are heavily influenced by the programming languages and tools we use. From time to time we should look at different paradigms, different programming languages, different tools simply to broaden our view and to get new and better ideas how to solve problems, even if we then come back to our main programming language.

Again from the comfy of beanbag.js, I watched Realtime in the browser using XMPP by Lloyd Watkin. He thinks XMPP “allows us to easily rebuild the decentralized web”. Slides are online.

Coming to an end, the organizers held back two really good talks:

The first one was GIFs vs Web Components. Glen Maddern implemented a library to control GIFs in every imaginable way. Born was the <x-gif> tag. See for yourself.

Followed by Know Your /’s in which Lindsay Eyink gave an overview over the early history of Silicon Valley. I can’t properly summarize it myself, so go watch the video when it is available.

The day was then coming to an end at yet another party, this time on a boat! There are photos of the conference and the parties on flickr. More to come for sure. The videos should be available on YouTube in the following weeks, I will probably link to them then. Here’s the full schedule for reject.js and the schedule for jsconfeu.

Thanks!

I had an amazing time and would love to come back next year for a full week of jsfest. Hope I can manage it somehow.

Thanks to the organizers of both conferences, thanks to all the awesome speakers, thanks for the people I met and talked too. Thanks to all attendees for making both these conferences such a joy!

Special thanks to Ola and Hendrik, who made it possible for me to attend the jsconfeu in the first place. And a shout-out to Jane, for whom this was the very first tech conference. Head over to her blog to read about her experience.

<3 <3 <3


Update 21.09.2014:

More things appear online: