Redis Sentinel & Redis Cluster - what?

(by )

In the last week there were several questions regarding Redis Sentinel and Redis Cluster, if one or the other will go away or if they need to be used in combination. This post tries to give a short and precise info about both and what they are used for.

Redis Sentinel

Redis Sentinel was born in 2012 and first released when Redis 2.4 was stable. It is a system designed to help managing Redis instances.

It will monitor your master & slave instances, notify you about changed behaviour, handle automatic failover in case a master is down and act as a configuration provider, so your clients can find the current master instance.

Redis Sentinel runs as a seperate program. You should have atleast 3 Sentinel instances monitoring a master instance and its slaves. Sentinel instances try to find consensus when doing a failover and only an odd number of instances will prevent most problems, 3 being the minimum. In this case one of the Sentinel instances can go down and a failover will still work as (hopefully) the other two instances reach consensus which slave to promote.

One thing about the configurable quorum: this is only the number of Sentinel who have to agree a master is down. You still need N/2 + 1 Sentinels to vote for a slave to be promoted (that N is the total number of all Sentinels ever seen for this pod).

A pod of Sentinels can monitor multiple Redis master & slave nodes. Just make sure you don’t mix up names, add slaves to the right master and so on.

Full documentation for Sentinel.

Redis Cluster

If we go by first commit, then Cluster is even older than Sentinel, dating back to 2011. There’s a bit more info in antirez’ blog. It’s released as stable with version 3.0 as of April 1st, 2015.

Redis Cluster is a data sharding solution with automatic management, handling failover and replication.

With Redis Cluster your data is split across multiple nodes, each one holding a subset of the full data. Slave instances replicate a single master and act as fallback instances. In case a master instance will become unavailable due to network splits or software/hardware crashes, the remaining Master nodes in the Cluster will register this and will reach a state triggering a failover. A suitable Slave of the unavailable Master node will then step up and will be promoted to takeover as a new Master.

You don’t need additional failover handling when using Redis Cluster and you should definitely not point Sentinel instances at any of the Cluster nodes. You also want to use a smart client library that knows about Redis Cluster, so it can automatically redirect you to the right nodes when accessing data.

Redis Cluster specification and Redis Cluster Tutorial.
I gave a talk about Redis Cluster at the PHPUGDUS meeting last month, my slides are on slidr.io.


Want to hear more about Redis, Redis Sentinel or Redis Cluster? Just invite me!

Using a Kindle for status information

(by )

Back in 2011 I got a Kindle 4 (the non-touch version) and for some time it was the primary device for reading, be it ebooks, technical documentation or slides and transcripts from university.

But then I was using it less and less and for the last one and a half years it basically layed around unused. While it is a good device for book reading, it isn’t for other content. It’s slow, it can’t handle PDFs properly (zooming is just awful) and adding notes is really annoying with that on-screen keyboard.

For some time now I have this link saved: Kindle Weather Display.

Well, what better to do with a lazy holiday then doing some hacking with the Kindle? And so I did and this is the current result: It displays the weather forecast.

For now it shows the weather forecast

As the original article is quite short on the precise steps to get this finished, I wanted to write them up here.

(Just in case: I’m not responsible if you break your kindle while hacking around with it.)

First you need to jailbreak your Kindle, this will make the following things a bit easier. You should get it done using this short guide. The next step is to set up SSH to get shell access on the Kindle. I used the USBnet variant described in the Kindle 4 NT Hacking Guide (yes, that’s the same as the Jailbreak one). Despite its name this can enable the SSH daemon on the WiFi interface too. Attach the Kindle via USB, mount it and then open the usbnet/etc/config and add:

K3_WIFI="true"

Now you can also enable auto-starting USBnet. Caution: As long as USBnet is running, you can’t mount the Kindle.

# the Kindle should be mounted into /mnt/sdb1
mv /mnt/sdb1/usbnet/DISABLED_auto /mnt/sdb1/usbnet/auto

Next, reboot your device. Once it’s back up you should be able to connect to it via SSH on the IP it has in your WiFi network.

ssh root@192.168.1.42

The root password is either mario or of the form fionaABCD. Use the Kindle root password tool to find out based on the serial number.

There’s just one more tool: Kite, the application launcher. You can get it in this forum post. Installation is easy once you got the kite.gz. Copy the kite file to the kindle, then execute it:

jer@brain$ gunzip kite.gz
jer@brain$ scp kite root@192.168.1.42:/tmp/
jer@brain$ ssh root@192.168.1.42
root@kindle# cd /tmp
root@kindle# chmod +x kite
root@kindle# ./kite

One thing to note: You just downloaded some binary blob from some random forum and executed it. But you did that with the jailbreak and USBnet above anyway. And hey, that’s how these things worked back in the old days, it actually was totally normal in the PSP scene too

Back to our project: Reboot the Kindle and in the start screen you should see some note that Kite is started as well. The Kindle will also contain some new directories:

root@kindle# ls -l /mnt/us/kite
drwxr-xr-x    2 root     root         8192 May 14 12:13 onboot
drwxr-xr-x    2 root     root         8192 May 14 11:57 ondrop

onboot is the relevant one. All scripts in there are executed by Kite on startup of the Kindle. That’s where we disable some stuff and display our image for the first time. Write the following code to a file init-weather.sh and place it in onboot (or just get it from the repository):

#!/bin/sh

/etc/init.d/framework stop
/etc/init.d/powerd stop
/mnt/us/weather/display-weather.sh

This will disable the framework (= the Kindle UI basically) and the power management daemon (= responsible for disabling WiFi and switching to the screensaver if idle for too long). In case you want to get back to the old state, just enable framework and powerd again (and first remove the init-weather.sh which will otherwise directly disable them again).

The display-weather.sh script now does the hard stuff, which is pretty easy: Clear the screen, get a new image, display it.

#!/bin/sh

cd "$(dirname "$0")"

rm -f display.png
eips -c
eips -c

if wget -q http://server/path/to/display.png; then
    eips -g display.png
else
    eips -g weather-image-error.png
fi

eips is the tool to write something on the screen or display an image.

Now to regularly and automatically get a new image, set up a cronjob:

root@kindle# mntroot rw
root@kindle# echo '0 7,19 * * * /mnt/us/weather/display-weather.sh' >> /etc/crontab/root
root@kindle# mntroot ro
root@kindle# /etc/init.d/cron restart

The script will now be executed every day at 7:00 and 19:00, showing a picture from the internet (well, at best it’s a picture you generated).

As this post is already getting quite long, I leave the server-side up to you. All files (for both the Kindle and the server part) are in the GitHub repository: kindle-weather-display. This is the final result: My Kindle hanging on the wall right under the calendar. :)

It's hanging at the wall


Thanks to @e2b for proofreading a draft of this post.

New releases of hiredis-py and hiredis-node

(by )

I just published hiredis-py v0.2.0 to PyPi and hiredis-node v0.3.0 to npm.

Both of these do not include many new features compared to the last release, but it still took me hours and hours to get this out, and that’s for one simple reason: We now have basic Windows support in hiredis and thus in hiredis-py and hiredis-node as well.

These two modules only use the parser functionality of hiredis and leave the socket stuff to the language itself. Since v0.12, this parser functionality in hiredis was extracted into seperate files, which made it easily possible to include the necessary compatibility code (if any) to use it on Windows as well.

What made these releases take so long to get finished was the CI process. I didn’t want to include support unless I can make sure it keeps working and for this I need to run the tests on the desired systems. But because I don’t personally own a Windows machine on which I could develop (nor would I want one) I had to use some external service for this. I was pointed to appveyor, basically the TRavis CI for Windows. Setting everything up and making sure tests run correctly took me quite some time. The last time I touched any compiler on a Windows machine is several years back, so I had to gather all needed information from the documentation and demo scripts from the Internet. And builds that take as long as 40 minutes for 6 different environments don’t really help to get started fast. The actual build per environment takes only 3 minutes, but even that is high compared to the Linux builds on Travis, that run in about a minute (that is for 3 environments).

I finally reached green builds now and I hope I can keep it that way. I will rely on these builds for releases from now on to support Windows as best as I can, but as said before, I have no machine to test these in more detail and I rely soly on user input if anything breaks beyond the simple compile and test appveyor now does.

At next I will release a new version of hiredis itself with several fixes and new features, but this may take a bit more time (I wanted to finish it this week, but I can’t promise that anymore).


You’re interest in Open Tech? Come to the otsconf in August! First batch of tickets goes on sale this Sunday, 5. April, 5:00 pm CEST.

U2F demo application

(by )

Two weeks ago I got my first Universal Second Factor Device. It’s an inexpensive small USB key: the FIDO U2F Security Key. This key can be used as a 2nd Factor Authentication device.

It uses the protocol as specified by the FIDO Alliance, which consists of Google, Microsoft, Yubico, Lenovo and others.

What it provides

The overview document states:

The FIDO U2F protocol enables relying parties to offer a strong cryptographic 2nd factor option for end user security.

After the user has registered their device, the application can request authentication using this key on login (or when it seems necessary, e.g. when changing some other security settings).

Right now it relies on a extension for Chrome to provide the JavaScript API: FIDO U2F (Universal 2nd Factor) extension. Hopefully this will soon be implemented directly in the browser.

How it works

The U2F protocol is not complex at all, making it easy to implement and verify its correctness. It consists of 2 phases: registration and autentication, both requiring explicit human interaction.

Registration

  1. The server choses a pseudo-random 32 byte challenge
  2. It sends this challenge, a version identifier and its appId to the browser
  3. The browser forwards this data and the origin of the challenge to the key after requesting access requiring human interaction
  4. The key assembles its public key, key handle and a signature. The signature includes the seen appId, a hash of the provided challenge and origin, its own public key and its key handle.
  5. The browser sends back this registration data to the server, where the certificate is checked, the signature validated and public key and key handle are saved.

The key is now registered for use with this origin and appId.

Authentication

  1. The server choses a pseudo-random 32 byte challenge for every possible key handle.
  2. This data is sent to the browser, including the appId
  3. The browser forwards this data to the key, including the origin
  4. The key is activated by human interaction, it then creates a signature of a hash of the appId, a counter value and a hash of the provided challenge and origin. This signature and the counter value is sent back to the browser, which submits it to the server
  5. The server verifies the signature using the previously saved public key and verifies that the counter value is larger than any previously seen counter for this key handle.

If all runs through the user is successfully authenticated based on his key.

The implementation

The small demo application does nothing more than authenticating a user by name and a password and authorizing access to the private section of the website. A user is then able to add second factor authentication through U2F devices by registering one or more keys for their account. If a user has U2F devices registered, the server requires additional authentication by providing the U2F key to the website,

I decided to built this small application using the Cuba framework, a small Rack-based web framework providing only the absolute basics necessary for this. Authentication is handled by Shield, user data is stored using Ohm. For correct generation and verification of the U2F data I rely on ruby-u2f, an implementation of the full specification. The code itself is quite small, there are some todos and unimplemented things still open, but from what I understand right now they are not security-impacting. But before you run this in production, please take your own measurement and check the implementation against the spec.

The following will only describe the U2F relevant parts. The rest should be straight forward.

Key registration

Before a user can use second factor authentication, they need to register their device with the service.

on get do
  registration_requests = u2f.registration_requests
  session[:challenges] = registration_requests.map(&:challenge)

  render "key_add",
    registration_requests: registration_requests
end

First we generate registration requests for the key to sign later. We then need to save the provided challenges into the session to be able to check them later again. These could also be saved directly into a database. We could also add sign requests for known key handles to later check if the key is already known, but for simplicity we don’t do this here.

Then we simply render our form, the important JavaScript part in the frontend is this:

var registerRequests = {{ registration_requests.to_json }};

var signRequests = [];

u2f.register(registerRequests, signRequests, function(registerResponse) {
    var form, reg;

    if (registerResponse.errorCode) {
        return alert("Registration error: " + registerResponse.errorCode);
    }

    form = document.forms[0];
    response = document.querySelector("[name=response]");

    response.value = JSON.stringify(registerResponse);

    form.submit();
});

First we pass in the register and sign requests as JSON to be inspected by JavaScript. We then call the u2f API provided by the browser (for now added by an extension). The browser handles all the complicated stuff of verifying the provided request, asking for the user’s permission to use the key, sending it to the key and returning back the signed data to the browser. Once this is done, the callback is called. All that’s left to do is sending this data back to the server. We use a simple hidden form for that.

On the server side the data is parsed and verified. Again, this is handled completely by the library. All we need to do is calling the right methods and saving the key handle and public key to our database.

on post, param("response") do |response|
  u2f_response = U2F::RegisterResponse.load_from_json(response)

  reg = begin
          u2f.register!(session[:challenges], u2f_response)
        rescue U2F::Error => e
          session[:error] =  "Unable to register: #{e.class.name}"
          redirect "/private/keys/add"
        ensure
          session.delete(:challenges)
        end

  Registration.create(:certificate => reg.certificate,
                      :key_handle  => reg.key_handle,
                      :public_key  => reg.public_key,
                      :counter     => reg.counter,
                      :user        => current_user)

  session[:success] = "Key added."
  redirect "/private/keys"
end

The user has now a registered U2F key and must provide this on the next login to be successfully authenticated.

Second Factor authentication

A user with a registered U2F device first needs to login using the usual way by providing a username and the password.

if login(User, username, password)
  if current_user.registrations.size > 0
    session[:notice] = "Please insert one of your registered keys to proceed."
    session[:user_prelogin] = current_user.id
    logout(User)
    redirect "/login/key"
  end

  # …
end

If the provided login data is correct and the user has U2F devices registered, we redirect him to the next page handling this.

In this second login step, we generate a sign request on the server:

# Fetch existing Registrations from your db
key_handles = user.registrations.map(&:key_handle)
if key_handles.empty?
  session[:notice] = "Please add a key first."
  redirect "/private/keys"
end

# Generate SignRequests
sign_requests = u2f.authentication_requests(key_handles)

and provide it to the user:

var signRequests = {{ sign_requests.to_json }};


u2f.sign(signRequests, function(signResponse) {
    var form, reg;

    if (signResponse.errorCode) {
        return alert("Authentication error: " + signResponse.errorCode);
    }

    form = document.forms[0];
    response = document.querySelector("[name=response]");

    response.value = JSON.stringify(signResponse);

    form.submit();
});

Again, we simply pass on this data to the browser API, which makes sure the device is actually present and then lets the key sign the provided data. Once it returns we then send on this data to the server.

If there is an error in the signing process we just alert the user for now. For a better user experience this should be handled more nicely, showing the user a proper error message and giving the option to try again.

On the server side we need to check that the key handle exists for the user, then let the library validate the signed authentication request against our previously saved challenge. If everything checks out fine, we can finally login the user and set the session. As the last step we’re also updating the saved counter for the given key handle. This way we can protect against reply attacks. New authentications are only valid if the sent counter is higher than our saved one.

u2f_response = U2F::SignResponse.load_from_json(response)

registration = user.registrations.find(key_handle: u2f_response.key_handle).first

unless registration
  session[:error] = "No matching key handle found."
  redirect "/login"
end

begin
  u2f.authenticate!(session[:challenges], u2f_response,
                    Base64.decode64(registration.public_key), registration.counter.to_i)

rescue U2F::Error => e
  session[:error] = "There was an error authenticating you: #{e}"
  redirect "/login"
ensure
  session.delete(:challenges)
end

authenticate(user)
registration.counter = u2f_response.counter
registration.save

And that’s it. That’s all it takes for a working U2F implementation.

(what’s not visible: the browser asks for permission to use the U2F key on registration and the simple key is only usable for a short time after insertion, so it needs to be reinserted for each login, requiring explicit human interaction)

The full code is available in the repository on GitHub: cuba-u2f-demo


Thanks to @soveran for proof-reading a draft of this post and of course for his work on Cuba.

The difference of Rust's thread::spawn and thread::scoped

(by )

So yesterday I gave a Rust introduction talk at the local hackerspace, CCCAC. The slides are already online. The talk went pretty well and I think I could convince a few people why the ideas in Rust are actually useful. Though I made one mistake in explaining a concurrency feature (see slide 30). As it turns out, the example as I explained it was different from the presented code and one of the attendees actually asked me about it.

// Careful, this example is not quite right.
use std::thread;
use std::sync::{Arc, Mutex};

fn main() {
    let numbers = Arc::new(Mutex::new(vec![1, 2, 3]));

    for i in 0..3 {
        let number = numbers.clone();

        let _ = thread::scoped(|| {
            let mut array = number.lock().unwrap();

            array[i] += 1;

            println!("numbers[{}] is {}", i, array[i]);
        });
    }
}

I used this example to explain why it is necessary to wrap the vector in a mutex and the mutex in an Arc to make it possible to write to it from several threads. The problem lies within the used thread abstraction: thread::scoped.

Spawn a new scoped thread, returning a JoinGuard for it. The join guard can be used to explicitly join the child thread (via join), returning Result, or it will implicitly join the child upon being dropped.

So in the case of the above code each thread is joined right after it was created and thus the threads don’t even run concurrently, making the Arc and Mutex unnecessary. The following shortened example will still work, though not show casing what I intended to:

use std::thread;

fn main() {
    let mut numbers = vec![1, 2, 3];

    for i in 0..3 {
        let number = &mut numbers;

        let _ = thread::scoped(|| {
            number[i] += 1;

            println!("numbers[{}] is {}", i, number[i]);
        });
    }
}

There is another in-built threading method: thread::spawn. Its documentation reads:

Spawn a new thread, returning a JoinHandle for it. The join handle will implicitly detach the child thread upon being dropped.

And this is actually what I need to correctly demonstrate what I wanted to: the use of Arc and Mutex to safely share writable access to shared memory through mutual exclusion. The following example works and has all necessary parts:

use std::thread;
use std::sync::{Arc, Mutex};

fn main() {
    let numbers = Arc::new(Mutex::new(vec![1, 2, 3]));

    let mut threads = vec![];
    for i in 0..3 {
        let number = numbers.clone();

        let cur = thread::spawn(move|| {
            let mut array = number.lock().unwrap();

            array[i] += 1;

            println!("numbers[{}] is {}", i, array[i]);
        });
        threads.push(cur);
    }

    for i in threads {
        let _ = i.join();
    }
}

Running it gives the expected output (your output might differ, the order is non-deterministic):

$ rustc concurrency.rs
$ ./concurrency
numbers[1] is 3
numbers[2] is 4
numbers[0] is 2

The Rust book contains a complete chapter on this topic: Concurrency, covering a bit more of the background and also the Channel concept.

Again, thanks to the CCCAC and for all people listening to me and quite some questions afterwards. For all who could not attend: the video should be up soon.