Extending Redis with Lua packages

(by )

Warning: If you patch your Redis as stated below, you won’t get much support from the Community.
Do not run this in production! You have been warned.


Redis comes with in-built support for Lua scripts. Using eval (or evalsha) it is possible to execute scripts right in the context of the Redis server. Redis provides some guarantees for these Lua scripts: Lua scripts are atomic and do not interfere with other scripts or normal commands. Redis tries its best to provide a sandbox you’re scripts are evaluated in, but it won’t stop you from doing stupid things.

This sandboxed mechanism also means that access to any external resources is not allowed. No IO, no external libraries (except thus already provided) and especially no compiled modules.

But what if you want to use a library anyway? Well, it’s actually not that hard. Three days ago a user came to the IRC channel (#redis on freenode) to ask how to use LGMP, an adapter to the GNU multiple precision arithmetic library, in Redis.

After some fiddling around I provided a solution:

First patch your local Redis checkout with the following patch:

diff --git i/deps/Makefile w/deps/Makefile
index 5a95545..9ec62be 100644
--- i/deps/Makefile
+++ w/deps/Makefile
@@ -58,8 +58,8 @@ ifeq ($(uname_S),SunOS)
  LUA_CFLAGS= -D__C99FEATURES__=1
 endif
 
-LUA_CFLAGS+= -O2 -Wall -DLUA_ANSI $(CFLAGS)
-LUA_LDFLAGS+= $(LDFLAGS)
+LUA_CFLAGS+= -O2 -Wall -DLUA_ANSI -DLUA_USE_DLOPEN $(CFLAGS)
+LUA_LDFLAGS+= -ldl $(LDFLAGS)
 
 lua: .make-prerequisites
  @printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
diff --git i/src/scripting.c w/src/scripting.c
index ef00eed..c7f3735 100644
--- i/src/scripting.c
+++ w/src/scripting.c
@@ -549,10 +549,8 @@ void luaLoadLibraries(lua_State *lua) {
     luaLoadLib(lua, "struct", luaopen_struct);
     luaLoadLib(lua, "cmsgpack", luaopen_cmsgpack);
 
-#if 0 /* Stuff that we don't load currently, for sandboxing concerns. */
     luaLoadLib(lua, LUA_LOADLIBNAME, luaopen_package);
     luaLoadLib(lua, LUA_OSLIBNAME, luaopen_os);
-#endif
 
}
 
 /* Remove a functions that we don't want to expose to the Redis scripting

This makes sure the provided Lua is compiled with support to load modules and it actually enables the package module in the embedded Lua interpreter (it also enables the OS module). The package module will then provide the require method used to load external modules.

Next compile your Redis as usual:

make distclean # Just in case it was compiled before
make

Next, grab a copy of the LGMP files from Wim Couwenberg (Direct link, .tar.bz2, make sure to grab the Lua 5.1 sources). Unpack it to a location, e.g. ~/code/lgmp and compile it:

gcc -O2 -Wall -lgmp -o c-gmp.so lgmp.c -shared -fPIC -L ~/code/redis/deps/lua/src/ -I ~/code/redis/deps/lua/src

Adjust the paths for -L and -I to point to the Lua bundled with Redis.

Once this is done, just start the Redis server (but make sure you do it from the directory with the gmp.lua and c-gmp.so):

~/code/redis/src/redis-server &

Alternatively copy gmp.lua and c-gmp.so to the libraries the following command gives you:

$ ~/code/redis/src/redis-cli eval "return {package.cpath, package.path}" 0
1) "./?.so;/usr/local/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/loadall.so"
2) "./?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;/usr/local/lib/lua/5.1/?.lua;/usr/local/lib/lua/5.1/?/init.lua"

As the last step, use the library:

$ ~/code/redis/src/redis-cli
127.0.0.1:6379> eval "local gmp=require'gmp'; return gmp.version" 0
"6.0.0"
127.0.0.1:6379> eval "local gmp=require'gmp'; local x = gmp.z(123); local y = gmp.z(456); return x:add(y):__tostring()"  0
"579"

And that’s it. That’s how you can load external modules into the embedded Lua interpreter. Again: Do not run this in production! It may break at any time, it may erase your hard drive or kill kittens.

(These instructions are also available in a short Gist.)

OpenTechSchool - Javascript for Beginners

(by )

2 month ago I coached at OpenTechSchool’s HTML & CSS for Absolute Beginners workshop. This weekend I once again helped at one of their workshops, this time Javascript for Absolute Beginners.

What should I say? It was again a great experience. It were less learners this time, but the ones that attended where very motivated and eager to learn new things. The material (Day 1, Day 2) was again provided by the OpenTechSchool movement and also translated into German. After just the first day, there were already great programs implemented, e.g. a star system, with rotating planets and moons.

The second day was a little less packed, but again all attendees worked their way through the material, asking interesting question and I did my best to answer them. But sometimes I couldn’t and needed help from the other coaches or even a quick online search to give a correct answer. (Would you know how to describe the difference between null and undefined in Javascript in about 1-2 sentences to a first-time programer?)

This time there was a small lottery. The winner already blogged about his experience: Cool JavaScript stuff I learned at the Open TechSchool workshop.

For this lottery I hacked a small and completely fair(!) winner-picker with the same tools/code also available to the learners. As requested the code is available online in a gist.

A big thanks again to the organizers of the OpenTechSchool Dortmund Team! Spread the word, attend meetups and workshops and use the chance to coach!

Changing the root password in recent SmartOS

(by )

Back in 2012 Jonathan Perkin wrote a little bit about SmartOS and the global zone, why and what in SmartOS is mounted read-only.

Back then /etc/shadow, the file containing the root password, was mounted with write permissions. It is not anymore:

# ls -l /etc/shadow
-r--------   1 root     root         560 May 12 19:51 /etc/shadow

And thus passwd will fail when trying to change the password. But it’s easy to circumvent this. It’s actually a lofs-mount as can be seen:

# mount | grep shadow
/etc/shadow on /usbkey/shadow read/write/setuid/devices/dev=1690008 on Mon May 12 20:05:36 2014

So to change your password use the following:

/usr/lib/cryptpass your-fancy-password
# replace crypt string in /usbkey/shadow
umount /etc/shadow
mount -F lofs /usbkey/shadow /etc/shadow
  1. will create the crypt string of your password. Make sure to remove it from your bash history afterwards
  2. place this string in /usbkey/shadow (it’s the long string between the first two :)
  3. unmount the file in place
  4. remount the right file back in place

And that’s it, the new root password is set.

key=value logs with nginx

(by )

In Six Ways to Make Your Production Logs More Useful @roidrage talked about how to make your logs much more useful. One of the proposed solutions was to use a format that is more structured and thus easier to read: key=value logging. This way it’s easy to parse for a human eye, but a machine will have no problems either. And of course your standard shell tools (with the combination of some basic regular expressions) will still work to find what you want in your logs.

This blog and all of my other sites run on nginx and with nginx it is very easy to use a custom log format. Just define a new log_format, give it a meaningful name and the layout you want. For example I use the following now:

log_format  keyvalue  'time=$time_iso8601 ip=$remote_addr user=$remote_user req="$request" '
                  'status=$status bytes_sent=$body_bytes_sent req_time=$request_time ref="$http_referer" '
                  'ua="$http_user_agent" forwarded="$http_x_forwarded_for"';

Put this in your http {} block and wherever you define a new access log file append keyvalue (or the name you have chosen) like this:

access_log  /var/log/nginx/access.log  keyvalue;

And voilà:

$ tail -1 /var/log/nginx/access.log
time=2014-04-15T18:20:54+02:00 ip=127.0.0.1 user=- req="GET / HTTP/1.1" status=301 bytes_sent=184 req_time=0.000 ref="-" ua="curl/7.36.0" forwarded="-"

If you want to know which variables are available, read the nginx docs.

Update: @rhoml pointed me to a full overview of available variables.

OpenTechSchool - Learn new things

(by )

Last weekend (29./30.03.) I coached beginners some HTML & CSS. This is about my experience.

A while back a friend of mine asked me if I wanted to help at OpenTechSchool Dortmund, coaching programming or to hold a talk. At that time I was a little busy with University stuff like writing exams, but I said I would be very happy to help.

Workshop schedule

Then about a week ago he reached out to me. One of their coaches had canceled and they needed a new one. I immediately said yes. Last friday then we had a short and fun coach meeting where I met the organizers. They worked hard to translate the complete material into German to make it easily understandable. The full material can be found online. It’s not quite perfect but it is a good starting point.

Saturday began quite early and by 11 o’clock the room at ständige vertretung was filled with about 20 beginners ready to start learning HTML & CSS. The OpenTechSchool encourages people to learn at their own pace, so my task was not to teach them but only assist when questions came up. This was a lot of fun for all, it was very interesting and great to see people getting so much fun by learning a new thing, implementing their own ideas and setting their creativity free. Oh, and I also learned a few things (typography can be an interesting topic).

After two full days of HTML & CSS and a very very funny evening in a local bar, the workshop came to an end and results started to show up. All attendees had something done, at least one website was successfully uploaded to the internet, some even worked on their sites after that.

If you want to know more about the OpenTechSchool and what they do go over and read about it. The next beginner’s workshop will be on 24th and 25th of May. Register at Meetup. Maybe I can find the time to be there as a coach as well (if the organizers want me as a coach of course ;)) There are also monthly “Learners Meetups” with talks, discussions and a lot of great people, next one just next week on 9th April.

A big thanks to the organizers Ola, Dennis, Leif, Tobias, Carsten and Hendrik. Another big thanks to the other coaches Lars and Dennis. And of course to all attendees, who had so much fun.