Fun with Redis

August 15, 2010

I’ve been using Redis for projects on and off for some time, and there are some little hacks I’ve been doing and never extracted from bigger projects. Yesterday I had to sit home for some time doing a job that involved some idle time waiting, so it was time to hack.

First, I nailed a small RestMQ using Sinatra and Redis. Here’s the gist for the first version, which already works nice along with RestMQ. You can use it to expose a small part of your broker to the outside world. Later, talking with a colleague at work, I changed it a bit and ended up having the whole queue list and hard/soft get (deletes the message or just reads it). Another gist.

Then it was time of extracting Message Queue and Load Balancing patterns from code to my branch of the Redis Cookbook . Apart from the basic algorithm for RestMQ, there is a pattern which I sometimes use to do load balancing and replica spreading. It uses scored sets and although it seems naive, works pretty well along with consistent hashing.

After that I fixed some issues on RestMQ and txredisapi, the first were related to configuration issues and the later related to publish/subscribe.

About Pub/Sub, I ended up extracting a small PubSub server using Websockets, Redis and Node.js. It was initially embedded in another proxy I tried for RestMQ but it works well alone. Check the code. A little bit of code twisting and it can turn into a very flexible actor-based library for node.js. And of course, the PubSub thingy can also use redis as a presence server.

/me deserves pizza

Advertisements

RestMQ has a unique endpoint for consumers which uses websockets. As such, I implemented websockets for cyclone and twisted some time ago. Last July there was an upgrade to the protocol to implement a ‘secure’ handshake. This new spec broke most of the implementations because it mixed the upgrade headers part and the first 8 bytes from the content.

I’ve upgraded both cyclone and txwebsockets to understand the ‘old’ spec (hixie 75) and the new one. Basically the handshake involves extracting numbers from two headers (Sec-Websocket-Key1 and Sec-Websocket-Key2), dividing the resulting number by the number of spaces, concatenating them with 8 bytes read from the socket and sending back the md5 digest of this mess back after the new headers. Code to test and calculate the handshake from the headers value can be found here .

Sounds cocky but that’s it. Using RestMQ, which builds on twisted, cyclone and a stack of well proven software, you can provide your applications with a robust and flexible queue over http protocol. It already was possible using COMET and GET/POST/DELETE requests, but now with websockets support it got to a new level.

Release gibberish apart there is a cool small app which streams twitter data to a html/css/js based app. Why all that work, would you say, if I can use it directly (well, not that easy because you need to provide username/pass to twitter streaming service) ? Well, for mashing up data and filtering it before delivery. Also, twitter is a convenient source of streaming data for tests, but usually you would roll up your own data source.

An actor based concurrency model or browser-based map/reduce is trivial, just like retrieving data, programming the proper javascript code to do all the work and post i back to a new queue.

Check the repository at http://github.com/gleicon/restmq_websockets and a live demo at http://cowcat.net/restmq_ws/. Note that it needs a websockets capable browser (chrome or a recent webkit build) and it may be down due to my bandwidth constraints 🙂

Web Sockets for Python

December 29, 2009

With all the fuzz around chrome and web sockets I’ve bundled up together a simple web sockets implementation for twisted. It’s a really easy way to interact with browsers.

In the next days I will run benchmarks, because I think it will stir up a bit the webserver scene a bit, and possibly, the assynchronous network frameworks too. On the bright side, its really easy to work on, and good ideas come up.

Enjoy and keep watching for new releases, as I have not finished it yet.

Repo URL: http://github.com/gleicon/txwebsockets