Fun with Redis

August 15, 2010

I’ve been using Redis for projects on and off for some time, and there are some little hacks I’ve been doing and never extracted from bigger projects. Yesterday I had to sit home for some time doing a job that involved some idle time waiting, so it was time to hack.

First, I nailed a small RestMQ using Sinatra and Redis. Here’s the gist for the first version, which already works nice along with RestMQ. You can use it to expose a small part of your broker to the outside world. Later, talking with a colleague at work, I changed it a bit and ended up having the whole queue list and hard/soft get (deletes the message or just reads it). Another gist.

Then it was time of extracting Message Queue and Load Balancing patterns from code to my branch of the Redis Cookbook . Apart from the basic algorithm for RestMQ, there is a pattern which I sometimes use to do load balancing and replica spreading. It uses scored sets and although it seems naive, works pretty well along with consistent hashing.

After that I fixed some issues on RestMQ and txredisapi, the first were related to configuration issues and the later related to publish/subscribe.

About Pub/Sub, I ended up extracting a small PubSub server using Websockets, Redis and Node.js. It was initially embedded in another proxy I tried for RestMQ but it works well alone. Check the code. A little bit of code twisting and it can turn into a very flexible actor-based library for node.js. And of course, the PubSub thingy can also use redis as a presence server.

/me deserves pizza

The rewrite ritual

June 6, 2010

Not that I am a fan of rewriting stuff, but there are some applications that I use to showcase technologies and from time to time I set up to rewrite them. As there is already a specification, its a matter of an exercise for new technologies. No over-specification and a very delimited time box.

Lately, two of these apps are a pastie clone and an url shortener. First I wrote them in ruby, them in python, python with twisted and now using Erlang and Mochiweb.

The code is on at uurl-erl and pasteme-erl repos, and as usual the html/css part is barely minimal. Also they are online at and

Both of them use Redis as a global increment and cache server, along with MongoDB for storing documents and stats. I find this to be a good match due to Redis atomic operations and pub/sub capabilities.

Last saturday I presented an updated version of what sent me the noSQL way at nosqlbr – the first noSQL conference in Brazil. Basically I talked about stuff I did and I saw when I gave consulting to legacy systems. Life is not pretty but there are ways out.

The event was great and I sent a recap to myNoSQL . I got really surprised that so many people got interested on noSQL and alternative/emergent technologies.

I’ve wanted to prototype an URL shortener some time ago to have a measure on how long it would take to do using a NoSQL DB to store stats and do analytics. The main reason would do it using cyclone and MongoDB, and later build over it to create a tracker, web analytics tool and maybe a NGINX module.

Read the rest of this entry »

Notes about Redis

January 31, 2010

I’ve come up with some notes about Redis, RestMQ and mongodb. Uploaded them to slideshare.

Yesterday I’ve presented this at Locaweb/Brazil as part of our NoSQL cycle. Many other people presented interesting new ideas and I’ve opted for this intro and a piece about Redis which I may publish soon.
NoSql Introduction
View more presentations from gleicon.

Redis and python

December 25, 2009

One of the byproducts of RestMQ for me is that I got involved in a redis client in more ways than just using it. Along with fiorix, I got to know more of twisted and how to use it along cyclone, which is a tornado-on-twisted port. That helped to change from a pure twisted.web RestMQ to a more flexible setup.

Along the road, there was the need for a different redis client, which could handle connection pools, and there was txredisapi, which benefited from the experience in tx-redis and the original python redis client. It worked so well that we implemented some other features.

The most promising feature is not on RestMQ yet, but it’s on the master repo a git, which is Consistent Hashing. Consistent Hashing, in its various forms, enable setups resembling sharding and data distribution between two or more instances of Redis.

This is importante not only from the data partition point of view, but also from the avaliability one. I strongly believe that the client, or a intermediate layer, can control the data distribution in this way and not the storage server itself. Using txredisapi, the beggining of such architecture can be used, and later on, data as server free space, speed and I/O capacity can be put on the mix to help to decide how to populate a new server.

I invite you to check txredisapi, restmq and cyclone. My repo at also has clones and (sometimes) branches of these projects, so feel free to send patches and ideas.