RubyConfBR recap

November 1, 2010

Last week I presented at Rubyconf here in Brazil. It was the first edition named Rubyconf, but the folks behind it already had a tradition with Rails Summit. It was a huge event, specially considering that the main focus was a programming language. I got no exact numbers, but people where talking about more than 600 attendants. The site still up and you can see for yourself that there was a lot of interesting people talking.

I was invited by Fabio Akita to speak about non-blocking I/O. It was the first time here in Brazil that I got the opportunity to talk of such topics, and I’m glad that all conferences I attended or presented this year had more technical topics than vendor related and superficial concepts related to methodologies.

I had the opportunity to talk with Jim Weirich and Blaine Cook about what they are doing, and other folks that I happen to see once in a while, even living in Brazil.

Nando Vieira‘s lightning talk was one of the best I saw, along with Blaine Cook talking about webfinger and Emerson Macedo about Node.JS

There goes my slide deck


RestMQ has a unique endpoint for consumers which uses websockets. As such, I implemented websockets for cyclone and twisted some time ago. Last July there was an upgrade to the protocol to implement a ‘secure’ handshake. This new spec broke most of the implementations because it mixed the upgrade headers part and the first 8 bytes from the content.

I’ve upgraded both cyclone and txwebsockets to understand the ‘old’ spec (hixie 75) and the new one. Basically the handshake involves extracting numbers from two headers (Sec-Websocket-Key1 and Sec-Websocket-Key2), dividing the resulting number by the number of spaces, concatenating them with 8 bytes read from the socket and sending back the md5 digest of this mess back after the new headers. Code to test and calculate the handshake from the headers value can be found here .

Sounds cocky but that’s it. Using RestMQ, which builds on twisted, cyclone and a stack of well proven software, you can provide your applications with a robust and flexible queue over http protocol. It already was possible using COMET and GET/POST/DELETE requests, but now with websockets support it got to a new level.

Release gibberish apart there is a cool small app which streams twitter data to a html/css/js based app. Why all that work, would you say, if I can use it directly (well, not that easy because you need to provide username/pass to twitter streaming service) ? Well, for mashing up data and filtering it before delivery. Also, twitter is a convenient source of streaming data for tests, but usually you would roll up your own data source.

An actor based concurrency model or browser-based map/reduce is trivial, just like retrieving data, programming the proper javascript code to do all the work and post i back to a new queue.

Check the repository at and a live demo at Note that it needs a websockets capable browser (chrome or a recent webkit build) and it may be down due to my bandwidth constraints 🙂

I’ve wanted to prototype an URL shortener some time ago to have a measure on how long it would take to do using a NoSQL DB to store stats and do analytics. The main reason would do it using cyclone and MongoDB, and later build over it to create a tracker, web analytics tool and maybe a NGINX module.

Read the rest of this entry »

Web Sockets for Python

December 29, 2009

With all the fuzz around chrome and web sockets I’ve bundled up together a simple web sockets implementation for twisted. It’s a really easy way to interact with browsers.

In the next days I will run benchmarks, because I think it will stir up a bit the webserver scene a bit, and possibly, the assynchronous network frameworks too. On the bright side, its really easy to work on, and good ideas come up.

Enjoy and keep watching for new releases, as I have not finished it yet.

Repo URL:

Redis and python

December 25, 2009

One of the byproducts of RestMQ for me is that I got involved in a redis client in more ways than just using it. Along with fiorix, I got to know more of twisted and how to use it along cyclone, which is a tornado-on-twisted port. That helped to change from a pure twisted.web RestMQ to a more flexible setup.

Along the road, there was the need for a different redis client, which could handle connection pools, and there was txredisapi, which benefited from the experience in tx-redis and the original python redis client. It worked so well that we implemented some other features.

The most promising feature is not on RestMQ yet, but it’s on the master repo a git, which is Consistent Hashing. Consistent Hashing, in its various forms, enable setups resembling sharding and data distribution between two or more instances of Redis.

This is importante not only from the data partition point of view, but also from the avaliability one. I strongly believe that the client, or a intermediate layer, can control the data distribution in this way and not the storage server itself. Using txredisapi, the beggining of such architecture can be used, and later on, data as server free space, speed and I/O capacity can be put on the mix to help to decide how to populate a new server.

I invite you to check txredisapi, restmq and cyclone. My repo at also has clones and (sometimes) branches of these projects, so feel free to send patches and ideas.

Twisted COMET

October 28, 2009

I was about to write a post about NGINX, Python, Twisted and COMET, but it got so long that I decided to break it in 2 or 3 parts.

The first one is a kind of follow-up to the last post . This time the subject is a script that will search for a given word in twitter, and update the results in a continuous fashion. Once you point your browser to http://localhost:8000/ it will start to receive the results from time to time, as a big download (which is what COMET is about).


Read the rest of this entry »