November 1, 2010
Last week I presented at Rubyconf here in Brazil. It was the first edition named Rubyconf, but the folks behind it already had a tradition with Rails Summit. It was a huge event, specially considering that the main focus was a programming language. I got no exact numbers, but people where talking about more than 600 attendants. The site still up and you can see for yourself that there was a lot of interesting people talking.
I was invited by Fabio Akita to speak about non-blocking I/O. It was the first time here in Brazil that I got the opportunity to talk of such topics, and I’m glad that all conferences I attended or presented this year had more technical topics than vendor related and superficial concepts related to methodologies.
I had the opportunity to talk with Jim Weirich and Blaine Cook about what they are doing, and other folks that I happen to see once in a while, even living in Brazil.
There goes my slide deck
August 15, 2010
I’ve been using Redis for projects on and off for some time, and there are some little hacks I’ve been doing and never extracted from bigger projects. Yesterday I had to sit home for some time doing a job that involved some idle time waiting, so it was time to hack.
First, I nailed a small RestMQ using Sinatra and Redis. Here’s the gist for the first version, which already works nice along with RestMQ. You can use it to expose a small part of your broker to the outside world. Later, talking with a colleague at work, I changed it a bit and ended up having the whole queue list and hard/soft get (deletes the message or just reads it). Another gist.
Then it was time of extracting Message Queue and Load Balancing patterns from code to my branch of the Redis Cookbook . Apart from the basic algorithm for RestMQ, there is a pattern which I sometimes use to do load balancing and replica spreading. It uses scored sets and although it seems naive, works pretty well along with consistent hashing.
After that I fixed some issues on RestMQ and txredisapi, the first were related to configuration issues and the later related to publish/subscribe.
About Pub/Sub, I ended up extracting a small PubSub server using Websockets, Redis and Node.js. It was initially embedded in another proxy I tried for RestMQ but it works well alone. Check the code. A little bit of code twisting and it can turn into a very flexible actor-based library for node.js. And of course, the PubSub thingy can also use redis as a presence server.
/me deserves pizza
August 7, 2010
RestMQ has a unique endpoint for consumers which uses websockets. As such, I implemented websockets for cyclone and twisted some time ago. Last July there was an upgrade to the protocol to implement a ‘secure’ handshake. This new spec broke most of the implementations because it mixed the upgrade headers part and the first 8 bytes from the content.
I’ve upgraded both cyclone and txwebsockets to understand the ‘old’ spec (hixie 75) and the new one. Basically the handshake involves extracting numbers from two headers (Sec-Websocket-Key1 and Sec-Websocket-Key2), dividing the resulting number by the number of spaces, concatenating them with 8 bytes read from the socket and sending back the md5 digest of this mess back after the new headers. Code to test and calculate the handshake from the headers value can be found here .
February 3, 2010
Seems like DSLs for web development are getting more solid in the python camp (alto I’m yet to see something like sinatra out of the box or something that spare me from too much CSS’ing).
Today I read about Bottle and Redis, and to keep things fresh, cooked up a PasteBin clone using Bottle and MongoDB. I may do it using Redis too, just to showcase the differences. It’s really easy to come up with prototypes using Bottle.
Check it here: http://github.com/gleicon/pasteme
December 29, 2009
With all the fuzz around chrome and web sockets I’ve bundled up together a simple web sockets implementation for twisted. It’s a really easy way to interact with browsers.
In the next days I will run benchmarks, because I think it will stir up a bit the webserver scene a bit, and possibly, the assynchronous network frameworks too. On the bright side, its really easy to work on, and good ideas come up.
Enjoy and keep watching for new releases, as I have not finished it yet.
Repo URL: http://github.com/gleicon/txwebsockets
December 25, 2009
One of the byproducts of RestMQ for me is that I got involved in a redis client in more ways than just using it. Along with fiorix, I got to know more of twisted and how to use it along cyclone, which is a tornado-on-twisted port. That helped to change from a pure twisted.web RestMQ to a more flexible setup.
Along the road, there was the need for a different redis client, which could handle connection pools, and there was txredisapi, which benefited from the experience in tx-redis and the original python redis client. It worked so well that we implemented some other features.
The most promising feature is not on RestMQ yet, but it’s on the master repo a git, which is Consistent Hashing. Consistent Hashing, in its various forms, enable setups resembling sharding and data distribution between two or more instances of Redis.
This is importante not only from the data partition point of view, but also from the avaliability one. I strongly believe that the client, or a intermediate layer, can control the data distribution in this way and not the storage server itself. Using txredisapi, the beggining of such architecture can be used, and later on, data as server free space, speed and I/O capacity can be put on the mix to help to decide how to populate a new server.
I invite you to check txredisapi, restmq and cyclone. My repo at http://github.com/gleicon also has clones and (sometimes) branches of these projects, so feel free to send patches and ideas.
November 17, 2009
…or how to do proper caching without heavily modifing your code.
I’ve been using this technique on django views and Juno for some time. It’s just a decorator which inspect the parameters given to a function to search for a key on memcached.
You will find two files on github: no_decorator_example.py and decorator_example_memcache.py. Both of them search on twitter for a given term. The first script executes the search everytime, and the second script implements a 60 second cache, which uses the parameter (search term) as key.
I’m sure there must be another way to deal with parameters without the wrapper class, but so far nothing I’ve tried gave the same result.
For Django views, one could go so far as using return HttpResponse(your_response, mimetype=’text_plain’) too.