websockets for python and twisted

RestMQ has a unique endpoint for consumers which uses websockets. As such, I implemented websockets for cyclone and twisted some time ago. Last July there was an upgrade to the protocol to implement a ‘secure’ handshake. This new spec broke most of the implementations because it mixed the upgrade headers part and the first 8 bytes from the content.

I’ve upgraded both cyclone and txwebsockets to understand the ‘old’ spec (hixie 75) and the new one. Basically the handshake involves extracting numbers from two headers (Sec-Websocket-Key1 and Sec-Websocket-Key2), dividing the resulting number by the number of spaces, concatenating them with 8 bytes read from the socket and sending back the md5 digest of this mess back after the new headers. Code to test and calculate the handshake from the headers value can be found here .

The rewrite ritual

Not that I am a fan of rewriting stuff, but there are some applications that I use to showcase technologies and from time to time I set up to rewrite them. As there is already a specification, its a matter of an exercise for new technologies. No over-specification and a very delimited time box.

Lately, two of these apps are a pastie clone and an url shortener. First I wrote them in ruby, them in python, python with twisted and now using Erlang and Mochiweb.

The code is on http://github.com/gleicon at uurl-erl and pasteme-erl repos, and as usual the html/css part is barely minimal. Also they are online at http://7co.cc and http://pasteme.7co.cc.

Both of them use Redis as a global increment and cache server, along with MongoDB for storing documents and stats. I find this to be a good match due to Redis atomic operations and pub/sub capabilities.

Websockets and RESTMQ – power to web applications

Sounds cocky but that’s it. Using RestMQ, which builds on twisted, cyclone and a stack of well proven software, you can provide your applications with a robust and flexible queue over http protocol. It already was possible using COMET and GET/POST/DELETE requests, but now with websockets support it got to a new level.

Release gibberish apart there is a cool small app which streams twitter data to a html/css/js based app. Why all that work, would you say, if I can use it directly (well, not that easy because you need to provide username/pass to twitter streaming service) ? Well, for mashing up data and filtering it before delivery. Also, twitter is a convenient source of streaming data for tests, but usually you would roll up your own data source.

An actor based concurrency model or browser-based map/reduce is trivial, just like retrieving data, programming the proper javascript code to do all the work and post i back to a new queue.

Check the repository at http://github.com/gleicon/restmq_websockets and a live demo at http://cowcat.net/restmq_ws/. Note that it needs a websockets capable browser (chrome or a recent webkit build) and it may be down due to my bandwidth constraints 🙂

Node.js and COMET

Today I started to study node.js, for these last days there was a lot of interesting posts about it.

So to begin understanding it, I setup to port two COMET based examples I did back when I started using twisted to do it.

Note that this may contain javascript misconceptions, as I’m not experienced with it and my js idiom is more geared towards python than proper js.

The first example is a basic comet server which prints a string to each connected user

simple comet

The other example sends the result of twitter searchs to all connected users:

twitter and comet

Node.JS presents a different toolset to solve the current problems we have. It used the excellent libev, among V8 and other cool stuff, and JS is a very modern and flexible language (at least more than I thought about it when trying to do web stuff).

Cheers

Web Sockets for Python

With all the fuzz around chrome and web sockets I’ve bundled up together a simple web sockets implementation for twisted. It’s a really easy way to interact with browsers.

In the next days I will run benchmarks, because I think it will stir up a bit the webserver scene a bit, and possibly, the assynchronous network frameworks too. On the bright side, its really easy to work on, and good ideas come up.

Enjoy and keep watching for new releases, as I have not finished it yet.

Repo URL: http://github.com/gleicon/txwebsockets

RESTMQ – REST/JSON/HTTP based message queue

That’s not a replay from the last post. I’ve pushed into my github repo an initial, cleaned up version of RestMQ. This is special because it not only uses python and twisted, but cyclone, the twisted-based implementation of Tornado. Repo URL:  http://github.com/gleicon/restmq

REST based message queue

I’ve been playing with lots of message queues lately, and with each of one, a protocol is favoured. There are modules for all protocols for a given MQ if you look further, but usually only one of them is more developed than others. That’s ok, because it’s hard to find willing people to help in open source projects.

Based on what I saw and used, a simple queue would suffice for most of cases. I tried Amazon SQS and the idea behind it is nice and simple. As I was looking to build a small “cloud-y” infra structure for my projects, I’ve started to build such queue using Google’s AppEngine. This was almost 1 year ago and just last week I found the original code, which immediatly I saw to clean, add API Keys and make it public.

A quick write-up on python decorators and memcached

…or how to do proper caching without heavily modifing your code.

I’ve been using this technique on django views and Juno for some time. It’s just a decorator which inspect the parameters given to a function to search for a key on memcached.

You will find two files on github: no_decorator_example.py and decorator_example_memcache.py. Both of them search on twitter for a given term. The first script executes the search everytime, and the second script implements a 60 second cache, which uses the parameter (search term) as key.

I’m sure there must be another way to deal with parameters without the wrapper class, but so far nothing I’ve tried gave the same result.

For  Django views, one could go so far as using return HttpResponse(your_response, mimetype=’text_plain’) too.

Enjoy.

Twisted COMET

I was about to write a post about NGINX, Python, Twisted and COMET, but it got so long that I decided to break it in 2 or 3 parts.

The first one is a kind of follow-up to the last post . This time the subject is a script that will search for a given word in twitter, and update the results in a continuous fashion. Once you point your browser to http://localhost:8000/ it will start to receive the results from time to time, as a big download (which is what COMET is about).

 

twisted and comet (comet in 60 seconds)

I’ve been looking into twisted to build a comet based app. It’s not a hard task, given that you can tune a lot of parameters (including which kind of reactor), but the basics are very interesting. Combining this approach along with a nginx based architecture has given me excellent results.