Twisted COMET

October 28, 2009

I was about to write a post about NGINX, Python, Twisted and COMET, but it got so long that I decided to break it in 2 or 3 parts.

The first one is a kind of follow-up to the last post . This time the subject is a script that will search for a given word in twitter, and update the results in a continuous fashion. Once you point your browser to http://localhost:8000/ it will start to receive the results from time to time, as a big download (which is what COMET is about).


We take advantage that thw browser render as it receives the data to force a scroll down after each search. Also it respects the last returned id, so the updates are minimal and with no old data.

The goal is a full application stack, using NGINX as a front end, a pool of twisted processes as backend and a shared session (hopefully) to broadcast updates between all backend instances.

Check out the code for at

There is a task to trigger the search, setup as

 loopingCall = task.LoopingCall(self.__update_search)
 loopingCall.start(20, False)

This task calls self.__update_search which first checks if there are anyone connected, and performs a call to to gather new results. Note that I used twisted’s own http client. A Deferred object is returned, and its callback is set to self.__process_search, which filter, and send the results to all conected users.

The list self.presence, is a list of requests objects. It acts as a central presence manager.  Each new request to render_GET ends up added to this list.

Twisted makes it easy to build these kind of programs. It pays off to know how deferreds and tasks works. Also, you can choose which reactor is better for your operation system.

I test it with $ twistd -ny and run like $  twistd –pidfile=/var/run/ –logfile=/var/log/comet.log  –reactor=epoll –uid=www-data –gid=www-data – .



Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: