October 28, 2009
I was about to write a post about NGINX, Python, Twisted and COMET, but it got so long that I decided to break it in 2 or 3 parts.
The first one is a kind of follow-up to the last post . This time the subject is a script that will search for a given word in twitter, and update the results in a continuous fashion. Once you point your browser to http://localhost:8000/ it will start to receive the results from time to time, as a big download (which is what COMET is about).
We take advantage that thw browser render as it receives the data to force a scroll down after each search. Also it respects the last returned id, so the updates are minimal and with no old data.
The goal is a full application stack, using NGINX as a front end, a pool of twisted processes as backend and a shared session (hopefully) to broadcast updates between all backend instances.
Check out the code for twitmonitor.py at http://github.com/gleicon/zenmachine.
There is a task to trigger the search, setup as
loopingCall = task.LoopingCall(self.__update_search) loopingCall.start(20, False)
This task calls self.__update_search which first checks if there are anyone connected, and performs a call to search.twitter.com to gather new results. Note that I used twisted’s own http client. A Deferred object is returned, and its callback is set to self.__process_search, which filter, and send the results to all conected users.
The list self.presence, is a list of requests objects. It acts as a central presence manager. Each new request to render_GET ends up added to this list.
Twisted makes it easy to build these kind of programs. It pays off to know how deferreds and tasks works. Also, you can choose which reactor is better for your operation system.
I test it with $ twistd -ny twitmonitor.py and run like $ twistd –pidfile=/var/run/comet.pid –logfile=/var/log/comet.log –reactor=epoll –uid=www-data –gid=www-data –python=twitmonitor.py .