MongoDB and Redis for an URL shortener

I’ve wanted to prototype an URL shortener some time ago to have a measure on how long it would take to do using a NoSQL DB to store stats and do analytics. The main reason would do it using cyclone and MongoDB, and later build over it to create a tracker, web analytics tool and maybe a NGINX module.

Testing Bottle and MongoDB

Seems like DSLs for web development are getting more solid in the python camp (alto I’m yet to see something like sinatra out of the box or something that spare me from too much CSS’ing).

Today I read about Bottle and Redis, and to keep things fresh, cooked up a PasteBin clone using Bottle and MongoDB. I may do it using Redis too, just to showcase the differences. It’s really easy to come up with prototypes using Bottle.

Check it here: http://github.com/gleicon/pasteme

Node.js and COMET

Today I started to study node.js, for these last days there was a lot of interesting posts about it.

So to begin understanding it, I setup to port two COMET based examples I did back when I started using twisted to do it.

Note that this may contain javascript misconceptions, as I’m not experienced with it and my js idiom is more geared towards python than proper js.

The first example is a basic comet server which prints a string to each connected user

simple comet

The other example sends the result of twitter searchs to all connected users:

twitter and comet

Node.JS presents a different toolset to solve the current problems we have. It used the excellent libev, among V8 and other cool stuff, and JS is a very modern and flexible language (at least more than I thought about it when trying to do web stuff).

Cheers

Notes about Redis

I’ve come up with some notes about Redis, RestMQ and mongodb. Uploaded them to slideshare.

NoSQL brief (gentle) intro and some SQL anti patterns

Yesterday I’ve presented this at Locaweb/Brazil as part of our NoSQL cycle. Many other people presented interesting new ideas and I’ve opted for this intro and a piece about Redis which I may publish soon.

NoSql Introduction

View more presentations from gleicon.

Treasure the past

Just a quick one: I’ve found code from my old repository, more than 10 years old. Uploaded it along with VideoDog to my github repository: http://github.com/gleicon/archive . Gotta love old stuff.

Web Sockets for Python

With all the fuzz around chrome and web sockets I’ve bundled up together a simple web sockets implementation for twisted. It’s a really easy way to interact with browsers.

In the next days I will run benchmarks, because I think it will stir up a bit the webserver scene a bit, and possibly, the assynchronous network frameworks too. On the bright side, its really easy to work on, and good ideas come up.

Enjoy and keep watching for new releases, as I have not finished it yet.

Repo URL: http://github.com/gleicon/txwebsockets

Redis and python

One of the byproducts of RestMQ for me is that I got involved in a redis client in more ways than just using it. Along with fiorix, I got to know more of twisted and how to use it along cyclone, which is a tornado-on-twisted port. That helped to change from a pure twisted.web RestMQ to a more flexible setup.

Along the road, there was the need for a different redis client, which could handle connection pools, and there was txredisapi, which benefited from the experience in tx-redis and the original python redis client. It worked so well that we implemented some other features.

The most promising feature is not on RestMQ yet, but it’s on the master repo a git, which is Consistent Hashing. Consistent Hashing, in its various forms, enable setups resembling sharding and data distribution between two or more instances of Redis.

This is importante not only from the data partition point of view, but also from the avaliability one. I strongly believe that the client, or a intermediate layer, can control the data distribution in this way and not the storage server itself. Using txredisapi, the beggining of such architecture can be used, and later on, data as server free space, speed and I/O capacity can be put on the mix to help to decide how to populate a new server.

I invite you to check txredisapi, restmq and cyclone. My repo at http://github.com/gleicon also has clones and (sometimes) branches of these projects, so feel free to send patches and ideas.

RESTMQ – REST/JSON/HTTP based message queue

That’s not a replay from the last post. I’ve pushed into my github repo an initial, cleaned up version of RestMQ. This is special because it not only uses python and twisted, but cyclone, the twisted-based implementation of Tornado. Repo URL:  http://github.com/gleicon/restmq

REST based message queue

I’ve been playing with lots of message queues lately, and with each of one, a protocol is favoured. There are modules for all protocols for a given MQ if you look further, but usually only one of them is more developed than others. That’s ok, because it’s hard to find willing people to help in open source projects.

Based on what I saw and used, a simple queue would suffice for most of cases. I tried Amazon SQS and the idea behind it is nice and simple. As I was looking to build a small “cloud-y” infra structure for my projects, I’ve started to build such queue using Google’s AppEngine. This was almost 1 year ago and just last week I found the original code, which immediatly I saw to clean, add API Keys and make it public.