Linger, a generally useful messaging server

Building software that involves coordinating multiple processes and servers, you need a reliable means of inter-process communication that is robust against process crash and network disconnects, and that allow for variation in message processing time. Hence, you need a message exchange system and possibly a messaging queue.

Messaging tools for inter-process and inter-computer communication are widely available, and include both free and commercial software, as well as paid services.

There are:

• The general purpose message brokers such as the integrated solution RabbitMQ, or the protocol building blocks ZeroMQ),
• The database as a queue (e.g., the in-memory database Redis, or any SQL database), and
• The task queue (e.g., Celery, or the classicbeanstalkd). Each cloud IaaS vendor has their own (often proprietary) messaging solutions, such as Amazon Simple Queue Service.

You can also use simple remote procedure calls (RPC), i.e., direct process-to-process communication such as XML-RPC. This can be combined with a reverse-proxy like nginx that can distribute RPCs using a round-robin routing scheme to multiple backend-processes.

Given the array of choices, it may require some thinking to pick the right tool for your project. In particular, as your requirements may go beyond what any single solution provides.

In my experience a simple solution is better, and for most of my projects I just want:

• A message queue for distributing messages to a group of workers.
• A (pub-sub) messaging system for passing messages from publishers to subscribers.
• To use HTTP, because this protocol is widely available in programming languages, and it is easily debuggable.
• A standalone program that I can easily install and run, even on a VPS with very limited resources.

So, I went ahead and developed a combined message queue and pub-sub service with HTTP API called Linger. The project is available as open-source on Github.

Linger is a generally useful messaging server that will work for almost any project on the scale from tiny to large, where you need a work queue or to pass messages between processes and servers.

The software has some nifty features, such as persistence to an sqlite database, and the possibility to set a high-level mark on the number of messages allowed to linger in a message queue.

The project's README file provides a detailed description of the protocol. As Python is a particular favorite programming language of mine, there is also a separate project with a Linger client for Python. But, the HTTP based protocol is so simple, that you can easily implement similar client code in any language.

If you work on a project where there is a need for communicating between multiple processes or servers, give Linger a try, I'm confident it will serve you reliably.