Thursday, July 24, 2014

A tryst with Message Queues Pt. 1

I need to give some background for this situation.
Its like this: we have a client server product. The server is written in .net and performs analysis on raw data collected from a number of clients and prepares various reports. The client software is written in c++/Qt and it interacts with various hardware controllers/ sensors to get the raw data.
There is a sync program written in .net that runs on the server and it pulls data from all the clients. The data is saved in servers PostgreSQL database for later processing.
Now, the powers to be decided that we need to remove this syncing program ( because it sunk the data into oblivion sometimes! :D ). So what's the replacement? Well, they decided on RabbitMQ as the queuing solution and decided that the clients would sync their own data to the queue and from there it will be fetched by the server.
So we as clients needed to have this functionality in our application. But after looking at the complexity of our own client, we decided to create a simple up sync application which will be completely separate from our client. Its sole responsibility will be to ensure that all data in the database is sent to the message Queue.
The powers that be decided to use RabbitMQ as messageQueuing server platform. Let me tell you, this is one software whose configuration is a headache. The horrible syntax required for the config file gets to you.
But I somehow survived to tell the tale. Anyway, since my domain knowledge about this was absolutely zero, I spent some time reason up the stuff and getting familiar wight the trade lingo.
Then I used the .net client to check out my clueless config and needed few more days to get it to my taste. After that I went on to compile libApr, and libStomp, and then use it all in my own application.
This would have turned out nightmarish but thankfully I got hold of the compilation commands on internet. Then I spent next couple of days writing the database interfacing stuff. Once I furnished that, I went on to integrate libStomp code.
That took few more days and finally I was sending some stuff to RabbitMQ. I didn't know how to check if correct data is going in. Looking around I found info about management plugin. So enabling it let me confirm the data. Some bugfixes later I was pushing thousands of rows successfully.
Mind you though I still didn't know exactly what the hell was happening inside LibStomp. I came to know that.
In next part.

No comments:

Post a Comment