Monday, July 28, 2014

A tryst with Message Queues Pt. 3

So once I have called it quits on the R&D, I started work on the integration thing. The problem has grown tenfold by now. The powers that be want to do a lot of messaging stuff and its become a list of at least half a dozen changes where messages are to be introduced. Now, there's another angle. I am serving notice period, the recruitment people are having hard time finding a replacement and its my last three weeks here. I should be doing the handover. Shouldn't be working on these new features - especially things as complex and less understood as this STOMP stuff. But what can you do when the powers that be want to push in code without worrying about the future of that code.
So I sat silently on this wet and cold Monday morning, trying to figure out the integration strategy. Yeah, the code is all C code, so better to keep it all together. Means a single function which connects to server, sends the message and disconnects. What can I pass? We need to fetch in settings from a config file. Alright I'll use the common config. And what parameters to this function? Maybe just the message will do? No, I need to pass the queue name too. Since there are going to be multiple queues.
OK. Plus I need to extern "C" all includes. Lets hope it doesn't screw up the already included stuff.
So I dive in the code. Add the libraries. Modify the build script to build these too.
Then I add support for the new config fields in the common config file. From there I add some more code to read the new configs.
From there I pull together the message sending code from other application which I have been using for R&D, put it all in function. Then I add some #ifdef's around all this code and add the linking and includes to the pro file.
So after messing with this for some time I finally have some messed up code ready to be compiled. Its going to be a nightmare to get this sauce working. But thankfully I have been many such nightmares already. Will be back soon with next part if I manage to get this spaghetti working.

Friday, July 25, 2014

A tryst with Message Queues Pt. 2

So far...
I'm sending data to a RabbitMQ broker as messages. These messages get queued up in a user configurable queue. Supposedly a consumer is going to process these. But the consumer is out of my view.
Doesn't matter.
Now there came a feature of receiving notifications from the consumer or some other system. So after some discussion, the powers to be decided that we are going to have a notification queue which the client has to read. And from it we get the notifications to be read. Now the shady part of business was I had no idea what the hell is going on the inside of libStomp. So I ask politely that I don't have any read code. And I get handed over a single stomp_read code snippet. (In whiny voice) They said it works!!! But it doesn't!!! I tried so hard!!! Damn!!!
So I looked into the protocol to understand why the hell I am no getting anything. After reading the protocol, I came to know that the implementation is fucked up in ways more than a few. First STOMP is a message send and receive protocol. Which means when we prepare a CONNECT frame, we are supposed to get a reply as CONNECTED along with server connection settings.
Now the original code sample which I was using for sending the data had all stomp_read calls commented out. And after I tried to run that sample with the read calls uncommented. And there I also ran into the same problem. That meant there could be either a server problem or a client problem.
I didn't have any other verified STOMP server, So I decided to check if RabbitMQ is the culprit. Thankfully at the stomp_ plugin page on RabbitMQ website, they have clearly given this procedure. After trying out the telnet routine I ran into an interesting problem.
At first CONNECT, I got invalid error command. Next any CONNECT worked out. After scratching my head for a while I remembered that putty does active negotiation for telnet connection. So I went into putty settings and tried out passive negotiation. That worked. And it also meant that RabbitMQ server is operating fine.
So the culprit must be libStomp. But even after extended search on the net I didn't get any occurrence of my problem.
Later after going through all the protocol stuff I tried out SUBSCRIBE command and even though RabbitMQ logs suggested that the operation has succeed, there was no response read.
That's the current state of things. I have given up on reading up stuff and instead I am going to integrate the working bits into our product. Later I'll see if the reading is working. Else I am thinking of writing my own STOMP message since its all text anyway.
Let's see!!!

Thursday, July 24, 2014

A tryst with Message Queues Pt. 1

I need to give some background for this situation.
Its like this: we have a client server product. The server is written in .net and performs analysis on raw data collected from a number of clients and prepares various reports. The client software is written in c++/Qt and it interacts with various hardware controllers/ sensors to get the raw data.
There is a sync program written in .net that runs on the server and it pulls data from all the clients. The data is saved in servers PostgreSQL database for later processing.
Now, the powers to be decided that we need to remove this syncing program ( because it sunk the data into oblivion sometimes! :D ). So what's the replacement? Well, they decided on RabbitMQ as the queuing solution and decided that the clients would sync their own data to the queue and from there it will be fetched by the server.
So we as clients needed to have this functionality in our application. But after looking at the complexity of our own client, we decided to create a simple up sync application which will be completely separate from our client. Its sole responsibility will be to ensure that all data in the database is sent to the message Queue.
The powers that be decided to use RabbitMQ as messageQueuing server platform. Let me tell you, this is one software whose configuration is a headache. The horrible syntax required for the config file gets to you.
But I somehow survived to tell the tale. Anyway, since my domain knowledge about this was absolutely zero, I spent some time reason up the stuff and getting familiar wight the trade lingo.
Then I used the .net client to check out my clueless config and needed few more days to get it to my taste. After that I went on to compile libApr, and libStomp, and then use it all in my own application.
This would have turned out nightmarish but thankfully I got hold of the compilation commands on internet. Then I spent next couple of days writing the database interfacing stuff. Once I furnished that, I went on to integrate libStomp code.
That took few more days and finally I was sending some stuff to RabbitMQ. I didn't know how to check if correct data is going in. Looking around I found info about management plugin. So enabling it let me confirm the data. Some bugfixes later I was pushing thousands of rows successfully.
Mind you though I still didn't know exactly what the hell was happening inside LibStomp. I came to know that.
In next part.