CSV HELL!!!

-
Recently I had the pleasure to work with CSV files. And it ended in a CSV hell for me! The functionality I wanted wasn't available anywhere (As often happens when you provide custom solutions to your clients!) The client didn't want to rely completely on the system! (I question the need for the system in such circumstances, but the client wanted the system! Brilliant! Kind of like 'hey, I'll be travelling in Bus but just in case lets have that train available and tracks laid out!)
Anyway, I couldn't get them to change their preference and so this hell started. Now the situation is the client uses OpenOffice and/or MS Excel to edit the CSV's. They have excelsheets for the data and they export that to csv. I assumed I could write a parser for CSV and that would be it. But when I got to the task I just couldn't figure out the correct logic. One logic worked for some and failed for some other data and the other worked for some and failed for some other.
After playing with it for couple of days, I was really frustrated enough that I dumped the csv thing and instead used single quotes as field delimiters and "|" as separators. This made my logic simple and in 2-3 days I had a working parser for these files.
Next hard part come when I tried exporting excel sheets from both MS Excel and Open Office - both would screw up the file somehow - sometimes it was the field delimiters, sometimes the separators. After wasting one more day on this, I finally resorted to manually editing the files. GEdit's find and replace function came really handy then.
Four more days later I had the supposed to be csv's ready and I finished the task. It shouldn't have taken  that much time but as it happens the raw data coming from client had so many problems in it, like improper fields, fields exceeding lengths(these brought in duplicate record errors in mysql), and many others.
All through and through, it was quite an experience for me to work on this challenge.

Comments

Post a Comment

Popular posts from this blog

Morning Quotes

-
Image
Source: www.brainyquote.com   I may be drunk, Miss, but in the morning I will be sober and you will still be ugly. Winston Churchill   Prayer is the key of the morning and the bolt of the evening. Mohandas Gandhi   I look upon death to be as necessary to our constitution as sleep. We shall rise refreshed in the morning. Benjamin Franklin  
Read more

QCalendarWidget CSS Stylesheeting

-
Recently I had to implement a calendar for a TouchScreen application. I resorted to css to modify the calendar widget's appearance so that it would be more accessible. Following is the code that enabled me to get a proper calendar. calendar->setStyleSheet("QMenu { font-size:20px; width: 300px; left: 20px; }"     "QToolButton {icon-size: 48px, 48px; font: 20px; height: 70px; width: 100px; }"     "QAbstractItemView {selection-background-color: rgb(255, 174, 0);}"     "QToolButton::menu-indicator{ width: 50px;}"     "QToolButton::menu-indicator:pressed,"     "QToolButton::menu-indicator:open{ top:10px; left: 10px;}"     "QTableView { font: 30px; }"     "QSpinBox { width: 100px; font: 20px;}"     "QSpinBox::up-button { width:40px;}"     "QSpinBox::down-button { width:40px;}");
Read more

Selecting new phone plan

-
I use a Reliance Jio SIM as my one and only cellphone number. I used to have another, but I found little use for that. So I discontinued that other SIM card. It was a Vodafone SIM and I had bought it thinking I would be using it when I visit my hometown. Turns out it's not usable there. So adios. Coming back to Jio, it was working so far so good for a while. And it was quite cheap. Then they increased the fares about 4-5 months ago. And it was costing ~Rs 7 per day. This was for unlimited Jio-Jio calling, ~300minutes free to other telcos, and 1.5GB daily cellular data. Now I got home internet, so I thought to cut costs on this. The current plan goes for INR555 for 84 days. This includes 1.4GB 4G cellular data per day, 300 min calling to other telcos, and jio-jio free. That's Rs 6.6 per day. Now I'm not even using 30minutes calling per day. And due to home internet my cellular usage has gone down. So I decided to go for some other cheaper plan. This one is INR1299 for 336 da...
Read more