Learnings from a project

Hi,

My latest project at my job has been going on for one full year now. It was predicted that it would be over in about six months but it has since dragged over to one year and the end is not yet in sight.

I got to learn a lot in this particular project.

Lesson 1: Don't ever go for short-cuts. They usually cut you short.

The upper management decided to go for code conversion. The existing product was coded in .Net. The upper management decided that they wanted to cut the costs. So they decided to shift to Linux. The hardware they settled on was an Arm board. And on top of it we would be using Qt/Linux to create the application. There were two approaches. a. Write fresh code while recreating the application. b. Use some automated tool to convert the code and then debug it.
Management didn't think enough and went straight for approach b. So we got hold of a tool that would convert the .Net code into C++. And the nightmare began.

Lesson 2: Don't trust any automated tool.

As all automated tools/ processes go, they are not replacements for human beings. But when your whole development strategy depends upon one single tool that may screw up everything if its not effective enough, there's a very high chance that you would end up on the wrong side of the park.
That's what happened here. The converted code was total junk. The team lost so much time on cleaning up the converted code and then making it even compilable, that the schedules got scrapped multiple times.

Lesson 3: Don't put more naive people to speed up development.

The management thought that if we could apply more manpower, the baby would be delivered in less time. So things became total mess when a number of new people joined the team and we didn't have any team structure to support them. Making them productive caused a drop in actual team efforts. And so time-lines slipped further.

Lesson 4: Don't think that development on ARM is easy or in any way related to x86 development.

Developing for arm is a totally different ballgame. You don't have the flexibility when you're working with X86. Here every time you have to make changes to the code, that involves compiling it, copying it to SD Card or similar media, and go to the board, test it, or maybe just get the debug logs and then come back to your workstation. Damn! It's enormously inefficient. And exhausting. And frustrating. And boring as hell especially when the same board is shard between 4-5 people. And sometimes even more if the feature you're working on(e.g. GPS), is working only on one certain setup.

Lesson 5: Debugging on ARM is hard, especially your code base is ugly and the product is a maze of unintelligible features.

By the time, we put together an intermediate release that would give some confidence to the Customers and the management, we reached a stage where we needed a good debugger. And we had none. And nobody bothered with these basic things that every developer takes for granted. So the team ended up putting 2x debug statements in the code, making a total mess of things.
When one of us frustrated guys made GDB work on the target, he was loaded with all debugging, ensuring that everybody else would keep their innovative ideas to themselves.

Lesson 6: Don't think that people are tools. They are not.

Through out this project the attitude of the management was that people are tools and they are to be used hard. Maybe thats why they ended up first hiring a number of people when there was not that much need and then firing half of them. Seriously people management is not a joke. But here it was made to be a joke.

Lesson 7: Don't think you can optimize anything in one or two weeks.

Optimization a C++ program is a whole different ballgame. If you are even thinking of optimization, be ready to dig into the annals of C++ and then drawn in details which seem to stretch from application code to system features. Get ready for long sessions of staring at 'top' output with CPU utilization and memory figures. Be ready to write system monitoring scripts and chart graphs of performance to confirm the suspicisions about a stray thread eating CPU.
Optimization is hard when your code base is not even stable and not feature complete. But the idiots in upper echelon will tell you otherwise. And so by the end of two allocated weeks you would end up with an even more unstable product than before. Remember writing code is easy. But writing good code is very hard. And the idiots in upper echelon don't know or care for good code.

Lesson 8: Don't think that code cleaning is worthless.

When you have number of people with varying skillset working on the code, it becomes very important to keep track of the code quality and maintain effective reviews. If you cannot maintain the code quality then you end up with code that is just totally unmaintainable. A total disaster. And then nobody can save your project from doom.

Lesson 9: Don't think hardware will behave.

Code that interact with hardware need to be extra robust. Because hardware is often notorious with giving you garbage/invalid data any time. And if someone with half-a-year of experience has written that code, it's going to fail. So ensure extra care when dealing with hardware - especially faulty hardware - which half of our development boards seem to be!

Lesson 10: Don't think that saving on hardware would be great, if you fail to see the cost of development and the product running late to market.

For any product the decision makers need to carefully consider how they are going to actually develop for any platform. And the tradeoffs need to be considered too. An X86/Linux based platform might cost a bit more but if you need the product ASAP, then it's easier with X86. There are a lot of tools available that can speed up your development. And there's not much cost of API/ABI translation as the one seen with Arm/Linux. Also you have the latest tools/ libraries available that make the life of developers easier.
But the decision makers of course have to consider all these options. Otherwise you end up with a project like ours where every day is a nightmare. And meeting deadlines is worse than fighting the undead.

Life in a traffic jam they say!!!