Saturday, August 29, 2015

Debug mode and release mode builds

Recently it just happened that the current product I'm working on had a crisis situation. What happened was testing found a number of inexplicable and inconsistent crashes in the product.
We didn't have a debugger back then. So to fix these things, we got gdb compiled for target. Then couple more weeks went by in us trying to find the best way to debug the application.
Ultimately we found that autogenerating core dumps for debug mode build is best solution. But as soon as we prepared debug mode build we found a number of other crashes in functionality that's properly working in release mode.
To find out the reason, after numerous internet searches, we found there are some initialisation differences in debug mode n badly written code can cause it.
So we went back n started fixing it.
Now I was thinking about the RCA for this debacle. And I realised that we all were going in the opposite way at this. Basically everybody develops in debug mode and after codebase is stable they give release in release mode.
We have been developing release mode itself. That's first issue. We didn't have a debugger that's second issue. And we didn't have a working debugging strategy, that's the third issue.
Anyway, we have incorporated these changes in development process. And things have become much better since.
So lesson learned, whenever a project is started have all general strategies in place like how its going to be developed, how it will be debugged, how it will be tested, especially its an embedded application. Since embedded development can complicate things further.

Wednesday, August 26, 2015

Viewing threads in top or ps


When we are dealing with multi threaded applications, we can use top or P's to see wayward threads.

PS has -L option to view threads.
Top has -H option to view threads.
You can use top -n 1 command in to create a simple 'top' logging script.