Sunday, December 20, 2015

Distroshare Ubuntu Imager - Remastersys replacement

I have been Remastersys to backup my customized distros for what seems like years. Its been a wonderful utility nicely tailored for this task.
But its development stopped few years back, and so there have been couple of alternatives for that.
Just recently I had to put together an ISO of my current OS setup on one of my laptops. So digging around I stumbled upon distroshare utility. This is a nice replacement for Remastersys. Its actually called Distroshare Ubuntu Imager.
After trying it out I found that it does exactly what I was looking for. And I successfully created a Live CD backup using distroshare.

Saturday, December 5, 2015

TCP/IP Client server persistent connections

So just last week I was moved to another project where I'm gonna be working on a client server program.
So we are going to be using TCP/IP for communication. And its exciting since its been over two years since I last dabbled in networking. Life is looking exciting.
Anyway so the problem was everywhere I looked, I could find only simple networking examples. So I dug in and put together some code to keep a connection alive for back and forth connection. What I did was basically keep the connection open for next request rather than closing it and reopening it.
This enabled me to have a continuous chat like communication between client and server.
Then I spent some time refactoring the code to make it streamlined.
But I still have to add error handling to it. I'll share it here when I have it ready. It was fun!!!

Monday, October 19, 2015

Limiting CPU percentage of a process

cpulimit is a utility available on Linux to limit the CPU time of any process.
The use is
cpulimit -l [℅ CPU] -e [binary name]
I found it useful to emulate slow embedded processor behavior on desktop.

Thursday, September 17, 2015

Qt Object Thread Affinity

Example code:

For last whole year I have been totally lost between threads, mutexes, shared data, pointers, signals and slots and blah blah blah.
Actually I have learned a heck of lot about all these things in these few months. But I think I haven't grasped everything yet. There are many combinations that I don't know work or not. And I think there are many other tricks I am yet to learn especially when it comes to debugging.

Anyway, this post is about QThread and object affinity. Well, the typical QThread example is way too simple. They show an object containing one worker function and that worker function running in that thread when they start the thread. Beyond that, nothing! What happens if you call any function on that object, what happens? What about concurrency? What about when the thread is busy? Will the slots get queued up? Lots and lots of questions. But the real reason for all these questions is that I faced all these issues in last one year. And I had to make sense of things as usual while trying to wrap up the work.

But it was some invaluable learning from System design point of view. Lost of threads probably equals concurrency issues. Too few threads, application response slow because of slots getting queued up. So it becomes imperative to make sense of things.

Anyway, to make sense of things I have prepared a small example. This is based off Qt4.8. I know it's not the latest and there's a fair chance that in Qt 5.4 which is the latest as of writing this, there may be some changes. But I hope things don't make this totally obsolete.

So lets refer to the code above. What I have done is I have created a small GUI application, that has a three buttons. They trigger three signals in the MainWindow class. I have another class called PObject. I have used objects of this Class to demonstrate how signals/slots, object usage and direct function calls work when the object is moved to another thread (lets say worker thread).

So we have the first object pobj. I am right away moving this to worker thread. And then connecting the three signals to three slots on this object. I have connected first signal with Qt::DirectConnection. That's because I wanted to see the exact effect of direct connection has one the slot execution.

I have taken another object called nobj which I have shuffled a bit here and there. Basically at first I didn't move nobj to worker thread. So the result was as follows: The slot connected with Qt::DirectConnection always ran on the main thread, means it runs on the thread from which the signal is emitted. The other slots run on the worker thread. Also if you press buttons 2 and 3 one after another inside five seconds, then the slot for Button3 is queued on worker thread.

All good till this point, and just like things used to work. One important thing is, the standard example showed that the function which was connected to worker thread's started() signal ran on the worker thread. And they didn't show any other function or slot running on worker thread, which made me assume that only one function (or rather slot) ran on the worker thread. But this is not true, the thread stays running and the slots keep getting executed on that thread. This example proved that.

Now my next question was what about if we call any function from object pobj from mainWindow? Ideally it should run on main thread? Or does it runs on worker thread? Maybe because pobj is moved to worker thread?
Well, the calls to someFunction() on pobj and nobj prove that it's not the case. If you invoke any pobj/nobj function from main, then it will run on main thread. This is logical since when we think about it a bit, actually running it on any other thread doesn't make sense. I mean we don't know if that thread is busy or not, how much load it has, and also we have control over main thread so might as well execute this on main.

The real question here is what if this function is modifying some data? Then this function being a slot, there's probability that it may get invoked on the thread as a result of a signal emit. And this could cause a crash.

So yes, there's a possibility of concurrency issue. So we need to ensure that whatever data we are accessing is locked down with the help of mutex.

Now what would happen if we tried to move nobj to worker thread while worker thread is already running? As it happens you can do that. And then the slots will execute on worker thread depending on their connection type. Except maybe you need to ensure that the object is moved to the thread and only after the signals / slots are connected.

Huh... that was exhausting. But I think I have finally gotten a good grip of QThreads and the various issues surrounding them.

Sunday, September 13, 2015

sudo apt-get remove linux-image-3.13.0-4*

*update: It worked!!!
In a hurry to remove multiple old linux kernel versions from my system I popped in above command. After a while I came back to the terminal to find all my kernels were un-installed!!! :J
Well, thankfully I noticed what has happened and right away reinstalled the latest one with this command:
sudo apt-get install linux-image-3.13.0-62-generic.
Haven't rebooted yet. Let see!

Qt global thread pool max thread count

Recently I encountered that some threads are not launching in our application using QConcurrentRun.

If you launched the same worker function over QThread it was working.  
So digging into this gave us this problem with defaults for Qt's global threadpool.

Basically Qt sets global threadpool's max thread count to the no of cores you have on your processor. We were easily running out of the default four max threads.

So we upped the count to 32 using following function: 


We set this at start of the program in main. Things have been fine since then!

Remove BOM (Byte-order mark) from a file

Got this script from following page:


set -o nounset
set -o errexit


if [ $(uname) == "SunOS" ] ; then
  if [ -x /usr/gnu/bin/sed ] ; then
    echo "Using GNU sed..."
  TMP_OPTS="-p "

if [ $(uname) == "Darwin" ] ; then
  TMP_OPTS="-t tmp"

  SED_EXEC="perl -pe"
  echo "Using perl..."


function usage() {
  echo "bom-remove [-adrx] [-s sed-name] [-e ext] files..."
  echo ""
  echo "  -a    Remove the BOM throughout the entire file."
  echo "  -e    Look only for files with the chosen extensions."
  echo "  -d    Do not overwrite original files and do not remove temp files."
  echo "  -r    Scan subdirectories."
  echo "  -s    Specify an alternate sed implementation."
  echo "  -x    Don't descend directories in other filesystems."

function checkExecutable() {
  if ( ! which "$1" > /dev/null 2>&1 ); then
    echo "Cannot find executable:" $1
    exit 4

function parseArgs() {
  while getopts "adfrs:e:x" flag
    case $flag in
      r) RECURSIVE=true ;;
      f) PROCESSING_FILES=true ;;
      s) SED_EXEC=$OPTARG ;;
      e) USE_EXT=true ; FILE_EXT=$OPTARG ;;
      d) DELETE_ORIG=false ; DELETE_FLAG="-d" ;;
      x) XDEV="-xdev" ;;
      *) echo "Unknown parameter." ; usage ; exit 2 ;;

  shift $(($OPTIND - 1))

  if [ $# == 0 ] ; then
    exit 2;

  # fixing darwin
  if [[ $ISDARWIN == true && $PROCESSALLFILE == false ]] ; then
    echo "Process all file is implicitly set on Darwin."


  if [ ! -n "$FILES" ]; then
    echo "No files specified. Exiting."

  if [ $RECURSIVE == true ]  && [ $PROCESSING_FILES == true ] ; then
    echo "Cannot use -r and -f at the same time."
    exit 1

  checkExecutable $SED_EXEC
  checkExecutable $TMP_CMD

function processFile() {
  if [ $(uname) == "Darwin" ] ; then
    TEMPFILENAME=$($TMP_CMD $TMP_OPTS"$(dirname "$1")")
  echo "Processing $1 using temp file $TEMPFILENAME"

  if [ $PROCESSALLFILE == false ] ; then
    cat "$1" | $SED_EXEC '1 s/\xEF\xBB\xBF//' > "$TEMPFILENAME"
    cat "$1" | $SED_EXEC 's/\xEF\xBB\xBF//g' > "$TEMPFILENAME"

  if [ $DELETE_ORIG == true ] ; then
    if [ ! -w "$1" ] ; then
      echo "$1 is not writable. Leaving tempfile."
      echo "Removing temp file..."
      mv "$TEMPFILENAME" "$1"

function doJob() {
  # Check if the script has been called from the outside.
  if [ $PROCESSING_FILES == true ] ; then
    for i in $(seq 1 ${#FILES[@]})
      echo ${FILES[$i-1]}
      processFile "${FILES[$i-1]}"

    # processing every file
for i in $(seq 1 ${#FILES[@]})
      # checking if file or directory exist
      if [ ! -e "$CURRFILE" ] ; then echo "File not found: $CURRFILE. Skipping..." ; continue ; fi
      # if a paremeter is a directory, process it recursively if RECURSIVE is set
      if [ -d "$CURRFILE" ] ; then
        if [ $RECURSIVE == true ] ; then
          if [ $USE_EXT == true ] ; then
            find "$CURRFILE" $XDEV -type f -name "*.$FILE_EXT" -exec "$0" $DELETE_FLAG $PROCESSALLFILE_FLAG -f "{}" \;
            find "$CURRFILE" $XDEV -type f -exec "$0" $DELETE_FLAG $PROCESSALLFILE_FLAG -f "{}" \;
          echo "$CURRFILE is a directory. Skipping..."
        processFile "$CURRFILE"

parseArgs "$@"

Assuming the script is in your $PATH and it's called bom-remove, you can "clean" a bunch of files invoking it this way:

$ bom-remove file-to-clean ...

If you want to clean the files in an entire directory, you can use the following syntax:

$ bom-remove -r dir-to-clean ...

If your sed installation is not in your $PATH or you have to use an alternate version, you can invoke the script with the following syntax:

$ bom-remove -s path/to/sed file-to-clean ...

If you want to clean a directory in which other file systems might be mounted, you can use the -x option so that the script does not descend them:

$ bom-remove -xr dir-to-clean ...

Seriously!!! This literally saved my day!!!

Saturday, August 29, 2015

Debug mode and release mode builds

Recently it just happened that the current product I'm working on had a crisis situation. What happened was testing found a number of inexplicable and inconsistent crashes in the product.
We didn't have a debugger back then. So to fix these things, we got gdb compiled for target. Then couple more weeks went by in us trying to find the best way to debug the application.
Ultimately we found that autogenerating core dumps for debug mode build is best solution. But as soon as we prepared debug mode build we found a number of other crashes in functionality that's properly working in release mode.
To find out the reason, after numerous internet searches, we found there are some initialisation differences in debug mode n badly written code can cause it.
So we went back n started fixing it.
Now I was thinking about the RCA for this debacle. And I realised that we all were going in the opposite way at this. Basically everybody develops in debug mode and after codebase is stable they give release in release mode.
We have been developing release mode itself. That's first issue. We didn't have a debugger that's second issue. And we didn't have a working debugging strategy, that's the third issue.
Anyway, we have incorporated these changes in development process. And things have become much better since.
So lesson learned, whenever a project is started have all general strategies in place like how its going to be developed, how it will be debugged, how it will be tested, especially its an embedded application. Since embedded development can complicate things further.

Wednesday, August 26, 2015

Viewing threads in top or ps


When we are dealing with multi threaded applications, we can use top or P's to see wayward threads.

PS has -L option to view threads.
Top has -H option to view threads.
You can use top -n 1 command in to create a simple 'top' logging script.