Thursday, September 17, 2015

Qt Object Thread Affinity









Example code:
https://drive.google.com/file/d/0BxorjNRCBW61ZEJhaEJST1RZa0k/view?usp=sharing


For last whole year I have been totally lost between threads, mutexes, shared data, pointers, signals and slots and blah blah blah.
Actually I have learned a heck of lot about all these things in these few months. But I think I haven't grasped everything yet. There are many combinations that I don't know work or not. And I think there are many other tricks I am yet to learn especially when it comes to debugging.

Anyway, this post is about QThread and object affinity. Well, the typical QThread example is way too simple. They show an object containing one worker function and that worker function running in that thread when they start the thread. Beyond that, nothing! What happens if you call any function on that object, what happens? What about concurrency? What about when the thread is busy? Will the slots get queued up? Lots and lots of questions. But the real reason for all these questions is that I faced all these issues in last one year. And I had to make sense of things as usual while trying to wrap up the work.

But it was some invaluable learning from System design point of view. Lost of threads probably equals concurrency issues. Too few threads, application response slow because of slots getting queued up. So it becomes imperative to make sense of things.

Anyway, to make sense of things I have prepared a small example. This is based off Qt4.8. I know it's not the latest and there's a fair chance that in Qt 5.4 which is the latest as of writing this, there may be some changes. But I hope things don't make this totally obsolete.

So lets refer to the code above. What I have done is I have created a small GUI application, that has a three buttons. They trigger three signals in the MainWindow class. I have another class called PObject. I have used objects of this Class to demonstrate how signals/slots, object usage and direct function calls work when the object is moved to another thread (lets say worker thread).

So we have the first object pobj. I am right away moving this to worker thread. And then connecting the three signals to three slots on this object. I have connected first signal with Qt::DirectConnection. That's because I wanted to see the exact effect of direct connection has one the slot execution.

I have taken another object called nobj which I have shuffled a bit here and there. Basically at first I didn't move nobj to worker thread. So the result was as follows: The slot connected with Qt::DirectConnection always ran on the main thread, means it runs on the thread from which the signal is emitted. The other slots run on the worker thread. Also if you press buttons 2 and 3 one after another inside five seconds, then the slot for Button3 is queued on worker thread.

All good till this point, and just like things used to work. One important thing is, the standard example showed that the function which was connected to worker thread's started() signal ran on the worker thread. And they didn't show any other function or slot running on worker thread, which made me assume that only one function (or rather slot) ran on the worker thread. But this is not true, the thread stays running and the slots keep getting executed on that thread. This example proved that.

Now my next question was what about if we call any function from object pobj from mainWindow? Ideally it should run on main thread? Or does it runs on worker thread? Maybe because pobj is moved to worker thread?
Well, the calls to someFunction() on pobj and nobj prove that it's not the case. If you invoke any pobj/nobj function from main, then it will run on main thread. This is logical since when we think about it a bit, actually running it on any other thread doesn't make sense. I mean we don't know if that thread is busy or not, how much load it has, and also we have control over main thread so might as well execute this on main.

The real question here is what if this function is modifying some data? Then this function being a slot, there's probability that it may get invoked on the thread as a result of a signal emit. And this could cause a crash.

So yes, there's a possibility of concurrency issue. So we need to ensure that whatever data we are accessing is locked down with the help of mutex.

Now what would happen if we tried to move nobj to worker thread while worker thread is already running? As it happens you can do that. And then the slots will execute on worker thread depending on their connection type. Except maybe you need to ensure that the object is moved to the thread and only after the signals / slots are connected.

Huh... that was exhausting. But I think I have finally gotten a good grip of QThreads and the various issues surrounding them.


Sunday, September 13, 2015

sudo apt-get remove linux-image-3.13.0-4*

*update: It worked!!!
In a hurry to remove multiple old linux kernel versions from my system I popped in above command. After a while I came back to the terminal to find all my kernels were un-installed!!! :J
Well, thankfully I noticed what has happened and right away reinstalled the latest one with this command:
sudo apt-get install linux-image-3.13.0-62-generic.
Haven't rebooted yet. Let see!

Qt global thread pool max thread count

Recently I encountered that some threads are not launching in our application using QConcurrentRun.

If you launched the same worker function over QThread it was working.  
So digging into this gave us this problem with defaults for Qt's global threadpool.

Basically Qt sets global threadpool's max thread count to the no of cores you have on your processor. We were easily running out of the default four max threads.

So we upped the count to 32 using following function: 

QThreadPool::globalInstance()->setMaxThreadCount(32);

We set this at start of the program in main. Things have been fine since then!

Remove BOM (Byte-order mark) from a file

Got this script from following page:
http://thegreyblog.blogspot.in/2010/09/shell-script-to-find-and-remove-bom.html


#!/bin/bash


set -o nounset
set -o errexit


DELETE_ORIG=true
DELETE_FLAG=""
RECURSIVE=false
PROCESSALLFILE=false
PROCESSING_FILES=false
PROCESSALLFILE_FLAG=""
SED_EXEC=sed
USE_EXT=false
FILE_EXT=""
TMP_CMD="mktemp"
TMP_OPTS="--tmpdir="
XDEV=""
ISDARWIN=false


if [ $(uname) == "SunOS" ] ; then
  if [ -x /usr/gnu/bin/sed ] ; then
    echo "Using GNU sed..."
    SED_EXEC=/usr/gnu/bin/sed
  fi
  TMP_OPTS="-p "
fi


if [ $(uname) == "Darwin" ] ; then
  TMP_OPTS="-t tmp"

  SED_EXEC="perl -pe"
  echo "Using perl..."
  ISDARWIN=true

fi


function usage() {
  echo "bom-remove [-adrx] [-s sed-name] [-e ext] files..."
  echo ""
  echo "  -a    Remove the BOM throughout the entire file."
  echo "  -e    Look only for files with the chosen extensions."
  echo "  -d    Do not overwrite original files and do not remove temp files."
  echo "  -r    Scan subdirectories."
  echo "  -s    Specify an alternate sed implementation."
  echo "  -x    Don't descend directories in other filesystems."
}


function checkExecutable() {
  if ( ! which "$1" > /dev/null 2>&1 ); then
    echo "Cannot find executable:" $1
    exit 4
  fi
}


function parseArgs() {
  while getopts "adfrs:e:x" flag
  do
    case $flag in
      a) PROCESSALLFILE=true ; PROCESSALLFILE_FLAG="-a" ;;
      r) RECURSIVE=true ;;
      f) PROCESSING_FILES=true ;;
      s) SED_EXEC=$OPTARG ;;
      e) USE_EXT=true ; FILE_EXT=$OPTARG ;;
      d) DELETE_ORIG=false ; DELETE_FLAG="-d" ;;
      x) XDEV="-xdev" ;;
      *) echo "Unknown parameter." ; usage ; exit 2 ;;
    esac
  done


  shift $(($OPTIND - 1))



  if [ $# == 0 ] ; then
    usage;
    exit 2;
  fi



  # fixing darwin
  if [[ $ISDARWIN == true && $PROCESSALLFILE == false ]] ; then
    PROCESSALLFILE=true
    echo "Process all file is implicitly set on Darwin."
  fi

  FILES=("$@")


  if [ ! -n "$FILES" ]; then
    echo "No files specified. Exiting."
  fi


  if [ $RECURSIVE == true ]  && [ $PROCESSING_FILES == true ] ; then
    echo "Cannot use -r and -f at the same time."
    usage
    exit 1
  fi


  checkExecutable $SED_EXEC
  checkExecutable $TMP_CMD
}


function processFile() {
  if [ $(uname) == "Darwin" ] ; then
    TEMPFILENAME=$($TMP_CMD $TMP_OPTS)
  else
    TEMPFILENAME=$($TMP_CMD $TMP_OPTS"$(dirname "$1")")
  fi
  echo "Processing $1 using temp file $TEMPFILENAME"


  if [ $PROCESSALLFILE == false ] ; then
    cat "$1" | $SED_EXEC '1 s/\xEF\xBB\xBF//' > "$TEMPFILENAME"
  else
    cat "$1" | $SED_EXEC 's/\xEF\xBB\xBF//g' > "$TEMPFILENAME"
  fi


  if [ $DELETE_ORIG == true ] ; then
    if [ ! -w "$1" ] ; then
      echo "$1 is not writable. Leaving tempfile."
    else
      echo "Removing temp file..."
      mv "$TEMPFILENAME" "$1"
    fi
  fi
}


function doJob() {
  # Check if the script has been called from the outside.
  if [ $PROCESSING_FILES == true ] ; then
    for i in $(seq 1 ${#FILES[@]})
    do
      echo ${FILES[$i-1]}
      processFile "${FILES[$i-1]}"
    done


  else
    # processing every file
for i in $(seq 1 ${#FILES[@]})
do
CURRFILE=${FILES[$i-1]}
      # checking if file or directory exist
      if [ ! -e "$CURRFILE" ] ; then echo "File not found: $CURRFILE. Skipping..." ; continue ; fi
     
      # if a paremeter is a directory, process it recursively if RECURSIVE is set
      if [ -d "$CURRFILE" ] ; then
        if [ $RECURSIVE == true ] ; then
          if [ $USE_EXT == true ] ; then
            find "$CURRFILE" $XDEV -type f -name "*.$FILE_EXT" -exec "$0" $DELETE_FLAG $PROCESSALLFILE_FLAG -f "{}" \;
          else
            find "$CURRFILE" $XDEV -type f -exec "$0" $DELETE_FLAG $PROCESSALLFILE_FLAG -f "{}" \;
          fi
        else
          echo "$CURRFILE is a directory. Skipping..."
        fi
      else
        processFile "$CURRFILE"
      fi
    done
  fi
}


parseArgs "$@"
doJob


Examples
Assuming the script is in your $PATH and it's called bom-remove, you can "clean" a bunch of files invoking it this way:

$ bom-remove file-to-clean ...

If you want to clean the files in an entire directory, you can use the following syntax:

$ bom-remove -r dir-to-clean ...

If your sed installation is not in your $PATH or you have to use an alternate version, you can invoke the script with the following syntax:

$ bom-remove -s path/to/sed file-to-clean ...

If you want to clean a directory in which other file systems might be mounted, you can use the -x option so that the script does not descend them:

$ bom-remove -xr dir-to-clean ...


Seriously!!! This literally saved my day!!!