Profiling Python Applications

I’ve been working on a project that’s written in Python, it continuously communicates with some external industrial equipment. It will poll the status of this equipment 4 times per second and also sends commands to them when requested to. My job this week was to raise the update rate to 5Hz…. I needed to make sure I had enough time to do this!

I decided before doing anything I should profile the code to find out how much time the main loop needs to run and what methods take the longest time. That way I’d know if the code can support 5Hz and if not what I can do about it.

Once again the Python standard library comes to the rescue, the cProfile module will monitor the execution of your program and generate a report. Below is an example of how to use it.

import cProfile"main()")

The next thing I did is write a simple bit of code that will print to stdout the current update rate of my application every second. It’s pretty much a Python port of the JavaScript library stats.js.

from __future__ import division
import time

class stats(object):
    def __init__(self):
        self.msMin = 1000
        self.msMax = 0
        self.msTime = 0
        self.fpsMin = 1000
        self.fpsMax = 0
        self.fps = 0;
        self.updates = 0
        self.startTime = int((time.time()+0.5)*1000)
        self.prevTime = int((time.time()+0.5)*1000)

    def begin(self):
        """Calling the method signifies the start of a frame
        self.startTime = int((time.time()+0.5)*1000)

    def end(self):
        """Calling this method signifies the end of a frame
        now = int((time.time()+0.5)*1000)

        self.msTime = now-self.startTime
        self.msMax = max(self.msMax, self.msTime)
        self.msMin = min(self.msMin, self.msTime)

        #print "ms: %i (%i - %i)" % (self.msTime, self.msMin, self.msMax)

        self.updates = self.updates + 1

        if now > (self.prevTime + 1000.0):
            self.fps = round((self.updates*1000.0)/float(now-self.prevTime))
            self.fpsMax = max(self.fpsMax, self.fps)
            self.fpsMin = min(self.fpsMin, self.fps)

            print "stats: %i fps (%.i fps - %i fps)" % (self.fps, self.fpsMin, self.fpsMax)

            self.prevTime = now
            self.updates = 0

        return now

        def update(self):
            self.startTime = self.end()

Simply add a call to begin() to the start of your loop and a corresponding call to end()  at the end of your loop. Now I get a nice counter that tells me how the code is performing. As I work on the code I can see how this affects the main loop performance.


Finding a File Descriptor Leak

My current project is a control system that runs on a small embedded PC running a Linux OS. I had a problem during development where I had to open/close two serial ports alternatively because they shared an interrupt. Doing this suddenly caused the software to crash after running for a few minutes. The problem turned out to be that I was leaking file descriptors!

I wrote a simple shell script that prints the the total file descriptors open on the serial ports. This helped me make sure my bug fix worked correctly.

while [ 1 -eq 1 ]; do
    lsof | awk '''
BEGIN { ttyS0 = ttyS1 = ttyS3 = ttyS4 = 0; }
/\/dev\/ttyS4/ { ttyS4++; }
/\/dev\/ttyS3/ { ttyS3++; }
/\/dev\/ttyS0/ { ttyS0++; }
/\/dev\/ttyS1/ { ttyS1++; }
    print " Descriptor    Counter ";
    print " ttyS4       " ttyS4;
    print " ttyS3       " ttyS3;
    print " ttyS1       " ttyS1;
    print " ttyS0       " ttyS0;
    sleep $TIME

In the end I manually control the serial port using the sys module instead of using pySerial. I still have no idea why pyserial started leaking resources.


Using CERN’s Root Library for Plotting

This is a post about something I did at least a year ago. I was modifying the firmware of a board as a customer wanted to change the way calibration of the product worked. The old method used a table containing points in a curve, each point contained the desired set point, and the actual value you get. The code then interpolated between the points in the curve as needed to get the actual value you need to set in order to get the desired value.

The customer wanted to use a single polynomial expression for calibration instead as that is what their existing products used. The new system would only need the coefficients of the polynomial instead of the large table of numbers. I wanted to experiment a bit with the polynomial code I had just written on the PC and at the time I did not know Scilab or numpy/matplotlib, but I had just read about CERN’s Root library which is a C++ library that includes really good plotting libraries. So I thought I’d try it out.

As you can see the plots look really good. Although I haven’t really used Root since I still remember it can sometimes consider it for doing plots even though I typically choose Gnuplot or matplotlib. On big advantage is that because it is a C++ library I can use the C code I wrote for the firmware directly without modification.


Setting Up A Mercurial Webserver

The version control system I use at work and home is Mercurial and I have been trying to learn more about it recently. What I want to do at the moment is have all of my repos stored in a central location so I can make sure they are backed up. I’d also like to be able to get the repos from any computer at work.

As it turns out Mercurial has a built-in webserver and this can handle multiple repositories. So created a directory on my computer where I put all my projects and then I’ve setup a script that will start-up the Mercurial webserver.

To set all this up I first created a directory, “/var/hg”, and in this directory is a sub directory called “repos” and a configuration file called “hgweb.config”. An example of the contents of this file are below.

repos/ = repos/
description=some description here
allow_archive = bz2 gz zip

After doing this I then put all of my repositories in the repos directory, each repository can have it’s own local web configuration. This is a quick shell snippet that will launch a Mercurial webserver.

hg serve –webdir-conf /var/hg/hgweb.config

Thats it! Well… mostly anyway. I have missed of some detail as I can’t remember exactly what I did. Perhaps I should start recording the things I do step by step here so I cant forget.


Paying with Processing (Part Two)

Last time I ran an example sketch and then wrote my own that prints “Hello World” in bright colours. Now I want to play around with some of the basic shapes that Processing provides. From the documentation it looks like the 2D primitives are the arc, ellipse, line, point, quad, rect, and triangle.

The first one I’m going to play with is the rect() primitive, so I wrote a simple sketch to draw one. All it does is set the window size then draw a rectangle somewhere in the window.

void setup() {
 size(640, 480);
 rect(30, 30, 155, 155);

The rect() function takes four parameters as inputs, the first two are the x and y co-ordinates of where to start drawing. The second two are the height and width of the rectangle. Then I thought it would be cool if I made it follow the mouse, this as it turns out is really easy. In the draw function you get a couple of variables you can use for this, mouseX and mouseY. They hold the x and y co-ordinates of where the mouse is so all I have to do is put the call to rect() into the draw loop then use the mouse co-ordinates to set that starting position of the rectangle.

rect(mouseX, mouseY, 155, 155);

This code worked ok but the viewport filled up with loads of rectangles as you moved the mouse really quickly. I wanted just a single rectangle that followed the mouse. So I added some extra lines of code that drew another rectangle over the whole viewport to clear it then draw the smaller rectangle. I also thought it would look much nicer if the mouse cursor was always located in the centre of the rectangle rather than at the top left-hand corner. So to do this I subtracted half the width of the rectangle from the x co-ordinate or the mouse cursor, and did the same with the height and y co-ordinate. This kept the mouse cursor roughly in the middle of the rectangle.

To draw the rectangle over the entire viewport I two other special variables, width and height. These are set to the width and height of the window you are using. The code for all of this is below.

void setup() {
  size(640, 480);

void draw() {
  rect(0, 0, width, height);
  fill(0, 0, 255);
  rect(mouseX-(155/2), mouseY-(155/2), 155, 155);

Well that’s all for now. See ya!


Plotting a Text File with Matplotlib and IPython

Earlier I was converting a Scilab simulation into C and I had the code emit a load of text files containing the data because I didn’t want to do any plotting or UI stuff in C. But I still wanted to plot the data so I can quickly check everything’s working. I also wanted to do some post processing of the data too… well that’s Python really shines in my opinion. I fired up IPython and used Numpy and Matplotlib.

 import numpy as np
 import matplotlib.pyplot as pyplot
 pyplot.plot( np.loadtxt("data/somedatafile.dat") )

Done! In four lines of code I have my plot. I love IPython, numpy and matplotlib, they allow you to get things done really fast. I also love that IPython even auto-completes file paths, so very handy :-).


Playing with Processing (Part One)

After a while playing with the Arduino I’ve become interested in Processing which is what the Arduino language and Arduino IDE is based on. Just as the Arduino is a thin layer on top of C, Processing is simply a library for Java that makes playing with computer graphics much easier. So today I’ve decided to play around with it a bit and write about it.

First of all I downloaded the Processing IDE from the Processing website, I downloaded the Linux version as I’m using Ubuntu but you can get a Windows or Mac OSX version as well. Then I decompressed the files and fired up the IDE by executing the “processing” file, I was then greeted by the IDE.

The Processing IDE

Looks good eh?? It looks almost idendical to the Arduino IDE. I then has a quick read through some of the documentation on the website. The first code snippet I ran was from Processing own tutorial, although I left a small bit of code out. The code for it is below, what it does is draw a circle wherever the mouse cursor currently is.

void setup() {
    size(560, 160);

void draw() {
    ellipse(mouseX, mouseY, 80, 80);

Yay, it worked!! I love how exciting it is to get instant feedback. I wasn’t sure exactly what the smooth() function did, 


I suspected it may have been something to do with anti-aliasing and it turns out I was right after looking at the reference. As you can see from the screen shot it can be used to make pretty snaking patterns.

I decided that I wanted to create a sketch that printed “Hello World” because that’s the first program most people write :-). I had a quick look around the documentation on the website and I found the text() function and some example code. I typed that into the IDE but I wanted to use a bigger font, it turns out the font file must be in the /data directory of the sketch.

There is a tool provided that will take a font on your computer and turn it into something Processing can use. To use this tool, go to the “Tools” menu and select “Create Font…” and then in the dialog box that pops up select the font you want and the filename then click OK.

void setup() {
  size(640, 160);
  background(254, 153, 0);

void draw() {
   PFont font = loadFont("Verdana-Bold-48.vlw");
   fill(0, 105, 253);
   text("Hello World", 15, 88);

Clicking the run button produces a large window with “Hello World” written in large letters. It worked! Yay!!! So far it’s been pretty fun playing around with Processing, I hope to do a lot more with it. I want to use it to talk to the Arduino. Perhaps plotting the voltage on the analogue pins over time.

That’s it for now, I hope to have another post on this later in the week. See ya!


Formatting Numbers using SI Prefixes

Well it’s time for another post. A couple of weeks ago I was working on an interface program for a rig at work. We were adding some extra features to a controller board and I thought while I was working on that I might as well make some changes to the interface program as well.  The feature I wanted to add to the program was to make it format some of the numbers that are displayed in the UI using SI prefixes and with units of measure.

So after a quick look on the web to see if I can find some example code on how to do this, I decided to have a go writing the code on my own without looking at other stuff too much. This is what I came up with, It’s a function written in C# that can be used to convert floating point numbers to formatted strings. So the number 0.102 will be printed as “102m”, and it can append units to the end like this – “102mA”.

public static string FormatStringEng(double input, string units, string format)
    string prefix = "";
    double value = 0.0;

    if (input >= 1e+12 && input < 1e+15) {
        prefix = "T"; value = (input / 1e+12);    // Tera (1e+12)
    } else if (input >= 1e+9 && input < 1e+12) {
        prefix = "G"; value = (input / 1e+9);     // Giga (1e+9
    } else if (input >= 1e+6 && input < 1e+9) {
        prefix = "M"; value = (input / 1e+6);     // Mega (1e+6)
    } else if (input >= 1e+3 && input < 1e+6) {
        prefix = "k"; value = (input / 1e+3);     // Kilo (1e+3)
    } else if ((input >= 1) && (input < 1e+6)) {
        prefix = ""; value = input;               // Unity
    } else if (input >= 1e-3 && input < 1) {
        prefix = "m"; value = (input * 1e+3);     // Milli (1e-3)
    } else if (input >= 1e-6 && input < 1e-3) {
        prefix = "μ"; value = (input * 1e+6);     // Micro (1e-6)
    } else if (input >= 1e-9 && input < 1e-6) {
        prefix = "n"; value = (input * 1e+9);     // Nano (1e-9)
    } else if (input >= 1e-12 && input < 1e-9) {
        prefix = "p"; value = (input * 1e+12);    // Pico (1e-12)
    } else if (input >= 1e-15 && input < 1e-12) {
        prefix = "f"; value = (input * 1e+15);    // Fempto (1e-15)

    return value.ToString() + prefix + units;

This code is working quite well so far. Haven’t had any problems with it… yet anyway. That’s it for now, see ya!


Playing with the Arduino UNO

Recently I ordered an Arduino UNO and two shields from Adafruit and two weeks ago it arrived… YAY! As soon as I signed for it I rushed to unpacked it! I was amazed at how small it is, it looked bigger in photos. So far over this couple of weeks I’ve only used a few times but those few times I have used I have been very impressed with it.

The first thing I did of course is get it flashing an LED, which only took about 10 minutes to get going. This was including the time to install the IDE on my laptop running Ubuntu 11.04. After a while I had an issue with the Ardunio IDE hanging whenever I click on the Tools menu and also an issue with the IDE reporting the serial port was locked by another application. I haven’t yet found a fix for this other than using my Fiancée’s laptop instead, which is running Windows Vista… I love cross platform software, it can be so handy sometimes.

After that I built the two shields, the first was a Protoshield with screw terminals, and the second was the data logging shield. These were both pretty easy to build and the instructions where excellent. The next thing I did was get the arduino sampling with the ADC, I did this by simply hooking a 10k pot to A0. Then I soldered an LM35 onto the data logging shield and hooked 3V3 to the VREF pin as recommended by Adafruit. At this point I was using the Serial port to get the samples out.

The next day I decided to follow the tutorials on how to use the SD card interface. Thanks to the excellent work by the authors of the various libraries this was very easy. The same for using the RTC. It all just came together and I had the Arduino sampling temperature with a timestamp to the SD card.

In conclusion I think the Arduino is fantastic and I hope to use it much more my future projects. I especially like the idea of the Arduino Pro and Nano because they can be embedded in a system somewhere.

1 3 4 5