Profiling Python Applications

I’ve been working on a project that’s written in Python, it continuously communicates with some external industrial equipment. It will poll the status of this equipment 4 times per second and also sends commands to them when requested to. My job this week was to raise the update rate to 5Hz…. I needed to make sure I had enough time to do this!

I decided before doing anything I should profile the code to find out how much time the main loop needs to run and what methods take the longest time. That way I’d know if the code can support 5Hz and if not what I can do about it.

Once again the Python standard library comes to the rescue, the cProfile module will monitor the execution of your program and generate a report. Below is an example of how to use it.

import cProfile
cProfile.run("main()")

The next thing I did is write a simple bit of code that will print to stdout the current update rate of my application every second. It’s pretty much a Python port of the JavaScript library stats.js.

from __future__ import division
import time

class stats(object):
    def __init__(self):
        self.msMin = 1000
        self.msMax = 0
        self.msTime = 0
        self.fpsMin = 1000
        self.fpsMax = 0
        self.fps = 0;
        self.updates = 0
        self.startTime = int((time.time()+0.5)*1000)
        self.prevTime = int((time.time()+0.5)*1000)

    def begin(self):
        """Calling the method signifies the start of a frame
        """
        self.startTime = int((time.time()+0.5)*1000)

    def end(self):
        """Calling this method signifies the end of a frame
        """
        now = int((time.time()+0.5)*1000)

        self.msTime = now-self.startTime
        self.msMax = max(self.msMax, self.msTime)
        self.msMin = min(self.msMin, self.msTime)

        #print "ms: %i (%i - %i)" % (self.msTime, self.msMin, self.msMax)

        self.updates = self.updates + 1

        if now > (self.prevTime + 1000.0):
            self.fps = round((self.updates*1000.0)/float(now-self.prevTime))
            self.fpsMax = max(self.fpsMax, self.fps)
            self.fpsMin = min(self.fpsMin, self.fps)

            print "stats: %i fps (%.i fps - %i fps)" % (self.fps, self.fpsMin, self.fpsMax)

            self.prevTime = now
            self.updates = 0

        return now

        def update(self):
            self.startTime = self.end()

Simply add a call to begin() to the start of your loop and a corresponding call to end()  at the end of your loop. Now I get a nice counter that tells me how the code is performing. As I work on the code I can see how this affects the main loop performance.

       

Finding a File Descriptor Leak

My current project is a control system that runs on a small embedded PC running a Linux OS. I had a problem during development where I had to open/close two serial ports alternatively because they shared an interrupt. Doing this suddenly caused the software to crash after running for a few minutes. The problem turned out to be that I was leaking file descriptors!

I wrote a simple shell script that prints the the total file descriptors open on the serial ports. This helped me make sure my bug fix worked correctly.

#!/bin/sh
 
TIME=1
 
while [ 1 -eq 1 ]; do
    clear
    lsof | awk '''
BEGIN { ttyS0 = ttyS1 = ttyS3 = ttyS4 = 0; }
/\/dev\/ttyS4/ { ttyS4++; }
/\/dev\/ttyS3/ { ttyS3++; }
/\/dev\/ttyS0/ { ttyS0++; }
/\/dev\/ttyS1/ { ttyS1++; }
END {
    print " Descriptor    Counter ";
    print " ttyS4       " ttyS4;
    print " ttyS3       " ttyS3;
    print " ttyS1       " ttyS1;
    print " ttyS0       " ttyS0;
}
'''
    sleep $TIME
done

In the end I manually control the serial port using the sys module instead of using pySerial. I still have no idea why pyserial started leaking resources.

       

Plotting a Text File with Matplotlib and IPython

Earlier I was converting a Scilab simulation into C and I had the code emit a load of text files containing the data because I didn’t want to do any plotting or UI stuff in C. But I still wanted to plot the data so I can quickly check everything’s working. I also wanted to do some post processing of the data too… well that’s Python really shines in my opinion. I fired up IPython and used Numpy and Matplotlib.

 import numpy as np
 import matplotlib.pyplot as pyplot
 pyplot.plot( np.loadtxt("data/somedatafile.dat") )
 pyplot.show()

Done! In four lines of code I have my plot. I love IPython, numpy and matplotlib, they allow you to get things done really fast. I also love that IPython even auto-completes file paths, so very handy :-).

         

Formatting Numbers using SI Prefixes

Well it’s time for another post. A couple of weeks ago I was working on an interface program for a rig at work. We were adding some extra features to a controller board and I thought while I was working on that I might as well make some changes to the interface program as well.  The feature I wanted to add to the program was to make it format some of the numbers that are displayed in the UI using SI prefixes and with units of measure.

So after a quick look on the web to see if I can find some example code on how to do this, I decided to have a go writing the code on my own without looking at other stuff too much. This is what I came up with, It’s a function written in C# that can be used to convert floating point numbers to formatted strings. So the number 0.102 will be printed as “102m”, and it can append units to the end like this – “102mA”.

public static string FormatStringEng(double input, string units, string format)
{
    string prefix = "";
    double value = 0.0;

    if (input >= 1e+12 && input < 1e+15) {
        prefix = "T"; value = (input / 1e+12);    // Tera (1e+12)
    } else if (input >= 1e+9 && input < 1e+12) {
        prefix = "G"; value = (input / 1e+9);     // Giga (1e+9
    } else if (input >= 1e+6 && input < 1e+9) {
        prefix = "M"; value = (input / 1e+6);     // Mega (1e+6)
    } else if (input >= 1e+3 && input < 1e+6) {
        prefix = "k"; value = (input / 1e+3);     // Kilo (1e+3)
    } else if ((input >= 1) && (input < 1e+6)) {
        prefix = ""; value = input;               // Unity
    } else if (input >= 1e-3 && input < 1) {
        prefix = "m"; value = (input * 1e+3);     // Milli (1e-3)
    } else if (input >= 1e-6 && input < 1e-3) {
        prefix = "ÎĽ"; value = (input * 1e+6);     // Micro (1e-6)
    } else if (input >= 1e-9 && input < 1e-6) {
        prefix = "n"; value = (input * 1e+9);     // Nano (1e-9)
    } else if (input >= 1e-12 && input < 1e-9) {
        prefix = "p"; value = (input * 1e+12);    // Pico (1e-12)
    } else if (input >= 1e-15 && input < 1e-12) {
        prefix = "f"; value = (input * 1e+15);    // Fempto (1e-15)
    }

    return value.ToString() + prefix + units;
}

This code is working quite well so far. Haven’t had any problems with it… yet anyway. That’s it for now, see ya!

     
1 2