Open Source

This is more a quick dump of some proof-of-concept code.  We’re in the process of writing communications drivers for an energy management system, many of which need to communicate with devices like Modbus energy meters.

Traditionally I’ve just used the excellent pymodbus library with its synchronous interface for batch-processing scripts, but this time I need real-time and I need to do things asynchronously.  I can either run the synchronous client in a thread, or, use the Twisted interface.

We’re actually using Tornado for our core library, and thankfully there’s an adaptor module to allow you to use Twisted applications.  But how do you do it?  Twisted code requires quite a bit of getting used to, and I’ve still not got my head around it.  I haven’t got my head fully around Tornado either.

So how does one combine these?

The following code pulls out the first couple of registers out of a CET PMC330A energy meter that’s monitoring a few circuits in our office. It is a stripped down copy of this script.

#!/usr/bin/env python
'''
Pymodbus Asynchronous Client Examples -- using Tornado
--------------------------------------------------------------------------

The following is an example of how to use the asynchronous modbus
client implementation from pymodbus.
'''
#---------------------------------------------------------------------------# 
# import needed libraries
#---------------------------------------------------------------------------# 
import tornado
import tornado.platform.twisted
tornado.platform.twisted.install()
from twisted.internet import reactor, protocol
from pymodbus.constants import Defaults

#---------------------------------------------------------------------------# 
# choose the requested modbus protocol
#---------------------------------------------------------------------------# 
from pymodbus.client.async import ModbusClientProtocol
#from pymodbus.client.async import ModbusUdpClientProtocol

#---------------------------------------------------------------------------# 
# configure the client logging
#---------------------------------------------------------------------------# 
import logging
logging.basicConfig()
log = logging.getLogger()
log.setLevel(logging.DEBUG)

#---------------------------------------------------------------------------# 
# example requests
#---------------------------------------------------------------------------# 
# simply call the methods that you would like to use. An example session
# is displayed below along with some assert checks. Note that unlike the
# synchronous version of the client, the asynchronous version returns
# deferreds which can be thought of as a handle to the callback to send
# the result of the operation.  We are handling the result using the
# deferred assert helper(dassert).
#---------------------------------------------------------------------------# 
def beginAsynchronousTest(client):
    io_loop = tornado.ioloop.IOLoop.current()

    def _dump(result):
        logging.info('Register values: %s', result.registers)
    def _err(result):
        logging.error('Error: %s', result)

    rq = client.read_holding_registers(0, 4, unit=1)
    rq.addCallback(_dump)
    rq.addErrback(_err)

    #-----------------------------------------------------------------------# 
    # close the client at some time later
    #-----------------------------------------------------------------------# 
    io_loop.add_timeout(io_loop.time() + 1, client.transport.loseConnection)
    io_loop.add_timeout(io_loop.time() + 2, io_loop.stop)

#---------------------------------------------------------------------------# 
# choose the client you want
#---------------------------------------------------------------------------# 
# make sure to start an implementation to hit against. For this
# you can use an existing device, the reference implementation in the tools
# directory, or start a pymodbus server.
#---------------------------------------------------------------------------# 
defer = protocol.ClientCreator(reactor, ModbusClientProtocol
        ).connectTCP("10.20.30.40", Defaults.Port)
defer.addCallback(beginAsynchronousTest)
tornado.ioloop.IOLoop.current().start()

Hi all,

This is more a note to self for future reference.  Qt has a nice handy reference counting memory management system by means of QSharedPointer and QWeakPointer.  The system is apparently thread-safe and seems to be totally transparent.

One gotcha though, is two QSharedPointer objects cannot share the same pointer unless one is cloned from the other (either directly or via QWeakPointer).  The other is that you must leave deletion of the object to QSharedPointer.  You’ve given it your precious pointer, it has adopted it and while you may call the object, it is no longer yours, so don’t go deleting it.

So you create an object, you want to pass a reference to yourself to some other object.  How?  Like this?

QSharedPointer<MyClass> MyClass::ref() {
    return QSharedPointer<MyClass>(this); /* NO! */
}

No, not like that! That will create QSharedPointer instances left right and centre. Not what you want to do at all. What you need to do, is create the initial reference, but then store a weak reference to it. Then all future calls, you simply call the toStrongRef method of the weak reference to get a QSharedPointer that’s linked to the first one you handed out.

Then, having done this, when you create your new object, you create it with the new keyword as normal, take a QSharedPointer reference to it, then forget all about the original pointer! You can get it back by calling the data method of the pointer object.

To make it simple, here’s a base class you can inherit to do this for you.

    #include <QWeakPointer>
    #include <QSharedPointer>

    /*!
     * Self-Reference helper.  This allows for objects to maintain
     * references to "this" via the QSharedPointer reference-counting
     * smart pointers.
     */
    template<typename T>
    class SelfRef {
        public:
            /*!
             * Get a strong reference to this object.
             */
            QSharedPointer<T>    ref()
            {
                QSharedPointer<T> this_ref(this->this_weak);
                if (this_ref.isNull()) {
                    this_ref = QSharedPointer<T>((T*)this);
                    this->this_weak = this_ref.toWeakRef();
                }
                return this_ref;
            }

            /*!
             * Get a weak reference to this object.
             */
            QWeakPointer<T>        weakRef() const
            {
                return this->this_weak;
            }
        private:
            /*! A weak reference to this object */
            QWeakPointer<T>        this_weak;
    };

Example usage:

#include <iostream>
#include <stdexcept>
#include "SelfRef.h"

class Test : public SelfRef<Test> {
        public:
                Test()
                {
                        std::cout << __func__ << std::endl;
                        this->freed = false;
                }
                ~Test()
                {
                        std::cout << __func__ << std::endl;
                        this->freed = true;
                }

                void test() {
                        if (this->freed)
                                throw std::runtime_error("Already freed!");
                        std::cout
                                << "Test object is at "
                                << (void*)this
                                << std::endl;
                }

                bool                    freed;
                QSharedPointer<Test>    another;
};

int main(void) {
        Test* a = new Test();
        if (a != NULL) {
                QSharedPointer<Test> ref1 = a->ref();
                if (!ref1.isNull()) {
                        QSharedPointer<Test> ref2 = a->ref();
                        ref2->test();
                }
                ref1->test();
        }
        a->test();
        return 0;
}

Note that the line before the return is a deliberate use after free bug to prove the pointer really was freed.  Also note that the idea of setting a boolean flag to indicate the constructor has been called only works here because nothing happens between that use after free attempt and the destructor being called.  Don’t rely on this to see if your object is being called after destruction.  This is what the output session from gdb looks like:

RC=0 stuartl@rikishi /tmp/qtsp $ make CXXFLAGS=-g
g++ -c -g -I/usr/share/qt4/mkspecs/linux-g++ -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtGui -I/usr/include/qt4 -I. -I. -o test.o test.cpp
g++ -Wl,-O1 -o qtsp test.o    -L/usr/lib64/qt4 -lQtGui -L/usr/lib64 -L/usr/lib64/qt4 -L/usr/X11R6/lib -lQtCore -lgthread-2.0 -lglib-2.0 -lpthread 
RC=0 stuartl@rikishi /tmp/qtsp $ gdb ./qtsp 
GNU gdb (Gentoo 7.7.1 p1) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./qtsp...done.
(gdb) r
Starting program: /tmp/qtsp/qtsp 
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Test
Test object is at 0x555555759c90
Test object is at 0x555555759c90
~Test
terminate called after throwing an instance of 'std::runtime_error'
  what():  Already freed!

Program received signal SIGABRT, Aborted.
0x00007ffff5820775 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff5820775 in raise () from /lib64/libc.so.6
#1  0x00007ffff5821bf8 in abort () from /lib64/libc.so.6
#2  0x00007ffff610cd75 in __gnu_cxx::__verbose_terminate_handler() ()
   from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6
#3  0x00007ffff6109ec8 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6
#4  0x00007ffff6109f15 in std::terminate() () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6
#5  0x00007ffff610a2e9 in __cxa_throw () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6
#6  0x0000555555555cea in Test::test (this=0x555555759c90) at test.cpp:20
#7  0x0000555555555315 in main () at test.cpp:41
(gdb) up
#1  0x00007ffff5821bf8 in abort () from /lib64/libc.so.6
(gdb) up
#2  0x00007ffff610cd75 in __gnu_cxx::__verbose_terminate_handler() ()
   from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6
(gdb) up
#3  0x00007ffff6109ec8 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6
(gdb) up
#4  0x00007ffff6109f15 in std::terminate() () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6
(gdb) up
#5  0x00007ffff610a2e9 in __cxa_throw () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6
(gdb) up
#6  0x0000555555555cea in Test::test (this=0x555555759c90) at test.cpp:20
20                                      throw std::runtime_error("Already freed!");
(gdb) up
#7  0x0000555555555315 in main () at test.cpp:41
41              a->test();
(gdb) quit
A debugging session is active.

        Inferior 1 [process 17906] will be killed.

Quit anyway? (y or n) y

You’ll notice it fails right on that second last line because the last QSharedPointer went out of scope before this.  This is why you forget all about the pointer once you create the first QSharedPointer.

To remove the temptation to use the pointer directly, you can make all your constructors protected (or private) and use a factory that returns a QSharedPointer to your new object.

A useful macro for doing this:

/*!
 * Create an instance of ClassName with the given arguments
 * and immediately return a reference to it.
 *
 * @returns	QSharedPointer<ClassName> object
 */
#define newRef(ClassName, args ...)	\
	((new ClassName(args))->ref().dynamicCast<ClassName>())

Just recently I’ve been looking into asynchronous programming.

Previously I had an aversion to asynchronous code due to the ugly twisted web of callback functions that it can turn into. However, after finding that having a large number of threads blocking on locks and semaphores still manages to thrash a machine, I’ve come to the conclusion that I should put aside my feelings and try it anyway.

Our codebase is written in Python 2.7, sadly, not new enough to have asyncio. However we do plan to eventually move to Python 3.x when things are a bit more stable in the Debian/Ubuntu department (Ubuntu 12.04 didn’t support it and there are a few sites that still run it, one or two still run 10.04).

That said, there’s thankfully a port of what became asyncio in the form of Trollius.

Reading through the examples though still had me lost and the documentation is not exactly extensive. In particular, coroutines and yielding. The yield operator is not new, it’s been in Python for some time, but until now I never really understood it or how it was useful in co-operative programming.

Thankfully, Sahand Saba has written a guide on how this all works:
http://sahandsaba.com/understanding-asyncio-node-js-python-3-4.html

I might put some more notes up as I learn more, but that guide explained a lot of the fundamentals behind a lot of event loop frameworks including asyncio.