Writing tests

Tests are very useful for writing good code. We have a number of tests checking the correctness of the pymeasure implementation. Those tests (located in the tests directory) are run automatically on our CI server, but you can also run them locally using pytest.

When adding instruments, your primary concern will be tests for the instrument driver you implement. We distinguish two groups of tests for instruments: the first group does not rely on a connected instrument. These tests verify that the implemented instrument driver exchanges the correct messages with a device (for example according to a device manual). We call those “protocol tests”. The second group tests the code with a device connected.

Implement device tests by adding files in the tests/instruments/... directory tree, mirroring the structure of the instrument implementations. There are other instrument tests already available that can serve as inspiration.

Protocol tests

In order to verify the expected working of the device code, it is good to test every part of the written code. The expected_protocol() context manager (using a ProtocolAdapter internally) simulates the communication with a device and verifies that the sent/received messages triggered by the code inside the with statement match the expectation.

import pytest

from pymeasure.test import expected_protocol

from pymeasure.instruments.extreme5000 import Extreme5000

def test_voltage():
    """Verify the communication of the voltage getter."""
    with expected_protocol(
        Extreme5000,
        [(":VOLT 0.345", None),
         (":VOLT?", "0.3000")],
    ) as inst:
        inst.voltage = 0.345
        assert inst.voltage == 0.3

In the above example, the imports import the pytest package, the expected_protocol and the instrument class to be tested.

The first parameter, Extreme5000, is the class to be tested.

When setting the voltage, the driver sends a message (":VOLT 0.345"), but does not expect a response (None). Getting the voltage sends a query (":VOLT?") and expects a string response ("0.3000"). Therefore, we expect two pairs of send/receive exchange. The list of those pairs is the second argument, the expected message protocol.

The context manager returns an instance of the class (inst), which is then used to trigger the behaviour corresponding to the message protocol (e.g. by using its properties).

If the communication of the driver does not correspond to the expected messages, an Exception is raised.

Note

The expected messages are without the termination characters, as they depend on the connection type and are handled by the normal adapter (e.g. VISAAdapter).

Some protocol tests in the test suite can serve as examples:

Testing a simple instrument: tests/instruments/keithley/test_keithley2000.py
Testing a multi-channel instrument: tests/instruments/tektronix/test_afg3152.py
Testing instruments using frame-based communication: tests/instruments/hcp/tc038.py

Test generator

In order to facilitate writing tests, if you already have working code and a device at hand, we have a Generator for tests. You can control your instrument with the TestGenerator as a middle man. It logs the method calls, the device communication and the return values, if any, and writes tests according to these log entries.

from pymeasure.generator import Generator
from pymeasure.instruments.hcp import TC038

generator = Generator()
inst = generator.instantiate(TC038, adapter, 'hcp', adapter_kwargs={'baud_rate': 9600})

As a first step, this code imports the Generator and generates a middle man instrument. The instantiate() method creates an instrument instance and logs the communication at startup. The Generator creates a special adapter for the communication with the device. It cannot inspect the instrument’s __init__(), however. Therefore you have to specify the all connection settings via the adapter_kwargs dictionary, even those, which are defined in __init__(). These adapter arguments are not written to tests. If you have arguments for the instrument itself, e.g. a RS485 address, you may give it as a keyword argument. These additional keyword arguments are included in the tests.

Now we can use inst as if it were created the normal way, i.e. inst = TC038(adapter), where adapter is some resource string. Having gotten and set some properties, and called some methods, we can write the tests to a file.

inst.information  # returns the 'information' property, e.g. 'UT150333 V01.R001111222233334444'
inst.setpoint = 20
assert inst.setpoint == 20
inst.setpoint = 60

generator.write_file(file)

The following data will be written to file:

import pytest

from pymeasure.test import expected_protocol
from pymeasure.instruments.hcp import TC038


def test_init():
    with expected_protocol(
            TC038,
            [(b'\x0201010WRS01D0002\x03', b'\x020101OK\x03')],
    ):
        pass  # Verify the expected communication.


def test_information_getter():
    with expected_protocol(
            TC038,
            [(b'\x0201010WRS01D0002\x03', b'\x020101OK\x03'),
             (b'\x0201010INF6\x03', b'\x020101OKUT150333 V01.R001111222233334444\x03')],
    ) as inst:
        assert inst.information == 'UT150333 V01.R001111222233334444'


@pytest.mark.parametrize("comm_pairs, value", (
    ([(b'\x0201010WRS01D0002\x03', b'\x020101OK\x03'),
      (b'\x0201010WWRD0120,01,00C8\x03', b'\x020101OK\x03')],
     20),
    ([(b'\x0201010WRS01D0002\x03', b'\x020101OK\x03'),
      (b'\x0201010WWRD0120,01,0258\x03', b'\x020101OK\x03')],
     60),
))
def test_setpoint_setter(comm_pairs, value):
    with expected_protocol(
            TC038,
            comm_pairs,
    ) as inst:
        inst.setpoint = value


def test_setpoint_getter():
    with expected_protocol(
            TC038,
            [(b'\x0201010WRS01D0002\x03', b'\x020101OK\x03'),
             (b'\x0201010WRDD0120,01\x03', b'\x020101OK00C8\x03')],
    ) as inst:
        assert inst.setpoint == 20.0

Device tests

It can be useful as well to test the code against an actual device. The necessary device setup instructions (for example: connect a probe to the test output) should be written in the header of the test file or test methods. There should be the connection configuration (for example serial port), too. In order to distinguish the test module from protocol tests, the filename should be test_instrumentName_with_device.py, if the device is called instrumentName.

To make it easier for others to run these tests using their own instruments, we recommend to use pytest.fixture to create an instance of the instrument class. It is important to use the specific argument name connected_device_address and define the scope of the fixture to only establish a single connection to the device. This ensures two things: First, it makes it possible to specify the address of the device to be used for the test using the --device-address command line argument. Second, tests using this fixture, i.e. tests that rely on a device to be connected to the computer are skipped by default when running pytest. This is done to avoid that tests that require a device are run when none is connected. It is important that all tests that require a connection to a device either use the connected_device_address fixture or a fixture derived from it as an argument.

A simple example of a fixture that returns a connected instrument instance looks like this:

@pytest.fixture(scope="module")
def extreme5000(connected_device_address):
    instr = Extreme5000(connected_device_address)
    # ensure the device is in a defined state, e.g. by resetting it.
    return instr

Note that this fixture uses connected_device_address as an input argument and will thus be skipped by automatic test runs. This fixture can then be used in a test functions like this:

def test_voltage(extreme5000):
    extreme5000.voltage = 0.345
    assert extreme5000.voltage == 0.3

Again, by specifying the fixture’s name, in this case extreme5000, invoking pytest will skip these tests by default.

It is also possible to define derived fixtures, for example to put the device into a specific state. Such a fixture would look like this:

@pytest.fixture
def auto_scaled_extreme5000(extreme5000):
    extreme5000.auto_scale()
    return extreme5000

In this case, do not specify the fixture’s scope, so it is called again for every test function using it.

To run the test, specify the address of the device to be used via the --device-address command line argument and limit pytest to the relevant tests. You can filter tests with the -k option or you can specify the filename. For example, if your tests are in a file called test_extreme5000_with_device.py, invoke pytest with pytest -k extreme5000 --device-address TCPIP::192.168.0.123::INSTR".

There might also be tests where manual intervention is necessary. In this case, skip the test by prepending the test function with a @pytest.mark.skip(reason="A human needs to press a button.") decorator.