How to test your PTB code automatically using Github

Tips & Tricks
1

Automated tests are a great way to make sure your MATLAB / PTB code doesn’t break when you make changes. And if you use Github to version your code, it can run MATLAB automatically every time (continuous integration, CI) you make a change and tell you if your tests pass or fail. Mathworks allows us to run MATLAB using CI without needing to setup any license…

However, PTB itself usually assumes real hardware to operate, and that means without some tweaks, it is hard to run automated tests. I have managed to get MATLAB + PTB working using automated testing using a few tweaks; and can even open a window on a system with no display, draw stimuli, flip etc.

Testing in MATLAB

I use class-based testing: Write Setup and Teardown Code Using Classes - MATLAB & Simulink – each test file is a class that contains many functions that each test one feature of your code. For example this test checks whether my fixation cross stimulus can be set up with an open PTB screen:

function testSetupWithScreen(testCase)
	assumeFalse(testCase, ~isempty(getenv('GITHUB_ACTIONS')), ...
				'Skipping PTB setup test in CI');
	sM = screenManager('windowed',[0 0 800 600], 'disableSyncTests', true);
	open(sM);
	cleanup = onCleanup(@() close(sM));
	f = fixationCrossStimulus('verbose', false);
	setup(f, sM);
	verifyTrue(testCase, f.isSetup, 'should be setup');
	reset(f);
end

I create a screenManager object (my opticka wrapper around Screen) with some properties, and open it, then this gets passed to my fixation-cross stimulus. The important part for the test is verifyTrue to make sure the state is as expected. If this is run on the desktop MATLAB with a real monitor and GPU, all is OK. Buuuut, running it remotely or via CI and PTB will complain / crash…

Solution 1–Use a virtual framebuffer

xvfb is a package that provides a virtual display for X11. We can therefore run MATLAB so that it thinks it has access to a display even when no display is present. So for example:

xvfb-run -a -s "-screen 0 1024x768x24" matlab -batch "addOptickaToPath; cd(optickaRoot); addpath('tests'); results = runtests('tests/TaskSequenceTest.m'); assertSuccess(results)"

…configures a 1024x768 screen and MATLAB and PTB now assume a display is attached and PTB initialisation can proceed. MATLAB runs the command using -batch to ensure path is ok and runs one test class. For Github, we need to install xvfbin the CI environment and then set up the DISPLAY environment variable so MATLAB can use it. This satisfies initialization for most of the PTB mex based functions…

Solution 2 – A screen.mex patch

However if we try to Screen('OpenWindow') using the virtual display, we hit a PTB crash. I got deepseek to evaluate the crash and make a patch (currently a pull-request). While waiting for Mario to evaluate whether this can be upstreamed, I simply include the patched screen.mex and copy it so it is used on Github.

Solution 3 – Testing user actions (eye movements etc.)

So, we have MATLAB+ PTB running on a virtual screen, capable of opening a window, drawing, flipping etc. but what about interaction? As an example, what if I want to test some code that manages an eyetracker fixation window? There is a X11 tool called xdotoolwhich can automate mouse moves, button presses etc. You can see an example of using xdotool to move a dummy eye inside a fixation window held for >0.1 secs:

 function testIsFixatedInWindow(testCase)
	assumeFalse(testCase, ~isempty(getenv('GITHUB_ACTIONS')), ...
				'Skipping PTB eyetracker test in CI');
	assumeTrue(testCase, exist(testCase.xdotoolPath) == 2 || ~isempty(which('xdotool')), 'xdotool must be available for mouse automation');
	[sM, eT, cleanup] = createDummySetup(testCase);
	% Set fixation window at center (0,0 deg) with radius 5 deg
	eT.updateFixationValues(0, 0, 0.1, 0.1, 10, true);
	eT.resetFixation;
	% Move mouse to center of window
	system(sprintf('%s mousemove --screen %d %d %d', ...
		testCase.xdotoolPath, sM.screen, ...
		sM.screenVals.xCenter, sM.screenVals.yCenter));
	WaitSecs(0.1);
	eT.getSample;
	WaitSecs(0.5);
	eT.getSample;
	[fixated, fixtime, searching] = eT.isFixated;
	verifyTrue(testCase, fixated, 'should be fixated when in window');
	verifyTrue(testCase, fixtime, 'should be fixated longer than the fix time');
	verifyFalse(testCase, searching, 'should not be searching when fixated');
end

See the full tests here…

Summary

Tests are a great way to stop code breaking unexpectedly. Getting Github to run tests every time you commit something keeps bugs from accumulating. With some setup, we can do almost every possible test on a headless remote server. Now, precise timing is not going to be testable, we should assume flip times running using software renderer on a virtual display are not meaningful. But still, being able to draw stimuli and test dynamic responses from subjects automatically is still a great help. We are limited to PTB 3.0.21 running remotely, but that is fine for most testing.

For drawing, it should be possible to provide reference images and the test whether the framebuffer contents matches within a margin of error (as PTBs current demo code does). I don’t do these kinds of tests yet…

My Github CI workflow is here for anyone wanting to see how to set this up: