Unit tests for PTB wrapper functions

Programming Help
1

Hey all,

A couple of colleagues and myself have been working on creating a set of “wrapper” functions for psychtoolbox: https://github.com/cpp-lln-lab/CPP_PTB

The motivation is partly that some of us were getting tired of always rewriting the same code to display a fixation cross or collect keyboard responses or having the same snippet of code to abort the experiment.

So in the spirit of “Don’t repeat yourself” we are working on a small library of functions we can reuse across projects.

I am looking for suggestions or pointers on how to wrote good “unit tests” for those wrapper functions to ensure that they do what they are supposed to do.

Ideally, I want those tests to run with some sort of continuous integrations (CI) like github actions or travis which would also make sure that those functions run with octave by default.

So to do that the wrapper functions should be somewhat “decoupled” from their PTB implementation because PTB won’t run on CI system.

I am unsure how to go about this (like most scientist I got 0 knowledge on software development and good coding practices), so any help or advice would be VERY welcome.

Thanks in advance.

Rémi

2

Hi Remi, did you ever get anywhere with this? Github allows (for free with public projects) a full release of MATLAB to run unit tests and other code using a Ubuntu virtual machine: GitHub - matlab-actions/overview: Automate your workflows with GitHub actions for MATLAB.

The VM obviously doesn’t contain any hardware, so most Screen or other core functions would currently fail. PTB would need to have some sort of trigger to silently drop errors (using something like an environment variable), or someone would need to make a “mock” Screen etc. command. But building this test-mode into PTB or building a mocked Screen, PsychPortAudio and other components would be a non-trivial time drain. It seems a modern CI / CD workflow will remain out of reach for PTB for the foreseeable future…

3

Hi Ian,

I furthermore wonder what the use of a mocked Screen etc would be. Answering the mail is not enough to test PTB. Its visual quality and timing (for Screen) that needs to be tested, automation of that would require capturing the video output of a video card and comparing to a reference.

4

Hi Dee, I completely agree in that I wouldn’t think we could automate testing for the core PTB functionality (interfacing with exotic hardware and precise timing of said interface). BUT the point of mocking is as long as the inputs and outputs are consistent, any user script that depends on PTB would run. For example say a user has a script that call Screen('Flip'). The idea would be that the rest of the user script could run, and when Flip was called a no-operation is performed, and as long as this mocked Flip accepts the same number of parameters and returns the same number of parameters, the calling script would not “know” no flip was actually performed. This would mean that CI tests could be run on the user script. This is similar to the dummy mode in your Titta toolbox, you can run the code even if no eyetracker is really connected…

I had a brief look at the C source to see if there was some way to automatically parse the inputs and outputs for the main PTB functions like Screen. It should be easy to parse e.g. ScreenSynopsis.c to generate a list of pseudo-functions to generate. The problem is if user code depends on specific characteristics: e.g. that vbl must increase monotonically. This would require some manual intervention…

Of course some sort of CI testing framework for PTB itself would also be ideal, ensuring no regressions on each commit; but as you point out, this would have to be a custom and complex franken-machine even for limited sub-system testing.

5

Hi Ian,

I see your point, i was only thinking about PTB itself. Also there there could be a use, doing some CI by automatically running all demos perhaps. But the class of bugs it would catch is very limited, and the work needed substantial.

What would be interesting is to try and automate testing using something like a VPixx device that can capture video/sound output and timing and feed it back to a testing script that then compares against expected output. That does require quite some funds, both for hardware and especially for man hours. Its not sexy and thus likely hard to fund…

Cheers,
Dee

6

In principle, one can do way more on Linux than stub-out functions. The OS can simulate substantial parts of the graphics/hw in software, specifically for the purpose of CI and development/testing. E.g., the Mesa OpenGL library has the llvmpipe software renderer which implements all of OpenGL 4.6 in software, it falls back to that if given hardware does not support hw acceleration. There are X-Servers like Xvfb that simulate a virtual framebuffer to operate on (cfe. xvfb-run octave), and at the lowest level there are the Linux VKMS video drivers which simulate displays and even stuff like the video refresh timing / timestamps etc. via. high-resolution timers (cfe. drm/vkms Virtual Kernel Modesetting — The Linux Kernel documentation). I haven’t tried any of this in practice, and ofc. we don’t know how that - apparently free to use for open-source projects - CI system provided by GitHub + Matlab is set up, i’m pretty sure it won’t go down to VKMS.

So in theory one could test PTB itself and code above it in a reasonably realistic fashion for many cases. If one would put in the huge amount of work of writing an automated test-suite that actually tests things worth testing. Efforts so far, from Dee iirc., mostly test things that don’t really need testing – the stuff that is most unlikely to actually break, but most easy to test.

But then, ofc. i do test PTB manually in areas that i work on, ie. where there is some chance old stuff could get broken by working on new stuff or improvements. And i manually run most tests and demos at least at their standard settings or common settings about once a year with the latest Matlab, on Linux, Windows, macOS, with some subset of hardware, because Mathworks pays us a little bit for that – not enough for this to be an actual profit, it barely covers our costs if nothing needs fixing as a consequence of those tests, but given the poor funding by our dear users, we take what we can get to stay afloat. The testing in 2021 took somewhere between 160 - 180 hours of manual tedious work. Following is the 1st paragraph of my report to Mathworks for the 2021 contract cycle:

“Compatibility of Matlab R2021b with current PTB was tested on Ubuntu Linux 20.04.3-LTS with AMD, NVidia and Intel graphics, on macOS 10.15.7 with AMD graphics, and on Windows 10 21H1 edition (most recent at time of testing this November) with AMD and NVidia graphics. No major problems were found in Psychtoolbox itself, but about 99 separate improvements or changes were made to tests and to demos and to minor functions to improve tests, or the general user experience. Close to 100 tests which are part of PTB were run on all operating systems, and almost all of our bundled demos (around 110 demos), were run on one OS and then spot-tested on others, when meaningful. These at basic test and demo settings. A full run of everything with every parameter setting would have probably taken thousands of work hours. This basic testing took somewhere between 160 and 180 work hours.”

The report for 2020 said: “Was tested and ensured in a similar manner as described in the 2021 contract for Ubuntu Linux 20.04.1/20.04.2 LTS, Windows 10 20H1, and macOS Catalina 10.15 for Matlab R2020b.”

I should note that testing rarely finds anything important broken in PTB itself, all major problems are 99% due to bugs introduced into the operating systems, graphics drivers etc. by OS or hardware vendors, mostly on macOS and Windows.

Ofc. the most interesting and valuable stuff to test is therefore how PTB runs on real operating systems with real hardware, interactions etc. and to find the bugs in the operating systems, drivers etc. So Dee is right that the biggest benefit would come from automated testing of the real sw+hw stacks, but this is by far the most time consuming and money consuming effort, not worth starting until/unless we have stable funding streams for building and longterm-maintenance of such things.

As far as manual testing goes that isn’t paid by somebody, I fully focus on Linux, usually pro-active testing with some very occasional minor spot testing on Windows or macOS. Tests on Windows and macOS are almost pointless - apart from tracking the degradation of these platforms - as results are not actionable: You can’t get any problems fixed properly by anybody, because nobody at Apple, Microsoft etc. cares. And you most often also can’t work around it in meaningful ways (with meaningful effort or at all), given the complexity and fault modes are increasing all the time.

On Linux i can proactively monitor and test, and fix bugs myself or sometimes with collaboration of upstream, or get them to fix something. It’s a tremendous amount of work and also not guaranteed to succeed always 100%, but at least one can do something about things and the starting situation is so much better, quality-wise, compared to the proprietary operating systems. E.g., pro-active testing/fixing/improvements for the upcoming April 2022 Ubuntu 22.04.0-LTS release already started in September last year and only preliminary concluded last Monday. Another testing cycle will follow closer to the release. A short skim of my todo list shows that this resulted in:

  • 9 separate bug fixes and improvements for the X-Server 21.1 in next Ubuntu (assuming/hoping it will ship with that if it makes the cut) and 2 bug fixes / improvements for X-Server 1.20.14, which may or may not show up in Ubuntu 20.04-LTS.

  • Improvements to smooth line rendering on AMD + Linux, and to MSAA for PTB on Linux and as a nice side effect on Windows.

  • Bug fix to the KDE desktop for 10 bpc deep color output.

  • 3 bug fixes for Portaudio for sounder sound.

  • A bug fix to Monado for use by our future OpenXR VR/AR driver.

This is an incomplete list of what’s needed to keep the highest-quality and most suitable OS for neuroscience, with the best QA and development processes, stay high-quality - minus all the things i might have missed, given the rate of change and my limited time and resources. Windows or macOS would need more work than that, except such work can’t be performed in the first place in a productive way, given the conditions imposed by the proprietary systems. I don’t even want to think about the number of hours that went into this (more than 300 hours in the last 6 months for sure) and the amount of delays this causes for actual paid contract work which brings in the money to keep us afloat. It added multiple months of delays…

The FOSS community has various CI systems for regression testing, so focusing on PTB+Linux for an automated CI system would also allow us to hook into those systems - We wouldn’t need to do all testing on our side, but offload to the upstreams and integrate with them, only doing the specialized stuff locally. Way more efficient and effective if done right than anything one could hope for with the proprietary OS’es.

But all this stuff stands and falls with the availability of major funding and stable long term funding for time and equipment and places to house such equipment. During the pandemic was the first time in a decade i couldn’t even use my old Datapixx to aid in highest quality testing, because it was locked away in a locked-down lab, and my last CRT monitor died after 20 years of great service.

In the end, our users will get what the pay for, and atm. they still get way more in terms of pro-active QA and resulting quality than they reasonably deserve. But maybe this insight won’t sink in until some real train-wrecks happen, if ever…

So yeah, a good hardware in the loop CI system would be as great as it is unattainable with our current resources and funding. And the usefulness of a CI that only tests trivial stuff unlikely to break in the first place, or to affect many users if it breaks, is questionable. Volunteers welcome ofc., e.g., for testing what the GitHub+Matlab CI can do, as long as they don’t create any extra unpaid work for myself.

-mario