How to record joystick/mouse motion in the background, sampling at > video refresh rate [was: Flip dontsync not working]

Hi All,

I’m creating a task where participants move a cursor from a centre point through one of 8 radially arranged targets by manipulating a joystick. Id like to sample the position data at higher than the 60Hz monitor refresh rate, but was initially limited by the syncing of flips. I got around this by setting the dontsync option to 1 (I realise this breaks timing, but Im relying on GetSecs and a photodiode for this) and things were working really well. However, Id done this while using the PsychDebugWindowConfiguration command. After removing the debug option, the sampling frequency is again limited to monitor refresh rate, despite the dontsync flag still being set to 1. I’m assuming that Ive missed a setting required to enable dontsync, but cant find any answers. Any help would be greatly appreciated. This is being run on Ubuntu 22.04 with an AMD Radeon RX6600.

cheers,

George

That is the wrong approach for sure, and there are way better ones, depending on which OS your script shall run on / be compatible with - Linux being the best choice as usual.

help PsychPaidSupportAndServices for a paid support membership token, and I’m happy to guide you through the good options. Otherwise good luck, the answer is within PTB’s documentation and demos.

-mario

Hi Mario, your input would be greatly appreciated, authentication token is below.

cheers,

George

T6TCR-DX-202427131859:06092c9361403195810a7875167098ddc72d63cc9a13b3d8a8c5fa6f2c6e0c6f

So far this token does not validate, which means payment has not yet been received. I’ll come back to you once it does.

That’s odd, as it was paid via credit card. I can follow up with our finance department if needed.

Validated now. It can take anywhere between hours and weeks from arrival of payment to when I can see it in my database, as there is a busy human in the loop other than me. Hence the advice to buy the membership well in advance.

So the reason this does not work, is that dontsync option 1 will indeed prevent your script from waiting until a flip completes, synchronized to the start of video refresh/vblank, at the cost of no longer getting a visual onset timestamp anymore. But there is only a limited amount of OpenGL image buffers your script can render to ahead of displaying them, typically 1 - 3 at most depending, on operating system, graphics card and system config. Once those are all filled and 'Flip’ped with new stimulus images, ‘Flip’ is forced by the operating system to block again, until at least one of the image buffers is consumed and shown by the display and freed up for reuse → you are back to lock-step with the video refresh at 60 Hz, or 60 presentation loop iterations per second. Essentially this can be useful if one only expects small “jitter” in drawing, but not to compensate for a situation where one wants to execute ones drawing loop at an average rate higher than the video refresh rate, like in your case where you want to sample the joystick faster than 60 samples per second. Using dontsync 2 would disable vsync and this throttling on many setups, but then your stimulus would be visually corrupted/undefined by tearing. PsychDebugConfiguration will cause routing of images through the desktop compositor for visual transparency, and disassociate the presentation cycle of your script from that of the compositor/display, again at the cost of no timing precision and the compositor likely dropping various stimulus frames at its discretion → Undefined stimulus presentation and timing again, where some of your stimuli may never show onscreen, and photo-diodes might get confused and fool you as well. So many ways to fool oneself with photo-diodes if one doesn’t understand the display system very well.

What you want to do is depending on operating system or the need for compatibility across operating systems. But seeing you chose Linux, that allows for the most elegant solution:

If you can restrict your script to Linux, and use our joystick / Gamepad support, you could simply record the joysticks motion in the background, independent of the execution and redraw rate of your Matlab display script. Our Keyboard queues on Linux can also record time-stamped mouse movement trajectories, and joysticks under Linux are just treated as unusual mice, so this works with joysticks as well.

Look at MouseMotionRecordingDemo.m and replace GetMouseIndices with GetGamepadIndices. [a,b] = GetGamepadIndices will list device name strings in variable ‘b’. Choose the one matching your joystick, say e.g., it is called ‘NameOfMyJoystick’, and then you can call
d = GetGamepadIndices('NameOfMyJoystick') in the modified MouseMotionRecordingDemo.m to get it to record joystick motioon and get an idea how this works. With timestamped events for each joystick movement.

The same technique would work for mice on MS-Windows, but not for joysticks atm… It would not work at all on macOS.

In your code you could use KbQueueStart/Stop to start/stop joystick movement data collection for a trial, KbEventFlush to discard data from a previous trial at the start of a new trial, and the kind of while loop shown in that demo inside the animation loop of your script, to fetch and store away whatever data is available during a single iteration of your drawing loop, and then use the last fetched item to update visible position of your stimulus cursor, redraw and then ‘Flip’ the updated stimulus image again conventionally, rinse, wash, repeat.

Let us know how that goes, or if you need a more detailed explanation. Or a less elegant / powerful solution that should work cross-platform though, by polling use of AsyncFlips.

-mario
[14 out of 30 minutes paid time used].

Hi Mario,

Thanks for the detailed reply, very much appreciated. I played around with the KbQueue commands previously but couldn’t figure out how to implement that kind of while loop in a way that would allow me to store movement data in the background. The issue I had was that the speed with which I could call KbEventGet was still limited by the flip rate, and I couldn’t see how to access older buffer entries that accumulated between flips. Is it possible to schedule the flip to a known point and run the while loop until that point? Apologies if Im missing something obvious here!

cheers,

George

Shouldn’t be limited if done properly. I’ve created a sample script that illustrates the structure of an experiment and the fetchData() helper function…

You still need to fix it for use with joystick, as I don’t have a joystick at hand right now, so it is written for mouse movement. And remove the fprintf() debug output and related code from fetchData() → Lines 104 - 118 and 93 can go for production code. But it gives the building blocks for incorporation into a trial loop. This is essentially one trial only.

Edit/Note: Wrt. joystick, have a look if evt.valuators() is giving you what you want , e.g., joystick deflection, instead of the x,y positions that the demos fetchData() function integrates and then returns from the valuator values. The demo was written and tested for mice after all, so the meaning of the valuators actually values will be different. We do ask for raw motion events, instead of the regular “cooked” events, as cooked events will be post-processed movements, applying pointer ballistics and similar stuff to make mouse movements and such smooth and efficient for the user, but probably not what you want for measuring the kinematics of a subjects motion.

[50 minutes out of 30 minutes paid time used - Paid support exhausted]

1 Like

Thanks so much Mario, this is a huge help!