I need to track the times of mouse button events (OSX ‘Apple Internal Keyboard / Trackpad’) during the playback of a movie, and would appreciate advice on how best to do this between frames. I’m a beginner to Psychtoolbox, but understand that OSX has a number of compatibility issues.
Usually, if tracking mouse events while playing audio, I can spin a while true loop and constantly retrieve GetMouse between PsychPortAudio('Start'...) and PsychPortAudio('Stop'...). With a movie, as per the below code snippet, it’s unclear how this fits in without affecting playback (missing frame deadlines); or missing button events due to the frame-by-frame loop.
Unlike keyboard events (using KbQueue), it doesn’t seem like there is a built in way of tracking mouse events asynchronously? If trying to use the trackpad device, I understandably get the error Specified device number is not a suitable keyboard type input device..
Is this related to the mess that is OSX? Or, is there a more sensible way of inserting mouse-event tracking below?
Thanks!
Screen('PlayMovie', movie, ...);
...
while 1
% Is there a way of filling gaps between frame presentation with GetMouse?
tex = Screen('GetMovieImage', win, movie ...);
if (tex < 0)
break;
elseif (tex == 0)
WaitSecs('YieldSecs', 0.05);
continue;
end
Screen('DrawTexture', sess.win, tex, ... );
Screen('Flip', win);
Screen('Close', tex);
...
end
...
Screen('CloseMovie', movie);
....
OSX Mojave 10.14.6 (sorry, I’m stuck with this for now)
Matlab R2019a
Thanks for the suggestion - I’ve tried this again, and still find that it only tracks some mouse events (no key list provided). Maybe, there’s an obvious reason for the inconsistency below!
When tapping the trackpad, no events are registered. This differs to GetMouse, where the buttons(1) in the return is toggled - this is what I’m keen to register during the movie.
When clicking down on the trackpad, events are generated:
I can confirm that taps are not reported as touchpad clicks on at least macOS 10.14 with the internal trackpads of the MBP 2010 and MBP 2017.
On Linux it is very likely to work, as tested on MBP 2010 with Ubuntu 18.04.4 LTS and MBP 2017 with Ubuntu 19.10. A single tap is just reported the same as clicking the trackpad ie. as Keycode 1 pressed+released. I can’t know how 3rd party trackpads will behave - don’t have any. But most likely the same ie. Linux works, macOS fails.
In addition, on Linux you can also detect double/triple clicks on the trackpad. And finger movement trajectories if you set up KbQueues properly, as demonstrated by MouseMotionRecordingDemo.m.
How easy it would be to install Linux on your Apple machine is highly dependent on the model. Apples machines used to be a good choice for Linux, but have become very hostile to 3rd party operating systems in the recent years.
Thanks Mario. In my case, taps are reported as mouse button press/release using GetMouse - why is this different when tracking the device using KbQueue? Should it behave the same? If so, I’d be interested in whether I can find a fix.
Unfortunately I don’t have the option to dual boot, and am unsure if a VM would run smoothly enough (sadly it’d run from an external HD). Might be time to find a separate machine…
It’s different, because it uses totally different api’s for its job. On macOS GetMouse uses the Carbon high-level GetCurrentButtonState() function, whereas everything KbQueue… uses the HID low-level frameworks. As such it is not expected to behave the same.
The implementation of user input is different on each OS, so one would expect OS differences, although for typical use PsychHID tries to abstract those differences away without losing too much control.
A VM wouldn’t work well at all, as it abstracts away the low-level bits we need to do a good job, and also reduces timing precision.
You can install Ubuntu Linux on an external disc drive though and dual-boot from that. I do this all the time for testing different hardware. With modern fast drives and connections it shouldn’t impact performance in a meaningful way.
Ofc how well it will run depends a lot on the type and year of your hardware and how much hostility Apple has added to the specific model.
I’ll consider dual-booting from an external drive, I have a spare(-ish) USB SSD which hopefully shouldn’t compromise performance significantly. Fingers crossed.
Please can you let me know how you often approach this?
I’ve tried both manually (hdiutil/diskutil/dd) and using software (Etcher/UNetbootin) and run into issues - two EFI Boot options appear on start-up, and both trigger some recovery-esque mode suggesting either the media is unusable or an update is required (which never finishes).
Because of the latest Apple security bullshit - them making it intentionally hard to even boot 3rd party OS’es that are cryptographically signed for secure use with secure boot: First you’d have to disable secure boot and enable booting from external drives in some security settings menu in macOS Recovery. See:
Then you might be able to boot a Ubuntu live/install system from your external USB stick (Ubuntu gives instructions on how to get it onto a USB stick). Then you could in the Ubuntu installer choose custom install and select your externally connected USB-SSD as target drive fro Ubuntu.
However, looking at this thread and some references there to the MBA 2019:
I’m pretty sure the out of the box experience will be quite unusable, unless you learn how to patch kernels yourself to run all kind of experimental prototype code. I’d guess it will be at least late 2020, or more likely 2021 or 2022 until something like Ubuntu will work out of the box with good feature set on the MBA 2018/2019.
In general all MacBook’s after the 2017 models are not well supported atm., thanks to Apples non-standard hw/fw design and general hostility towards other OS’es. I have a MBP 2017 which works ok enough to be useful for PTB development and testing - and better working for that than on macOS - under Linux for myself. But it is not perfect, and getting it to the current state took me 2 full days of work.
Ofc. you can try booting the Ubuntu live system from the USB stick an play with it, to see if it is workable at all before you’d try to actually install it on a external drive. Ubuntu 19.10 would be the choice atm. A simple “sudo apt install octave-psychtoolbox-3” would even install octave + ptb into the live system for a quick test run with ptb.
But if the keyboard or touchpad would not work in the first place, i guess that might be the point to give up, and simply get some well designed and Linux friendly laptop that would work much better. If i had to guess, i’d guess you could probably get a better suited PC laptop machine than the MBA for not much more than half the prize.
Thanks Mario for the detailed answer - I agree that this looks like it might be a nightmare on any recent OSX.
I’ve managed to get hold of a reasonably high spec desktop through work, and am installing Ubuntu ready to get started. I’ll let you know how it goes, one outstanding challenge will be finding a suitable trackpad or equivalent device for monitoring taps.
EDIT: I should have asked, if you have any advice on the above (for reliable performance, sensitivity and linux compatibility), it’d be much appreciated!
Even a low end machine should be able to play typical movie files with ease. As usual, Intel or AMD graphics is strongly recommended over NVidia for care-free operation.
Wrt. external trackpads, i only have tested our touch input once with a original Apple Magic trackpad around October 2017 that was laying around on a shelf in the lab, and at least that worked at that time (cfe. help TouchInput). Haven’t had access to one since, and of course it is possible that Apple worked relentless since then to break it for other operating systems. Googling suggests though that the Magic trackpad 2 is also supported.
EDIT: Now, SimpleMovieDemo behaves (I re-installed gstreamer). Unfortunately it’s still playing up on my custom .mkv playback (blocking on GetMovieImage) - I suspect I’m handling the audio incorrectly, tbc…
Fingers crossed. I’m currently getting timeouts on GetMovieImage but hopefully will suss it out soon. I’m testing on two devices and the other has an Intel 620, which might behave a bit better…
For example,
>> SimpleMovieDemo
...
PTB-INFO: Advanced Micro Devices, Inc. [AMD/ATI] - Raven Ridge [Radeon Vega Series / Radeon Vega Mobile Series] GPU found. Trying to establish low-level access...
PTB-INFO: Unsupported AMD gpu detected. No low-level access, because the gpu is too new or too old. [pciid = 0x15dd]
PTB-INFO: Failed to map GPU low-level control registers for screenId 0 [Success].
PTB-INFO: Beamposition timestamping on NVidia and AMD gpu's, and other special functions on AMD gpu's, disabled.
PTB-INFO: If you are using the open-source graphics drivers, then this usually doesn't matter for typical use.
PTB-INFO: OpenGL-Renderer is X.Org :: AMD RAVEN (DRM 3.33.0, 5.3.0-40-generic, LLVM 9.0.0) :: 4.5 (Compatibility Profile) Mesa 19.2.8
PTB-INFO: VBL startline = 1080 , VBL Endline = -1
PTB-INFO: Will try to use OS-Builtin OpenML sync control support for accurate Flip timestamping.
PTB-INFO: Measured monitor refresh interval from VBLsync = 16.639919 ms [60.096445 Hz]. (50 valid samples taken, stddev=0.000488 ms.)
PTB-INFO: Reported monitor refresh interval from operating system = 16.640042 ms [60.096001 Hz].
PTB-INFO: Small deviations between reported values are normal and no reason to worry.
PTB-ERROR: In Screen('GetMovieImage') for movie 0: Timed out while waiting for new frame after 5 seconds!
Fwiw,
04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Raven Ridge [Radeon Vega Series / Radeon Vega Mobile Series] (rev c5)```
So our stanard demo movie for SimpleMovieDemo works now? That means you installed a previously missing gstreamer package, i think gst-libav or such, which decodes certain proprietary formats like H264.
That hang usually means some decoding problem for your movie format.
A good test would be to play your movie form a terminal via “gst-play-1.0 moviefilename” to see if it is related to your movie.
It could also block on audio if PsychPortAudio would be used in parallel, taking exclusive access to the sound card, taking it away from GStreamer. ‘OpenMovie’ has some optional ‘specialFlags’ setting iirc. to tell not to play movie sound. You can’t use sound from movies and PsychPortAudio at the same time.
Yep - this is the problem, unlike OSX I need to explicitly close any PsychPortAudio handle before starting a movie.
Also, to get tapping working reliably during movie/audio playback, by chance, I found it only works when using xinput --test in another window. I don’t know why!
Now to figure out why synaptic isn’t behaving for certain trackpad interractions…
You can also tell OpenMovie that the to-be-opened movie won’t have any sound/need for sound output. Unless you’d want to play sound from the movie and PsychPortAudio simultaneously.
The tapping thing is weird, i don’t need to do anything with xinput to make it work reliably, now also tested on a HP laptops touchpad. Fwiw, on Linux you can also use our multi-touch input functionality and abuse a touchpad as “touchscreen”: help TouchInput explains how. A bit clunky, you’d have to tell the synaptic driver to ignore the touchpad, but doable if you need multi-touch input. Maybe tuning some touch parameters via “synclient” or a similar tool would be enough for your purpose.