I’ve encountered a very strange issue when I play audio and video in an experiment.
I have an audio sound, a movie, and another audio clip playing sequentially. And I have to use PsychPortAudio(‘Open’) before the first sound play and PsychPortAudio(‘Close’) between the first sound and movie. Then, use PsychPortAudio(‘Open’) again before the second sound play. This is because if I don’t close the handle after the first sound play, movie will be played without audio.
The issue that I have is that the sound device (e.g. speaker) used in the first time PsychPortAudio(‘Open’) disappeared from the device list (from PsychPortAudio(‘GetDevices’)) after the movie playback, ended by Screen(‘CloseMovie’).
If I don’t play any movie between the two sound, the sound device used in the first PsychPortAudio(‘Open’) is still available in the subsequent PsychPortAudio(‘Open’).
So, it seems the movie playing affected sound device. Whenever the movie plays, there are error messages in the terminal (I launched matlab from there). Here is the error message:
I used Psychtoolbox 3.0.19 on a machine running Ubuntu 23.04. MATLAB version is R2023a. And I installed Psychtoolbox following the the instruction of Subversion-based installation method on the PTB website.
The issue seems to be caused by GStreamer. My GStreamer version is 1.22.
Thanks!
Update
It seems the movie play temporarily disabled the audio device. If I set an interval of a few seconds between the movie offset and 2nd audio playing (FillBuffer), the original audio device will be available. If I play the 2nd audio file immediately after the movie playback, the device will disappear.
This issue is independent of PTB version, as I replicated this issue under 3.0.17 PTB on the same machine.
Looks like Mathworks introduced a new compatibility problem into Matlab, by shipping their own outdated/incompatible libfreetype.so library that overrides the good system installed library and causes this - unrelated to your sound problem though, but should cause playback failure of various movie formats which use libgstlibav.so for decoding. Maybe a new problem specific to Ubuntu 23.04 and similar modern distributions. I only tested Ubuntu 23.04 on one machine for an hour with Octave, so I wouldn’t know or care if there is new Matlab related trouble on a not officially supported Ubuntu version.
Not a supported distribution, only most minimally tested, some known problems, as I mentioned to you before. Not tested with Matlab at all, as that is way more work.
It would be also caused by almost any audio playing, e.g., a youtube video in a webbrowser, a music player, some system feedback sound (“you have new mail”). Our GStreamer movie playback engine like most apps uses the Pulseaudio desktop sound server, which will take over exclusive control of the soundchip, so it becomes unavailable for PsychPortAudio GetDevice enumeration until the Pulseaudio server releases it again. Auto-release happens a few seconds after being idle, I think somewhere around 3 - 5 seconds.
There may be one way around it atm., if one does not care about audio timing precision at all:
PsychPortAudio('EngineTunables', [], [], [], 0) as very first PsychPortAudio command in a session. Prevents PsychPortAudio from grabbing an audio hardware device for exclusive access, thereby allowing other sound apps to continue to work, e.g., the Pulseaudio sound server and thereby our movie playback engine.
Do the GetDevices, find the deviceindex of the device with DeviceName “default” and open that device in PsychPortAudio Open. Do your thing and keep the audio device open.
→ This way PsychPortAudio will also use the Pulseaudio sound server for sound output, collaborating with all other running sound applications and GStreamer movie playback. Downside is inaccurate timing, potentially dozens of milliseconds off, or even more, as this way of operation was never designed or even tested/validated for any kind of timing precision.
This is all a known limitation which becomes apparent on Linux. I tried to work with somebody early last year and in bits over last year to help get a Linux Pulseaudio backend into Portaudio with proper timing, but that collaborative work progressed too slowly.
The utterly disgusting and disappointing lack of financial support by the majority of our users means a lack of funding → lack of time to spend on such improvements, as I have to prioritize anything that brings in money and pays the bills over other work.
Pulseaudio, now at an age of 18 years, is on the way out anyway for desktop Linux. Recent/Current distributions, e.g., Ubuntu 23.04, all switch(ed) to the new Pipewire audio and media server, so there isn’t much point in improving the Pulseaudio situation anymore anyway.
The same limitation exists for the high precision timing backends on MS-Windows, e.g., WASAPI. And while not immediately apparent on macOS (or sometimes on Windows, depending on specific usage), any shared use of the sound system will impair timing precision and trustworthiness. PsychPortAudio’s precise timing was only ever validated for exclusive use of sound cards, not shared modes, where there is always some kind of sound server / software mixer running which can badly interfere with timing, unless specifically designed to not do so. E.g., on Linux one can manually install and setup the JACK audio server and use it with PsychPortAudio if one needs shared use of audio and more precise timing, as JACK is meant as a pro audio server specifically for use by musicians etc., where timing also matters for live performances, recording sessions etc. But JACK is not the default on typical Linux distributions, as it is not as plug & play and beginner-proof as Pulseaudio or Pipewire.
It sounds exciting that the current Linux distros are using Pipewire. Would PsychPortAudio use Pipewire as the backend in the future?
I’m not familiar with sound cards or related engines. I can see Pipewire listed after running PsychPortAudio(‘GetDevices’). Should I choose Pipewire as the audio device, over other devices, such as systemdefault or usb sound card?
Thanks again, and we will budget financial support to ptb in the near future!
It uses portaudio, so whatever backends the installed libportaudio exposes, PsychPortAudio can use. If Psychtoolbox is still maintained by the time a future Ubuntu LTS with “Pipewire by default” was released and the base-funding by our users is much much better than what it is now (like at least 25x as much) then I can consider active work on improvements in this area. If not, then this will be the least of the problems the neuroscience community will face when it comes to demanding/high quality data collection.
One way to combine relatively good timing precision/low latency with the ability to share the soundcard with other apps (or our GStreamer support) right now, is to use the jack backend of PsychPortAudio/portaudio. Portaudio’s jack backend – according to my code review and some light testing on some setup a few years ago – does have the kind of timing/timestamping support that PsychPortAudio needs. One way is to install/setup a real jack 2 server, the other way is a pipewire emulated jack server.
Pipewire is supposed to be a reasonably compatible replacement for both pulseaudio and jack, iow. Pipewire has pulseaudio and jack frontends and can thereby pretend to be a pulseaudio or jack server.
I only touch-tested this for a few minutes quickly in my light test of Ubuntu 23.04 a few weeks ago, but iirc. there is some command-line tool pwjack or pw-jack or pipewire-jack or something similar to that, which allows to launch at least octave (a la pw-jack octave --gui or something like that) with some jack client library interposer injected. If launched this way, then PsychPortAudio GetDevices will also list some Jack audio devices. One choose those to use Pipewire via Jack with probably reasonable timing/latency etc.
Note that timing precision/reliabilty/accuracy etc. was only properly tested with PsychPortAudio when using a native sound device in exclusive mode (ALSA on Linux, WASAPI (and ASIO in the past) on Windows, Coreaudio on macOS), so all these shared modes may or may not work with correct timing. Also due to lack of funding, the last full validation test is a while ago. In the past, I revalidated this stuff against each major new OS release, but if our users don’t care enough about the quality of their tax-payer funded research to support us financially even a little bit then I can’t waste my time and money on extensive free validation work anymore.
Maybe, I don’t know, don’t care at this time. Our driver or portaudio does not have a native pipewire backend, so whatever that is is an emulation of unknown quality.