Priority Support Code: SMXY5-SB-20223185039:172c27b88cdf2b14d0e36f6d7673b52536bc53cd613984fa26b7ff10829afe7f
I am looking for help on two separate questions on the following system:
Surface Pro 6
3.0.18 - Flavor: Debian package - psychtoolbox-3 (3.0.18.4dfsg1-1~nd20.04+1)
Linux Ubuntu 20.04 with linux-surface kernel
1. Audio quality issue: I am playing simple beeps created using the MakeBeep function and there is cracking/popping sound intermittently (with 25 to 50% of the beeps). This occurs regardless of whether I use latency mode 0 or 2 when opening PsychPortAudio. How can I improve the sound quality/get rid of this popping? Example code:
isi = 0.4;
soundStart = GetSecs + isi
PsychPortAudio('FillBuffer',wdwprop.pahandle,[params.stimulus.tone;params.stimulus.tone]);
for i=1:ntap
toneTime(i) = PsychPortAudio('Start',wdwprop.pahandle,1,soundStart,1);
soundStart = toneTime(i) + isi;
end
2. Pen input on Surface Pro 6. I have created an experiment that uses the TouchEventQueue to record a series of touches while presenting stimuli in a loop and this is working well. I would like to change the input to use the stylus pen instead of the finger but am having trouble figuring out how to use mouse input with Event Queues (rather than checking in a loop at the same time as stimulus presentation). Is this possible (and if so, how)?
For item 2: you can pass any device index from GetMouseIndices (typing this from memory, function is called something like that) to KbQueueCreate and that way use event queues with mice.
Which version of the Linux surface kernel is this?
Wrt. 2, what Dee says. If the stylus pen currently works properly for you as a mouse on your Microsoft Surface Pro 6 (does it?), then [idx,name] = GetMouseIndices() will give you a list of input device names + idx indices. Once you’ve found the name of the stylus, you can do a
stylusIndex = GetMouseIndices([], 'NameOfMyStylus']) with “NameOfMyStyles” replaced by the right name string from the name list. This way stylusIndex will always have the right index to pass into KbQueueCreate.
From there on, you can get button clicks/contact of the stylus like key press events from KbQueue, and for recording the 2D motion of the stylus (or any touchpad or mouse), look at MouseMotionRecordingDemo.m where you’d replace line 37 with d = stylusIndex accordingly. Generally what is reported under the name “Virtual core pointer” or via GetMouseIndices('masterPointer') will be what corresponds to the mouse cursor motion on the screen – the union of all stylus,touchpad,mice etc., whereas selection by product name like above will only query the stylus.
As you can see, the KbQueueCreate call takes some special parameters to enable (x,y) position reporting, or avoid applying pointer acceleration ballistics if it is a mouse device.
Wrt. question 1: Do you hear intermittent crackles or pops in other audio demos, e.g., PsychPortAudioTimingTest, BasicSoundOutputDemo, etc.? Or only in your own code with your specific tones?
Does it only happen on the Surface Pro or also on other models of computers?
Assuming you use Matlab, does it also happen under octave? I just tried and i can force the latency down to 5 msecs without getting any crackles/dropouts under Octave, whereas latency mode 2 maps to 10 msecs and latency mode 0 maps to 34 msecs.
Are all beep tones the same, or do they vary in length, frequency etc. and there’s a pattern to which ones “pop”?
One thing to keep in mind is that the sine wave from y = MakeBeep(...) will not always end on a zero sample at the end, ie. whereas y(1) == 0, y(end) can be anything, depending if a signal of selected duration and frequency will end with fitting in an integral number of sine waves of the given period or not. So a crackle at sound offset could be expected. Cfe. BasicSoundPhaseShiftDemo.m lines 110 - 142 for some elaboration on this.
[Time spent so far: 110 Minutes - Priority support way exceeded.]
Thanks, I had missed the MouseMotionRecordingDemo. That all works well. On my system the x and y don’t seem to be in relative/change values but instead some sort of absolute position values (this is true using the Demo too). Is there a straightforward way to find out how to map to pixels, or should I just touch a known location to calibrate…?
As for the sound issue, I am using just a single sound/beep 100ms and I did add a 10ms ramp to the beginning and end so I’m not getting any clicking in that regard. The popping is inconsistent and only occurs on a few out of a string of beeps. I do hear it in the basic sound output demo.
The popping occurs on a Surface Pro 6 with kernel version “Linux 5.16.10-surfaec x86_64”
Interestingly, it works fine on a Surface Pro 7 with kernel version “Linux 5.14.15-surface x86_64” (but of course the pen is not supported on the Pro 7… which is why the Pro 6).
The demo was written with mice and touchpads in mind, which are relative motion reporting devices, but the stylus is likely an absolute position reporting device, like a touch screen. I don’t have a stylus for my Surface Pro, so i wouldn’t know. Its already nice to know that the stylus works properly.
One thing you can try is KbQueueCreate like in the demo, but without the flags value 4 at the end, so it doesn’t report raw motion values, but processed ones. This may give you direct x,y screen pixel positions in evt.X and evt.Y if you are lucky. Raw values are important for relative motion devices like mice to avoid things like the application of acceleration profiles or inertia (“pointer ballistics”) which is useful for good mousing, but not so much if one wants to measure true motion trajectories of a mouse for studies of arm kinematics/motor function etc. Using processed events may be more convenient for a stylus.
Then there are the valuators, like used in that demo. The numValuators parameter to KbQueueCreate determines how many of them you want reported. Cranking it up from 3 in the demo to a higher number may give interesting/useful additional information for your stylus. E.g., there could be ones reporting things like pen pressure, angles relative to the surface, rotations/tilt, proximity - all depending on the specific stylus device and tablet.
[~,~,~,~,~,valinfo] = GetMouse([], stylusIndex) will report back a valinfo struct array, with one entry for each evt.Valuators(i), e.g., valinfo(i) for i'th valuator. You get fields like .label with a description of what is reported, and .min and .max with the range of values, and .resolution with how fine the axis values are resolved (in units per meter iirc), and .mode to tell if the device reports relative or absolute motion/position. min and max allow to map the reported stylus position from device units to screen units. E.g., a touch screen or stylus may have a more fine grained position reporting than the screen resolution.
Important to note for the interested readers is that this motion reporting is way more limited on MS-Windows and unsupported on macOS, so this is another area where one wants to use Linux for best results.
Ok, so a hardware issue or overload issue then that causes occasional glitching, nothing in your script itself. I assume you don’t have anything very compute-intense running in parallel? Does BasicSoundOutputDemo report any non-zero XRuns in its output? Terminal output of grep --context=5 hda /var/log/syslog?
Terminal output of pactl list sinks ?
Any difference if you choose a different sampling rate out of the set (44100, 48000)?
Any difference when disabling wifi?
Does sound playback with our movie demos, e.g., PlayMoviesDemo and “c” for cool work glitch-free? Does playback of Youtube videos in firefox etc. work glitch-free?
Or some driver issue? For reference, i tested my Surface Pro 6 today with Linux 5.15.5-surface with no glitches whatsoever, even for aggressive low-latency mode 3 (5 msecs target) on Ubuntu 20.04.4-LTS. I’ve never experienced glitches before either, also using the tablet a lot to play music, watch Youtube/Netflix/…, video conferencing, etc. Currently running 5.15.5-surface, 5.14.14-surface, or 5.3.18-surface. So maybe try if installing such a kernel helps, in case some problematic change was introduced into your 5.16-surface kernel?
Note that the paid-for time contingent for priority support is far exceeded by now - over twice as paid for. I’d appreciate if you could buy one more membership. In any case though, unless simply switching to one of the kernels mentioned above resolves the problem, or something trivial springs to my mind in response to your responses, it would be necessary to buy at least two more licenses for additional time before we can proceed. Ofc. another cheapo workaround would be buying and plugging in an external USB sound card…
Any news on this? I now also tested Linux 5.16.11-surface and don’t experience any crackles or other sound artifacts. I only tested with Octave though, not with Matlab.
Any difference if you choose a different sampling rate out of the set (44100, 48000)?
No difference
Any difference when disabling wifi?
No difference
Does sound playback with our movie demos, e.g., PlayMoviesDemo and “c” for cool work glitch-free?
Seems to work fine
Does playback of Youtube videos in firefox etc. work glitch-free?
Yes, it seems to work ok.
Or some driver issue? For reference, i tested my Surface Pro 6 today with Linux 5.15.5-surface with no glitches whatsoever, even for aggressive low-latency mode 3 (5 msecs target) on Ubuntu 20.04.4-LTS. I’ve never experienced glitches before either, also using the tablet a lot to play music, watch Youtube/Netflix/…, video conferencing, etc. Currently running 5.15.5-surface, 5.14.14-surface, or 5.3.18-surface. So maybe try if installing such a kernel helps, in case some problematic change was introduced into your 5.16-surface kernel?
Haven’t tried rolling back kernel yet
After all the above, I discovered that if I remove and ran just the sound portions of the task separately from any other part of the task I am not having the clipping-like sounds so it does seem to be something about the task itself. By adding back other pieces of the task step by step it turns out that setting the priority of the onscreen window to 1 causes the clipping sound. Again, this is true only for this particular system, not a Surface 7 which I set up similarly a while back (with older kernel version).
Ok, your results can make sense, depending on what’s going on otherwise in your script and what Priority() level you are using. Current PTB 3.0.18.4 uses a realtime priority of 1 for the audio processing, just as Priority(1) or Priority(MaxPriority(whatever)); would do. So your main script could compete with the audio thread, and cause timing issues and audio dropouts iff the machine were otherwise under realtime load. KbQueueCreate/Start or TouchQueueCreate/Start for touch input or digitizer stylus input would create a thread with priority of Priority() + 1. So essentially a realtime priority of 1 if you don’t use the Priority() command, or 2 if you used Priority(1). So you’d have at least one higher priority thread than the audio thread if you use touch/digitizer input and the Priority(1) command. As ps -Haxo class,comm,rtprio | grep FF in a terminal shows, there are quite a few other realtime threads running on the Surface Pro 6, with higher realtime priorities, e.g., the mei threads related to the Intel management engine which is involved in touch/digitizer input on the Surface Pro 6, or ssam for Surface specific activities, or card0-crtc0 for driving the touchscreen display.
So it is conceivable that sometimes there might be more realtime threads requiring to run than there are physical cpu cores and prioritization could kick off/starve the audio processing thread of cpu access at the wrong moment for the benefit of another same- or higher realtime priority thread. If you get unlucky, audio glitches could happen at low enough latency settings.
I used to have an optimization to bump realtime priority for PsychPortAudio to Priority() + 4, but that got lost around late 2018 with some driver restructuring. Maybe i need to add that in again, to give timing-sensitive audio processing an extra priority boost over other tasks. That would be for a future PTB release though.
Atm. you could simply avoid the Priority() command, if you don’t see any adverse effects on general timing of your script. Also, given you installed from NeuroDebian, have you run PsychLinuxConfiguration once, to enable some additional timing optimizations?
Another thing i just realized is that i’m a bit surprised that your posted sample code works at all, as you call toneTime(i) = PsychPortAudio('Start',wdwprop.pahandle,1,soundStart,1); in a tight loop without using the PsychPortAudio('Stop', wdwprop.pahandle, 1); function to wait for one tone to end playback before starting the next one. Wouldn’t our driver output a warning on each loop iteration and cut playback short this way?
Anyway, problem solved. We’d appreciate if you could buy one more priority support license, as working on this took way longer than what a single license pays for.
Rainy sunday, so i had another look at this: I’ve changed the realtime priority setup method on Linux with ALSA backend. The freshly compiled PsychPortAudio mex files can be found here atm., until the next beta release comes out:
Would be interesting to know if this fixes your audio glitches when using the Priority() command. It now tries to boost the realtime priority to Priority() + 4 instead of always using priority 1, which could cause competition with the PTB mainthread or other threads like our touch input processing. High priority system threads could still preempt it though.
Also, i was not ever able to reproduce your audio glitches on essentially the same hardware and software under Octave, so this must be some special interplay somewhere - or maybe only happen under Matlab?