Ubuntu + Nvidia sync trouble

[natalia]

Dear all, dear Mario,

we frequently (but not always) observe sync failure on our Dual-monitor Ubuntu setup with various scripts, including PerceptualVBLSyncTest.
I understand that this is one of the most common issues and the one hardest to track, but I would at least appreciate a hint on where to look for trouble first (there are so many options).
I know that NVidia is not recommended but this is what we have. If it really turns out to be the problem, we will try to replace it.

Here is the system information:

PsychtoolboxVersion: WARNING - Could not query additional version information from SVN – svn tools not properly installed?!?

ans =

'3.0.16'

Installed addons/adjustments: linux lowlatency, PsychLinuxConfiguration, XOrgConfCreator

Occasionally, PTB prints a message about installing LinuxGameMode:

PTB-INFO: Failed to request additional performance tuning from operating system.
PTB-INFO: This is because the optional “FeralInteractive gamemode” package is not installed
PTB-INFO: and set up yet. If you want to have these extra optimizations, then read
PTB-INFO: the setup instructions in “help LinuxGameMode”.

Further information:
OS: Ubuntu 18.04.4 LTS
MATLAB 2019b
GPU: GeForce GT 1030/PCIe/SSE2
NVidia driver 440.100

PTB output:
PTB-INFO: This is Psychtoolbox-3 for GNU/Linux X11, under Matlab 64-Bit (Version 3.0.16 - Build date: Mar 26 2020).
PTB-INFO: OS support status: Linux 5.3.0-61-generic Supported.
PTB-INFO: Type ‘PsychtoolboxVersion’ for more detailed version information.
PTB-INFO: Most parts of the Psychtoolbox distribution are licensed to you under terms of the MIT License, with
PTB-INFO: some restrictions. See file ‘License.txt’ in the Psychtoolbox root folder for the exact licensing conditions.

PTB-INFO: Connected to NVidia GP108 [GeForce GT 1030] GPU of NV-130 family with 4 display heads.

PTB-INFO: OpenGL-Renderer is NVIDIA Corporation :: GeForce GT 1030/PCIe/SSE2 :: 4.6.0 NVIDIA 440.100
PTB-INFO: VBL startline = 1080 , VBL Endline = 1124
PTB-INFO: Measured monitor refresh interval from beamposition = 16.667123 ms [59.998358 Hz].
PTB-INFO: Will use beamposition query for accurate Flip time stamping.
PTB-INFO: Measured monitor refresh interval from VBLsync = 16.666755 ms [59.999683 Hz]. (298 valid samples taken, stddev=0.582320 ms.)
PTB-INFO: Reported monitor refresh interval from operating system = 16.666667 ms [60.000000 Hz].
PTB-INFO: Small deviations between reported values are normal and no reason to worry.

WARNING: Couldn’t compute a reliable estimate of monitor refresh interval! Trouble with VBL syncing?!?

----- ! PTB - ERROR: SYNCHRONIZATION FAILURE ! -----

[Ian-Max-Andolina]

I have tried everything on my RTX-2070 and T1000 systems, including all the recommended optimisations (low-latency kernel, clean install, gamemode priority etc.), you can see some of this in the following issue: https://github.com/kleinerm/Psychtoolbox-3/issues/168

One thing that was a reliable observation — the first PTB run in a fresh MATLAB invocation normally returns sync stddev just below the threshold, but from the next run on the sync testing always fails. Does this happen for you? I never managed to solve this.

My solution is to only use AMD cards with the open-source drivers for experiment data (where I never get sync errors), and then just test if I’m running on NVidia and increase the sync tolerance as my NVidia machines are only development systems:

gpuinfo = opengl('data');
if IsLinux && ~isempty(regexpi(gpuinfo.Vendor,'NVIDIA','ONCE'))
   Screen('Preference','SyncTestSettings', 0.001); %up to 1ms std variability
end

[mariokleiner]

Fwiw, since two months i have a NVidia GeForce GTX 1650 (lowest end/lowest performance/lowest power consumption) Turing class gpu in a AMD Ryzen 5 2400 G 499 Euro PC under Ubuntu 20.04-LTS with Linux 5.4-lowlatency and NVidia driver 440.100 - the same drivers Natalia has.

This one works mostly fine timing-wise on a 2nd X-Screen, also on Matlab R2019a, also on repeated use. So that driver works on at least one Ubuntu version with at least one display on one type of PC hardware.

But yeah, using NVidia with the proprietary drivers is a matter of luck. If it works great, great. If it wouldn’t work on my machine i would just be as helpless as anybody else who is trapped with proprietary graphics drivers.

Ian’s solution would normally cause PTB to miss any broken timing, so not useful for real data collection on legacy operating systems. On Linux it may be a little bit less dangerous, as PTB should be at least able to detect and warn about some problems it would not be able to detect on other OS’es.

Still, AMD or Intel hw is much more robust thanks to the open-source drivers.

@natalia one thing you could try if you wanted to stick to the NVidia for data collection is if the open-source nouveau driver works well enough for your needs. Timing and timing precision should be far more robust. Performance however will be probably about 5% of the proprietary driver in demanding stimulation scenarios on a GeForce 800/900/1000/1600/2000 card.

But then i think your types of visual stimuli as far as i remember are not super taxing on the gpu, so it might work well enough.

Also, the low-latency kernel is not used, otherwise this line

PTB-INFO: OS support status: Linux 5.3.0-61-generic Supported.

would read -lowlatency instead of -generic. Try first if that helps. You’d need to install:

sudo apt install linux-low latency-hwe-18.04 and reboot.``

I assume this is a dual-X-Screen setup created via XOrgConfCreator, but only screen 1 is used for visual stimulation?

If you wanted to try the standard open-source nouveau driver, you’d need to first run XOrgConfSelector and select 0 and logout+login again to disable the dual-X-Screen configuration, then switch to the open-source driver and reboot, then rerun XOrgConfCreator and XOrgConfSelector to create a new config file for dual-x-screen, as NVidia’s proprietary driver uses a config file format incompatible with the open-source driver and trying to run on nouveau with a config file created for NVidia would end badly - probably with a dark display.

Driver switching:

1. In a terminal launch: sudo software-properties-gtk OR run that app from the GUI.

2. Go to tab “Additional Drivers”

3. Select “Using X.Org X server – Nouveau …”

4. “Apply changes” or restart or whatever it wants. A reboot is needed in any case.

5. After a reboot you hopefully still have a working display, now running under nouveau instead of the proprietary driver.

If you buy new hardware, prefer Intel onboard (only on Linux for low to medium performance needs, on Windows it sucks badly), or AMD for best performance and feature set (at least on Linux, on Windows the results are mixed and a bit a matter of luck, as any other choice wrt. timing).

-mario

[natalia]

We have now installed linux-low latency-hwe-18.04 (by the way, originally we followed the official ptb instructions and installed sudo apt install linux-lowlatency , which apparently did not work for some reason ).

Of course, it did not help. Nvidia cards work fine on two other Linux machines we have, but with this one, we happen to have bad luck. Although using nouveau is an option for me, the current machine is a shared resource, so to accommodate potentially more demanding experiments and ensure excellent timing we decided to get an AMD card.

Thanks Ian and Mario for comments/suggestions.

Best,
Natalia

[mariokleiner]

@natalia

We have now installed linux-low latency-hwe-18.04 (by the way, originally we followed the official ptb instructions and installed sudo apt install linux-lowlatency , which apparently did not work for some reason ).

That’s because of kernel version numbers. If you install a fresh Ubuntu LTS of version number x.04.y with y == 0 or 1, e.g., Ubuntu 18.04.0 LTS or Ubuntu 18.04.1 LTS, or if you upgrade to Ubuntu x.04.y from an earlier version, e.g., from Ubuntu 17.10 to 18.04.something, then the standard hardware stack from Ubuntu x.04.0, e.g., Ubuntu 18.04.0 LTS will be used, as that is the conservative choice. If you install Ubuntu x.04.y with y >= 2 from a fresh download from ubuntu.com, ie. Ubuntu 18.04.2 or 18.04.3 or 18.04.4 …, you will get the so called HWE (Hardware Enablement) stack. The HWE brings a more modern version of the Linux kernel, X display server and OpenGL graphics libraries, with new improvements and up to date device drivers for the latest hardware.

This HWE will get updated every six months to newer versions taken from intermediate Ubuntu versions (18.04.2 LTS got the kernel etc. from Ubuntu 18.10, 18.04.3 got the stuff from Ubuntu 19.04, 18.04.4 has the stuff from Ubuntu 19.10, 18.04.5 will get the stuff from Ubuntu 20.04.0 LTS), so you stay up to date with the latest low-level tech and drivers. The non-LTS Ubuntu variants basically serve as beta test versions for the next small HWE upgrade of the LTS versions. In that sense, the HWE is a less conservative choice, as in more modern, but faster moving. You can also trigger installation of the HWE yourself by installing the software packages with *-hwe-18.04 in their name. This leaves people with the choice if they want somewhat old but proven over time stuff in the “engine room” of their computer, or if they want the new and fancy, but less long-term tested.

So what happened is when you installed linux-lowlatency, it installed the low-latency variant of the Linux kernel originally shipping with Ubuntu 18.04.0 LTS – the conservative choice. That would be Linux 4.15-lowlatency. But as your original Ubuntu installation was probably something like 18.04.3 LTS or 18.04.4 LTS, it already came with the HWE stack, and so the standard kernel shipping with that would be the more modern Linux 5.3-generic. The bootloader picks the Linux kernel with the highest version number by default when booting the machine, and as 5.3-generic is a higher version number than 4.15-lowlatency, it started with the 5.3 generic kernel. So the lowlatency kernel was on there, but not picked by default at boot. Installing linux-lowlatency-hwe-18.04 otoh. will install the low latency variant of Linux 5.3, so now that one is picked as default kernel.

So the instructions on our website are a bit inaccurate, not explaining the whole hwe thing, but then if one explains too much, people don’t pay attention and drown in walls of text (like this one). Maybe i need to think of a way to automate this setup more if i find time to think about it…

@natalia

Of course, it did not help. Nvidia cards work fine on two other Linux machines we have, but with this one, we happen to have bad luck. Although using nouveau is an option for me, the current machine is a shared resource, so to accommodate potentially more demanding experiments and ensure excellent timing we decided to get an AMD card.

At the moment i’d probably recommend AMD Polaris or Vega family cards. Polaris (Radeon 500 series - Wikipedia) is the conservative choice - as this is currently the most well tested gpu with Psychtoolbox - i have two of these.

Vega (Radeon RX Vega series - Wikipedia) should work equally well with Psychtoolbox, and is faster and technically a bit more advanced, especially if one wants to use PTB’s variable refresh rate support with suitable FreeSync / DisplayPort adaptive sync displays for very fine-grained visual stimulus timing (Psychtoolbox-3 - VRRSupport). However, i never had the chance to actually test PTB with a Vega gpu.

Use of the latest generation Navi gpu’s (Radeon RX 5000 series - Wikipedia) is probably not the recommended safe choice yet, as new gpu’s tend to be less mature, with a higher potential for display driver bugs etc., and PTB’s low-level tricks do not currently work with those gpu’s, and i don’t know if or when i will implement support for these.

-mario

[natalia]

Just to follow up on this: we now got AMD Radeon Pro WX3100 , which works like magic. Thanks everyone for the help/advice.