Screen latency high except on first call to screen('flip') for fresh onscreen window

Programming Help
1

My experiment involves realtime monitoring/processing of camera images coming in at 80fps, and the moment a certain feature is detected in an image (say image_frame1) some very basic shapes are printed to a display monitor (240 hz refresh rate).

Using these same cameras, I can measure the latency between when I send the screen(‘Flip’) command and when it actually appears on the display monitor.

Strangely, on a freshly opened onscreen window, the latency between finished execution of the Screen(‘Flip’) command and when things appear on the display monitor is ~4-5 milliseconds. However, on subsequent trials this latency jumps to ~18 milliseconds UNLESS I close and reopen a fresh onscreen window between each trial. Note that it does not continue to incrementally increase throughout the trials - it remains steady at an average of 18 milliseconds for the remaining trials.

This is important for my experiment as I need to have a total latency of 50 milliseconds or less, which would include:

  1. first detecting the onset of the phenomenon in the camera (4.8 milliseconds after onset)
  2. Grabbing two more images (at 80fps) to gather needed data (additional 25 milliseconds)
  3. doing computations on the image, drawing textures, and Screen(‘Flip’) (additional 10ms)
  4. stimulus appearing on the screen (5 milliseconds for first screen flip, 18ms for all subsequent screen flips)
    Altogether, if the display monitor latency is 5ms, everything works (~45ms total latency). However, a display monitor latency of 18-20 milliseconds is ~20% increase in total latency (~60ms total latency).

Does anyone know why something like this might happen? To be clear, why is the first screen(‘flip’) on a fresh window faster than subsequent flips? I have included some important details about my setup and experiment below.

SYSTEM DETAILS
OS: windows 10 enterprise (64 bit)
processor: Intel Core i7-8700K @3.70 GHz
ram: 16 GB
GPU: NVIDIA GeForce GTX 1070
Matlab version: 2020a
PTB version: 3.0.16
display monitor: AW2518Hf (240Hz, display with on-screen window for stimulus presentation)
side monitor: P2719h (60 Hz, no on-screen window)

CODE NOTES:

  • because I have two monitors attached and I’m only opening the onscreen window on the 240Hz monitor, I use Screen(‘Preference’, ‘SkipSyncTests’, 1);
    before calling
    Screen(‘OpenWindow’, 2, [0 0 0]);
    to open the new window
  • ALL textures are pre-drawn after first opening the window
  • the textures are simple circles or rings, premade using Screen(‘MakeTexture’,…) - they are MxNx4 matrices to define color and transparency
  • Screen(‘BlendFunction’,params.win,GL_SRC_ALPHA,GL_ONE_MINUS_SRC_ALPHA); - this is the function call I use to turn on the transparency features for my textures
  • after grabbing the last image from the cameras (image_frame1) and performing some computations on the image data (whose computation I am measuring) I only use Screen(‘DrawTexture’) commands to put the pre-made textures on the off-screen buffer
  • To flip to the screen, I am using: Screen(‘Flip’,params.win,0,0,2);
    specifically with the [dontsync] option set to 2, which displays the stimulus immediately without syncing to the screen refresh rate
  • between each trial there are 3 textures containing text that are made and drawn to the screen (I doubt that these piling up is causing the slow down, as the display monitor latency is ~18 milliseconds even on the second screen(‘flip’), and doesn’t increase even with 40-50 subsequent trials)
2

18ms is suspiciously much like the IFI at 60hz. If you run with only the 240Hz monitor attached, does it work then?

1 Like
3

Thanks for the quick help! Will give that a shot and see how it goes

4

You must not Screen(‘Preference’, ‘SkipSyncTests’, 1); if you care about any visual timing.
You must not specify dontsync=1 or dontsync=2 if you care about any visual timing.
That dontsync=2 flag meant that visual onset timing was totally undefined, and you had no way of finding out if completion of a flip happened when you thought it happened.

If your system doesn’t work properly and without errors or warnings under those conditions then it is broken and needs fixing.
Especially on Windows, multi-display setups are often broken or fragile timing-wise, so as Dee said, unplug every other monitor than the 240 Hz display monitor and then retry.
Also all the optimizations in help SyncTrouble, as needed.
Then you’ll see where you stand.

In general i strongly recommend use of Linux with non-NVidia graphics for timing-sensitive (multi-display) tasks.

-mario

5

Hi All,

Speaking of screen latency, how reliable are the outputs of the Screen(‘Flip’) function?

I am interested in the duration of the flip execution, and whether I missed presenting my stimuli at the next possible refresh cycle - e.g., Screen(‘Flip’, windowPtr, 0).

I am checking the VBLTimestamp, FlipTimestamp, and Missed.
I calculate FlipExecDur = FlipTimestamp - VBLTimestamp. I get values less than 0.001 (i.e., less than 1 ms). Is that typical?

For Missed, I rarely get values over 0. Am I right to assume that, when the Missed value is less than 0, my stimuli were presented at the next possible refresh cycle?

Thank you,
Natasa

6

PTB has the most sophisticated software timestamping mechanisms possible, far ahead of any other vision science software i have ever heard of. The timestamps relate to when a new 'Flip’ped stimulus image leaves the video output of your graphics card, ie. the top-leftmost 1st pixel. Psychtoolbox/PsychDocumentation/ has the ECVP2013 intro pdf with a basic explanation of how timing and timestamping should be used, the ECVP2010 pdf has a poster with more details, albeit a bit dated after almost 12 years.

However, your specific display device can add latency, depending on settings and type.

And how precise the “signal leaves video output” timestamps are depends on setup, operating system and graphics card, and specific operating system or graphics driver bugs and flaws, all not fixable by us.

The best choice for great timing is Linux (with AMD or Intel graphics strongly preferred, although NVidia is workable in most cases) due to built-in superior timing mechanisms.

Windows usually can work ok with single-display setups on AMD or NVidia, is badly broken with Intel graphics, problematic on multi-display or mixed DPI and various other scenarios.

macOS on Apple Silicon M1/M2 etc. is a disaster atm. and possibly for a long time to come. macOS on Intel Macs with AMD or Intel gpu’s and PsychtoolboxKernelDriver installed and properly configured usually works ok’ish on most machines, but there are outliers due to the numerous Apple macOS operating system bugs.

That would just give you an estimate of the duration of the vertical blank interval, typically less than 1 msec, but meaningless for your purpose.

VBLSyncTest.m demonstrates timed presentation and how to check timestamps at least on fixed refresh rate displays, ``VRRTest.m` shows the same for variable refresh rate mode, which is currently only fully supported on Linux + AMD.

If you have further questions, → help PsychPaidSupportAndServices will tell you how to buy a support membership for up to 30 minutes more of me answering questions.

-mario

1 Like
7

Hi Mario,

Super! Thank you so much for this comprehensive reply. I’ll check out the VBLSyncTest.m.

Best,
Natasa

8

The best way to thank me is for your lab to buy a yearly support membership, because that actually contributes to keeping the lights on for Psychtoolbox.

Best,
-mario

9

Hi!

I’m sorry, I am coming back to this issue.

Can one refresh cycle last longer (e.g., 3 ms longer) than the prescribed refresh rate and the next cycle be shorter (e.g., 3 ms shorter)? This is what I am getting from the [VBLTimestamp] = Screen(‘Flip’, windowPtr).

I am confused about this - I thought the refresh rate was “hardwired”.

Thank you,
Natasa

UPDATE: I’ve just re-read your previous reply. You say that the timestamps are related to when the signal leaves the video output of the graphics card so these variations are possible.

10

No, only update timing of the display itself could see some content dependent variation, but that would only be detectable by suitable photo-sensors etc.

The timestamps reported by Flip would fluctuate exactly like you described if high-precision timestamping wouldn’t work / is miconfigured / disabled due to skipsynctests settings or sync failures, so you see OS scheduling noise and other random delays and variations. E.g., on a Apple macOS IntelMac machine without properly installed/configured/working PsychtoolboxKernelDriver.