How to "properly" use pipeline to apply spatial (x, y) offsets to dual stereo displays (haploscope)

Programming Help
,
1

Problem Description

I’m developing experiments for haploscopes using stereomode 4. Experiment, software, and timing in general working well, but I’ve been applying per monitor (left, right) spatial calibration offsets (x, y) to individual draw commands. As you can imagine, doing so adds some complexity to the code and I think we will hit some limits using this approach in the future with more sophisticated spatial calibration methods or advanced stimulus presentation. In the words of Raymond Hettinger, “there must be a better way!”

I’ve attempted to search the FAQs and forums, primarily in the programming help section (e.g. 4435), but have had mixed success in finding relevant posts. I’ve also been reading over PsychImaging.m, SetCompressedStereoSideBySideParameters, SCREENHookFunctions.c and various demos. While I didn’t see an existing task for applying spatial offsets (x,y), I did find a task ‘FlipHorizontal’ (and other examples) that use “Builtin:IdentityBlit”.

Using this information, I created a naive proof of concept (github gist) that visually appears to provide the results I’m after by hijacking the FlipHorizontal task to apply spatial offsets but I’m now trying to learn how to do this “properly” as attempts to do without hijacking FlipHorizontal have not succeeded.

I use FlipHorizontal when running on actual haploscope, but I do not use FlipHorizontal when on development machines. If I naively attempt to append a builtin blitter without creating the flip horizontal task, I don’t get the desired behavior (graphics render but no offsets applied) nor do I receive any errors or warnings. I’m assuming I’m missing a step during the PrepareConfiguration stage prior to OpenWindow.

Paid Support

I have included string per directions for requesting paid support for this issue.

F5P6CUU9-2022915141825:ecb9ef71f7312bddd20e36d47dc6401df4240acfe885c16b35caf2e6ae04549b

Hardware/Software Configuration

I have also attempted to include requested information per Psychtoolbox Forum page.

>> PsychtoolboxVersion

ans =

'3.0.18 - Flavor: beta - Corresponds to SVN Revision 12862

For more info visit:

https://github.com/Psychtoolbox-3/Psychtoolbox-3'
  • Develops on Windows 10 using various graphic configurations and understanding that we likely will not achieve frame accurate timing of stimulus.
  • Deploy to lab machines running Fedora 34 with AMD graphics per PTB guidance and regularly verify timing using oscope+photodetectors.
  • We try to keep the same MATLAB version on all development and deploy machines, at this time we are primarily running MATLAB R2021b.

Thank you all for the hard work on this project and for your help with my question.

Sincerely,

Matt Pare
Meta Reality Labs

2

By calibration methods, do you mean warping the stimuli in screen space per monitor (for example like you would do with old fashion CRT’s)?

If so, see

http://psychtoolbox.org/docs/ImageUndistortionDemo

and related functions.

P

4

Hi Peter, thank you for the response.

No, in our case we are applying simple X, Y offsets to each monitor. No warp at this stage.

In either case, I’m trying to learn more about the hook processing chain.

-Matt

5

Ok, I think you just need to learn how to set up frustums then. Ping me an email if you like.

P

6

Hello Matt,

thanks for the paid support request.

Is the function SetStereoSideBySideParameters what you want to have? It allows runtime change of the spatial offsets in stereo modes 4 and 5.

(Uncommented) sample code in lines 243-250 of ImagingStereoDemo.m.

Best,
-mario

7

Hi Mario, thank you for your response. After inspecting, SetStereoSideBySideParameters, I think it is one of the demos that I used as inspiration for my posted example code. I’ll take a look at SetStereoSideBySideParameters again to see if I can use it to identify the error in my posted example code.

-Matt

8

Inspired by your question, I made a bunch of improvements to SetStereoSideBySideParameters.m and related PsychImaging.m and RemapMouse.m for an upcoming PTB release. You could already grab them from my GitHub:

https://raw.githubusercontent.com/kleinerm/Psychtoolbox-3/master/Psychtoolbox/PsychGLImageProcessing/SetStereoSideBySideParameters.m

This now also allows to specify spatial left/right stereo offsets in units of pixels instead of normalized units, by setting the new optional offsetUnit parameter as ‘pixels’, as that seems to be the only thing your example Gist code does differently from what SetStereoSideBySideParameters() provides by default, so this improved version should satisfy your needs?

Wrt. why you needed ‘FlipHorizontal’ task in your Gist example code: The various image buffers and the basic topology of the image processing graph for the imaging pipeline is built statically during PsychImaging(‘OpenWindow’,…), based on all the PsychImaging ‘AddTask’ statements and other requirements. Those ‘Hookfunction’ calls you used are used to dynamically add/remove/modify image processing slots - and to a more limited degree extra buffers for intermediate results - into places of the graph whose basic structure/topology remains fixed after ‘OpenWindow’.

If PsychImaging doesn’t know at OpenWindow time what you want it to do, it may optimize the parts of the processing graph or image buffers away that your Hookfunction call in your sample script would need, so they turn into no-ops or errors. A lot of logic inside PsychImaging.m and Screen’s C code is there to optimize away unneeded processing steps and especially gpu buffer copies, both to keep VRAM consumption low, but also to save gpu processing time/bandwidth during each Flip. The PsychImaging.m subfunction FinalizeConfiguration has most of the logic to translate user-provided task specifications/requirements into configuration flags for building the processing graph, with some more logic inside the ‘OpenWindow’ implementation inside PsychImaging.m.

Internally PsychImaging(‘Openwindow’) calls FinalizeConfiguration, does its thing, then calls Screen OpenWindow with proper config flags and settings, Screen internally (C code) builds the graph and does basic setup, then PsychImaging.m uses its internal PostConfiguration subfunction (helper functions called by PostConfiguration) to fill and configure all those Hookfunction slots with shaders, lookup tables, etc. to finalize the whole processing graph which will then later execute during each Screen(‘Flip’).

In your case: You use the StereoLeftCompositingBlit hook, which only exists if per-view image processing is requested by specifying the imagingMode flag kPsychNeedImageProcessing during Screen('OpenWindow', ..., imagingMode);. You can see this flag being added around PsychImaging.m lines 3385-3406 when PsychImaging detects the presence of a task that uses viewType ‘LeftView’, ‘RightView’, or ‘AllViews’. Your scripts request PsychImaging('AddTask', 'AllViews', 'FlipHorizontal'); triggers proper setup because of ‘AllViews’. A hack would have been in this case to pass in the following:
``

winParms = struct( …
screenId=0, …
color=0.5, …
rect=, …
pixelSize=, …
numberOfBuffers=, …
stereoMode=4, …
multisample=, …
imagingmode=kPsychNeedImageProcessing …
);

``

to manually specify an additional kPsychNeedImageProcessing flag, but this is not generally recommended.

In the code for SetStereoSideBySideParameters.m you see it using hookfunction StereoCompositingBlit instead, which applies in a later stage of the graph to merge together images from two image buffers (left/right eye) into one buffer.

Slide 38 in Psychtoolbox/PsychDocumentation/PTBTutorial-ECVP2013.pdf illustrates the rough topology of the processing graph for a stereomode 4. This is not a complete picture but the best we have at hand - from year 2013. Your script tries to plug into the “Left or Mono image processing chain”, which doesn’t exist without tasks of type ‘AllViews’ or ‘LeftView’ → no op.

By now you’ve used up the first 2 of the 10 support memberships you bought.

Best,
-mario

1 Like
9

Hi Mario,

Thank you so much Mario for the improvements and the detailed high level explanation! I can’t wait to put these enhancements and knowledge to work. Unfortunately I need to resolve a issue with font rendering for “small” height fonts first (I’ll create a separate post).

Apologies for my delayed response, I’ve been out of office for a few days.

Sincerely,
Matt

10

Hi Mario,

I wanted to follow up to let you know that the guidance to use hook StereoCompositingBlit instead of StereoLeftCompositingBlit, StereoRightCompositingBlit indeed helped me achieve my implementation goal. I also found the slide you were referring to that outlines the hook processing pipeline chain and understand better how and when things happen. Thank you so much!

For future me and for other readers, if you decide to use SetStereoSideBySideParameters() directly, you will need to manually include the screen width to account for the left image width when specifying rightOffset. Otherwise, the right image will not be visible or other unexpected results may occur. SetStereoSideBySideParameters() does this automatically if you allow it to use defaults.

Thanks again Mario, I really appreciate the spot on support!

-Matt