Automatically invoke fragment shader in every flip

Programming Help
1

I’d like to add a fragment shader to the processing pipeline so that it is executed automatically in each flip. I managed to run the fragment shader explicitly by specifying it as an argument during a call of Screen(‘DrawTexture’), but I failed to run it automatically in each flip.

To do so, I added the following task to PsychImaging.m:

floc = find(mystrcmp(reqs, 'EnableTetraShader'));
if ~isempty(floc)
    shader = LoadGLSLProgramFromFiles('TetraShaderTrans', 1);
    glUseProgram(shader);
    glUniform1i(glGetUniformLocation(shader, 'Image'), 0);
    glUniform1f(glGetUniformLocation(shader, 'Offset'), 640);
    glUseProgram(0);
    % Append our new shader and enable chain:
    if outputcount > 0
        % Need a bufferflip command:
        Screen('HookFunction', win, 'AppendBuiltin', 'FinalOutputFormattingBlit', 'Builtin:FlipFBOs', '');
    end    
    Screen('HookFunction', win, 'PrependShader', 'FinalOutputFormattingBlit', 'TetraShader', shader)
    Screen('HookFunction', win, 'Enable', 'FinalOutputFormattingBlit');    
    outputcount = outputcount + 1;
end

and I called the corresponding task before opening the screen window:

PsychImaging('PrepareConfiguration');
PsychImaging('AddTask', 'General', 'EnableTetraShader');
PsychImaging('OpenWindow', screenNumber, 127.5);

According to the output of Screen(‘HookFunction’, win, ‘DumpAll’), the fragment shader is registered, but it doesn’t seem to be doing anything during the flips:

Hook chain FinalOutputFormattingBlit is currently enabled.
Following hook slots are assigned to this hook-chain:
=====================================================
Slot 0: Id='TetraShader' : GLSL-Shader      : id=1 , luttex1=0 , blitter=
=====================================================

Is there something wrong with setting up the hook chain?

Here is the code of the fragment shader:

/* TetraShaderTrans.frag.txt - Shader for tetrachromatic projector:
 *
 *
 */ 

#extension GL_ARB_texture_rectangle : enable

uniform sampler2DRect Image;
uniform float Offset;
/*out vec4 fragColor;*/

void main()
{
    /* Get default texel read position (x,y): x is column, y is row of image. */
    vec2 pos = gl_FragCoord.xy;
    
    /* transpose x and y*/
    pos = vec2(pos.y, pos.x);
    pos.x = 1080-pos.x;    

    int channel = int(floor((pos.y-0.5)/Offset));
    float start = mod(pos.y-0.5,Offset);    

    pos.y = (start)*3.0+0.5;
    vec4 redChannel = texture2DRect(Image,pos);
    pos.y = pos.y + 1.0;
    vec4 greenChannel = texture2DRect(Image,pos);
    pos.y = pos.y + 1.0;
    vec4 blueChannel = texture2DRect(Image,pos);    
        
    if (channel == 0) {
        gl_FragColor = vec4(redChannel.r, greenChannel.r, blueChannel.r, 1.0);
    }
    if (channel == 1) {
        gl_FragColor = vec4(redChannel.g, greenChannel.g, blueChannel.g, 1.0);
    }
    if (channel == 2) {
        gl_FragColor = vec4(redChannel.b, greenChannel.b, blueChannel.b, 1.0);
    }
    if (channel == 3) {
        gl_FragColor = vec4(redChannel.a, greenChannel.a, blueChannel.a, 1.0);
    }
}

Best,
Alexander

2

I’d love to hear more detail about the purpose and setup of this, and possibly if it would be useful to integrate properly into PTB for the benefit of other users as well, if you’d want to contribute a pull request, after some potential cleanup? Sound like you try to drive some custom display with 4 color channels?

The setup code for the PostConfiguration() seems reasonable, but what seems to be missing is the setup code in the FinalizeConfiguration() subfunction in PsychImaging.m

Cfe. the handling of the FinalFormatting task in lines 3428-3432 for the kind of setup code that would be needed at a minimum for your shader. Otherwise the pipeline stages you intend to use will be bypassed during ‘Flip’ processing.

Might be that using …

PsychImaging('AddTask', 'FinalFormatting', 'EnableTetraShader');

…would do the trick, ie. replacing ‘General’ in ‘AddTask’ by ‘FinalFormatting’.

We’d appreciate some financial contribution for this advice as described at:
https://www.psychtoolbox.net/#service

-mario

3

Thanks a lot, that solved the problem! You are right, the purpose is to drive a custom 4-channel display that receives a 2560x1080x3 input to show a 1920x1080x4 image. The four channels are arranged in four columns across the larger horizontal window and the three RGB channels. The shader is for our convenience so that we can use the regular screen dimensions and the alpha channel as the fourth primary when drawing stimuli. I’m not sure if other people will need that functionality, so it’s up to you if you want to include it officially. Anyway, I just bought a support membership.

What is a bit strange though is that I have to transpose the coordinates only when the shader is directly invoked during the DrawTexture command, but not when it is automatically invoked during the flip. I added an additional parameter to the shader:

/* TetraShader.frag.txt - Shader for tetrachromatic projector:
 *      Converts fragment coordinates from (h,v,4) to (h/3*4,v,3)
 *          TransposeCoordinates: Vertical screen size to transpose x and y (necessary when directly applied to DrawTexture command)
 *                                0 to keep x and y (necessary when applied at flip)
 *          HorizontalOffset: Applied to split the image horizontally in four columns. Should correspond to screen width/3.
 *
 */ 

#extension GL_ARB_texture_rectangle : enable

uniform sampler2DRect Image;
uniform int TransposeCoordinates;
uniform float HorizontalOffset;

void main()
{
    int channel;
    vec4 redChannel;
    vec4 greenChannel;
    vec4 blueChannel;

    /* Get default texel read position (x,y): x is column, y is row of image. */
    vec2 pos = gl_FragCoord.xy;
    
    /* Transpose x and y*/
    if (TransposeCoordinates > 0) {
        pos = vec2(pos.y, pos.x);
        pos.x = TransposeCoordinates-pos.x;    

        channel = int(floor((pos.y-0.5)/HorizontalOffset));
        float start = mod(pos.y-0.5,HorizontalOffset);    

        pos.y = (start)*3.0+0.5;
        redChannel = texture2DRect(Image,pos);
        pos.y = pos.y + 1.0;
        greenChannel = texture2DRect(Image,pos);
        pos.y = pos.y + 1.0;
        blueChannel = texture2DRect(Image,pos);    
    } 
    /* Keep x and y */
    if (TransposeCoordinates == 0) {
        channel = int(floor((pos.x-0.5)/HorizontalOffset));
        float start = mod(pos.x-0.5,HorizontalOffset);    

        pos.x = (start)*3.0+0.5;
        redChannel = texture2DRect(Image,pos);
        pos.x = pos.x + 1.0;
        greenChannel = texture2DRect(Image,pos);
        pos.x = pos.x + 1.0;
        blueChannel = texture2DRect(Image,pos);    
    }        

    if (channel == 0) {
        gl_FragColor = vec4(redChannel.r, greenChannel.r, blueChannel.r, 1.0);
    }
    if (channel == 1) {
        gl_FragColor = vec4(redChannel.g, greenChannel.g, blueChannel.g, 1.0);
    }
    if (channel == 2) {
        gl_FragColor = vec4(redChannel.b, greenChannel.b, blueChannel.b, 1.0);
    }
    if (channel == 3) {
        gl_FragColor = vec4(redChannel.a, greenChannel.a, blueChannel.a, 1.0);
    }
}

Thanks,
Alexander

4

I see, some customized DLP projector? 3 horizontally adjacent pixels in the stimulus image are packed into the three color channels of an output pixel, and the four color channels of the source pixels are put into corresponding output locations in the four horizontal 640 pixel wide stripes to drive the 4 color channels of the projector.

Btw. i may misremember, but as a little optimization, you could replace the four if (channel == 0) ... by one statement that uses channel to index into the four .r .g .b .a channels as the vec4’s can also be indexed as 4 element arrays if i remember correctly:

gl_FragColor = vec4(redChannel[channel], greenChannel[channel], blueChannel[channel], 1.0);

Shaders are usually more efficient if they can avoid a lot of branching instructions like if-else. Not sure if it matters in your case.

Another tip would be to specify the optional fbOverrideRect parameter of PsychImaging('OpenWindow', ....., fbOverrideRect); as [0 0 1920 1080]. This should make the window size appear as the true output size 1920x1080 of your projector and avoid weirdness or complications, e.g., when centering a stimulus image in a window, it will be properly centered instead of shifted, because PTB has the right idea about true output dimensions.

These are the kind of setup steps we’d implement automatically if we’d add proper support for such tetrachromatic display devices, also dealing with stereo display setups and such.

That’s because Screen('MakeTexture', ...) by default uploads texture images in a transposed layout that optimizes for fast texture creation, taking caching and memory access properties of typical cpu’s into account. Depending on image dimensions, processor type and processor memory cache hierarchy, it can speedup texture creation by more than 20x, especially on low end machines with smaller caches, ymmv. PTB’s drawing functions know how to deal with this. For simplicity, you could simply specify the optional textureOrientation parameter of MakeTexture as 1 (see Screen MakeTexture? builtin help). Then the transpose in your shader should not be needed, as the transpose will be done efficiently on the gpu at MakeTexture time, simplifying your shader.

-mario

5

Yes, that’s correct.

I tried that, but then the fourth column at pixels from 1920 to 2560 doesn’t contain any image. I think the fragment shader has to go through the full resolution of 2560x3 pixels and remap that to the internal 1920x4 format. If the screen width is only 1920 it will never reach the fourth column.

Thanks for the advice about optimizing the shader and the call for Screen('MakeTexture'...), that worked well.

6

Ah, that was the wrong parameter. I think what should work better than specifying fbOverrideRect is this:

In ‘OpenWindow’, specify the optional imagingmode parameter as kPsychNeedClientRectNoFitter and the clientRect parameter as [0 0 1920 1080]

Yet another more elegant/high-level way would be to use a custom panelfitter task, as explained in the corresponding PsychImaging help section (cfe. PanelFitterDemo.m for other more common uses):

PsychImaging('AddTask', 'General', 'UsePanelFitter', [1920, 1080], 'Custom', [0 0 1920 1080], [0 0 1920 1080]);

But that does the same as the first low-level proposal, just with more processing overhead for this special case where source and destination region are identical, probably adding up to one millisecond of processing time. Beauty takes its time…

Edit: Note that this additional setup for convenience will make direct use of the shader with MakeTexture + DrawTexture impossible though, as DrawTexture won’t have the full “real” framebuffer to draw to anymore. Otoh., I assume the automatic ‘Flip’ method is what you wanted for all uses anyway, so it may not matter.

In principle all this kind of setup for use cases is what PsychImaging is meant to do, so it would make sense to integrate all this stuff properly into some future PTB release. But for that I’d need your permission to use all your code as a starting point for refinement - that you donate your code under PTB’s standard MIT license for inclusion. Also, more details about the specific projector and setup. I guess integrating only makes much sense if that hardware setup is easy enough to replicate for other labs, so upstream integration would benefit n > 1 labs.

-mario