# Automatically invoke fragment shader in every flip

**URL:** <https://psychtoolbox.discourse.group/t/automatically-invoke-fragment-shader-in-every-flip/4435>\
**Category:** Programming Help\
**Created:** [May 19, 2022, 1:56pm UTC](https://psychtoolbox.discourse.group/t/automatically-invoke-fragment-shader-in-every-flip/4435 "2022-05-19T13:56:16Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![acschuetz](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@acschuetz](https://psychtoolbox.discourse.group/u/acschuetz)\
**Post date:** [May 19, 2022, 1:56pm UTC](https://psychtoolbox.discourse.group/t/automatically-invoke-fragment-shader-in-every-flip/4435/1 "2022-05-19T13:56:16Z")

</div>

I’d like to add a fragment shader to the processing pipeline so that it is executed automatically in each flip. I managed to run the fragment shader explicitly by specifying it as an argument during a call of Screen(‘DrawTexture’), but I failed to run it automatically in each flip.

To do so, I added the following task to PsychImaging.m:

```auto
floc = find(mystrcmp(reqs, 'EnableTetraShader'));
if ~isempty(floc)
    shader = LoadGLSLProgramFromFiles('TetraShaderTrans', 1);
    glUseProgram(shader);
    glUniform1i(glGetUniformLocation(shader, 'Image'), 0);
    glUniform1f(glGetUniformLocation(shader, 'Offset'), 640);
    glUseProgram(0);
    % Append our new shader and enable chain:
    if outputcount > 0
        % Need a bufferflip command:
        Screen('HookFunction', win, 'AppendBuiltin', 'FinalOutputFormattingBlit', 'Builtin:FlipFBOs', '');
    end    
    Screen('HookFunction', win, 'PrependShader', 'FinalOutputFormattingBlit', 'TetraShader', shader)
    Screen('HookFunction', win, 'Enable', 'FinalOutputFormattingBlit');    
    outputcount = outputcount + 1;
end

```

and I called the corresponding task before opening the screen window:

```auto
PsychImaging('PrepareConfiguration');
PsychImaging('AddTask', 'General', 'EnableTetraShader');
PsychImaging('OpenWindow', screenNumber, 127.5);

```

According to the output of Screen(‘HookFunction’, win, ‘DumpAll’), the fragment shader is registered, but it doesn’t seem to be doing anything during the flips:

```auto
Hook chain FinalOutputFormattingBlit is currently enabled.
Following hook slots are assigned to this hook-chain:
=====================================================
Slot 0: Id='TetraShader' : GLSL-Shader : id=1 , luttex1=0 , blitter=
=====================================================

```

Is there something wrong with setting up the hook chain?

Here is the code of the fragment shader:

```auto
/* TetraShaderTrans.frag.txt - Shader for tetrachromatic projector:
 *
 *
 */ 

#extension GL_ARB_texture_rectangle : enable

uniform sampler2DRect Image;
uniform float Offset;
/*out vec4 fragColor;*/

void main()
{
    /* Get default texel read position (x,y): x is column, y is row of image. */
    vec2 pos = gl_FragCoord.xy;
    
    /* transpose x and y*/
    pos = vec2(pos.y, pos.x);
    pos.x = 1080-pos.x;    

    int channel = int(floor((pos.y-0.5)/Offset));
    float start = mod(pos.y-0.5,Offset);    

    pos.y = (start)*3.0+0.5;
    vec4 redChannel = texture2DRect(Image,pos);
    pos.y = pos.y + 1.0;
    vec4 greenChannel = texture2DRect(Image,pos);
    pos.y = pos.y + 1.0;
    vec4 blueChannel = texture2DRect(Image,pos);    
        
    if (channel == 0) {
        gl_FragColor = vec4(redChannel.r, greenChannel.r, blueChannel.r, 1.0);
    }
    if (channel == 1) {
        gl_FragColor = vec4(redChannel.g, greenChannel.g, blueChannel.g, 1.0);
    }
    if (channel == 2) {
        gl_FragColor = vec4(redChannel.b, greenChannel.b, blueChannel.b, 1.0);
    }
    if (channel == 3) {
        gl_FragColor = vec4(redChannel.a, greenChannel.a, blueChannel.a, 1.0);
    }
}

```

Best,  
Alexander

---

<div class="post-metadata">

**Author:** ![mariokleiner](https://avatars.discourse-cdn.com/v4/letter/m/13edae/32.png) [@mariokleiner](https://psychtoolbox.discourse.group/u/mariokleiner)\
**Post date:** [May 23, 2022, 6:26pm UTC](https://psychtoolbox.discourse.group/t/automatically-invoke-fragment-shader-in-every-flip/4435/2 "2022-05-23T18:26:14Z")

</div>

I’d love to hear more detail about the purpose and setup of this, and possibly if it would be useful to integrate properly into PTB for the benefit of other users as well, if you’d want to contribute a pull request, after some potential cleanup? Sound like you try to drive some custom display with 4 color channels?

The setup code for the PostConfiguration() seems reasonable, but what seems to be missing is the setup code in the FinalizeConfiguration() subfunction in PsychImaging.m

Cfe. the handling of the FinalFormatting task in lines 3428-3432 for the kind of setup code that would be needed at a minimum for your shader. Otherwise the pipeline stages you intend to use will be bypassed during ‘Flip’ processing.

Might be that using …

`PsychImaging('AddTask', 'FinalFormatting', 'EnableTetraShader');`

…would do the trick, ie. replacing ‘General’ in ‘AddTask’ by ‘FinalFormatting’.

We’d appreciate some financial contribution for this advice as described at:  
[https://www.psychtoolbox.net/#service](https://www.psychtoolbox.net/#service)

-mario

---

<div class="post-metadata">

**Author:** ![acschuetz](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@acschuetz](https://psychtoolbox.discourse.group/u/acschuetz)\
**Post date:** [May 24, 2022, 7:51am UTC](https://psychtoolbox.discourse.group/t/automatically-invoke-fragment-shader-in-every-flip/4435/3 "2022-05-24T07:51:12Z")

</div>

> [@mariokleiner](#):
>
> PsychImaging(‘AddTask’, ‘FinalFormatting’, ‘EnableTetraShader’);

Thanks a lot, that solved the problem! You are right, the purpose is to drive a custom 4-channel display that receives a 2560x1080x3 input to show a 1920x1080x4 image. The four channels are arranged in four columns across the larger horizontal window and the three RGB channels. The shader is for our convenience so that we can use the regular screen dimensions and the alpha channel as the fourth primary when drawing stimuli. I’m not sure if other people will need that functionality, so it’s up to you if you want to include it officially. Anyway, I just bought a support membership.

What is a bit strange though is that I have to transpose the coordinates only when the shader is directly invoked during the DrawTexture command, but not when it is automatically invoked during the flip. I added an additional parameter to the shader:

```auto
/* TetraShader.frag.txt - Shader for tetrachromatic projector:
 * Converts fragment coordinates from (h,v,4) to (h/3*4,v,3)
 * TransposeCoordinates: Vertical screen size to transpose x and y (necessary when directly applied to DrawTexture command)
 * 0 to keep x and y (necessary when applied at flip)
 * HorizontalOffset: Applied to split the image horizontally in four columns. Should correspond to screen width/3.
 *
 */ 

#extension GL_ARB_texture_rectangle : enable

uniform sampler2DRect Image;
uniform int TransposeCoordinates;
uniform float HorizontalOffset;

void main()
{
    int channel;
    vec4 redChannel;
    vec4 greenChannel;
    vec4 blueChannel;

    /* Get default texel read position (x,y): x is column, y is row of image. */
    vec2 pos = gl_FragCoord.xy;
    
    /* Transpose x and y*/
    if (TransposeCoordinates > 0) {
        pos = vec2(pos.y, pos.x);
        pos.x = TransposeCoordinates-pos.x;    

        channel = int(floor((pos.y-0.5)/HorizontalOffset));
        float start = mod(pos.y-0.5,HorizontalOffset);    

        pos.y = (start)*3.0+0.5;
        redChannel = texture2DRect(Image,pos);
        pos.y = pos.y + 1.0;
        greenChannel = texture2DRect(Image,pos);
        pos.y = pos.y + 1.0;
        blueChannel = texture2DRect(Image,pos);    
    } 
    /* Keep x and y */
    if (TransposeCoordinates == 0) {
        channel = int(floor((pos.x-0.5)/HorizontalOffset));
        float start = mod(pos.x-0.5,HorizontalOffset);    

        pos.x = (start)*3.0+0.5;
        redChannel = texture2DRect(Image,pos);
        pos.x = pos.x + 1.0;
        greenChannel = texture2DRect(Image,pos);
        pos.x = pos.x + 1.0;
        blueChannel = texture2DRect(Image,pos);    
    }        

    if (channel == 0) {
        gl_FragColor = vec4(redChannel.r, greenChannel.r, blueChannel.r, 1.0);
    }
    if (channel == 1) {
        gl_FragColor = vec4(redChannel.g, greenChannel.g, blueChannel.g, 1.0);
    }
    if (channel == 2) {
        gl_FragColor = vec4(redChannel.b, greenChannel.b, blueChannel.b, 1.0);
    }
    if (channel == 3) {
        gl_FragColor = vec4(redChannel.a, greenChannel.a, blueChannel.a, 1.0);
    }
}

```

Thanks,  
Alexander

---

<div class="post-metadata">

**Author:** ![mariokleiner](https://avatars.discourse-cdn.com/v4/letter/m/13edae/32.png) [@mariokleiner](https://psychtoolbox.discourse.group/u/mariokleiner)\
**Post date:** [May 24, 2022, 6:09pm UTC](https://psychtoolbox.discourse.group/t/automatically-invoke-fragment-shader-in-every-flip/4435/4 "2022-05-24T18:09:13Z")

</div>

> [@acschuetz](#):
>
> > [@mariokleiner](#):
> >
> > PsychImaging(‘AddTask’, ‘FinalFormatting’, ‘EnableTetraShader’);
> 
> Thanks a lot, that solved the problem! You are right, the purpose is to drive a custom 4-channel display that receives a 2560x1080x3 input to show a 1920x1080x4 image. The four channels are arranged in four columns across the larger horizontal window and the three RGB channels. The shader is for our convenience so that we can use the regular screen dimensions and the alpha channel as the fourth primary when drawing stimuli. I’m not sure if other people will need that functionality, so it’s up to you if you want to include it officially. Anyway, I just bought a support membership.

I see, some customized DLP projector? 3 horizontally adjacent pixels in the stimulus image are packed into the three color channels of an output pixel, and the four color channels of the source pixels are put into corresponding output locations in the four horizontal 640 pixel wide stripes to drive the 4 color channels of the projector.

Btw. i may misremember, but as a little optimization, you could replace the four `if (channel == 0) ...` by one statement that uses `channel` to index into the four .r .g .b .a channels as the vec4’s can also be indexed as 4 element arrays if i remember correctly:

`gl_FragColor = vec4(redChannel[channel], greenChannel[channel], blueChannel[channel], 1.0);`

Shaders are usually more efficient if they can avoid a lot of branching instructions like if-else. Not sure if it matters in your case.

Another tip would be to specify the optional `fbOverrideRect` parameter of `PsychImaging('OpenWindow', ....., fbOverrideRect);` as `[0 0 1920 1080]`. This should make the window size appear as the true output size 1920x1080 of your projector and avoid weirdness or complications, e.g., when centering a stimulus image in a window, it will be properly centered instead of shifted, because PTB has the right idea about true output dimensions.

These are the kind of setup steps we’d implement automatically if we’d add proper support for such tetrachromatic display devices, also dealing with stereo display setups and such.

> [@](#):
>
> What is a bit strange though is that I have to transpose the coordinates only when the shader is directly invoked during the DrawTexture command, but not when it is automatically invoked during the flip. I added an additional parameter to the shader:

That’s because `Screen('MakeTexture', ...)` by default uploads texture images in a transposed layout that optimizes for fast texture creation, taking caching and memory access properties of typical cpu’s into account. Depending on image dimensions, processor type and processor memory cache hierarchy, it can speedup texture creation by more than 20x, especially on low end machines with smaller caches, ymmv. PTB’s drawing functions know how to deal with this. For simplicity, you could simply specify the optional `textureOrientation` parameter of MakeTexture as 1 (see `Screen MakeTexture?` builtin help). Then the transpose in your shader should not be needed, as the transpose will be done efficiently on the gpu at MakeTexture time, simplifying your shader.

-mario

---

<div class="post-metadata">

**Author:** ![acschuetz](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@acschuetz](https://psychtoolbox.discourse.group/u/acschuetz)\
**Post date:** [May 27, 2022, 7:14am UTC](https://psychtoolbox.discourse.group/t/automatically-invoke-fragment-shader-in-every-flip/4435/5 "2022-05-27T07:14:16Z")

</div>

> [@mariokleiner](#):
>
> I see, some customized DLP projector? 3 horizontally adjacent pixels in the stimulus image are packed into the three color channels of an output pixel, and the four color channels of the source pixels are put into corresponding output locations in the four horizontal 640 pixel wide stripes to drive the 4 color channels of the projector.

Yes, that’s correct.

> [@mariokleiner](#):
>
> Another tip would be to specify the optional `fbOverrideRect` parameter of `PsychImaging('OpenWindow', ....., fbOverrideRect);` as `[0 0 1920 1080]` . This should make the window size appear as the true output size 1920x1080 of your projector and avoid weirdness or complications, e.g., when centering a stimulus image in a window, it will be properly centered instead of shifted, because PTB has the right idea about true output dimensions.

I tried that, but then the fourth column at pixels from 1920 to 2560 doesn’t contain any image. I think the fragment shader has to go through the full resolution of 2560x3 pixels and remap that to the internal 1920x4 format. If the screen width is only 1920 it will never reach the fourth column.

Thanks for the advice about optimizing the shader and the call for `Screen('MakeTexture'...)`, that worked well.

---

<div class="post-metadata">

**Author:** ![mariokleiner](https://avatars.discourse-cdn.com/v4/letter/m/13edae/32.png) [@mariokleiner](https://psychtoolbox.discourse.group/u/mariokleiner)\
**Post date:** [May 27, 2022, 6:24pm UTC](https://psychtoolbox.discourse.group/t/automatically-invoke-fragment-shader-in-every-flip/4435/6 "2022-05-27T18:24:30Z")

</div>

Ah, that was the wrong parameter. I think what should work better than specifying `fbOverrideRect` is this:

In ‘OpenWindow’, specify the optional `imagingmode` parameter as `kPsychNeedClientRectNoFitter` and the `clientRect` parameter as `[0 0 1920 1080]`

Yet another more elegant/high-level way would be to use a custom panelfitter task, as explained in the corresponding PsychImaging help section (cfe. PanelFitterDemo.m for other more common uses):

`PsychImaging('AddTask', 'General', 'UsePanelFitter', [1920, 1080], 'Custom', [0 0 1920 1080], [0 0 1920 1080]);`

But that does the same as the first low-level proposal, just with more processing overhead for this special case where source and destination region are identical, probably adding up to one millisecond of processing time. Beauty takes its time…

**Edit: Note that this additional setup for convenience will make direct use of the shader with MakeTexture + DrawTexture impossible though, as DrawTexture won’t have the full “real” framebuffer to draw to anymore. Otoh., I assume the automatic ‘Flip’ method is what you wanted for all uses anyway, so it may not matter.**

In principle all this kind of setup for use cases is what PsychImaging is meant to do, so it would make sense to integrate all this stuff properly into some future PTB release. But for that I’d need your permission to use all your code as a starting point for refinement - that you donate your code under PTB’s standard MIT license for inclusion. Also, more details about the specific projector and setup. I guess integrating only makes much sense if that hardware setup is easy enough to replicate for other labs, so upstream integration would benefit n \> 1 labs.

-mario