Synchronous Audio-visual speech stimuli presentation

Programming Help
1

Hi everyone,

I am designing an experiment where I will present AV speech ( a video of a person narrating a story) in a background of a mixture of noise. I need to play the audio and the visual of the AV talker separately because I will manipulate the sound level of the audio in real-time (so, to my understanding can’t use the playmovie function to present the video
:().

I am trying to present each frame of the video on the screen with the audio file (audio of the visual + other noise) playing through PsychportAudio, however, the visual is going out of sync with its audio after like 5-6 seconds. The visual lags the audio.

Does anyone know how to solve this issue? I am attaching my code snippet here but if you have done something similar before, I will be very grateful if you could share your script or point me to a direction. I am using Psychtoolbox 3.0.19.16 in Matlab 2022a on a Windows 11 laptop.

%% Initialize

% Screen
Screen('Preference','SyncTestSettings' ,0.002,50,0.1,5);
[hz, outRect, win0] = open_screen(2)

% Initialize Sound
InitializePsychSound;
pahandle = PsychPortAudio('Open', deviceID,1, 2,Fs, 2);

% load audio file
[sound_mix,fs] = audioread('sound_file.wav');

% Give instructions
instructions;

%% Experiment trials
StartTime = GetSecs;
for tt = start_trial:totaltrials
   
    % add some delay
    goSecs = GetSecs+1+rand/2;
    
    % Open Movie
    moviefile = 'path_to_\video_file.mp4';
    v = VideoReader(moviefile); 
    frame1 = read(v,1);
    first_frame = 1;
    buffer_time = 1;
                  
    % Wait for jitter to be over
    WaitSecs('UntilTime',goSecs);
    
    % Take control over processor
    Priority(2);
    
    % Schedule sound playback for future time to sync with visual
    PsychPortAudio('FillBuffer', pahandle, sound_mix');
    PsychPortAudio('Start',pahandle,1,GetSecs+buffer_time);
   
    goSecs = GetSecs+buffer_time-(1/hz);
    
    %% Stimulus playback loop 
    tstart = GetSecs; 
    for jj = 1:v.NumFrames 

        % Read the frame
        frame = read(v,jj);
        % Make the texture
        tex=Screen('MakeTexture', win0, frame);
        
        % Draw the new texture to off-screen window:
        Screen('DrawTexture', win0, tex, [], outRect);
        
        % Wait for sound playback to flip the first frame
        if first_frame
            WaitSecs('UntilTime',goSecs);
        end
        
        % Update display:
        t=Screen('Flip', win0);
        
        % Measure when that happened re: estimate of auditory onset
        if first_frame
            first_frame = 0;
            v_lag(tt) = buffer_time-(t-tstart);
            disp(v_lag(tt))
        end
        
        % Show the frame again (monitor fs = 60 frame rate = 30 Hz)
        Screen('DrawTexture', win0, tex,[],outRect);
        Screen('Flip',win0);
        
        % Get rid of the texture
        Screen('Close',tex);
    end
    
    % stop sound playback
    PsychPortAudio('Stop',pahandle);
    
    % clear display:
    Screen('Flip', win0);
    
    % Wait for first frame to be over
    WaitSecs('UntilTime',goSecs);
    Priority(0);
        
end
%% Finish up and close experiment

Thanks!

2

You don’t want to be creating your video textures within the loop. You should pre-create them. It’s probably that the time it takes to do all that is longer than the frame rate. You can measure this with a few GetSecs calls interspersed within your loop. That’s the best way to debug timing errors, to see how long each step is taking.

Keith

3

I’d recommend you have a look at some of the movie demos include din Psychtoolbox. I think they will really help you understand things better.

At the moment there are numerous ways in which the code could be improved and it would take someone a bunch of time to go through everything.

Have a look at the demos included in PTB and I think things will be much clearer for you.

P