Movie playback w/GStreamer - timing problem

1
Hi All --

I have a problem with timing of video playback. I'm running my code on a 27" iMac (10.8.3), with MATLAB 2012b, the latest version of psychtoolbox (3.0.10, 3713), and GStreamer (0.10.36).

For some reason, there's an occasional delay of a half a second between the time the first frame of the video is displayed and the second. It always occurs at this moment in the video playback, and the delay is always a half a second, regardless of whether I'm running a 4 minute video or a 12 minute video. It's very intermittent, as in maybe once every 5 runs. And, it usually (but not always) comes the first time I run the code. I'm running a *.mov file.

Anyone have any suggestions, or experienced a similar delay?


Also, possibly unrelated to the above issue, I've installed the PsychtoolboxKernelDriver, and get what I thought was the confirmation that the driver is being used:

PTB-INFO: Connection to Psychtoolbox kernel support driver instance #0 (Revision 0) established.
PTB-INFO: Connection to Psychtoolbox kernel support driver instance #1 (Revision 0) established.

But, it still gives me the warning:

"Deficient Apple OS/X 10.7 or later detected: Would use more fragile CoreVideo timestamping instead of precise vbl-irq timestamping as fallback. Installation of the PsychtoolboxKernelDriver is strongly recommended if you care about precise visual onset timestamping or timing. See 'help PsychtoolboxKernelDriver' for instructions."

Does this mean that it is or isn't using the driver?

Thanks very much,
Dan
2
--- In psychtoolbox@yahoogroups.com, "dkennedyiu" <dkennedy80@...> wrote:
>
> Hi All --
>
> I have a problem with timing of video playback. I'm running my code on a 27" iMac (10.8.3), with MATLAB 2012b, the latest version of psychtoolbox (3.0.10, 3713), and GStreamer (0.10.36).
>
> For some reason, there's an occasional delay of a half a second between the time the first frame of the video is displayed and the second. It always occurs at this moment in the video playback, and the delay is always a half a second, regardless of whether I'm running a 4 minute video or a 12 minute video. It's very intermittent, as in maybe once every 5 runs. And, it usually (but not always) comes the first time I run the code. I'm running a *.mov file.
>
> Anyone have any suggestions, or experienced a similar delay?
>

Enough free memory installed in the machine?

Is it possibly related to the specific movie file? Sometimes it looks random, but just because trials / movie files are randomized.

Does it happen with our demos?

Is this movies with or withound sound?

One way that always works without sound is setting the optional 'async' flag in Screen('OpenMovie', ...) to 4 and possibly set 'preloadSecs' to a value > 1 second. This changes the buffering behaviour and may help if your system has occassional timing glitches. E.g., starting playback, then waiting a second or so, then entering the actual playback loop. This will allow the system to initialize and prebuffer a few 'preloadSecs' seconds of video to compensate for system performance glitches.

For short movies it can also make sense to preload all movie frames into textures, like in LoadMovieIntoTexturesDemo, to have full control over display timing.

With sound is a bit more problematic. Higher 'preloadSecs' may help, the 'async' flag 4 may help, but could cause audio-video sync problems, depending on the movie files encoding, if there's too much delay between Screen('Playmovie') and the first Screen('GetMovieImage'), because audio might start before the first video frame is presented.

The performance behaviour often depends on the type of movie encoding used, keyframe placement etc. Proper movie encoding for a given task is more an art than a science.

>
> Also, possibly unrelated to the above issue, I've installed the PsychtoolboxKernelDriver, and get what I thought was the confirmation that the driver is being used:
>
> PTB-INFO: Connection to Psychtoolbox kernel support driver instance #0 (Revision 0) established.
> PTB-INFO: Connection to Psychtoolbox kernel support driver instance #1 (Revision 0) established.
>
> But, it still gives me the warning:
>
> "Deficient Apple OS/X 10.7 or later detected: Would use more fragile CoreVideo timestamping instead of precise vbl-irq timestamping as fallback. Installation of the PsychtoolboxKernelDriver is strongly recommended if you care about precise visual onset timestamping or timing. See 'help PsychtoolboxKernelDriver' for instructions."
>
> Does this mean that it is or isn't using the driver?
>

It's using the driver if it reports no warnings wrt. to unsupported or somehow broken beamposition queries afterwards, which would trigger the fallback method. The warning is a boilerplate, always shown on 10.7 or later to tell you what would happen if it couldn't use the driver. At the time the warning is printed, the toolbox doesn't yet know if the kernel driver will work properly for your display setup. Maybe i should scrap the warning, that OSX is deficient for demanding graphics apps is not really news to anyone anymore and may just be confusing.

-mario
3
--- In psychtoolbox@yahoogroups.com, Matthew Edmundson <medmunds@...> wrote:
>
> "This sounds more like just startup stutter. If you "clear all" and thereby
> "clear Screen" etc., you force a complete reinitialization of the playback
> engine for each movie. There can be delays every time a function is called
> the first time during execution of a script, especially after a clear.
> Similar for movie playback, e.g., decoder threads need to be started,
> libraries/codecs loaded, memory buffers must be allocated, etc. Just add a
> warmup trial at the beginning of your session, maybe playing some movie
> clip of a few seconds before the real session starts and see if the problem
> goes away."
>
> Thanks for this. I may give this warmup video workaround a shot if we get
> more situations with startup stutter occurring.
>

It's not really a workaround but best practice, not only for movie playback but in general. If Matlab runs through a script the first time, it will be slower, because stuff needs to be read from harddisc, which takes non-deterministic time, m-files need to be possibly partially jit-compiled during their first loop iterations, mex files need to get all their required support libraries loaded and linked, etc.

Warmup trials are not only for your subjects, but also for the equipment.

>
> "This last video queue is only there with async flag 4. Other queues are
> available before that final queue, but how many, where, and how exactly (if
> at all) the preloadSecs flag affects the capacity of that queue is
> dependent on the video and audio codecs, demultiplexers etc. - It depends
> both on the exact format and encoding of your movies and even a bit on the
> operating system and potentially hardware in use."
>
> Can you give some examples of codecs and/or container formats that you know
> from your work in the past have filled this queue, even if it was just on
> Linux or Mac OS X? Maybe you've used H.264 contained in MP4? What software
> would you have used to encode these videos?
>

I seldomly encode movies myself. I use test movies provided by users in the past, or generated by our own movie writing code, as shown in ImagingStereoDemo if the 'writemovie' flag is set, or movies recorded by our VideoRecordingDemo. I also test with movie trailers, or video created with Quicktime-Pro under osx.

When encoding with ptb's built in movie writer, default codec and muxer settings are chosen to facilitate use for psychophysics. The problem here is that the actual set of available codecs and muxers depends on your GStreamer installation and varies a lot across operating systems and GStreamer distributions, so the best settings may not work. On Linux you usually get a very complete set of codecs, OSX with the source build from Homebrew is ok iirc, as is the old OSSBuilds GStreamer for 32-Bit Matlab on Windows. The GStreamer SDK currently has the smallest set of supported encoders. Again, i mostly tested this on Linux, benchmarking such stuff is annoying enough on one OS, i didn't want to repeat this on the other OS'es and don't care too much about the other os'es for high performance applications anymore.

But mp4 container with H264 video and AAC audio is a common combination as a starting point. However, there are enough tunable parameters for that combo already to get you in trouble, or to spend time tinkering.

The last project i was involved in involved playback of stereoscopic HD 1920x1080 progressive scan video at sustained 60 fps, synchronized to the movements of a robotic motion simulator. The video material was in a ISO MP4 container, with MPEG-4 AAC audio and H264/AVC video. I wasn't involved in recording and encoding, but they used Handbrake ( http://http://handbrake.fr/ ) for encoding.

I think Tobias Wolf posted a few bis about encoding in the past. He uses ffmpeg directly iirc.

> If you aren't sure, I can do some tests myself, it just takes time to
> encode into all these different formats to test, so if I could start with a
> known quantity it would help.
>

LoadMovieIntoTexturesDemo has benchmark modes that allow to check how well different parts of the playback behave with different movies and settings.

> Like I said, I see no evidence of this queue filling up any significant
> amount of memory with Xvid AVIs on Windows. It could be that I'm just
> misusing the functions here, but I think it's more likely that this is
> something that's video format or OS specific.
>

I don't think i ever tested queueing behaviour a lot, beyond the very specific applications we needed. E.g., the application i mentioned used a preloadSecs of multiple seconds to buffer that amount, but it normally doesn't use the audio in the movie, so audio-video sync isn't a problem. Instead it loads the audio as a .wav file and plays it via PsychPortAudio, just making sure that audio and video playback start perfectly in sync. This causes audio and video to drift apart over time, but for trials of no more than maybe a minute or two, the error on our setup was still below ~ 40 msecs, good enough for the purpose.

As i said, the last queue is controlled via the async=4 flag. This is demonstrated e.g., in LoadMovieIntoTexturesDemo, but this is mostly unsuitable for audio+video playback. There are other queues in the playback pipeline, but their size depends on many factors. We essentially pass 'preloadSecs' as a high-level hint and then the specific pipeline somehow derives the sizes of the different queues from that and from other parameters. 'preloadSecs' for those latter queues is a misleading name, as the engine will not neccessarily prebuffer 'preloadSecs' seconds of data before playback starts. It is more like an upper limit of what can be buffered in certain situations, and it is also possible that certain codecs or older versions of GStreamer ignore this hint. E.g., the GStreamer from OSSBuilds that you have installed if you use 32-Bit Matlab on Windows is rather old and outdated, last updated over 2 years ago. Could be that it ignores the hint completely. Our source code says you need at least version v0.10.31 of the 'playbin2' plugin.

But i don't think that tweaking the buffering behaviour will gain you much for your approach. Where it might help is if you really play a movie in a loop for multiple interations, because then the 2nd iteration could use all the buffered data from the first iteration if it fits fully into memory. Or for random seeks in the material, once the whole video has been buffered. What you described in your other post doesn't sound as if the whole movie would get buffered, given your unconventional use of the loop flag. You'd only have the few seconds from your seek position to the end of the movie, not the following seconds from the start of the movie after it looped back to the beginning, so probably not much gain there.

-mario

> Best,
> Matthew
>
>
>
> On Thu, May 2, 2013 at 11:06 PM, Mario <mario.kleiner@...>wrote:
>
> > **
> >
> >
> >
> >
> > --- In psychtoolbox@yahoogroups.com, Matthew Edmundson <medmunds@>
> > wrote:
> > >
> > > Hi Mario,
> > >
> > > Not to hijack this thread, but I've also been trying to use background
> > > preloading of videos myself to see if it might help some performance
> > > problems we're having, but I can find no evidence that any preloading is
> > > actually occurring. I'm following the OpenMovie Docs page instructions,
> > > and according to it I'm supposed to open the movies with the async flag
> > set
> > > to 1 (or 2, or 4) and then wait a bit (say pause(10)), and then open them
> > > again with async flag set to 0 to get the pointers. I'm setting
> > preloadSecs
> > > to -1, but have also tried other values greater than 1.
> > >
> > > The problem is, I notice no increase in RAM usage on the system at all
> > > until the *second* OpenMovie is called with async flag set to 0, and even
> > > then it is only a very minor bump, say 6 MB per video. The videos I'm
> > > trying to preload are Xvids which are all larger than 20 MB, so this
> > > doesn't exactly make sense in terms of fully preloading the videos.
> > >
> >
> > What preloadSecs does on GStreamer is it sets the size of internal memory
> > buffers
> > meant for caching of data and also sets the maximum capacity of an
> > internal queue for decoded video frames if that async flag is set to
> > include the value 4 - an "or" of values.
> >
> > If your movies don't have sound then that async 4 setting will cause video
> > frames to get decoded and queued in an internal queue of up to
> > 'preloadSecs' seconds after 'PlayMovie' got called. Then each
> > 'GetMovieImage' call pulls the oldest decoded frame from that queue, so if
> > you gave it enough head start, it could pull frames from the queue and
> > survive stalls or slowdowns of the decoding process of up to preloadSecs
> > seconds. If you have sound and do the same it will work the same, but as
> > sound output would start almost immediately after 'PlayMovie', but you
> > would pause a bit before 'GetMovieImage' + 'DrawTexture' + 'Flip' you'd
> > have sound playback getting ahead of video playback.
> >
> > This last video queue is only there with async flag 4. Other queues are
> > available before that final queue, but how many, where, and how exactly (if
> > at all) the preloadSecs flag affects the capacity of that queue is
> > dependent on the video and audio codecs, demultiplexers etc. - It depends
> > both on the exact format and encoding of your movies and even a bit on the
> > operating system and potentially hardware in use.
> >
> >
> > > We're using Windows, btw.
> >
> > My condolence ;-).
> >
> > I mostly test and tune this stuff on Linux, and then some on OSX. Windows
> > gets only minimal effort, because trying to make pigs fly is so tiresome.
> >
> > The PlayMoviesDemo also has some optional flags 'pixelFormat' and
> > 'maxThreads' which allow optimizations if the video material is very taxing
> > for the system, e.g., very high resolution, high framerate, stereo etc. The
> > bottleneck always depends on the task, and clever choice of movie encoding
> > can boost performance a lot, but "clever" is not very well defined, a lot
> > of trial and error if you're not an expert in video formats.
> >
> > best,
> > -mario
> >
> > > Best,
> > > Matthew
> > >
> > >
> > > On Wed, May 1, 2013 at 12:16 PM, dkennedyiu <dkennedy80@> wrote:
> > >
> > > > **
> >
> > > >
> > > >
> > > > Hi Mario --
> > > >
> > > > Thanks very much for the reply.
> > > >
> > > > The videos I'm playing have sound.
> > > >
> > > > And, In terms of free memory, I have 32 GB of RAM, so that shouldn't be
> > > > the problem.
> > > >
> > > > I'm only playing 1 long-ish video at a time (~4 to 12 minutes, w/o
> > > > interruption), so there's no randomization. I'll investigate more to
> > see if
> > > > it's a problem specific to these specific video files. Also, I'll also
> > > > check whether I can reproduce the problem with your demos, and report
> > back.
> > > >
> > > > Here's one more interesting bit of info -- If I run my script in a
> > loop 50
> > > > times, I've noticed that the timing error only happens on the first
> > > > run-through of the video. Not in the next 49 runs. If I clear all,
> > clc, and
> > > > then rerun the loop, I might (but not always) again get the half-second
> > > > timing error on the first run and no where else. The weird thing is
> > that I
> > > > run the "clear all" command within the script at the end of each run.
> > So,
> > > > no idea what's happening here.
> > > >
> > > > And, the "Deficient Apple OS/X 10.7" warning is helpful, but I was just
> > > > confused since it didn't go away after installing the recommended
> > driver.
> > > > Thanks for clarifying!
> > > >
> > > > Dan
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>
4
"v0.10.7 is not the same as the version of the plugins. The 64-Bit Matlab ptb uses the GStreamer SDK, which is more modern. Unfortunately it has some shortcomings wrt. the old OSSBuilds version. Mostly it doesn't ship a set of plugins that is as complete as OSSBuilds. E.g., video capture and video recording is completely broken, whereas the OSSBuilds version works. This will be hopefully fixed in the next SDK release, whenever that will happen. It also lacks many high performance video encoders, e.g., x264enc for h264 encoding, so creation of movie files at least for high performance is pretty painful or impossible. On the plus side it allows multi-threaded decoding in theory, on the minus side that is highly ineffective under Windows. I hope that most problems will be fixed in the next SDK release, because the videocapture bugs are believed to be fixed and i submitted a working bug fix for the multi-threaded decoding, which may or may not get accepted for the SDK, depending if my solution is deemed elegant and future proof enough or not. Eventually i'd like to get rid of the OSSBuilds version, once the SDK is mostly equivalent in functionality."

This is very informative. We are using the 32-bit version of Matlab on Windows along with that OSSBuilds version. I'm drawing a blank as to why I kept us with the 32-bit version, because I did test both, so I'm going to have to revisit my notes. Maybe I should re-visit it. One thing to note is that we actually use Mathworks' Computer Vision System Toolbox in concert with the old-school Xvid VfW encoder to compress our videos. It might have been that there was a bug with the 64-bit side of MATLAB on that front, which made me decide to just keep it simple and stick with the 32-bit version across all parts? Our other programmer, back when we started, also might have not been aware of PsychToolbox's capabilities on the encoding front, or maybe the exact functionality that he wanted wasn't present back then. I'll have to run this by him.


"But here you have typical Windows problems which make this process slow and painful on Windows: a) The build environment is rather hostile to FOSS software originally developed and tuned for Linux. On Linux or OSX you could just swap out the GStreamer libraries for whatever suits your needs. On Windows only the specific libraries will work that i've chosen for building, so it is a all or nothing decision. b) Building the SDK from source is possible on Windows, but it is a long, involved and painful procedure, so nobody really wants to do this, certainly not me. This is why on Windows you'll always have to wait the longest for bug-fixes or improvements, you are mostly dependent on the schedules and decisions of the upstream developers. On OSX you have the choice between the SDK or you can get often more up to date stuff from Homebrew, compiled from source within half an hour or so. And on Linux it is even simpler and has the highest performance because that is the main development and testing platform."

Again, very informative. Thanks for this.


"I don't think much tech savyness is still needed nowadays. But switching from OSX to Windows for Psychtoolbox would be strange. I can't think of many things the Windows ptb would do better, but i can think of many things it did and does worse."

One reason we moved things from OSX to Windows was because as far as the encoding side of things, our video pre-editing is done with the Computer Vision System Toolbox, and Mathworks hasn't developed the ability for their function to pass the signal to an encoder on the Mac side, so you can only export uncompressed. Sure, we could've exported uncompressed videos and then encoded them using other software, but that's a hassle. We also could've possibly developed some sort of script or wrapper to do this for us (maybe even using the capabilities of PsychToolbox), but at the time, the functionality was already there on the Windows side of the Computer Vision System Toolbox, so we just went with it. Even though our custom playback package, on the other hand, doesn't require the Computer Vision System Toolbox, I didn't want to have us pre-editing our videos one platform, and then playing them back on another, so I argued for just doing it all on the Windows side. Given that most of our other video presentation and editing software is also on the Windows side, and our primary investigator and post-doc prefer Windows, I wanted to streamline things as much as possible for what they are used to. You may be right about the need for less tech savviness wrt. using Linux and PsychToolbox these days, but on the front of video codecs and container formats, there is only so much disparate workarounds that psychology researchers will take from their lab tech! As much as the Xvid encoder and the AVI container formats are outdated, these are a known quantity in our lab as far as our other software systems we use, so I wanted to use the same format for everything as much as was possible.


"With GStreamer it doesn't really preload, except when downloading from a network, e.g., video streaming. What it does is caching of data, so you could use it to get an effect similar to preloading. E.g., one thing that might work, but i never tested it, would be to play the whole movie once from the beginning to the end, with sound muted, discarding the video frames instead of displaying them, maybe running the playback at 10x the speed or so to speed up the process. This would fill the internal buffer caches etc. If you would then rewind and play normally and allocated enough buffer, then it would probably be able to skip some parts of the loading, parsing and demuxing of data. The pipeline would still need to run most of the decoding of the preparsed data, because buffering fully decoded data would use up way too much memory."

This is intriguing, and sounds somewhat similar to the way we are (ab)using the loop flag. Mind you, our method doesn't pre-parse the entire file beforehand, but it at least has all the videos playing in memory before the onset occurs, to a similar effect of having some of the buffer allocations already loaded. You've now made me consider the far out possibility that to avoid any looping idiosyncrasies we could if we had to just pre-edit in some blank space onto the beginning of all our videos, effectively doubling the length of each video so that we could pre-position the start point without having to loop at all. I'll have to continue this part of the discussion with you in the other thread as necessary, as you'd need to know more about our specific project to understand all the ins and outs of why this might or might not be useful.


"Another manual way of getting most of the preloading effect would be to make sure all movie files you use are cached in the operating systems file system cache before your session starts. You could simply fopen() the movie file and then fread() the whole file, fclose()'ing the file and discarding what you read. This would force the OS to load the whole file into RAM, where it would stay as long as you have plenty enough RAM installed. Or you install a SSD drive to minimize disc drive access delays."

Excellent tricks there. It hadn't occurred to me to essentially make a RAMdisk of all of our videos. :) Speaking of an SSD, we do have a smaller capacity one in the machine for the OS, but for sake of disk space, I've not been storing the stimuli on it. I could give that a shot to see if it makes any difference.

"Another thing you should be aware of is that when you say you "start multiple movies at the same time" in your other forum thread: With default settings, when the async flag is ~= 4, start of playback of a movie will actually happen the first time you call Screen('GetMovieImage'). Screen('PlayMovie') will do most of the preparations for a fast start of playback, but it won't start it. This is because otherwise a delay of more than a few milliseconds between 'PlayMovie' and the first 'GetMovieImage' call would already lead to dropped video frames and audio-video sync problems."

This might be critical. I don't think I accounted for this in my tests. I'll see what it does.

"The other problem is of course that you can't really start playback of all movies simultaneously. Starting each one will take a bit of time, so there will be a small offset between them from the start."

Yes, in fact, I've measured it. On our setup, the time between the first video and the last video displaying "simultaneously" is, if I recall, under 20 ms, and not perceptible to the naked eye at full speed. I have considered discussing with our other programmer about randomizing the order of our Screen function calls pertaining to which video is played first by MATLAB to counterbalance this disparity, but at the moment it's a low priority.

"In the (probably) next PTB release, Quicktime support will be completely removed, so with that compatibility obstacle out of the way we can take more advantage of GStreamer features in the future. But people like you should file feature requests with descriptions of their special use cases, so i know about special needs once i find time to work on improving our GStreamer support."

Will do. It's at the following link, correct?:



On Fri, May 10, 2013 at 8:11 PM, Mario <mario.kleiner@...> wrote:



--- In psychtoolbox@yahoogroups.com, Matthew Edmundson <medmunds@...> wrote:
> "It is more like an upper limit of what can be buffered in certain
> situations, and it is also possible that certain codecs or older versions
> of GStreamer ignore this hint. E.g., the GStreamer from OSSBuilds that you
> have installed if you use 32-Bit Matlab on Windows is rather old and
> outdated, last updated over 2 years ago. Could be that it ignores the hint
> completely. Our source code says you need at least version v0.10.31 of the
> 'playbin2' plugin."
>
> I think that probably answers it -- GStreamer from OSSBuilds is at v0.10.7.
> Luckily, we haven't so far hit the ceiling yet on what this older version
> can do for us, for our current needs, that is. Our investigators have been

v0.10.7 is not the same as the version of the plugins. The 64-Bit Matlab ptb uses the GStreamer SDK, which is more modern. Unfortunately it has some shortcomings wrt. the old OSSBuilds version. Mostly it doesn't ship a set of plugins that is as complete as OSSBuilds. E.g., video capture and video recording is completely broken, whereas the OSSBuilds version works. This will be hopefully fixed in the next SDK release, whenever that will happen. It also lacks many high performance video encoders, e.g., x264enc for h264 encoding, so creation of movie files at least for high performance is pretty painful or impossible. On the plus side it allows multi-threaded decoding in theory, on the minus side that is highly ineffective under Windows. I hope that most problems will be fixed in the next SDK release, because the videocapture bugs are believed to be fixed and i submitted a working bug fix for the multi-threaded decoding, which may or may not get accepted for the SDK, depending if my solution is deemed elegant and future proof enough or not. Eventually i'd like to get rid of the OSSBuilds version, once the SDK is mostly equivalent in functionality.

But here you have typical Windows problems which make this process slow and painful on Windows: a) The build environment is rather hostile to FOSS software originally developed and tuned for Linux. On Linux or OSX you could just swap out the GStreamer libraries for whatever suits your needs. On Windows only the specific libraries will work that i've chosen for building, so it is a all or nothing decision. b) Building the SDK from source is possible on Windows, but it is a long, involved and painful procedure, so nobody really wants to do this, certainly not me. This is why on Windows you'll always have to wait the longest for bug-fixes or improvements, you are mostly dependent on the schedules and decisions of the upstream developers. On OSX you have the choice between the SDK or you can get often more up to date stuff from Homebrew, compiled from source within half an hour or so. And on Linux it is even simpler and has the highest performance because that is the main development and testing platform.


> opposed to Linux (it is a relatively small lab that hasn't maintained
> consistent tech savviness over time, so it would have to be a significant
> functionality setback to cause them to make a long term decision to go that
> route.) As far as Mac OS is concerned, we initially began development on
> the Mac side, but ran into other design setbacks (some related to
> PsychToolbox, some unrelated) that caused us to move things to Windows.
>

I don't think much tech savyness is still needed nowadays. But switching from OSX to Windows for Psychtoolbox would be strange. I can't think of many things the Windows ptb would do better, but i can think of many things it did and does worse.


> "But i don't think that tweaking the buffering behaviour will gain you much
> for your approach. Where it might help is if you really play a movie in a
> loop for multiple interations, because then the 2nd iteration could use all
> the buffered data from the first iteration if it fits fully into memory."
>
> Yeah, our program would have to incorporate some sort of smart look-ahead
> buffering to future trials to account for the different movie files we use
> to benefit from that -- not within the scope for us at the moment, but I
> agree with your thinking.
>
> "Or for random seeks in the material, once the whole video has been
> buffered. What you described in your other post doesn't sound as if the
> whole movie would get buffered, given your unconventional use of the loop
> flag. You'd only have the few seconds from your seek position to the end of
> the movie, not the following seconds from the start of the movie after it
> looped back to the beginning, so probably not much gain there."
>
> I'll have to get back with you on this point. On first glance it sounds
> like a plausible description, but if my memory serves me correctly, in my
> tests I was calling preloadSecs -1 as well as the rest of the steps from
> the Docs all prior to any definition of 'SetMovieTimeIndex'. In other
> words, it should have buffered the entire video into the memory before it
> had any knowledge of where the program was going to be going with seeking
> or looping.
>

The point is that no preloading in the literal sense happens. 'preloadSecs' is a historical misnomer, 'bufferSecs' would be a better name now. In the past PTB used Apple's Quicktime for movie playback. With Quicktime 'preloadSecs' really controlled the amount of preloading before start of playback, as there was a specific Quicktime function to do that. In reality it never worked very well. Movies with soundtracks simply caused a hard crash of most Matlab versions under most OSX versions if you tried, and movies without sound tracks only worked occassionally for some specific video encodings. But at least the intention of the parameter was correct.

With GStreamer it doesn't really preload, except when downloading from a network, e.g., video streaming. What it does is caching of data, so you could use it to get an effect similar to preloading. E.g., one thing that might work, but i never tested it, would be to play the whole movie once from the beginning to the end, with sound muted, discarding the video frames instead of displaying them, maybe running the playback at 10x the speed or so to speed up the process. This would fill the internal buffer caches etc. If you would then rewind and play normally and allocated enough buffer, then it would probably be able to skip some parts of the loading, parsing and demuxing of data. The pipeline would still need to run most of the decoding of the preparsed data, because buffering fully decoded data would use up way too much memory. A few seconds of movie would be enough to consume a Gigabyte of RAM.

Here's some more background info:
<http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/chapter-buffering.html>

When you use the async=4 flag and start playback early, buffering of fully decoded video will happen, as explained before, but anything than preloading a few seconds this way would bring your system to a halt rather quickly due to the amount of consumed RAM.

And some GStreamer pipelines may do some real preloading, which may be effected by the 'preloadSecs' parameter, but there is no guarantee.

In most situations i think explicit preloading makes no sense, which is why it doesn't happen with PTB by default.

Another manual way of getting most of the preloading effect would be to make sure all movie files you use are cached in the operating systems file system cache before your session starts. You could simply fopen() the movie file and then fread() the whole file, fclose()'ing the file and discarding what you read. This would force the OS to load the whole file into RAM, where it would stay as long as you have plenty enough RAM installed.

Or you install a SSD drive to minimize disc drive access delays.

Another thing you should be aware of is that when you say you "start multiple movies at the same time" in your other forum thread: With default settings, when the async flag is ~= 4, start of playback of a movie will actually happen the first time you call Screen('GetMovieImage'). Screen('PlayMovie') will do most of the preparations for a fast start of playback, but it won't start it. This is because otherwise a delay of more than a few milliseconds between 'PlayMovie' and the first 'GetMovieImage' call would already lead to dropped video frames and audio-video sync problems.

The other problem is of course that you can't really start playback of all movies simultaneously. Starting each one will take a bit of time, so there will be a small offset between them from the start.

That's why controlling sound (with PsychPortAudio) and video playback separately would probably make sense for you. Then you could take more advantage of real internal video buffering via the async=4 flag to get stuff synchronized properly.

In the (probably) next PTB release, Quicktime support will be completely removed, so with that compatibility obstacle out of the way we can take more advantage of GStreamer features in the future. But people like you should file feature requests with descriptions of their special use cases, so i know about special needs once i find time to work on improving our GStreamer support.

-mario


> Thanks again for your detailed responses here.
>
> Best,
> Matthew
>
>
> On Thu, May 9, 2013 at 7:15 PM, Mario <mario.kleiner@...>wrote:

>
> > **
> >
> >
> >
> >
> > --- In psychtoolbox@yahoogroups.com, Matthew Edmundson <medmunds@>
> > wrote:
> > >
> > > "This sounds more like just startup stutter. If you "clear all" and
> > thereby
> > > "clear Screen" etc., you force a complete reinitialization of the
> > playback
> > > engine for each movie. There can be delays every time a function is
> > called
> > > the first time during execution of a script, especially after a clear.
> > > Similar for movie playback, e.g., decoder threads need to be started,
> > > libraries/codecs loaded, memory buffers must be allocated, etc. Just add
> > a
> > > warmup trial at the beginning of your session, maybe playing some movie
> > > clip of a few seconds before the real session starts and see if the
> > problem
> > > goes away."
> > >
> > > Thanks for this. I may give this warmup video workaround a shot if we get
> > > more situations with startup stutter occurring.
> > >
> >
> > It's not really a workaround but best practice, not only for movie
> > playback but in general. If Matlab runs through a script the first time, it
> > will be slower, because stuff needs to be read from harddisc, which takes
> > non-deterministic time, m-files need to be possibly partially jit-compiled
> > during their first loop iterations, mex files need to get all their
> > required support libraries loaded and linked, etc.
> >
> > Warmup trials are not only for your subjects, but also for the equipment.
> >
> >
> > >
> > > "This last video queue is only there with async flag 4. Other queues are
> > > available before that final queue, but how many, where, and how exactly
> > (if
> > > at all) the preloadSecs flag affects the capacity of that queue is
> > > dependent on the video and audio codecs, demultiplexers etc. - It depends
> > > both on the exact format and encoding of your movies and even a bit on
> > the
> > > operating system and potentially hardware in use."
> > >
> > > Can you give some examples of codecs and/or container formats that you
> > know
> > > from your work in the past have filled this queue, even if it was just on
> > > Linux or Mac OS X? Maybe you've used H.264 contained in MP4? What
> > software
> > > would you have used to encode these videos?
> > >
> >
> > I seldomly encode movies myself. I use test movies provided by users in
> > the past, or generated by our own movie writing code, as shown in
> > ImagingStereoDemo if the 'writemovie' flag is set, or movies recorded by
> > our VideoRecordingDemo. I also test with movie trailers, or video created
> > with Quicktime-Pro under osx.
> >
> > When encoding with ptb's built in movie writer, default codec and muxer
> > settings are chosen to facilitate use for psychophysics. The problem here
> > is that the actual set of available codecs and muxers depends on your
> > GStreamer installation and varies a lot across operating systems and
> > GStreamer distributions, so the best settings may not work. On Linux you
> > usually get a very complete set of codecs, OSX with the source build from
> > Homebrew is ok iirc, as is the old OSSBuilds GStreamer for 32-Bit Matlab on
> > Windows. The GStreamer SDK currently has the smallest set of supported
> > encoders. Again, i mostly tested this on Linux, benchmarking such stuff is
> > annoying enough on one OS, i didn't want to repeat this on the other OS'es
> > and don't care too much about the other os'es for high performance
> > applications anymore.
> >
> > But mp4 container with H264 video and AAC audio is a common combination as
> > a starting point. However, there are enough tunable parameters for that
> > combo already to get you in trouble, or to spend time tinkering.
> >
> > The last project i was involved in involved playback of stereoscopic HD
> > 1920x1080 progressive scan video at sustained 60 fps, synchronized to the
> > movements of a robotic motion simulator. The video material was in a ISO
> > MP4 container, with MPEG-4 AAC audio and H264/AVC video. I wasn't involved
> > in recording and encoding, but they used Handbrake (
> > http://http://handbrake.fr/ ) for encoding.
> >
> > I think Tobias Wolf posted a few bis about encoding in the past. He uses
> > ffmpeg directly iirc.
> >
> >
> > > If you aren't sure, I can do some tests myself, it just takes time to
> > > encode into all these different formats to test, so if I could start
> > with a
> > > known quantity it would help.
> > >
> >
> > LoadMovieIntoTexturesDemo has benchmark modes that allow to check how well
> > different parts of the playback behave with different movies and settings.
> >
> >
> > > Like I said, I see no evidence of this queue filling up any significant
> > > amount of memory with Xvid AVIs on Windows. It could be that I'm just
> > > misusing the functions here, but I think it's more likely that this is
> > > something that's video format or OS specific.
> > >
> >
> > I don't think i ever tested queueing behaviour a lot, beyond the very
> > specific applications we needed. E.g., the application i mentioned used a
> > preloadSecs of multiple seconds to buffer that amount, but it normally
> > doesn't use the audio in the movie, so audio-video sync isn't a problem.
> > Instead it loads the audio as a .wav file and plays it via PsychPortAudio,
> > just making sure that audio and video playback start perfectly in sync.
> > This causes audio and video to drift apart over time, but for trials of no
> > more than maybe a minute or two, the error on our setup was still below ~
> > 40 msecs, good enough for the purpose.
> >
> > As i said, the last queue is controlled via the async=4 flag. This is
> > demonstrated e.g., in LoadMovieIntoTexturesDemo, but this is mostly
> > unsuitable for audio+video playback. There are other queues in the playback
> > pipeline, but their size depends on many factors. We essentially pass
> > 'preloadSecs' as a high-level hint and then the specific pipeline somehow
> > derives the sizes of the different queues from that and from other
> > parameters. 'preloadSecs' for those latter queues is a misleading name, as
> > the engine will not neccessarily prebuffer 'preloadSecs' seconds of data
> > before playback starts. It is more like an upper limit of what can be
> > buffered in certain situations, and it is also possible that certain codecs
> > or older versions of GStreamer ignore this hint. E.g., the GStreamer from
> > OSSBuilds that you have installed if you use 32-Bit Matlab on Windows is
> > rather old and outdated, last updated over 2 years ago. Could be that it
> > ignores the hint completely. Our source code says you need at least version
> > v0.10.31 of the 'playbin2' plugin.
> >
> > But i don't think that tweaking the buffering behaviour will gain you much
> > for your approach. Where it might help is if you really play a movie in a
> > loop for multiple interations, because then the 2nd iteration could use all
> > the buffered data from the first iteration if it fits fully into memory. Or
> > for random seeks in the material, once the whole video has been buffered.
> > What you described in your other post doesn't sound as if the whole movie
> > would get buffered, given your unconventional use of the loop flag. You'd
> > only have the few seconds from your seek position to the end of the movie,
> > not the following seconds from the start of the movie after it looped back
> > to the beginning, so probably not much gain there.
> >
> > -mario
> >
> > > Best,
> > > Matthew
> > >
> > >
> > >
> > > On Thu, May 2, 2013 at 11:06 PM, Mario <mario.kleiner@>wrote:

> > >
> > > > **
> >
> > > >
> > > >
> > > >
> > > >
> > > > --- In psychtoolbox@yahoogroups.com, Matthew Edmundson <medmunds@>
> > > > wrote:
> > > > >
> > > > > Hi Mario,
> > > > >
> > > > > Not to hijack this thread, but I've also been trying to use
> > background
> > > > > preloading of videos myself to see if it might help some performance
> > > > > problems we're having, but I can find no evidence that any
> > preloading is
> > > > > actually occurring. I'm following the OpenMovie Docs page
> > instructions,
> > > > > and according to it I'm supposed to open the movies with the async
> > flag
> > > > set
> > > > > to 1 (or 2, or 4) and then wait a bit (say pause(10)), and then open
> > them
> > > > > again with async flag set to 0 to get the pointers. I'm setting
> > > > preloadSecs
> > > > > to -1, but have also tried other values greater than 1.
> > > > >
> > > > > The problem is, I notice no increase in RAM usage on the system at
> > all
> > > > > until the *second* OpenMovie is called with async flag set to 0, and
> > even
> > > > > then it is only a very minor bump, say 6 MB per video. The videos I'm
> > > > > trying to preload are Xvids which are all larger than 20 MB, so this
> > > > > doesn't exactly make sense in terms of fully preloading the videos.
> > > > >
> > > >
> > > > What preloadSecs does on GStreamer is it sets the size of internal
> > memory
> > > > buffers
> > > > meant for caching of data and also sets the maximum capacity of an
> > > > internal queue for decoded video frames if that async flag is set to
> > > > include the value 4 - an "or" of values.
> > > >
> > > > If your movies don't have sound then that async 4 setting will cause
> > video
> > > > frames to get decoded and queued in an internal queue of up to
> > > > 'preloadSecs' seconds after 'PlayMovie' got called. Then each
> > > > 'GetMovieImage' call pulls the oldest decoded frame from that queue,
> > so if
> > > > you gave it enough head start, it could pull frames from the queue and
> > > > survive stalls or slowdowns of the decoding process of up to
> > preloadSecs
> > > > seconds. If you have sound and do the same it will work the same, but
> > as
> > > > sound output would start almost immediately after 'PlayMovie', but you
> > > > would pause a bit before 'GetMovieImage' + 'DrawTexture' + 'Flip' you'd
> > > > have sound playback getting ahead of video playback.
> > > >
> > > > This last video queue is only there with async flag 4. Other queues are
> > > > available before that final queue, but how many, where, and how
> > exactly (if
> > > > at all) the preloadSecs flag affects the capacity of that queue is
> > > > dependent on the video and audio codecs, demultiplexers etc. - It
> > depends
> > > > both on the exact format and encoding of your movies and even a bit on
> > the
> > > > operating system and potentially hardware in use.
> > > >
> > > >
> > > > > We're using Windows, btw.
> > > >
> > > > My condolence ;-).
> > > >
> > > > I mostly test and tune this stuff on Linux, and then some on OSX.
> > Windows
> > > > gets only minimal effort, because trying to make pigs fly is so
> > tiresome.
> > > >
> > > > The PlayMoviesDemo also has some optional flags 'pixelFormat' and
> > > > 'maxThreads' which allow optimizations if the video material is very
> > taxing
> > > > for the system, e.g., very high resolution, high framerate, stereo
> > etc. The
> > > > bottleneck always depends on the task, and clever choice of movie
> > encoding
> > > > can boost performance a lot, but "clever" is not very well defined, a
> > lot
> > > > of trial and error if you're not an expert in video formats.
> > > >
> > > > best,
> > > > -mario
> > > >
> > > > > Best,
> > > > > Matthew
> > > > >
> > > > >
> > > > > On Wed, May 1, 2013 at 12:16 PM, dkennedyiu <dkennedy80@> wrote:
> > > > >
> > > > > > **
> > > >
> > > > > >
> > > > > >
> > > > > > Hi Mario --
> > > > > >
> > > > > > Thanks very much for the reply.
> > > > > >
> > > > > > The videos I'm playing have sound.
> > > > > >
> > > > > > And, In terms of free memory, I have 32 GB of RAM, so that
> > shouldn't be
> > > > > > the problem.
> > > > > >
> > > > > > I'm only playing 1 long-ish video at a time (~4 to 12 minutes, w/o
> > > > > > interruption), so there's no randomization. I'll investigate more
> > to
> > > > see if
> > > > > > it's a problem specific to these specific video files. Also, I'll
> > also
> > > > > > check whether I can reproduce the problem with your demos, and
> > report
> > > > back.
> > > > > >
> > > > > > Here's one more interesting bit of info -- If I run my script in a
> > > > loop 50
> > > > > > times, I've noticed that the timing error only happens on the first
> > > > > > run-through of the video. Not in the next 49 runs. If I clear all,
> > > > clc, and
> > > > > > then rerun the loop, I might (but not always) again get the
> > half-second
> > > > > > timing error on the first run and no where else. The weird thing is
> > > > that I
> > > > > > run the "clear all" command within the script at the end of each
> > run.
> > > > So,
> > > > > > no idea what's happening here.
> > > > > >
> > > > > > And, the "Deficient Apple OS/X 10.7" warning is helpful, but I was
> > just
> > > > > > confused since it didn't go away after installing the recommended
> > > > driver.
> > > > > > Thanks for clarifying!
> > > > > >
> > > > > > Dan
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>