Unable to save video with custom Gstreamer pipe

Programming Help
1

Dear Psychtoolbox Community,

I am trying to record and display video from a webcam at the same time with Psychtoolbox.

With the Logitech C920 webcams I have, for good resolution and frame rate (e.g. 1280 x 720 at 30 fps) I need to query the device for MJPG format (as verified with v4l2-ctl). I can do so using ffmpeg or gstreamer directly (see example code later). With default Psychtoolbox video capture behavior (using e.g. VideoRecordingDemo) the device returns lower fps, seemingly corresponding to its limits with its default YUYV format.

I can plug in a custom Gstreamer pipe for Screen which gives me the expected quality and frame rate (example code below) but then I am unable to save the video, I can only display it. A file with the specified name is opened, but it always remains 0 bytes. What am I doing wrong?

System
Ubuntu 18.04.5 LTS
Intel Core i5-6500 CPU
Intel HD Graphics 530 (SKL GT2)
Gnome 3.28.2
GStreamer 1.14.5
Matlab 2017a
PsychtoolboxVersion
3.0.16 - Flavor: Debian package - psychtoolbox-3 (3.0.16.20200326.dfsg1-1~nd18.04+1)

Minimal code to reproduce:

Insert the following lines to VideoRecordingDemo.m instead of line 227:

% Custom Gstreamer pipeline definition:
capturebinspec = 'v4l2src device=/dev/video0 ! jpegdec ! videoconvert';
Screen('SetVideoCaptureParameter', -1, sprintf('SetNextCaptureBinSpec=%s', capturebinspec));
grabber = Screen('OpenVideoCapture', win, -9, [0 0 640 480], [], [], [], codec, withsound, [], 8);

The demo seemingly works as expected but does not save out the video, leaves a 0 bytes file.

The output of VideoRecordingDemo.m with the custom Gstreamer pipe at debug verbosity is here:
customDemo_output

A more involved, but still minimal example to reproduce the problem is here:
minExample.m

Gstreamer code that both displays and saves the video (I would like to achieve similar behavior with Psychtoolbox):

gst-launch-1.0 -v v4l2src device=/dev/video0 \
    ! jpegdec \
    ! video/x-raw,width=1280,height=720,framerate=30/1 \
    ! tee name=t t. \
    ! queue ! xvimagesink sync=false t. \
    ! queue ! videoconvert ! x264enc tune=zerolatency ! h264parse \
    ! matroskamux ! filesink location='test.mkv' sync=false

What I tried and did not work:

  • Including / leaving out the ! videoconvert part from the gstreamer specification
  • Playing around with pixel depth values, recording flags, resolutions

Thank you for any advice,

adam boncz

2

Update on the problem:

(1) Earlier reported issue:
The problem seems related to this github issue that was closed due to inactivity:
https://github.com/Psychtoolbox-3/Psychtoolbox-3/issues/545
I do not observe the memory leak, but the problem with saving the file is the same.

(2) YUY2 format with custom Gstreamer pipe:
The problem does not seem to be related to the image format, as I see the same inability to save video with a custom Gstreamer pipe requesting YUY2 frames:

capturebinspec = 'v4l2src device=/dev/video0 ! video/x-raw,format=YUY2 ! videoconvert';
Screen('SetVideoCaptureParameter', -1, sprintf('SetNextCaptureBinSpec=%s', capturebinspec));

As with the MJPG format, I can see the live display but the target file remains 0 bytes at the end.

(3) Workarounds:

  • I can save the frames with matlab’s built-in VideoWriter. However, this is slow, somewhat reducing the frame rate.
  • I tried supplying a Gstreamer pipe with tee to Psychtoolbox, so that I could save the video from Gstreamer directly, but that is never treated as a valid capturebin input.
  • I suppose there could be an OpenGL-solution, saving the textures out efficiently with wrapped OpenGL commands, however, I have no idea yet how to do that…
3

Opened an issue on github for this question: https://github.com/Psychtoolbox-3/Psychtoolbox-3/issues/687

4

Hi,

given the history of this specific issue and what you tried already unsuccessfully, i expect this to be non-trivial enough that it will take me more than an hour to read up on what you posted, what was in that GitHub issue, have a look at our C code, experiment myself, etc. A proper solution could even take changes to the C code. I’m short on time this week, as you can guess from the non-response so far. Next week i’ll also be quite busy, with no spare cycles for any non-trivial free work, so only paid-for services have priority at the moment.

To give this any priority, i’d therefore ask you - or more your lab actually - to buy a “Community membership with priority support” from our new commercial services website:

Specifically this link takes you to the store for the right item:

We are still working out some kinks in the process, as this business model is very new - less than a week old, so once your lab bought that and you have the “Order Id” and license key, contrary to what the text on that site says do not post the license key here on the forum or GitHub issue tracker, but instead execute this script in Octave or Matlab:

https://raw.githubusercontent.com/Psychtoolbox-3/Psychtoolbox-3/master/Psychtoolbox/PsychDocumentation/PsychPaidSupportAndServices.m

The script will ask you for the order id and license key and generate a suitable support request token, which you can post here to prove you are eligible for priority support wrt. questions and bugs. As noted there, it may take until next week though until your key can be activated and your request dealt with.

Thanks for your support,
-mario

5

Hi Mario,

thank you for the response.

I fully understand. The new business model looks exciting, the lab will look into getting a membership.

best,

6

Just in case anyone runs into this problem: this issue has been addressed in the new (3.0.17) release (as usual, thanks to Mario).

7

Only for Octave 5.2 on Ubuntu 20.04 LTS and later so far. A fix release for Ubuntu 18.04 LTS Octave or Matlab will follow in a later beta.

Wrt. to thank you’s. What a nicer way to thank me than by providing post-hoc the authentication token you get by your lab buying a priority support membership for 150 Euros + VAT? The total work time to fix your problem was way more than 10 hours causing costs on our side exceeding 1000 Euros, so it is nice to get something back that is more useful than a thank you, e.g., a small contribution to those 1000 Euros.

Have a nice weekend,
-mario

8

I got the message the first time and agree with it. This is not my grant/project though - I asked the PI and the answer was a “no” for this year’s budget, but it is planned for next year. Will happily send you the token once we have it. Frankly, the lab should pay you stg. like a subscription in general, not just for support, after all, we don’t seem to have a problem with paying for e.g. matlab.

best,
adam

9

Good, say thanks to your PI for planning this. The “community membership with priority support” is mostly meant as a subscription with a bit of “fire insurance” added. The priority support / “fire insurance” component is just there to make it more easy for labs to justify it to their funding sources and admins. If everybody would make full use of the priority support contingent this would not be viable. So far the uptake is underwhelming with only 17 such items sold in 2 weeks. I just hope this is because we started this offering rather late in the year, possibly too late, when many labs may have already spent all their funding for 2020. Otherwise Psychtoolbox will continue to be severely underfunded and we have to think about much stricter measures to make this work. I’d prefer carrot like the last 16 years, instead of stick, but i can’t act as a massively underpaid and overworked one-person charity forever. The easiest cost cutting measure and life improvement would be likely to cut down support for macOS and Windows users though and focus even more on Linux users.

More to the point: The videocapture improvements are now released with the latest PTB beta for Matlab and older Linux distributions like Ubuntu 18.04 LTS as well.