Verify Priority level from system command line

1
Dear PsychToolbox Wizards,

Context:
- Ubuntu 14.04 with low-latency kernel 4.4.0
- Lots of processor cores and RAM , no more than about 15% of either in use
- 8 Matlab processes running in parallel , synched through c-language system calls
- Up to date Psych Toolbox and Matlab 2015b
- lubuntu-desktop environment for lightweight GUI
- Average 8MByte/s udp packets arriving at Gigabit ethernet card
- Ran PsychLinuxConfiguration , but prior to kernel 4.4.0 upgrade

I'm fighting with skipped frames. The number of skipped frames oscillates between none, and several per second, suggesting some kind of scheduling issue. I've used Priority( 1 ) in only one Matlab process, the one that Screen ( 'Flip<s>' , .. ) images to the test screen ; the change is verified by Priority with no input args.

However, then I can't seem to verify the priority change from the system command line. I've tried the 'top' command and 'ps -a -o pid,pri,cmd'. But neither of these reports a different priority for the Matlab process that actually ran Priority( 1 ). top says they all have a priority of 20, ps says they're all at 19.

It may not necessarily be process scheduling, as the kernel is busy dealing with a lot of incoming packets. On the other hand, reducing the network traffic doesn't seem to reduce skipped frames. Neither does reducing the number of parallel processes seem to help.

So, the question is how to properly check the priority of the Matlab process that has run Priority( 1 ), from the system command line. If I can do that properly, then I might be able to rule out a scheduling issue.

Best,
Jackson


2
Having looked into things further, I see that Priority( ) ultimately results in a call to pthread_setschedparam( ). So, realtime priority is given only to the PsychToolbox thread, while all other threads in the Matlab process retain a real-time priority of 0. They will be subject to process scheduling based on the nice value, I believe. Would a sensible approach then be to reduce the nice value (i.e. increase non-realtime priority) of the Matlab process that flips images to screen? With any luck, the actions taken by PsychLinuxConfiguration will have provided me with the permissions to do this.
3
Hi Mario,

That's great advice to get started on, thanks! I reckon a bit of fine tuning will get everything to work well. Skipped frames start to go away after a long burn-in period.

Q: So why do you run 8 Matlab processes and what do they do?

A: The idea is to have different Matlab-language "threads", so to speak, to run an ephys experiment. They all run via the ptb3-matlab command with -nodesktop, -nojvm seems to block beamposition time stamping. Their jobs are as follows...

Mat1 - User interface GUI, includes subject performance plots, neural peri-stimulus time histograms, neural response tuning curves, buffers incoming real-time data from all shared mem for end-of-trial analysis

Mat2 - Master Psych Toolbox, opens Psych Toolbox window and runs the animation loop, keeps track of the state of the trial, provides current stimulus locations via dedicated POSIX shared memory

Mat3 - Eye position capture from remote eye tracker, passes this into dedicated POSIX shared memory for use by other Matlabs

Mat4 - Targeting, compares eye positions versus stimulus positions, reads from shared mem

Mat5 - Spike time capture, and start/stop recording on remote data collection system. Writes spike times to dedicated POSIX shared mem.

Mat6 - USB-DAQ output, drives reward pump but also generates 'labelled' TTL pulses for synchronisation with remote data recording system ; by labelled, I mean a 16-bit unsigned integer that signals which event is occurring

Mat7 - Real-time eye position plot alongside stimulus locations, reads shared mem

Mat8 - Real-time spike event plot in register with trial events, reads shared mem

<parent process, compiled C> - Sets up all IPC, forks child Matlab processes, then syncs up their behaviour with Unix pipes, acts as a server that receives and broadcasts trial events in a hub and spoke network

So there's a fair number of read/write calls to the kernel. There may be some room for me to reduce the frequency of calls, which might free up some kernel time. For instance, I might buffer incoming data for a little longer and then write out bigger chunks of data to the shared memory less frequently ; we'd be talking about an extra kilobyte or two than it currently transfers, if that, so I expect that there would not be much time lost due to the additional data transferred.

Q: What gpu + driver?
A: Graphics card is the Nvidia Quadro K2200, using the latest proprietary driver available (375, I think). Did I mention that I open the Psych Toolbox window in stereo-mode 6 (red-green anaglyph)? There are two X-screens. Screen 0 provides the user interface and online plots through a DisplayPort link on a flatscreen monitor. Screen 1 hosts the Psych Toolbox window that shows the visual stimulus through a VGA link on a CRT monitor (85Hz refresh rate, 1600x1200 resolution).

Q: A first question would be if it happens with only one Matlab instance and without network traffic.
A: This deserves some more testing that it's received. I find that the type of stimulus can lead to more or less frame skips. Random dot stimuli run well, unless I have a large number of dots. Procedural Sine gratings cause skips ; I've tried reducing the texture support as much as I sensibly can, but I don't know that this made a difference. Again, it deserves more testing, but I believe that drawing to the frame buffer is all happening quickly.

Cheers,
Jackson
4
ps - Use Matlab's animated line object for eye positions. It's simple to use and seems quick enough.
5
XXXXXX---In PSYCHTOOLBOX@yahoogroups.com, <jsdpag@...> wrote :

Hi Zack,

Thanks for the tip re Nvidia driver. Maybe I should try downgrading as well. While I've not quantified it properly, I think there is some improvement from giving real time priority to Xorg, and dropping the Matlab nice value to -20 doesn't hurt, either.

-> I can't think of a reason of why dropping Matlab's nice value would help. If you increase the priority of Matlab threads which are irrelevant to PTB's functioning, you are increasing the competition for cpu resources for PTB's relevant threads. So if that really somehow helps, i'd like to see some measurement results. If you meant boosting the priority by lower nice values, but not making those threads actual realtime then maybe, who knows, at least it shouldn't hurt. With having cross process communication between the visual stimulation Matlab and the others, you could run into issues where one of the non-realtime Matlabs makes the rt one wait for data, at least if the pipe's run full.

Another thing you could try is to use octave for some of these tasks, as it is less resource intense than Matlab, and as far as my experience goes, it seems to be less problematic as well, especially for the latest Matlab versions. E.g., i found this note in my notes.txt file: "Matlab R2015a and R2016b: matlab -nojvm on Linux --> Doesn't enable the hw accelerated graphics drivers, but uses the Mesa software fallback!!! WTF?!?" - Maybe that related to your comment about not enabling PTB's high precision timestamping? Matlab is not involved in that at all, apart from increasingly being a hazard to well working setups on all platforms.

-> I also assume you set the performance level of the NVidia card to "maximum" in the control GUI, or via xorg.conf options, to avoid gpu power-management from optimizing for power savings instead of speed and low latency? There are certain graphics loads where a gpu can be underutilized (on average over a multiple second time window) at a performance level n+1, but a bit overutilized at level n, and then the drivers power management starts to toggle between level n and n+1 and you effectively lose performance or increase latency although the card would manage just fine if it stayed at level n+1.

-> All the plotting and gui activity on the other x-screens could slow down the rendering of ptb, as the gpu is a shared resource and you get competition and there is no priorization similar to what Priority() does for the cpu or main memory. Given that you chose a gpu from NVidia which requires proprietary drivers to function, there's not much one could do to advice you here, but all the plotting going on could be a large factor.

I would really look first into how one Matlab behaves, compared to multiple instances, as there is a lot potential for interference.

Another question is if you really need 8 Matlab instances for all these tasks? I don't know what makes sense for your setup, but some of the tasks sound a bit as if an extra Matlab is overkill? And each Matlab creates on the order of 21 threads even if it does nothing, so the logic "i have 8 cores, so 8 Matlab's should be optimal" doesn't really apply.

-mario

6
Also not fully understanding what you're up to, it does sound like you could do with significantly less matlabs, many have hardly anything to do. if you want minimum latency for things like getting eye position into the right process, you an simply do what everybody does with the eyelink, getlatestsample or read the whole queue. You gain nothing, but probably lose, by having a separate matlab pipe that to you. Whats the logic behind this?

Cheers,
Dee

On Tue, Oct 10, 2017 at 2:30 AM, mario.kleiner@... [PSYCHTOOLBOX] <PSYCHTOOLBOX@yahoogroups.com> wrote:


XXXXXX---In PSYCHTOOLBOX@yahoogroups.com, <jsdpag@...> wrote :

Hi Zack,

Thanks for the tip re Nvidia driver. Maybe I should try downgrading as well. While I've not quantified it properly, I think there is some improvement from giving real time priority to Xorg, and dropping the Matlab nice value to -20 doesn't hurt, either.

-> I can't think of a reason of why dropping Matlab's nice value would help. If you increase the priority of Matlab threads which are irrelevant to PTB's functioning, you are increasing the competition for cpu resources for PTB's relevant threads. So if that really somehow helps, i'd like to see some measurement results. If you meant boosting the priority by lower nice values, but not making those threads actual realtime then maybe, who knows, at least it shouldn't hurt. With having cross process communication between the visual stimulation Matlab and the others, you could run into issues where one of the non-realtime Matlabs makes the rt one wait for data, at least if the pipe's run full.

Another thing you could try is to use octave for some of these tasks, as it is less resource intense than Matlab, and as far as my experience goes, it seems to be less problematic as well, especially for the latest Matlab versions. E.g., i found this note in my notes.txt file: "Matlab R2015a and R2016b: matlab -nojvm on Linux --> Doesn't enable the hw accelerated graphics drivers, but uses the Mesa software fallback!!! WTF?!?" - Maybe that related to your comment about not enabling PTB's high precision timestamping? Matlab is not involved in that at all, apart from increasingly being a hazard to well working setups on all platforms.

-> I also assume you set the performance level of the NVidia card to "maximum" in the control GUI, or via xorg.conf options, to avoid gpu power-management from optimizing for power savings instead of speed and low latency? There are certain graphics loads where a gpu can be underutilized (on average over a multiple second time window) at a performance level n+1, but a bit overutilized at level n, and then the drivers power management starts to toggle between level n and n+1 and you effectively lose performance or increase latency although the card would manage just fine if it stayed at level n+1.

-> All the plotting and gui activity on the other x-screens could slow down the rendering of ptb, as the gpu is a shared resource and you get competition and there is no priorization similar to what Priority() does for the cpu or main memory. Given that you chose a gpu from NVidia which requires proprietary drivers to function, there's not much one could do to advice you here, but all the plotting going on could be a large factor.

I would really look first into how one Matlab behaves, compared to multiple instances, as there is a lot potential for interference.

Another question is if you really need 8 Matlab instances for all these tasks? I don't know what makes sense for your setup, but some of the tasks sound a bit as if an extra Matlab is overkill? And each Matlab creates on the order of 21 threads even if it does nothing, so the logic "i have 8 cores, so 8 Matlab's should be optimal" doesn't really apply.

-mario




7
Hi Mario,

I'm afraid that your hunch about Mathworks might just be right.

They say that "there was a change in R2013a which means that all graphics now depends on a layer of Java (https://www.mathworks.com/help/matlab/release-notes.html#btsurqv-6). As such, you will need Java enabled to render in MATLAB." The link says that "When you start MATLAB with the -nojvm startup option, Handle Graphics® functionality is not supported." I don't know how that affects linking the OpenGL libraries, but I will raise it with Mathworks.

And I'll start thinking hard about adding an Octave switch to my launch program. Does this mean that the Matlab PTB3 installation is treated separately on the local system from the Octave PTB3 installation, and that both can co-exist on the same system?

Thanks, as ever, for your help in troubleshooting.

Cheers,
Jackson
8
Hi Mario,

I'm afraid that your hunch about Mathworks might just be right.

They say that "there was a change in R2013a which means that all graphics now depends on a layer of Java (https://www.mathworks.com/help/matlab/release-notes.html#btsurqv-6). As such, you will need Java enabled to render in MATLAB." The link says that "When you start MATLAB with the -nojvm startup option, Handle Graphics® functionality is not supported." I don't know how that affects linking the OpenGL libraries, but I will raise it with Mathworks.

->  Badly, apparently. Quite an achievement to break this, must have been a lot of work, but with enough determination the sky's the limit.

And I'll start thinking hard about adding an Octave switch to my launch program. Does this mean that the Matlab PTB3 installation is treated separately on the local system from the Octave PTB3 installation, and that both can co-exist on the same system?

-> Yes.

-mario


Thanks, as ever, for your help in troubleshooting.

Cheers,
Jackson
9
Okay, I should have thought of this ages ago when you told me how to use top for viewing individual threads. I've monitored the single Matlab process that uses PTB for drawing and flipping, like this:

top -H -d 0.5 -p $p , where $p is the Matlab PID.

What I have observed is that when I get an animation freeze, it is not any Matlab thread that gobbles CPU time ... it is the set of PTB mainthreads. These guys can suddenly each shoot up to 100% CPU. But since there's over 20 PTB threads on the go, this makes the entire Matlab process eat up over 1000% of the CPU time. This will cause the current frame to skip, but the skip lasts for hundreds to thousands of milliseconds. Whether or not the threads have a CPU spike is unpredictable, but seems related to the number of dots I'm drawing. Some trials run smooth with no CPU spike, others do.

The current build I'm running is:

3.0.14 - Flavor: Debian package - psychtoolbox-3 (3.0.14.20170611.dfsg1-1~nd14.04+1)

The question is how to troubleshoot this. I've not previously seen freezing behaviour if I run a single, conventional ptb3-matlab session, but I'll run this again at a more demanding setting to see what happens. If using PTB functions in more than one Matlab process is the problem, then I will have to write my own mex wrapper functions to access timing and waiting system calls.

Cheers,
Jackson