Daq Toolbox and multiprocessing

1
Dear PTB Community,

Are the Daq Toolbox functions "thread-safe"?

The rationale, here, is that I need to perform both DAQ input and output operations in a timely but asynchronous manner. Rather than pile all the jobs together into a single PTB program, I would rather run simultaneous copies of Matlab, and give each copy a different program to run. In this case, one program would be dedicated to DAQ output, another would be dedicated to DAQ input, and a final program would draw the stimulus.

We are using Linux on a multi-processor, multi-core machine with one USB-1208fs. It is possible, then, that the DAQ could receive an input and output command at the same time. If the underlying implementation is thread-safe then this wouldn't matter. The operations would automatically queue up and occur as quickly as the device could handle them. If they are not thread-safe, then sad things would happen.

This leads to a broader question of the "thread-safety" of PsychToolbox. I would not allow more than one copy of Matlab to run Screen( ) commands, and the like. But sometimes PTB seems to do a bit of initialisation. And that may or may not be process-specific. Could one copy of PTB Matlab accidentally reset system or graphics settings that had been placed by another?

Cheers,
Jackson
2
Alright, I've managed to test a couple of simple things.

First, I tried to access the same USB-1208fs from two different instances of Matlab with DaqBlinkLED. The first Matlab to run the function caused the LED to blink. But the second Matlab returned an error saying that it could not access the DAQ. So, it seems that only one instance of Matlab/PsychToolbox can work with a given USB-DAQ at a time. I haven't tested whether different input or output functions could be accessed.

Instead, I plugged in a second USB-1208fs ... and DaqAInScan crashed. The reason is that it does not take into account which interface ID belongs to which device. This can be corrected by adding code to filter ID's by the device's serial number (as opposed to report serial numbers, which come from PsychHID).

To modify DaqAInScan that comes with PTB 3.0.12.429704077, do the following:

Line 441 - Define persistent variable to keep serial number

persistent  daqsno ;

Line 483 - Initialise DAQ serial number

daqsno = AllHIDDevices( daq ).serialNumber ;

Line 498 - Keep device indeces for next step

kk = strmatch ( SN , AllSNs ) ;
InterfaceInds = transpose ( kk ) ;

Line 499 - Filter interface IDs by DAQ serial number

kk = strcmp ( daqsno , { AllHIDDevices( kk ).serialNumber } ) ;
InterfaceInds = InterfaceInds ( kk ) ;

Line 536 - Add  elseif to if statement, to check serial number of daq

elseif  ( options.continue || options.end )  &&  ~strcmp ( daqsno , AllHIDDevices( daq ).serialNumber )

    error ( 'DAQ with serial number %s expected' , daqsno )

Once this is done, DaqAInScan will only use interface ID's from the device specified by the daq argument.

This allowed me to operate two different USB-1208fs devices from two different instances of Matlab/PTB on the same computer (Linux , don't know how this works otherwise). One Matlab read and plotted analogue input in real time ; the other sat there blinking the DAQ's LED , just because it could.

With this setup, I hope to devote one Matlab/DAQ pair to reading analogue input, and another to generating digital output signals.

Cheers,
Jackson

---In psychtoolbox@yahoogroups.com, <jsdpag@...> wrote :

Dear PTB Community,

Are the Daq Toolbox functions "thread-safe"?

The rationale, here, is that I need to perform both DAQ input and output operations in a timely but asynchronous manner. Rather than pile all the jobs together into a single PTB program, I would rather run simultaneous copies of Matlab, and give each copy a different program to run. In this case, one program would be dedicated to DAQ output, another would be dedicated to DAQ input, and a final program would draw the stimulus.

We are using Linux on a multi-processor, multi-core machine with one USB-1208fs. It is possible, then, that the DAQ could receive an input and output command at the same time. If the underlying implementation is thread-safe then this wouldn't matter. The operations would automatically queue up and occur as quickly as the device could handle them. If they are not thread-safe, then sad things would happen.

This leads to a broader question of the "thread-safety" of PsychToolbox. I would not allow more than one copy of Matlab to run Screen( ) commands, and the like. But sometimes PTB seems to do a bit of initialisation. And that may or may not be process-specific. Could one copy of PTB Matlab accidentally reset system or graphics settings that had been placed by another?

Cheers,
Jackson
3
Can you send the modified DaqAInScan.m, which i assume also includes the potential fixes to continuous scan mode? What you did sounds reasonable, but i can't test any of this without the required hardware and also won't have time to think about this for at least a week. But if the code changes make sense i could merge it into the next beta (all your contributions to PTB are automatically MIT licensed - that's the only way you can contribute, btw.)

I'd expect all OS'es to usually only allow exclusive access by one actor to a given USB device interface, so i'd expect to only one application at a time being able to access a given device, at least the way our access code in PsychHID is written atm.

"Thread-safety" btw. is not what you think it means. Thread safety means that some library or code etc. is safe against concurrent execution by different parallel processing threads *within* one single application, so it wouldn't apply to multiple Matlab instances. If you have multiple instances then the operating system and its device drivers are the arbiters of who or how many such instances can have shared or exclusive access to any given system resource. However, for low level USB devices like such DAQ boxes, which don't have a dedicated OS layer or driver to muitiplex concurrent access, it makes sense to only allow one actor per device interface or even just per device. How OS'es handle this is highly OS (and sometimes OS version) dependent, so the only safe assumption is 1 application per device.

In general you can run multiple PTB's in separate Matlab or Octave instances in parallel, if they don't access the same resources in some exclusive way. E.g., 1 PTB per physical soundcard, one PTB onscreen window per (X-)Screen, but you could have multiple fullscreen PTB windows on separate X-Screens, or multiple non-fullscreen windows wherever. Things like gamma settings will affect everything on one display though, access to special hardware is often exclusive like with the DAQ boxes...

-mario

---In PSYCHTOOLBOX@yahoogroups.com, <jsdpag@...> wrote :

Alright, I've managed to test a couple of simple things.

First, I tried to access the same USB-1208fs from two different instances of Matlab with DaqBlinkLED. The first Matlab to run the function caused the LED to blink. But the second Matlab returned an error saying that it could not access the DAQ. So, it seems that only one instance of Matlab/PsychToolbox can work with a given USB-DAQ at a time. I haven't tested whether different input or output functions could be accessed.

Instead, I plugged in a second USB-1208fs ... and DaqAInScan crashed. The reason is that it does not take into account which interface ID belongs to which device. This can be corrected by adding code to filter ID's by the device's serial number (as opposed to report serial numbers, which come from PsychHID).

To modify DaqAInScan that comes with PTB 3.0.12.429704077, do the following:

Line 441 - Define persistent variable to keep serial number

persistent  daqsno ;

Line 483 - Initialise DAQ serial number

daqsno = AllHIDDevices( daq ).serialNumber ;

Line 498 - Keep device indeces for next step

kk = strmatch ( SN , AllSNs ) ;
InterfaceInds = transpose ( kk ) ;

Line 499 - Filter interface IDs by DAQ serial number

kk = strcmp ( daqsno , { AllHIDDevices( kk ).serialNumber } ) ;
InterfaceInds = InterfaceInds ( kk ) ;

Line 536 - Add  elseif to if statement, to check serial number of daq

elseif  ( options.continue || options.end )  &&  ~strcmp ( daqsno , AllHIDDevices( daq ).serialNumber )

    error ( 'DAQ with serial number %s expected' , daqsno )

Once this is done, DaqAInScan will only use interface ID's from the device specified by the daq argument.

This allowed me to operate two different USB-1208fs devices from two different instances of Matlab/PTB on the same computer (Linux , don't know how this works otherwise). One Matlab read and plotted analogue input in real time ; the other sat there blinking the DAQ's LED , just because it could.

With this setup, I hope to devote one Matlab/DAQ pair to reading analogue input, and another to generating digital output signals.

Cheers,
Jackson

---In psychtoolbox@yahoogroups.com, <jsdpag@...> wrote :

Dear PTB Community,

Are the Daq Toolbox functions "thread-safe"?

The rationale, here, is that I need to perform both DAQ input and output operations in a timely but asynchronous manner. Rather than pile all the jobs together into a single PTB program, I would rather run simultaneous copies of Matlab, and give each copy a different program to run. In this case, one program would be dedicated to DAQ output, another would be dedicated to DAQ input, and a final program would draw the stimulus.

We are using Linux on a multi-processor, multi-core machine with one USB-1208fs. It is possible, then, that the DAQ could receive an input and output command at the same time. If the underlying implementation is thread-safe then this wouldn't matter. The operations would automatically queue up and occur as quickly as the device could handle them. If they are not thread-safe, then sad things would happen.

This leads to a broader question of the "thread-safety" of PsychToolbox. I would not allow more than one copy of Matlab to run Screen( ) commands, and the like. But sometimes PTB seems to do a bit of initialisation. And that may or may not be process-specific. Could one copy of PTB Matlab accidentally reset system or graphics settings that had been placed by another?

Cheers,
Jackson
4
Hi Mario,

Thanks, that all makes a lot of sense. I've attached my copy of DaqAInScan.m to this reply, since I don't yet know how to add to GitHub without messing that up on you. Changes are summed up as:

- Workaround for deprecated bitcmp
- Remembers Daq device serial number and uses its device IDs, if multiple devices connected
- Allows partial reads if checking for consecutive reports
- options.nodiscard to keep all data , skipping check for consecutive reports

In reply to your other post about integer types ( response to 19610 ), we tried both ways on Matlab R2012a. The answer using uint16 matched the outcome of the deprecated bitcmp. I can't say that I've done extensive testing, or worked out all the logic of this.

Cheers,
Jackson
5
ps - I've had to clean up another part of the function. The original DaqAInScan.m builds the output argument 'data' by looped concatenation when its size is known. It also fails to reshape params.times the same way as data, which is a pain if you read more than one input channel. The original code from lines 845 to 869 is:

% Return times.
params.times=[reports.time];
% Extract the data from the reports.
data=[];
for k=1:length(reports)
    data=[data reports(k).report(1:bytes)];
end

% Combine two bytes for each reading.
data = double(data(1:2:end))+double(data(2:2:end))*256;
% Discard any extra 16-bit words at the end of the last report.
if length(data)>c*options.count
    if 2*(length(data)-options.count*c)>60
        fprintf('Trimming off an extra %.1f sample/channel, %.0f bytes.\n',length(data)/c-options.count,2*(length(data)-options.count*c));
    end
    data=data(1:c*options.count);
end
if c*options.count>length(data)
    if isfinite(options.count)
        fprintf('Missing %.1f sample/channel, %.0f bytes.\n',options.count-length(data)/c,2*(options.count*c-length(data)));
    end
    options.count=floor(length(data)/c);
    data=data(1:c*options.count);
end
data=reshape(data,c,options.count)';

Here are edits to build data efficiently and reshape params.times (attached to this post):

% Return times.
params.times=[reports.time];

% Extract the data from the reports.
% Pre-allocate memory for speed -- js
data = zeros ( bytes , length ( reports ) ) ;
for k = 1 : length ( reports )
    data( : , k ) = reports( k ).report( 1 : bytes ) ;
end

% Combine two bytes for each reading.
% Pre-allocated data is already type double. Use matrix multiplication to
% combine the two bytes -- js
data = [ 1 , 256 ]  *  data ;

% Discard any extra 16-bit words at the end of the last report.
% We have to reshape params.times as well , or else they're out of alignment -- js
if length(data)>c*options.count
    if 2*(length(data)-options.count*c)>60
        fprintf('Trimming off an extra %.1f sample/channel, %.0f bytes.\n',length(data)/c-options.count,2*(length(data)-options.count*c));
    end
    data=data(1:c*options.count);
    params.times = params.times ( 1 : c * options.count ) ;
end
if c*options.count>length(data)
    if isfinite(options.count)
        fprintf('Missing %.1f sample/channel, %.0f bytes.\n',options.count-length(data)/c,2*(options.count*c-length(data)));
    end
    options.count=floor(length(data)/c);
    data=data(1:c*options.count);
    params.times = params.times ( 1 : c * options.count ) ;
end
data=reshape(data,c,options.count)';

% params.times now takes on the same shape as data -- js
params.times = reshape ( params.times , c , options.count )' ;

Cheers,
Jackson
6
XX---In PSYCHTOOLBOX@yahoogroups.com, <jsdpag@...> wrote :

Hi Mario,

Thanks, that all makes a lot of sense. I've attached my copy of DaqAInScan.m to this reply, since I don't yet know how to add to GitHub without messing that up on you. Changes are summed up as:

- Workaround for deprecated bitcmp
- Remembers Daq device serial number and uses its device IDs, if multiple devices connected
- Allows partial reads if checking for consecutive reports
- options.nodiscard to keep all data , skipping check for consecutive reports

In reply to your other post about integer types ( response to 19610 ), we tried both ways on Matlab R2012a. The answer using uint16 matched the outcome of the deprecated bitcmp. I can't say that I've done extensive testing, or worked out all the logic of this.

-> I split your code up in two separate commits to make it more manageable. I also rewrote the bitcmp replacement differently, as your solution doesn't work on Octave 3.8, which is a requirement. If i understood the reason for the handling correctly, this should work as well and be more simple. Also i think the threshold for negating numbers was wrong with > 2048 instead of >= 2048 to detect the negative-sign bit 12. The different treatment for differential channels vs. single-ended channels looks very weird to me, so i hope that that code was correct in the first place. I can't test any of this due to lack of hardware, so just assume you did the right thing if it works on your setup.

thanks,
-mario

7
Hi Mario,

I agree, params.times should not be reshaped. And since 'bytes' can be 2 or 62, the way I've combined the two bytes with matrix multiplication is wrong.

In testing we ran into a whole bunch of new problems. In sum, reports seem to be generated one channel at a time for each sample over time , hence why data is reshaped (line 914). But if you use DaqAInScanContinue with wantLiveData = 1 and call DaqAInScanContinue repeatedly then you can't guarantee that the first report comes from the first channel. If you assume that, then sampled data gets assigned to the wrong column of output argument 'data'.

We solved that problem by assigning values to both the rows and columns of 'data' based on the reports' serial numbers. Then we found out that the report serial number never exceeds 65335 before starting again from 0 ; presumably PsychHID or the DAQ device uses an unsigned 16-bit integer to count serial numbers. We used a desperate hack to solve that problem. If we can write a more general solution then we'll submit this to github.

I couldn't find any mention of the the reports 'serial' field in the PsychHID documentation, or from my delvings into the source code, so I don't know where it comes from. If it is maintained by PsychHID then it might be worth using an unsigned 64-bit integer to count serial numbers, instead.

Best,
Jackson
8
XX---In PSYCHTOOLBOX@yahoogroups.com, <jsdpag@...> wrote :

Hi Mario,

I agree, params.times should not be reshaped. And since 'bytes' can be 2 or 62, the way I've combined the two bytes with matrix multiplication is wrong.

-> Ok. So we won't merge this stuff.

In testing we ran into a whole bunch of new problems. In sum, reports seem to be generated one channel at a time for each sample over time , hence why data is reshaped (line 914). But if you use DaqAInScanContinue with wantLiveData = 1 and call DaqAInScanContinue repeatedly then you can't guarantee that the first report comes from the first channel. If you assume that, then sampled data gets assigned to the wrong column of output argument 'data'.

-> Means the code of yours i just merged is broken? Or just not that reliable?

We solved that problem by assigning values to both the rows and columns of 'data' based on the reports' serial numbers. Then we found out that the report serial number never exceeds 65335 before starting again from 0 ; presumably PsychHID or the DAQ device uses an unsigned 16-bit integer to count serial numbers. We used a desperate hack to solve that problem. If we can write a more general solution then we'll submit this to github.

I couldn't find any mention of the the reports 'serial' field in the PsychHID documentation, or from my delvings into the source code, so I don't know where it comes from. If it is maintained by PsychHID then it might be worth using an unsigned 64-bit integer to count serial numbers, instead.

-> It comes from line 791 of DaqAInScan.m, a 16-Bit serial number extracted from the last 2 Bytes of the HID report, provided by at least the USB-1208FS according to code comments. One would hope all USB-DAQ devices provide such a number, but who knows?

This code commit of mine to the Linux 3D OpenGL library solves the same problem for mapping slightly scrambled/out of order 32-Bit serial numbers to 64-Bit numbers. You could translate that solution to the problem of mapping slightly scrambled 16-Bit numbers to, e.g., 32-Bit numbers, just using different thresholds like 2^14 for a 16 bit number instead of 2^30 for a 32 bit number in that code:

https://cgit.freedesktop.org/mesa/mesa/commit/?id=cc5ddd584d17abd422ae4d8e83805969485740d9

Btw. for future submissions, please make sure to use the same coding style as the rest of the code, e.g., wrt. blank space.

thanks,
-mario