problem loading gamma table

dear mario
i load a special gamma table, with fine steps over a limited range, in order to get better than 8-bit resolution in my low-contrast image, but I'm still getting 8-bit performance. i think i'm taking all the advice you've given me on this topic over the past 2 years, including the best macOS computer to buy. i was confident that this would work, and I'm at a loss to guess what i might be doing wrong.

MacBook Pro (Retina, 15-inch, Mid 2015)
AMD Radeon R9 M370X 2048 MB
macOS 10.12.3
MATLAB 9.1.0.441655 (R2016b)
Psychtoolbox 3.0.14 - Flavor: beta - Corresponds to SVN Revision 5452
​I used a Cambridge Research Systems colorimeter to measure the luminance produced by each gamma value that I loaded into the CLUT, using EnableCLUTMapping and
Screen('LoadNormalizedGammaTable',...).
Here are the graphs of the luminance vs. gamma.
With 8-bit performance one expects 255 luminance steps across the whole range, and 4 steps over 1/64 of the range, and 2 steps over 1/128 of the range. That is exactly what I get, though I anticipated getting much better performance. I'd be happy with just dither, simulating better-than-8-bit performance.
Inline image 2
PTB-INFO: OpenGL-Renderer is ATI Technologies Inc. :: AMD Radeon R9 M370X OpenGL Engine :: 2.1 ATI-1.48.21
PTB-INFO: Renderer has 2048 MB of VRAM and a maximum 1949 MB of texture memory.
PTB-INFO: VBL startline = 1800 , VBL Endline = -1
PTB-INFO: Beamposition queries unsupported or defective on this system. Using basic timestamping as fallback: Timestamps returned by Screen('Flip') will be less robust and accurate.
PTB-INFO: Measured monitor refresh interval from VBLsync = 0.000000 ms [inf Hz]. (0 valid samples taken, stddev=0.000000 ms.)
PTB-INFO: Small deviations between reported values are normal and no reason to worry.
PTB-WARNING: ==================================================================================================================
PTB-WARNING: DESKTOP COMPOSITOR IS ACTIVE! ALL FLIP STIMULUS ONSET TIMESTAMPS WILL BE VERY LIKELY UNRELIABLE AND LESS ACCURATE!
PTB-WARNING: STIMULUS ONSET TIMING WILL BE UNRELIABLE AS WELL, AND GRAPHICS PERFORMANCE MAY BE REDUCED!
PTB-WARNING: DO NOT USE THIS MODE FOR RUNNING REAL EXPERIMENT SESSIONS WITH ANY REQUIREMENTS FOR ACCURATE TIMING!
PTB-WARNING: ==================================================================================================================
PTB-INFO: All display tests and calibrations disabled. Assuming a refresh interval of 60.000000 Hz. Timing will be inaccurate!
PTB-INFO: Psychtoolbox imaging pipeline starting up for window with requested imagingmode 1027 ...
PTB-INFO: Will use 8 bits per color component framebuffer for stimulus drawing.
PTB-INFO: Will use 8 bits per color component framebuffer for stimulus post-processing (if any).
Compiling all shaders matching RGBMultiLUTLookupCombine_FormattingShader * into a GLSL program.
Building a fragment shader:Reading shader from file /Applications/Psychtoolbox/PsychOpenGL/PsychGLSLShaders/RGBMultiLUTLookupCombine_FormattingShader.frag.txt ...
EnableCLUTMapping: Using a 32 bit float CLUT -> 23 bits effective linear mapping precision per color channel.

By the way, using EnableCLUTMapping, I notice that what is returned by ReadNormalizedCLUT is not affected by whatever I load with LoadNormalizedCLUT.

My complete test program, 123 lines, is below. It produced the Figure above. Can you give me advice on how to achieve 10-bit performance?

Best
denis

function TestLoadGamma
% TestLoadGamma
% Measure the precision of the OUTPUT of the gamma table. This will include
% precision achieved by dither, as implemented by the video driver.
% Denis Pelli March 28, 2017

crsColorimeter=1; % ColorCAL II XYZ
if crsColorimeter
%% OPEN PORT FOR COLORIMETER
% Cambridge Research Systems
%
ob=instrfind;
fclose(ob);
portName='/dev/cu.usbmodem0001';
if exist('s1','var')
fclose(s1);
delete(s1)
end
clear s1;
% Open ColorCAL II virtual serial port.
s1 = serial(portName);
pause(0.1);
fopen(s1);
addpath(fullfile(fileparts(mfilename('fullpath')),'../ColorCAL MKII')); % folder in same directory as this M file
else
s1=[];
end

logBase2OfFractions=[0 6 7];
useFractionOfScreen=0.2;
BackupCluts;
Screen('Preference','SkipSyncTests',2);
try
%% OPEN WINDOW
screen = 0;
screenBufferRect = Screen('Rect',screen);
PsychImaging('PrepareConfiguration');
PsychImaging('AddTask','General','UseRetinaResolution');
PsychImaging('AddTask','AllViews','EnableCLUTMapping',256,1); % clutSize,high res
if ~useFractionOfScreen
[window,screenRect] = PsychImaging('OpenWindow',screen,255);
else
[window,screenRect] = PsychImaging('OpenWindow',screen,255,round(useFractionOfScreen*screenBufferRect));
end
%% MEASURE LUMINANCE AT EACH GAMMA VALUE
steps=31;
for j=1:length(logBase2OfFractions)
logBase2OfFraction=logBase2OfFractions(j);
for i=0:steps
if logBase2OfFraction>0
g=0.5;
else
g=0;
end
g=g+2^-logBase2OfFraction*i/steps;
assert(g<=1)
bits(j).test(i+1).gamma=g;
bits(j).test(i+1).L=LuminanceOfGamma(s1,window,g); % Read photometer
end
end
catch
if ~isempty(s1)
% Close the serial port
fclose(s1);
delete(s1);
clear s1
end
sca
psychrethrow(psychlasterror);
end
if ~isempty(s1)
% Close the serial port
fclose(s1);
delete(s1);
clear s1
end
Screen('Close',window);
close all
RestoreCluts
sca

%% PLOT RESULTS
figure
for j=1:length(logBase2OfFractions)
logBase2OfFraction=logBase2OfFractions(j);
subplot(1,length(logBase2OfFractions),i)
plot([bits(j).test.gamma],[bits(j).test.L]);
title(sprintf('Range/%d',2^logBase2OfFraction));
xlabel('gamma');
ylabel('Luminance (cd/m^2)');
xlim([bits(j).test(1).gamma bits(j).test(end).gamma]);
end
end

function L=LuminanceOfGamma(s1,window,g)
% L=LuminanceOfGamma(s1,window,g)
% MEASURE LUMINANCE PRODUCED BY ONE GAMMA VALUE
persistent CORRMAT
if ~isempty(s1) && isempty(CORRMAT)
% Get ColorCAL II XYZ correction matrix (CRT=1 WLED LCD=2 OLED=3):
CORRMAT = crsGetColorCALCorrectionMatrix(s1,2);
pause(0.2);
end
loadOnNextFlip = 2; % REQUIRED for reliable LoadNormalizedGammaTable.
gamma=[0:255;0:255;0:255]'/255;
iPixel=128;
for j=-4:4
gamma(1+iPixel+j,1:3)=[g g g];
end
Screen('LoadNormalizedGammaTable',window,gamma,loadOnNextFlip);
Screen('FillRect',window,iPixel);
Screen('Flip',window);
if ~isempty(s1)
XYZ=crsGetColorCALMeasurement(s1,CORRMAT)'; % Read photometer
pause(0.1);
L=XYZ(2);
else
% No photometer. Simulate 8-bit performance.
L=200*round(g*256)/256;
end
fprintf('g% .4f, %.4f cd/m^2n',g,L);
end




photo
Denis Pelli
Professor of Psychology & Neural Science, New York University
+1-646-258-7524 | denis.pelli@... | http://psych.nyu.edu/pelli/ | Skype: denispelli | http://denispelli.com
XX---In PSYCHTOOLBOX@yahoogroups.com, <denis.pelli@...> wrote :

dear mario
thanks. first step is to install the kernel driver. Following the instructions in "Help PsychtoolboxKernelDriver", to disable cryptographic kext signature verification, I typed the following into Terminal:

Santayana:~ denispelli$ sudo nvram boot-args="kext-dev-mode=1"

I get the following error:

nvram: Error setting variable - 'boot-args': (iokit/common) general error

Has the command changed? Do I still need it?


-> No, you generally should not need this, as the driver is cryptographically signed for us by Cambridge Research Systems. Just install the driver, wait a minute, try if PTB detects the driver after a "clear all".


And if it didn't work, that tip wouldn't work anyway, since 10.11 or 10.12 Apple changed the override again in an incompatible way. I'll remove that tip as we don't support 10.10 anymore. Nowadays you'd need to completely disable SIP (System integrity protection) and the internet tells about that.


-> However, on your  MacBookAir installing the driver is pointless, as the driver does not support Intel graphics chips anymore by default because accessing them would be dangerous. Intel gpu based macs under OSX therefore don't get any high-precision timestamping / advanced problem detection / most workarounds / other goodies. Intel graphics under Linux will work with bells & whistles, as the required features are built into the OS.


ciao,

-mario




Best

denis

p.s.

The above is on my MacBook Air. I'll try the MacBook Pro in my office tomorrow.







Denis Pelli
Professor of Psychology & Neural Science, New York University
+1-646-258-7524 | denis.pelli@... | http://psych.nyu.edu/pelli/ | Skype: denispelli | http://denispelli.com

On Tue, Mar 28, 2017 at 11:52 PM, mario.kleiner@... [PSYCHTOOLBOX] <PSYCHTOOLBOX@yahoogroups.com> wrote:
 




XX---In PSYCHTOOLBOX@yahoogroups.com, <denis.pelli@...> wrote :

dear mario
i load a special gamma table, with fine steps over a limited range, in order to get better than 8-bit resolution in my low-contrast image, but I'm still getting 8-bit performance. i think i'm taking all the advice you've given me on this topic over the past 2 years, including the best macOS computer to buy. i was confident that this would work, and I'm at a loss to guess what i might be doing wrong.

-> Hi Denis,

i cannot imagine i ever recommended buying an Apple computer in the past couple of years? But AMD graphics was the least broken one in the past, and the one where we have the most control and workarounds iff the PsychtoolboxKernelDriver is installed and working, if that is what you meant? Not sure if that is still true though, the brokeness of recent AMD graphics under recent OSX versions seems to have reached the same level as NVidia brokeness. If the kernel driver is installed it would warn you about broken/untrustworthy visual timing - without any ability to fix anything about it though.

MacBook Pro (Retina, 15-inch, Mid 2015)
AMD Radeon R9 M370X 2048 MB
macOS 10.12.3
MATLAB 9.1.0.441655 (R2016b)
Psychtoolbox 3.0.14 - Flavor: beta - Corresponds to SVN Revision 5452
​I used a Cambridge Research Systems colorimeter to measure the luminance produced by each gamma value that I loaded into the CLUT, using EnableCLUTMapping and
Screen(' LoadNormalizedGammaTable',...) . 
Here are the graphs of the luminance vs. gamma.
With 8-bit performance one expects 255 luminance steps across the whole range, and 4 steps over 1/64 of the range, and 2 steps over 1/128 of the range. That is exactly what I get, though I anticipated getting much better performance. I'd be happy with just dither, simulating better-than-8-bit performance.

-> I think you misunderstood the use case for EnableCLUTMapping. It is a good way (= as in "the only way nowadays") to do CLUT animation with well defined timing, or other random remapping of pixel color values. It is not the method of choice for gamma correction or for squeezing out more precision via gamma table tweaking, as it doesn't operate on the hardware gamma tables at all, so would be limited to 8 bpc precision on a standard 8 bpc framebuffer. Might explain the 256 levels you get, also explains why you don't see any effect on ReadNormalizedGammaTable.

-> Use LoadNormalizedGammatable, but with the 'loadonNextFlip' flag of 0 or 1 to actually set the hardware tables. These have 10 bpc precision on your gpu for 256 -> 1024 levels mapping. If dithering is enabled by default on the machine, that might give simulated 10 bpc over DVI, HDMI or DisplayPort on 8 bpc panels. If it isn't enabled by default, one could use the PsychtoolboxKernelDriver + some Screen('ConfigureDisplay', 'Dithering', ...) command with the right magic number to override the OS, similar to the trickery we used to get 11 bpc or 12 bpc mode on your Linux laptop -- assuming Hoermet provides the measurements to confirm that trick actually worked.

Another thing to test would be if PsychImaging's 'EnableNative10BitFramebuffer' would do the trick of providing a dithering based simulated 10 bpc framebuffer on AMD hardware with macOS 10.12 on 8 bpc panels. I still didn't get any testing feedback about PTB's new 10 bpc framebuffer support on OSX. It certainly would work on Linux, both via dithering and also with native true 10 bpc on suitable external 10 bpc HDMI or DisplayPort panels.

Oh btw., PTB has a ColorCal2() driver for the CRS ColorCal II which might make such measurements more simple.

-mario





Inline image 2
PTB-INFO: OpenGL-Renderer is ATI Technologies Inc. :: AMD Radeon R9 M370X OpenGL Engine :: 2.1 ATI-1.48.21
PTB-INFO: Renderer has 2048 MB of VRAM and a maximum 1949 MB of texture memory.
PTB-INFO: VBL startline = 1800 , VBL Endline = -1
PTB-INFO: Beamposition queries unsupported or defective on this system. Using basic timestamping as fallback: Timestamps returned by Screen('Flip') will be less robust and accurate.
PTB-INFO: Measured monitor refresh interval from VBLsync = 0.000000 ms [inf Hz]. (0 valid samples taken, stddev=0.000000 ms.)
PTB-INFO: Small deviations between reported values are normal and no reason to worry.
PTB-WARNING: ============================== ============================== ============================== ========================
PTB-WARNING: DESKTOP COMPOSITOR IS ACTIVE! ALL FLIP STIMULUS ONSET TIMESTAMPS WILL BE VERY LIKELY UNRELIABLE AND LESS ACCURATE!
PTB-WARNING: STIMULUS ONSET TIMING WILL BE UNRELIABLE AS WELL, AND GRAPHICS PERFORMANCE MAY BE REDUCED!
PTB-WARNING: DO NOT USE THIS MODE FOR RUNNING REAL EXPERIMENT SESSIONS WITH ANY REQUIREMENTS FOR ACCURATE TIMING!
PTB-WARNING: ============================== ============================== ============================== ========================
PTB-INFO: All display tests and calibrations disabled. Assuming a refresh interval of 60.000000 Hz. Timing will be inaccurate!
PTB-INFO: Psychtoolbox imaging pipeline starting up for window with requested imagingmode 1027 ...
PTB-INFO: Will use 8 bits per color component framebuffer for stimulus drawing.
PTB-INFO: Will use 8 bits per color component framebuffer for stimulus post-processing (if any).
Compiling all shaders matching RGBMultiLUTLookupCombine_ FormattingShader * into a GLSL program.
Building a fragment shader:Reading shader from file /Applications/Psychtoolbox/ PsychOpenGL/PsychGLSLShaders/ RGBMultiLUTLookupCombine_ FormattingShader.frag.txt ...
EnableCLUTMapping: Using a 32 bit float CLUT -> 23 bits effective linear mapping precision per color channel.

By the way, using EnableCLUTMapping, I notice that what is returned by ReadNormalizedCLUT is not affected by whatever I load with LoadNormalizedCLUT.

My complete test program, 123 lines, is below. It produced the Figure above. Can you give me advice on how to achieve 10-bit performance?

Best
denis

function TestLoadGamma
% TestLoadGamma
% Measure the precision of the OUTPUT of the gamma table. This will include
% precision achieved by dither, as implemented by the video driver.
% Denis Pelli March 28, 2017

crsColorimeter=1; % ColorCAL II XYZ
if crsColorimeter
   %% OPEN PORT FOR COLORIMETER
   % Cambridge Research Systems
   %
   ob=instrfind;
   fclose(ob);
   portName='/dev/cu. usbmodem0001';
   if exist('s1','var')
      fclose(s1);
      delete(s1)
   end
   clear s1;
   % Open ColorCAL II virtual serial port.
   s1 = serial(portName);
   pause(0.1);
   fopen(s1);
   addpath(fullfile(fileparts( mfilename('fullpath')),'../ ColorCAL MKII')); % folder in same directory as this M file
else
   s1=[];
end

logBase2OfFractions=[0 6 7];
useFractionOfScreen=0.2;
BackupCluts;
Screen('Preference',' SkipSyncTests',2);
try
   %% OPEN WINDOW
   screen = 0;
   screenBufferRect = Screen('Rect',screen);
   PsychImaging(' PrepareConfiguration');
   PsychImaging('AddTask',' General','UseRetinaResolution' );
   PsychImaging('AddTask',' AllViews','EnableCLUTMapping', 256,1); % clutSize,high res
   if ~useFractionOfScreen
      [window,screenRect] = PsychImaging('OpenWindow', screen,255);
   else
      [window,screenRect] = PsychImaging('OpenWindow', screen,255,round( useFractionOfScreen* screenBufferRect));
   end
   
   %% MEASURE LUMINANCE AT EACH GAMMA VALUE
   steps=31;
   for j=1:length( logBase2OfFractions)
      logBase2OfFraction= logBase2OfFractions(j);
      for i=0:steps
         if logBase2OfFraction>0
            g=0.5;
         else
            g=0;
         end
         g=g+2^-logBase2OfFraction*i/ steps;
         assert(g<=1)
         bits(j).test(i+1).gamma=g;
         bits(j).test(i+1).L= LuminanceOfGamma(s1,window,g); % Read photometer
      end
   end
catch
   if ~isempty(s1)
      % Close the serial port
      fclose(s1);
      delete(s1);
      clear s1
   end
   sca
   psychrethrow(psychlasterror);
end
if ~isempty(s1)
   % Close the serial port
   fclose(s1);
   delete(s1);
   clear s1
end
Screen('Close',window);
close all
RestoreCluts
sca

%% PLOT RESULTS
figure
for j=1:length( logBase2OfFractions)
   logBase2OfFraction= logBase2OfFractions(j);
   subplot(1,length( logBase2OfFractions),i)
   plot([bits(j).test.gamma],[ bits(j).test.L]);
   title(sprintf('Range/%d',2^ logBase2OfFraction));
   xlabel('gamma');
   ylabel('Luminance (cd/m^2)');
   xlim([bits(j).test(1).gamma bits(j).test(end).gamma]);
end
end

function L=LuminanceOfGamma(s1,window, g)
% L=LuminanceOfGamma(s1,window, g)
% MEASURE LUMINANCE PRODUCED BY ONE GAMMA VALUE
persistent CORRMAT
if ~isempty(s1) && isempty(CORRMAT)
   % Get ColorCAL II XYZ correction matrix (CRT=1 WLED LCD=2 OLED=3):
   CORRMAT = crsGetColorCALCorrectionMatrix (s1,2);
   pause(0.2);
end
loadOnNextFlip = 2; % REQUIRED for reliable LoadNormalizedGammaTable.
gamma=[0:255;0:255;0:255]'/ 255;
iPixel=128;
for j=-4:4
   gamma(1+iPixel+j,1:3)=[g g g];
end
Screen(' LoadNormalizedGammaTable', window,gamma,loadOnNextFlip);
Screen('FillRect',window, iPixel);
Screen('Flip',window);
if ~isempty(s1)
   XYZ= crsGetColorCALMeasurement(s1, CORRMAT)'; % Read photometer
   pause(0.1);
   L=XYZ(2);
else
   % No photometer. Simulate 8-bit performance.
   L=200*round(g*256)/256;
end
fprintf('g% .4f, %.4f cd/m^2\n',g,L);
end




Denis Pelli
Professor of Psychology & Neural Science, New York University
+1-646-258-7524 | denis.pelli@... | http://psych.nyu.edu/pelli/ | Skype: denispelli | http://denispelli.com