Experimental direct OpenGL-to-Metal presentation backend for Psychtoolbox on Apple Silicon

Bugs & Features
,
1

Motivation

I have been investigating the presentation-timing problems affecting Psychtoolbox on Apple Silicon under recent macOS versions. In particular, ordinary drawing and unscheduled flips can appear to work, while some scheduled presentation tests exhibit half-rate or irregular behavior.

As an experiment, I created a small component called PsychMetal. It retains PTB’s normal Screen drawing functions but replaces window creation, matrix-to-texture creation, presentation, and cleanup with a direct macOS Metal presentation path.

Basic interface

The supported interface is deliberately small:

[w, rect, ifi] = PsychMetal('OpenWindow', screenNumber);
texture = PsychMetal('MakeTexture', w, imageMatrix);
vbl = PsychMetal('Flip', w);
history = PsychMetal('Diagnostic', w);
PsychMetal('Close', w);

All actual drawing continues to use the normal PTB Screen functions with the returned handle w. This includes shapes, dots, text, textures, rotation, blending, and other ordinary drawing operations.

For a basic PTB program, the required substitutions are:

Screen('OpenWindow', ...)   -> PsychMetal('OpenWindow', ...)
Screen('MakeTexture', ...)  -> PsychMetal('MakeTexture', ...)
Screen('Flip', ...)         -> PsychMetal('Flip', ...)
Screen('CloseAll')          -> PsychMetal('Close', w)

For example:

[w, rect, ifi] = PsychMetal('OpenWindow', max(Screen('Screens')));

Screen('FillRect', w, [128 128 128]);
Screen('DrawDots', w, xy, sizes, colors);

vbl = PsychMetal('Flip', w);

history = PsychMetal('Diagnostic', w);
PsychMetal('Close', w);

Textures returned by PsychMetal('MakeTexture') are ordinary PTB texture handles and are drawn and closed normally:

texture = PsychMetal('MakeTexture', w, imageMatrix);
Screen('DrawTexture', w, texture, [], destination, angle);
Screen('Close', texture);

Presentation path

PsychMetal creates two IOSurfaces that are shared between OpenGL and Metal. These surfaces are attached alternately to the OpenGL framebuffer used by PTB drawing commands.

After drawing, the completed IOSurface is presented through CAMetalDisplayLink in a native fullscreen Metal window. The other IOSurface is immediately attached as the next drawing buffer.

The resulting path is approximately:

PTB Screen drawing through OpenGL
             ↓
double-buffered IOSurfaces
             ↓
native Metal presentation

PsychMetal does not call Screen('Flip') or use PTB’s Vulkan/MoltenVK presentation path. The OpenGL framebuffer attachment is the IOSurface subsequently sampled by Metal, so there is no separate full-frame OpenGL copy into an intermediate surface. Metal performs the final IOSurface-to-drawable rendering pass.

The returned drawing rectangle is the physical Retina framebuffer size rather than a smaller logical resolution that is subsequently enlarged.

Why Flip returns a projected timestamp

One important difference concerns timestamps.

To sustain one presentation per refresh, Metal keeps frames queued in advance. On my system, the display link normally accepts a submitted frame with a target approximately two refresh intervals in the future.

At that point the target presentation time is known, but the actual presentedTime does not yet exist because the frame has not been displayed. Metal supplies the confirmed timestamp through a callback only after the future presentation occurs.

PsychMetal('Flip', w) therefore returns the calibrated projected presentation time. It does not wait for the later confirmed timestamp.

Waiting inside every Flip for the actual timestamp would stall the MATLAB or Octave loop while the queued frame crosses those future refreshes. This would prevent subsequent frames from being submitted sufficiently far in advance and could reduce the presentation rate. Confirmed timestamps are therefore collected asynchronously.

Diagnostic timestamp history

After or outside the presentation loop:

history = PsychMetal('Diagnostic', w);

returns one record for every issued Flip, including:

history.flipNumber
history.frameID
history.projectedTimestamp
history.actualTimestamp
history.actualStatus
history.targetErrorMs
history.projectionLeadMs
history.scheduledAt
history.confirmationCallbackTime
history.confirmationDelayMs
history.displayLinkTick
history.commandStatus

The projected timestamp returned during presentation can therefore be compared with Metal’s subsequently confirmed timestamp for the same Flip.

Occasionally Apple returns no valid presentedTime for a frame. PsychMetal records the actual timestamp as unavailable in that case. A missing timestamp is not automatically classified as a dropped physical frame, particularly when the projected timestamps and display-link frame counts remain consecutive.

Results so far

On an M4 MacBook Air with a 60 Hz display, a 3,600-frame fullscreen test produced:

  • 3,600 valid scheduled presentations

  • median presentation interval of approximately 16.6667 ms

  • 99th-percentile interval of approximately 16.6668 ms

  • no skipped refreshes in that run

  • approximately 1.2 ms median drawing time

  • approximately 13 ms median synchronous Flip-call duration

The packaged binaries were tested under both native ARM64 GNU Octave and native ARM64 MATLAB.

Repeated opening, presentation, cleanup, reopening, and MEX unloading were also tested. Two consecutive MATLAB runs each produced 30 valid presentations, 29 consecutive frame intervals, and no skipped refreshes.

These remain software timestamp tests. Independent photodiode validation is required before relying on the timestamps for experimental stimulus timing.

Current limitations

PsychMetal deliberately implements only a restricted feature set:

  • Apple-silicon macOS

  • native fullscreen presentation

  • selection of an attached PTB screen number

  • 8-bit monoscopic drawing

  • one new presentation per refresh

  • ordinary PTB drawing commands

  • textures created from MATLAB or Octave matrices

  • projected timestamps during presentation

  • asynchronously collected confirmed timestamps

It does not currently implement:

  • the when argument

  • dontclear or dontsync

  • asynchronous Flip

  • multiflip

  • multiple-refresh presentation intervals

  • stereo

  • HDR

  • DataPixx or other specialized display hardware

  • the full functionality of Screen('Flip')

Source and download

Source repository:

Packaged experimental release:

The repository is MIT-licensed and includes:

  • PsychMetal.m

  • the complete Objective-C++ MEX source

  • precompiled ARM64 binaries for MATLAB and Octave

  • a Makefile and build scripts

  • an original direct-presentation demonstration

  • installation, conversion, diagnostic, and build documentation

It does not include or redistribute Psychtoolbox source code. Psychtoolbox is an external dependency.

After downloading, add the extracted PsychMetal directory to the MATLAB or Octave path:

addpath('/path/to/PsychMetal')

Then test it with:

PsychMetalDirectDemo(600)

The precompiled binary for the current host should be selected automatically. Both binaries can also be rebuilt from source:

make all

or separately:

make octave
make matlab

Because a browser may apply a macOS quarantine attribute to downloaded MEX files, a user who trusts the downloaded source and binaries may need to run:

cd /path/to/PsychMetal
xattr -dr com.apple.quarantine .

Scope, intent, and possible future direction

This experiment is not intended to replace or compete with Psychtoolbox. PTB provides a mature, cross-platform system with a much broader collection of drawing, timing, input, audio, imaging, and specialized-hardware functionality.

I also understand the importance of maintaining one coherent cross-platform implementation rather than developing and supporting independent versions of PTB for macOS, Linux, and Windows.

The difficulty is that the preferred modern graphics API is now platform-dependent. Metal is native to macOS, Vulkan is available natively on Linux and Windows, and Direct3D is native to Windows. Vulkan applications can run on macOS through MoltenVK, but that introduces a Vulkan-to-Metal translation layer—the layer this experiment deliberately bypasses for presentation.

PsychMetal is therefore not a proposal for a separate macOS implementation of the whole toolbox. I am simply experimenting with the smallest practical direct Metal presentation path to see whether it can provide reliable frame pacing and useful timestamps while retaining PTB’s existing drawing interface.

The current implementation still relies on PTB’s OpenGL drawing functions. Apple deprecated OpenGL several years ago and may eventually remove it. A possible longer-term architecture could preserve one common, user-facing PTB drawing API while implementing its lower-level operations through platform backends—for example, Vulkan on Linux and Windows and Metal on macOS. Textures, shapes, dots, text, rotation, blending, and antialiasing could then behave consistently even though their native implementations differed.

That would be a substantially larger undertaking. For now, this is a focused experiment and diagnostic tool, shared in case the implementation or timing results are useful for understanding the current macOS behavior or informing future backend work.