Hm, testing using Ubuntu 22.04 on my Uno (actually it is a chinese clone of a genuine arduino) I get a digitalRead mean of 4.1ms ( ±0.001 SE) and my Seeeduino Xiao as 0.62ms ( ±0.0006 SE):
The Xiao at least has pretty low variance. The Uno is slightly bimodal, and ~4ms would not be so acceptable in a tight display loop. Neither show any increased variance over time. I think your M1 laptop is most of this problem, and on something native it would work more reliably.
As a workaround you can “warm up” the board by doing some digital reads in a loop before any experiment. I tend to do this with all my PTB functions, making sure MATLAB has everything in memory etc.