Good. The config file you created should be correct and things seem to work.
Wrt. uninstalling the 3rd party AMD drivers, there should be an amdgpu-uninstall script on your machine that you can run + reboot to go back to the default Ubuntu drivers, but this is unlikely to be the culprit for the timestamping error you get occassionally. Those 3rd party AMD drivers are just more up to date / recent versions of the open-source drivers that Ubuntu ships with by default. Functionally they are not worse.
More likely is a slight problem with your computers clock, e.g., caused by using Network time protocol NTP for automatic time and date setting and sync across machines, in combination with local network issues: See this thread for discussion and tips for resolution: Inconsistent Clock Behavior on Laptop
Or, less likely, you specify a tWhen target stimulus onset time in Screen('Flip', ..., tWhen,...); that is so far into the future that substantial clock error accumulates. E.g., for flips that are dozens of seconds or minutes into the future.
-mario