Good decision.
The “Polaris” gpu family is well tested, and RX500 cards are the consumer variants of that family, so should work fine. The “Radeon Pro” variants are tested by myself (WX4100 infrequently, Radeon Pro 560 in a MacBookPro 2017 frequently), and used successfully quite a lot by various labs and Ian Andolina @iandol, as described in multiple posts here. I also test frequently with a built-in onboard Raven Ridge gpu as part of a PC with AMD Ryzen 5 2400G processor.
My friend Natalia uses a WX 3100 with great success, after experiencing the same problems as yours with a NVidia:
In principle all AMD and Intel gpu’s under Linux are supposed to be compatible with all Psychtoolbox versions, as long as there are no driver bugs in your Linux distribution itself, and historically there haven’t been nearly as many bugs with the Linux open-source drivers than as on Windows or on Apples trainwreck. Also the Linux OSS drivers are most frequently tested by myself and i’m contributing to driver development and debugging myself, to make sure the quality standard on Linux stays as high as possible.
In practice, ofc. i can only frequently test a small subset of gpu’s which i have available in my machines, and atm. these are Intel chips from the Skylake/Kabylake series, and AMD gpu’s from the Polaris and Raven Ridge series. In the past, gpu’s from the Sea Islands series and HD-4000/5000 series all the way back to the year 2006 were successfully tested and should still work fine, but that seems less relevant to buying decisions in 2020.
What is different wrt. PTB is how well special features like high color precision framebuffers or (sub-)millisecond fine-grained visual timing is supported, if squeezing out more precision relies on special PTB low-level hardware tricks, or on hardware improvements in the graphics cards themselves. Here the Polaris gpu’s are most well tested atm. Their successors from the Vega gpu family should be even better for fine-grained timing by use of FreeSync when using PTBs ‘UseFineGrainedTiming’ support on such setups, due to hardware improvements, and should still work with PTB’s special bag of tricks, but i never had the chance to access a Vega for testing.
The latest generation of AMD gpu’s, the integrated chips in AMD Ryzen processors and later, and the discrete “Navi” gpu family is not supported by PTB’s bag of low-level tricks and tweaks at the moment, but that is mostly because those tricks are generally not needed anymore, as the standard open-source drivers already do the same job, so i try to spare myself the maintenance overhead for that functionality. The hardware of the latest generations does contain other improvements over Polaris and Vega though, which can be useful for some scenarios.
-mario