We have several Mac Pro 5,1 towers, one of which (6-core) is in daily use for video production running under High Sierra. It's been substantially updated with a PCIe boot and other SSD drives, a USB 3 card for high bandwidth external storage (replacing our old FireWire drives), etc. Our other Mac Pro towers (4-core) are fully functional but no longer in service.Inspired by this thread, I went out and bought a new toy on eBay....
I'm conscious of the issue that at some point productivity software updates will force the production unit to be updated to Mojave and beyond. That will mean replacing the original Radeon 5770 video card with one that's Metal-compatible.
Like you, I've bought an AMD Sapphire Pulse Radeon RX 580 card for that and have been experimenting with it in one of the leftover 4-core Mac Pros. Unfortunately, this RX 580 is not one with the much-referenced, but rarely if ever encountered in the wild, Apple-specific firmware. As such, it has some known drawbacks. There's no startup chime, and the screen remains completely black until the OS loads fully. As a result, the Option key startup drive selector is inoperative, although, once fully booted, you can use Startup Disk in System Preferences. It's probably not possible to boot into the recovery partition, although I haven't tested this yet. If it should become necessary, I can reinstall the original 5770 video card and regain access.
I assume you're faced with those same limitations?
There is a way to hack the RX 580 card's existing firmware to make the Mac recognize it as fully compatible, but the tools to do this only run under native Windows (including Boot Camp, but excluding VM's).
I have an internal SATA SSD set up with both High Sierra and Mojave, each on its own partition. Mac firmware is currently at 220.127.116.11.0, since I have not installed the latest Mojave update yet.
One of the things that I have not yet been successful with is to set up a Boot Camp partition, encountering an error message saying this isn't supported in my configuration. I still have some other options to try for getting Boot Camp to work. Assuming I'm eventually successful, I hope to try the RX 580 card firmware hack to get full compatibility.
This kind of experimentation is enabled by the Mac Pro's modular construction, since it's easy to set up different drive configurations using other Macs and then physically load them into the test tower.