I used to do something similar. Passing GPU between host and VM without rebooting is a major pain in the ass. What I did instead was had a Linux hypervisor and 3 VMs (Linux, Windows, and MacOS). I would swap between the 3 VMs, and they each had access to my GPU. It was fun to set up and somewhat convenient, but got really annoying as it was my only workstation at the time.
I would highly suggest to just accept dual-booting and if it takes too long, get a faster SSD and/or faster RAM.
I’ve since gone Linux full-time, and I have no complaints. None of the games I can no longer play would be worth having Windows to deal with. I thought I would miss them at first, but I’m happy playing what’s available.
really weird that they only included a discord link, but here is the repo: https://github.com/dittofeed/dittofeed