I was having this same, frustrating issue. Tried all sorts of things to no avail.
I did find a workaround that doesn't require fiddling before every flight:
You have to install Voice Meeter Banana in order to do this; so yes, another app eating up resources, but it works. Once you have that installed, all you essentially need to do is direct Vox to play through Voice Meeter's virtual input, rather than your actual sound card. Within Voice Meeter, you can then control the volume of the voices you have assigned to Vox without any auto-adjustments taking place.
Hope this helps somebody.