I’m surprised the volume gets cut unless they are both on at the same time.
TRS and TRRS are just descriptions of the connector structure Tip-Ring-Sleeve (or 2 Ring for TRRS). The standard 3.5mm audio connector you see on headphone cables or the y connector you are using is a TRS connector. If you want to ensure isolation you need to use a 2 (or more) in to 1 out switch.
I assume you are maybe using component Y-Pb-Pr for both consoles, how are you switching between the video? It’s rare to see a component switch without any audio ports as well.