It is your cable for sure if you lost a color for a bit and then it came back after messing with the cable. So I would replace the VGA cable from the Toro to the OSSC first.
I use an Extron 7SC for my retro consoles and it outputs as RGBHV. I have that into AV3 on my OSSC and do not recall having to flip any switches for that to work. I did find that with the Extron, it was best to set the OSSC to passthru mode for the 480p (What my extron upscales and outputs everything at), as that provided a slightly sharper picture than with line2x etc enabled.
But yes, if you have screen flipping and colors missing…that is a sign of a likely faulty cable.