That gradient is the OSSC’s test screen, which appears on boot and disappears when it receives a sync signal. Since the OSSC is still displaying its test screen, back up to the OSSC and the N64.
You are correct that AV1-RGBS is the correct input for typical SCART sources; so, if the test screen isn’t going away, take a look at the OSSC’s LCD screen and its LED. Is the LED solid red? Does the LCD say “NO SYNC”?
Also, what kind of RGB mod was installed, and how was it installed? If the luma pin was, for whatever reason, severed, then your cable would be useless. I was also under the impression that people who get their N64s RGB-modded are recommended CSync cables; did you decide on a sync-on-luma cable from a recommendation?