I don’t think there’s a known incompatibility that could make the OSSC fail to pick up a sync-on-luma signal from the N64. You should contact the modder. They should be able to tell you what cable works with your console. Sync-on-luma is probably a safe bet, though. You could check that the cable is actually wired for sync-on-luma.