I think its because your TV does not accept the 48.92Hz on the HDMI input which is quite far from nominal 50Hz. Since you have an NTSC N64 it would be best to buy an Everdrive 64 and play NTSC games. PAL versions of N64 games are horrible.
But if you really want to play your PAL games then you need a PAL N64 or another display like a monitor which generally are more accepting of off-spec refresh rates.
As a last resort you could try to set the TX mode of the OSSC to DVI.