There are a few options for using an N64 with the OSSC.
First, if you’re not willing to mod your console, there is the RetroTINK 2X plus an HDMI to VGA converter, which I mentioned earlier.
If you are willing to mod, you can install an RGB mod, like NoAffinity mentioned, which can either be a relatively simple board and install if you have an early NS1-serialed model; or a slightly more complicated install with Tim Worthington’s board, which will work on any N64.
Lastly, you can do what I intend to do, and skip the RGB mod altogether and go straight for the UltraHDMI mod, which you can either use directly on your HDMI receiver/TV, or you can use an HDMI to VGA converter to run it into the OSSC. The UltraHDMI can do line-doubling on its own, but it does have a direct passthrough mode that will output at the native video dimensions (and optionally also the original, non-normalized sync signal), which you can run into the OSSC for line multiplication and/or adding scanlines, or whatever you want to do.