Line5x does a couple tricks internally to generate active width of 1920/1600. For 320×240 mode, horizontal parameters are multiplied by 5, so with default settings H.samplerate=2130, H.active=1600, H.backporch=245, H.synclen=155. However, these would not be valid settings for 1920-column mode (1920+245+155>2130) so H.synclen is reduced by 120, H.backporch by 160, and H.active increased by 2*160. The final parameters thus are H.samplerate=2130, H.active=1920, H.backporch=85, H.synclen=35, H.frontporch=90.
If you set H.samplerate=386 for N64, then notice that 5*386=1930, leaving practically no time for sync & porch signals. Setting output format to 1600×1200 will solve this, but you also need to keep H.synclen>24 (same applies for all L5 optim. modes) to avoid underflowing it.
This is crucial information to understanding Advanced Timing.
I had no idea that the line multiplier multiplies the base timing parameters after the sampling.
Please add this to the Wiki.
The Wiki page is very vague about signal path and advanced timing.
It would be great to have more information so that more users can make their TV work with the OSSC.