First things first, de-blurred N64 content will look the best at 1280×960 (4x mode), since the optimal sampling rate is 386.75 (386.75 * 4 = 1547). The OSSC will round to the nearest integer, so it won’t be perfect in 5x mode.
Also, by pre-deblur N64 RBG board, do you mean Tim Worthington’s board or a simple amp mod using a THS7314 chip (or similar)? If there is a Video LPF higher up in the chain, you want to disable the Video LPF on the OSSC, since it affects phase settings. A THS7314 chip will already have one. However, a THS7316 chip doesn’t have a filter and a THS7374 might have it turned off or on. If I remember correctly the Tim Worthington boards use a THS7374 with the low pass filter enabled. Make sure you only have one filter in the chain before dialing in settings. The “Auto” setting on the OSSC will never turn the filter off, so make sure it’s set to “Off” if necessary.
One thing that you can also try is to turn off “Allow TVP HPLL2x” in Sampling opt.
On: Enables 2x H-PLL, generally reducing jitter at the price of inaccurate sampling phase due to internal bug. [default]
Off: Disables 2x H-PLL, which might help with Line5x stability.
It usually doesn’t matter much, but for accurate sampling phase for the N64, I’ve found it to be necessary.
Another thing is that text has a tendency to look bad with de-blur. Especially games that render in 640×240 (GoldenEye for example, which looks terrible with de-blur). For these titles you’re better off using the generic modes. While there are optimal timings for 640×240, it just doesn’t look great. OoT should look fine for the most part though, since the text is very blocky.
Results at 5x should look a little better than in the picture you provided, but it will never be 100% perfect. 4x will be much closer to a perfect result. PAL consoles also look the best at 4x since the optimal sampling rate is very close to 397.25.
Give the settings I mentioned a try and see if results improve.