Looks like the smoothing filter RetroTINK applies.
For Gamecube, might just be better to upgrade to a mark 2 GCHD. GCHD MK 2 and GCDual have full RGB analogue out. I route mine through the OSSC and use the 480p x2 mode, I like the look better than the digital output from GCDual (without 480p x2) and there’s no loss in quality that I can see with a shielded cable.
For N64 I’d say the only reason to bypass UltraHDMI was to minimise input lag, in which case you can use direct mode with a HDMI to VGA adapter. N64 is an odd beast and better suited to the UltraHDMI’s custom scaling and debluring system.
DC HDMI, I saw Dan do some comparison shots, things are very close between OSSC and DC HDMI. If you don’t need the convenience of HDMI on your Dreamcast OSSC is still a great way to hook it up (especially via Toro SCART).