Alright this is going to be long-winded because I’m drunk… Sorry.
From what I understand the OSSC outputs full range RGB, so HDMI black level being on “high” (or “normal” for Samsung TVs) “should” be correct… Otherwise you’re at risk of crushing (losing) detail at the extremes of the dark/light areas. What you’re seeing on that image is the Megadrive’s jailbars being exaggerated because the brightness is set too high.
I could be getting too technical here, but basically imagine that you have a source outputting the same test pattern that you see when you turn on the OSSC – a single image with a gradient going from white to black. A full range RGB signal will have 256 (0-255) steps between the whitest part of the gradient and the blackest part. A limited range RGB signal will have 220 (16-235) steps between the same white/black parts.
You have to tell the TV what sort of RGB range it should be expecting so it knows to map the correct values to the correct brightness range on the panel… If the TV HDMI black level is set to “low” then it expects a value of 16 and below to be completely black and outputs to the TV accordingly, the same goes for the high end and white. If it’s receiving a full range signal (while set to think it’s limited) then it’s culling information, and you’ll get an image that seems extremely vibrant/high-contrast but at the expense of any information that’s in those dark/light areas. This is why you’ve stopped seeing the lines in your second image, it’s culling info in the dark areas.
High (normal) black level = expects a full range (0-255) RGB signal
Low black level = expects a limited range (16-235) RGB signal.
The reason this setting exists is because traditionally TVs (broadcast TV etc) have always worked within a limited range, whereas PC monitors have always been full range. The HDMI black level option is there to allow your TV to correctly map PC-esque full range sources (like the OSSC) that don’t have an option to change to limited range. Hell, even modern consoles output limited range by default.
Problem is, analogue sources (especially consoles) don’t exactly adhere to a 0-255 level in the same way that digital sources do. On most Samsung TVs “45” should be the neutral brightness setting, where black is actually black (and not a slight shade of grey) without losing information in the dark areas, but with some consoles connected to the OSSC you’ll probably need to set the brightness down a couple of notches to compensate – mine’s around 42/43. This used to be the same on TVs when it came to the contrast setting, set it too high and you start to lose detail in the white areas – however, nowadays most TVs can have the contrast set to full/100 and not cull any information.
It’s just a guess – but Samsung TVs usually save different profiles (per source) depending on what’s connected. If something’s in HDMI mode (and adheres to certain resolutions/refresh rates) then the TV sees it as a standard input, you can choose to turn on game mode, change colour temperature, adjust zoom/resize options etc… If it’s in DVI mode then it probably sees it as a PC input, which disables a LOT of processing options – including game mode. You’ll also probably find that HDMI black level being greyed out also means it’s on “normal” (high) because it’s expecting a PC input.
The TV will have different settings saved/applied when it switches between normal/game mode and PC mode (brightness/contrast/any dynamic crap that’s turned on) and it’s likely one of those settings that’s blowing out the contrast, not HDMI black level. I know this because the 61Hz I get out of the RGB modded PAL N64 (running NTSC software) forces the TV into PC mode, disabling colour options etc – and loading different settings to the ones that pop up when it’s in game mode.
Long story short – while many of your TVs options may vary, turn on game mode, put HDMI black level on “high”, brightness on “43”, contrast on “95”, sharpness on “0” (unless it’s in PC mode, at which point it’s probably between 25-50 – thanks Samsung), colour between 50-55 (and temperature on warm1), colour space on “native” and turn off any of the stuff like dynamic colour/contrast/clear motion/clean view/film mode and all that crap…
As for why it looks OK with the Framemeister? God knows – maybe it’s mapping colours in a different way?