What kind of TV do you have? Have you tried with both passthrough and line2x mode?
You should also try cutting off the top+bottom pixel from the image in line2x to produce a standard 540p resolution. The default (in line2x) is 960×544, which is non-standard and some TVs may not like (passthrough mode of 480×272 is also non-standard). This wiki article covers how to do this, but tl; dr: with the PSP mode enabled and set to line2x, in the advanced settings change
V. Active 270
V. Backporch 135
Also if you have a different source for a 540p image, you should test your TV with that source to see if your TV supports 540p at all.
If that doesn’t fix it, and your TV does support “normal” 540p signals, then it’s possible that your TV doesn’t like the very large offscreen regions of the image (much larger than usual), which isn’t something that can be fixed with the OSSC.