Those are total line counts of the incoming resolution, not simply visible lines (so 480p will show as 525p, 576p will show as 625p, etc.); 423 lines seems appropriate for 384p.
My assumption is that your display doesn’t like the resolution, refresh rate, or both. Assuming a resolution of 496×384, line doubled, that comes to 992×768, which is not any sort of standard PC or TV resolution. If the OSSC is framing this as 1024×768, that might have better compatibility, but I imagine there are TVs out there that don’t accept this over HDMI.
As for refresh rate, 57.66Hz is enough off-spec from 60Hz that I can see displays outright rejecting it.
You might need something like a Framemeister or Extron DSC 301 HD to scale and framerate convert this to use on your display.