I think the problem with 320×240 optimized in 5x is that exceeds some max pixel clock limitation of what the display can handle. My Philips TV behaves the same.
Generic mode 5x has h.samplerate of 2046 (actually this is the same as 256×240 optimized 4:3: 341×6 = 2046). With a NES/SNES (262 lines) the pixel clock is roughly 160.8 MHz.
Optimized 320×240 has total output samplerate of above 2100 (e.g. 427×5 = 2135). With a MD/Genesis (263 lines) the pixel clock is roughly 168.4 MHz. Somewhere in that territory is where my display will not accept the signal anymore.
(Keep in mind that the standard pixel clock for 1080p is 148.5 MHz, and for WUXGA (1920×1200@60Hz) it’s 153.9 MHz, so we are already above those from the beginning.)
Regarding your PAL 1080p 4x tweak, I can add that while the 2200 samplerate/1920 active setting does indeed work on my TV, it prevents me from forcing 4:3 so I get wrong AR. So I tweaked it back to a 4:3 format by h.samplerate at 1938 with 1440 h.active (sync/backporch roughly 170/210). My TV recognizes this as a 1440×1080 @ 50Hz signal and displays it beautifully.