I assumed so because what you posted is an invalid mode (active > samplerate ! )
Samplerate = active + frontporch + sync + backporch
This relation should not be violated. I have no idea how the OSSC treats the signal with those parameters.
The reason I guess your display does not accept the 320×240 optim, might have to do with it’s upper limit for pixelclock, or other incompatibility. For example my TV does not accept total samplerate over 2150-ish. In 5x mode this limits 320 optim mode to 430 samplerate (5×430 = 2150), and it is unstable even before that.
However, in 512 optim mode, it’s only multiplied 3x horizontally in 5x mode, 2x in others. (See the table in optimal timing wiki page) So you will have much lower total samplerate.
So try with lower line multiplication mode.
Also, from the MAME documentation for toaplan2 board:
/* video hardware */
SCREEN(config, m_screen, SCREEN_TYPE_RASTER);
m_screen->set_raw(27_MHz_XTAL/4, 432, 0, 320, 262, 0, 240);
320 signifies active h.video (the others dotclock, dots per line, total lines, active lines, everything you needed 😉 )
Edit: regarding raiden, the 14.3 and 12 mhz clocks are only for audio. The video area seems to be 256×240, so that would indicate a dotclock below 10mhz. 5 or 6.67 perhaps?