Thanks a lot for your contributions BonzoBits!
The interesting thing is that Lx4 mode might actually be one of the more compatible modes. This is because, due to how the scanline count works out, it’s actually really close to VESA timings from the start. Here is the VESA timings for 1280×1024 (non-reduced blanking):
VESA 1280x1024 H.Samplerate | 1712 | V.Total | 1063 | H.Active | 1280 | V.Active | 1024 | H.FrontP | 80 | V.FrontP | 3 | H.Sync | 136 | V.Sync | 7 | H.BackP | 216 | V.BackP | 29 |
Note that LO in linex4 is 4×262= 1048 or 4×263= 1052, which is pretty close to 1063 (e.g. a lot closer than in Lx5).
And both optimized modes work as 427×4= 1708, and 341×5= 1705, apparently within tolerance also.