Some more findings. In lineX2 the left mask is always 1 pixel wider than the right mask. For example with H.mask set to 6 with linex2 256×240 optimized the left and right borders should both be 18 pixels but they are 19 & 17.
The same thing with the latency test where the boxes should be 90 the top left one is 91 and bottom right is 89 pixels. So this must be an issue with hcnt_act being off.
Delaying only hcnt_2x by 2 cycles will increase the left border by 1 pixel and decrease the right border. However in 480pX2 which had no issue now also has a border of 2 pixels. Delaying hcnt_2x by 1 cycle had no effect in 240pX2.