I just tried it with my single N64 PAL game: Clay fighter 63 1/3 and my capture card didn’t have any problems with V.active 288. I just increased V.active to 288 and decreased V.backporch to center the image. It resulted in 1600/1920×1440. I think your TV does not accept these resolutions which is odd since it should handle up to 4k.
How does this profile work exactly? Can I imagine the OSSC sends a 1600/1920×1200 frame to the TV and by the advanced timing settings we’re “fiddling” the image in?
The OSSC sends all the lines to the display so in lineX5 PAL mode 1565 lines which you can consider the frame. Then with V.active it tells the display how many of those lines should be visible. So the display will crop the resolution itself.