Thanks for the suggestion paulb_nl! I attempted this on 256×240 optim. a while back with no luck and didn’t consider trying it in 320×240 optim. The method almost works, but I lose picture at exactly H. active 512. My CRT can display 511 fine, but the capture card doesn’t like it. H. active 510 works for both outputs, but that’s a net 1 pixel loss for 256×240 content (and I prefer shifting the image to the right 1 pixel and crop the left border, so that’s a net 2 pixel loss).
Unlike the generic scaler, turning on AV3 interlacefix didn’t allow for higher horizontal resolutions, and I’m not sure what other options I can tweak to make it work. What I see looks good though, and it suits my preferences at the moment.