Update on the firmware: the latest firmware resolved the glitch I was having with 5x mode. However, on the vanilla firmware my TV is compatible with the NES and SNES in 5x mode. With this custom firmware, they only display up to 4x. Genesis and N64 work fine in 5x.
The fact that the TV works with the vanilla firmware for those 2 consoles and but not this firmware hints that something’s still up with the way this firmware treats 5x.