I would say Modes 2 and 3 look blocky because of the way bob deinterlacing works. This type of deinterlacing (or filling in the missing field with black lines) is absolutely necessary to produce a 480p signal without adding any perceivable lag and providing compatibility for TVs that don’t accept 480i over HDMI.
Where Mode 1 is concerned, it’s just digitizing the 480i signal and passing it along, and it seems like your capture device or capture software is doing some form of deinterlacing. (If you haven’t already, you’ll probably want to verify the encoding settings so that it’s deinterlacing to 60fps and not 30fps.) Like I said, the RT2X in this instance might not be properly setting some metadata flags, which would be something out of your control. You should get in touch with Mike Chi; he might be able to help diagnose the problem and potentially provide a solution.