The Rec. 601/Rec.709 setting is for the input color space for converting the YPbPr signal to RGB. http://junkerhq.net/xrgb/index.php?title=OSSC#YPbPr_input_Color_Space
The signal from the Wii that the OSSC receives at its input is 480p so it should be set to Auto or Rec 601.
You should stick to TX mode RGB for best quality and accurate colors. If you select YCbCr444 then it will convert the output from RGB to YCbCr Limited Range (16-235) and the receiving device has to convert it from YCbCr to RGB again choosing Rec 601/709 so you’ll want to avoid that.
EDIT: Too late 🙂