You will get accurate colors as long as you leave “YPbPr in Color Space” at auto.
“YPbPr in Color Space” means “YPbPr input Color Space” (that’s how it appears in the manual), so it is only used for interpreting incoming YPbPr signals (I believe the OSSC uses RGB internally). Leaving it at auto will let the OSSC correctly use Rec. 601 for SD and ED content and Rec. 709 for HD content.
As for TX Mode, you should leave it set to HDMI (RGB) unless you need some specific compatibility offered by the other two output modes, such as if your TV or downstream video processor cannot handle 4:4:4 RGB (Use YCbCr 4:4:4), or your AVR (like mine) won’t fall back to TOSLINK when the OSSC is in HDMI mode (use DVI).
As for using Rec. 601 for output, marqs will probably have a better explanation, but, my [limited] understanding is that HDMI (ostensibly) allows for indicating the color model and color space, and Rec. 601 is a generic standard for encoding YCbCr/YPbPr (whereas 709 is specific to HDTV 720p/1080i/1080p modes); so, as long as the display understands that the color model is YCbCr and the color space is Rec. 601, it should be able to accurately decode the video.