Hmm. That programmer resembles the one from Seeedstudio: Atmel AVRISP STK500 USB ISP Programmer, which is also being sold by Digikey. According to the PDF manual available on Digikey, the programmer type should be set to STK500. This may or may not have an effect.
What drivers are you using for the programmer? Have you tried the libusb32 or libusb-K driver (using Zadig)?
The voltage should be set to 5V. Having target mode off is mainly to protect the programmer itself from burning out from interaction with the PSU. If there is no jumper for it, you can check if the 5V pin outputs any power (you try lighting up an LED through a resistor with it, for example).
Using Atmel/Microchip Studio should be fine, but you may want to try reading the old firmware off the Koryuu just to see that the programmer works reliably. If that works without any errors, flashing a new firmware image should as well. As long as you do not change any fuse settings, bricking the MCU is very unlikely.