It looks better with the TV scaling because the N64 looks horrible and blocking. What you are seeing with the OSSC is more accurate to what the N64 is actually outputting. Some people like that, others (like yourself) find it to look quite horrible. Nearly all early 3D systems look pretty bad when scaled this way. The TV, as mentioned above, has a proper scaler that can interpolate the pixels instead of just doubling/tripling/etc them. This allows it to make the image look smoother and less blocky, but at the cost of additional input lag.
This is easy to replicate in Photoshop, as shown in this image: http://www.chrismadden.co.uk/inkline-press/create/create3/figs/6-18.html. Nearest neighbor is the rough equivalent of what the OSSC does. It provides very sharp lines, but a blocky image overall. The others look somewhat blurry up close, but when you back out, what you actually see are smoother lines.