It might have something to do with the horizontal blur applied to most N64 games. I imagine vertical scanlines help because they help break up the blur. You didn’t say which RGB mod you have in your N64, but, if you have boards by Tim Worthington or borti, you should be able to flash a firmware that has VI De-blur functionality; that should help clear up the image. (If you have an UltraHDMI, you can enable it in the menu.)
Your TV’s scaler is an actual scaler, not a line multiplier like the OSSC; so, while the OSSC is designed to simply multiply the size of a given pixel by 2/3/4/5x, which results in the blockiness but has no perceivable lag, your TV’s scaler is designed to enlarge the image to fit its panel while smoothing out those rough edges, because most people who need that scaling are using it for video content and 3D gaming, where simple line-doubling doesn’t make sense.