I don’t think that would be possible without at least a partial framebuffer, and I’m not sure if the OSSC’s current hardware is sufficient to add one. Right now it just stores a single line at a time to multiply, but windowboxing would require enough memory to buffer all the lines it receives while it’s busy drawing the borders.
You don’t need a framebuffer so much as enough memory for a larger line buffer, similar to how the Hi-Def NES handles the same problem (4x scale being a 960p image in a 1080p frame). But that would still appear to require more RAM than the OSSC has available.
In addition to additional RAM, that’d also require precise and flexible (not tailored for just single source) timing generation to achieve perfect framelock while outputting non-integer multiple of source scanlines.