All 4 SD card data pins are wired to FPGA so one could get good read/write performance with a suitable controller, but bandwidth probably isn’t the biggest worry. I think erase speed is the bottleneck since it would block other SD card accesses – even if you managed to time erases to vblank, I doubt it’d be sufficient length even for the smallest sector erase. Another issue is SD card retention as such flash is typically specced to last only ~100k erase cycles.
You might be able to sitestep both issues with a massive SD card, though. With a good lossless compression scheme and low-res content you could maybe run the system for good 5 hours before a 256GB microSD would get full :). That’s assuming changes between read and write modes wouldn’t incur notable latency penalties which might also break the scheme.