1 2 3 4 5 File Explorer drag & drop ########## (112 MBps) Copy-Item ####### (82 MBps) Built in SFTP client ###### (70 MBps) Built in robocopy ## (25 MBps) WSL 2 rsync # (13 MBps)
In table form:
Tool Speed (MBps) Difference Drag and drop ~112 — Copy-Item ~82 -27% sftp ~70 -37% robocopy ~25 MBps -78% rsync (WSL 2) ~13 MBps -88%
I feel like I’m losing my mind.
It all started innocently enough with a misconfigured NAS. I’ve been making some investments into the homelab recently, both for professional reasons and personal curiosity; by now I consider a couple of long CAT 7 cables and some unmanaged 1 Gbps Ethernet switches to be some of the best money I’ve spent this year. Every device in our home has a cozy data autobahn to rush down to talk to every other device in our home, and what’s more, I’ve actually seen rsync copy files from my NAS to my Debian daily driver at 100+ MBps quite regularly.
But even after I un-mis-configured the NAS, I was surprised to find that rsync on Windows Subsystem for Linux pulled in data at a paltry 9 MBps, for an ideal use case (one large file). Removing the -z compression flag moved us up by 30-50%, which is commendable, but not the order of magnitude increase I was doubtfully hoping for.
Was something wrong with the connection? Getting up from the strandmon would have required physical movement, so instead I just took that same exact file via the File Explorer network share, and just Ctrl-C Ctrl-V’d it onto my desktop - and I was immediately hitting 111-112 MBps again! There goes that theory. The calls are coming from inside the house.
I don’t have much occasion to use these skills these days, but I actually became a somewhat proficient PowerShell user and Windows admin back in my first job. My curiosity was fully piqued by this point so I busted out Old Reliable - robocopy, rsync before rsync. It’s been built in since Windows 7 and handles an absurd number of weird edge cases due to its longevity, so you would expect it to be, well, fast, right. Like this is one of those Laws of Lindy Software that everyone supposedly knows; software that was created to be demanding on system resources 30 years ago is blazing fast to the point of instantaneous today, that’s why everyone loves Vim and Emacs over VS Code, etc.
Not so, apparently! robocopy doesn’t print its progress in seconds out of the box, but it can restart things from where they were last left off, so a few date && robocopy and Ctrl+C’s later, I had a pretty reliable estimate of… 25 MBps.
My jaw just about hit the floor, reader. It was here I began to realize that something was rotten in the state of Denmark. You probably shouldn’t be launching 10- or 100-hour long copy jobs between Windows machines in the first place if you have any other options, but on the other hand, civilization advances when we extend the number of important operations which we can perform without thinking about them, and I would sure like to know that the venerable old tool I would script such enormous jobs around wouldn’t do it at one-fifth the speed of doing it by hand. No, I didn’t investigate robocopy ’s myriad other options in detail, but in this situation I do not think I have to - we are comparing default behaviors against default behaviors! This is also why I didn’t go super deep into the WSL question; I should be able to assume that WSL comes shipped with sane defaults that don’t decimate my network connection because I didn’t set the MTU to 1337 by hand or what have you.
... continue reading