4x faster network file sync with rclone (vs rsync)
For the past couple years, I have transported my 'working set' of video and project data to and from work on an external Thunderbolt NVMe SSD.
But it's always been slow when I do the sync. In a typical day, I may generate a new project folder with 500-1000 individual files, and dozens of them may be 1-10 GB in size.
The Thunderbolt drive I had was capable of well over 5 GB/sec, and my 10 Gbps network connection is capable of 1 GB/sec. I even upgraded my Thunderbolt drive to Thunderbolt 5 lately... though that was not the bottleneck.
I used the following rsync command to copy files from a network share mounted on my Mac to the drive (which I call "Shuttle"):
rsync -au --progress --stats /Volumes/mercury/* /Volumes/Shuttle/Video_Projects
mercury is so named because it's a fast NVMe-backed NAS volume on my Arm NAS (all my network volumes are named after celestial bodies).
As a test, I deleted one of the dozen or so active projects off my 'Shuttle' drive, and ran my rsync copy:
$ time rsync -au --progress --stats /Volumes/mercury/* /Volumes/Shuttle/Video_Projects Radxa Orion O6/ Radxa Orion O6/.DS_Store 6148 100% 4.80MB/s 00:00:00 (xfer#1, to-check=1582/3564) Radxa Orion O6/Micro Center Visit Details.pages 141560 100% 9.83MB/s 00:00:00 (xfer#2, to-check=1583/3564) Radxa Orion O6/Radxa Orion O6.md 19817 100% 1.89MB/s 00:00:00 (xfer#3, to-check=1584/3564) Radxa Orion O6/BIOS and Images/ Radxa Orion O6/BIOS and Images/orion-o6-bios-0.2.2-1.zip 3916964 100% 83.32MB/s 00:00:00 (xfer#4, to-check=1586/3564) Radxa Orion O6/BIOS and Images/orion-o6-bios-9.0.0-apr-11.7z 4341505 100% 112.62MB/s 00:00:00 (xfer#5, to-check=1587/3564) Radxa Orion O6/Scratch/ Radxa Orion O6/Scratch/bios 9.0.0 - screen acpi and debian 12 attempt.mp4 254240026 100% 114.11MB/s 00:00:02 (xfer#6, to-check=1589/3564) ... Number of files: 3564 Number of files transferred: 122 Total file size: 244284287846 B Total transferred file size: 62947785101 B Unmatched data: 62947785101 B Matched data: 0 B File list size: 444318 B File list generation time: 9.155 seconds File list transfer time: 0.078 seconds Total sent: 62955918871 B Total received: 2728 B sent 62955918871 bytes received 2728 bytes 128990035 bytes/sec total size is 244284287846 speedup is 3.88 real 8:17.57 user 3:13.14 sys 2:45.45
The full copy took over 8 minutes, for a total of about 59 GiB of files copied. There are two problems:
... continue reading