CDC File Transfer

CDC File Transfer

Born from the ashes of Stadia, this repository contains tools for syncing and streaming files from Windows to Windows or Linux. The tools are based on Content Defined Chunking (CDC), in particular FastCDC, to split up files into chunks.

History

At Stadia, game developers had access to Linux cloud instances to run games. Most developers wrote their games on Windows, though. Therefore, they needed a way to make them available on the remote Linux instance.

As developers had SSH access to those instances, they could use scp to copy the game content. However, this was impractical, especially with the shift to working from home during the pandemic with sub-par internet connections. scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression.

To help this situation, we developed two tools, cdc_rsync and cdc_stream , which enable developers to quickly iterate on their games without repeatedly incurring the cost of transmitting dozens of GBs.

CDC RSync

cdc_rsync is a tool to sync files from a Windows machine to a Linux device, similar to the standard Linux rsync. It is basically a copy tool, but optimized for the case where there is already an old version of the files available in the target directory.

It quickly skips files if timestamp and file size match.

It uses fast compression for all data transfer.

... continue reading