ts_zip: Text Compression using Large Language Models
ts_zip
A GPU is necessary to get a reasonable speed. 4 GB of RAM is required.
It is slower than conventional compressors (compression and decompression speed: up to 1 MB/s on a RTX 4090).
Only text files are supported. Binary files won't be compressed much. The currently used language model (RWKV 169M v4) was trained mostly on English texts. Other languages are supported including source code.
It is experimental so no backward compability should be expected between the various versions.
See also ts_sms which is optimized for the compression of small messages.
Compression Ratio
The compression ratio is given in bits per byte (bpb).
File Original size
... continue reading