< Go Back

Archive, Compress & Decompress Huge Files via zstd, tar, split and pv

Extreme speeds of efficiently backing up and organizing our information!

The following tutorial is how we backup our drives in an efficient means with zstd and tar via a Linux operating system or powershell if you are using Windows.

We folded/archived, compressed, decompressed and extracted 180GB of data within ~30 minutes!

Package Requirements:

tar
pv
zstd
dosfsck
cat
split

optional: Check the filesize in chunks for the FAT32 drive


sudo dosfsck -v -n /dev/sda1


**File size is 4294967295 bytes, cluster chain length is 0 bytes.**

Folding and Compressing the files into one file efficiently

Option 1: A simpler way of compressing a tar is


pv Desktop.tar | zstd >> Desktop.zst

Option 2: Folding, compressing and Splitting the tar.zst into files to be transported between drives

We make use of tar with the -I option and zstd, while splitting and visualizing a progress counter with pv.


tar -I zstd -cf - /home | (pv -p --timer --rate --bytes | split --bytes=4294967295 - /run/media/Backups/System/Linux/2024/Home.backup.tar.zst)

Option 3: Another example of backing up multiple files/folders in our home directory:


tar -I zstd -cf - Documents/ Downloads/ Music/ Pictures/ Public/ Templates/ Videos/ .ssh/ .bashrc .gitconfig .nvidia-settings-rc | (pv -p --timer --rate --bytes | split --bytes=4294967295 - /run/media/Backups/System/Linux/2024/Home.backup.tar.zst)

Decompress and Unfold All Parts

The following also makes use of cat that injects all the parts into zstd [SEE REFERENCES]

Option 1: Decompress multiple .zst files into a tar


cat *.zst* | zstd -d > /home/odinzu/dest.tar

Option 2: A more complex command for FAT32 drives

This is useful if your /home directory isn't big enough to fit the tar file; so we decompress the *.zst's* and *split* to a FAT32 larger drive with enough space => then, we extract the .tar files to the smaller drive i.e. /home.

Make a new directory for the part.tar* files


mkdir tar-archives && cd tar-archives

then, run from current directory and be sure paths are correct i.e. Backups/tar-archives/


cat ../Home*.zst* | zstd -d | (pv -p --timer --rate --bytes | split --bytes=4294967295 - ./home.part.tar)

Next, we extract multiple .tar files into our /home directory from our external drive

So be in the directory you are extracting to i.e. /home/username/, then do the following:


cat /run/media/Backups/System/Linux/Arch/username/tar-archives/home*.tar* | (pv | tar -xf - -C /home/username/)

Option 3: for other use cases that are less complicated

Extract the archive into specific directory


tar -xvf dest.tar -C /opt/files

References

Github Zstd Issue: https://github.com/facebook/zstd/issues/4014#issuecomment-2032786590

02 Apr 2024