Speeding up archive operations on OS X and Linux

I work with a lot of virtual machine images and other big data sets that needed to be archived, or moved around. If I run a command like tar -cjf archive.tar.bz2 directory it will churn away on a single core.

Given that all my machines have a lot of cores, it would be better if the tar command made use of all available cores. Thankfully someone created a tool called pigz (pronounced "pig-zee") that will allow the use of these CPU cores.

There are 2 parts to the setup for pigz: get the utility, and force tar to use it. On my Mac I grabbed pigz from brew:

brew install pigz

Then I added an alias so that tar uses this by default:

alias tar='tar --use-compress-program=pigz'

Remember that you might not want to use an alias if your profile travels with you to en­vi­ron­ments where the use of all cores could create problems.