{"id":439,"date":"2014-11-10T08:07:45","date_gmt":"2014-11-10T08:07:45","guid":{"rendered":"http:\/\/www.tech-g.com\/?p=439"},"modified":"2025-12-26T15:35:04","modified_gmt":"2025-12-26T15:35:04","slug":"tar-and-compress-directory-on-the-fly-with-multi-threading","status":"publish","type":"post","link":"https:\/\/www.voodoo.business\/blog\/2014\/11\/10\/tar-and-compress-directory-on-the-fly-with-multi-threading\/","title":{"rendered":"Tar and compress directory on the fly with multi threading"},"content":{"rendered":"<p>There is not much to it. the tar command piped into any compression program of your choice.<\/p>\n<p>For speed, rather than using gzip, you can use pigz (to employ more processors \/ processor cores), or pbzip2 which is slower but compresses more<\/p>\n<p>cd to the directory where your folder is in <\/p>\n<p>then<\/p>\n<p>tar -c mysql | pbzip2 -vc > \/hds\/dbdirzip.tar.bz2<\/p>\n<p>for more compression<br \/>\ntar -c mysql | pbzip2 -vc -9 > \/hds\/dbdirzip.tar.bz2 <\/p>\n<p>for more compression and to limit CPUs to 6 instead of 8, or 3 instead of 4, or whatever you want to use, since the default is to use all cores<br \/>\ntar -c mysql | pbzip2 -vc -9 -p6 > \/hds\/dbdirzip.tar.bz2 <\/p>\n<p>tar cvf &#8211; mysql | pigz -9 > \/hds\/dbdirzip.tar.gz<\/p>\n<p>Or to limit the number of processors to 6 for example<br \/>\ntar cvf &#8211; mysql | pigz -9 -p6 > \/hds\/dbdirzip.tar.gz<\/p>\n<p>Now, if you want to compress a single file to a different directory<\/p>\n<p>pbzip2 -cz somefile > \/another\/directory\/compressed.bz2<\/p>\n","protected":false},"excerpt":{"rendered":"<p>There is not much to it. the tar command piped into any compression program of your choice. For speed, rather than using gzip, you can use pigz (to employ more processors \/ processor cores), or pbzip2 which is slower but compresses more cd to the directory where your folder is in then tar -c mysql [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[219],"tags":[],"class_list":["post-439","post","type-post","status-publish","format-standard","hentry","category-file-compression"],"_links":{"self":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts\/439","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/comments?post=439"}],"version-history":[{"count":6,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts\/439\/revisions"}],"predecessor-version":[{"id":707,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts\/439\/revisions\/707"}],"wp:attachment":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/media?parent=439"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/categories?post=439"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/tags?post=439"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}