The problem with DD is that it copies the whole disk, In reality, the disk could have 10GBs but that dump file has to be of the disk size, lets say 100GBs<\/p>\n\n\n\n
So, how do we get a dump file that is only around 10GBs in size.<\/p>\n\n\n\n
The answer is simple. Compressing a zero fill file is very efficient (almost nothing).<\/p>\n\n\n\n
So, frst we create a zero fill file. with the following command, i recommend you stop the fill while there is still a bit of space on the disk especially if the disk has a database running that could need to insert.. so stop the running fill with ctrl+c before you actually fill the whole disk<\/p>\n\n\n\n
cat \/dev\/zero > zero3.fill;sync;sleep 1;sync;\n<\/pre>\n\n\n\nAt this point, you can either delete the zero.fill file or not, It will not make a difference in the dump size, deleting is recommended, but it wont make much of a difference.<\/p>\n\n\n\n
Notes
sync flushed any remaining buffer in memory to the hard drive
If the process stops for any reason, keep the file already written and make a second one and a third and whatever it takes, do not delete the existing one, just make sure almost all of your disk’s free space is occupied by zero fill files.<\/p>\n\n\n\nNow, to DD and compression on the fly (So that you won’t need much space on the target drive)<\/p>\n\n\n\n
If you want to monitor the dump, you can use pv<\/p>\n\n\n\n
dd if=\/dev\/sdb | pv -s SIZEOFDRIVEINBYTES | pigz --fast > \/targetdrive\/diskimage.img.gz\n<\/pre>\n\n\n\nOr if you like, you can use parallel BZIP2 like so, in this example this is a 2TB hard drive<\/p>\n\n\n\n
dd if=\/dev\/sda | pv -s 2000398934016 | pbzip2 --best > \/somefolder\/thefile.img.bz2<\/pre>\n\n\n\nwithout the monitoring<\/p>\n\n\n\n
dd if=\/dev\/sdb | pigz --fast > \/targetdrive\/diskimage.img.gz\n<\/pre>\n\n\n\nNow, to dump this image back to a hard drive<\/p>\n\n\n\n
Note that using pigz for the decompression in this situation is not recommended, somthing along the lines of this<\/p>\n\n\n\n
DO NOT USE this one, use the one with gunzip<\/pre>\n\n\n\npigz -d \/hds\/www\/vzhost.img.gz | pv -s SIZEOFIMAGEINBYTES | dd of=\/dev\/sdd\n<\/pre>\n\n\n\nWill work, but it will decompress the file in place before sending it through the pipe, so the recommended way to do it on the fly is with gunzip, this is also true because there are no benefits from parallel gzip while decompressing<\/p>\n\n\n\n
gunzip -c \/hds\/www\/vzhost.img.gz | pv -s SIZEOFIMAGEINBYTES | dd of=\/dev\/sdb\n<\/pre>\n\n\n\nOr<\/p>\n\n\n\n
pigz -d \/hds\/www\/vzhost.img.gz | dd of=\/dev\/sdd\n<\/pre>\n\n\n\nMy records
The following are irrelevant to you, this is strictly for my records<\/p>\n\n\n\nmount -t ext4 \/dev\/sdb1 \/hds\n\ndd if=\/dev\/sdc | pv -s 1610612736000 | pigz --fast > \/hds\/www\/vzhost.img.gz\n<\/pre>\n\n\n\nOne that covers doing for part of a disk<\/p>\n\n\n\n
Assume i want to copy the first 120GB of a large drive where my windows partition lives, I want it compressed and i want the free space cleared<\/p>\n\n\n\n
first, in windows use SDELETE to zero empty space<\/p>\n\n\n\n
sdelete -z c:<\/pre>\n\n\n\nNow, mount the disk on a linux partition<\/p>\n\n\n\n
dd if=\/dev\/sdb bs=512 count=235000000 | pigz --fast > \/hds\/usb1\/diskimage.img.gz\ndd if=\/dev\/sdb bs=512 count=235000000 | pbzip2 > \/hds\/usb1\/diskimage.img.gz\n<\/pre>\n\n\n\nIf it is advanced format, you would probably do
dd if=\/dev\/sdb of=\/hds\/usb1\/firstpartofdisk.img bs=4096 count=29000000<\/p>\n\n\n\nor something like that<\/p>\n\n\n\n
Now, if we have a disk image with the extension (.bin.gz) and we want to extract it to a different directory, we can pipe it as follows<\/p>\n\n\n\n
gunzip -c \/pathto\/my_disk.bin.gz > \/targetdir\/my_disk.bin<\/pre>\n","protected":false},"excerpt":{"rendered":"The problem with DD is that it copies the whole disk, In reality, the disk could have 10GBs but that dump file has to be of the disk size, lets say 100GBs So, how do we get a dump file that is only around 10GBs in size. The answer is simple. Compressing a zero fill […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-466","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts\/466","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/comments?post=466"}],"version-history":[{"count":12,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts\/466\/revisions"}],"predecessor-version":[{"id":3444,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts\/466\/revisions\/3444"}],"wp:attachment":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/media?parent=466"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/categories?post=466"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/tags?post=466"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}