{"id":824,"date":"2017-09-24T05:10:06","date_gmt":"2017-09-24T05:10:06","guid":{"rendered":"http:\/\/www.tech-g.com\/?p=824"},"modified":"2017-09-24T05:10:06","modified_gmt":"2017-09-24T05:10:06","slug":"finding-duplicate-files-on-nas-storage","status":"publish","type":"post","link":"https:\/\/www.voodoo.business\/blog\/2017\/09\/24\/finding-duplicate-files-on-nas-storage\/","title":{"rendered":"Finding duplicate files on NAS storage"},"content":{"rendered":"<p>I have this bad habit of copying things so many times when modifying them, when that is a large database, we talking many gigas, so here is a script to find those duplicate files among many hard drives and telling you which ones are duplicates, moving and deleting and symbolic linking is done manually after.<\/p>\n<p>1- this script is PHP-CLI, so make sure that is installed on your computer<br \/>\n2- this script runs the find command, make sure it can execute that program<br \/>\n3- you run the script with the path parameter, but will need to edit the script to change the 1GB size i have hard coded<\/p>\n<p>What this script does is<\/p>\n<p>1- find files with size greater than 1GB (find \/hds -size +1G)<br \/>\n2- Store the files in database with size<br \/>\n3- retrieve the files ordered by size<br \/>\n4- if 2 files have exactly the same size, calculates MD5sum for the first MB of the file<br \/>\n5- If the MD5 of the first MB of the files are the same, calculate the whole MD5<br \/>\n6- If they turn out to be duplicates, they are printed to the command line<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I have this bad habit of copying things so many times when modifying them, when that is a large database, we talking many gigas, so here is a script to find those duplicate files among many hard drives and telling you which ones are duplicates, moving and deleting and symbolic linking is done manually after. [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-824","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts\/824","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/comments?post=824"}],"version-history":[{"count":1,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts\/824\/revisions"}],"predecessor-version":[{"id":825,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts\/824\/revisions\/825"}],"wp:attachment":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/media?parent=824"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/categories?post=824"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/tags?post=824"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}