So I have a bunch of external drives plus my home PC which all have various combinations of mostly the same files all kept in a slightly different file structure. plus a bunch of other files that are unique to that drive.

Like for music I have 95% of the files duplicated in several different places but each place has a few extra / fewer albums and some albums might be in Various in one place and soundtrack in the other

I want to make sure that I have one copy of each and every file, but not three or four of everything in slightly different places. Just using windows copy/paste will not work as if the folder structure is slightly different it will just copy it accross

So I am not really looking for pure back-up software either, I guess I want something that will scan the files I have, recognize a new unique file and copy it across to a new master location.

I tried searching for file aggregation , but was not having any luck

Suggestions?

robocopy into a fresh location?

edit: good luck with the syntax though: https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy

The commands look scary indeed

<table>
<tbody>
<tr>
<td>/xx</td>
<td class=””>Excludes extra files and directories.</td>
</tr>
<tr>
<td>/xl</td>
<td class=””>Excludes “lonely” files and directories.</td>
</tr>
</tbody>
</table>

is it a lonely file??

Easy on linux, run md5sum on every file, then pipe the list through sort and uniq and from that rebuild a list of one copy of each unique file, which you then find and copy to the clean drive.

footflaps

Easy on linux, run md5sum on every file, then pipe the list through sort and uniq and from that rebuild a list of one copy of each unique file, which you then find and copy to the clean drive.

I’m a Linux programmer by trade and TBH that doesn’t sound that easy.

Well not compared to copying the lot to a single drive in windows, and running dupeguru. 🙂

Most likely there will be issues with escaping any weird characters (speech marks for example) in filenames with that approach too.

I’m a Linux programmer by trade and TBH that doesn’t sound that easy.

Under 50 lines of bash…..

Pretty trivial for bash programming.