Powershell to display duplicate files

Question:

I have a task to check if new files are imported for the day in a shared location folder and alert if any duplicate files and no recursive check needed.

Below code displays all the file details with size which are 1 day old However I need only files with the same size as I cannot compare them using name.

Answer:

I didn’t like the big DOS-like script answer written here, so here’s an idiomatic way of doing it for Powershell:

From the folder you want to find the duplicates, just run this simple set of pipes

Which will show all files and their hashes that match other files.
Each line does the following:

  • get files
    • from current directory (use -Path $directory otherwise)
    • recursively (if not wanted, remove -Recurse)
  • group based on file size
  • discard groups with less than 2 files
  • grab all those files
  • get hashes for each
  • group based on hash
  • discard groups with less than 2 files
  • get all those files

Add | %{ $_.path } to just show the paths instead of the hashes.
Add | %{ $_.path -replace "$([regex]::escape($(pwd)))",'' } to only show the relative path from the current directory (useful in recursion).

For the question-asker specifically, don’t forget to whack in | Where-Object {$_.CreationTime -gt (Get-Date).AddDays(-1)} after the gci so you’re not comparing files you don’t want to consider, which might get very time-consuming if you have a lot of coincidentally same-length files in that shared folder.

Finally, if you’re like me and just wanted to find dupes based on name, as google will probably take you here too:

gci -Recurse -file | Group-Object name | Where-Object { $_.Count -gt 1 } | select -ExpandProperty group | %{ $_.fullname }

Source:

Powershell to display duplicate files by licensed under CC BY-SA | With most appropriate answer!

Leave a Reply