Question:
I have a task to check if new files are imported for the day in a shared location folder and alert if any duplicate files and no recursive check needed.
Below code displays all the file details with size which are 1 day old However I need only files with the same size as I cannot compare them using name.
1 2 3 4 5 |
$Files = Get-ChildItem -Path E:\Script\test | Where-Object {$_.CreationTime -gt (Get-Date).AddDays(-1)} $Files | Select-Object -Property Name, hash, LastWriteTime, @{N='SizeInKb';E={[double]('{0:N2}' -f ($_.Length/1kb))}} |
Answer:
I didn’t like the big DOS-like script answer written here, so here’s an idiomatic way of doing it for Powershell:
From the folder you want to find the duplicates, just run this simple set of pipes
1 2 3 4 5 6 7 8 9 |
Get-ChildItem -Recurse -File ` | Group-Object -Property Length ` | ?{ $_.Count -gt 1 } ` | %{ $_.Group } ` | Get-FileHash ` | Group-Object -Property Hash ` | ?{ $_.Count -gt 1 } ` | %{ $_.Group } |
Which will show all files and their hashes that match other files.
Each line does the following:
- get files
- from current directory (use
-Path $directory
otherwise) - recursively (if not wanted, remove
-Recurse
)
- from current directory (use
- group based on file size
- discard groups with less than 2 files
- grab all those files
- get hashes for each
- group based on hash
- discard groups with less than 2 files
- get all those files
Add | %{ $_.path }
to just show the paths instead of the hashes.
Add | %{ $_.path -replace "$([regex]::escape($(pwd)))",'' }
to only show the relative path from the current directory (useful in recursion).
For the question-asker specifically, don’t forget to whack in | Where-Object {$_.CreationTime -gt (Get-Date).AddDays(-1)}
after the gci
so you’re not comparing files you don’t want to consider, which might get very time-consuming if you have a lot of coincidentally same-length files in that shared folder.
Finally, if you’re like me and just wanted to find dupes based on name, as google will probably take you here too:
gci -Recurse -file | Group-Object name | Where-Object { $_.Count -gt 1 } | select -ExpandProperty group | %{ $_.fullname }