Ever seen a program that will look for duplicate files?

You can talk about almost anything that you want to on this board.

Moderator: Moderators

Post Reply
User avatar
Marscaleb
Posts: 207
Joined: Fri Sep 11, 2015 10:39 am
Contact:

Ever seen a program that will look for duplicate files?

Post by Marscaleb » Sat Mar 07, 2020 2:37 pm

So I've got a bunch of files I've copied over from an old computer. Most - but not all - of these files already exist on my current computer somewhere else. And going through these files manually is going to take some time...

Anyone ever seen a program that can scan through some files in different folders and look for duplicate files?
I'd like to set something up to look at all the files in one directory and see if there is anything in another that matches.
Even if it was just checking the file names, it would give me a huge leg-up just to know where to look to find that file's brother.

User avatar
Fisher
Posts: 1070
Joined: Sat Jul 04, 2015 9:58 am
Location: -29.794229 -55.795374

Re: Ever seen a program that will look for duplicate files?

Post by Fisher » Sat Mar 07, 2020 3:48 pm

I use fslint on my Ubuntu machine for quite some time with excellent results.
For Windows, I remember that there was a tool that did some thing similar, I think it was on Win98 (the last windows I used at home) installation CD.
Hope this helps. :-)

tepples
Posts: 21980
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Ever seen a program that will look for duplicate files?

Post by tepples » Sat Mar 07, 2020 6:34 pm

Back in 2014, I wrote a Python program to search for duplicate files in a directory and its subdirectories. Today I ported it to Python 3 now that Python 2 has reached its end of support.
Attachments
dedupe.py.zip
(2.34 KiB) Downloaded 70 times

Oziphantom
Posts: 827
Joined: Tue Feb 07, 2017 2:03 am

Re: Ever seen a program that will look for duplicate files?

Post by Oziphantom » Sat Mar 07, 2020 10:42 pm

any fill diffing tool worth its salt... even WinDiff does it.

User avatar
tokumaru
Posts: 11702
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: Ever seen a program that will look for duplicate files?

Post by tokumaru » Sun Mar 08, 2020 10:32 am

I'm pretty sure I've used WinMerge for this in the past. There's also a portable version, in case you're like me and don't like to litter your Windows installation.

User avatar
Dwedit
Posts: 4309
Joined: Fri Nov 19, 2004 7:35 pm
Contact:

Re: Ever seen a program that will look for duplicate files?

Post by Dwedit » Sun Mar 08, 2020 11:06 am

I usually use "Duplicate File Cleaner" (an older version that was still freeware), but WinMerge is great for comparing pure text files.
Here come the fortune cookies! Here come the fortune cookies! They're wearing paper hats!

User avatar
Marscaleb
Posts: 207
Joined: Fri Sep 11, 2015 10:39 am
Contact:

Re: Ever seen a program that will look for duplicate files?

Post by Marscaleb » Sun Mar 15, 2020 9:51 pm

So after Fisher suggested fslint I did a search for equivalent programs in windows and was directed to "dupeGuru." It didn't work quite like I expected and just this weekend got around to reading through the documentation to get it figured out. And I don't much care for it's interface; I wound up sorting through a bunch of files manually.

Tepples, just from reading the description in the header, that is a handy tool that looks much smarter than DupeGuru. How do I run a python script? I'm a little disappointed that all the files have to be in the same directory though; that means I'm gonna have to copy these files over to the same directory and scan basically my whole hard drive.

Just taking a cursory glance at Winmerge and Windiff, they look like they are more designed for comparing the content within the files and showing them to the user. I've just got a bunch of backups from old drives and I'm really just looking for any file that I don't still have. So it's not about looking at two text files so I can see what changed, but just telling "hey, these files in this directory can be found in these locations" and then I guess an easy way to delete the ones I don't need so I can easily see the ones that I don't have.
Maybe that's what they do. Maybe that's what DupeGuru does but I'm not giving it a proper chance.
Maybe the real problem is that I wind up only working on this when it's late at night and I'm half asleep and can't think straight, like right now.

tepples
Posts: 21980
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Ever seen a program that will look for duplicate files?

Post by tepples » Sun Mar 15, 2020 10:10 pm

First you have to install Python. If using Windows, install it from python.org.

The rest depends on how much you know of things like the command prompt and the current working directory.

User avatar
Dwedit
Posts: 4309
Joined: Fri Nov 19, 2004 7:35 pm
Contact:

Re: Ever seen a program that will look for duplicate files?

Post by Dwedit » Mon Mar 16, 2020 4:02 pm

If you're looking for things like freeware utilities *in general*, go to http://fileforum.betanews.com/ and use the Search feature. After entering your search terms, you can turn on the Freeware or Open Source only filter. But as usual, watch out for "bundled software" that hooks onto the installers.
Here come the fortune cookies! Here come the fortune cookies! They're wearing paper hats!

93143
Posts: 1146
Joined: Fri Jul 04, 2014 9:31 pm

Re: Ever seen a program that will look for duplicate files?

Post by 93143 » Mon Mar 16, 2020 5:10 pm

Marscaleb wrote:
Sun Mar 15, 2020 9:51 pm
I've just got a bunch of backups from old drives and I'm really just looking for any file that I don't still have.
So they'd have the same name? Doesn't Windows automatically figure that out for you if you paste them into the same directory?

Post Reply