Takeout Champ
How to Identify Duplicate Files by Content
You export your Google account with Takeout, wait for the email, and download a pile of zip files. Inside are more zips, folders inside folders, and photos that seem to appear in three places. You want to know which files are actually the same. Sorting by name doesn't help, because the names don't tell you.
This is the standard problem with any large file collection that has been moved, exported, synced, or backed up more than once. The reliable fix is to identify duplicates by what the files contain, not by what they are called.
Why filenames and dates are unreliable
The obvious first move is to look for files with the same name. It fails in both directions.
- Same content, different names. A photo saved as
IMG_2041.jpgin one folder andIMG_2041 (1).jpgorvacation-final.jpgin another is still the same photo. Name matching misses it. - Same name, different content. Cameras and phones reuse numbering. Two completely different photos can both be called
IMG_0001.jpg. Name matching would flag them as duplicates, and deleting one would lose a real file.
File size and modified date have the same problem. Two files can share a size by coincidence. Dates change when a file is copied, extracted from an archive, or re-exported. These attributes are useful clues, but none of them proves two files are identical.
How content-based duplicate detection works
The approach is to read each file's bytes and compute a fingerprint, usually a hash. A hash function takes the whole contents of a file and produces a short, fixed-length value. If two files have the same contents, they produce the same value. If even one byte differs, the value will almost certainly be different.
That gives you a simple process:
- Walk through every file, including those inside nested archives.
- Compute a fingerprint for each one from its contents.
- Group files that share a fingerprint.
- Keep one from each group and treat the rest as duplicates.
Because the fingerprint ignores the filename, a renamed copy still lands in the same group as the original. And because it covers the full contents, two different photos with the same name are kept apart.
What this does and doesn't catch
Content fingerprinting finds true duplicates, meaning files whose contents are identical. It is not the same as finding photos that merely look alike. If you resize a picture or recompress it at a different quality, the bytes change, so it is a different file as far as a hash is concerned. That is usually what you want when cleaning up an archive. You are removing redundant copies, not making judgment calls about which similar photos you prefer.
Practical things to watch for
Even with a good method, a few details matter when you clean up a real archive.
Decide which copy to keep. Once files are grouped, something has to determine which one survives. Original timestamps and metadata matter here, especially for photos, where the date is part of how you find things later. Whatever process you use, make sure it doesn't wipe those out.
Expect files that can't be read. In any large export, a few files will be corrupted, truncated, or in a format your tool can't process. It's better to have those flagged than silently skipped or silently dropped.
Think about where the scanning happens. Hashing requires reading every file. For a Google Takeout export, that means your photos, email, contacts, and documents. Many online tools ask you to upload them first. If the archive is personal, it's worth checking whether the work can be done on your own machine instead.
Duplicates span sessions. Big exports often arrive as several zips, and you may not process them all in one sitting. If your tool forgets what it has already seen, a file in the second batch won't be recognized as a copy of one from the first.
How Takeout Champ handles this
Takeout Champ was built for the situation described at the start: a Google Takeout export that arrived as a sprawl of nested zips with no easy way to tell what's duplicated. It also accepts any zip, folder, or batch of files.
Here is how it deals with the points above:
- Content-based fingerprinting. It identifies duplicates by what's inside the files, not by filename. That means renamed and re-exported copies are caught, not just files that happen to share a name.
- Local processing. All scanning and hashing happens in your browser, on your device. Your files are never uploaded. What persists is a small local fingerprint, a hash plus metadata rather than the file itself, and you can clear it at any time.
- Memory across sessions. Because those fingerprints are kept locally, Takeout Champ remembers duplicates between sessions. This helps when your export came in multiple parts and you process them separately.
- Original timestamps and metadata preserved. Deduplicating doesn't strip the dates and details that make a photo archive searchable.
- Sorting into categories. Files are organized into Photos, Videos, Gmail, Contacts, Calendar, Documents, and Audio, so the result is something you can browse.
- Flagging unprocessable files. If a file can't be handled, it is flagged rather than quietly ignored.
- A clean zip at the end. The organized, duplicate-free result is repackaged into a single organized zip.
It's free to process up to 3GB, and there's no account to create.
Wrapping up
If you want to know which files are really the same, stop comparing names and start comparing contents. A content hash is a simple, dependable way to do it, and it works the same whether you have a few hundred files or a Takeout export with nested archives. Whatever tool you use, check that it keeps your original dates, flags what it can't read, and doesn't require sending your private files somewhere else to do the job.