且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

如何在AWS S3存储区中查找重复的文件?

更新时间:2022-10-19 12:50:54

在Amazon S3中没有find duplicate命令。



但是,您执行以下操作:




  • 检索对象列表在桶中

  • 查找具有相同的对象 ETag (checksum)和大小



他们(非常可能)是重复的对象。


Is there a way to recursively find duplicate files in an Amazon S3 bucket? In a normal file system, I would simply use:

fdupes -r /my/directory

There is no "find duplicates" command in Amazon S3.

However, you do do the following:

  • Retrieve a list of objects in the bucket
  • Look for objects that have the same ETag (checksum) and Size

They would (extremely likely) be duplicate objects.