且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

从目录中获取单个文件的最有效/最快速的方法

更新时间:2023-11-25 16:34:04

创建文件时,将最新文件的名称添加到文本文件中存储的列表中。当您要阅读/处理/删除文件时:


  1. 打开文本文件

  2. 设置文件名到列表顶部的名称。

  3. 从列表顶部删除名称

  4. 关闭文本文件

  5. 处理文件名。 / li>


What is the most efficient and fastest way to get a single file from a directory using Python?

More details on my specific problem:
I have a directory containing a lot of pregenerated files, and I just want to pick a random one. Since I know that there's no really efficient way of picking a random file from a directory other than listing all the files first, my files are generated with an already random name, thus they are already randomly sorted, and I just need to pick the first file from the folder.

So my question is: how can I pick the first file from my folder, without having to load the whole list of files from the directory (nor having the OS to do that, my optimal goal would be to force the OS to just return me a single file and then stop!).

Note: I have a lot of files in my directory, hence why I would like to avoid listing all the files to just pick one.

Note2: each file is only picked once, then deleted to ensure that only new files are picked the next time (thus ensuring some kind of randomness).

SOLUTION

I finally chose to use an index file that will store:

  • the index of the current file to be picked (eg: 1 for file1.ext, 2 for file2.ext, etc..)
  • the index of the last file generated (eg: 1999 for file1999.ext)

Of course, this means that my files are not generated with a random name anymore, but using a deterministic incrementable pattern (eg: "file%s.ext" % ID)

Thus I have a near constant time for my two main operations:

  • Accessing the next file in the folder
  • Counting the number of files that are left (so that I can generate new files in a background thread when needed).

This is a specific solution for my problem, for more generic solutions, please read the accepted answer.

Also you might be interested into these two other solutions I've found to optimize the access of files and directory walking using Python:

when creating the files add the name of the newest file to a list stored in a text file. When you want to read/process/delete a file:

  1. Open the text file
  2. Set filename to the name on the top of the list.
  3. Delete the name from the top of the list
  4. Close the text file
  5. Process filename.