问题描述
我特别需要使用with open语句来打开文件，因为我需要一起打开几百个文件并使用K-way merge进行合并.我了解，理想情况下，我应该将K保持在较低水平，但是我没有预见到这个问题.
I specifically need to use with open statement for opening the files, because I need to open a few hundred files together and merge them using K-way merge. I understand, ideally I should have kept K low, but I did not foresee this problem.
由于我有一个截止日期，因此现在不能从头开始.因此，在这一点上，我需要非常快速的I/O，该I/O不能将文件的整个/巨大部分存储在内存中(因为有成百上千个文件，每个文件约10MB).我只需要一次阅读一行即可进行K路合并.减少内存使用是我目前的主要重点.
Starting from scratch is not an option now as I have a deadline to meet. So at this point, I need very fast I/O that does not store the whole/huge portion of file in memory (because there are hundreds of files, each of ~10MB). I just need to read one line at a time for K-way merge. Reducing memory usage is my primary focus right now.
我了解到with open是最有效的技术，但是我无法理解如何在单个with open语句中将所有文件一起open.对不起我的初学者无知！
I learned that with open is the most efficient technique, but I cannot understand how to open all the files together in a single with open statement. Excuse my beginner ignorance!
更新:此问题已解决.事实证明，问题根本不在于我如何打开文件.我发现过多的内存使用是由于低效率的垃圾回收所致.我根本没有使用with open.我使用常规的f=open()和f.close().垃圾收集挽救了这一天.
Update: This problem was solved. It turns out the issue was not about how I was opening the files at all. I found out that the excessive memory usage was due to inefficient garbage collection. I did not use with open at all. I used the regular f=open() and f.close(). Garbage collection saved the day.
推荐答案
使用内置的 contextmanger 函数修饰符将"with语句上下文管理器的工厂函数"定义为文档状态.例如:
It's fairly easy to write your own context manager to handle this by using the built-in contextmanger function decorator to define "a factory function for with statement context managers" as the documentation states. For example:
from contextlib import contextmanager
@contextmanager
def multi_file_manager(files, mode='rt'):
""" Open multiple files and make sure they all get closed. """
files = [open(file, mode) for file in files]
yield files
for file in files:
file.close()
filenames = 'file1', 'file2', 'file3'
with multi_file_manager(filenames) as files:
a = files[0].readline()
b = files[2].readline()
...
这篇关于如何使用"with open"打开多个文件(事先未知的文件数)?陈述?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！