本文介绍了当多个进程尝试同时写入文件然后从文件读取时如何防止竞争条件的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我有以下代码(为了清楚起见而简化):

I have the following code (simplified for clarity):

import os
import errno
import imp


lib_dir = os.path.expanduser('~/.brian/cython_extensions')
module_name = '_cython_magic_5'
module_path = os.path.join(lib_dir, module_name + '.so')
code = 'some code'

have_module = os.path.isfile(module_path)
if not have_module:
    pyx_file = os.path.join(lib_dir, module_name + '.pyx')

    # THIS IS WHERE EACH PROCESS TRIES TO WRITE TO THE FILE.  THE CODE HERE
    # PREVENTS A RACE CONDITION.
    try:
        fd = os.open(pyx_file, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
    except OSError as e:
        if e.errno == errno.EEXIST:
            pass
        else:
            raise
    else:
        os.fdopen(fd, 'w').write(code)

# THIS IS WHERE EACH PROCESS TRIES TO READ FROM THE FILE.  CURRENTLY THERE IS A
# RACE CONDITION.
module = imp.load_dynamic(module_name, module_path)

(上面的一些代码是从这个答案借来的.)

(Some of the above code is borrowed from this answer.)

当同时运行多个进程时,此代码仅导致一个进程打开并写入 pyx_file(假设 pyx_file 尚不存在).问题在于,当这个进程正在写入 pyx_file 时,其他进程尝试加载 pyx_file -- 在后面的进程中会引发错误,因为在它们读取时 pyx_filecode>pyx_file,不完整.(具体来说,会引发 ImportErrors,因为进程正在尝试导入文件的内容.)

When several processes are run at once, this code causes just one to open and write to pyx_file (assuming pyx_file does not already exist). The problem is that as this process is writing to pyx_file, the other processes try to load pyx_file -- errors are raised in the latter processes, because at the time they read pyx_file, it's incomplete. (Specifically, ImportErrors are raised, because the processes are trying to import the contents of the file.)

避免这些错误的最佳方法是什么?一种想法是让进程在 while 循环中不断尝试导入 pyx_file 直到导入成功.(这个解决方案似乎不太理想.)

What's the best way to avoid these errors? One idea is to have the processes keep trying to import pyx_file in a while loop until the import is successful. (This solution seems suboptimal.)

推荐答案

这样做的方法是每次打开它时都使用排他锁.写入器在写入数据时持有锁,而读取器会阻塞,直到写入器通过 fdclose 调用释放锁.如果文件被部分写入并且写入过程异常退出,这当然会失败,因此如果无法加载模块,应该显示适当的错误删除文件:

The way to do this is to take an exclusive lock each time you open it. The writer holds the lock while writing data, while the reader blocks until the writer releases the lock with the fdclose call. This will of course fail if the file has been partially written and the writing process exits abnormally, so a suitable error to delete the file should be displayed if the module can't be loaded:

import os
import fcntl as F

def load_module():
    pyx_file = os.path.join(lib_dir, module_name + '.pyx')

    try:
        # Try and create/open the file only if it doesn't exist.
        fd = os.open(pyx_file, os.O_CREAT | os.O_EXCL | os.O_WRONLY):

        # Lock the file exclusively to notify other processes we're writing still.
        F.flock(fd, F.LOCK_EX)
        with os.fdopen(fd, 'w') as f:
            f.write(code)

    except OSError as e:
        # If the error wasn't EEXIST we should raise it.
        if e.errno != errno.EEXIST:
            raise

    # The file existed, so let's open it for reading and then try and
    # lock it. This will block on the LOCK_EX above if it's held by
    # the writing process.
    with file(pyx_file, "r") as f:
        F.flock(f, F.LOCK_EX)

    return imp.load_dynamic(module_name, module_path)

module = load_module()

这篇关于当多个进程尝试同时写入文件然后从文件读取时如何防止竞争条件的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

05-28 19:01
查看更多