Python垃圾收集有时在Jupyter Notebook中不起作用

问题描述

我使用一些Jupyter笔记本电脑时，RAM一直用光了，我似乎无法释放不再需要的内存.这是一个示例:

I'm constantly running out of RAM with some Jupyter Notebooks and I seem to be unable to release memory that is no longer needed. Here is an example:

import gc
thing = Thing()
result = thing.do_something(...)
thing = None
gc.collect()

您可以假定，thing使用大量内存来做某事，然后我不再需要它.我应该能够释放它使用的内存.即使它没有写入我可以从笔记本中访问的任何变量，垃圾回收器也无法正确释放空间.我发现的唯一解决方法是将result写入pickle，重新启动内核，从pickle加载result，然后继续.当运行长笔记本时，这确实很不方便.如何正确释放内存?

As you can presume, thing uses a lot of memory to do something and then I don't need it anymore. I should be able to release the memory it uses. Even though it doesn't write to any variables that I can access from my notebook, garbage collector isn't freeing up space properly. The only workaround I've found is writing result into a pickle, restarting kernel, loading result from pickle, and continuing. This is really inconvenient when running long notebooks. How can I free up memory properly?

推荐答案

此处存在许多问题.第一个是IPython(Jupyter在幕后使用的东西在看到类似Out[67]之类的东西时会保留对对象的附加引用.实际上，您可以使用该语法来调用该对象并对其执行某些操作，例如str(Out[67]).第二个问题是Jupyter似乎保留了自己对输出变量的引用，因此只能对IPython进行完全重置.但这与重新启动笔记本没有太大区别.

There are a number of issues at play here. The first is that IPython (what Jupyter uses behind the scenes keeps additional references to objects when you see something like Out[67]. In fact you can use that syntax to recall the object and do something with it. eg. str(Out[67]). The second problem is that Jupyter seems to be keeping its own reference of output variables, so only a full reset of IPython will work. But that's not much different to just restarting the notebook.

尽管有解决方案！我编写了一个可以运行的函数，该函数将清除所有变量，但明确要求保留的变量除外.

There is a solution though! I wrote a function that you can run that will clear all variables, except the ones you explicitly ask to keep.

def my_reset(*varnames):
    """
    varnames are what you want to keep
    """
    globals_ = globals()
    to_save = {v: globals_[v] for v in varnames}
    to_save['my_reset'] = my_reset  # lets keep this function by default
    del globals_
    get_ipython().magic("reset")
    globals().update(to_save)

您将以如下方式使用它:

You would use it like:

x = 1
y = 2
my_reset('x')
assert 'y' not in globals()
assert x == 1

下面，我写了一个笔记本，向您展示了幕后发生的事情以及如何使用weakref模块真正删除某些内容时的情况.您可以尝试运行它，看看它是否可以帮助您了解发生了什么.

Below I wrote a notebook that shows you a little bit of what is going on behind the scenes and how you can see when something has truly been deleted by using the weakref module. You can try running it to see if it helps you understand what is going on.

In [1]: class MyObject:
            pass

In [2]: obj = MyObject()

In [3]: # now lets try deleting the object
        # First, create a weak reference to obj, so we can know when it is truly deleted.
        from weakref import ref
        from sys import getrefcount
        r = ref(obj)
        print("the weak reference looks like", r)
        print("it has a reference count of", getrefcount(r()))
        # this prints a ref count of 2 (1 for obj and 1 because getrefcount
        # had a reference to obj)
        del obj
        # since obj was the only strong reference to the object, it should have been
        # garbage collected now.
        print("the weak reference looks like", r)

the weak reference looks like <weakref at 0x7f29a809d638; to 'MyObject' at 0x7f29a810cf60>
it has a reference count of 2
the weak reference looks like <weakref at 0x7f29a809d638; dead>

In [4]: # lets try again, but this time we won't print obj, will just do "obj"
        obj = MyObject()

In [5]: print(getrefcount(obj))
        obj

2
Out[5]: <__main__.MyObject at 0x7f29a80a0c18>

In [6]: # note the "Out[5]". This is a second reference to our object
        # and will keep it alive if we delete obj
        r = ref(obj)
        del obj
        print("the weak reference looks like", r)
        print("with a reference count of:", getrefcount(r()))

the weak reference looks like <weakref at 0x7f29a809db88; to 'MyObject' at 0x7f29a80a0c18>
with a reference count of: 7

In [7]: # So what happened? It's that Out[5] that is keeping the object alive.
        # if we clear our Out variables it should go away...
        # As it turns out Juypter keeps a number of its own variables lying around,
        # so we have to reset pretty everything.

In [8]: def my_reset(*varnames):
            """
            varnames are what you want to keep
            """
            globals_ = globals()
            to_save = {v: globals_[v] for v in varnames}
            to_save['my_reset'] = my_reset  # lets keep this function by default
            del globals_
            get_ipython().magic("reset")
            globals().update(to_save)

        my_reset('r') # clear everything except our weak reference to the object
        # you would use this to keep "thing" around.

Once deleted, variables cannot be recovered. Proceed (y/[n])? y

In [9]: print("the weak reference looks like", r)

the weak reference looks like <weakref at 0x7f29a809db88; dead>

这篇关于Python垃圾收集有时在Jupyter Notebook中不起作用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！