Django 1.6和Celery 3.0内存泄漏

问题描述

将Django升级到1.6之后，我的芹菜工作者正在消耗RAM。
似乎分配给工作人员的内存在每个任务之后都不会释放，并且会增加。

After upgrading Django to 1.6, my celery worker is eating up RAM.Seems that the memory allocated for the workers isn't released and grows after every task.

相关设置：

# DB:
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'somedb',
        'USER': '',
        'PASSWORD': '',
        'HOST': 'localhost',
        'PORT': '',
    }
}


# CELERY SETTINGS:
CELERY_RESULT_BACKEND = 'redis://'
BROKER_URL = 'redis://'

相关软件包版本：

Django==1.6
celery==3.0.24
django-celery==3.0.23
billiard==2.7.3.34
kombu==2.5.16
redis==2.7.6

在我的本地环境（带有 DEBUG = False ）中手动运行该工作器以及在芹菜与Upstart一起运行的分阶段环境中都发生过。

Happens in both my local env (with DEBUG=False) running the worker manually and in a staging environment where celery is running with Upstart.

更新

1. 尝试设置 autocommit = False 没有成功。


2. 可能与Django版本升级无关，而与某些设置或第三者软件包有关，我必须升级才能切换到1.6。

1. Tried setting autocommit=False with no success.
2. Could be it's not related to the Django version upgrade, but to some setting or 3rd party package that I had to upgrade to make the switch to 1.6.

推荐答案

事实证明，内存泄漏并非直接由Django升级或Celery引起。

It turns out the memory leak was not directly caused by the Django upgrade or Celery.

挖掘发现，令人惊讶的是，由于我将 django-debug-toolbar 从 0.9.4 升级到了<$ c，所以发生了celery worker内存泄漏。 $ c> 0.11.0 （这是Django 1.6兼容性所必需的）。

After a lot of digging around I found that, surprisingly, the celery worker memory leak happens because I upgraded django-debug-toolbar from 0.9.4 to 0.11.0 (which is needed for Django 1.6 compatibility).

仍然不知道是什么导致了这个问题，或者为什么会这样仅发生在celery worker进程中，而不发生在应用程序服务器进程中（Gunicorn）。

Still no idea what exactly caused this issue, or why it only happens in the celery worker processes and not in the app server ones (Gunicorn).

从已安装的应用和中间件中删除 django-debug-toolbar 解决了该问题。至少是暂时的。

Removing django-debug-toolbar from the installed apps and middleware solves the issue. At least temporarily.

这篇关于Django 1.6和Celery 3.0内存泄漏的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！