内存泄漏时使用字符串

内存泄漏时使用字符串

关注
发信
关注(28)粉丝(399)

内存泄漏时使用字符串< Python中128KB？

本文介绍了内存泄漏时使用字符串&lt; Python中128KB？的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述 29岁程序员，3月因学历无情被辞！ 原标题：内存泄露打开文件<在Python中是128KB？ 原始问题 我在运行Python脚本时发现内存泄漏。这是我的脚本： 导入sys 导入时间 类MyObj（object）： def __init __（self，filename）：开放（文件名）为f： self.att = f.read（） def myfunc（文件名）： mylist = [MyObj（文件名）for x in xrange（100）] len（mylist） return [] def main（）： filename = sys.argv [1] myfunc（文件名） time.sleep（3600） if __name__ =='__main__'： main（） 主函数调用 myfunc（），它创建一个包含100个对象的列表，每个对象打开并且读取一个文件。从 myfunc（）返回后，我希望从100-条目列表和的内存读取文件被释放，因为它们不再被引用。但是，当我使用 ps 命令检查内存使用情况时，Python进程使用大约10,000 KB 的内存多于从一个Python进程运行脚本中有第12行和第13行注释掉。 奇怪的是，内存泄漏（如果是这样的话）似乎只发生文件< ;大小128KB。我创建了一个bash脚本来运行这个脚本，其文件大小为，大小从1KB到200KB，当文件大小达到128KB时，内存增加停止。 这里是bash脚本： $ $ $ $ echo $ PID RSS S TTY TIME COMMAND> output.txt for i in`seq 1 200`; 做 python debug_memory.pydata / stuff _ $ {i} K.txt& pid = $！ 睡眠0.1 ps -e -O rss | grep $ pid | grep -v grep>> output.txt kill $ pid 完成 以下是bash脚本： PID RSS S TTY TIME COMMAND 28471 5552 S pts / 16 00:00:00 python debug_memory .py data / stuff_1K.txt 28477 5656 S pts / 16 00:00:00 python debug_memory.py data / stuff_2K.txt 28483 5756 S pts / 16 00:00:00 python debug_memory.py data / stuff_3K.txt 28488 5852 S pts / 16 00:00:00 python debug_memory.py data / stuff_4K.txt 28494 5952 S pts / 16 00:00:00 python debug_memory.py data / stuff_5K.txt 28499 6052 S pts / 16 00:00:00 python debug_memory.py data / stuff_6K.txt 28505 6156 S pts / 16 00:00:00 python debug_memory.py data / stuff_7K。 txt 28511 6256 S pts / 16 00:00:00 python debug_memory.py data / stuff_8K.txt 28516 6356 S pts / 16 00:00:00 python debug_memory.py data / stuff_9K.txt 28522 6452 S pts / 16 00:00:00 python debug_memory.py data / stuff_10K.txt 28527 6552 S pts / 16 00:00:00 python debug _memory.py data / stuff_11K.txt 28533 6656 S pts / 16 00:00:00 python debug_memory.py data / stuff_12K.txt 28539 6756 S pts / 16 00:00:00 python debug_memory。 py data / stuff_13K.txt 28544 6852 S pts / 16 00:00:00 python debug_memory.py data / stuff_14K.txt 28550 6952 S pts / 16 00:00:00 python debug_memory.py data /stuff_15K.txt 28555 7056 S pts / 16 00:00:00 python debug_memory.py data / stuff_16K.txt 28561 7156 S pts / 16 00:00:00 python debug_memory.py data / stuff_17K .txt 28567 7252 S pts / 16 00:00:00 python debug_memory.py data / stuff_18K.txt 28572 7356 S pts / 16 00:00:00 python debug_memory.py data / stuff_19K.txt 28578 7452 S pts / 16 00:00:00 python debug_memory.py data / stuff_20K.txt 28584 7556 S pts / 16 00:00:00 python debug_memory.py data / stuff_21K.txt 28589 7652 S pts / 16 00:00:00 python debug_memory.py data / stuff_22K.txt 28595 7756 S pts / 16 00:00:00 python debug_memory.py data / stuff_23K.txt 28600 7852 S pts / 16 00:00:00 python debug_memory.py data / stuff_24K.txt 28606 7952 S pts / 16 00:00:00 python debug_memory.py data / stuff_25K.txt 28612 8052 S pts / 16 00:00:00 python debug_memory.py data / stuff_26K.txt 28617 8152 S pts / 16 00:00:00 python debug_memory.py data / stuff_27K.txt 28623 8252 S pts / 16 00： 00:00 python debug_memory.py data / stuff_28K.txt 28629 8356 S pts / 16 00:00:00 python debug_memory.py data / stuff_29K.txt 28634 8452 S pts / 16 00:00： 00 python debug_memory.py data / stuff_30K.txt 28640 8556 S pts / 16 00:00:00 python debug_memory.py data / stuff_31K.txt 28645 8656 S pts / 16 00:00:00 python debug_memory.py data / stuff_32K.txt 28651 8756 S pts / 16 00:00:00 python debug_memory.py data / stuff_33K.txt 28657 8856 S pts / 16 00:00:00 python debug_memory。 py data / stuff_34K.txt 28662 8956 S pts / 16 00:00:00 python debug_memory.py data / stuff_35K.txt 28668 9056 S pts / 16 00:00:00 python debug_memory.py data / stuff_36K .txt 28674 9156 S pts / 16 00:00:00 python debug_memory.py data / stuff_37K.txt 28679 9256 S pts / 16 00:00:00 python debug_memory.py data / stuff_38K.txt 28685 9352 S pts / 16 00:00:00 python debug_memory.py data / stuff_39K.txt 28691 9452 S pts / 16 00:00:00 python debug_memory.py data / stuff_40K.txt 28696 9552 S pts / 16 00:00:00 python debug_memory.py data / stuff_41K.txt 28702 9656 S pts / 16 00:00:00 python debug_memory.py data / stuff_42K.txt 28707 9756 S pts / 16 00:00:00 python debug_memory.py data / stuff_43K.txt 28713 9852 S pts / 16 00:00:00 python debug_memory.py data / stuff_44K.txt 28719 9952 S pts / 16 00:00:00 python debug_memory.py data / stuff_45K.txt 28724 10052 S pts / 16 00:00:00 python debug_memory.py data / stuff_46K.txt 28730 10156 S pts / 16 00:00:00 python debug_memory.py data / stuff_47K.txt 28739 10256 S pts / 16 00:00:00 python debug_memory.py data / stuff_48K.txt 28746 10352 S pts / 16 00:00:00 python deb ug_memory.py data / stuff_49K.txt 28752 10452 S pts / 16 00:00:00 python debug_memory.py data / stuff_50K.txt 28757 10556 S pts / 16 00:00:00 python debug_memory。 py data / stuff_51K.txt 28763 10656 S pts / 16 00:00:00 python debug_memory.py data / stuff_52K.txt 28769 10752 S pts / 16 00:00:00 python debug_memory.py data /stuff_53K.txt 28774 10852 S pts / 16 00:00:00 python debug_memory.py data / stuff_54K.txt 28780 10952 S pts / 16 00:00:00 python debug_memory.py data / stuff_55K .txt 28786 11052 S pts / 16 00:00:00 python debug_memory.py data / stuff_56K.txt 28791 11152 S pts / 16 00:00:00 python debug_memory.py data / stuff_57K.txt 28797 11256 S pts / 16 00:00:00 python debug_memory.py data / stuff_58K.txt 28802 11356 S pts / 16 00:00:00 python debug_memory.py data / stuff_59K.txt 28808 11452 S pts / 16 00:00:00 python debug_memory.py data / stuff_60K.txt 28814 11556 S pts / 16 00:00:00 python debug_memory.py data / stuff_61K.txt 28819 11656 S p ts / 16 00:00:00 python debug_memory.py data / stuff_62K.txt 28825 11752 S pts / 16 00:00:00 python debug_memory.py data / stuff_63K.txt 28831 11852 S pts / 16 00:00:00 python debug_memory.py data / stuff_64K.txt 28836 11956 S pts / 16 00:00:00 python debug_memory.py data / stuff_65K.txt 28842 12052 S pts / 16 00 ：00：00 python debug_memory.py data / stuff_66K.txt 28847 12152 S pts / 16 00:00:00 python debug_memory.py data / stuff_67K.txt 28853 12256 S pts / 16 00:00 ：00 python debug_memory.py data / stuff_68K.txt 28859 12356 S pts / 16 00:00:00 python debug_memory.py data / stuff_69K.txt 28864 12452 S pts / 16 00:00:00 python debug_memory.py data / stuff_70K.txt 28871 12556 S pts / 16 00:00:00 python debug_memory.py data / stuff_71K.txt 28877 12652 S pts / 16 00:00:00 python debug_memory .py data / stuff_72K.txt 28883 12756 S pts / 16 00:00:00 python debug_memory.py data / stuff_73K.txt 28889 12856 S pts / 16 00:00:00 python debug_memory.py数据/ stuff_7 4K.txt 28894 12952 S pts / 16 00:00:00 python debug_memory.py data / stuff_75K.txt 28900 13056 S pts / 16 00:00:00 python debug_memory.py data / stuff_76K。 txt 28906 13156 S pts / 16 00:00:00 python debug_memory.py data / stuff_77K.txt 28911 13256 S pts / 16 00:00:00 python debug_memory.py data / stuff_78K.txt 28917 13352 S pts / 16 00:00:00 python debug_memory.py data / stuff_79K.txt 28922 13452 S pts / 16 00:00:00 python debug_memory.py data / stuff_80K.txt 28928 13556 S pts / 16 00:00:00 python debug_memory.py data / stuff_81K.txt 28934 13652 S pts / 16 00:00:00 python debug_memory.py data / stuff_82K.txt 28939 13752 S pts / 16 00:00:00 python debug_memory.py data / stuff_83K.txt 28945 13852 S pts / 16 00:00:00 python debug_memory.py data / stuff_84K.txt 28951 13952 S pts / 16 00:00:00 python debug_memory.py data / stuff_85K.txt 28956 14052 S pts / 16 00:00:00 python debug_memory.py data / stuff_86K.txt 28962 14152 S pts / 16 00:00:00 python d ebug_memory.py data / stuff_87K.txt 28967 14256 S pts / 16 00:00:00 python debug_memory.py data / stuff_88K.txt 28973 14352 S pts / 16 00:00:00 python debug_memory。 py data / stuff_89K.txt 28979 14456 S pts / 16 00:00:00 python debug_memory.py data / stuff_90K.txt 28984 14552 S pts / 16 00:00:00 python debug_memory.py data /stuff_91K.txt 28990 14652 S pts / 16 00:00:00 python debug_memory.py data / stuff_92K.txt 28996 14756 S pts / 16 00:00:00 python debug_memory.py data / stuff_93K .txt 29001 14852 S pts / 16 00:00:00 python debug_memory.py data / stuff_94K.txt 29007 14956 S pts / 16 00:00:00 python debug_memory.py data / stuff_95K.txt 29012 15052 S pts / 16 00:00:00 python debug_memory.py data / stuff_96K.txt 29018 15156 S pts / 16 00:00:00 python debug_memory.py data / stuff_97K.txt 29024 15252 S pts / 16 00:00:00 python debug_memory.py data / stuff_98K.txt 29029 15360 S pts / 16 00:00:00 python debug_memory.py data / stuff_99K.txt 29035 15456 S pts / 16 00:00:00 python debug_memory.py data / stuff_100K.txt 29040 15556 S pts / 16 00:00:00 python debug_memory.py data / stuff_101K.txt 29046 15652 S pts / 16 00:00:00 python debug_memory.py data / stuff_102K.txt 29052 15756 S pts / 16 00:00:00 python debug_memory.py data / stuff_103K.txt 29057 15852 S pts / 16 00 ：00：00 python debug_memory.py data / stuff_104K.txt 29063 15952 S pts / 16 00:00:00 python debug_memory.py data / stuff_105K.txt 29069 16056 S pts / 16 00:00 ：00 python debug_memory.py data / stuff_106K.txt 29074 16152 S pts / 16 00:00:00 python debug_memory.py data / stuff_107K.txt 29080 16256 S pts / 16 00:00:00 python debug_memory.py data / stuff_108K.txt 29085 16356 S pts / 16 00:00:00 python debug_memory.py data / stuff_109K.txt 29091 16452 S pts / 16 00:00:00 python debug_memory .py data / stuff_110K.txt 29097 16552 S pts / 16 00:00:00 python debug_memory.py data / stuff_111K.txt 29102 16652 S pts / 16 00:00:00 python debug_memory.p y data / stuff_112K.txt 29108 16756 S pts / 16 00:00:00 python debug_memory.py data / stuff_113K.txt 29113 16852 S pts / 16 00:00:00 python debug_memory.py data /stuff_114K.txt 29119 16952 S pts / 16 00:00:00 python debug_memory.py data / stuff_115K.txt 29125 17056 S pts / 16 00:00:00 python debug_memory.py data / stuff_116K .txt 29130 17156 S pts / 16 00:00:00 python debug_memory.py data / stuff_117K.txt 29136 17256 S pts / 16 00:00:00 python debug_memory.py data / stuff_118K.txt 29141 17356 S pts / 16 00:00:00 python debug_memory.py data / stuff_119K.txt 29147 17452 S pts / 16 00:00:00 python debug_memory.py data / stuff_120K.txt 29153 17556 S pts / 16 00:00:00 python debug_memory.py data / stuff_121K.txt 29158 17656 S pts / 16 00:00:00 python debug_memory.py data / stuff_122K.txt 29164 17756 S pts / 16 00:00:00 python debug_memory.py data / stuff_123K.txt 29170 17856 S pts / 16 00:00:00 python debug_memory.py data / stuff_124K.txt 29175 17952小号pts / 16 00:00:00 python debug_memory.py data / stuff_125K.txt 29181 18056 S pts / 16 00:00:00 python debug_memory.py data / stuff_126K.txt 29186 18152 S pts / 16 00:00:00 python debug_memory.py data / stuff_127K.txt 29192 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_128K.txt 29198 5456 S pts / 16 00 ：00：00 python debug_memory.py data / stuff_129K.txt 29203 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_130K.txt 29209 5452 S pts / 16 00:00 ：00 python debug_memory.py data / stuff_131K.txt 29215 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_132K.txt 29220 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_133K.txt 29226 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_134K.txt 29231 5452 S pts / 16 00:00:00 python debug_memory .py data / stuff_135K.txt 29237 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_136K.txt 29243 5456 S pts / 16 00:00:00 python debug_memory.p y data / stuff_137K.txt 29248 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_138K.txt 29254 5452 S pts / 16 00:00:00 python debug_memory.py data /stuff_139K.txt 29260 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_140K.txt 29265 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_141K .txt 29271 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_142K.txt 29276 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_143K.txt 29282 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_144K.txt 29288 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_145K.txt 29293 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_146K.txt 29299 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_147K.txt 29305 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_148K.txt 29310 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_149K.txt 29316 5452小号pts / 16 00:00:00 python debug_memory.py data / stuff_150K.txt 29321 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_151K.txt 29327 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_152K.txt 29333 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_153K.txt 29338 5456 S pts / 16 00 ：00：00 python debug_memory.py data / stuff_154K.txt 29344 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_155K.txt 29349 5452 S pts / 16 00:00 ：00 python debug_memory.py data / stuff_156K.txt 29355 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_157K.txt 29361 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_158K.txt 29366 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_159K.txt 29372 5456 S pts / 16 00:00:00 python debug_memory .py data / stuff_160K.txt 29378 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_161K.txt 29383 5460 S pts / 16 00:00:00 python debug_memory.p y data / stuff_162K.txt 29389 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_163K.txt 29394 5456 S pts / 16 00:00:00 python debug_memory.py data /stuff_164K.txt 29400 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_165K.txt 29406 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_166K .txt 29411 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_167K.txt 29417 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_168K.txt 29423 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_169K.txt 29428 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_170K.txt 29434 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_171K.txt 29439 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_172K.txt 29445 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_173K.txt 29451 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_174K.txt 29456 5452小号pts / 16 00:00:00 python debug_memory.py data / stuff_175K.txt 29463 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_176K.txt 29483 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_177K.txt 29489 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_178K.txt 29496 5452 S pts / 16 00 ：00：00 python debug_memory.py data / stuff_179K.txt 29501 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_180K.txt 29507 5452 S pts / 16 00:00 ：00 python debug_memory.py data / stuff_181K.txt 29512 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_182K.txt 29518 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_183K.txt 29524 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_184K.txt 29529 5452 S pts / 16 00:00:00 python debug_memory .py data / stuff_185K.txt 29535 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_186K.txt 29541 5452 S pts / 16 00:00:00 python debug_memory.p y data / stuff_187K.txt 29546 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_188K.txt 29552 5456 S pts / 16 00:00:00 python debug_memory.py data /stuff_189K.txt 29557 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_190K.txt 29563 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_191K .txt 29569 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_192K.txt 29574 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_193K.txt 29580 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_194K.txt 29586 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_195K.txt 29591 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_196K.txt 29597 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_197K.txt 29602 5452 S pts / 16 00:00:00 python debug_memory.py data / stuff_198K.txt 29608 5456 S pts / 16 00:00:00 python debug_memory.py data / stuff_199K.txt 29614 5452小号pts / 16 00:00:00 python debug_memory.py data / stuff_200K.txt 有人可以解释什么正在发生？为什么我在使用文件（128KB）时看到内存使用增加？ 我的完整测试环境位于以下位置： https://github.com/saltycrane/debugging-python-memory-usage/tree / 50f73358c7a84a504333ce9c4071b0f3537bbc0f 我在Ubuntu 12.04上运行Python 2.7.3。 UPDATE 1 此问题并非特定于使用大小小于128K的文件。我得到相同的结果，将对象属性设置为与从文件读取的大小相同的值。这是更新后的代码： import sys 进口时间 class MyObj（object）： def __init __（self，size_kb）： self.att =''* int（size_kb）* 1024 $ b def myfunc（ size_kb）： mylist = [xObjects（size_kb）for xrange（100）] len（mylist） return [] def main（）： size_kb = sys.argv [1] myfunc（size_kb） time.sleep（3600） if __name__ == '__main__'： main（） 运行此脚本会得到类似的结果。更新后的测试环境位于以下位置： https：// UPDATE 2 我进一步简化了我的测试脚本：1.删除类并简单地创建一个字符串列表2.删除 myfunc（）并使用 del 删除 mylist 对象 import sys import time $ b def main（）： size_kb = sys.argv [1] mylist = [] for x in xrange（100）： mystr =''* int（size_kb）* 1024 mylist.append（mystr） del mylist time.sleep（3600） if __name__ =='__main__'： main（） 我的简化脚本也给出了类似的结果s到原始。 但是，如果我不创建一个单独的字符串变量，我没有看到增加了内存。以下是不会在内存中增加的脚本： 导入sys 导入时间 def main（）： size_kb = sys.argv [1] mylist = [] for xrange（100）： mylist.append（''* int（size_kb）* 1024） del mylist time.sleep（3600） if __name__ =='__main__'： main（） 更新后的测试环境位于： https://github.com/saltycrane/debugging- python-memory-usage / tree / 423ca6a50dccbe32572a9d0dea1068ddcb06663b 更多问题： 其他人是否可以重现我的结果？ 预计是否出现内存增加 ps ？ 关于发生的事情的提示 我发现了一些关于免费列表的有趣信息，它们看起来像，它们可能与此问题有关： $ b a href =http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm =nofollow noreferrer>为什么没有' t当我删除大型对象时，Python会释放内存？ 我该如何在Python中显式释放内存？ Python内存管理 - Theano v0.6rc3文档 链接： UPDATE 3 我简化了Update 2中的代码。添加行 del mystr 在脚本的最后释放内存。 （请参阅： https：// github .com / saltycrane / debugging-python-memory-usage / blob / dd058e4774802cae7cbfca520fb835ea46b645e8 / debug_memory_leaks.py ）我更新了脚本，使其足够复杂问题。 以下代码仍存在问题。 最新的代码/环境位于此处： https：/ /github.com/saltycrane/debugging-python-memory-usage/tree/fc0c8ce9ba621cb86b6abb93adf1b297a7c0230b import gc import sys import time $ b def main（）： size_kb = sys.argv [1] mylist = [ ] for x in xrange（100）： mystr =''* int（size_kb）* 1024 mydict = {'mykey'：mystr} mylist.append（mydict ） del mystr del mydict del mylist gc.collect（） time.sleep（3600） if __name__ =='__main__'： main（） 我还运行了其他一些环境的脚本。奇怪的结果是从一个干净的virtualenv中运行。在这种情况下，内存下降发生在260KB而不是128KB。请参阅 https://github.com/saltycrane/debugging-python-memory - 使用/树/ 52fbd5d57ff45affdcd70623ddb74fa1f1ffbbc2 环境： Ubuntu 12.04 64位，系统Python 2.7.3：原始运行 Ubuntu 12.04 64位，Python 3.3.0源代码编译：类似结果 科学Linux 6 64位，Python 2.6.6：类似的结果 从virtualenv：内存下载的Ubuntu 12.04 64位Python 2.7.3发生在260KB而不是128KB li> 更多参考资料： http://revista.python.org.ar/2/en/html/ memory-fragmentation.html http： //www.evanjo nes.ca/python-memory.html http://mail.python.org/pipermail/python-dev/2004-October/049480.html （注意：这是从2004年开始） http：//mail.python。 org / pipermail / python-dev / 2006-March / 061991.html http://www.evanjones.ca/memoryallocator/ http://www.evanjones.ca/memory-allocator.pdf http://hg.python.org/releasing/2.7.3/file/7bb96963d067 /Objects/obmalloc.c 在阅读了其中的一些内容后，我看到了对竞技场ize256KB。 也许这是相关的吗？ $ b 更新4（大部分已解决） schlenk发现内存使用在128KB下降的原因。 128KB是内存分配函数（malloc？）使用mmap而不是使用sbrk增加程序中断的点。 有趣的是，阈值可以通过环境变量来改变。 我运行了一个测试，将 MALLOC_MMAP_THRESHOLD _ 环境变量设置为个不同的值，并且内存使用的下降与该值匹配。 此处查看结果： https://github.com/saltycrane/debugging-python-memory-usage/blob/97d93cd165a139a6b6f96720de63a92561dd2f05/output_debug_memory_leaks.py.txt I仍然想知道它是否预期我的脚本的行为对于字符串值泄漏内存< 128KB。 更多链接： mallopt（3） - Linux手册页面（来自schlenk） Python内存管理和TCMalloc |推网络 Re ：将x设置为None并且del x在python 2.7.1（HPUX 11.23，ia64）中不释放内存«蟒蛇»«ActiveState列表存档 问题3526：在SunOS和AIX上定制malloc实现 - Python跟踪器 将malloc中的mmap / brk阈值设为动态以提高性能 注意：根据最后两个链接，使用mmap会有的性能（速度）命中解决方案您可能只需点击linux内存分配器的默认行为即可。 基本上Linux有两种分配策略，sbrk（）用于小块内存，mmap（）用于大型r块。 sbrk（）分配的内存块不能轻易返回到系统，而基于mmap（）的可以（只是取消映射页面）。 所以如果你分配一个内存块大于libc中的malloc（）分配器决定在sbrk（）和mmap（）之间切换的值，您会看到这种效果。请参阅mallopt（）调用，尤其是MMAP_THRESHOLD（ http：// man7。 org / linux / man-pages / man3 / mallopt.3.html ）。 更新 回答你的额外问题：是的，如果内存分配器的工作方式与Linux上的libc工作方式相同，那么预计会出现这种内存泄漏。如果您使用Windows LowFragmentationHeap，它可能不会泄漏，在AIX上类似，具体取决于哪个malloc配置。也许其他分配器之一（tcmalloc等）也解决了这些问题。 sbrk（）非常快，但存在内存碎片问题。 CPython不能做太多的事情，因为它没有压缩垃圾收集器，只是简单的引用计数。 Python提供了一些减少缓冲区分配的方法，请参阅例如这里的博客文章： http://eli.thegreenplace.net/2011/11/28/less-copies-in-python-with-the-buffer-protocol-and-memoryviews/ Original title: Memory leak opening files < 128KB in Python?Original questionI see what I think is a memory leak when running my Python script. Here is my script:import sysimport timeclass MyObj(object): def __init__(self, filename): with open(filename) as f: self.att = f.read()def myfunc(filename): mylist = [MyObj(filename) for x in xrange(100)] len(mylist) return []def main(): filename = sys.argv[1] myfunc(filename) time.sleep(3600)if __name__ == '__main__': main()The main function calls myfunc() which creates a list of 100 objects that each open andread a file. After returning from myfunc(), I'd expect memory from the 100-item list andfrom reading the file to be freed since they are no longer referenced. However, when Icheck the memory usage using the ps command, the Python process uses about 10,000 KBmore memory than a Python process run from a script with lines 12 and 13 commented out.The strange thing is that the memory leak (if that's what it is) only seems to occurfor files <128KB in size. I created a bash script to run this script with files rangingin size from 1KB to 200KB and the memory increase stopped when the files size hit 128KB.Here is the bash script:#!/bin/bashecho "PID RSS S TTY TIME COMMAND" > output.txtfor i in `seq 1 200`;do python debug_memory.py "data/stuff_${i}K.txt" & pid=$! sleep 0.1 ps -e -O rss | grep $pid | grep -v grep >> output.txt kill $piddoneHere is the output of the bash script:PID RSS S TTY TIME COMMAND28471 5552 S pts/16 00:00:00 python debug_memory.py data/stuff_1K.txt28477 5656 S pts/16 00:00:00 python debug_memory.py data/stuff_2K.txt28483 5756 S pts/16 00:00:00 python debug_memory.py data/stuff_3K.txt28488 5852 S pts/16 00:00:00 python debug_memory.py data/stuff_4K.txt28494 5952 S pts/16 00:00:00 python debug_memory.py data/stuff_5K.txt28499 6052 S pts/16 00:00:00 python debug_memory.py data/stuff_6K.txt28505 6156 S pts/16 00:00:00 python debug_memory.py data/stuff_7K.txt28511 6256 S pts/16 00:00:00 python debug_memory.py data/stuff_8K.txt28516 6356 S pts/16 00:00:00 python debug_memory.py data/stuff_9K.txt28522 6452 S pts/16 00:00:00 python debug_memory.py data/stuff_10K.txt28527 6552 S pts/16 00:00:00 python debug_memory.py data/stuff_11K.txt28533 6656 S pts/16 00:00:00 python debug_memory.py data/stuff_12K.txt28539 6756 S pts/16 00:00:00 python debug_memory.py data/stuff_13K.txt28544 6852 S pts/16 00:00:00 python debug_memory.py data/stuff_14K.txt28550 6952 S pts/16 00:00:00 python debug_memory.py data/stuff_15K.txt28555 7056 S pts/16 00:00:00 python debug_memory.py data/stuff_16K.txt28561 7156 S pts/16 00:00:00 python debug_memory.py data/stuff_17K.txt28567 7252 S pts/16 00:00:00 python debug_memory.py data/stuff_18K.txt28572 7356 S pts/16 00:00:00 python debug_memory.py data/stuff_19K.txt28578 7452 S pts/16 00:00:00 python debug_memory.py data/stuff_20K.txt28584 7556 S pts/16 00:00:00 python debug_memory.py data/stuff_21K.txt28589 7652 S pts/16 00:00:00 python debug_memory.py data/stuff_22K.txt28595 7756 S pts/16 00:00:00 python debug_memory.py data/stuff_23K.txt28600 7852 S pts/16 00:00:00 python debug_memory.py data/stuff_24K.txt28606 7952 S pts/16 00:00:00 python debug_memory.py data/stuff_25K.txt28612 8052 S pts/16 00:00:00 python debug_memory.py data/stuff_26K.txt28617 8152 S pts/16 00:00:00 python debug_memory.py data/stuff_27K.txt28623 8252 S pts/16 00:00:00 python debug_memory.py data/stuff_28K.txt28629 8356 S pts/16 00:00:00 python debug_memory.py data/stuff_29K.txt28634 8452 S pts/16 00:00:00 python debug_memory.py data/stuff_30K.txt28640 8556 S pts/16 00:00:00 python debug_memory.py data/stuff_31K.txt28645 8656 S pts/16 00:00:00 python debug_memory.py data/stuff_32K.txt28651 8756 S pts/16 00:00:00 python debug_memory.py data/stuff_33K.txt28657 8856 S pts/16 00:00:00 python debug_memory.py data/stuff_34K.txt28662 8956 S pts/16 00:00:00 python debug_memory.py data/stuff_35K.txt28668 9056 S pts/16 00:00:00 python debug_memory.py data/stuff_36K.txt28674 9156 S pts/16 00:00:00 python debug_memory.py data/stuff_37K.txt28679 9256 S pts/16 00:00:00 python debug_memory.py data/stuff_38K.txt28685 9352 S pts/16 00:00:00 python debug_memory.py data/stuff_39K.txt28691 9452 S pts/16 00:00:00 python debug_memory.py data/stuff_40K.txt28696 9552 S pts/16 00:00:00 python debug_memory.py data/stuff_41K.txt28702 9656 S pts/16 00:00:00 python debug_memory.py data/stuff_42K.txt28707 9756 S pts/16 00:00:00 python debug_memory.py data/stuff_43K.txt28713 9852 S pts/16 00:00:00 python debug_memory.py data/stuff_44K.txt28719 9952 S pts/16 00:00:00 python debug_memory.py data/stuff_45K.txt28724 10052 S pts/16 00:00:00 python debug_memory.py data/stuff_46K.txt28730 10156 S pts/16 00:00:00 python debug_memory.py data/stuff_47K.txt28739 10256 S pts/16 00:00:00 python debug_memory.py data/stuff_48K.txt28746 10352 S pts/16 00:00:00 python debug_memory.py data/stuff_49K.txt28752 10452 S pts/16 00:00:00 python debug_memory.py data/stuff_50K.txt28757 10556 S pts/16 00:00:00 python debug_memory.py data/stuff_51K.txt28763 10656 S pts/16 00:00:00 python debug_memory.py data/stuff_52K.txt28769 10752 S pts/16 00:00:00 python debug_memory.py data/stuff_53K.txt28774 10852 S pts/16 00:00:00 python debug_memory.py data/stuff_54K.txt28780 10952 S pts/16 00:00:00 python debug_memory.py data/stuff_55K.txt28786 11052 S pts/16 00:00:00 python debug_memory.py data/stuff_56K.txt28791 11152 S pts/16 00:00:00 python debug_memory.py data/stuff_57K.txt28797 11256 S pts/16 00:00:00 python debug_memory.py data/stuff_58K.txt28802 11356 S pts/16 00:00:00 python debug_memory.py data/stuff_59K.txt28808 11452 S pts/16 00:00:00 python debug_memory.py data/stuff_60K.txt28814 11556 S pts/16 00:00:00 python debug_memory.py data/stuff_61K.txt28819 11656 S pts/16 00:00:00 python debug_memory.py data/stuff_62K.txt28825 11752 S pts/16 00:00:00 python debug_memory.py data/stuff_63K.txt28831 11852 S pts/16 00:00:00 python debug_memory.py data/stuff_64K.txt28836 11956 S pts/16 00:00:00 python debug_memory.py data/stuff_65K.txt28842 12052 S pts/16 00:00:00 python debug_memory.py data/stuff_66K.txt28847 12152 S pts/16 00:00:00 python debug_memory.py data/stuff_67K.txt28853 12256 S pts/16 00:00:00 python debug_memory.py data/stuff_68K.txt28859 12356 S pts/16 00:00:00 python debug_memory.py data/stuff_69K.txt28864 12452 S pts/16 00:00:00 python debug_memory.py data/stuff_70K.txt28871 12556 S pts/16 00:00:00 python debug_memory.py data/stuff_71K.txt28877 12652 S pts/16 00:00:00 python debug_memory.py data/stuff_72K.txt28883 12756 S pts/16 00:00:00 python debug_memory.py data/stuff_73K.txt28889 12856 S pts/16 00:00:00 python debug_memory.py data/stuff_74K.txt28894 12952 S pts/16 00:00:00 python debug_memory.py data/stuff_75K.txt28900 13056 S pts/16 00:00:00 python debug_memory.py data/stuff_76K.txt28906 13156 S pts/16 00:00:00 python debug_memory.py data/stuff_77K.txt28911 13256 S pts/16 00:00:00 python debug_memory.py data/stuff_78K.txt28917 13352 S pts/16 00:00:00 python debug_memory.py data/stuff_79K.txt28922 13452 S pts/16 00:00:00 python debug_memory.py data/stuff_80K.txt28928 13556 S pts/16 00:00:00 python debug_memory.py data/stuff_81K.txt28934 13652 S pts/16 00:00:00 python debug_memory.py data/stuff_82K.txt28939 13752 S pts/16 00:00:00 python debug_memory.py data/stuff_83K.txt28945 13852 S pts/16 00:00:00 python debug_memory.py data/stuff_84K.txt28951 13952 S pts/16 00:00:00 python debug_memory.py data/stuff_85K.txt28956 14052 S pts/16 00:00:00 python debug_memory.py data/stuff_86K.txt28962 14152 S pts/16 00:00:00 python debug_memory.py data/stuff_87K.txt28967 14256 S pts/16 00:00:00 python debug_memory.py data/stuff_88K.txt28973 14352 S pts/16 00:00:00 python debug_memory.py data/stuff_89K.txt28979 14456 S pts/16 00:00:00 python debug_memory.py data/stuff_90K.txt28984 14552 S pts/16 00:00:00 python debug_memory.py data/stuff_91K.txt28990 14652 S pts/16 00:00:00 python debug_memory.py data/stuff_92K.txt28996 14756 S pts/16 00:00:00 python debug_memory.py data/stuff_93K.txt29001 14852 S pts/16 00:00:00 python debug_memory.py data/stuff_94K.txt29007 14956 S pts/16 00:00:00 python debug_memory.py data/stuff_95K.txt29012 15052 S pts/16 00:00:00 python debug_memory.py data/stuff_96K.txt29018 15156 S pts/16 00:00:00 python debug_memory.py data/stuff_97K.txt29024 15252 S pts/16 00:00:00 python debug_memory.py data/stuff_98K.txt29029 15360 S pts/16 00:00:00 python debug_memory.py data/stuff_99K.txt29035 15456 S pts/16 00:00:00 python debug_memory.py data/stuff_100K.txt29040 15556 S pts/16 00:00:00 python debug_memory.py data/stuff_101K.txt29046 15652 S pts/16 00:00:00 python debug_memory.py data/stuff_102K.txt29052 15756 S pts/16 00:00:00 python debug_memory.py data/stuff_103K.txt29057 15852 S pts/16 00:00:00 python debug_memory.py data/stuff_104K.txt29063 15952 S pts/16 00:00:00 python debug_memory.py data/stuff_105K.txt29069 16056 S pts/16 00:00:00 python debug_memory.py data/stuff_106K.txt29074 16152 S pts/16 00:00:00 python debug_memory.py data/stuff_107K.txt29080 16256 S pts/16 00:00:00 python debug_memory.py data/stuff_108K.txt29085 16356 S pts/16 00:00:00 python debug_memory.py data/stuff_109K.txt29091 16452 S pts/16 00:00:00 python debug_memory.py data/stuff_110K.txt29097 16552 S pts/16 00:00:00 python debug_memory.py data/stuff_111K.txt29102 16652 S pts/16 00:00:00 python debug_memory.py data/stuff_112K.txt29108 16756 S pts/16 00:00:00 python debug_memory.py data/stuff_113K.txt29113 16852 S pts/16 00:00:00 python debug_memory.py data/stuff_114K.txt29119 16952 S pts/16 00:00:00 python debug_memory.py data/stuff_115K.txt29125 17056 S pts/16 00:00:00 python debug_memory.py data/stuff_116K.txt29130 17156 S pts/16 00:00:00 python debug_memory.py data/stuff_117K.txt29136 17256 S pts/16 00:00:00 python debug_memory.py data/stuff_118K.txt29141 17356 S pts/16 00:00:00 python debug_memory.py data/stuff_119K.txt29147 17452 S pts/16 00:00:00 python debug_memory.py data/stuff_120K.txt29153 17556 S pts/16 00:00:00 python debug_memory.py data/stuff_121K.txt29158 17656 S pts/16 00:00:00 python debug_memory.py data/stuff_122K.txt29164 17756 S pts/16 00:00:00 python debug_memory.py data/stuff_123K.txt29170 17856 S pts/16 00:00:00 python debug_memory.py data/stuff_124K.txt29175 17952 S pts/16 00:00:00 python debug_memory.py data/stuff_125K.txt29181 18056 S pts/16 00:00:00 python debug_memory.py data/stuff_126K.txt29186 18152 S pts/16 00:00:00 python debug_memory.py data/stuff_127K.txt29192 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_128K.txt29198 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_129K.txt29203 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_130K.txt29209 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_131K.txt29215 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_132K.txt29220 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_133K.txt29226 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_134K.txt29231 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_135K.txt29237 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_136K.txt29243 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_137K.txt29248 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_138K.txt29254 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_139K.txt29260 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_140K.txt29265 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_141K.txt29271 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_142K.txt29276 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_143K.txt29282 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_144K.txt29288 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_145K.txt29293 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_146K.txt29299 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_147K.txt29305 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_148K.txt29310 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_149K.txt29316 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_150K.txt29321 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_151K.txt29327 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_152K.txt29333 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_153K.txt29338 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_154K.txt29344 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_155K.txt29349 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_156K.txt29355 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_157K.txt29361 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_158K.txt29366 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_159K.txt29372 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_160K.txt29378 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_161K.txt29383 5460 S pts/16 00:00:00 python debug_memory.py data/stuff_162K.txt29389 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_163K.txt29394 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_164K.txt29400 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_165K.txt29406 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_166K.txt29411 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_167K.txt29417 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_168K.txt29423 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_169K.txt29428 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_170K.txt29434 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_171K.txt29439 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_172K.txt29445 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_173K.txt29451 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_174K.txt29456 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_175K.txt29463 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_176K.txt29483 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_177K.txt29489 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_178K.txt29496 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_179K.txt29501 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_180K.txt29507 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_181K.txt29512 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_182K.txt29518 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_183K.txt29524 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_184K.txt29529 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_185K.txt29535 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_186K.txt29541 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_187K.txt29546 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_188K.txt29552 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_189K.txt29557 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_190K.txt29563 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_191K.txt29569 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_192K.txt29574 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_193K.txt29580 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_194K.txt29586 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_195K.txt29591 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_196K.txt29597 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_197K.txt29602 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_198K.txt29608 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_199K.txt29614 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_200K.txtCan someone explain what is happening? Why do I see an increase in memory usagewhen using files <128KB?My full test environment is located here:https://github.com/saltycrane/debugging-python-memory-usage/tree/50f73358c7a84a504333ce9c4071b0f3537bbc0fI am running Python 2.7.3 on Ubuntu 12.04.UPDATE 1This issue is not specific to working with files <128K in size. I get the sameresults setting the object attribute to a value the same size as was read fromthe file. Here is the updated code:import sysimport timeclass MyObj(object): def __init__(self, size_kb): self.att = ' ' * int(size_kb) * 1024def myfunc(size_kb): mylist = [MyObj(size_kb) for x in xrange(100)] len(mylist) return []def main(): size_kb = sys.argv[1] myfunc(size_kb) time.sleep(3600)if __name__ == '__main__': main()Running this script gives similar results. The updated test environment is located here:https://github.com/saltycrane/debugging-python-memory-usage/tree/59b7ff61134dfc11c4195e9201b2c1728ed4fcceUPDATE 2I simplified my test script further by: 1. removing the class and simply creating a list of strings 2. removing myfunc() and using del to delete the mylist objectimport sysimport timedef main(): size_kb = sys.argv[1] mylist = [] for x in xrange(100): mystr = ' ' * int(size_kb) * 1024 mylist.append(mystr) del mylist time.sleep(3600)if __name__ == '__main__': main()My simplified script also gives similar results to the original.However, if I don't create a separate string variable, I don't seean increase in memory. Here is the script that does not create anincrease in memory:import sysimport timedef main(): size_kb = sys.argv[1] mylist = [] for x in xrange(100): mylist.append(' ' * int(size_kb) * 1024) del mylist time.sleep(3600)if __name__ == '__main__': main()The updated test environment is located here:https://github.com/saltycrane/debugging-python-memory-usage/tree/423ca6a50dccbe32572a9d0dea1068ddcb06663bMore questions:Can someone else reproduce my results?Is the increase in memory seen by ps expected?Hints about what is happeningI discovered some interesting information about "free lists" that seem likethey could be related to this issue:Why doesn't Python release the memory when I delete a large object?How can I explicitly free memory in Python?Python Memory Management — Theano v0.6rc3 documentationFrom the last link:UPDATE 3I oversimplified the code in Update 2. Adding the line del mystr at the endof the script freed the memory.(See: https://github.com/saltycrane/debugging-python-memory-usage/blob/dd058e4774802cae7cbfca520fb835ea46b645e8/debug_memory_leaks.py)I updated the script to be sufficiently complicated to demonstrate the issue.The issue still exists in the following code.The latest code/environment is located here: https://github.com/saltycrane/debugging-python-memory-usage/tree/fc0c8ce9ba621cb86b6abb93adf1b297a7c0230bimport gcimport sysimport timedef main(): size_kb = sys.argv[1] mylist = [] for x in xrange(100): mystr = ' ' * int(size_kb) * 1024 mydict = {'mykey': mystr} mylist.append(mydict) del mystr del mydict del mylist gc.collect() time.sleep(3600)if __name__ == '__main__': main()I also ran the script is some other environments. The strange result wasrunning from within a clean virtualenv. In this case, the memory dropoffoccurred at 260KB instead of 128KB. See https://github.com/saltycrane/debugging-python-memory-usage/tree/52fbd5d57ff45affdcd70623ddb74fa1f1ffbbc2Environments:Ubuntu 12.04 64-bit, system Python 2.7.3: original runUbuntu 12.04 64-bit, Python 3.3.0 compiled from source: similar resultsScientific Linux 6 64-bit, Python 2.6.6: similar resultsUbuntu 12.04 64-bit, Python 2.7.3 from a virtualenv: memory dropoff occurs at 260KB instead of 128KBMore references:http://revista.python.org.ar/2/en/html/memory-fragmentation.htmlhttp://www.evanjones.ca/python-memory.htmlhttp://mail.python.org/pipermail/python-dev/2004-October/049480.html (Note: this is from 2004)http://mail.python.org/pipermail/python-dev/2006-March/061991.htmlhttp://www.evanjones.ca/memoryallocator/http://www.evanjones.ca/memory-allocator.pdfhttp://hg.python.org/releasing/2.7.3/file/7bb96963d067/Objects/obmalloc.cAfter reading some of these, I see a reference to an "arena size" of 256KB.Maybe that is related?UPDATE 4 (MOSTLY SOLVED)schlenk uncovered the reason the memory usage drops off at 128KB.128KB is the point at which "memory allocation functions" (malloc?)use mmap instead of increasing the program break using sbrk.Interestingly, the threshold can be changed via an environment variable.I ran a test setting the MALLOC_MMAP_THRESHOLD_ environment variable todifferent values and the dropoff in memory usage matched that value.See here for results:https://github.com/saltycrane/debugging-python-memory-usage/blob/97d93cd165a139a6b6f96720de63a92561dd2f05/output_debug_memory_leaks.py.txtI would still like to know if it expected behavior for my script toleak memory for string values < 128KB.A few more links:mallopt(3) - Linux manual page (from schlenk)Python memory management and TCMalloc | Pushing the WebRe: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) « python-list « ActiveState List ArchivesIssue 3526: Customized malloc implementation on SunOS and AIX - Python trackerMake the mmap/brk threshold in malloc dynamic to improve performanceNote: According to the last two links, there is a performance (speed) hit forusing mmap instead of sbrk. 解决方案 You might simply hit the default behaviour of the linux memory allocator.Basically Linux has two allocation strategies, sbrk() for small blocks of memory and mmap() for larger blocks. sbrk() allocated memory blocks cannot easily be returned to the system, while mmap() based ones can (just unmap the page).So if you allocate a memory block larger than the value where the malloc() allocator in your libc decides to switch between sbrk() and mmap() you see this effect. See the mallopt() call, especially the MMAP_THRESHOLD (http://man7.org/linux/man-pages/man3/mallopt.3.html).UpdateTo answer your extra question: yes, it is expected that you leak memory that way, if the memory allocator works like the libc one on Linux. If you used Windows LowFragmentationHeap instead, it would probably not leak, similar on AIX, depending on which malloc is configured. Maybe one of the other allocators (tcmalloc etc.) also fix such issues. sbrk() is blazingly fast, but has issues with memory fragmentation. CPython cannot do much about it, as it does not have a compacting garbage collector, but simple reference counting.Python offers a few methods to reduce the buffer allocations, see for example the blog post here: http://eli.thegreenplace.net/2011/11/28/less-copies-in-python-with-the-buffer-protocol-and-memoryviews/ 这篇关于内存泄漏时使用字符串&lt; Python中128KB？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！ 上岸，阿里云！
08-14 16:11
Powered by LMLPHP ©2024 内存泄漏时使用字符串 0.035431