小时后出现

小时后出现

关注
发信
关注(28)粉丝(399)

Spark Streaming 应用程序运行 24 小时后出现 OOM

本文介绍了Spark Streaming 应用程序运行 24 小时后出现 OOM的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我正在使用 spark 1.5.0 并开发一个 spark 流应用程序.应用程序从 HDFS 读取文件，将 rdd 转换为数据帧并对每个数据帧执行多个查询.

该应用程序完美运行了大约 24 小时，然后崩溃了.应用程序主日志/驱动程序日志显示:

线程dag-scheduler-event-loop"中的异常java.lang.OutOfMemoryError:超出GC开销限制在 java.lang.Class.getDeclaredMethods0(Native Method)在 java.lang.Class.privateGetDeclaredMethods(Class.java:2701)在 java.lang.Class.getDeclaredMethod(Class.java:2128)在 java.io.ObjectStreamClass.getInheritableMethod(ObjectStreamClass.java:1442)在 java.io.ObjectStreamClass.access$2200(ObjectStreamClass.java:72)在 java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:508)在 java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)在 java.security.AccessController.doPrivileged(Native Method)在 java.io.ObjectStreamClass.(ObjectStreamClass.java:472)在 java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)在 java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)在 java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)在 java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)在 java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)在 java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)在 java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)在 java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)在 java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)在 java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)在 java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)在 scala.collection.immutable.$colon$colon.writeObject(List.scala:379)在 sun.reflect.GeneratedMethodAccessor1511.invoke(来源不明)在 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)在 java.lang.reflect.Method.invoke(Method.java:497)在 java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)在 java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)在 java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)在 java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)在 java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)在 java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)在 java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)在 java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)线程JobGenerator"中的异常 java.lang.OutOfMemoryError:超出 GC 开销限制在 java.util.zip.ZipCoder.getBytes(ZipCoder.java:80)在 java.util.zip.ZipFile.getEntry(ZipFile.java:310)在 java.util.jar.JarFile.getEntry(JarFile.java:240)在 sun.net.www.protocol.jar.URLJarFile.getEntry(URLJarFile.java:128)在 sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:132)在 sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)在 java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:238)在 java.lang.Class.getResourceAsStream(Class.java:2223)在 org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:38)在 org.apache.spark.util.ClosureCleaner$.getInnerClosureClasses(ClosureCleaner.scala:81)在 org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:187)在 org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)在 org.apache.spark.SparkContext.clean(SparkContext.scala:2032)在 org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:314)在 org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:313)在 org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)在 org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)在 org.apache.spark.rdd.RDD.withScope(RDD.scala:306)在 org.apache.spark.rdd.RDD.map(RDD.scala:313)在 org.apache.spark.streaming.dstream.MappedDStream$$anonfun$compute$1.apply(MappedDStream.scala:35)在 org.apache.spark.streaming.dstream.MappedDStream$$anonfun$compute$1.apply(MappedDStream.scala:35)在 scala.Option.map(Option.scala:145)在 org.apache.spark.streaming.dstream.MappedDStream.compute(MappedDStream.scala:35)在 org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:350)在 org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:350)在 scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)在 org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:349)在 org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:349)在 org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:399)在 org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:344)在 org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:342)在 scala.Option.orElse(Option.scala:257)

我收集了驱动程序堆转储，它说可能的内存泄漏来自 org.apache.spark.sql.execution.ui.SQLListener

也在我的应用程序主 URL 中，我可以看到数千个 SQL 选项卡，例如:->SQL 1, SQL2 .. SQL 2000 以及这些选项卡的数量不断增加.

有谁知道为什么这些SQL选项卡不断增加并建议GC异常.谢谢

解决方案

1.5.0 中存在一些内存泄漏问题:SPARK-11126、SPARK-10155.

根据您的描述，您正在点击 SPARK-11126.>

您需要升级到 1.5.2 或为您的 Spark 应用补丁.

I am using spark 1.5.0 and working on a spark streaming application . The application reads files from HDFS , converts rdd into dataframe and execute multiple queries on each dataframe.

The application runs perfectly for around 24 hours and then it crashes.The application master logs / driver logs shows :

Exception in thread "dag-scheduler-event-loop" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethod(Class.java:2128)
at java.io.ObjectStreamClass.getInheritableMethod(ObjectStreamClass.java:1442)
at java.io.ObjectStreamClass.access$2200(ObjectStreamClass.java:72)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:508)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    at scala.collection.immutable.$colon$colon.writeObject(List.scala:379)
    at sun.reflect.GeneratedMethodAccessor1511.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
Exception in thread "JobGenerator" java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.util.zip.ZipCoder.getBytes(ZipCoder.java:80)
    at java.util.zip.ZipFile.getEntry(ZipFile.java:310)
    at java.util.jar.JarFile.getEntry(JarFile.java:240)
    at sun.net.www.protocol.jar.URLJarFile.getEntry(URLJarFile.java:128)
    at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:132)
    at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)
    at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:238)
    at java.lang.Class.getResourceAsStream(Class.java:2223)
    at org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:38)
    at org.apache.spark.util.ClosureCleaner$.getInnerClosureClasses(ClosureCleaner.scala:81)
    at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:187)
    at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)
    at org.apache.spark.SparkContext.clean(SparkContext.scala:2032)
    at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:314)
    at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:313)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
    at org.apache.spark.rdd.RDD.map(RDD.scala:313)
    at org.apache.spark.streaming.dstream.MappedDStream$$anonfun$compute$1.apply(MappedDStream.scala:35)
    at org.apache.spark.streaming.dstream.MappedDStream$$anonfun$compute$1.apply(MappedDStream.scala:35)
    at scala.Option.map(Option.scala:145)
    at org.apache.spark.streaming.dstream.MappedDStream.compute(MappedDStream.scala:35)
    at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:350)
    at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:350)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
    at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:349)
    at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:349)
    at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:399)
    at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:344)
    at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:342)
    at scala.Option.orElse(Option.scala:257)

I collected driver heap dump and it says possible memory leak is from org.apache.spark.sql.execution.ui.SQLListener

Also in my applciation master url , i can see thousands of SQL tabs eg:-> SQL 1, SQL2 .. SQL 2000 and these number of tabs keeps increasing.

Does any one know why these SQL tabs keep increasing and suggestion for GC exception.Thanks

解决方案

There are some memory leak issues in 1.5.0: SPARK-11126, SPARK-10155.

According to your description, you are hitting SPARK-11126.

You need to upgrade to 1.5.2 or apply the patches for your Spark.

这篇关于Spark Streaming 应用程序运行 24 小时后出现 OOM的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！

08-01 04:42
Powered by LMLPHP ©2024 小时后出现 0.040900