ALS部署Spark集群入坑记

[Stage 236:>                                                      (0 + 0) / 400]17/12/04 09:45:55 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: java.lang.StackOverflowError

java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1841)

java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1534)

java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)

java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)

java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)

java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)

java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)

java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)

java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)

java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)

java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)

java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)

java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)

java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)

java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)

java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)

java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)

java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)

java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)

java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)

java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)

java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)

java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)

java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)

java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)

java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)

scala.collection.immutable.$colon$colon.writeObject(List.scala:379)

sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

栈溢出原因：由于ASL的迭代次数过多，DAG过深，导致RDD的lineage，栈溢出。

现有解决：1.迭代次数减少。

2.利用checkpoint？

参考：

1. 记一次spark mllib stackoverflow踩坑
2. 一个 KCore 算法引发的 StackOverflow 奇案
3. Spark出现java.lang.stackoverflowerror的解决方法
4. Spark sql解析异常java.lang.StackOverflowError处理
5. 由于长RDD谱系,Stackoverflow

其他：

1. spark出现task不能序列化错误的解决方法 org.apache.spark.SparkException: Task not serializable