" ClassNotFoundException：sun.security.provider.Sun"在Google App Engine中运行Google Cloud Dataflow管道时

问题描述

在我们的Dataflow管道中， DoFn 包含一个类型，其中 Random 字段指向 SecureRandom 实例，并且该字段在使用 DataflowPipelineRunner 时在Dataflow服务中运行时无法反序列化。 （下面的堆栈跟踪）

我们使用默认的ctor创建 SecureRandom ，这会发生一个实例使用 sun.security.provider.Sun 作为其 java.security.Provider （请参阅）。 SecureRandom 扩展随机，这是可序列化的。

尝试反序列化此类时数据流服务扼流圈，因为它无法创建 sun.security.provider.Sun 。

仔细观察堆栈跟踪，我发现反序列化通过 com.google.apphosting.runtime.security.UserClassLoader 发生，现在我的理论是这个类加载器不允许加载 sun。* 类，或者至少这个特定的 sun。* class。

  java.lang.IllegalArgumentException：无法在com.google.cloud.dataflow.sdk.util反序列化com.example.Example@13e88d 
 .SerializableUtils.deserializeFromByteArray（SerializableUtils.java:73）
，com.google.cloud.dataflow.sdk.util.SerializableUtils.clone（SerializableUtils.java:88）
，com.google.cloud.dataflow .sdk.transforms.ParDo $ Bound。< init>（ParDo.java:683）
 [...] 
由以下原因引起： java.lang.ClassNotFoundException：sun.security.provider.Sun 
 at com.google.apphosting.runtime.security.UserClassLoader.loadClass（UserClassLoader.java:442）
 at java.lang.ClassLoader.loadClass （ClassLoader.java:375）
 at java.lang.Class.forName0（Native Method）
 [...] 
  

解决方案

问题在于 sun.security.provider.Sun 未显示App Engine JRE白名单，因此类加载程序无法实例化它的实例：

但幸运的是，仍然可以在同一个环境中使用新的SecureRandom（）。

要解决这个问题，我们添加了一个自定义使用 Random 字段将de / serialization挂钩到类。简单示例：

  class示例实现了Serializable {
 
 //查看{@link #writeObject上的注释}为什么这是暂时的。 
 //应该被视为最终的，但不能这样声明。 
私人暂时随机随机; 
 
 // 
 // [班级的勇气在这里...] 
 // 
 $ b $ / ** 
 *序列化钩来处理瞬态随机场。 
 * / 
 private void writeObject（ObjectOutputStream out）throws IOException {
 out.defaultWriteObject（）; 
 if（random instanceof SecureRandom）{
 //写一个null来告诉readObject（）在反序列化期间创建一个新的
 // SecureRandom; null可以安全地使用
 //作为占位符，因为构造函数不允许为null 
 // Randoms。 
 // 
 //数据流云环境不会反序列化
 //使用sun.security.provider.Sun 
 //作为Provider的SecureRandom实例，因为它是一个系统
 //不在App Engine白名单上的类：
 // https://cloud.google.com/appengine/docs/java/jrewhitelist 
 out.writeObject（null）; 
} else {
 out.writeObject（random）; 
} 
} 
 
 / ** 
 *反序列化钩子初始化瞬态随机场。 
 * / 
 private void readObject（ObjectInputStream in）
 throws IOException，ClassNotFoundException {
 in.defaultReadObject（）; 
 Object newRandom = in.readObject（）; 
 if（newRandom == null）{
 // writeObject（）会在原始字段为
 // SecureRandom时写入null;创建一个新的实例来替换它。请参阅writeObject（）中的
 //注释以了解背景。 
 random = new SecureRandom（）; 
 random.nextDouble（）; //强制播种
} else {
 random =（Random）newRandom; 
} 
} 
} 
  

A DoFn in our Dataflow pipeline contains a type with a Random field pointing to a SecureRandom instance, and that field fails to deserialize when running in the Dataflow service using DataflowPipelineRunner. (stack trace below)

We create the SecureRandom using its default ctor, which happens to hand back an instance that uses sun.security.provider.Sun as its java.security.Provider (see SecureRandom#getProvider). SecureRandom extends Random, which is serializable.

The Dataflow service chokes when trying to deserialize this class because it can't create sun.security.provider.Sun.

Looking closer at the stack trace, I see that deserialization happens through com.google.apphosting.runtime.security.UserClassLoader, and now my theory is that this classloader doesn't allow loading of sun.* classes, or at least this particular sun.* class.

java.lang.IllegalArgumentException: unable to deserialize com.example.Example@13e88d
    at com.google.cloud.dataflow.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:73)
    at com.google.cloud.dataflow.sdk.util.SerializableUtils.clone(SerializableUtils.java:88)
    at com.google.cloud.dataflow.sdk.transforms.ParDo$Bound.<init>(ParDo.java:683)
    [...]
    Caused by: java.lang.ClassNotFoundException: sun.security.provider.Sun
    at com.google.apphosting.runtime.security.UserClassLoader.loadClass(UserClassLoader.java:442)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:375)
    at java.lang.Class.forName0(Native Method)
    [...]

解决方案

The problem is that sun.security.provider.Sun doesn't appear on the App Engine JRE whitelist, so the classloader can't instantiate instances of it:

https://cloud.google.com/appengine/docs/java/jrewhitelist

But luckily you can still say new SecureRandom() in the same environment.

To work around the problem, we added a custom de/serialization hook to the class with the Random field. Simplified example:

class Example implements Serializable {

  // See comments on {@link #writeObject} for why this is transient.
  // Should be treated as final, but can't be declared as such.
  private transient Random random;

  //
  // [Guts of the class go here...]
  //

  /**
   * Serialization hook to handle the transient Random field.
   */
  private void writeObject(ObjectOutputStream out) throws IOException {
    out.defaultWriteObject();
    if (random instanceof SecureRandom) {
      // Write a null to tell readObject() to create a new
      // SecureRandom during deserialization; null is safe to use
      // as a placeholder because the constructor disallows null
      // Randoms.
      //
      // The dataflow cloud environment won't deserialize
      // SecureRandom instances that use sun.security.provider.Sun
      // as their Provider, because it's a system
      // class that's not on the App Engine whitelist:
      // https://cloud.google.com/appengine/docs/java/jrewhitelist
      out.writeObject(null);
    } else {
      out.writeObject(random);
    }
  }

  /**
   * Deserialization hook to initialize the transient Random field.
   */
  private void readObject(ObjectInputStream in)
      throws IOException, ClassNotFoundException {
    in.defaultReadObject();
    Object newRandom = in.readObject();
    if (newRandom == null) {
      // writeObject() will write a null if the original field was
      // SecureRandom; create a new instance to replace it. See
      // comments in writeObject() for background.
      random = new SecureRandom();
      random.nextDouble(); // force seeding
    } else {
      random = (Random) newRandom;
    }
  }
}

这篇关于" ClassNotFoundException：sun.security.provider.Sun"在Google App Engine中运行Google Cloud Dataflow管道时的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！