问题描述
我正在开发一个Spark应用程序,我已经习惯了Spring作为一个依赖注入框架。现在,我坚持了这个问题,该处理部分使用了Spring的@Autowired功能,但它是序列化和反序列化星火
I'm developing a Spark Application and I'm used to Spring as a Dependency Injection Framework. Now I'm stuck with the problem, that the processing part uses the @Autowired functionality of Spring, but it is serialized and deserialized by Spark.
所以下面code使我陷入麻烦:
So the following code gets me into trouble:
Processor processor = ...; // This is a Spring constructed object
// and makes all the trouble
JavaRDD<Txn> rdd = ...; // some data for Spark
rdd.foreachPartition(processor);
处理器看起来像:
The Processor looks like that:
public class Processor implements VoidFunction<Iterator<Txn>>, Serializeable {
private static final long serialVersionUID = 1L;
@Autowired // This will not work if the object is deserialized
private transient DatabaseConnection db;
@Override
public void call(Iterator<Txn> txns) {
... // do some fance stuff
db.store(txns);
}
}
所以我的问题是:它甚至有可能使用类似弹簧结合星火?如果没有,什么是最优雅的方式做这样的事情?任何帮助AP preciated!
So my question is: Is it even possible to use something like Spring in combination with Spark? If not, what is the most elegant way to do something like that? Any help is appreciated!
推荐答案
的补充:的要直接干预反序列化的部分,而无需修改自己的类使用以下的。该项目autowires当它被弹簧反序列化你的bean。
Added: To interfere the deserialization part directly without modifying your own classes use the following spring-spark project. This projects autowires your bean when it gets deserialized by spring.
另一个可能的黑客攻击将迫使春
注入的情况下,你的分贝
数据库连接它的无效
。
Another possible hack would be to force Spring
to inject your db
database connection in case it's null
.
@Autowired // This will not work if the object is deserialized
private transient DatabaseConnection db;
@Override
public void call(Iterator<Txn> txns) {
... // do some fance stuff
getDb().store(txns);
}
private DatabaseConnection getDb() {
if(db == null) {
WebApplicationContext webApplicationContext = WebApplicationContextUtils.getWebApplicationContext(MySpringUtils.getServletContext());
AutowireCapableBeanFactory autowireCapableBeanFactory = webApplicationContext.getAutowireCapableBeanFactory();
autowireCapableBeanFactory.configureBean(this, "databaseConnection");
}
return db;
}
和则需要
@Component
public class MySpringUtils {
@Autowired
private ServletContext servletContext;
private static ServletContext staticContext;
@PostConstruct
public void init() {
this.staticContext = servletContext;
}
public static ServletContext getServletContext() {
return staticContext;
}
}
但是,这听起来像一个黑客,因为分贝
似乎被弹簧非托管,如这不是一个数据源
- 这似乎有些奇怪。因此,这可能不是最佳的解决方案。
But this sounds like a hack, because db
seems unmanaged by Spring, as in that is not a DataSource
- which seems somewhat odd. Therefore, this might not be the best possible solution.
这篇关于星火一起使用Spring的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!