将Spring Boot应用程序jar提交到Spark-Submit

问题描述

我是Spark的新手，我正在尝试尝试Spark提交.我在spring boot中创建了一个应用程序，使用mvn package创建了一个jar.但是，当我尝试将jar提交给spark-submit时，它找不到主类.但是主类存在于jar中.

I am pretty new to spark, and i am trying to try out spark submit. I created an application in spring boot , used mvn package to create a jar . But when i am trying to submit the jar to spark-submit , it is not able to find the Main class . But the main class is present in the jar.

 spark-submit --class com.dip.sparkapp.SparkappApplication --master local target/sparkapp-0.0.1-SNAPSHOT.jar

推荐答案

实际上，在您发布此消息的同一天，我们遇到了同样的问题.我们的解决方案是为Maven使用shade插件来编辑我们的构建.我们发现，当使用spring-boot-maven插件打包时，它将我们的类嵌套在BOOT-INF/spark不喜欢的类中.我将粘贴相关部分，以便您可以在自己的应用程序上进行尝试-祝您好运！

We ran into the same problem, actually, on the same day you posted this. Our solutions was to use the shade plugin for maven to edit our build a bit. We found that when packaging with the spring-boot-maven plugin it nested our classes in BOOT-INF/classes which spark didn't like. I'll paste the relevant section so you can try it out on your own application -- good luck!

<build>
   <plugins>
      <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <dependencies>
                <dependency>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-maven-plugin</artifactId>
                    <version>${spring-boot-version}</version>
                </dependency>
            </dependencies>
            <configuration>
                <keepDependenciesWithProvidedScope>false</keepDependenciesWithProvidedScope>
                <createDependencyReducedPom>false</createDependencyReducedPom>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
                <transformers>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>META-INF/spring.handlers</resource>
                    </transformer>
                    <transformer
                            implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
                        <resource>META-INF/spring.factories</resource>
                    </transformer>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>META-INF/spring.schemas</resource>
                    </transformer>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                        <mainClass>${start-class}</mainClass>
                    </transformer>
                </transformers>
            </configuration>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

这篇关于将Spring Boot应用程序jar提交到Spark-Submit的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！