如何在Zeppelin中使用来自S3的依赖项?

问题描述

是否有一种方法可以将Zeppelin的依赖项添加到S3上的存储桶中?尝试过z.load(s3n://...)和z.addRepo(some_name).url(s3n://...)，但它们似乎没有完成任务..

Is there a way to add jars that are in a bucket on S3 as a dependency of Zeppelin? tried z.load(s3n://...) and z.addRepo(some_name).url(s3n://...) but they don't seem to do the job..

推荐答案

您可以从S3下载jar，然后将其放在本地FS上.可以在％dep 解释器内完成，例如:

You could download jars from S3 and put it on the local FS. It could be done inside %dep interpreter like this:

%dep
import com.amazonaws.services.s3.AmazonS3Client
import java.io.File
import java.nio.file.{Files, StandardCopyOption}

val dest = "/tmp/dependency.jar"
val s3 = new AmazonS3Client()
val stream = s3.getObject("buckename", "path.jar").getObjectContent

Files.copy(stream, new File(dest).toPath, StandardCopyOption.REPLACE_EXISTING)

z.load(dest)

注意:您必须生成胖子jar，即包括默认情况下未提供的所有自定义依赖项(例如，当您的项目中有多个模块时).在maven中，可以使用maven-shade-plugin来实现:

Note: You must generate fat jar, i.e. include all custom dependencies not provided by default (for example when you have multiple modules in your project). In maven it could be implemented with maven-shade-plugin:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.4.2</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <artifactSet>
                    <includes>
                        <include>com.yourcompany:*</include>
                    </includes>
                </artifactSet>
            </configuration>
        </execution>
    </executions>
</plugin>

这篇关于如何在Zeppelin中使用来自S3的依赖项?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！