度量系统无法识别应用程序jar中的自定义源/接收器

首先，我在 org.apache.spark.metrics.sink 包中创建了一个 CustomGraphiteSink 类: package org.apache.spark.metrics.sink;公共类CustomGraphiteSink扩展了GraphiteSink {} 然后我在 metrics.properties 中指定了该类 *.sink.graphite.class = org.apache.spark.metrics.sink.CustomGraphiteSink 并通过以下方式将此文件传递给spark-submit:-conf spark.metrics.conf = metrics.properties Followup from here.I've added Custom Source and Sink in my application jar and found a way to get a static fixed metrics.properties on Stand-alone cluster nodes. When I want to launch my application, I give the static path - spark.metrics.conf="/fixed-path/to/metrics.properties". Despite my custom source/sink being in my code/fat-jar - I get ClassNotFoundException on CustomSink.My fat-jar (with Custom Source/Sink code in it) is on hdfs with read access to all.So here's what all I've already tried setting (since executors can't find Custom Source/Sink in my application fat-jar):spark.executor.extraClassPath = hdfs://path/to/fat-jarspark.executor.extraClassPath = fat-jar-name.jarspark.executor.extraClassPath = ./fat-jar-name.jarspark.executor.extraClassPath = ./spark.executor.extraClassPath = /dir/on/cluster/* (although * is not at file level, there are more directories - I have no way of knowing random application-id or driver-id to give absolute name before launching the app)It seems like this is how executors are getting initialized for this case (please correct me if I am wrong) - Driver tells here's the jar location - hdfs://../fat-jar.jar and here are some properties like spark.executor.memory etc.N number of Executors spin up (depending on configuration) on clusterStart downloading hdfs://../fat-jar.jar but initialize metrics system in the mean time (? - not sure of this step)Metrics system looking for Custom Sink/Source files - since it's mentioned in metrics.properties - even before it's done downloading fat-jar (which actually has all those files) (this is my hypothesis)ClassNotFoundException - CustomSink not found!Is my understanding correct? Moreover, is there anything else I can try? If anyone has experience with custom source/sinks, any help would be appreciated. 解决方案 I stumbled upon the same ClassNotFoundException when I needed to extend existing GraphiteSink class and here's how I was able to solve it.First, I created a CustomGraphiteSink class in org.apache.spark.metrics.sink package:package org.apache.spark.metrics.sink;public class CustomGraphiteSink extends GraphiteSink {}Then I specified the class in metrics.properties*.sink.graphite.class=org.apache.spark.metrics.sink.CustomGraphiteSinkAnd passed this file to spark-submit via:--conf spark.metrics.conf=metrics.properties 这篇关于度量系统无法识别应用程序jar中的自定义源/接收器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！