问题描述
我正在公司防火墙后面的Windows服务器中运行MiNiFi.我的NiFi在Hortonworks群集上运行(3个节点，分别为:sj46，sj47，sj48).
I am running MiNiFi in a windows server which is behind my company's firewall. My NiFi is running on Hortonworks cluster (3 nodes ex:sj46,sj47,sj48).
我正在尝试将数据从MiNiFi发送到NiFi.
I am trying to send data from the MiNiFi to NiFi.
- MiNiFi版本:0.4
- NiFi版本:1.5
在MiNiFi中，我具有ListFile处理器，FetchFile处理器和RemoteProcessGroup.
In MiNiFi, i have ListFile processor, FetchFile processor and RemoteProcessGroup.
RemoteProcessGroup:
- URL: https://sjxx46.xxx:6689/nifi,https://sjxx47.xxx:6689/nifi,https://sjxx48.xxx:6689/nifi
- TransportProtocol:HTTP
- URL:https://sjxx46.xxx:6689/nifi,https://sjxx47.xxx:6689/nifi,https://sjxx48.xxx:6689/nifi
- TransportProtocol: HTTP
NiFi.properties:
nifi.remote.input.host=
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.input.secure=True
nifi.remote.input.socket.port=6698
nifi.cluster.node.protocol.port=6699
nifi.web.http.port=
nifi.web.https.host=sj46.xxxx
nifi.web.https.port=6689
MiNiFi config.yml
MiNiFi Config Version: 3
Flow Controller:
name: min2510
comment: ''
Core Properties:
flow controller graceful shutdown period: 10 sec
flow service write delay interval: 500 ms
administrative yield duration: 30 sec
bored yield duration: 10 millis
max concurrent threads: 1
variable registry properties: ''
FlowFile Repository:
partitions: 256
checkpoint interval: 2 mins
always sync: false
Swap:
threshold: 20000
in period: 5 sec
in threads: 1
out period: 5 sec
out threads: 4
Content Repository:
content claim max appendable size: 10 MB
content claim max flow files: 100
always sync: false
Provenance Repository:
provenance rollover time: 1 min
implementation: org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository
Component Status Repository:
buffer size: 1440
snapshot frequency: 1 min
Security Properties:
keystore: ''
keystore type: ''
keystore password: ''
key password: ''
truststore: ''
truststore type: ''
truststore password: ''
ssl protocol: ''
Sensitive Props:
key:
algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
provider: BC
Processors:
- id: 50420aac-ef66-30ac-0000-000000000000
name: FetchFile
class: org.apache.nifi.processors.standard.FetchFile
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 0 sec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 0
auto-terminated relationships list:
- failure
- not.found
- permission.denied
Properties:
Completion Strategy: None
File to Fetch: ${absolute.path}/${filename}
Log level when file not found: ERROR
Log level when permission denied: ERROR
Move Conflict Strategy: Rename
Move Destination Directory:
- id: ae4799d5-073c-3dd6-0000-000000000000
name: ListFile
class: org.apache.nifi.processors.standard.ListFile
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 0 sec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 0
auto-terminated relationships list: []
Properties:
File Filter: '[^\.].*'
Ignore Hidden Files: 'true'
Include File Attributes: 'true'
Input Directory: C:\minifi to EFT
Input Directory Location: Local
Maximum File Age:
Maximum File Size:
Minimum File Age: 0 sec
Minimum File Size: 0 B
Path Filter:
Recurse Subdirectories: 'true'
target-system-timestamp-precision: auto-detect
Controller Services: []
Process Groups: []
Input Ports: []
Output Ports: []
Funnels: []
Connections:
- id: 95dde9e7-96a4-3bea-0000-000000000000
name: FetchFile/success/eef78a5f-cc55-1d33-0000-0000481921e3
source id: 50420aac-ef66-30ac-0000-000000000000
source relationship names:
- success
destination id: eef78a5f-cc55-1d33-0000-0000481921e3
max work queue size: 10000
max work queue data size: 1 GB
flowfile expiration: 0 sec
queue prioritizer class: ''
- id: 2ba13196-5ac6-3966-0000-000000000000
name: ListFile/success/FetchFile
source id: ae4799d5-073c-3dd6-0000-000000000000
source relationship names:
- success
destination id: 50420aac-ef66-30ac-0000-000000000000
max work queue size: 10000
max work queue data size: 1 GB
flowfile expiration: 0 sec
queue prioritizer class: ''
Remote Process Groups:
- id: 6bd4b280-646b-3b93-0000-000000000000
name: ''
url: https://sj46.xxxx:6689/nifi
comment: ''
timeout: 30 sec
yield period: 10 sec
transport protocol: HTTP
proxy host: ''
proxy port: ''
proxy user: ''
proxy password: ''
local network interface: ''
Input Ports:
- id: eef78a5f-cc55-1d33-0000-0000481921e3
name: From MiNiFi
comment: ''
max concurrent tasks: 1
use compression: false
Output Ports: []
NiFi Properties Overrides: {}
在MiNiFi Windows服务器中，打开了诸如6688、6689和6699之类的端口.
in the MiNiFi windows server, the ports such as 6688,6689 and 6699 are opened.
但是如果我尝试在MiNiFi和NiFi上进行telnet，则无法建立.
but if i try to telnet on MiNiFi and NiFi, it was not established.
minifi-app.log
2019-03-27 15:50:09,152 WARN [Remote Process Group 6bd4b280-646b-3b93-0000-000000000000 Thread-1] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://sj46.xxxx:6689/nifi-api due to org.apache.http.conn.HttpHostConnectException: Connect to sj46.xxxx:6689 [sj46.xxxx/x.x.x.x] failed: Connection timed out: connect
2019-03-27 15:50:09,160 WARN [Remote Process Group 6bd4b280-646b-3b93-0000-000000000000 Thread-1] o.a.n.remote.StandardRemoteProcessGroup Unable to communicate with remote instance RemoteProcessGroup[https://sj46.xxxx:6689/nifi]
org.apache.nifi.controller.exception.CommunicationsException: org.apache.nifi.controller.exception.CommunicationsException: Unable to communicate with Remote NiFi at URI https://sj46.xxxx:6689/nifi due to: Connect to sj46.xxxx:6689 [sj46.xxxx/x.x.x.x] failed: Connection timed out: connect
at org.apache.nifi.remote.StandardRemoteProcessGroup.refreshFlowContents(StandardRemoteProcessGroup.java:882)
at org.apache.nifi.remote.StandardRemoteProcessGroup.lambda$initialize$0(StandardRemoteProcessGroup.java:193)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.nifi.controller.exception.CommunicationsException: Unable to communicate with Remote NiFi at URI https://sj46.xxxx:6689/nifi due to: Connect to sj46.xxxx:6689 [sj46.xxxx/x.x.x.x] failed: Connection timed out: connect
at org.apache.nifi.remote.StandardRemoteProcessGroup.refreshFlowContents(StandardRemoteProcessGroup.java:844)
... 8 common frames omitted
有关如何解决此问题的任何指示?
Any pointers on how to troubleshoot this issue?
通过从MiNiFi服务器禁用防火墙解决了上述问题.现在我遇到了另一个类似下面的问题:
the above issue got resolved by disabling the firewall from MiNiFi server.now i am getting another issue like below:
WARN [Remote Process Group 6bd4b280-646b-3b93-0000-000000000000 Thread-1] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://sj46.xxxx:6689/nifi-api due to javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
WARN [Remote Process Group 6bd4b280-646b-3b93-0000-000000000000 Thread-1] o.a.n.remote.StandardRemoteProcessGroup Unable to connect to RemoteProcessGroup[https://sj46.xxxx:6689/nifi] due to javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
通过将SSl证书(您可以在NiFi.properties中，尤其是keystore.jks和truststore.jks中引用)证书导入到MiNiFi，已解决了上述SSLHandshakeException问题.我们需要在MiNiFi中手动修改config.yml.
the above issue of SSLHandshakeException has been resolved by importing the SSl certificate (which you can refer in NiFi.properties especially keystore.jks and truststore.jks) to the MiNiFi. we need to manually modify config.yml in MiNiFi.
在此问题之后，我面临以下问题:
after this issue, i was facing the below one:
ERROR [Timer-Driven Process Thread-6] o.a.n.c.t.ContinuallyRunConnectableTask RemoteGroupPort[name=From MiNiFi,targets=https://sj46.xxxx:6689/nifi] failed to process session due to java.lang.RuntimeException: java.lang.NoSuchMethodError: org.apache.http.impl.client.HttpClientBuilder.setSSLContext(Ljavax/net/ssl/SSLContext;)Lorg/apache/http/impl/client/HttpClientBuilder;
ERROR [Timer-Driven Process Thread-6] o.a.n.c.t.ContinuallyRunConnectableTask
java.lang.RuntimeException: java.lang.NoSuchMethodError: org.apache.http.impl.client.HttpClientBuilder.setSSLContext(Ljavax/net/ssl/SSLContext;)Lorg/apache/http/impl/client/HttpClientBuilder;
通过将MiNiFi版本从0.4更改为0.3，已经解决了SSLContext问题.
the SSLContext issue has been resolved by changing the MiNiFi version from 0.4 to 0.3.
现在，我可以成功地从MiNiFi服务器获取数据到NiFi了.
Now i am successfully able to fetch data from MiNiFi server to the NiFi.
谢谢.
推荐答案
我看到您的远程端口配置为6698.
I see that your remote port is configured at 6698.
-
尝试从运行MiNiFi的计算机上通过6698远程登录NiFi主机.尝试查看在NiFi主机上配置的入站规则. (通过远程登录到特定端口上的远程主机是为了检查您是否能够通过该特定端口与该主机进行通信.这与计算机上的特定端口是否打开无关.)
Try to telnet the NiFi Host on 6698 from the machine where MiNiFi is running. Try to see the inbound rules configured on the NiFi Host. (Telnetting to a remote host on a specific port is to check whether you are able to communicate with that host over that specific port. It has got nothing to do with whether the specific port on your machine is open or not. )
在#Site下的nifi.properties中将NiFi主机名添加到站点属性"中
Add the NiFi Host name in nifi.properties under #Site to Site Properties
nifi.remote.input.host =运行NiFi的主机
我建议，只是为了消除对NiFi Web URL端口和远程输入端口的常见模式的迷惑，请尝试在nifi.properties中进行更改
I would suggest, just to remove the mist of confusion over the common patterns of NiFi Web URL Port and Remote Input Port, try changing this in nifi.properties
nifi.remote.input.socket.port = 1026
还要在config.yml的输入端口"下添加以下部分:
Also add the below section under Input Ports in config.yml:
属性:
端口:6698 主机名:
Properties:
Port: 6698 Host Name:
有关config.yml和nifi.properties修改参考，请检查.
For config.yml and nifi.properties modification references, check this.
这篇关于MiNiFi-NiFi无法与主机通信:端口连接超时的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！