elasticsearch - elasticsearch 7无法将映射放在索引logstash类型事件和拒绝映射更新上

我们有一个带有7.3.2版本的elasticsearch的新集群。我们使用rsyslog将客户端日志发送到elasticsearch节点。

通过删除版本7中弃用的事件和类型，我将Logstash模板从6更新为7。

我面临的错误。

[2020-03-17T09:13:08,861][DEBUG][o.e.c.s.MasterService    ] [prod-apm-elasticsearch103.example.com] processing [put-mapping[events]]: took [2ms] no change in cluster state
[2020-03-17T09:13:08,964][DEBUG][o.e.c.s.MasterService    ] [prod-apm-elasticsearch103.example.com] processing [put-mapping[events]]: execute
[2020-03-17T09:13:08,967][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [prod-apm-elasticsearch103.example.com] failed to put mappings on indices [[[logstash-2020.03.17/7_uGNP-iSCOxGczC2_xvfA]]], type [events]
java.lang.IllegalArgumentException: Rejecting mapping update to [logstash-2020.03.17] as the final mapping would have more than 1 type: [_doc, events]
    at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:272) ~[elasticsearch-7.3.2.jar:7.3.2]
    at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:238) ~[elasticsearch-7.3.2.jar:7.3.2]
    at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:687) ~[elasticsearch-7.3.2.jar:7.3.2]
    at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:310) ~[elasticsearch-7.3.2.jar:7.3.2]
    at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:210) [elasticsearch-7.3.2.jar:7.3.2]
    at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) [elasticsearch-7.3.2.jar:7.3.2]
    at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.3.2.jar:7.3.2]
    at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.3.2.jar:7.3.2]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) [elasticsearch-7.3.2.jar:7.3.2]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.3.2.jar:7.3.2]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.3.2.jar:7.3.2]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:835) [?:?]

以下是rsyslog转发器配置。

#This is the same as the default template, but allows for tags longer than 32 characters.
#See http://www.rsyslog.com/sende-messages-with-tags-larger-than-32-characters/ for an explanation
template (name="LongTagForwardFormat" type="string" string="<%PRI%>%TIMEGENERATED:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%")

# this is for index names to be like: logstash-YYYY.MM.DD
template(name="logstash-index" type="list") {
  constant(value="logstash-")
  property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
  constant(value=".")
  property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
  constant(value=".")
  property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
}


action(type="mmjsonparse")

template(name="plain-syslog" type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"containerId\":\"") property(name="hostname")
      constant(value="\",\"host\":\"")        property(name="$myhostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"tag\":\"")         property(name="programname" format="json") #name of process
      constant(value="\",")
      property(name="$!all-json" position.from="2")
}


# ship logs to Elasticsearch, contingent on having an applog_es_server defined
local0.* action(type="omelasticsearch"
  template="plain-syslog"
  #template="logstash"
  searchIndex="logstash-index"
  dynSearchIndex="on"
  #asyncrepl="off"
  bulkmode="on"
  queue.dequeuebatchsize="250"
  queue.type="linkedlist"
  queue.filename="syslog-elastic"
  queue.maxdiskspace="1G"
  queue.highwatermark="10000"
  queue.lowwatermark="5000"
  queue.size="2000000"
  queue.timeoutEnqueue="0"
  queue.timeoutshutdown="5000"
  queue.saveonshutdown="on"
  action.resumeretrycount="1"
  server=["prod-routing101.example.com:9200"]

rsyslogd 8.1908.0(aka 2019.08)编译为:
平台:x86_64-pc-linux-gnu
平台(lsb_release -d):
FEATURE_REGEXP:是
GSSAPI Kerberos 5支持:否
FEATURE_DEBUG(调试版本，慢速代码):否
支持32位原子操作:是
支持64位原子操作:是
内存分配器:系统默认值
运行时检测(慢速代码):否
uuid支持:是
系统支持:否
配置文件:/etc/rsyslog.conf
PID文件:/var/run/rsyslogd.pid
RainerScript整数的位数:64

请帮助我解决问题。

有关我使用的模板的信息，请引用elasticsearch mapper_parsing_exception Root mapping definition has unsupported parameters

提前致谢。

最佳答案

在rsyslog conf(applog forwader)中添加searchType="_doc"后工作。

# ship logs to Elasticsearch, contingent on having an applog_es_server defined
local0.* action(type="omelasticsearch"
  template="plain-syslog"
  #template="logstash"
  searchIndex="logstash-index"
  dynSearchIndex="on"
  searchType="_doc"
  #asyncrepl="off"
  bulkmode="on"
  queue.dequeuebatchsize="250"
  queue.type="linkedlist"
  ...
  ...
  ...

关于elasticsearch - elasticsearch 7无法将映射放在索引logstash类型事件和拒绝映射更新上，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/60723455/