使用时间表时Elasticsearch/Logstash复制输出

问题描述

我将Elasticsearch与Logstash结合使用.我想在数据库更改时更新索引.因此，我决定使用LS schedule .但是每隔1分钟输出将附加到数据库表记录中.示例:合同表有2行.前1分钟总计:2，总输出后1分钟是:4；我该如何解决?

Im using Elasticsearch with Logstash. I want to update indexes when database changes. So i decided to use LS schedule. But every 1 minute output appended by database table records. Example: contract table has 2 rows.First 1 minute total: 2, 1 minute after total output is : 4;How can i solve this?

有我的配置文件.命令是 bin/logstash -f contract.conf

There is my config file. Command is bin/logstash -f contract.conf

input {
        jdbc {
            jdbc_connection_string => "jdbc:postgresql://localhost:5432/resource"
            jdbc_user => "postgres"
            jdbc_validate_connection => true
            jdbc_driver_library => "/var/www/html/iltodgeree/logstash/postgres.jar"
            jdbc_driver_class => "org.postgresql.Driver"
            statement => "SELECT * FROM contracts;"
            schedule => "* * * * *"
            codec => "json"
        }
    }

    output {
        elasticsearch {
            index => "resource_contracts"
            document_type => "metadata"
            hosts => "localhost:9200"
        }
    }

推荐答案

您需要通过指定document_id设置并使用合同表中的ID字段来修改输出.这样，您将永远不会得到重复.

You need to modify your output by specifying the document_id setting and use the ID field from your contracts table. That way, you'll never get duplicates.

output {
    elasticsearch {
        index => "resource_contracts"
        document_type => "metadata"
        document_id => "%{ID_FIELD}"
        hosts => "localhost:9200"
    }
}

如果contracts表中有更新时间戳，则可以如下所示在输入中修改SQL语句，以便仅复制最近更改的记录:

Also if you have an update timestamp in your contracts table, you can modify the SQL statement in your input like below in order to only copy the records that changed recently:

        statement => "SELECT * FROM contracts WHERE timestamp > :sql_last_value;"

这篇关于使用时间表时Elasticsearch/Logstash复制输出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！