问题描述
我在弹性搜索引擎中加载了事件日志，并使用Kibana对其进行了可视化处理。我的事件日志实际上存储在Google Big Query表中。目前，我正在将json文件转储到Google存储桶中，并将其下载到本地驱动器。然后使用logstash，将json文件从本地驱动器移至弹性搜索引擎。
I have the event logs loaded in elasticsearch engine and I visualise it using Kibana. My event logs are actually stored in the Google Big Query table. Currently I am dumping the json files to a Google bucket and download it to a local drive. Then using logstash, I move the json files from the local drive to the elastic search engine.
现在，我正在尝试通过在Google大查询和弹性搜索之间建立联系来实现流程自动化。根据我的阅读，我了解到有一个输出连接器，可将数据从弹性搜索发送到Google大查询，但反之则不然。只是想知道我是否应该将json文件上传到kubernete集群，然后在集群和Elastic搜索引擎之间建立连接。
Now, I am trying to automate the process by establishing the connection between google big query and elastic search. From what I have read, I understand that there is a output connector which sends the data from elastic search to Google big query but not vice versa. Just wondering whether I should upload the json file to a kubernete cluster and then establish the connection between the cluster and Elastic search engine.
在此方面的任何帮助将不胜感激。
Any help with this regard would be appreciated.
推荐答案
Apache Beam具有用于BigQuery和Elastic Search的连接器，我将使用DataFlow明确地执行此操作，因此您无需实现复杂的ETL和临时存储。您可以使用 BigQueryIO.Read.from 从BigQuery读取数据（如果性能很重要，请查看此内容），然后使用 ElasticsearchIO.write（）
Apache Beam has connectors for BigQuery and Elastic Search, I would definitly perform this using DataFlow so you don´t need to implement a complex ETL and staging storage. You can read the data from BigQuery using BigQueryIO.Read.from (take a look to this if performance is important BigQueryIO Read vs fromQuery) and load it into ElasticSearch using ElasticsearchIO.write()
请参阅此如何从BigQuery Dataflow中读取数据
Refer this how read data from BigQuery Dataflow
弹性搜索索引
已更新2019-06-24
UPDATED 2019-06-24
今年最近发布了BigQuery Storage API，该API改进了从BigQuery提取数据的并行性，并由DataFlow原生支持。请参阅了解更多详情。
Recently this year was release BigQuery Storage API which improve the parallelism to extract data from BigQuery and is natively supported by DataFlow. Refer to https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-api for more details.
从文档中
这篇关于使用Google Big Query进行弹性搜索的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！