本文介绍了如何从直线访问 Metastore?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我需要做一些 SQL 查询(如 此处) 直接来自 Metasore.
PS:SHOW/DESCRIBE 命令不够.

I need to do some SQL queries (as here) directly from Metasore.
PS: the commands SHOW/DESCRIBE are not enough.

如何从它作为数据库启用访问,或者Metastore的数据库名称是什么?......在如今(2019年)有可能吗?

How to enable access from it as database, or what the database name of Metastore? ... In nowadays (2019) it is possible?

注意事项

  • 什么是Metastore?
    对我来说是 Hive 架构的一个非常重要的元素,最终用户需要对其进行一些访问... 所有 Hive 实现都需要一个 Metastore 服务,它存储元数据.它是使用关系表实现的数据库.默认情况下,Hive 使用内置的 Derby SQL 服务器", 1.
    当然,您需要在您的上下文中一个标准"Metastore.在我公司的 Hadoop 集群上,我们计划标准化 Metastore(本地和长期标准),可能是 PostgreSQL 以及 (PostgREST API 用于外部使用它的一些 SQL 视图).

  • What is Metastore?
    For me is a very important element of the Hive architecture, final user need some access to it... "All Hive implementation need a metastore service, where it stores metadata. It is implemented using tables in relational database. By default, Hive uses built-in Derby SQL server", 1.
    Of course, you need in your context a "standard" Metastore. On my corporation's Hadoop cluster we are planning to standardize Metastore (local and long term standard), perhaps PostgreSQL and also an (PostgREST API for external consume of some SQL-Views from it).

当 Metastore 是一个长期的本地标准时,SQL 定义(表名等)将是稳定的,Metastore 查询将是可靠的.

The SQL definitions (table names, etc.) will be stable and Metastore queries will be reliable when Metastore is a long-term local standard.

Metastore 它与 Hive 紧密相连,它是一个 Java API,但 Metastore 也是一个标准的 RDBMS,并为外部宇宙提供标准连接(通过 SQL).
PS:我对 Metastore 的兴趣在于这种外部环境.

The Metastore it is closely connected to Hive, where it is a Java API, but Metastore is also a standard RDBMS and offers standard connection (by SQL) for the external universe.
PS: my interest on Metastore is in this external context.

推荐答案

Spark-shell 解决方案

Spark 访问 Metastore 引擎,它有一个一流元数据方法,返回数据帧,schema 属性,公开名称、类型等,并提供 getComment 方法.

Spark-shell solution

Spark access Metastore under the hood, it have a first class metadata method, that returns a dataframe, the schema property, that expose names, types, etc. and offers getComment method.

参见https://stackoverflow.com/a/57857021/287948

这篇关于如何从直线访问 Metastore?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

08-04 16:24
查看更多