问题描述
我需要做一些 SQL 查询(如 此处) 直接来自 Metasore.
PS:SHOW/DESCRIBE 命令不够.
I need to do some SQL queries (as here) directly from Metasore.
PS: the commands SHOW/DESCRIBE are not enough.
如何从它作为数据库启用访问,或者Metastore的数据库名称是什么?......在如今(2019年)有可能吗?
How to enable access from it as database, or what the database name of Metastore? ... In nowadays (2019) it is possible?
注意事项
什么是Metastore?
对我来说是 Hive 架构的一个非常重要的元素,最终用户需要对其进行一些访问... 所有 Hive 实现都需要一个 Metastore 服务,它存储元数据.它是使用关系表实现的数据库.默认情况下,Hive 使用内置的 Derby SQL 服务器", 1.
当然,您需要在您的上下文中一个标准"Metastore.在我公司的 Hadoop 集群上,我们计划标准化 Metastore(本地和长期标准),可能是 PostgreSQL 以及 (PostgREST API 用于外部使用它的一些 SQL 视图).
What is Metastore?
For me is a very important element of the Hive architecture, final user need some access to it... "All Hive implementation need a metastore service, where it stores metadata. It is implemented using tables in relational database. By default, Hive uses built-in Derby SQL server", 1.
Of course, you need in your context a "standard" Metastore. On my corporation's Hadoop cluster we are planning to standardize Metastore (local and long term standard), perhaps PostgreSQL and also an (PostgREST API for external consume of some SQL-Views from it).
当 Metastore 是一个长期的本地标准时,SQL 定义(表名等)将是稳定的,Metastore 查询将是可靠的.
The SQL definitions (table names, etc.) will be stable and Metastore queries will be reliable when Metastore is a long-term local standard.
Metastore 它与 Hive 紧密相连,它是一个 Java API,但 Metastore 也是一个标准的 RDBMS,并为外部宇宙提供标准连接(通过 SQL).
PS:我对 Metastore 的兴趣在于这种外部环境.
The Metastore it is closely connected to Hive, where it is a Java API, but Metastore is also a standard RDBMS and offers standard connection (by SQL) for the external universe.
PS: my interest on Metastore is in this external context.
推荐答案
Spark-shell 解决方案
Spark 访问 Metastore 引擎,它有一个一流元数据方法,返回数据帧,schema
属性,公开名称、类型等,并提供 getComment
方法.
Spark-shell solution
Spark access Metastore under the hood, it have a first class metadata method, that returns a dataframe, the schema
property, that expose names, types, etc. and offers getComment
method.
参见https://stackoverflow.com/a/57857021/287948
这篇关于如何从直线访问 Metastore?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!