如何列出AWS Glue目录中的所有数据库和表?

问题描述

我在AWS Glue控制台中创建了一个开发端点，现在我可以在gumpyspark控制台中访问SparkContext和SQLContext.

I created a Development Endpoint in the AWS Glue console and now I have access to SparkContext and SQLContext in gluepyspark console.

如何访问目录并列出所有数据库和表?通常的sqlContext.sql("show tables").show()不起作用.

How can I access the catalog and list all databases and tables? The usual sqlContext.sql("show tables").show() does not work.

CatalogConnection类，但我不知道它在哪个软件包中.我尝试从awsglue.context导入，但没有成功.

What might help is the CatalogConnection Class but I have no idea in which package it is. I tried importing from awsglue.context and no success.

推荐答案

我花了几个小时试图找到有关CatalogConnection类的一些信息，但没有找到任何东西. (即使在aws-glue-lib存储库 https://github.com/awslabs/aws-glue -libs )

I spend several hours trying to find some info about CatalogConnection class but haven't found anything. (Even in the aws-glue-lib repository https://github.com/awslabs/aws-glue-libs)

就我而言，我需要在Glue Job Script控制台中使用表名

In my case I needed table names in Glue Job Script console

最后，我使用boto库并通过Glue客户端检索数据库和表名:

Finally I used boto library and retrieved database and table names with Glue client:

import boto3


client = boto3.client('glue',region_name='us-east-1')

responseGetDatabases = client.get_databases()

databaseList = responseGetDatabases['DatabaseList']

for databaseDict in databaseList:

    databaseName = databaseDict['Name']
    print '\ndatabaseName: ' + databaseName

    responseGetTables = client.get_tables( DatabaseName = databaseName )
    tableList = responseGetTables['TableList']

    for tableDict in tableList:

         tableName = tableDict['Name']
         print '\n-- tableName: '+tableName

重要的是正确设置区域

参考:get_databases- http://boto3.readthedocs.io/en/latest/reference/services/glue.html#Glue.Client.get_databases

Reference:get_databases - http://boto3.readthedocs.io/en/latest/reference/services/glue.html#Glue.Client.get_databases

get_tables- http://boto3.阅读thedocs.io/en/latest/reference/services/glue.html#Glue.Client.get_tables

get_tables - http://boto3.readthedocs.io/en/latest/reference/services/glue.html#Glue.Client.get_tables

这篇关于如何列出AWS Glue目录中的所有数据库和表?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！