使用PySpark从Azure Blob存储中读取CSV文件

问题描述

我正在尝试使用Microsoft Azure上的PySpark HDInsight群集来做一个机器学习项目.要在群集上运行，请使用Jupyter笔记本.另外，我将数据(一个csv文件)存储在Azure Blob存储中.

I'm trying to do a machine learning project using a PySpark HDInsight cluster on Microsoft Azure. To operate on my cluster a use a Jupyter notebook. Also, I have my data (a csv file), stored on the Azure Blob storage.

根据文档，我文件路径的语法为:

According to the documentation the syntax of the path to my file is:

path = 'wasb[s]://[email protected]/movies_plus_genre_info_2.csv'

但是，当我尝试使用以下命令读取csv文件时:

However, when i try to read the csv file with the following command:

csvFile = spark.read.csv(path, header=True, inferSchema=True)

我收到以下错误:

'java.net.URISyntaxException: Illegal character in scheme name at index 4: wasb[s]://[email protected]/movies_plus_genre_info_2.csv'

这是该错误的屏幕截图，看起来像在笔记本中:

Here is a screenshot of the the error looks like in the notebook:

关于如何解决此问题的任何想法?

Any ideas on how to fix this?

推荐答案

它是(未加密):

wasb://...

或(加密):

wasbs://...

不是

wasb[s]://...

这篇关于使用PySpark从Azure Blob存储中读取CSV文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！