问题描述
目标:我试图在我的pycharm IDE中正确解释apache-spark pyspark.
Goal: I am trying to get apache-spark pyspark to be appropriately interpreted within my pycharm IDE.
问题:我目前收到以下错误:
Problem: I currently receive the following error:
ImportError: cannot import name accumulators
我正在关注以下博客，以帮助我完成整个过程. http://renien.github.io/blog/accessing-pyspark-pycharm/
I was following the following blog to help me through the process. http://renien.github.io/blog/accessing-pyspark-pycharm/
由于我的代码采用了我个人尝试过的除外路径:except:只是为了查看确切的错误是什么.
Due to the fact my code was taking the except path I personally got rid of the try: except: just to see what the exact error was.
在此之前，我收到以下错误:
Prior to this I received the following error:
ImportError: No module named py4j.java_gateway
只需在bash中键入"$ sudo pip install py4j"即可解决此问题.
This was fixed simply by typing '$sudo pip install py4j' in bash.
我的代码当前如下所示:
My code currently looks like the following chunk:
import os
import sys
# Path for spark source folder
os.environ['SPARK_HOME']="[MY_HOME_DIR]/spark-1.2.0"
# Append pyspark to Python Path
sys.path.append("[MY_HOME_DIR]/spark-1.2.0/python/")
try:
from pyspark import SparkContext
print ("Successfully imported Spark Modules")
except ImportError as e:
print ("Can not import Spark Modules", e)
sys.exit(1)
我的问题:
1.此错误的根源是什么?原因是什么? 2.如何解决该问题，以便可以在pycharm编辑器中运行pyspark.
My Questions:
1. What is the source of this error? What is the cause? 2. How do I remedy the issue so I can run pyspark in my pycharm editor.
注意:我在pycharm中使用的当前解释器是Python 2.7.8(〜/anaconda/bin/python)
NOTE: The current interpreter I use in pycharm is Python 2.7.8 (~/anaconda/bin/python)
提前谢谢！
唐
推荐答案
首先，设置您的环境var
Firstly, set your environment var
export SPARK_HOME=/home/.../Spark/spark-2.0.1-bin-hadoop2.7
export PYTHONPATH=$SPARK_HOME/python/:$SPARK_HOME/python/lib/py4j-0.10.3-src.zip:$PYTHONPATH
PATH="$PATH:$JAVA_HOME/bin:$SPARK_HOME/bin:$PYTHONPATH"
确保使用自己的版本名称
make sure that you use your own version name
然后重新启动！确认您的设置很重要.
and then, restart! it is important to validate you setting.
这篇关于pyspark ImportError:无法导入名称累加器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！