您当前的位置: 首页 >  Python
  • 3浏览

    0关注

    478博文

    0收益

  • 0浏览

    0点赞

    0打赏

    0留言

私信
关注
热门博文

PySpark ERROR: Python in worker has different version 3.9 than that in driver 3.8

高精度计算机视觉 发布时间:2022-07-12 09:44:19 ,浏览量:3

在wiindows环境下,使用PySpark的时候报错:

Python in worker has different version 3.9 than that in driver 3.8, PySpark cannot run with different minor versions

File "E:\Anaconda3\envs\tf38\Lib\site-packages\pyspark\python\lib\pyspark.zip\pyspark\worker.py", line 473, in main
Exception: Python in worker has different version 3.9 than that in driver 3.8, PySpark cannot run with different minor versions. Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

根据报错内容,设置了以下变量,

PYSPARK_PYTHON=E:\Anaconda3\envs\tf38\python.exe

问题解决。

PS: 根据官方的说法,如果设置了PYSPARK_DRIVER_PYTHON,则PYSPARK_DRIVER_PYTHON会优先。我这里没有设置,所以程序直接用的PYSPARK_PYTHON。

在Linux下,一般需要在~/.bashrc中设置,例如

export PYSPARK_PYTHON=~/Anaconda3/envs/tf38/python

另外,找到一篇介绍PYSPARK_DRIVER_PYTHON的文章,里面有pycharm的设置方法,

Demystify Pyspark_driver_Python Comfortably - Python Pool

本文结束 

关注
打赏
1659856567
查看更多评论
立即登录/注册

微信扫码登录

0.4345s