（Docker）中缺少GOMP_parallel或GOMP_4.0 jupyter / all-spark-notebook：R和Python

问题描述

我有一个关于GOMP的问题，我认为与gcc有关。

我使用这个图像作为一些实验的基础。看起来我使用了一个基于这个未修改图像的良好版本（4.9.2），但是我得到了一些与Cython相关的软件包的奇怪行为。

示例1

在未修改的jupyter / all-spark-notebook docker镜像上运行时，尝试运行Python时，出现以下问题：

  import lightfm 
 
 Traceback（最近一次调用最后一次）：
 
文件，第1行， in 
 
文件/opt/conda/lib/python3.4/site-packages/lightfm/init.py，第1行，来自.lightfm的
 import LightFM 
 
文件/opt/conda/lib/python3.4/site-packages/lightfm/lightfm.py，第7行，来自.lightfm_fast导入的
（CSRMatrix，FastLightFM，
 
 ImportError：/opt/conda/lib/python3.4/site-packages/lightfm/lightfm_fast.cpython-34m.so：
 undefined symbol：GOMP_parallel 
  

我曾试过的事情：

1. 文件lightfm.py并移动所有的d依赖于一行，如下所示：

   
   
   

   来自.lightfm_fast导入（CSRMatrix，FastLightFM，
fit_logistic，predict_lightfm，
fit_warp ，fit_bpr，fit_warp_kos）

（CSRMatrix，FastLightFM，fit_logistic，predict_lightfm，fit_warp，fit_bpr，fit_warp_kos）的$ b

同样的错误。



.lightfm 更改为 lightfm 以更改相对导入。同样的错误。



 检查gcc和内核版本：gcc 4.9.2 Ubuntu 14.04 Linux 00846c176840 3.13.0-67-generic＃110 -Ubuntu SMP Fri Oct 23 13:24:41 UTC 2015 x86_64 GNU / Linux 
  

但是我想如果你只需要拖动docker镜像并进行pip安装 lightfm 它应该精确地复制错误。 示例2





运行 pip install xgboost 时，安装成功，但是在笔记本中尝试执行：

  ------------------------------ --------------------------------------------- 
 OSError Traceback（最近一次调用最后一次）
< ipython-input-2-afdaff4619ce> in< module>（）
 ----> 1在< module>中导入xgboost 
 
 /home/jovyan/.local/lib/python3.5/site-packages/xgboost-0.4-py3.5.egg/xgboost/__init__.py（ ）
 9进口os 
 10 
 ---> 11。从核心进口DMatrix，增加
 12，来自训练进口火车，cv 
 13。在< module>（）中导入价格
 
 /home/jovyan/.local/lib/python3.5/site-packages/xgboost-0.4-py3.5.egg/xgboost/core.py 
 81 
 82＃全局加载XGBoost库
 ---> 83 _LIB = _load_lib（）
 84 
 85 def _check_call（ret）：
 
 /home/jovyan/.local/lib/python3.5/site-packages/xgboost-在_load_lib（）中，0.4-py3.5.egg / xgboost / core.py 
 75如果len（lib_path）== 0：
 76返回None 
 ---> 77 lib = ctypes.cdll.LoadLibrary（lib_path [0]）
 78 lib.XGBGetLastError.restype = ctypes.c_char_p 
 79返回lib 
 $ b $ / opt / conda / lib /在LoadLibrary（self，name）中的python3.5 / ctypes / __ init__.py 
 423 
 424 def LoadLibrary（self，name）：
  - > 425返回self._dlltype（名称）
 426 
 427 cdll = LibraryLoader（CDLL）
 
 /opt/conda/lib/python3.5/ctypes/__init__.py in __init__ （self，name，mode，handle，use_errno，use_last_error）
 345 
 346如果句柄是None：
  - > 347 self._handle = _dlopen（self._name，mode）
 348 else：
 349 self._handle =句柄
 
 OSError：/opt/conda/bin/../ lib / libgomp.so.1：未找到版本`GOMP_4.0'（必需/home/jovyan/.local/lib/python3.5/site-packages/xgboost-0.4-py3.5.egg/xgboost/libxgboost .so）
  

我刚刚注意到这是一个反复出现的模式，有时可能相当有限，但我不明白编译器是否知道这实际上是一个图像问题，还是这不是一个真正的问题，而是一个设计决策。任何想法？



非常感谢！





我试过的东西：






1. 我试过
   conda install -c https://conda.anaconda.org/anaconda gcc 在终端
   但无济于事。

另外，当我试图做 install.packages（'xgboost' ）（相当于）在R中，我得到这个错误：



错误：.onLoad在'xgboost'的loadNamespace（）失败，详细信息：
调用：dyn.load（文件，DLLpath = DLLpath，...）
错误：无法加载共享对象'/usr/local/spark-1.6.0-bin-hadoop2.6/R /lib/xgboost/libs/xgboost.so'：
/usr/local/spark-1.6.0-bin-hadoop2.6/R/lib/xgboost/libs/xgboost.so：undefined symbol：GOMP_parallel




sudo apt-get install libgomp1 也没有帮助我因为它已经安装了。





1. 根据项目Jupyter的建议，我
   运行 conda list 并找到以下行：
   libgcc 4.8.5 1 r



但是没有gcc我也有glib。
但是在终端：

  jovyan @ aaaaaaa：〜/ work $ gcc --version 
 
 gcc（Debian 4.9.2-10）4.9.2 
版权所有（C）2014 Free Software Foundation，Inc. 
这是免费软件;请参阅复制条件的来源。没有任何b $ b保修;甚至不适用于适销性或特定美元的适用性b $ b  

更新：我现在认为检测到的编译器实际上是一个较早的版本，所以也许这是问题的根源，因为在运行时

 ％install_ext https ：//raw.githubusercontent.com/rasbt/watermark/master/watermark.py 
％load_ext水印
＃显示此环境的水印
％watermark -d -m -v -p numpy，matplotlib -g 
  

我收到以下回应：

  CPython 2.7.11 
 IPython 4.1.2 
 
 numpy 1.10.4 
 matplotlib 1.5.1 
 
编译器：GCC 4.4.7 20120313（Red Hat 4.4.7-1）
系统：Linux 
发行版：4.4.5-15.26.amzn1.x86_64 
 machine：x86_64 
处理器：
 CPU核心：4 
解释器：64位
 Git哈希：
  

还有其他想法吗？


解决方案

大多数可能作者已经找到了答案，但是我仍然会将其留在这里，因为它可能为某人节省时间。





这个问题的确与gcc的错误版本有关用过的。在 all-spark-notebook 之上安装xgboost之前运行此命令帮助我解决了gomp问题：

 运行conda install -y gcc 
  

您可以看到工作解决方案在我的 all-spark-notebook 扩展名。




I have a question regarding GOMP, that I figured is related to gcc.

I am using this image as a base for some experiments. It seems that I am using an alright version (4.9.2) based on this unmodified image, but I get some strange behaviour with a number of Cython related packages.

Example 1

When running this on the unmodified jupyter/all-spark-notebook docker image, when attempting to run Python I get the following issue,

import lightfm

Traceback (most recent call last):

File "", line 1, in

File "/opt/conda/lib/python3.4/site-packages/lightfm/init.py", line 1, in 
from .lightfm import LightFM

File "/opt/conda/lib/python3.4/site-packages/lightfm/lightfm.py", line 7, in 
from .lightfm_fast import (CSRMatrix, FastLightFM,

ImportError: /opt/conda/lib/python3.4/site-packages/lightfm/lightfm_fast.cpython-34m.so:
undefined symbol: GOMP_parallel

Things I have tried:

1. Going to the file lightfm.py and moving all of the dependencies onto one line like so:

   from .lightfm_fast import (CSRMatrix, FastLightFM,fit_logistic, predict_lightfm,fit_warp, fit_bpr, fit_warp_kos)

to

from .lightfm_fast import (CSRMatrix, FastLightFM, fit_logistic, predict_lightfm, fit_warp, fit_bpr, fit_warp_kos).

Same error.

Also tried ".lightfm" to "lightfm" to change from relative import. Same error.

Checking gcc and kernel versions: gcc 4.9.2 Ubuntu 14.04 Linux 00846c176840 3.13.0-67-generic #110-Ubuntu SMP Fri Oct 23 13:24:41 UTC 2015 x86_64 GNU/Linux

But I think if you just pull the docker image and do a pip install lightfm it should replicate the error precisely.

Example 2

When running pip install xgboost, the installation succeeds, but when in notebook I attempt to do:

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-2-afdaff4619ce> in <module>()
----> 1 import xgboost

/home/jovyan/.local/lib/python3.5/site-packages/xgboost-0.4-py3.5.egg/xgboost/__init__.py in <module>()
      9 import os
     10 
---> 11 from .core import DMatrix, Booster
     12 from .training import train, cv
     13 from . import rabit

/home/jovyan/.local/lib/python3.5/site-packages/xgboost-0.4-py3.5.egg/xgboost/core.py in <module>()
     81 
     82 # load the XGBoost library globally
---> 83 _LIB = _load_lib()
     84 
     85 def _check_call(ret):

/home/jovyan/.local/lib/python3.5/site-packages/xgboost-0.4-py3.5.egg/xgboost/core.py in _load_lib()
     75     if len(lib_path) == 0:
     76         return None
---> 77     lib = ctypes.cdll.LoadLibrary(lib_path[0])
     78     lib.XGBGetLastError.restype = ctypes.c_char_p
     79     return lib

/opt/conda/lib/python3.5/ctypes/__init__.py in LoadLibrary(self, name)
    423 
    424     def LoadLibrary(self, name):
--> 425         return self._dlltype(name)
    426 
    427 cdll = LibraryLoader(CDLL)

/opt/conda/lib/python3.5/ctypes/__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error)
    345 
    346         if handle is None:
--> 347             self._handle = _dlopen(self._name, mode)
    348         else:
    349             self._handle = handle

OSError: /opt/conda/bin/../lib/libgomp.so.1: version `GOMP_4.0' not found (required by /home/jovyan/.local/lib/python3.5/site-packages/xgboost-0.4-py3.5.egg/xgboost/libxgboost.so)

I have just noticed that this is a recurring pattern and at times can be quite limiting, but I don't understand compilers that well to know if this actually a problem with the image or if that's not really an 'issue', but rather a design decision. Any ideas?

Much appreciated!

Things I have tried:

1. I triedconda install -c https://conda.anaconda.org/anaconda gcc in the terminalbut to no avail.

2. Also when I tried to do install.packages('xgboost') (equivalent) in R, I got this error:

   Error : .onLoad failed in loadNamespace() for 'xgboost', details:call: dyn.load(file, DLLpath = DLLpath, ...)error: unable to load shared object '/usr/local/spark-1.6.0-bin-hadoop2.6/R/lib/xgboost/libs/xgboost.so':/usr/local/spark-1.6.0-bin-hadoop2.6/R/lib/xgboost/libs/xgboost.so: undefined symbol: GOMP_parallel

sudo apt-get install libgomp1 also did not help me as it was already installed.

1. Upon suggestion of someone from project Jupyter I ran conda list and found this line:libgcc 4.8.5 1 r

But no gcc I also have glib.But in terminal:

jovyan@aaaaaaa:~/work$ gcc --version      

gcc (Debian 4.9.2-10) 4.9.2                                                             
Copyright (C) 2014 Free Software Foundation, Inc.                                       
This is free software; see the source for copying conditions.  There is NO              
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR 

UPDATE: I now think that the detected compiler is actually an earlier version, so perhaps that is the source of the problem, because upon running

%install_ext https://raw.githubusercontent.com/rasbt/watermark/master/watermark.py
%load_ext watermark
# show a watermark for this environment
%watermark -d -m -v -p numpy,matplotlib -g

I get the following response:

CPython 2.7.11
IPython 4.1.2

numpy 1.10.4
matplotlib 1.5.1

compiler   : GCC 4.4.7 20120313 (Red Hat 4.4.7-1)
system     : Linux
release    : 4.4.5-15.26.amzn1.x86_64
machine    : x86_64
processor  : 
CPU cores  : 4
interpreter: 64bit
Git hash   : 

Any other thoughts?

解决方案

Most probably author did already find the answer, however I will still leave it here as it may potentially save time for someone.

The issue is indeed related to the wrong version of gcc used. Running this command before installing xgboost on top of all-spark-notebook helped me to fix the gomp issue:

RUN conda install -y gcc

You can see the working solution in my extension of all-spark-notebook on github.

这篇关于（Docker）中缺少GOMP_parallel或GOMP_4.0 jupyter / all-spark-notebook：R和Python的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！