问题描述
我正在尝试使用 cmake 构建一个程序.出于多种原因,必须使用静态库而不是动态库来构建程序,而且我需要使用 PyTorch,所以这就是我所做的:
I'm trying to build a program using cmake. For several reasons, the program must be built using static libraries rather than dynamic libraries, and I need to use PyTorch so this is what I've done:
- 下载并安装了 PyTorch 静态库(我在正确的路径中找到了
libtorch.a
,在/home/me/pytorch/torch/lib
) - 使用以下内容制作
CMakeLists.txt
:
cmake_minimum_required(VERSION 3.5.1 FATAL_ERROR)
project(example-app LANGUAGES CXX)
find_package(Torch REQUIRED)
add_executable(example-app example-app.cpp argparse/argparse.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}" -static -fopenmp)
set_property(TARGET example-app PROPERTY CXX_STANDARD 14)
仅供参考,example-app.cpp
是包含 main 函数的文件,argparse/
是一个目录,其中包含 example 中调用的函数的一些源代码-app.cpp
FYI, example-app.cpp
is the file with the main function, and argparse/
is a directory with some source code for functions called in example-app.cpp
它一直工作到 cmake -DCMAKE_PREFIX_PATH=/home/me/pytorch/torch ..
,但是下面的 build
会产生一些错误,说找不到参考一些函数,即以fbgemm::
开头的函数.fbgemm
是(据我所知)某种用于实现 PyTorch 的 GEMM 库.
It works until cmake -DCMAKE_PREFIX_PATH=/home/me/pytorch/torch ..
, but the following build
incurs some errors, saying it could not find the reference to some functions, namely functions starting with fbgemm::
. fbgemm
is (as long as I know) some sort of GEMM library used in implementing PyTorch.
在我看来,在链接静态 PyTorch 库时,其内部库(如 fbgemm
内容)尚未正确链接,但我不是 cmake
方面的专家老实说并不完全确定.
It seems to me that while linking the static PyTorch library, its internal libraries like fbgemm
stuff have not been linked properly, but I'm not an expert on cmake
and honestly not entirely sure.
我做错了什么,还是有解决这个问题的方法?任何帮助或朝正确方向推动的努力将不胜感激.
Am I doing something wrong, or is there a workaround for this problem? Any help or push in the right direction would be greatly appreciated.
附言
确切的错误没有被发布,因为它太长了,但它主要由
未定义引用 ~
错误组成.如果查看错误消息可能对某些人有所帮助,我很乐意编辑问题并将其发布.
The exact error has not been posted because it is way too long, but it consists of mostly
undefined reference to ~
errors. If looking at the error message might be helpful for some people, I'd be happy to edit the question and post it.
build
如果我从代码中删除需要库函数的部分而不注释掉 #include <torch/torch.h>
来自 example-app.cpp
.
build
ing and running the file works fine if I remove the parts that require the library's functions from the code without commenting out #include <torch/torch.h>
from example-app.cpp
.
推荐答案
最近通过 PyTorch 的静态链接经历了类似的过程,老实说它不太漂亮.
Lately went through similar process with static linking of PyTorch and to be honest it wasn't too pretty.
我将概述我已采取的步骤(您可以在 torchlambda 中找到确切的源代码,这里是CMakeLists.txt它还包括 AWS 开发工具包和 AWS Lambda 静态构建),/a> 是从源代码构建
pytorch
的脚本(通过 /scripts/build_mobile.sh
克隆和构建,仅支持 CPU)),虽然它只支持 CPU(尽管如果您需要 CUDA,类似的步骤应该没问题,它至少会让您入门).
首先,您需要预先构建的静态库文件(所有都需要是静态的,因此没有
.so
,只有那些带有的文件.
扩展是合适的).
Tbh 我一直在寻找
PyTorch
在 安装上提供的那些页面,但只有shared
版本.在一个 GitHub 问题中,我找到了一种下载它们的方法,如下所示:
Tbh I've been looking for those provided by
PyTorch
on installation page, yet there is only shared
version.In one GitHub issue I've found a way to download them as follows:
而不是下载(此处通过
wget
)共享库:
Instead of downloading (here via
wget
) shared libraries:
$ wget https://download.pytorch.org/libtorch/cu101/libtorch-shared-with-deps-1.4.0.zip
您将
shared
重命名为 static
(如在这个问题中),所以它会变成:
you rename
shared
to static
(as described in this issue), so it would become:
$ wget https://download.pytorch.org/libtorch/cu101/libtorch-static-with-deps-1.4.0.zip
然而,当你下载它时,
lib
文件夹下没有 libtorch.a
(没有找到 libcaffe2.a
这个问题),所以我剩下的就是从源代码明确构建.
Yet, when you download it there is no
libtorch.a
under lib
folder (didn't find libcaffe2.a
either as indicated by this issue), so what I was left with was building explicitly from source.
如果您以某种方式拥有这些文件(如果有,请提供您从何处获取这些文件),您可以跳过下一步.
If you have those files somehow (if so, please provide where you got them from please), you can skip the next step.
对于 CPU 版本,我使用了 /pytorch/scripts/build_mobile.sh 文件,如果需要 GPU 支持,您可以以此为基础构建您的版本(也许您只需将
-DUSE_CUDA=ON
传递给此脚本,但不确定).
For CPU version I have used /pytorch/scripts/build_mobile.sh file, you can base your version off of this if GPU support is needed (maybe you only have to pass
-DUSE_CUDA=ON
to this script, not sure though).
最重要的是
cmake
的 -DBUILD_SHARED_LIBS=OFF
以便将所有内容构建为 static
库.您还可以检查 脚本build_mobile.sh
的参数也是如此.
Most important is
cmake
's -DBUILD_SHARED_LIBS=OFF
in order to build everything as static
library. You can also check script from my tool which passes arguments to build_mobile.sh
as well.
默认情况下,在上面运行将在
/pytorch/build_mobile/install
中为您提供静态文件,其中有您需要的一切.
Running above will give you static files in
/pytorch/build_mobile/install
by default where there is everything you need.
现在您可以将上述构建文件复制到
/usr/local
(最好不要,除非您使用 Docker
作为 torchlambda
)或设置从您的 CMakeLists.txt
中访问它的路径,如下所示:
Now you can copy above build files to
/usr/local
(better not to unless you are using Docker
as torchlambda
) or set path to it from within your CMakeLists.txt
like this:
set(LIBTORCH "/path/to/pytorch/build_mobile/install")
# Below will append libtorch to path so CMake can see files
set(CMAKE_PREFIX_PATH "${CMAKE_PREFIX_PATH};${LIBTORCH}")
现在其余的都很好除了
target_link_libraries
,应该是(如这个问题,请参阅此处列出的相关问题以获取更多参考)与 -Wl,--whole-archive
链接器标志一起使用,这让我想到这个:
Now the rest is fine except
target_link_libraries
, which should be (as indicated by this issue, see related issues listed there for additional reference) used with -Wl,--whole-archive
linker flag, which brought me to this:
target_link_libraries(example-app PRIVATE -lm
-Wl,--whole-archive "${TORCH_LIBRARIES}"
-Wl,--no-whole-archive
-lpthread
${CMAKE_DL_LIBS})
您可能不需要
-lm
、-lpthread
或 ${CMAKE_DL_LIBS}
,尽管我在 亚马逊 Linux AMI.
You may not need either of
-lm
, -lpthread
or ${CMAKE_DL_LIBS}
, though I needed it when building on Amazon Linux AMI.
现在您可以开始构建应用程序了.标准的
libtorch
方式应该没问题,但这是我使用的另一个命令:
Now you are off to building your application. Standard
libtorch
way should be fine but here is another command I used:
mkdir build &&
cd build &&
cmake .. &&
cmake --build . --config Release
上面将创建
build
文件夹,其中 example-app
二进制文件现在应该是安全的.
Above will create
build
folder where example-app
binary should be now safely located.
最后使用
ld build/example-app
来验证 PyTorch
中的所有内容都是静态链接的,参见 上述问题点5.
,您的输出应该看起来相似.
Finally use
ld build/example-app
to verify everything from PyTorch
was statically linked, see aforementioned issue point 5.
, your output should look similar.
这篇关于cmake - 链接静态库 pytorch 在构建过程中找不到其内部函数的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!