Docker Practice

1. Docker编译安装Python

1.1 Python编译所需必要

  1. 必须手动编译安装openssl(yum install openssl 不生效), 否则python编译完成后, pip install package 会报SSH错误
  2. 版本选择, 在对应链接里下载最新的3.2.x源码即可

1.2 TensorRT-8.6 与CUDA 及 CUDANN

  1. TensorRT与CUDA和CUDNN版本依赖非常强, 如果对应不上, 后续做TRT-ENGINE转换的时候会有很多莫名其妙的错误

1.3 打镜像

  • 使用docker pull nvidia/cuda:12.2.2-cudnn8-devel-centos7将对应的基础镜像拉下来
  • 新建空目录Docker, 下载对应的安装包, 放入Docker/envs/下
  • 使用tar -xvf cudnn-linux-x86_64-8.8.1.3_cuda12-archive.tar.xz解压得到cudnn-linux-x86_64-8.8.1.3_cuda12-archive

文件目录结构如下:

1
2
3
4
5
6
7
8
9
10
11
12
Docker/
├── .dockerignore
├── Dockerfile
└── envs
    ├── Python-3.11.7.tgz
    ├── TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0.tar.gz
    ├── cudnn-linux-x86_64-8.8.1.3_cuda12-archive
    │   ├── LICENSE
    │   ├── include
    │   └── lib
    ├── cudnn-linux-x86_64-8.8.1.3_cuda12-archive.tar.xz
    └── openssl-3.2.0.tar.gz

.dockerignore选择不将部分文件传至docker build的context:

1
2
envs/cudnn-linux-x86_64-8.8.1.3_cuda12-archive.tar.xz
envs/cudnn-linux-x86_64-8.8.1.3_cuda12-archive/*

环境变量:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
alias ls='ls -a'
alias ll='ls -al'
#alias vim='vim -u ~/.vimrc'

# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi

export EDITOR=vim
#export JAVA_HOME=/home/s/apps/inference_server/envs/java-11.0.21
#export CLASSPATH=.:${JAVA_HOME}/lib
#export PATH=${JAVA_HOME}/bin:$PATH
export PATH=$PATH:/usr/local/cuda/bin
export CUDA_HOME=$CUDA_HOME:/usr/local/cuda
export LD_LIBRARY_PATH=/usr/local/TensorRT-8.6.1.6/targets/x86_64-linux-gnu/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH

export LANG="en_US.utf8"
export LC_ALL="en_US.utf8"

编写Dockerfile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
FROM nvidia/cuda:12.2.2-cudnn8-devel-centos7
#将cudnn依赖拷入镜像, 并将下载好的源代码同时拷入
COPY ["envs/cudnn-linux-x86_64-8.8.1.3_cuda12-archive/include/*", "/usr/local/include/"]
COPY ["envs/cudnn-linux-x86_64-8.8.1.3_cuda12-archive/lib/*", "/usr/local/lib64/"]
COPY ["envs/*", "/root"]

#yum update 和yum install 一定是成对使用
RUN yum update -y && \
    yum install -y epel-release \
                patch \
                python-devel \
                gdbm-devel \
                db4-devel \
                libpcap-devel \
                xz-devel \
                snappy-devel \
                libssl-dev \
                libarchive-devel \
                libcurl-devel \
                zlib-devel \
                bzip2-devel \
                openssl-devel \
                openssl11-devel \
                ncurses-devel \
                sqlite-devel \
                readline-devel \
                libffi-devel \
                tk-devel \
                wget \
                gcc \
                gcc-c++ \
                make \
                perl-CPAN \
                libgomp \
                tzdata \
                less \
                vim && \
    mkdir -p /home/hoshinory/envs && \
    ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    ln -sf /usr/local/cuda/targets/x86_64-linux/include /usr/local/cuda/include && \
    tar -zvxf /root/TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0.tar.gz -C /usr/local/ && \
    tar -zvxf /root/Python-3.11.7.tgz -C /home/hoshinory/envs && \
    tar -zvxf /root/openssl-3.2.0.tar.gz -C /home/hoshinory/envs

WORKDIR /home/hoshinory/envs/openssl-3.2.0
#必须安装perl-IPC-Cmd, 否则openssl编译会报错
RUN yum update -y && \
    yum install -y perl-IPC-Cmd 
#初始化openssl 编译环境, 安装目录为/usr/local/openssl, 可以根据个人需要调整
RUN ["./Configure", "--prefix=/usr/local/openssl"]
#初始化和编译必须分开RUN, 合在一起会报错, 具体原因不清楚
RUN make -j8 && \
    make install && \
    ln -sf /usr/local/openssl/lib64 /usr/local/openssl/lib && \
    ln -sf /usr/local/openssl/include/openssl /usr/include/openssl && \
    ln -sf /usr/local/openssl/lib/libssl.so.3 /usr/local/lib64/libssl.so && \
    ln -sf /usr/local/openssl/bin/openssl /usr/bin/openssl && \
    echo /usr/local/openssl/lib >> /etc/ld.so.conf && \
    ldconfig -v

WORKDIR /home/hoshinory/envs/Python-3.11.7
#初始化python编译环境, 必须带上openssl路径, 否则pip install 会报错
RUN ["./configure", "--prefix=/home/hoshinory/envs/python3.11", "--with-openssl=/usr/local/openssl", "--with-openssl-rpath=auto"]
#同样的, 初始化和编译过程必须分开RUN, 否则报错
RUN make clean && \
    make -j8 && \
    make install && \
    ln -sf /home/hoshinory/envs/python3.11/bin/python3.11 /usr/bin/python3 && \
    ln -sf /home/hoshinory/envs/python3.11/bin/pip3.11 /usr/bin/pip3 && \
    ln -sf /home/hoshinory/envs/python3.11/bin/python3-config /usr/bin/python3-config

#安装python对应的package, 主要是torch
RUN pip3 install --upgrade pip -i http://pypi.douban.com/simple --trusted-host pypi.douban.com && \
    pip3 install tensorflow==2.15 -i http://pypi.douban.com/simple --trusted-host pypi.douban.com && \
    pip3 install /usr/local/TensorRT-8.6.1.6/python/tensorrt-8.6.1-cp311-none-linux_x86_64.whl && \
    pip3 install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121 && \
    pip3 install torchserve -i http://pypi.douban.com/simple --trusted-host pypi.douban.com && \
    pip3 install torch-model-archiver -i http://pypi.douban.com/simple --trusted-host pypi.douban.com && \
    pip3 install torch-workflow-archiver -i http://pypi.douban.com/simple --trusted-host pypi.douban.com && \
    pip3 install numpy -i http://pypi.douban.com/simple --trusted-host pypi.douban.com && \
    pip3 install pandas -i http://pypi.douban.com/simple --trusted-host pypi.douban.com && \
    pip3 install scikit-learn -i http://pypi.douban.com/simple --trusted-host pypi.douban.com && \
    pip3 install gensim -i http://pypi.douban.com/simple --trusted-host pypi.douban.com && \
    pip3 install nvgpu -i http://pypi.douban.com/simple --trusted-host pypi.douban.com && \
    pip3 install PyYAML -i http://pypi.douban.com/simple --trusted-host pypi.douban.com && \
    ln -sf /home/hoshinory/envs/python3.11/bin/torchserve /usr/bin/torchserve && \
    ln -sf /home/hoshinory/envs/python3.11/bin/torchrun /usr/bin/torchrun && \
    ln -sf /home/hoshinory/envs/python3.11/bin/torch-model-archiver /usr/bin/torch-model-archiver && \
    ln -sf /home/hoshinory/envs/python3.11/bin/torch-workflow-archiver /usr/bin/torch-workflow-archiver

#设置这个, 主要是因为torchserve 运行时中文解码问题, 设置不对, 中文会全部变成问号
ENV LC_ALL="en_US.utf8" \
    LANG="en_US.utf8"

SystemTools > Docker
#Docker #Python #Conda
Docker Practice
https://hoshinory.com/20240227/SystemTools/Docker/DockerExamples/
Author
hoshinory
Posted on
February 27, 2024
Licensed under