quick guide for dl4cv book

quick guide for dl4cv book

Quick Guide

quick guide to configure and install deep learning environments on server for dl4cv book

prepare

tools

  • MobaXterm (for windows)
  • ssh + vscode

for windows:
drop files to MobaXterm to upload to server
use zip format

commands

view disk

du -d 1 -h
df -h

view files and count

wc -l data.csv

# count how many folders
ls -lR | grep '^d' | wc -l
17

# count how many jpg files
ls -lR | grep '.jpg' | wc -l
1360

# view 10 images 
ls train | head
ls test | head

link datasets

# link 
ln -s srt dest
ln -s dl4cv/datasets/ datasets

tmux for background tasks

tmux new -s notebook
tmux ls 
tmux attach -t notebook
tmux detach

wget download

# wget 
# continue donwload
wget -c url 

# background donwload for large file
wget -b -c url
tail -f wget-log

gpu and cpu usage

watch -n 1 nvidia-smi
top 

tips about training large model

terminal 1:

tmux new -s train
conda activate keras

time python train_alexnet.py

terminal 2:

tmux detach

tmux attach -t train

and then close vscode, otherwise bash training process will exit when we close vscode.

cuda driver and toolkits

see cuda-toolkit for cuda driver version

cudatookit version depends on cuda driver version.

check cuda driver version

> cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  418.43  Tue Feb 19 01:12:11 CST 2019
GCC version:  gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.11) 

>nvidia-smi
Wed Aug 21 17:15:12 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.43       Driver Version: 418.43       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+

> nvidia-smi -L
GPU 0: Quadro RTX 8000 (UUID: GPU-acb01c1b-776d-cafb-ea35-430b3580d123)
GPU 1: Quadro RTX 8000 (UUID: GPU-df7f0fb8-1541-c9ce-e0f8-e92bccabf0ef)
GPU 2: Quadro RTX 8000 (UUID: GPU-67024023-20fd-a522-dcda-261063332731)
GPU 3: Quadro RTX 8000 (UUID: GPU-7f9d6a27-01ec-4ae5-0370-f0c356327913)

install conda

./Anaconda3-2019.03-Linux-x86_64.sh 
[yes]
[yes]

config channels

conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/msys2/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/menpo/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/

conda config --set show_channel_urls yes

install libraries

conclusions:

  • py37/keras: conda install -y tensorflow-gpu keras
  • py37/torch: conda install -y pytorch torchvision
  • py36/mxnet: conda install -y mxnet

common libraries

conda install -y scikit-learn pandas matplotlib pillow opencv
pip install imutils progressbar pydot

pip install imutils to avoid downgrade for tensorflow-gpu

py37

cudatoolkit               10.0.130                  0    
cudnn                     7.6.0                cuda10.0_0    
tensorflow-gpu            1.13.1 

py36

cudatoolkit        anaconda/pkgs/main/linux-64::cudatoolkit-10.1.168-0
cudnn              anaconda/pkgs/main/linux-64::cudnn-7.6.0-cuda10.1_0
tensorboard        anaconda/pkgs/main/linux-64::tensorboard-1.14.0-py36hf484d3e_0
tensorflow         anaconda/pkgs/main/linux-64::tensorflow-1.14.0-gpu_py36h3fb9ad6_0
tensorflow-base    anaconda/pkgs/main/linux-64::tensorflow-base-1.14.0-gpu_py36he45bfe2_0
tensorflow-estima~ anaconda/cloud/conda-forge/linux-64::tensorflow-estimator-1.14.0-py36h5ca1d4c_0
tensorflow-gpu     anaconda/pkgs/main/linux-64::tensorflow-gpu-1.14.0-h0d30ee6_0

imutils only support 36 and 37.
mxnet only support 35 and 36.

details

# remove py35
conda remove -n py35 --all

conda info --envs

conda create -n py37 python==3.7
conda activate py37

# common libraries
conda install -y scikit-learn pandas pillow opencv
pip install imutils

# imutils
conda search imutils  
# py36 and py37

# Name                       Version           Build  Channel             
imutils                        0.5.2          py27_0  anaconda/cloud/conda-forge
imutils                        0.5.2          py36_0  anaconda/cloud/conda-forge
imutils                        0.5.2          py37_0  anaconda/cloud/conda-forge

# tensorflow-gpu and keras
conda install -y tensorflow-gpu keras

# install pytorch
conda install -y pytorch torchvision

# install mxnet
# method 1: pip
pip search mxnet
mxnet-cu80[mkl]/mxnet-cu90[mkl]/mxnet-cu91[mkl]/mxnet-cu92[mkl]/mxnet-cu100[mkl]/mxnet-cu101[mkl]

# method 2: conda
conda install mxnet
# py35 and py36

TensorFlow Object Detection API

home page: home page

download tensorflow models and rename models-master to tfmodels

vim ~/.bashrc

export PYTHONPATH=/home/kezunlin/dl4cv:/data_1/kezunlin/tfmodels/research:$PYTHONPATH

source ~/.bashrc

jupyter notebook

conda activate py37
conda install -y jupyter 

install kernels

python -m ipykernel install --user --name=py37
Installed kernelspec py37 in /home/kezunlin/.local/share/jupyter/kernels/py37

config for server

python -c "import IPython;print(IPython.lib.passwd())"
Enter password: 
Verify password: 
sha1:ef2fb2aacff2:4ea2998699638e58d10d594664bd87f9c3381c04

jupyter notebook --generate-config
Writing default config to: /home/kezunlin/.jupyter/jupyter_notebook_config.py

vim .jupyter/jupyter_notebook_config.py

c.NotebookApp.ip = '*'  
c.NotebookApp.password = u'sha1:xxx:xxx' 
c.NotebookApp.open_browser = False 
c.NotebookApp.port = 8888 
c.NotebookApp.enable_mathjax = True

run jupyter on background

tmux new -s notebook
jupyter notebook
# ctlr+b+d exit session and DO NOT close session
# ctlr+d exit session and close session

access web and input password

test

py37

import cv2
cv2.__version
import tensorflow as tf
import keras 
import torch
import torchvision

cat .keras/keras.json

{
    "epsilon": 1e-07,
    "floatx": "float32",
    "backend": "tensorflow",
    "image_data_format": "channels_last"
}

py36

import mxnet

train demo

export

# use CPU only
export CUDA_VISIBLE_DEVICES=""

# use gpu 0 1
export CUDA_VISIBLE_DEVICES="0,1"

code

import os
os.environ['CUDA_VISIBLE_DEVICES'] = "0,1"

start train

python train.py

./keras folder

view keras models and datasets

ls .keras/
datasets  keras.json  models

models saved to /home/kezunlin/.keras/models/
datasets saved to /home/kezunlin/.keras/datasets/

models lists

xxx_kernels_notop.h5 for include_top = False
xxx_kernels.h5 for include_top = True

Datasets

mnist

cifar10

to skip download

wget http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
mv ~/Download/cifar-10-python.tar.gz ~/.keras/datasets/cifar-10-batches-py.tar.gz

to load data

(x_train, y_train), (x_test, y_test) = cifar10.load_data()

flowers-17

animals

panda images are WRONG !!!

counts

ls -lR animals/cat | grep ".jpg" | wc -l
1000
ls -lR animals/dog | grep ".jpg" | wc -l
1000
ls -lR animals/panda | grep ".jpg" | wc -l
1000

kaggle cats vs dogs

solutions

caltech101

download background

wget -b -c http://www.vision.caltech.edu/Image_Datasets/Caltech101/101_ObjectCategories.tar.gz

Kaggle API

install and config

see kaggle-api

conda activate keras
conda install kaggle

# download kaggle.json
mv kaggle.json ~/.kaggle/kaggle.json
chmod 600 ~/.kaggle/kaggle.json

cat kaggle.json
{"username":"xxx","key":"yyy"}

or by export

export KAGGLE_USERNAME=xxx
export KAGGLE_KEY=yyy

tips

  1. go to account and select ‘Create API Token’ and keras.json will be downloaded.
  2. Ensure kaggle.json is in the location ~/.kaggle/kaggle.jsonto use the API.

check version

kaggle --version
Kaggle API 1.5.5

commands overview

commands

kaggle competitions {list, files, download, submit, submissions, leaderboard}
kaggle datasets {list, files, download, create, version, init}
kaggle kernels {list, init, push, pull, output, status}
kaggle config {view, set, unset}

download datasets

kaggle competitions download -c dogs-vs-cats

show leaderboard

kaggle competitions leaderboard dogs-vs-cats --show
teamId  teamName                           submissionDate       score    
------  ---------------------------------  -------------------  -------  
71046  Pierre Sermanet                    2014-02-01 21:43:19  0.98533  
66623  Maxim Milakov                      2014-02-01 18:20:58  0.98293  
72059  Owen                               2014-02-01 17:04:40  0.97973  
74563  Paul Covington                     2014-02-01 23:05:20  0.97946  
74298  we've been in KAIST                2014-02-01 21:15:30  0.97840  
71949  orchid                             2014-02-01 23:52:30  0.97733  

set default competition

kaggle config set --name competition --value dogs-vs-cats
- competition is now set to: dogs-vs-cats

submit

kaggle c submissions
- Using competition: dogs-vs-cats
- No submissions found

kaggle c submit -f ./submission.csv -m "first submit"

competition has already ended, so can not submit.

Reference

History

  • 20190821: created.