Kezunlin's Blog

Live and Learn

0%

Posted on Edited on In

Guide

wechat

1
2
3
4
5
6
7
wget -O- https://deepin-wine.i-m.dev/setup.sh | sh

sudo apt-get install deepin.com.wechat
#版本还是比较新的，这下比网页版微信方便多了。（发帖时Windows上的微信最新是2.8.0版）

#安装QQ只需要一条命令：
sudo apt-get install deepin.com.qq.im

screen recorder

obs

see obs

1
2
3
4
5
sudo apt install ffmpeg

sudo add-apt-repository ppa:obsproject/obs-studio
sudo apt update
sudo apt install obs-studio

Reference

History

  • 2020/06/05: created.

Posted on Edited on In

Guide

prepare ubuntu 18.04

主机支持8个口才能使用GeForce RTX 2080 Ti(11GB显存)
Ubuntu 18.04(刻录Ubuntu 18.04系统并通过U盘安装.)
Mem: 64GB
CPU: 2 core 10
GPU: RTX 2080 Ti

download

  • Anaconda3-2019.03-Linux-x86_64.sh

ssh

1
2
3
4
sudo apt-get -y install openssh-server
sudo ap-get -y install net-tools
sudo ifconfig
# 192.168.0.114

install packages

1
2
3
4
5
6
sudo apt -y install git vim tree build-essential cmake-gui
sudo apt-get -y install libleveldb-dev liblmdb-dev libsnappy-dev libhdf5-serial-dev 
sudo apt-get -y install libopenblas-dev liblapack-dev libatlas-base-dev
apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libhdf5-serial-dev protobuf-compiler
apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev

Nvidia driver

Disable the Nouveau drivers

To install the Display Driver, the Nouveau drivers must first be disabled. 

1
2
3
4
5
6
7
8
9
10
11
12
13
# nouveau is loaded
lsmod | grep nouveau

vim /etc/modprobe.d/blacklist-nouveau.conf 
blacklist nouveau
options nouveau modeset=0

#Regenerate the kernel initramfs:
sudo update-initramfs -u
sudo reboot 

# reboot and verify that the Nouveau drivers are not loaded.
lsmod | grep nouveau

install driver by CUDA_linux.run(not)

The Runfile installation installs the NVIDIA Driver, the CUDA Toolkit, and CUDA Samples, via an interactive ncurses-based interface.

1
2
3
4
5
6
7
sudo apt-get purge nvidia-*

sudo ./cuda_10.1.243_418.87.00_linux.run --help
sudo ./cuda_10.1.243_418.87.00_linux.run --driver

# test 
nvidia-smi

/var/log/nvidia-installer.log
/var/log/cuda-installer.log

ERROR: The Nouveau kernel driver is currently in use by your system. This driver is incompatible with the NVIDIA driver, and must be disabled before proceeding.

install driver by PPA(推荐)

see ubuntu_nvidia_ppa

1
2
3
4
5
6
7
8
9
sudo add-apt-repository ppa:graphics-drivers/ppa
sudp apt-get update

sudo apt-cache search nvidia-driver-*
# nvidia-driver-418
# nvidia-driver-440

sudo apt-get -y install nvidia-driver-440
sudo reboot

check dirver

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
nvidia-smi
Thu Jun  4 17:12:59 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.82       Driver Version: 440.82       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:17:00.0 Off |                  N/A |
| 30%   42C    P8    19W / 250W |      1MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:65:00.0  On |                  N/A |
| 31%   44C    P8    10W / 250W |     80MiB / 11018MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    1      1309      G   /usr/lib/xorg/Xorg                            21MiB |
|    1      1399      G   /usr/bin/gnome-shell                          56MiB |
+-----------------------------------------------------------------------------+

cuda-toolkit(9.1)

1
2
3
4
5
6
7
8
sudo apt-get install nvidia-cuda-toolkit

nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Nov__3_21:07:56_CDT_2017
Cuda compilation tools, release 9.1, V9.1.85

conda

1
2
3
./Anaconda3-2019.03-Linux-x86_64.sh 
[yes]
[yes]

install torch

1
2
3
4
conda create -n py37 python==3.7
conda activate py37

conda install -y pytorch torchvision

opencv

1
2
3
4
5
6
7
8
9
pip install opencv-contrib-python

python
Python 3.7.0 (default, Oct  9 2018, 10:31:47) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.__version__
'4.2.0'

qt

1
conda install pyqt

other libraries

1

Reference

History

  • 2020/06/04: created.

Posted on Edited on In

WLS for windows

  • Enable WSL for Windows
1
2
3
4
5
6
# start powershell as admin
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

# or by 控制面板的"启动或关闭Windows功能"

#重启电脑
  • Microsoft Store - Search Ubuntu16.04/Ubuntu18.04 and Install
  • CMD+R \wsl$
  • VS Code + WSL Bash (CMD,Powershell)

windows 10 配置默认Linux发行版本

1
2
3
4
5
6
7
8
cmd> wsl
cmd> wslconfig /l
Ubuntu-16.04(默认)
Ubuntu-18.04

cmd> wslconfig /s Ubuntu-18.04
Ubuntu-18.04(默认)
Ubuntu-16.04

卸载WSL Ubuntu 16.04

1
2
3
4
5
# Windows Search ---- Ubuntu 16.04 LTS --- 菜单选择卸载

# check if uninstall successfully
cmd> wslconfig /s Ubuntu-18.04
Ubuntu-18.04(默认)

Reference

History

  • 2020/3/20: created.

Posted on Edited on In

Guide

atexit

1
2
3
4
5
6
7
8
def exit_handler():
    print("exit')

import atexit
atexit.register(exit_handler)

if __name__ == '__main__':
    print("OK)

ctrl+c/ ctrl+z

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import signal
from sys import exit

def handler(signal_received, frame):
    # Handle any cleanup here
    print('SIGINT or CTRL-C detected. Exiting gracefully')
    
    exit(0)

if __name__ == '__main__':
    # Tell Python to run the handler() function when SIGINT is recieved
    signal.signal(signal.SIGINT, self.ctrl_exit_handler) # ctlr + c
    signal.signal(signal.SIGTSTP, self.ctrl_exit_handler) # ctlr + z

    print('Running. Press CTRL-C to exit.')
    while True:
        # Do nothing and hog CPU forever until SIGINT received.
        pass

Reference

History

  • 2020/3/20: created.

Posted on Edited on In

Guide

Ubuntu 16.04

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
deb http://mirrors.aliyun.com/ubuntu/ xenial main
deb-src http://mirrors.aliyun.com/ubuntu/ xenial main

deb http://mirrors.aliyun.com/ubuntu/ xenial-updates main
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates main

deb http://mirrors.aliyun.com/ubuntu/ xenial universe
deb-src http://mirrors.aliyun.com/ubuntu/ xenial universe

deb http://mirrors.aliyun.com/ubuntu/ xenial-updates universe
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates universe

deb http://mirrors.aliyun.com/ubuntu/ xenial-security main
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security main

deb http://mirrors.aliyun.com/ubuntu/ xenial-security universe
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security universe

Ubuntu 18.04

1
2
3
4
5
6
7
8
9
10
11
12
13
14
deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse

Reference

History

  • 2020/3/20: created.

Posted on Edited on In

pyppetter

1
2
3
4
# python3 -c 'import pyppeteer; pyppeteer.chromium_downloader.download_chromium()'
        
# [W:pyppeteer.chromium_downloader] chromium extracted to: /home/ubuntu/.local/share/pyppeteer/local-chromium/575458

demo 1

1
2
3
4
5
6
7
import requests
from pyquery import PyQuery as pq
 
url = 'http://quotes.toscrape.com/js/'
response = requests.get(url)
doc = pq(response.text)
print('Quotes:', doc('.quote').length)

0

demo 2

1
2
3
4
5
6
7
8
9
10
11
12
13
import asyncio
from pyppeteer import launch
from pyquery import PyQuery as pq
 
async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('http://quotes.toscrape.com/js/')
    doc = pq(await page.content())
    print('Quotes:', doc('.quote').length)
    await browser.close()
 
asyncio.get_event_loop().run_until_complete(main())

10

demo 3

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import asyncio
from pyppeteer import launch
 
async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('http://quotes.toscrape.com/js/')
    await page.screenshot(path='example.png')
    await page.pdf(path='example.pdf')
    dimensions = await page.evaluate('''() => {
        return {
            width: document.documentElement.clientWidth,
            height: document.documentElement.clientHeight,
            deviceScaleFactor: window.devicePixelRatio,
        }
    }''')
 
    print(dimensions)
    # >>> {'width': 800, 'height': 600, 'deviceScaleFactor': 1}
    await browser.close()
 
asyncio.get_event_loop().run_until_complete(main())

demo 4

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
#!/usr/bin/python
# -*- coding: UTF-8 -*-

import asyncio
from pyppeteer import launch
 
async def main():
    # https://github.com/miyakogi/pyppeteer/issues/171
    browser = await launch({
            'args': ['--no-sandbox'],
        }
    )
    page = await browser.newPage()
    url="http://127.0.0.1:5000/tool?1784076_1_1_1_1_1"
    await page.goto(url)
    #await page.waitFor(3000) # wait for 3 seconds

    #===================================================
    # JS逻辑, 如果echart图像render finished,则在html页面中动态添加echartReadyDiv
    # 表示图像ready，可以download/send to server.
    await page.waitForSelector('#echartReadyDiv')
    #===================================================

    # 页面渲染完毕后，开始截图
    # 如果没有加载完毕就生产image了，内容不完整
    await page.screenshot(path='example.png') 
    #await page.pdf(path='example.pdf')
    dimensions = await page.evaluate('''() => {
        return {
            width: document.documentElement.clientWidth,
            height: document.documentElement.clientHeight,
            deviceScaleFactor: window.devicePixelRatio,
        }
    }''')
 
    print(dimensions)
    # >>> {'width': 800, 'height': 600, 'deviceScaleFactor': 1}
    await page.close()
    await browser.close()
 
asyncio.get_event_loop().run_until_complete(main())

Page APIs:

  • page.waitFor(3000) # wait for 3 seconds
  • page.waitForSelector(‘#echartReadyDiv’) # wait for selector
  • page.page.waitForXPath(‘//*[@id=”echartReadyDiv”]’) # wait for xpath

Reference

History

  • 2020/1/17: created.

Posted on Edited on In

crontab

syntax

sudo crontab -e

1
2
# every 5 minutes
*/5 * * * * /home/user/git/repo/python-demo/run.sh
  * * * * * command to be executed
  - - - - -
  | | | | |
  | | | | ----- Day of week (0 - 7) (Sunday=0 or 7)
  | | | ------- Month (1 - 12)
  | | --------- Day of month (1 - 31)
  | ----------- Hour (0 - 23)
  ------------- Minute (0 - 59)

crontab FAQs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
# (1) how to edit crontab file ?
sudo crontab -e 
sudo crontab -l 

# (2) how to enable crontab log 
sudo vim /etc/rsyslog.d/50-default.conf
cron.*              /var/log/cron.log

# restart rsyslog
sudo service rsyslog restart
# now we can see 
tail -f /var/log/cron.log


# (3) how to run a crontab job and see ENVS
sudo crontab -e 

* * * * * env > /home/kezunlin/env.log  
# 由root用户生成
# -rw-r--r-- 1 root root 411 3月  17 17:37 /home/kezunlin/env.log

#结果如下
LC_TIME=zh_CN.UTF-8
#MYFB_ROOT=/home/kezunlin/git/repo/myproject/
#HOME=/root
#PYTHON_BIN=/home/kezunlin/anaconda3/bin/python
LC_MONETARY=zh_CN.UTF-8
LOGNAME=root
#PATH=/usr/bin:/bin
LC_ADDRESS=zh_CN.UTF-8
LANG=en_US.UTF-8
LC_TELEPHONE=zh_CN.UTF-8
LC_NAME=zh_CN.UTF-8
SHELL=/bin/sh
LC_MEASUREMENT=zh_CN.UTF-8
LC_IDENTIFICATION=zh_CN.UTF-8
PWD=/root
#PYTHONPATH=$PYHONPATH:$MYFB_ROOT../
LC_NUMERIC=zh_CN.UTF-8
LC_PAPER=zh_CN.UTF-8

# 可以看到,系统执行crontab job则只会加载默认的ENVS+ /etc/environment中的ENVS，不会设置任何用户ENVS
# 如果scripts中使用了来自.bashrc, .profile中的ENVs，肯定会导致导致scripts执行失败
# **方案**
# 将用户自定义ENVS放置在/etc/environment


# 其他示例
# crontab 中最好使用绝对路径 (不能使用~等,  可以使用$HOME,  source需要用.代替)
* * * * * sleep 5s && echo "yo"
* * * * * env >> $HOME/env.log 
* * * * * export LC_ALL=nb_NO.UTF-8; sleep 5s && echo "yo"
*/5 * * * * /home/ubuntu/scripts/run5.sh


# (5) 既然crontab可以加载/etc/environment中的ENVS，那么可以在crontab中使用来自/etc/environment中的ENVS，比如
*/1 * * * * $MYFB_ROOT/scripts/run5.sh  #OK 可以替换下面的路径
*/1 * * * * /home/ubuntu/project/scripts/run5.sh


# (6) crontab和手动运行scripts的区别
# A: 手动运行,开启terminal,除了加载系统/etc/environment中的ENVS,
# 还会加载当前user的.bashrc .profile文件,设置好ENVS,可以在scripts中使用
# B: 然而crontab则不会加载用户ENVS
# ***所以crontab执行的scripts中最好使用绝对路径，不要使用用户ENVS，否则执行失败***
# 或者将用户ENVS写入到/etc/environment中

crontab 终极方案

ENVS

vim /etc/environment

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# vim /etc/environment （可以被crontab加载,.bashrc则不行) 
# ~/git会被crontab替换为/root/git/repo/myproject，所以使用/home/kezunlin代替

# (1) ubuntu 16.04
MYFB_ROOT=/home/kezunlin/git/repo/myproject/
PYTHONPATH=$PYHONPATH:$MYFB_ROOT../
PYTHON_BIN=/home/kezunlin/anaconda3/bin/python

# (2) WSL   
MYFB_ROOT=/home/ubuntu/git/myproject/
PYTHONPATH=$PYHONPATH:$MYFB_ROOT../
PYTHON_BIN=/usr/bin/python3

# (3) cloud server 
MYFB_ROOT=/home/ubuntu/git/myproject/
PYTHONPATH=$PYHONPATH:$MYFB_ROOT../
PYTHON_BIN=/usr/bin/python3

crontab -e

1
2
3
4
5
6
# scripts的路径可以使用来自/etc/environment的ENVS
# 可以统一各个不同平台的写法
# for ubuntu 16.04 / WSL / cloud server
* * * * * env > $MYFB_ROOT/scripts/env.log
*/5 * * * * $MYFB_ROOT/scripts/run5.sh  
*/10 * * * * $MYFB_ROOT/scripts/run10.sh

Scripts

scripts/run5.sh

1
2
3
4
5
6
#!/bin/sh
. /etc/environment
# MYFB_ROOT  PYTHONPATH  PYTHON_BIN
#env > $MYFB_ROOT/env_test.log
cd $MYFB_ROOT  # 代码中使用了.flag.txt必须cd到此目录
$PYTHON_BIN run5.py >> log/log5.txt

scripts/run10.sh

1
2
3
4
5
6
#!/bin/sh
. /etc/environment
##env > $MYFB_ROOT/env_test.log
# MYFB_ROOT  PYTHONPATH  PYTHON_BIN
cd $MYFB_ROOT  # 代码中使用了.flag.txt必须cd到此目录
$PYTHON_BIN run10.py >> log/log10.txt

Reference

History

  • 2020/1/17: created.

Posted on Edited on In

Guide

error

when r = requests.get(url), error occur:

requests.exceptions.ProxyError: HTTPConnectionPool(host=’127.0.0.1’, port=8888):
Max retries exceeded with url: http://www.baidu.com/ (Caused by ProxyError(‘Cannot
connect to proxy.’, NewConnectionError(‘<urllib3.connection.HTTPConnection object
at 0x7f5611990080>: Failed to establish a new connection: [Errno 111] Connection
refused’)))

solutions

1
2
3
4
5
6
$ env | grep -i proxy 

export http_proxy='127.0.0.1:8888'
export https_proxy='127.0.0.1:8888'
export ftp_proxy=''
export socks_proxy=''

edit ~/.bashrc 

1
2
3
4
export http_proxy=''
export https_proxy=''
export ftp_proxy=''
export socks_proxy=''

and run source ~.bashrc

Now OK.

or by code 

1
2
proxies = { "http": None, "https": None}
requests.get("http://baidu.com", proxies=proxies)

Reference

History

  • 2019/11/08: created.