Kezunlin's Blog

Guide

Linux/Unix Address Space

BSS: Block Started by Symbol

program memory area

program memory area:

• code: executable instructions （和 const 字符串常量？？？）
• initialized-data: static/global 初始化不为０的
• zero-initialized data(BSS): static/global 初始化为０的
• heap： malloc/new 动态分配，大小不固定
• stack：　非static的local variable; functin parameter; return value

bss它不占用程序文件的大小，但是占用程序运行时的内存空间。
zero-initialized data(BSS)区域的变量不占用exe的磁盘空间，其内容由操作系统初始化（清零），而initialized-data需要占用exe的磁盘空间，其内容由程序初始化
同一个program的N个不同process,在内存中共享同一个code/text segment，同时都有自己的initialized-data,zero-initialized data(BSS)对应的data　segment
字符串常量

address space segment

address space segment

• text
• data
• stack

1. text,data,stack占用３块不同的地址空间。
2. initialized-data，zero-initialized data(BSS)，heap占用同一块连续的物理内存空间;
3. heap地址空间从小到大增长；stack的地址空间从大往小减小;

Heap，BSS，Data这三个段在物理内存中是连续存放的，可以这么理解：这三个是一体的。Text、Stack是独立存放的

executable file section

executable file section

• .text: 在进程最初被加载到内存中开始，该节的大小就被固定，只读readonly。
• .data: 该节的大小在运行时固定的
• .bss: 该节的大小在运行时固定的
• heap: 从内存的低地址向高地址增长
• stack: 从内存的高地址向低地址增长
• .rodata: readonly data section

eg

size a.out 
   text	   data	    bss	    dec	    hex	filename
   1551	    600	     40	   2191	    88f	a.out

a.out加载到内存中的大小为2191 bytes.

rodata

(1) const修饰的global放在.rodata, const修饰的local例外，没有放入常量区。
(３)对于global/local字符串常量，编译器会去掉重复的常量，让程序的每个字符串常量只有一份，放在.rodata。（数组内容为字符串的除外）

1
2
3
4
5
6
7
8
9
10
11
12

const int num = 100; // .rodata 

char *str = "123456789"; // 123456789 ===>.rodata LC0
char *str1 = "helloworld"; // helloworld===>.rodata LC1

int main()
{
　 const int local_num = 99; // 没有放入.rodata
  char* a = "helloworld";  // helloworld===>.rodata LC1
  char b[10] = "helloworld";// helloworld　没有放入.rodata
  return 1;
}

Example code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

#include <stdio.h>  
  
int add(int x, int y)  
{  
    return x + y;  
}  
  
int aaa,bbb;  // auto zero-intialized(.bss)
int ccc000 = 0; // auto zero-initialized(.bss)
int ccc111 = 1;  // initialized(.data)

char *str1 = "123"; // str1--->.data   123--->.rodata A
char *str2 = "xyz"; //  str2--->.data   xyz--->.rodata B
char str3[] = "abc"; // str3--->.data  
char *str4; // .bss

static int hhh; // .bss
static int iii00 = 0; // .bss
static int iii99 = 99; // .data

const int NUM99 = 99; // .rodata
const int NUM00 = 0; // .rodata
  
int main()  
{  
    static int jjj; // .bss
    static int kkk=1; // .data

    const int local_const00 = 0; // 
    const int local_const99 = 99; // 

    char *str5 = "xyz"; // str5===> stack,  xyz===> .rodata B
    char str6[] = "abc"; // str6--->stack,  abc ===> stack


    int local = 2;  // stack
    int *local_new = new int(5); // heap
    delete local_new;
    return 0;  
}    

output

g++ hello.cpp
nm a.out

0000000000601078 B aaa
000000000060107c B bbb
000000000060106c B __bss_start
0000000000601080 B ccc000
0000000000601048 D ccc111
0000000000601070 b completed.7594
0000000000601038 D __data_start
0000000000601038 W data_start
00000000004005e0 t deregister_tm_clones
0000000000400660 t __do_global_dtors_aux
0000000000600e08 t __do_global_dtors_aux_fini_array_entry
0000000000601040 D __dso_handle
0000000000600e18 d _DYNAMIC
000000000060106c D _edata
00000000006010a0 B _end
00000000004007a4 T _fini
0000000000400680 t frame_dummy
0000000000600e00 t __frame_dummy_init_array_entry
0000000000400910 r __FRAME_END__
0000000000601000 d _GLOBAL_OFFSET_TABLE_
                 w __gmon_start__
00000000004007c4 r __GNU_EH_FRAME_HDR
0000000000400530 T _init
0000000000600e08 t __init_array_end
0000000000600e00 t __init_array_start
00000000004007b0 R _IO_stdin_used
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
0000000000600e10 d __JCR_END__
0000000000600e10 d __JCR_LIST__
                 w _Jv_RegisterClasses
00000000004007a0 T __libc_csu_fini
0000000000400730 T __libc_csu_init
                 U __libc_start_main@@GLIBC_2.2.5
00000000004006ba T main
0000000000400620 t register_tm_clones
                 U __stack_chk_fail@@GLIBC_2.4
00000000004005b0 T _start
0000000000601050 D str1
0000000000601058 D str2
0000000000601060 D str3
0000000000601088 B str4
0000000000601070 D __TMC_END__
00000000004006a6 T _Z3addii
                 U _ZdlPv@@GLIBCXX_3.4
0000000000601090 b _ZL3hhh
0000000000601094 b _ZL5iii00
0000000000601064 d _ZL5iii99
00000000004007c0 r _ZL5NUM00
00000000004007bc r _ZL5NUM99
                 U _Znwm@@GLIBCXX_3.4
0000000000601098 b _ZZ4mainE3jjj
0000000000601068 d _ZZ4mainE3kkk

other commands

1
2

strings a.out
readelf -s a.out

Reference

• Linux段管理，BSS段，data段，.rodata段，text段
• c-tutorials-different-segments-program-have
• Linux/Unix process address space

History

• 20181031: created.

Series

• Part 1: compile darknet on ubuntu 16.04
• Part 2: compile darknet on windows 10
• Part 3: compile caffe-yolov3 on ubuntu 16.04

Guide

Darknet is an open source neural network framework written in C and CUDA. It is fast, easy to install, and supports CPU and GPU computation.

optional dependencies:

• opencv
• gpu (CUDA/CUDNN)

repo intro

see darknet
This repository supports:

• both Windows(VS 2015 (v140)) and Linux (GCC>=4.9)
• CPU and GPU (with CC >= 3.0)
• both OpenCV 2.x.x and OpenCV <= 3.4.0 (3.4.1 and higher isn’t supported)
• both cuDNN v5-v7
• CUDA >= 7.5
• also create SO-library on Linux and DLL-library on Windows

compile

my system requirements (same as caffe on ubuntu 16.04)

• ubuntu 16.04
• GeForce 1060 (6G)
• cuda: 9.2
• cudnn: 7.1.4
• opencv: 3.3.0
• openmp

commands

1
2
3
4

git clone https://github.com/AlexeyAB/darknet.git

cd darknet
vim Makefile 

options:

GPU=1
CUDNN=1
CUDNN_HALF=0
OPENCV=0
AVX=0
OPENMP=1
LIBSO=1

now compile with

1

make -j8

OK. now we have darknet, darknet.so and uselib which use darknet.so

fix error

CUDNN error:

Try to load cfg: ./ped.cfg, weights: ./ped.weights, clear = 0 
   layer   filters  size/strd(dil)      input                output
   0 
 cuDNN status Error in: file: ./src/convolutional_layer.c : () : line: 301 : build time: Sep 24 2019 - 10:28:26 
cuDNN Error: CUDNN_STATUS_BAD_PARAM

fix

Makefile options:

CUDNN=0

and compile again

1

make -j8

see here

pretrained models

dataset

• coco: 80 classes
• voc: 20 classes

model

• yolov3: 0-106 layers

• yolov2: 0-31 layers

• yolov3.cfg (236 MB COCO Yolo v3) - requires 4 GB GPU-RAM: yolov3.weights

• yolov2.cfg (194 MB COCO Yolo v2) - requires 4 GB GPU-RAM: yolov2.weights

demo

yolov2

1

sh ./image_yolov2.sh 

or

1

./darknet detector test ./cfg/coco.data ./cfg/yolov2.cfg ./yolov2.weights data/dog.jpg -i 0 -thresh 0.2

output

 29 conv   1024  3 x 3 / 1    13 x  13 x1280   ->    13 x  13 x1024 3.987 BF
  30 conv    425  1 x 1 / 1    13 x  13 x1024   ->    13 x  13 x 425 0.147 BF
  31 detection
mask_scale: Using default '1.000000'
Total BFLOPS 29.475 
Loading weights from ./yolov2.weights...
 seen 32 
Done!
data/dog.jpg: Predicted in 0.000000 milli-seconds.
dog: 79%
bicycle: 84%
truck: 77%
Not compiled with OpenCV, saving to predictions.png instead

view results

1

eog prediction.png

yolov3

1

sh ./image_yolov3.sh 

or

1

./darknet detector test ./cfg/coco.data ./cfg/yolov3.cfg ./yolov3.weights data/dog.jpg -i 0 -thresh 0.25

output

104 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256 1.595 BF
 105 conv    255  1 x 1 / 1    52 x  52 x 256   ->    52 x  52 x 255 0.353 BF
 106 yolo
Total BFLOPS 65.864 
Loading weights from ./yolov3.weights...
 seen 64 
Done!
data/dog.jpg: Predicted in 0.000000 milli-seconds.
bicycle: 99%
dog: 100%
truck: 93%
Not compiled with OpenCV, saving to predictions.png instead

view results

1

eog prediction.png

uselib

1

./uselib data/dog.jpg

output

100 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256 1.595 BF
 101 conv    128  1 x 1 / 1    52 x  52 x 256   ->    52 x  52 x 128 0.177 BF
 102 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256 1.595 BF
 103 conv    128  1 x 1 / 1    52 x  52 x 256   ->    52 x  52 x 128 0.177 BF
 104 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256 1.595 BF
 105 conv    255  1 x 1 / 1    52 x  52 x 256   ->    52 x  52 x 255 0.353 BF
 106 yolo
Total BFLOPS 65.864 
Loading weights from yolov3.weights...
 seen 64 
Done!
object names loaded 
input image or video filename: dog - obj_id = 16,  x = 123, y = 223, w = 196, h = 319, prob = 0.998
truck - obj_id = 7,  x = 474, y = 87, w = 216, h = 78, prob = 0.931
bicycle - obj_id = 1,  x = 117, y = 124, w = 451, h = 308, prob = 0.99

darknet.py

1

python darknet.py

Reference

• pjreddie darknet yolo
• darknet install
• darknet for linux and windows

History

• 20181031: created.

Guide

from cuda-gpus

Tesla

Quadro

NVS

GeForce

TEGRA/Jetson

jetson products

• Jetson TX1 (fp32,fp16)
• Jetson TX2 (fp32,fp16)
• Jetson AGX Xavier (fp32,fp16,int8,dla)
• Jetson Nano (Jetbot)

hardware precision support matrix

see tensorrt-support-matrix

Xaiver: 30W,性能和能效分别比前代产品 NVIDIA Jetson TX2 高出 20 倍和 10 倍

JetPack SDK

Jetson Development Pack (JetPack SDK)

JetPack 4.1.1 Developer Preview

JetPack Components

• OS Image: JetPack includes a sample file system derived from Ubuntu.
• Libraries: CUDA/CUDNN/TensorRT/MultiMedia API/VisionWorks and OpenCV
• Developer Tools: CUDA tools/NVIDIA Nsight Systems/NVIDIA Nsight Graphics
• Samples
• Documents

JetPack Installer

JetPack 4.1.1
Now supports Host computer running Ubuntu 18.04 or Ubuntu 16.04.
JetPack installer can now share host computer’s internet connection to Jetson device via USB Type C cable during install.

Jetson DeepStream SDK

Reference

• cuda gpus
• drivers
• Jetson AGX Xavier

History

• 20181029: created.

Iterable && Iterator

• 可迭代对象（iterable），只定义了__iter__方法； 字符串、列表、元组、字典、文件；可以通过iter(iterable)方法获取iterator对象，也可以通过list(iterable) for xxx in iterable间接调用__iter__方法

• 迭代器（iterator）， Iteration Protocol: 定义了__iter__和__next__两个方法，__iter__返回迭代器本身(用于for loop)，__next__方法返回下一个元素，如果没有元素了，抛出StopIteration异常； for python2, use next; for python3, use __next__

  iterator = iter(l) #
  iterator2 = l.__iter__() 
  
  list(l)
  for xxx in l:
  

yrange

例子1：iterable和iterator是同一个对象。

y = iterable()
list(y) 
list(y)
for i in y

只有第一次输出所有值；后续输出未空。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

class yrange:
    def __init__(self, n):
        self.i = 0
        self.n = n

    def __iter__(self):
        print("__iter__1")
        return self

    def next(self):
        if self.i < self.n:
            i = self.i
            self.i += 1
            return i
        else:
            raise StopIteration()

当使用list(iterable)的时候，会调用iterable的__iter__方法返回iterator

output

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

y = yrange(3)
0
y = yrange(3)
1
y = yrange(3)
2
y = yrange(3)
StopIteration 

list(yrange(3))
__iter__1
[0, 1, 2]

sum(yrange(3))
__iter__1
3


y = yrange(3)

list(y)
__iter__1
    [0, 1, 2]

list(y)
__iter__1
    []
    
y = yrange(3)

list(y.__iter__())
__iter__1
__iter__1
    [0, 1, 2]

list(y.__iter__())
__iter__1
__iter__1
    []

zrange

例子2：iterable和iterator是不同对象。

z = iterable()
list(z)
list(z)
for i in z

调用N次，都会输出所有值。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

class zrange:
    def __init__(self, n):
        self.n = n

    def __iter__(self):
        print("__iter__1")
        return zrange_iter(self.n)

class zrange_iter:
    def __init__(self, n):
        self.i = 0
        self.n = n

    def __iter__(self):
        print("__iter__2")
        # Iterators are iterables too.
        # Adding this functions to make them so.
        return self

    def next(self):
        if self.i < self.n:
            i = self.i
            self.i += 1
            return i
        else:
            raise StopIteration()

output

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

z = zrange(3)
list(z)
__iter__1
[0, 1, 2]

list(z)
__iter__1
[0, 1, 2]

z = zrange(3)
list(z.__iter__())
__iter__1
__iter__2
[0, 1, 2]

list(z.__iter__())
__iter__1
__iter__2
[0, 1, 2]

Generator

Generator functions are ordinary functions defined using yield instead of return. When called, a generator function returns a generator object, which is a kind of iterator - it has a next() method. When you call next(), the next value yielded by the generator function is returned.

use the word “generator” to mean the genearted object and “generator function” to mean the function that generates it.

generator也是一个iterator。 Generator functions简化了iterator的创建。只需要yield就可以代替实现iterator的__iter__和next方法。

1
2
3
4
5

def yrange(n):
    i = 0
    while i < n:
        yield i
        i += 1

output

1
2
3
4
5
6
7
8
9
10
11
12
13

>>> y = yrange(3)
>>> y
<generator object yrange at 0x401f30>
>>> y.next()
0
>>> y.next()
1
>>> y.next()
2
>>> y.next()
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
StopIteration

How to work

When a generator function is called, it returns a generator object without even beginning execution of the function.
When next method is called for the first time, the function starts executing until it reaches yield statement. The yielded value is returned by the next call.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

>>> def foo():
...     print "begin"
...     for i in range(3):
...         print "before yield", i
...         yield i
...         print "after yield", i
...     print "end"
...
>>> f = foo() # 不执行任何语句，返回generator object
>>> f.next() # 执行语句直到yield,返回结果
begin
before yield 0
0
>>> f.next() # 从上一次yield语句的下一句开始执行语句直到再次到达yield,返回结果
after yield 0
before yield 1
1
>>> f.next() # 从上一次yield语句的下一句开始执行语句直到再次到达yield,返回结果
after yield 1
before yield 2
2
>>> f.next() # 从上一次yield语句的下一句开始执行语句，由于没有再次到达yield所以抛出StopIteration异常
after yield 2
end
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
StopIteration
>>>

再看一下例子

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

def integers():
    """Infinite sequence of integers."""
    i = 1
    while True:
        yield i
        i = i + 1

def squares():
    for i in integers():
        yield i * i

def take(n, seq):
    """Returns first n values from the given sequence."""
    seq = iter(seq)
    result = []
    try:
        for i in range(n):
            result.append(seq.next())
    except StopIteration:
        pass
    return result

print take(5, squares()) # prints [1, 4, 9, 16, 25]

Generator Expressions

Generator Expressions are generator version of list comprehensions.
They look like list comprehensions, but returns a generator back instead of a list.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

a = [x for x in range(5)]
a
[0, 1, 2, 3, 4]


a = (x*x for x in range(5))
a
<generator object <genexpr> at 0x0000000005232630>

sum(a)
10

sum((x*x for x in range(10)))
#如果只有一个参数，generator expression的()可以省略
sum(x*x for x in range(10))

pyt = ((x, y, z) for z in integers() 
                    for y in xrange(1, z) 
                            for x in range(1, y) 
                                if x*x + y*y == z*z)

pyt
<generator object <genexpr> at 0x0000000005232828>

take(5,pyt)
[(3, 4, 5), (6, 8, 10), (5, 12, 13), (9, 12, 15), (8, 15, 17)]

Example: Reading multiple files

python提供的file对象就是一个iterator对象

1
2
3

f = open('./1.txt')
f.next()
f.next()

使用generator简化代码
old

1
2
3
4
5
6
7
8
9
10

def cat(filenames):
    for f in filenames:
        for line in open(f):
            print line,

def grep(pattern, filenames):
    for f in filenames:
        for line in open(f):
            if pattern in line:
                print line,

new with generator

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

def readfiles(filenames):
    for f in filenames:
        for line in open(f):
            yield line

def grep(pattern, lines):
    return (line for line in lines if pattern in line)

def printlines(lines):
    for line in lines:
        print line,

def main(pattern, filenames):
    lines = readfiles(filenames)
    lines = grep(pattern, lines)
    printlines(lines)

Itertools

1
2
3
4
5
6
7
8
9
10
11
12

import itertools
it1 = iter([1, 2, 3])
it2 = iter([4, 5, 6])
for v in itertools.chain(it1, it2):
    print v

for x, y in itertools.izip(["a", "b", "c"], [1, 2, 3]:
    print x, y

#a 1
#b 2
#c 3

Reference

• iterators
• difference-between-pythons-generators-and-iterators

History

• 20181029: created.

Series

• Part 1: compile protobuf-cpp on ubuntu 16.04
• Part 2: compile protobuf-cpp on windows 10
• Part 3: protobuf usage

Guide

compile

1
2
3
4
5
6

git clone https://github.com/protocolbuffers/protobuf.git     
cd protobuf
git submodule update --init --recursive
# third_party/benchmark third_party/googletest

cd cmake && mkdir build && cd build && cmake-gui ..

with options

CMAKE_INSTALL_PREFIX C:/Program Files/protobuf
protobuf_BUILD_SHARED_LIBS ON

Static Linking vs DLL

Static linking is now the default for the Protocol Buffer libraries. Due to issues with Win32’s use of a separate heap for each DLL, as well as binary compatibility issues between different versions of MSVC’s STL library, it is recommended that you use static linkage only.
However, it is possible to build libprotobuf and libprotoc as DLLs if you really want. To do this, do the following:

• Add an additional flag -Dprotobuf_BUILD_SHARED_LIBS=ON when invoking cmake
• Follow the same steps as described in the above section.
• When compiling your project, make sure to #define PROTOBUF_USE_DLLS.

compile ALL_BUILD with VS 2015 and install to C:/Program Files/protobuf with dynamic libraries.

usage

1
2
3

protoc --cpp_out=. ./point_cloud.proto 
protoc --java_out=./java/ ./proto/helloworld.proto
protoc --go_out=./go/ ./proto/helloworld.proto

CMakeLists.txt

1
2
3
4

find_package(Protobuf REQUIRED)
add_definitions( -DPROTOBUF_USE_DLLS ) 
MESSAGE( [Main] " PROTOBUF_INCLUDE_DIRS = ${PROTOBUF_INCLUDE_DIRS}")   
MESSAGE( [Main] " PROTOBUF_LIBRARIES = ${PROTOBUF_LIBRARIES}")  

Reference

• protobuf on windows

History

• 20181029: created.

Set up

1
2
3
4
5
6

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import numpy as np
import tensorflow as tf

Tensor Values

tensor, rank, shape, value

1
2
3
4
5
6
7
8
9
10
11

1. # a rank 0 tensor; a scalar with shape [],

[1., 2., 3.] # a rank 1 tensor; a vector with shape [3]

[[1., 2., 3.], 
 [4., 5., 6.]] # a rank 2 tensor; a matrix with shape [2, 3]

[
    [[1., 2., 3.]], 
     [[7., 8., 9.]]
] # a rank 3 tensor with shape [2, 1, 3]

[[[1.0, 2.0, 3.0]], [[7.0, 8.0, 9.0]]]

TensorFlow Core Walkthrough

You might think of TensorFlow Core programs as consisting of two discrete sections:

• Building the computational graph (a tf.Graph).
• Running the computational graph (using a tf.Session).

Graph

A computational graph is a series of TensorFlow operations arranged into a graph. The graph is composed of two types of objects.

• tf.Operation (or “ops”): The nodes of the graph. Operations describe calculations that consume and produce tensors.
• tf.Tensor: The edges in the graph. These represent the values that will flow through the graph. Most TensorFlow functions return tf.Tensors.

1
2
3
4
5
6
7
8

# tf.Tensors do not have values

a = tf.constant(3.0, dtype=tf.float32)
b = tf.constant(4.0) # also tf.float32 implicitly
total = a + b
print(a)
print(b)
print(total)

Tensor("Const_4:0", shape=(), dtype=float32)
Tensor("Const_5:0", shape=(), dtype=float32)
Tensor("add_2:0", shape=(), dtype=float32)

TensorBoard

visualizing a computation graph.

1
2

writer = tf.summary.FileWriter('.')
writer.add_graph(tf.get_default_graph()) # events.out.tfevents.{timestamp}.{hostname}

Now, in a new terminal, launch TensorBoard with the following shell command:

tensorboard --logdir .

then access http://localhost:6006/#graphs

Session

To evaluate tensors, instantiate a tf.Session object, informally known as a session. A session encapsulates the state of the TensorFlow runtime, and runs TensorFlow operations. If a tf.Graph is like a .py file, a tf.Session is like the python executable.

1
2

sess = tf.Session()
print(sess.run(total))

7.0

1

print(sess.run({'ab':(a, b), 'total':total}))

{'total': 7.0, 'ab': (3.0, 4.0)}

1
2
3
4
5
6
7

# 第一次，第二次run的vec结果都不一样；第三次先生成vec，然后得到out1和out2
vec = tf.random_uniform(shape=(3,))
out1 = vec + 1
out2 = vec + 2
print(sess.run(vec))
print(sess.run(vec))
print(sess.run((out1, out2)))

[0.3936273  0.7793083  0.46128905]
[0.9422312  0.03195083 0.4434662 ]
(array([1.9042395, 1.5705729, 1.4484392], dtype=float32), array([2.9042397, 2.5705729, 2.4484391], dtype=float32))

Feeding

A placeholder is a promise to provide a value later, like a function argument.

1
2
3
4
5
6

x = tf.placeholder(tf.float32)
y = tf.placeholder(tf.float32)
z = x + y
print(x)
print(y)
print(z)

Tensor("Placeholder_2:0", dtype=float32)
Tensor("Placeholder_3:0", dtype=float32)
Tensor("add_6:0", dtype=float32)

1
2

print(sess.run(z, feed_dict={x: 3, y: 4.5}))
print(sess.run(z, feed_dict={x: [1, 3], y: [2, 4]}))

7.5
[3. 7.]

1
2

# placeholders throw an error if no value is fed to them.
# print(sess.run(z))

Datasets

1
2
3
4
5
6
7
8

my_data = [
    [0, 1,],
    [2, 3,],
    [4, 5,],
    [6, 7,],
]
slices = tf.data.Dataset.from_tensor_slices(my_data)
next_item = slices.make_one_shot_iterator().get_next()

Reaching the end of the data stream causes Dataset to throw an tf.errors.OutOfRangeError. For example, the following code reads the next_item until there is no more data to read:

1
2
3
4
5

while True:
  try:
    print(sess.run(next_item))
  except tf.errors.OutOfRangeError:
    break

[0 1]
[2 3]
[4 5]
[6 7]

If the Dataset depends on stateful operations you may need to initialize the iterator before using it, as shown below:

1
2
3
4
5
6
7
8
9
10
11

r = tf.random_normal([10,3])
dataset = tf.data.Dataset.from_tensor_slices(r)
iterator = dataset.make_initializable_iterator()
next_row = iterator.get_next()

sess.run(iterator.initializer) # initialize iterator
while True:
  try:
    print(sess.run(next_row))
  except tf.errors.OutOfRangeError:
    break

[-0.27240568  0.41916385  0.8312674 ]
[-0.37411043  0.39321673  0.44852588]
[-2.502281   -0.35216704 -0.42525485]
[ 0.16175386  0.4726008  -0.02786499]
[-0.252643    0.47525632  1.3186764 ]
[ 0.2424109  1.3193169 -0.4673948]
[ 0.43207827  0.31257248 -0.609174  ]
[-0.80196303 -1.100821    2.44067   ]
[0.79778594 0.34206432 1.0677754 ]
[ 0.47419003 -0.7259666   0.8751686 ]

Layers

Creating Layers

1
2
3

x = tf.placeholder(tf.float32, shape=[None, 3])
linear_model = tf.layers.Dense(units=1) # Layer
y = linear_model(x)

Initializing Layers

The layer contains variables that must be initialized before they can be used.

1
2

init = tf.global_variables_initializer()
sess.run(init)

Executing Layers

1

print(sess.run(y, {x: [[1, 2, 3],[4, 5, 6]]}))

[[ 4.5850883]
 [10.428741 ]]

Layer Function shortcuts

For each layer class (like tf.layers.Dense) TensorFlow also supplies a shortcut function (like tf.layers.dense).

1
2
3
4
5
6
7

x = tf.placeholder(tf.float32, shape=[None, 3])
y = tf.layers.dense(x, units=1)

init = tf.global_variables_initializer()
sess.run(init)

print(sess.run(y, {x: [[1, 2, 3], [4, 5, 6]]}))

[[2.4798913]
 [6.7798343]]

Feature columns

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

features = {
    'sales' : [[5], [10], [8], [9]],
    'department': ['sports', 'sports', 'gardening', 'gardening']}

department_column = tf.feature_column.categorical_column_with_vocabulary_list(
        'department', ['sports', 'gardening'])
department_column = tf.feature_column.indicator_column(department_column)
print(department_column)

columns = [
    tf.feature_column.numeric_column('sales'),
    department_column
]

inputs = tf.feature_column.input_layer(features, columns)
print(inputs)

_IndicatorColumn(categorical_column=_VocabularyListCategoricalColumn(key='department', vocabulary_list=('sports', 'gardening'), dtype=tf.string, default_value=-1, num_oov_buckets=0))
Tensor("input_layer/concat:0", shape=(4, 3), dtype=float32)

Running the inputs tensor will parse the features into a batch of vectors.

Feature columns can have internal state, like layers, so they often need to be initialized. Categorical columns use tf.contrib.lookup internally and these require a separate initialization op, tf.tables_initializer.

1
2
3
4

var_init = tf.global_variables_initializer()
table_init = tf.tables_initializer()
sess = tf.Session()
sess.run((var_init, table_init))

(None, None)

1

print(sess.run(inputs))

[[ 1.  0.  5.]
 [ 1.  0. 10.]
 [ 0.  1.  8.]
 [ 0.  1.  9.]]

Training

Define the data

1
2

x = tf.constant([[1], [2], [3], [4]], dtype=tf.float32)
y_true = tf.constant([[0], [-1], [-2], [-3]], dtype=tf.float32)

Define the model

1
2
3

linear_model = tf.layers.Dense(units=1)

y_pred = linear_model(x)

1
2
3
4
5
6
7

sess = tf.Session()
init = tf.global_variables_initializer()
sess.run(init)

# The model hasn't yet been trained, so the four "predicted" values aren't very good. 
# Here's what we got; your own output will almost certainly differ:
print(sess.run(y_pred))

[[0.5062338]
 [1.0124676]
 [1.5187014]
 [2.0249352]]

Loss

To optimize a model, you first need to define the loss. We’ll use the mean square error (MSE), a standard loss for regression problems.

1
2
3

loss = tf.losses.mean_squared_error(labels=y_true, predictions=y_pred)

print(sess.run(loss))

10.484383

Training

TensorFlow provides optimizers implementing standard optimization algorithms. These are implemented as sub-classes of tf.train.Optimizer. They incrementally change each variable in order to minimize the loss. The simplest optimization algorithm is gradient descent, implemented by tf.train.GradientDescentOptimizer.

1
2

optimizer = tf.train.GradientDescentOptimizer(0.01)
train = optimizer.minimize(loss) # training operation

1
2
3

for i in range(100):
  _, loss_value = sess.run((train, loss))
  print(loss_value)

10.484383
7.366995
5.2033534
3.7015016
2.6588542
1.9348423
1.4319282
1.0824323
0.83939326
0.67022556
0.5523189
0.46998408
0.41233516
0.37181824
0.34319195
0.32281947
0.30817705
0.29751378
0.28961438
0.2836359
0.27899322
0.2752804
0.27221578
0.26960373
0.26730868
0.2652365

1

print(sess.run(y_pred))

[[-0.66274  ]
 [-1.3211429]
 [-1.9795457]
 [-2.6379485]]

Complete program

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

x = tf.constant([[1], [2], [3], [4]], dtype=tf.float32)
y_true = tf.constant([[0], [-1], [-2], [-3]], dtype=tf.float32)

linear_model = tf.layers.Dense(units=1)

y_pred = linear_model(x)
loss = tf.losses.mean_squared_error(labels=y_true, predictions=y_pred)

optimizer = tf.train.GradientDescentOptimizer(0.01)
train = optimizer.minimize(loss)

init = tf.global_variables_initializer()

sess = tf.Session()
sess.run(init)

for i in range(100):
  _, loss_value = sess.run((train, loss))
  #print(loss_value)

print(sess.run(y_pred))

[[-0.5594496]
 [-1.2710916]
 [-1.9827336]
 [-2.6943758]]

Reference

• intro

History

• 20181026: created.

Guide

file example

读取文件处理传统写法,需要确保文件正常打开之后，即使出现异常，也能够正常关闭。写法不够优雅。

1
2
3
4
5
6
7
8
9
10
11
12
13
14

try:
    file = open("./12.txt")
except:
    print("can not open file")
    exit(1)

try:
    data = file.read()
    # ... do somethings
except:
    print("exception")
finally:
    print("OK. close file")
    file.close()

with语句时用于对try except finally 的优化，让代码更加美观。同时确保即使执行异常，finally中的语句也可以执行。

with-as 重写

1
2

with open("./1.txt") as file:
    data = file.read()

with-as 原理

基本思想是with所求值的对象必须有一个__enter__()方法，一个__exit__()方法。

原理：

1. with后面的语句被求值后返回对象A;
2. 调用对象A的__enter__()方法将返回值赋给as后面的变量B；
3. 执行with后面的代码块;
4. 当with后面的代码块全部被执行完之后（或者由于抛出异常没有正常习执行完毕），最终都会调用A对象的__exit__()方法进行清理工作。

1
2
3
4
5
6
7
8
9
10
11
12
13

class Sample:
    def __enter__(self):
        print "In __enter__()"
        return "Foo"
 
    def __exit__(self, type, value, trace):
        print "In __exit__()"
 
def get_sample():
    return Sample()
 
with get_sample() as s:
    print "sample:", s

output

In __enter__()
sample: Foo
In __exit__()

steps:

1. 执行with后面的语句get_sample()返回对象sample
2. 执行sample对象的__enter__()方法，返回string类型的”Foo”赋值给as后面的s变量
3. 执行代码块，打印变量”sample”的值为 “Foo”
4. 代码块执行完毕，调用sample对象的__exit__()方法进行清理工作

with 处理异常

with除了可以进行清理工作之外，真正强大之处是可以处理异常。Sample类的__exit__方法有三个参数：val, type 和 trace。 这些参数在异常处理中相当有用。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

class Sample:
    def __enter__(self):
        return self
 
    def __exit__(self, type, value, trace):
        print "type:", type
        print "value:", value
        print "trace:", trace
 
    def do_something(self):
        bar = 1/0
        return bar + 10
 
with Sample() as sample:
    sample.do_something()

output

type: <type 'exceptions.ZeroDivisionError'>
value: integer division or modulo by zero
trace: <traceback object at 0x0000000004A5AA08>

ZeroDivisionErrorTraceback (most recent call last)
<ipython-input-9-282d3906c5ac> in <module>()
     13 
     14 with Sample() as sample:
---> 15     sample.do_something()

<ipython-input-9-282d3906c5ac> in do_something(self)
      9 
     10     def do_something(self):
---> 11         bar = 1/0
     12         return bar + 10
     13 

ZeroDivisionError: integer division or modulo by zero

在执行代码块sample.do_something()的时候由于异常抛出，与之关联的type，value和stack trace传给__exit__()方法，因此抛出的ZeroDivisionError异常被打印出来了。

file 对象

python中的file对象已经实现了__enter__()和__exit__()方法

1

file = open("./1.txt")

1

file.__enter__()

<open file './1.txt', mode 'r' at 0x00000000049FF030>

1

file.read(1)

1

file.__exit__()

1

file.read(1)

ValueErrorTraceback (most recent call last)

<ipython-input-19-d0e1662399bb> in <module>()
----> 1 file.read(1)


ValueError: I/O operation on closed file

Reference

• python with as的用法

History

• 20181023: created.

Guide

Superdog soft

2个dog:

• 开发狗：Superdog公司给客户公司(kezunlin)提供的开发狗，包含公司的vendor_code，相当于管理员。不要遗失。
• 超级狗：可以通过软件写入feature_id交付用户。可以有多个超级狗。

如何给超级狗写入特征id?

提示：写入特征之前，必须确保开发狗+超级狗都连接在电脑。

（1）安装windows版本加密狗软件到C:\Program Files (x86)\SafeNet\SuperDog\2.3\。
（2）在线下载开发商代码。
（3）打开授权管理工具，添加特征。例如feature_id=101,然后编程到超级狗。

开发商代码会写入vendor_code到C:\Program Files (x86)\SafeNet\SuperDog\2.3\VendorCodes\BYAUY.hvc

Superdog sample

参见 C:\Program Files (x86)\SafeNet\SuperDog\2.3\Samples\Licensing\C

Superdog C SDK

从API中整理出C版本的SDK，供后面使用。

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        2014/6/30      8:43           9521 dog_api.h
-a----       2016/11/10     19:10        3297280 dog_windows_3150436.dll
-a----       2016/11/10     19:10           5668 dog_windows_3150436.lib
-a----       2016/11/10     19:10        3632128 dog_windows_x64_3150436.dll
-a----       2016/11/10     19:10           5530 dog_windows_x64_3150436.lib

superdog-config.cmake

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

set(SUPERDOG_FOUND TRUE) # auto 
set(SUPERDOG_VERSION 2.3.0)
set(SUPERDOG_ROOT_DIR "C:/car_libs/superdog-c-2.3")

find_path(SUPERDOG_INCLUDE_DIR NAMES superdog/dog_api.h PATHS "${SUPERDOG_ROOT_DIR}/include") 
mark_as_advanced(SUPERDOG_INCLUDE_DIR) # show entry in cmake-gui

find_library(SUPERDOG_LIBRARY NAMES dog_windows_x64_3150436.lib PATHS "${SUPERDOG_ROOT_DIR}/lib") 
mark_as_advanced(SUPERDOG_LIBRARY) # show entry in cmake-gui

# use xxx_INCLUDE_DIRS and xxx_LIBRARIES in CMakeLists.txt
set(SUPERDOG_INCLUDE_DIRS ${SUPERDOG_INCLUDE_DIR} )
set(SUPERDOG_LIBRARIES ${SUPERDOG_LIBRARY} )

message( "superdog-config.cmake " ${SUPERDOG_ROOT_DIR})

Code Example

code

MyDogApiImpl.h

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

#pragma once
#include <iostream>
#include <string>

#include <superdog/dog_api.h>

#include "errorprinter.h"

class MyDogApiImpl 
{
public:
	MyDogApiImpl(int feature_id, const std::string& vendor_code);

	int login();

	int logout();

private:
	dog_handle_t   handle;
	int m_feature_id;
	std::string m_vendor_code;

	ErrorPrinter err;
};

MyDogApiImpl.cpp

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#include "mydog_api_impi.h"

MyDogApiImpl::MyDogApiImpl(int feature_id, const std::string& vendor_code)
{
	m_feature_id = feature_id;
	m_vendor_code = vendor_code;
}

int MyDogApiImpl::login()
{
	dog_status_t status = dog_login(
		m_feature_id,
		(dog_vendor_code_t *)m_vendor_code.c_str(),
		&handle
	);
	err.printError(status);
	return (int)status;
}

int MyDogApiImpl::logout()
{
	dog_status_t status = dog_logout(handle);
	err.printError(status);
	return (int)status;
}

test

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

#include <iostream>
#include <string>

#include <gtest/gtest.h>
#include <glog/logging.h>

#include "mydog/mydog.h"

using namespace std;

int test_api()
{
	int feature_id = 111;
	std::string vendor_code = "xxx"; // from `VendorCodes\BYAUY.hvc`
	
	MyDogApi dog(feature_id, vendor_code);
	int status = dog.login(); // 0  OK, other failed (feature id error, vendor_code error, NO SuperDog)

	std::cout << "status = " << status << std::endl;

	int status2 = dog.logout();
	std::cout << "status2 = " << status2 << std::endl;
	return 0;
}

TEST(test, test_api) {
	test_api();
}

Python version supported

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

#include "mydog_api.h"

#define BUILD_PYTHON_MYDOG

#ifdef BUILD_PYTHON_MYDOG

#include <pybind11/pybind11.h>

namespace py = pybind11;

int add(int i, int j) {
	return i + j;
}

struct Pet {
	Pet(const std::string &name) : name(name) { }
	void setName(const std::string &name_) { name = name_; }
	const std::string &getName() const { return name; }

	std::string name;
};

PYBIND11_MODULE(mydog, m)
{
	// optional module docstring
	m.doc() = "mydog plugin for python ";

	// FUNCTIONS
	// expose add function, and add keyword arguments and default arguments
	m.def("add", &add, "A function which adds two numbers", py::arg("i") = 1, py::arg("j") = 2);

	// DATA
	// exporting variables
	m.attr("the_answer") = 42;
	py::object world = py::cast("World");
	m.attr("what") = world;


	// CLASSES
	/*
	py::class_<Pet>(m, "Pet")
	.def(py::init<const std::string &>())
	.def("setName", &Pet::setName)
	.def("getName", &Pet::getName);
	*/

	// MyDog
	py::class_<MyDogApi>(m, "MyDogApi")
		.def(py::init<int, const std::string &>())
		.def("login", &MyDogApi::login)
		.def("logout", &MyDogApi::logout);
}


/*
// =====================1
pybind11_add_module(${PYMYDOG_TARGET}
${HEADER_FILES}
${SOURCE_FILES}

# supder dog (dog_login,dog_logout)
${SUPERDOG_LIBRARIES}
)

// =====================2
add_library(${PYMYDOG_TARGET} MODULE
${HEADER_FILES}
${SOURCE_FILES}
)

target_link_libraries(${PYMYDOG_TARGET}
pybind11::module
${SUPERDOG_LIBRARIES}
)
*/

#endif // BUILD_PYTHON_MYDOG

cmake for C++ version

1
2
3
4
5
6
7
8
9
10
11

find_package(SUPERDOG REQUIRED) # user-defined supderdog 2.3
include_directories(${SUPERDOG_INCLUDE_DIRS})

add_library(${MYDOG_TARGET} 
	${HEADER_FILES}
	${SOURCE_FILES}
)

target_link_libraries (${MYDOG_TARGET}   
	${SUPERDOG_LIBRARIES}
)

cmake for Python version

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

find_package(SUPERDOG REQUIRED) # user-defined supderdog 2.3
include_directories(${SUPERDOG_INCLUDE_DIRS})

find_package(pybind11 CONFIG REQUIRED)
include_directories(${pybind11_INCLUDE_DIRS})

add_library(${MYDOG_TARGET} 
	${HEADER_FILES}
	${SOURCE_FILES}
)

target_link_libraries (${MYDOG_TARGET}   
	pybind11::module
	${SUPERDOG_LIBRARIES}
)

rename mydog.dll to mydog.pyd

test pymydog

1
2
3
4
5
6
7
8

import mydog # mydog.pyd

feature_id = 111
vendor_code = "xxx"

dog = mydog.MyDogApi(feature_id,vendor_code)
print(dog.login())
print(dog.logout())

with files

dog_windows_x64_3150436.dll
mydog.py
mydog.pyd

Reference

History

• 20181022: created.

Guide

• windows 10 64-bit
• crypto++ 7.0
• vs 2015

compile

download crypto++ 7.0

open cryptest.sln with Visual Studio 2015

change Windows SDK Version from 8.1 to 10.0

compile cryptodll with Release x64 and we get cryptopp.dll and cryptopp.lib

sdk

copy headers to include, copy libs to lib and dlls to dll like this:

cryptopp-config.cmake

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

set(CRYPTOPP_FOUND TRUE) # auto 
set(CRYPTOPP_VERSION 7.0.0)
set(CRYPTOPP_ROOT_DIR "C:/car_libs/cryptopp-7.0")

find_path(CRYPTOPP_INCLUDE_DIR NAMES cryptopp/aes.h PATHS "${CRYPTOPP_ROOT_DIR}/include") 
mark_as_advanced(CRYPTOPP_INCLUDE_DIR) # show entry in cmake-gui

find_library(CRYPTOPP_LIBRARY NAMES cryptopp.lib PATHS "${CRYPTOPP_ROOT_DIR}/lib") 
mark_as_advanced(CRYPTOPP_LIBRARY) # show entry in cmake-gui

# use xxx_INCLUDE_DIRS and xxx_LIBRARIES in CMakeLists.txt
set(CRYPTOPP_INCLUDE_DIRS ${CRYPTOPP_INCLUDE_DIR} )
set(CRYPTOPP_LIBRARIES ${CRYPTOPP_LIBRARY} )

message( "cryptopp-config.cmake " ${CRYPTOPP_ROOT_DIR})

Example

see cryptopp-7.0/dlltest.cpp

Code

cipher.h

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

#pragma once

#pragma warning( disable: 4275 ) 
#pragma warning( disable: 4819 )

#include <iostream>

#include <cryptopp/dll.h>
#include <cryptopp/cryptlib.h>
#include <cryptopp/aes.h>
#include <cryptopp/filters.h>
#include <cryptopp/modes.h>

using namespace std;

USING_NAMESPACE(CryptoPP)

class CipherApiImpl 
{
public:
	enum MODE
	{
		AES,
		DES
	};

public:
	CipherApiImpl(MODE mode = MODE::AES);

	int get_filesize(const char *filename);

	int save(const std::string& filepath, const std::string& str);
	std::string load(const std::string& filepath);
	
	std::string encrypt_string(const std::string& plain_text);
	std::string decrypt_string(const std::string& cipher_text);

	std::string encrypt_file(const std::string& infile);
	std::string decrypt_file(const std::string& infile);

	int encrypt_file(const std::string& infile, const std::string& outfile);
	int decrypt_file(const std::string& infile, const std::string& outfile);

protected:
	std::string tohex(const char& c);
	std::string tohexs(const std::string& str);

	char tochar(const char& hex1, const char& hex2);
	std::string tochars(const std::string& hex);

private:
	// 32 + 16
	const byte aes_key[CryptoPP::AES::MAX_KEYLENGTH] = {
		0x01,0x23,0x45,0x67,0x89,0xab,0xcd,0xef,
		0x01,0x23,0x45,0x67,0x89,0xab,0xcd,0xef,
		0x01,0x23,0x45,0x67,0x89,0xab,0xcd,0xef,
		0x01,0x23,0x45,0x67,0x89,0xab,0xcd,0xef
	};
	const byte aes_iv[CryptoPP::AES::BLOCKSIZE] = {
		0x12,0x34,0x56,0x78,0x90,0xab,0xcd,0xef,
		0x12,0x34,0x56,0x78,0x90,0xab,0xcd,0xef
	};

	// 24+8
	const byte des_key[CryptoPP::DES_EDE3::MAX_KEYLENGTH] = {
		0x01,0x23,0x45,0x67,0x89,0xab,0xcd,0xef,
		0x01,0x23,0x45,0x67,0x89,0xab,0xcd,0xef,
		0x01,0x23,0x45,0x67,0x89,0xab,0xcd,0xef
	};
	const byte des_iv[CryptoPP::DES_EDE3::BLOCKSIZE] = {
		0x12,0x34,0x56,0x78,0x90,0xab,0xcd,0xef
	};

	CryptoPP::CFB_FIPS_Mode<CryptoPP::AES>::Encryption aes_encryption;
	CryptoPP::CFB_FIPS_Mode<CryptoPP::AES>::Decryption aes_decryption;

	CryptoPP::CFB_FIPS_Mode<CryptoPP::DES_EDE3>::Encryption des_encryption;
	CryptoPP::CFB_FIPS_Mode<CryptoPP::DES_EDE3>::Decryption des_decryption;

	MODE m_mode;
};

cipher.cpp

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285

#include "cipher_api_impl.h"

#include <sys/stat.h>

CipherApiImpl::CipherApiImpl(MODE mode)
{
	m_mode = mode;
	
	// for des
	des_encryption.SetKeyWithIV(des_key, sizeof(des_key), des_iv);
	des_decryption.SetKeyWithIV(des_key, sizeof(des_key), des_iv);

	// for aes
	aes_encryption.SetKeyWithIV(aes_key, sizeof(aes_key), aes_iv);
	aes_decryption.SetKeyWithIV(aes_key, sizeof(aes_key), aes_iv);
}

#pragma region hex/char

std::string CipherApiImpl::tohex(const char& c)
{
	/*
	00,01,02,...0F   [0,1,2,...15]
	10,11,12,...1F   [16,17,...31]
	...
	F0,F1,F2,...FF	[240,241,...255]
	*/

	/*
	char ch[2] = { 0 };
	sprintf(ch, "%02x", static_cast<byte>(c)); //  char ---> 16 hex
	return std::string(ch);
	*/
	const char HEX[] = "0123456789ABCDEF";
	const int HEX_BASE = 16;

	std::string c_hex;

	int uc = (unsigned char)(c);
	int a = uc / HEX_BASE;
	int b = uc % HEX_BASE;
	c_hex.push_back(HEX[a]);
	c_hex.push_back(HEX[b]);

	return c_hex;
}

std::string CipherApiImpl::tohexs(const std::string& str)
{
	const char HEX[] = "0123456789ABCDEF";
	const int HEX_BASE = 16;

	std::string hex_result;
	for (size_t i = 0; i < str.size(); i++)
	{
		std::string c_hex;
		int uc = (unsigned char)(str[i]);
		int a = uc / HEX_BASE;
		int b = uc % HEX_BASE;
		c_hex.push_back(HEX[a]);
		c_hex.push_back(HEX[b]);

		hex_result += c_hex;
	}
	return hex_result;
}

char CipherApiImpl::tochar(const char& hex1, const char& hex2)
{
	const int HEX_BASE = 16;

	int h1 = (int)hex1;
	int h2 = (int)hex2;

	// F1 ===> 16,1===>15*16+1 = 241 ===> char
	char uc = (char)(h1*HEX_BASE + h2);
	return uc;
}

std::string CipherApiImpl::tochars(const std::string& hex)
{
	if (hex.size() % 2 != 0)
	{
		return "";
	};

	std::map<char, int> hex_int = {
		{ '0' , 0 },
		{ '1', 1 },
		{ '2', 2 },
		{ '3', 3 },
		{ '4', 4 },
		{ '5', 5 },
		{ '6', 6 },
		{ '7', 7 },
		{ '8', 8 },
		{ '9', 9 },
		{ 'A', 10 },
		{ 'B', 11 },
		{ 'C', 12 },
		{ 'D', 13 },
		{ 'E', 14 },
		{ 'F', 15 },
	};

	std::string char_result;
	const int HEX_BASE = 16;

	for (size_t i = 0; i < hex.size(); i += 2)
	{
		int h1 = hex_int.at(hex[i]);   // 'F' ===>15
		int h2 = hex_int.at(hex[i + 1]); // '1'===>1

										 // F1 ===> 15,1===>15*16+1 = 241 ===> char
										 //std::cout << "===============\n";
		char uc = (char)(h1*HEX_BASE + h2);
		//std::cout << h1 <<","<<h2 << std::endl;
		//std::cout << uc << std::endl;
		char_result += uc;
	}
	return char_result;
}
#pragma endregion

#pragma region encrypt_string and decrypt_string

int CipherApiImpl::get_filesize(const char *filename)
{
	struct stat f_stat;
	if (stat(filename, &f_stat) == -1)
	{
		return -1;
	}
	return f_stat.st_size;
}

/*
std::string load_text_not_use(const std::string& filepath)
{
std::string str;
ifstream in(filepath, ios::in | ios::binary);

std::string line;
std::string content;
while (getline(in, line))
{
//std::cout << "line.size()=" << line.size() << std::endl;
if (line.size() > 0) {
content += line + "\n";
}
line.clear();
}

in.close();

return content;
}
*/

int CipherApiImpl::save(const std::string& filepath, const std::string& str)
{
	ofstream out(filepath, ios::out | ios::binary); // binary mode (default text mode)
	out.write(str.c_str(), str.length());
	out.close();
	return 1;
}

std::string CipherApiImpl::load(const std::string& filepath)
{
	int filesize = get_filesize(filepath.c_str());

	char *buffer = new char[filesize];

	ifstream in(filepath, ios::in | ios::binary); // binary mode (default text mode)
	in.read(buffer, filesize);
	in.close();

	// std::string content(buffer); // ERROR for binary file
	std::string content;
	for (size_t i = 0; i < filesize; i++)
	{
		content.push_back(buffer[i]);
	}
	delete[] buffer;

	return content;
}

/*
//==========================================================================
std::string cipher_text;
CryptoPP::AES::CipherApiImpl aesEncryption(key, CryptoPP::AES::MAX_KEYLENGTH);
CryptoPP::CBC_Mode_ExternalCipher::CipherApiImpl cbcEncryption(aesEncryption, iv);
CryptoPP::StreamTransformationFilter stfEncryptor(cbcEncryption, new CryptoPP::StringSink(cipher_text));
stfEncryptor.Put(reinterpret_cast<const unsigned char*>(plain_text.c_str()), plain_text.length());
stfEncryptor.MessageEnd();

std::string cipher_text_hex = tohexs(cipher_text);
return cipher_text_hex;
//==========================================================================
CryptoPP::AES::Decryption aesDecryption(key, CryptoPP::AES::MAX_KEYLENGTH);
CryptoPP::CBC_Mode_ExternalCipher::Decryption cbcDecryption(aesDecryption, iv);
CryptoPP::StreamTransformationFilter stfDecryptor(cbcDecryption, new CryptoPP::StringSink(decryptedText));
stfDecryptor.Put(reinterpret_cast<const unsigned char*>(cipher_text.c_str()), cipher_text.size());
stfDecryptor.MessageEnd();

return decrypted_text;
//==========================================================================

byte ciphertext[24];
byte decrypted[24];

CFB_FIPS_Mode<DES_EDE3>::Encryption encryption_DES_EDE3_CFB;
encryption_DES_EDE3_CFB.SetKeyWithIV(key, sizeof(key), iv);
encryption_DES_EDE3_CFB.ProcessString(ciphertext, plaintext, 24);

CFB_FIPS_Mode<DES_EDE3>::Decryption decryption_DES_EDE3_CFB;
decryption_DES_EDE3_CFB.SetKeyWithIV(key, sizeof(key), iv);
decryption_DES_EDE3_CFB.ProcessString(decrypted, ciphertext, 24);
*/

std::string CipherApiImpl::encrypt_string(const std::string& plain_text)
{
	size_t size = plain_text.size();
	const byte* in_byte = (const unsigned char*)(plain_text.c_str());

	byte* out_byte = new byte[size];

	des_encryption.ProcessString(out_byte, in_byte, size);

	std::string cipher_text;
	for (size_t i = 0; i < size; i++)
	{
		cipher_text.push_back(out_byte[i]);
	}
	delete[] out_byte;

	return cipher_text;
}

std::string CipherApiImpl::decrypt_string(const std::string& cipher_text)
{
	size_t size = cipher_text.size();
	const byte* in_byte = (const unsigned char*)(cipher_text.c_str());
	byte* out_byte = new byte[size];

	des_decryption.ProcessString(out_byte, in_byte, size);

	std::string plain_text;
	for (size_t i = 0; i < size; i++)
	{
		plain_text.push_back(out_byte[i]);
	}
	delete[] out_byte;

	return plain_text;
}

std::string CipherApiImpl::encrypt_file(const std::string& infile)
{
	std::string plain_text = load(infile);
	std::string cipher_text = encrypt_string(plain_text);
	return cipher_text;
}

std::string CipherApiImpl::decrypt_file(const std::string& infile)
{
	std::string cipher_text = load(infile);
	std::string plain_text = decrypt_string(cipher_text);
	return plain_text;
}

int CipherApiImpl::encrypt_file(const std::string& infile, const std::string& outfile)
{
	std::string cipher_text_hex = encrypt_file(infile);
	return save(outfile, cipher_text_hex);
}

int CipherApiImpl::decrypt_file(const std::string& infile, const std::string& outfile)
{
	std::string plain_text = decrypt_file(infile);
	return save(outfile, plain_text);
}

#pragma endregion

test.cpp

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

#include <iostream>
#include <string>

#include <gtest/gtest.h>
#include <glog/logging.h>

#pragma warning( disable: 4819 ) 
#pragma warning( disable: 4244 ) 
#pragma warning( disable: 4267 ) 

#include "cipher/cipher.h"

using namespace std;

int test_cipher_api()
{
	CipherApi cipher;

	std::string text = "[hello zhuzhu dashen !]\n[1234567]\n[123]\n1\n";
	cout << "text : " << text << endl;
	cout << "text size: " << text.size() << endl;
	cipher.save("./tmp/text.txt", text);

	std::cout << "==============encrypt/decrypt string======================\n";

	std::string cipherHex = cipher.encrypt_string(text);
	//cout << "cipher : " << cipherHex << endl;
	cipher.save("./tmp/cipherHex.txt", cipherHex);

	std::string text2 = cipher.decrypt_string(cipherHex);
	cout << "text2 : " << text2 << endl;
	cout << "text2 size: " << text2.size() << endl;
	cipher.save("./tmp/text2.txt", text2);

	// load_text 
	std::string text_load = cipher.load("./tmp/text2.txt");
	cout << "text_load : " << text_load << endl;
	cout << "text_load size: " << text_load.size() << endl;

	// encrypt/decrypt text file
	std::cout << "==============encrypt/decrypt text file======================\n";
	cipher.encrypt_file("./tmp/text.txt", "./tmp/text_encryption.txt");
	cipher.decrypt_file("./tmp/cipherHex.txt", "./tmp/cipherHex_decryption.txt");

	// encrypt/decrypt prototxt 
	std::cout << "==============encrypt/decrypt prototxt======================\n";
	cipher.encrypt_file("./tmp/small/deploy.prototxt", "./tmp/small/deploy_encryption.prototxt");
	cipher.decrypt_file("./tmp/small/deploy_encryption.prototxt", "./tmp/small/deploy_new.prototxt");

	std::cout << "==============encrypt/decrypt caffemodel======================\n";
	cipher.encrypt_file("./tmp/small/recon__iter_140000.caffemodel", "./tmp/small/recon__iter_140000_encryption.caffemodel");
	cipher.decrypt_file("./tmp/small/recon__iter_140000_encryption.caffemodel", "./tmp/small/recon__iter_140000_new.caffemodel");
	return 0;
}

TEST(cipher_test, test_cipher_api) {
	test_cipher_api();
}

CMakeLists.txt

1
2
3
4
5
6
7

find_package(CRYPTOPP REQUIRED)

MESSAGE( [Main] " CRYPTOPP_INCLUDE_DIRS = ${CRYPTOPP_INCLUDE_DIRS}")
MESSAGE( [Main] " CRYPTOPP_LIBRARIES = ${CRYPTOPP_LIBRARIES}")

include_directories(${CRYPTOPP_INCLUDE_DIRS})
target_link_libraries (demo  ${CRYPTOPP_LIBRARIES})

Reference

• cryptopp github

History

• 20181018: created.

Guide

param input && stdin

linux命令可以从两个地方读取要处理的内容

• 通过命令行参数输入,通过main函数的argc,argv获取
• 标准输入stdin,通过scanf或者cin获取

然后产生输出

• 标准输出stdout,通过printf或者cout输出

命令类型

• 部分命令可以接受参数输入＋标准输入stdin，比如cat、grep等。
• 部分命令只接受参数输入，不接受标准输入stdin，比如kill、rm等。