Kezunlin's Blog

Live and Learn

0%

tensorflow basic introduction

Posted on Edited on In

Set up

1
2
3
4
5
6
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import numpy as np
import tensorflow as tf

Tensor Values

tensor, rank, shape, value

1
2
3
4
5
6
7
8
9
10
11
1. # a rank 0 tensor; a scalar with shape [],

[1., 2., 3.] # a rank 1 tensor; a vector with shape [3]

[[1., 2., 3.], 
 [4., 5., 6.]] # a rank 2 tensor; a matrix with shape [2, 3]

[
    [[1., 2., 3.]], 
     [[7., 8., 9.]]
] # a rank 3 tensor with shape [2, 1, 3]
[[[1.0, 2.0, 3.0]], [[7.0, 8.0, 9.0]]]

TensorFlow Core Walkthrough

You might think of TensorFlow Core programs as consisting of two discrete sections:

  • Building the computational graph (a tf.Graph).
  • Running the computational graph (using a tf.Session).

Graph

A computational graph is a series of TensorFlow operations arranged into a graph. The graph is composed of two types of objects.

  • tf.Operation (or “ops”): The nodes of the graph. Operations describe calculations that consume and produce tensors.
  • tf.Tensor: The edges in the graph. These represent the values that will flow through the graph. Most TensorFlow functions return tf.Tensors.
1
2
3
4
5
6
7
8
# tf.Tensors do not have values

a = tf.constant(3.0, dtype=tf.float32)
b = tf.constant(4.0) # also tf.float32 implicitly
total = a + b
print(a)
print(b)
print(total)
Tensor("Const_4:0", shape=(), dtype=float32)
Tensor("Const_5:0", shape=(), dtype=float32)
Tensor("add_2:0", shape=(), dtype=float32)

TensorBoard

visualizing a computation graph.

1
2
writer = tf.summary.FileWriter('.')
writer.add_graph(tf.get_default_graph()) # events.out.tfevents.{timestamp}.{hostname}

Now, in a new terminal, launch TensorBoard with the following shell command:

tensorboard --logdir .

then access http://localhost:6006/#graphs

Session

To evaluate tensors, instantiate a tf.Session object, informally known as a session. A session encapsulates the state of the TensorFlow runtime, and runs TensorFlow operations. If a tf.Graph is like a .py file, a tf.Session is like the python executable.

1
2
sess = tf.Session()
print(sess.run(total))
7.0

1
print(sess.run({'ab':(a, b), 'total':total}))
{'total': 7.0, 'ab': (3.0, 4.0)}

1
2
3
4
5
6
7
# 第一次，第二次run的vec结果都不一样；第三次先生成vec，然后得到out1和out2
vec = tf.random_uniform(shape=(3,))
out1 = vec + 1
out2 = vec + 2
print(sess.run(vec))
print(sess.run(vec))
print(sess.run((out1, out2)))
[0.3936273  0.7793083  0.46128905]
[0.9422312  0.03195083 0.4434662 ]
(array([1.9042395, 1.5705729, 1.4484392], dtype=float32), array([2.9042397, 2.5705729, 2.4484391], dtype=float32))

Feeding

A placeholder is a promise to provide a value later, like a function argument.

1
2
3
4
5
6
x = tf.placeholder(tf.float32)
y = tf.placeholder(tf.float32)
z = x + y
print(x)
print(y)
print(z)
Tensor("Placeholder_2:0", dtype=float32)
Tensor("Placeholder_3:0", dtype=float32)
Tensor("add_6:0", dtype=float32)

1
2
print(sess.run(z, feed_dict={x: 3, y: 4.5}))
print(sess.run(z, feed_dict={x: [1, 3], y: [2, 4]}))
7.5
[3. 7.]

1
2
# placeholders throw an error if no value is fed to them.
# print(sess.run(z))

Datasets

1
2
3
4
5
6
7
8
my_data = [
    [0, 1,],
    [2, 3,],
    [4, 5,],
    [6, 7,],
]
slices = tf.data.Dataset.from_tensor_slices(my_data)
next_item = slices.make_one_shot_iterator().get_next()

Reaching the end of the data stream causes Dataset to throw an tf.errors.OutOfRangeError. For example, the following code reads the next_item until there is no more data to read:

1
2
3
4
5
while True:
  try:
    print(sess.run(next_item))
  except tf.errors.OutOfRangeError:
    break
[0 1]
[2 3]
[4 5]
[6 7]

If the Dataset depends on stateful operations you may need to initialize the iterator before using it, as shown below:

1
2
3
4
5
6
7
8
9
10
11
r = tf.random_normal([10,3])
dataset = tf.data.Dataset.from_tensor_slices(r)
iterator = dataset.make_initializable_iterator()
next_row = iterator.get_next()

sess.run(iterator.initializer) # initialize iterator
while True:
  try:
    print(sess.run(next_row))
  except tf.errors.OutOfRangeError:
    break
[-0.27240568  0.41916385  0.8312674 ]
[-0.37411043  0.39321673  0.44852588]
[-2.502281   -0.35216704 -0.42525485]
[ 0.16175386  0.4726008  -0.02786499]
[-0.252643    0.47525632  1.3186764 ]
[ 0.2424109  1.3193169 -0.4673948]
[ 0.43207827  0.31257248 -0.609174  ]
[-0.80196303 -1.100821    2.44067   ]
[0.79778594 0.34206432 1.0677754 ]
[ 0.47419003 -0.7259666   0.8751686 ]

Layers

Creating Layers

1
2
3
x = tf.placeholder(tf.float32, shape=[None, 3])
linear_model = tf.layers.Dense(units=1) # Layer
y = linear_model(x)

Initializing Layers

The layer contains variables that must be initialized before they can be used. 

1
2
init = tf.global_variables_initializer()
sess.run(init)

Executing Layers

1
print(sess.run(y, {x: [[1, 2, 3],[4, 5, 6]]}))
[[ 4.5850883]
 [10.428741 ]]

Layer Function shortcuts

For each layer class (like tf.layers.Dense) TensorFlow also supplies a shortcut function (like tf.layers.dense). 

1
2
3
4
5
6
7
x = tf.placeholder(tf.float32, shape=[None, 3])
y = tf.layers.dense(x, units=1)

init = tf.global_variables_initializer()
sess.run(init)

print(sess.run(y, {x: [[1, 2, 3], [4, 5, 6]]}))
[[2.4798913]
 [6.7798343]]

Feature columns

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
features = {
    'sales' : [[5], [10], [8], [9]],
    'department': ['sports', 'sports', 'gardening', 'gardening']}

department_column = tf.feature_column.categorical_column_with_vocabulary_list(
        'department', ['sports', 'gardening'])
department_column = tf.feature_column.indicator_column(department_column)
print(department_column)

columns = [
    tf.feature_column.numeric_column('sales'),
    department_column
]

inputs = tf.feature_column.input_layer(features, columns)
print(inputs)
_IndicatorColumn(categorical_column=_VocabularyListCategoricalColumn(key='department', vocabulary_list=('sports', 'gardening'), dtype=tf.string, default_value=-1, num_oov_buckets=0))
Tensor("input_layer/concat:0", shape=(4, 3), dtype=float32)

Running the inputs tensor will parse the features into a batch of vectors.

Feature columns can have internal state, like layers, so they often need to be initialized. Categorical columns use tf.contrib.lookup internally and these require a separate initialization op, tf.tables_initializer.

1
2
3
4
var_init = tf.global_variables_initializer()
table_init = tf.tables_initializer()
sess = tf.Session()
sess.run((var_init, table_init))
(None, None)

1
print(sess.run(inputs))
[[ 1.  0.  5.]
 [ 1.  0. 10.]
 [ 0.  1.  8.]
 [ 0.  1.  9.]]

Training

Define the data

1
2
x = tf.constant([[1], [2], [3], [4]], dtype=tf.float32)
y_true = tf.constant([[0], [-1], [-2], [-3]], dtype=tf.float32)

Define the model

1
2
3
linear_model = tf.layers.Dense(units=1)

y_pred = linear_model(x)
1
2
3
4
5
6
7
sess = tf.Session()
init = tf.global_variables_initializer()
sess.run(init)

# The model hasn't yet been trained, so the four "predicted" values aren't very good. 
# Here's what we got; your own output will almost certainly differ:
print(sess.run(y_pred))
[[0.5062338]
 [1.0124676]
 [1.5187014]
 [2.0249352]]

Loss

To optimize a model, you first need to define the loss. We’ll use the mean square error (MSE), a standard loss for regression problems.

1
2
3
loss = tf.losses.mean_squared_error(labels=y_true, predictions=y_pred)

print(sess.run(loss))
10.484383

Training

TensorFlow provides optimizers implementing standard optimization algorithms. These are implemented as sub-classes of tf.train.Optimizer. They incrementally change each variable in order to minimize the loss. The simplest optimization algorithm is gradient descent, implemented by tf.train.GradientDescentOptimizer. 

1
2
optimizer = tf.train.GradientDescentOptimizer(0.01)
train = optimizer.minimize(loss) # training operation
1
2
3
for i in range(100):
  _, loss_value = sess.run((train, loss))
  print(loss_value)
10.484383
7.366995
5.2033534
3.7015016
2.6588542
1.9348423
1.4319282
1.0824323
0.83939326
0.67022556
0.5523189
0.46998408
0.41233516
0.37181824
0.34319195
0.32281947
0.30817705
0.29751378
0.28961438
0.2836359
0.27899322
0.2752804
0.27221578
0.26960373
0.26730868
0.2652365

1
print(sess.run(y_pred))
[[-0.66274  ]
 [-1.3211429]
 [-1.9795457]
 [-2.6379485]]

Complete program

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
x = tf.constant([[1], [2], [3], [4]], dtype=tf.float32)
y_true = tf.constant([[0], [-1], [-2], [-3]], dtype=tf.float32)

linear_model = tf.layers.Dense(units=1)

y_pred = linear_model(x)
loss = tf.losses.mean_squared_error(labels=y_true, predictions=y_pred)

optimizer = tf.train.GradientDescentOptimizer(0.01)
train = optimizer.minimize(loss)

init = tf.global_variables_initializer()

sess = tf.Session()
sess.run(init)

for i in range(100):
  _, loss_value = sess.run((train, loss))
  #print(loss_value)

print(sess.run(y_pred))
[[-0.5594496]
 [-1.2710916]
 [-1.9827336]
 [-2.6943758]]

Reference

History

  • 20181026: created.