TensorFlow 2.0 + Keras Crash Course

Ich habe heute einen großartiger TensorFlow 2.0 + Keras Crash Course für Forscher im Bereich des tiefen Lernens/Deep Learning durch den Schöpfer von Keras (Bibliothek für neuronale Netzwerke), François Chollet, entdeckt.

Code wird auch in einem Colab-Notizbuch bereitgestellt.

Ich werde unten ein Copy Paste von tf.keras for Researchers: Crash Course.ipynb machen. Viel Spaß!

!pip install tf-nightly-gpu-2.0-preview

import tensorflow as tf

tf.__version__

1) The first class you need to know is Layer. A Layer encapsulates a state (weights) and some computation (defined in the call method).

from tensorflow.keras.layers import Layer

class Linear(Layer):
„““y = w.x + b“““

def __init__(self, units=32, input_dim=32):
super(Linear, self).__init__()
w_init = tf.random_normal_initializer()
self.w = tf.Variable(
initial_value=w_init(shape=(input_dim, units), dtype=’float32′),
trainable=True)
b_init = tf.zeros_initializer()
self.b = tf.Variable(
initial_value=b_init(shape=(units,), dtype=’float32′),
trainable=True)

def call(self, inputs):
return tf.matmul(inputs, self.w) + self.b

# Instantiate our layer.
linear_layer = Linear(4, 2)

# The layer can be treated as a function.
# Here we call it on some data.
y = linear_layer(tf.ones((2, 2)))
assert y.shape == (2, 4)

# Weights are automatically tracked under the `weights` property.
assert linear_layer.weights == [linear_layer.w, linear_layer.b]

2) The add_weight method gives you a shortcut for creating weights.

3) It’s good practice to create weights in a separate build method, called lazily with the shape of the first inputs seen by your layer. Here, this pattern prevents us from having to specify input_dim in the constructor:

class Linear(Layer):
„““y = w.x + b“““

def __init__(self, units=32):
super(Linear, self).__init__()
self.units = units

def build(self, input_shape):
self.w = self.add_weight(shape=(input_shape[-1], self.units),
initializer=’random_normal‘,
trainable=True)
self.b = self.add_weight(shape=(self.units,),
initializer=’random_normal‘,
trainable=True)

def call(self, inputs):
return tf.matmul(inputs, self.w) + self.b

# Instantiate our lazy layer.
linear_layer = Linear(4)

# This will also call `build(input_shape)` and create the weights.
y = linear_layer(tf.ones((2, 2)))

4) You can automatically retrieve the gradients of the weights of a layer by calling it inside a GradientTape. Using these gradients, you can update the weights of the layer, either manually, or using an optimizer object. Of course, you can modify the gradients before using them, if you need to.

# Prepare a dataset.
(x_train, y_train), _ = tf.keras.datasets.mnist.load_data()
dataset = tf.data.Dataset.from_tensor_slices(
(x_train.reshape(60000, 784).astype(‚float32‘) / 255, y_train))
dataset = dataset.shuffle(buffer_size=1024).batch(64)

# Instantiate our linear layer (defined above) with 10 units.
linear_layer = Linear(10)

# Instantiate a logistic loss function that expects integer targets.
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

# Instantiate an optimizer.
optimizer = tf.keras.optimizers.SGD(learning_rate=1e-3)

# Iterate over the batches of the dataset.
for step, (x, y) in enumerate(dataset):

# Open a GradientTape.
with tf.GradientTape() as tape:

# Forward pass.
logits = linear_layer(x)

# Loss value for this batch.
loss = loss_fn(y, logits)

# Get gradients of weights wrt the loss.
gradients = tape.gradient(loss, linear_layer.trainable_weights)

# Update the weights of our linear layer.
optimizer.apply_gradients(zip(gradients, linear_layer.trainable_weights))

# Logging.
if step % 100 == 0:
print(step, float(loss))

5) Weights created by layers can be either trainable or non-trainable. They’re exposed in trainable_weights and non_trainable_weights. Here’s a layer with a non-trainable weight:

class ComputeSum(Layer):
„““Returns the sum of the inputs.“““

def __init__(self, input_dim):
super(ComputeSum, self).__init__()
# Create a non-trainable weight.
self.total = tf.Variable(initial_value=tf.zeros((input_dim,)),
trainable=False)

def call(self, inputs):
self.total.assign_add(tf.reduce_sum(inputs, axis=0))
return self.total

my_sum = ComputeSum(2)
x = tf.ones((2, 2))

y = my_sum(x)
print(y.numpy()) # [2. 2.]

y = my_sum(x)
print(y.numpy()) # [4. 4.]

assert my_sum.weights == [my_sum.total]
assert my_sum.non_trainable_weights == [my_sum.total]
assert my_sum.trainable_weights == []

6) Layers can be recursively nested to create bigger computation blocks. Each layer will track the weights of its sublayers (both trainable and non-trainable).

# Let’s reuse the Linear class
# with a `build` method that we defined above.

class MLP(Layer):
„““Simple stack of Linear layers.“““

def __init__(self):
super(MLP, self).__init__()
self.linear_1 = Linear(32)
self.linear_2 = Linear(32)
self.linear_3 = Linear(10)

def call(self, inputs):
x = self.linear_1(inputs)
x = tf.nn.relu(x)
x = self.linear_2(x)
x = tf.nn.relu(x)
return self.linear_3(x)

mlp = MLP()

# The first call to the `mlp` object will create the weights.
y = mlp(tf.ones(shape=(3, 64)))

# Weights are recursively tracked.
assert len(mlp.weights) == 6

7) Layers can create losses during the forward pass. This is especially useful for regularization losses. The losses created by sublayers are recursively tracked by the parent layers.

class ActivityRegularization(Layer):
„““Layer that creates an activity sparsity regularization loss.“““

def __init__(self, rate=1e-2):
super(ActivityRegularization, self).__init__()
self.rate = rate

def call(self, inputs):
# We use `add_loss` to create a regularization loss
# that depends on the inputs.
self.add_loss(self.rate * tf.reduce_sum(inputs))
return inputs

# Let’s use the loss layer in a MLP block.

class SparseMLP(Layer):
„““Stack of Linear layers with a sparsity regularization loss.“““

def __init__(self):
super(SparseMLP, self).__init__()
self.linear_1 = Linear(32)
self.regularization = ActivityRegularization(1e-2)
self.linear_3 = Linear(10)

def call(self, inputs):
x = self.linear_1(inputs)
x = tf.nn.relu(x)
x = self.regularization(x)
return self.linear_3(x)

mlp = SparseMLP()
y = mlp(tf.ones((10, 10)))

print(mlp.losses) # List containing one float32 scalar

8) These losses are cleared by the top-level layer at the start of each forward pass — they don’t accumulate. layer.losses always contain only the losses created during the last forward pass. You would typically use these losses by summing them before computing your gradients when writing a training loop.

# Losses correspond to the *last* forward pass.
mlp = SparseMLP()
mlp(tf.ones((10, 10)))
assert len(mlp.losses) == 1
mlp(tf.ones((10, 10)))
assert len(mlp.losses) == 1 # No accumulation.

# Let’s demonstrate how to use these losses in a training loop.

# Prepare a dataset.
(x_train, y_train), _ = tf.keras.datasets.mnist.load_data()
dataset = tf.data.Dataset.from_tensor_slices(
(x_train.reshape(60000, 784).astype(‚float32‘) / 255, y_train))
dataset = dataset.shuffle(buffer_size=1024).batch(64)

# A new MLP.
mlp = SparseMLP()

# Loss and optimizer.
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
optimizer = tf.keras.optimizers.SGD(learning_rate=1e-3)

for step, (x, y) in enumerate(dataset):
with tf.GradientTape() as tape:

# Forward pass.
logits = mlp(x)

# External loss value for this batch.
loss = loss_fn(y, logits)

# Add the losses created during the forward pass.
loss += sum(mlp.losses)

# Get gradients of weights wrt the loss.
gradients = tape.gradient(loss, mlp.trainable_weights)

# Update the weights of our linear layer.
optimizer.apply_gradients(zip(gradients, mlp.trainable_weights))

# Logging.
if step % 100 == 0:
print(step, float(loss))

9) Running eagerly is great for debugging, but you will get better performance by compiling your computation into static graphs. Static graphs are a researcher’s best friends. You can compile any function by wrapping it in a tf.function decorator.

# Prepare our layer, loss, and optimizer.
mlp = MLP()
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
optimizer = tf.keras.optimizers.SGD(learning_rate=1e-3)

# Create a training step function.

@tf.function # Make it fast.
def train_on_batch(x, y):
with tf.GradientTape() as tape:
logits = mlp(x)
loss = loss_fn(y, logits)
gradients = tape.gradient(loss, mlp.trainable_weights)
optimizer.apply_gradients(zip(gradients, mlp.trainable_weights))
return loss

# Prepare a dataset.
(x_train, y_train), _ = tf.keras.datasets.mnist.load_data()
dataset = tf.data.Dataset.from_tensor_slices(
(x_train.reshape(60000, 784).astype(‚float32‘) / 255, y_train))
dataset = dataset.shuffle(buffer_size=1024).batch(64)

for step, (x, y) in enumerate(dataset):
loss = train_on_batch(x, y)
if step % 100 == 0:
print(step, float(loss))