{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4NDd5Gr2s7eO"
      },
      "source": [
        "This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n",
        "\n",
        "**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n",
        "\n",
        "This notebook was generated for TensorFlow 2.6."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pHwcOO72s7eR"
      },
      "source": [
        "# Introduction to Keras and TensorFlow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uQ78X2F-s7eS"
      },
      "source": [
        "## What's TensorFlow?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Va091rY9s7eS"
      },
      "source": [
        "## What's Keras?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KW9AA9vYs7eS"
      },
      "source": [
        "## Keras and TensorFlow: A brief history"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OqQoZwZ4s7eT"
      },
      "source": [
        "## Setting up a deep-learning workspace"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z0T5uljTs7eT"
      },
      "source": [
        "### Jupyter notebooks: The preferred way to run deep-learning experiments"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DRscNGMHs7eT"
      },
      "source": [
        "### Using Colaboratory"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5Rl1VYmIs7eT"
      },
      "source": [
        "#### First steps with Colaboratory"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9U3fLEWIs7eU"
      },
      "source": [
        "#### Installing packages with pip"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cPSHQGNys7eU"
      },
      "source": [
        "#### Using the GPU runtime"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r1_N3QHfs7eU"
      },
      "source": [
        "## First steps with TensorFlow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v4mi07AQs7eU"
      },
      "source": [
        "#### Constant tensors and variables"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8cpIXK6Qs7eV"
      },
      "source": [
        "**All-ones or all-zeros tensors**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hBW3xV1zs7eV"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "x = tf.ones(shape=(2, 1))\n",
        "print(x)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jF4ohxVJs7eW"
      },
      "outputs": [],
      "source": [
        "x = tf.zeros(shape=(2, 1))\n",
        "print(x)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UMkc-7BSs7eW"
      },
      "source": [
        "**Random tensors**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XOYHWTKds7eW"
      },
      "outputs": [],
      "source": [
        "x = tf.random.normal(shape=(3, 1), mean=0., stddev=1.)\n",
        "print(x)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "S_BP-gvOs7eW"
      },
      "outputs": [],
      "source": [
        "x = tf.random.uniform(shape=(3, 1), minval=0., maxval=1.)\n",
        "print(x)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5-118060s7eW"
      },
      "source": [
        "**NumPy arrays are assignable**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iPKpDnN5s7eW"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "x = np.ones(shape=(2, 2))\n",
        "x[0, 0] = 0."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "voe723ySs7eX"
      },
      "source": [
        "**Creating a TensorFlow variable**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "at3qS9MZs7eX"
      },
      "outputs": [],
      "source": [
        "v = tf.Variable(initial_value=tf.random.normal(shape=(3, 1)))\n",
        "print(v)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "siscN826s7eX"
      },
      "source": [
        "**Assigning a value to a TensorFlow variable**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WKzAxJ2Zs7eX"
      },
      "outputs": [],
      "source": [
        "v.assign(tf.ones((3, 1)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aQk7Ti5Js7eX"
      },
      "source": [
        "**Assigning a value to a subset of a TensorFlow variable**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Bep_dVPbs7eX"
      },
      "outputs": [],
      "source": [
        "v[0, 0].assign(3.)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ZZT_7Nxs7eX"
      },
      "source": [
        "**Using `assign_add`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z0wO-z8rs7eY"
      },
      "outputs": [],
      "source": [
        "v.assign_add(tf.ones((3, 1)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_N86nv9Hs7eY"
      },
      "source": [
        "#### Tensor operations: Doing math in TensorFlow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1On8Gtzvs7eY"
      },
      "source": [
        "**A few basic math operations**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dQwAyyaWs7eY"
      },
      "outputs": [],
      "source": [
        "a = tf.ones((2, 2))\n",
        "b = tf.square(a)\n",
        "c = tf.sqrt(a)\n",
        "d = b + c\n",
        "e = tf.matmul(a, b)\n",
        "e *= d"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gn0l0_gUs7eY"
      },
      "source": [
        "#### A second look at the GradientTape API"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LeTjtaG7s7eY"
      },
      "source": [
        "**Using the `GradientTape`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5tluWG43s7eY"
      },
      "outputs": [],
      "source": [
        "input_var = tf.Variable(initial_value=3.)\n",
        "with tf.GradientTape() as tape:\n",
        "   result = tf.square(input_var)\n",
        "gradient = tape.gradient(result, input_var)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MEmkKFocs7eZ"
      },
      "source": [
        "**Using `GradientTape` with constant tensor inputs**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "crUdSiKDs7eZ"
      },
      "outputs": [],
      "source": [
        "input_const = tf.constant(3.)\n",
        "with tf.GradientTape() as tape:\n",
        "   tape.watch(input_const)\n",
        "   result = tf.square(input_const)\n",
        "gradient = tape.gradient(result, input_const)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-R5hs-Wls7eZ"
      },
      "source": [
        "**Using nested gradient tapes to compute second-order gradients**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "l2MvdqAos7eZ"
      },
      "outputs": [],
      "source": [
        "time = tf.Variable(0.)\n",
        "with tf.GradientTape() as outer_tape:\n",
        "    with tf.GradientTape() as inner_tape:\n",
        "        position =  4.9 * time ** 2\n",
        "    speed = inner_tape.gradient(position, time)\n",
        "acceleration = outer_tape.gradient(speed, time)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OJneaGjSs7eZ"
      },
      "source": [
        "#### An end-to-end example: A linear classifier in pure TensorFlow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bZTp7Fnns7eZ"
      },
      "source": [
        "**Generating two classes of random points in a 2D plane**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-bfakuc4s7eZ"
      },
      "outputs": [],
      "source": [
        "num_samples_per_class = 1000\n",
        "negative_samples = np.random.multivariate_normal(\n",
        "    mean=[0, 3],\n",
        "    cov=[[1, 0.5],[0.5, 1]],\n",
        "    size=num_samples_per_class)\n",
        "positive_samples = np.random.multivariate_normal(\n",
        "    mean=[3, 0],\n",
        "    cov=[[1, 0.5],[0.5, 1]],\n",
        "    size=num_samples_per_class)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LXS0J9XGs7ea"
      },
      "source": [
        "**Stacking the two classes into an array with shape (2000, 2)**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xozsa3YAs7ea"
      },
      "outputs": [],
      "source": [
        "inputs = np.vstack((negative_samples, positive_samples)).astype(np.float32)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gNprL5Cps7ea"
      },
      "source": [
        "**Generating the corresponding targets (0 and 1)**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7QAfaRMPs7ea"
      },
      "outputs": [],
      "source": [
        "targets = np.vstack((np.zeros((num_samples_per_class, 1), dtype=\"float32\"),\n",
        "                     np.ones((num_samples_per_class, 1), dtype=\"float32\")))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UxY8o8Ras7ea"
      },
      "source": [
        "**Plotting the two point classes**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "00XVjqAAs7ea"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.scatter(inputs[:, 0], inputs[:, 1], c=targets[:, 0])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_z2xILOms7ea"
      },
      "source": [
        "**Creating the linear classifier variables**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B2NqzfILs7ea"
      },
      "outputs": [],
      "source": [
        "input_dim = 2\n",
        "output_dim = 1\n",
        "W = tf.Variable(initial_value=tf.random.uniform(shape=(input_dim, output_dim)))\n",
        "b = tf.Variable(initial_value=tf.zeros(shape=(output_dim,)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w3D-a9bms7eb"
      },
      "source": [
        "**The forward pass function**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zloltDSos7eb"
      },
      "outputs": [],
      "source": [
        "def model(inputs):\n",
        "    return tf.matmul(inputs, W) + b"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ODdZOVzAs7eb"
      },
      "source": [
        "**The mean squared error loss function**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ciQn9WZ5s7eb"
      },
      "outputs": [],
      "source": [
        "def square_loss(targets, predictions):\n",
        "    per_sample_losses = tf.square(targets - predictions)\n",
        "    return tf.reduce_mean(per_sample_losses)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qAWfq0G6s7eb"
      },
      "source": [
        "**The training step function**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RAx5fRlss7eb"
      },
      "outputs": [],
      "source": [
        "learning_rate = 0.1\n",
        "\n",
        "def training_step(inputs, targets):\n",
        "    with tf.GradientTape() as tape:\n",
        "        predictions = model(inputs)\n",
        "        loss = square_loss(targets, predictions)\n",
        "    grad_loss_wrt_W, grad_loss_wrt_b = tape.gradient(loss, [W, b])\n",
        "    W.assign_sub(grad_loss_wrt_W * learning_rate)\n",
        "    b.assign_sub(grad_loss_wrt_b * learning_rate)\n",
        "    return loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A6fRbNdks7eb"
      },
      "source": [
        "**The batch training loop**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zbPFQ68As7eb"
      },
      "outputs": [],
      "source": [
        "for step in range(40):\n",
        "    loss = training_step(inputs, targets)\n",
        "    print(f\"Loss at step {step}: {loss:.4f}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xlLSq_0fs7ec"
      },
      "outputs": [],
      "source": [
        "predictions = model(inputs)\n",
        "plt.scatter(inputs[:, 0], inputs[:, 1], c=predictions[:, 0] > 0.5)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k0Hfx2fFs7ec"
      },
      "outputs": [],
      "source": [
        "x = np.linspace(-1, 4, 100)\n",
        "y = - W[0] /  W[1] * x + (0.5 - b) / W[1]\n",
        "plt.plot(x, y, \"-r\")\n",
        "plt.scatter(inputs[:, 0], inputs[:, 1], c=predictions[:, 0] > 0.5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "586ElIF8s7ec"
      },
      "source": [
        "## Anatomy of a neural network: Understanding core Keras APIs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eobbH-ups7ec"
      },
      "source": [
        "### Layers: The building blocks of deep learning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZRJutsrHs7ec"
      },
      "source": [
        "#### The base Layer class in Keras"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vg1giUoGs7ec"
      },
      "source": [
        "**A `Dense` layer implemented as a `Layer` subclass**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1Vltf78js7ec"
      },
      "outputs": [],
      "source": [
        "from tensorflow import keras\n",
        "\n",
        "class SimpleDense(keras.layers.Layer):\n",
        "\n",
        "    def __init__(self, units, activation=None):\n",
        "        super().__init__()\n",
        "        self.units = units\n",
        "        self.activation = activation\n",
        "\n",
        "    def build(self, input_shape):\n",
        "        input_dim = input_shape[-1]\n",
        "        self.W = self.add_weight(shape=(input_dim, self.units),\n",
        "                                 initializer=\"random_normal\")\n",
        "        self.b = self.add_weight(shape=(self.units,),\n",
        "                                 initializer=\"zeros\")\n",
        "\n",
        "    def call(self, inputs):\n",
        "        y = tf.matmul(inputs, self.W) + self.b\n",
        "        if self.activation is not None:\n",
        "            y = self.activation(y)\n",
        "        return y"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cT73TrKrs7ec"
      },
      "outputs": [],
      "source": [
        "my_dense = SimpleDense(units=32, activation=tf.nn.relu)\n",
        "input_tensor = tf.ones(shape=(2, 784))\n",
        "output_tensor = my_dense(input_tensor)\n",
        "print(output_tensor.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CiN-7t1cs7ed"
      },
      "source": [
        "#### Automatic shape inference: Building layers on the fly"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mjWzAsl2s7ed"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras import layers\n",
        "layer = layers.Dense(32, activation=\"relu\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OkuKxivds7ed"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras import models\n",
        "from tensorflow.keras import layers\n",
        "model = models.Sequential([\n",
        "    layers.Dense(32, activation=\"relu\"),\n",
        "    layers.Dense(32)\n",
        "])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IKWfNWnzs7ed"
      },
      "outputs": [],
      "source": [
        "model = keras.Sequential([\n",
        "    SimpleDense(32, activation=\"relu\"),\n",
        "    SimpleDense(64, activation=\"relu\"),\n",
        "    SimpleDense(32, activation=\"relu\"),\n",
        "    SimpleDense(10, activation=\"softmax\")\n",
        "])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZrzQRECxs7ed"
      },
      "source": [
        "### From layers to models"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8s_1bGbTs7ed"
      },
      "source": [
        "### The \"compile\" step: Configuring the learning process"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "O5d8U6Pos7ed"
      },
      "outputs": [],
      "source": [
        "model = keras.Sequential([keras.layers.Dense(1)])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"mean_squared_error\",\n",
        "              metrics=[\"accuracy\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pAgvlH54s7ed"
      },
      "outputs": [],
      "source": [
        "model.compile(optimizer=keras.optimizers.RMSprop(),\n",
        "              loss=keras.losses.MeanSquaredError(),\n",
        "              metrics=[keras.metrics.BinaryAccuracy()])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jtUgTvy_s7ed"
      },
      "source": [
        "### Picking a loss function"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nfIT68wrs7ee"
      },
      "source": [
        "### Understanding the fit() method"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Spamf-KCs7ee"
      },
      "source": [
        "**Calling `fit()` with NumPy data**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rCZbvEp3s7ee"
      },
      "outputs": [],
      "source": [
        "history = model.fit(\n",
        "    inputs,\n",
        "    targets,\n",
        "    epochs=5,\n",
        "    batch_size=128\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J_xCOjAus7ee"
      },
      "outputs": [],
      "source": [
        "history.history"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v3sAVaoMs7ee"
      },
      "source": [
        "### Monitoring loss and metrics on validation data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hhp_sRuYs7ee"
      },
      "source": [
        "**Using the `validation_data` argument**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UWCLS86ds7ee"
      },
      "outputs": [],
      "source": [
        "model = keras.Sequential([keras.layers.Dense(1)])\n",
        "model.compile(optimizer=keras.optimizers.RMSprop(learning_rate=0.1),\n",
        "              loss=keras.losses.MeanSquaredError(),\n",
        "              metrics=[keras.metrics.BinaryAccuracy()])\n",
        "\n",
        "indices_permutation = np.random.permutation(len(inputs))\n",
        "shuffled_inputs = inputs[indices_permutation]\n",
        "shuffled_targets = targets[indices_permutation]\n",
        "\n",
        "num_validation_samples = int(0.3 * len(inputs))\n",
        "val_inputs = shuffled_inputs[:num_validation_samples]\n",
        "val_targets = shuffled_targets[:num_validation_samples]\n",
        "training_inputs = shuffled_inputs[num_validation_samples:]\n",
        "training_targets = shuffled_targets[num_validation_samples:]\n",
        "model.fit(\n",
        "    training_inputs,\n",
        "    training_targets,\n",
        "    epochs=5,\n",
        "    batch_size=16,\n",
        "    validation_data=(val_inputs, val_targets)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6q3dszBys7ee"
      },
      "source": [
        "### Inference: Using a model after training"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qkLWlh02s7ee"
      },
      "outputs": [],
      "source": [
        "predictions = model.predict(val_inputs, batch_size=128)\n",
        "print(predictions[:10])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aP1VZIdMs7ef"
      },
      "source": [
        "## Summary"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "chapter03_introduction-to-keras-and-tf.i",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}