First Neural Network with Keras

Lately, I have been on a DataCamp spree after unlocking a two-month free unlimited trial through Microsoft’s Visual Studio Dev Essentials program. If you haven’t already, make sure to check it out, as it offers a plethora of tools, journal subscriptions, and software packages for developers. Anyhow, one of the courses I decided to check out on DataCamp was titled “Introduction to Deep Learning with Python,” which covered basic concepts in deep learning such as forward and backward propagation. The latter half of the tutorial was devoted to the introduction of the Keras API and the implementation of neural networks. I created this notebook immediately after finishing the tutorial for memory retention and self-review purposes.

First, we begin by importing the keras library as well as other affiliated functions in the module. Note that Keras uses TensorFlow as backend by default. The warning in the code block below appears because this notebook was written on Google Colab, which informs users that the platform will be switching over to TensorFlow 2 in the future.

import keras
from keras.datasets import mnist
from keras.layers import Dense, Dropout
from keras.models import Sequential
from keras.callbacks import EarlyStopping

As you might be able to guess from one of the imported modules, the objective of the neural network will be to classify hand-written digits. In doing so, we will be dealing with a classic in machine learning literature known as the the MNIST data set, which contains images of hand-written digits from 0 to 9, each hand-labeled by researchers. The num_class variable denotes the total number of class labels available in the classification task, which is 10. epochs specifies the number of iterations the gradient descent algorithm will run for.

num_class = 10
epochs = 15

Let’s begin by loading the data from keras.datasets.

(X_train, y_train), (X_test, y_test) = mnist.load_data()

Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz
11493376/11490434 [==============================] - 1s 0us/step

Now we have to slightly modify the loaded data so that its dimensions and values are made suitable to be fed into a neural network. Changing the dimensionality of data can be achieved through the reshape function, which takes in the number of rows and columns as its argument. We convert the numbes into type float32, then normalize it so that its values are all between 0 and 1. Although we won’t get into too much detail as to why normalization is important, an elementary intuition we might develop is that normalization effectively squishes all values into the same bound, making data much more processable. We also implement one-hot encoding on y through the keras.utils.to_categorical function.

X_train = X_train.reshape(60000, 784)
X_test = X_test.reshape(10000, 784)
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
y_train = keras.utils.to_categorical(y_train, num_class)
y_test = keras.utils.to_categorical(y_test, num_class)

Let’s quickly check if the necessary adjustments were made by looking up the dimensions of X_train and y_train, respectively.

X_train.shape

(60000, 784)

y_train.shape

(60000, 10)

Looks like the data has been reshaped successfully. Now, it’s finally time to get into the nuts and bolts of a neural network. The simplest neural network is the Sequential model, which means that every neuron in one layer is connected to all other neurons in the previous layer. Building a simple neural network is extremely easy in a high level API like Keras. The model below has 784 input nodes. The input layer is then connected to a hidden layer with 512 neurons, which is then connected to a second hidden layer with also 512 neurons. Note that the hidden layer uses the relu function as its activation function. The dropout layers ensure that our model does not overfit to the data. The last output layer has 10 neurons, each corresponding to digits from 0 to 9. The activation fuction of this last layer is the softmax function, which allows us to interpret the final results as a categorical distribution.

model = Sequential()
model.add(Dense(512, activation='relu', input_shape=(784,)))
model.add(Dropout(0.2))
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(num_class, activation='softmax'))

Let’s double check that the layers have been formed correctly as per our intended design.

model.summary()

Model: "sequential_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_6 (Dense)              (None, 512)               401920    
_________________________________________________________________
dropout_5 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_7 (Dense)              (None, 512)               262656    
_________________________________________________________________
dropout_6 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_8 (Dense)              (None, 10)                5130      
=================================================================
Total params: 669,706
Trainable params: 669,706
Non-trainable params: 0
_________________________________________________________________

Everything looks good, which means we are now ready to compile and train our model. Before we do that, however, it is always a good idea to use the EarlyStopping module to ensure that gradient descent stops when no substantial weight adjustments are being made to our model. In other words, when the model successfully finds the local minimum (or preferably the global minimum), the early_stopping_monitor will kick in and stop gradient descent from proceeding with further epochs.

early_stopping_monitor = EarlyStopping(patience=2)

We are now ready to go! Let’s compile the model by making some configurations, namely the optimizer, loss, and metrics. Simply put, an optimizer specifies which flavor of the gradient descent algorithm we want to choose. The simplest version is known as sgd, or the stochastic gradient descent. adam can be considered an improved version of the stochastic gradient descent in that its learning rate changes depending on the slope of the loss function, defined here as cross entropy. If you recall, cross entropy is basically a measurement of the pseudo-distance between two distributions, i.e. how different two distributions are. But because cross entropy is often not easy to intuitively wrap our minds around, let’s pass the accuracy metric to the compile function, as shown below.

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

It’s time to train the neural network with the training data, X_train and y_train, over a specified number of epochs. As promised, we will use the early_stopping_monitor to stop graident descent from making unnecessary computations down the road. We also specify that X_test and y_test are components of the validation set.

history = model.fit(X_train, y_train, epochs=epochs, callbacks=[early_stopping_monitor], validation_data=(X_test, y_test))

Train on 60000 samples, validate on 10000 samples
Epoch 1/15
60000/60000 [==============================] - 8s 129us/step - loss: 0.2149 - acc: 0.9349 - val_loss: 0.0992 - val_acc: 0.9688
Epoch 2/15
60000/60000 [==============================] - 7s 124us/step - loss: 0.1048 - acc: 0.9676 - val_loss: 0.0815 - val_acc: 0.9750
Epoch 3/15
60000/60000 [==============================] - 7s 122us/step - loss: 0.0820 - acc: 0.9743 - val_loss: 0.0907 - val_acc: 0.9726
Epoch 4/15
60000/60000 [==============================] - 7s 123us/step - loss: 0.0669 - acc: 0.9794 - val_loss: 0.0795 - val_acc: 0.9782
Epoch 5/15
60000/60000 [==============================] - 7s 122us/step - loss: 0.0576 - acc: 0.9823 - val_loss: 0.0883 - val_acc: 0.9767
Epoch 6/15
60000/60000 [==============================] - 7s 123us/step - loss: 0.0498 - acc: 0.9843 - val_loss: 0.0704 - val_acc: 0.9799
Epoch 7/15
60000/60000 [==============================] - 7s 123us/step - loss: 0.0470 - acc: 0.9851 - val_loss: 0.0752 - val_acc: 0.9814
Epoch 8/15
60000/60000 [==============================] - 7s 122us/step - loss: 0.0425 - acc: 0.9867 - val_loss: 0.0857 - val_acc: 0.9800

Keras shows us how much our neural network improves over each epoch. This is convenient, but can we do better? The answer is a sure yes. Let’s quickly plot a graph to see how model accuracy improves over time, while cross entropy loss decreases with more epochs.

%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
plt.style.use('seaborn')

fig = plt.figure()
plt.subplot(2,1,1)
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('Model Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'])

plt.subplot(2,1,2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Cross Entropy Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'])

plt.tight_layout()

As the last step, we might want to save our trained model. This can be achieved with a single line of code.

model.save('path_to_my_model.h5')

We can load pre-saved models as well.

new_model = keras.models.load_model('path_to_my_model.h5')

That’s it for today! Obviously there are a lot more we can do with keras, such as building deeper neural networks or non-sequential models such as CNN or GAN, but these are topics we might look at a later date when I grow more proficient with the Keras API and deep learning in general. For now, consider this to be a gentle introduction to neural networks with Keras.

Thanks for reading! Catch you up in the next one.