Deep Learning With Python

Intro to Theano

Theano is a Python library for fast numerical computation to aid in the development of deep learning models. At its heart, Theano is a compiler for mathematical expressions in Python. It knows how to take your structures and turn them into very efficient code that uses NumPy and efficient native libraries to run as fast as possible on CPUs or GPUs. The actual syntax of Theano expressions is symbolic, which can be off-putting to beginners used to normal software development. Specifically, expression are defined in the abstract sense, compiled and later actually used to make calculations. In this lesson, your goal is to install Theano and write a small example that demonstrates the symbolic nature of Theano programs. For example, you can install Theano using pip as follows:

sudo pip install Theano

A small example of a Theano program that you can use as a starting point is listed below:

import theano
from theano import tensor

# declare two symbolic floating-point scalars
a = tensor.dscalar()
b = tensor.dscalar()

# create a simple expression
c = a + b

# convert the expression into a callable object that takes (a,b)
# values as input and computes a value for c
f = theano.function([a,b], c)

# bind 1.5 to 'a', 2.5 to 'b', and evaluate 'c'
assert 4.0 == f(1.5, 2.5)

Learn more about Theano on the Theano homepage.

Intro to TensorFlow

TensorFlow is a Python library for fast numerical computing created and released by Google. Like Theano, TensorFlow is intended to be used to develop deep learning models. With the backing of Google, perhaps used in some of its production systems and used by the Google DeepMind research group, it is a platform that we cannot ignore. Unlike Theano, TensorFlow does have more of a production focus with a capability to run on CPUs, GPUs and even very large clusters. In this lesson, your goal is to install TensorFlow become familiar with the syntax of the symbolic expressions used in TensorFlow programs. For example, you can install TensorFlow using pip. There are many different versions of TensorFlow, specialized for each platform. Select the right version for your platform on the TensorFlow installation webpage. A small example of a TensorFlow program that you can use as a starting point is listed below:

import tensorflow as tf

# declare two symbolic floating-point scalars
a = tf.placeholder(tf.float32)
b = tf.placeholder(tf.float32)

# create a simple symbolic expression using the add function
add = tf.add(a, b)

# bind 1.5 to ' a ' , 2.5 to ' b ' , and evaluate ' c '
sess = tf.Session()
binding = {a: 1.5, b: 2.5}
c = sess.run(add, feed_dict=binding)
print(c)

Learn more about TensorFlow on the TensorFlow homepage.

Intro to Keras

A difficulty of both Theano and TensorFlow is that it can take a lot of code to create even very simple neural network models. These libraries were designed primarily as a platform for research and development more than for the practical concerns of applied deep learning. The Keras library addresses these concerns by providing a wrapper for both Theano and TensorFlow. It provides a clean and simple API that allows you to define and evaluate deep learning models in just a few lines of code. Because of the ease of use and because it leverages the power of Theano and TensorFlow, Keras is quickly becoming the go-to library for applied deep learning. The focus of Keras is the concept of a model. The life-cycle of a model can be summarized as follows:

1. Define your model. Create a Sequential model and add configured layers.

2. Compile your model. Specify loss function and optimizers and call the compile() function on the model.

3. Fit your model. Train the model on a sample of data by calling the fit() function on the model.

4. Make predictions. Use the model to generate predictions on new data by calling functions such as evaluate() or predict() on the model.

Your goal for this lesson is to install Keras. For example, you can install Keras using pip:

sudo pip install keras

Start to familiarize yourself with the Keras library ready for the upcoming lessons where we will implement our first model. You can learn more about the Keras library on the Keras homepage.

Crash Course in Multi-Layer Perceptrons

Artificial neural networks are a fascinating area of study, although they can be intimidating when just getting started. The field of artificial neural networks is often just called neural networks or multi-layer Perceptrons after perhaps the most useful type of neural network. The building block for neural networks are artificial neurons. These are simple computational units that have weighted input signals and produce an output signal using an activation function. Neurons are arranged into networks of neurons. A row of neurons is called a layer and one network can have multiple layers. The architecture of the neurons in the network is often called the network topology. Once configured, the neural network needs to be trained on your dataset. The classical and still preferred training algorithm for neural networks is called stochastic gradient descent. Your goal for this lesson is to become familiar with neural network terminology. Dig a little deeper into terms like neuron, weights, activation function, learning rate and more.

Develop Your First Neural Network in Keras

Keras allows you to develop and evaluate deep learning models in very few lines of code. In this lesson, your goal is to develop your first neural network using the Keras library. Use a standard binary (two-class) classification dataset from the UCI Machine Learning Repository, like the Pima Indians onset of diabetes or the ionosphere datasets. Piece together code to achieve the following:

1. Load your dataset using NumPy or Pandas.

2. Define your neural network model and compile it.

3. Fit your model to the dataset.

4. Estimate the performance of your model on unseen data.

To give you a massive head-start below is a complete working example that you can use as a starting point. It assumes that you have downloaded the Pima Indians dataset to your current working directory with the filename pima-indians-diabetes.csv.

# Create first network with Keras
from keras.models import Sequential
from keras.layers import Dense
import numpy

# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)

# load pima indians dataset
dataset = numpy.loadtxt("pima-indians-diabetes.csv", delimiter=",")
# split into input (X) and output (Y) variables
X = dataset[:,0:8]
Y = dataset[:,8]

# create model
model = Sequential()
model.add(Dense(12, input_dim=8, kernel_initializer='uniform', activation='relu'))
model.add(Dense(8, kernel_initializer='uniform', activation='relu'))
model.add(Dense(1, kernel_initializer='uniform', activation='sigmoid'))

# Compile model
model.compile(loss='binary_crossentropy' , optimizer='adam', metrics=['accuracy'])

# Fit the model
model.fit(X, Y, epochs=150, batch_size=10)

# evaluate the model
scores = model.evaluate(X, Y)
print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

Now develop your own model on a different dataset, or adapt this example. Learn more about the Keras API for simple model development.

Use Keras Models With Scikit-Learn

The scikit-learn library is a general purpose machine learning framework in Python built on top of SciPy. Scikit-learn excels at tasks such as evaluating model performance and optimizing model hyperparameters in just a few lines of code. Keras provides a wrapper class that allows you to use your deep learning models with scikit-learn. For example, an instance of KerasClassifier class in Keras can wrap your deep learning model and be used as an Estimator in scikit-learn. When using the KerasClassifier class, you must specify the name of a function that the class can use to define and compile your model. You can also pass additional parameters to the constructor of the KerasClassifier class that will be passed to the model.fit() call later, like the number of epochs and batch size. In this lesson, your goal is to develop a deep learning model and evaluate it using k-fold cross validation. For example, you can define an instance of the KerasClassifier and the custom function to create your model as follows:

# Function to create model, required for KerasClassifier
def create_model():
    # Create model
    model = Sequential()
    ...
    # Compile model
    model.compile(...)
    return model

# create classifier for use in scikit-learn
model = KerasClassifier(build_fn=create_model, epochs=150, batch_size=10)

# evaluate model using 10-fold cross validation in scikit-learn
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed)
results = cross_val_score(model, X, Y, cv=kfold)

Learn more about using your Keras deep learning models with scikit-learn on the Wrappers for the Sciki-Learn API webpage.

Plot Model Training History.

You can learn a lot about neural networks and deep learning models by observing their performance over time during training. Keras provides the capability to register callbacks when training a deep learning model. One of the default callbacks that is registered when training all deep learning models is the History callback. It records training metrics for each epoch. This includes the loss and the accuracy (for classification problems) as well as the loss and accuracy for the validation dataset if one is set. The history object is returned from calls to the fit() function used to train the model. Metrics are stored in a dictionary in the history member of the object returned. Your goal for this lesson is to investigate the history object and create plots of model performance during training. For example, you can print the list of metrics collected by your history object as follows:

# list all data in history
history = model.fit(...)
print(history.history.keys())

You can learn more about the History object and the callback API in Keras. Well done, you made it to the half-way point.

Save Your Best Model During Training With Checkpointing.

Reduce Overfitting With Dropout Regularization.

Lift Performance With Learning Rate Schedules.

Crash Course in Convolutional Neural Networks.

Handwritten Digit Recognition.

Object Recognition in Small Photographs.

Improve Generalization With Data Augmentation.