Visualising MLP Data Transformation
I was trying to explain how a neural learns to perform certain tasks by performing a series of transformations so decided to put a little notebook together to visualise this process.
from sklearn.datasets import make_moons, make_circles
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
%matplotlib inline
X,y = make_circles(n_samples=100000, noise=0.2, factor=0.5, random_state=1)
cm = plt.cm.RdBu
cm_bright = ListedColormap(['#FF0000', '#0000FF'])
ax = plt.subplot()
ax.scatter(X[:100, 0], X[:100, 1], c=y[:100], cmap=cm_bright,edgecolors='k')
import tensorflow as tf
from tensorflow import keras
Use Tensorflow sequentail API to build the neural network. It is worth experimenting here and working out what is the minimum number of dimensions required in the hidden layer to be able to separate these data. I have added an extra linear layer with 2 nodes to visualise how the input datapoints have been transformed. This extra layer does not really have any impact on the neural network performance.
model = keras.models.Sequential()
model.add(keras.layers.InputLayer(2))
model.add(keras.layers.Dense(6,activation="relu"))
model.add(keras.layers.Dense(2,activation="linear"))
model.add(keras.layers.Dense(1,activation="sigmoid"))
Compile the model using binary cross entropy loss because it is a two class problem.
model.compile(
optimizer=keras.optimizers.RMSprop(), # Optimizer
# Loss function to minimize
loss=keras.losses.BinaryCrossentropy(),
# List of metrics to monitor
metrics=[keras.metrics.BinaryAccuracy()],
)
print("Fit model on training data")
history = model.fit(
X,
y,
batch_size=32,
epochs=5,
validation_split=0.2,
)
X_test,y_test = make_circles(n_samples=100, noise=0.2, factor=0.5, random_state=1)
Keras provides an API for getting individual layer outputs for feature inspections
extractor = keras.Model(inputs=model.inputs,
outputs=[layer.output for layer in model.layers])
features = extractor(X_test)
# print(features[0].shape, features[1].shape, features[2].shape)
Visualise the hidden layer that's doing all the work.
# fig = plt.figure()
# ax = fig.add_subplot(111, projection='3d')
# ax.scatter(features[0][:, 0], features[0][:, 1], features[0][:,2], cmap=cm_bright,edgecolors='k')
fig = plt.figure(figsize=(14,8))
ax1 = fig.add_subplot(131)
ax1.scatter(features[0][:,0], features[0][:,1], c=y_test, cmap=cm_bright, edgecolors='k')
ax2 = fig.add_subplot(132)
ax2.scatter(features[0][:,2], features[0][:,3], c=y_test, cmap=cm_bright, edgecolors='k')
ax3 = fig.add_subplot(133)
ax3.scatter(features[0][:,0], features[0][:,5], c=y_test, cmap=cm_bright, edgecolors='k')
Visualise the linear layer to see how the data have been transformed for classification
fig = plt.figure(figsize=(14,8))
ax = fig.add_subplot(111)
ax.scatter(features[1][:, 0], features[1][:, 1], c=y_test, cmap=cm_bright,edgecolors='k')
We can see the data have been transformed such that they are easily separable for binary classification
from scipy.interpolate import CubicSpline
import numpy as np
x_ = np.vstack((X_test[:,0], features[0].numpy()[:,0], features[1].numpy()[:,0]))
print(x_.shape)
# for i in range(len(X_test)):
# print(X_test[i,0], features[0].numpy()[i,0], features[1].numpy()[i,0])
# x_ = np.concatenate(X_test[i,0], features[0].numpy()[i,0], features[1].numpy()[i,0])
# print(x_)