Lab Six: CNNs

Prince Ndhlovu and Kirby Cravens

1. Data Preparation

In [1]:
import pandas as pd
import numpy as np
import seaborn as sns
%matplotlib inline 
from matplotlib import pyplot as plt
import os
import pickle
from PIL import Image
from matplotlib.pyplot import imshow
from IPython.display import display
import sklearn
from sklearn.metrics import roc_curve
from sklearn.metrics import auc
from sklearn.model_selection import train_test_split
from sklearn.model_selection import StratifiedShuffleSplit
from sklearn.model_selection import StratifiedKFold
import cv2
import random 
import keras
from keras import backend as K
from keras.utils import to_categorical
from keras.models import Sequential
from keras.layers import Reshape
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Conv2D, Conv3D, MaxPooling2D, MaxPooling3D
from keras.preprocessing.image import ImageDataGenerator
from keras.callbacks import EarlyStopping
from sklearn import metrics as mt
from keras.models import load_model
from tensorflow.keras.regularizers import l2
In [2]:
filepath = '/Users/princendhlovu/Downloads/brain_tumor_dataset/'

nontumor = os.listdir(filepath + 'no' )
tumor = os.listdir(filepath + 'yes' )
no_std_path = filepath + 'no/'
yes_std_path = filepath + 'yes/'

tumor_images = []
non_tumor_images = []
target = []

h,w = (80,75)
# read the images into the respective arrays
def addImageToArray(label,std_path,imageFilePath,imageArrays):
    for pic in imageFilePath:
        image_array = cv2.imread(std_path + pic, cv2.IMREAD_GRAYSCALE)
#         print(image_array.shape[0]/image_array.shape[1])
        image_array = cv2.resize(image_array, (w,h))
        imageArrays.append(image_array.flatten())
        if(label == 'yes'):
            target.append(1)
        if(label == 'no'):
            target.append(0)
In [3]:
non_tumor_images.clear()
tumor_images.clear()
target.clear()
addImageToArray('yes', yes_std_path,tumor,tumor_images)
addImageToArray('no', no_std_path,nontumor,non_tumor_images)

X_data = non_tumor_images + tumor_images

# print('frequencies',target.astype(np.int).value_counts())

X_data = np.array(X_data)
target = np.array(target)

rstate = np.random.get_state()
np.random.shuffle(X_data)
np.random.set_state(rstate)
np.random.shuffle(target)


print("Images readIn:", len(X_data))

unique, counts = np.unique(target, return_counts=True)

print(np.asarray((unique, counts)).T)
Images readIn: 253
[[  0  98]
 [  1 155]]
In [4]:
# Class distributions
labels_df = pd.DataFrame(data=target)
labels_df.hist()
plt.title("Class Distribution")
Out[4]:
Text(0.5, 1.0, 'Class Distribution')

Evaluating Performance

When radiologists are diagnosing patients who might have brain tumors, they have to minimize the risk of "missing" brain tumors which might give their clients false confidence causing them to abandon their medication and other necessary therapy sessions.In most cases if the tumor is unnoticed, it might be malignant and spread to other parts of the brain or body causing more damage. In this dataset were are trying to minimize the number of false negatives where the model declares an image tumor free when it is not as this is very detrimental to the present and future health or wellbeing of a person. We also do not want to diagnose people who are cancer free with tumors that they do not have, that is we want to minimize the number of false positives as they can cause healthy people to take medication that they do not need leading to health complications, substantial stress and even fatalities due to the irrelevant therapy and medication that they would be receiving. False positives are not desirable but the presence of false negatives has far wide reaching effects that cannot be mitigated and often times reduce the human lifespan by a considerable amount. Therefor were are going to use recall as an evaluation metric for our CNN model. Recall gives us a good picture of how good our model is at predicting true positives and reducing the number of false negatives and it is given by:
$$ \begin{align} Recall = \frac{True Positives}{True Positives + False Negatives} \end{align} $$

Splitting the Data

We had a small dataset of 253 images and we used image transformations to increase to 1253 images. This data expansion helps the model to experience wide variations of image orientations that would make it good at generalizing to new data that it has not seen before.It is unlikely that our model will see images or scans that have been taken in a certain orientation so we are going to divide our data using the StratifiedKFold with 5 folds. The images would be shuffled and and distributed among the 5 folds to expose our model. This would help reduce the variances during the classification process.Then we would evaluate our model with an 80 - 20 split of training and testing respectively.

2. Modeling

Data Expansion

In [5]:
#create new sample data
from keras.preprocessing.image import ImageDataGenerator

datagen = ImageDataGenerator(featurewise_center=False,
    samplewise_center=False,
    featurewise_std_normalization=False, #adding to each pixel
    samplewise_std_normalization=False,  #adding to each picture 
    zca_whitening=False,
    rotation_range=5, # used, Int. Degree range for random rotations.                        
    width_shift_range=0.1, # used, Float (fraction of total width). Range for random horizontal shifts.
    height_shift_range=0.1, # used,  Float (fraction of total height). Range for random vertical shifts.
    shear_range=0., # Float. Shear Intensity (Shear angle in counter-clockwise direction as radians)
    zoom_range=0.,
    channel_shift_range=0.,
    fill_mode='nearest',
    cval=0.,
    horizontal_flip=True,
    vertical_flip=True,
    rescale=None)

# X_data = X_data/255 - 0.5

#expand dimensions
X_data = np.expand_dims(X_data.reshape((-1,h, w)), axis=3)

datagen.fit(X_data)



new_images = datagen.flow(X_data, target, batch_size=1)
print('new images',len(new_images))
print('original data',len(X_data))
print(X_data.shape)

# for tmp in new_images:
#     imshow(tmp[0].squeeze(),cmap='bone')
#     break
new images 253
original data 253
(253, 80, 75, 1)
In [6]:
j = 0
for img in new_images:
    if j == 1000:
        break
    j += 1
    if j % 100 == 0:
        print(f'Appended {j} images')
    X_data = np.vstack((X_data,np.expand_dims(img[0][0].squeeze().reshape((-1,h,w)), axis=3)))
    target = np.append(target,img[1][0])
Appended 100 images
Appended 200 images
Appended 300 images
Appended 400 images
Appended 500 images
Appended 600 images
Appended 700 images
Appended 800 images
Appended 900 images
Appended 1000 images
In [7]:
classes = {0:"No Tumor", 1:"Tumor"}
# classes = {0:"Daisy", 1:"Dandelion", 2:"Rose", 3:"Sunflower", 4:"Tulip"}

# select random images to visualize

random.seed(1)

plt.figure(figsize=(2.0 * 7, 2.3 * 3))
plt.subplots_adjust(bottom=0, left=.01, right=.99, top=.90, hspace=.35)
random_sample = random.sample(range(0,X_data.shape[0]), k=21)
for n,i in enumerate(random_sample):
        plt.subplot(3,7, n + 1)
        imshow(X_data[i].reshape((h,w)))
        plt.title(classes[target[i]], size=12)
        plt.xticks(())
        plt.yticks(())
In [8]:
# Class distributions
labels_df = pd.DataFrame(data=target)
labels_df.hist()
plt.title("Class Distribution")
Out[8]:
Text(0.5, 1.0, 'Class Distribution')
In [9]:
unique, counts = np.unique(target, return_counts=True)

print(np.asarray((unique, counts)).T)
[[  0 486]
 [  1 767]]
In [76]:
#We could not use the recall from the keras model so, i used this one from:
#https://datascience.stackexchange.com/questions/45165/how-to-get-accuracy-f1-precision-and-recall-for-a-keras-model
    
def recall_m(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    recall = true_positives / (possible_positives + K.epsilon())
    return recall

ResNet

In [11]:
kfold_data = StratifiedKFold(n_splits=5, random_state=42, shuffle=True).split(X_data, target)
X_train, X_test, y_train, y_test = train_test_split(X_data, target, test_size= 0.20, random_state=42)

y_cat = to_categorical(target)
y_train_cat = to_categorical(y_train)
y_test_cat = to_categorical(y_test)

NUM_CLASSES = 2
In [14]:
%%time

# now lets use the LeNet architecture with batch norm
# We will also use ReLU where approriate and drop out 
from tensorflow.keras.layers import Add, Input
from tensorflow.keras.layers import average, concatenate
from tensorflow.keras.models import Model

l2_lambda = 0.000001
input_holder = Input(shape=(h, w, 1))

# start with a conv layer
x = Conv2D(filters=32,
               input_shape = (h,w,1),
               kernel_size=(3,3),
               kernel_initializer='he_uniform', 
               kernel_regularizer=l2(l2_lambda),
               padding='same', 
               activation='relu', 
               data_format="channels_last")(input_holder)

x = MaxPooling2D(pool_size=(2, 2), data_format="channels_last")(x)

x = Conv2D(filters=32,
               kernel_size=(3,3),
               kernel_initializer='he_uniform', 
               kernel_regularizer=l2(l2_lambda),
               padding='same', 
               activation='relu', 
               data_format="channels_last")(x)

x_split = MaxPooling2D(pool_size=(2, 2), data_format="channels_last")(x)

x = Conv2D(filters=64,
               kernel_size=(1,1),
               kernel_initializer='he_uniform', 
               kernel_regularizer=l2(l2_lambda),
               padding='same', 
               activation='relu', 
               data_format="channels_last")(x_split)

x = Conv2D(filters=64,
               kernel_size=(3,3),
               kernel_initializer='he_uniform', 
               kernel_regularizer=l2(l2_lambda),
               padding='same', 
               activation='relu', 
               data_format="channels_last")(x)

x = Conv2D(filters=32,
               kernel_size=(1,1),
               kernel_initializer='he_uniform', 
               kernel_regularizer=l2(l2_lambda),
               padding='same', 
               activation='relu', 
               data_format="channels_last")(x)

# now add back in the split layer, x_split (residual added in)
x = Add()([x, x_split])
x = Activation("relu")(x)

x = MaxPooling2D(pool_size=(2, 2), data_format="channels_last")(x)

x = Flatten()(x)
x = Dropout(0.25)(x)
x = Dense(256)(x)
x = Activation("relu")(x)
x = Dropout(0.5)(x)
x = Dense(NUM_CLASSES)(x)
x = Activation('softmax')(x)

resnet1 = Model(inputs=input_holder,outputs=x)

resnet1.summary()
Model: "functional_1"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_2 (InputLayer)            [(None, 80, 75, 1)]  0                                            
__________________________________________________________________________________________________
conv2d (Conv2D)                 (None, 80, 75, 32)   320         input_2[0][0]                    
__________________________________________________________________________________________________
max_pooling2d (MaxPooling2D)    (None, 40, 37, 32)   0           conv2d[0][0]                     
__________________________________________________________________________________________________
conv2d_1 (Conv2D)               (None, 40, 37, 32)   9248        max_pooling2d[0][0]              
__________________________________________________________________________________________________
max_pooling2d_1 (MaxPooling2D)  (None, 20, 18, 32)   0           conv2d_1[0][0]                   
__________________________________________________________________________________________________
conv2d_2 (Conv2D)               (None, 20, 18, 64)   2112        max_pooling2d_1[0][0]            
__________________________________________________________________________________________________
conv2d_3 (Conv2D)               (None, 20, 18, 64)   36928       conv2d_2[0][0]                   
__________________________________________________________________________________________________
conv2d_4 (Conv2D)               (None, 20, 18, 32)   2080        conv2d_3[0][0]                   
__________________________________________________________________________________________________
add (Add)                       (None, 20, 18, 32)   0           conv2d_4[0][0]                   
                                                                 max_pooling2d_1[0][0]            
__________________________________________________________________________________________________
activation (Activation)         (None, 20, 18, 32)   0           add[0][0]                        
__________________________________________________________________________________________________
max_pooling2d_2 (MaxPooling2D)  (None, 10, 9, 32)    0           activation[0][0]                 
__________________________________________________________________________________________________
flatten (Flatten)               (None, 2880)         0           max_pooling2d_2[0][0]            
__________________________________________________________________________________________________
dropout (Dropout)               (None, 2880)         0           flatten[0][0]                    
__________________________________________________________________________________________________
dense (Dense)                   (None, 256)          737536      dropout[0][0]                    
__________________________________________________________________________________________________
activation_1 (Activation)       (None, 256)          0           dense[0][0]                      
__________________________________________________________________________________________________
dropout_1 (Dropout)             (None, 256)          0           activation_1[0][0]               
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 2)            514         dropout_1[0][0]                  
__________________________________________________________________________________________________
activation_2 (Activation)       (None, 2)            0           dense_1[0][0]                    
==================================================================================================
Total params: 788,738
Trainable params: 788,738
Non-trainable params: 0
__________________________________________________________________________________________________
CPU times: user 108 ms, sys: 21.9 ms, total: 130 ms
Wall time: 147 ms
In [16]:
resnet1.compile(loss='categorical_crossentropy', # 'categorical_crossentropy' 'mean_squared_error'
                optimizer='adam', # 'adadelta' 'rmsprop'
                metrics=[recall_m])



cnn3 = []
for i, (train_data, test_data) in enumerate(kfold_data):
    res_history1 = resnet1.fit(X_data[train_data], y_cat[train_data], batch_size=16, 
                      epochs=50, verbose=1,
                      validation_data=(X_data[test_data],y_cat[test_data]),
                      callbacks=[EarlyStopping(monitor='val_loss', patience=4)]
                     )
    
    cnn3.append(res_history1)
Epoch 1/50
63/63 [==============================] - 5s 84ms/step - loss: 89.8077 - recall_m: 0.5665 - val_loss: 2.0339 - val_recall_m: 0.6665
Epoch 2/50
63/63 [==============================] - 5s 79ms/step - loss: 2.8412 - recall_m: 0.6157 - val_loss: 1.0910 - val_recall_m: 0.6087
Epoch 3/50
63/63 [==============================] - 5s 80ms/step - loss: 1.4096 - recall_m: 0.6065 - val_loss: 0.8228 - val_recall_m: 0.6158
Epoch 4/50
63/63 [==============================] - 5s 80ms/step - loss: 0.9073 - recall_m: 0.6504 - val_loss: 0.6466 - val_recall_m: 0.6822
Epoch 5/50
63/63 [==============================] - 5s 76ms/step - loss: 0.7325 - recall_m: 0.6857 - val_loss: 0.6555 - val_recall_m: 0.6626
Epoch 6/50
63/63 [==============================] - 5s 75ms/step - loss: 0.6024 - recall_m: 0.7141 - val_loss: 0.6513 - val_recall_m: 0.6683
Epoch 7/50
63/63 [==============================] - 5s 75ms/step - loss: 0.5913 - recall_m: 0.7046 - val_loss: 0.6348 - val_recall_m: 0.6683
Epoch 8/50
63/63 [==============================] - 5s 80ms/step - loss: 0.5647 - recall_m: 0.7143 - val_loss: 0.6567 - val_recall_m: 0.6314
Epoch 9/50
63/63 [==============================] - 5s 80ms/step - loss: 0.5567 - recall_m: 0.7373 - val_loss: 0.6325 - val_recall_m: 0.6644
Epoch 10/50
63/63 [==============================] - 6s 89ms/step - loss: 0.4544 - recall_m: 0.7897 - val_loss: 0.6407 - val_recall_m: 0.6548
Epoch 11/50
63/63 [==============================] - 5s 82ms/step - loss: 0.4732 - recall_m: 0.7651 - val_loss: 0.6490 - val_recall_m: 0.6431
Epoch 12/50
63/63 [==============================] - 5s 80ms/step - loss: 0.4502 - recall_m: 0.7895 - val_loss: 0.6590 - val_recall_m: 0.6605
Epoch 13/50
63/63 [==============================] - 5s 81ms/step - loss: 0.4156 - recall_m: 0.7901 - val_loss: 0.6534 - val_recall_m: 0.6566
Epoch 1/50
63/63 [==============================] - 5s 85ms/step - loss: 0.4905 - recall_m: 0.7921 - val_loss: 0.3456 - val_recall_m: 0.9006
Epoch 2/50
63/63 [==============================] - 5s 80ms/step - loss: 0.4598 - recall_m: 0.7786 - val_loss: 0.3799 - val_recall_m: 0.8672
Epoch 3/50
63/63 [==============================] - 5s 79ms/step - loss: 0.4480 - recall_m: 0.8020 - val_loss: 0.3585 - val_recall_m: 0.8516
Epoch 4/50
63/63 [==============================] - 5s 81ms/step - loss: 0.4119 - recall_m: 0.8073 - val_loss: 0.3942 - val_recall_m: 0.8594
Epoch 5/50
63/63 [==============================] - 5s 81ms/step - loss: 0.4080 - recall_m: 0.8141 - val_loss: 0.3713 - val_recall_m: 0.8498
Epoch 1/50
63/63 [==============================] - 5s 78ms/step - loss: 0.3957 - recall_m: 0.8216 - val_loss: 0.2349 - val_recall_m: 0.9297
Epoch 2/50
63/63 [==============================] - 5s 78ms/step - loss: 0.3954 - recall_m: 0.8125 - val_loss: 0.2467 - val_recall_m: 0.8961
Epoch 3/50
63/63 [==============================] - 5s 85ms/step - loss: 0.3412 - recall_m: 0.8473 - val_loss: 0.2727 - val_recall_m: 0.8945
Epoch 4/50
63/63 [==============================] - 5s 86ms/step - loss: 0.3114 - recall_m: 0.8572 - val_loss: 0.2629 - val_recall_m: 0.9000
Epoch 5/50
63/63 [==============================] - 5s 79ms/step - loss: 0.3350 - recall_m: 0.8507 - val_loss: 0.3087 - val_recall_m: 0.8766
Epoch 1/50
63/63 [==============================] - 5s 81ms/step - loss: 0.3572 - recall_m: 0.8379 - val_loss: 0.1978 - val_recall_m: 0.9453
Epoch 2/50
63/63 [==============================] - 5s 76ms/step - loss: 0.3370 - recall_m: 0.8622 - val_loss: 0.2103 - val_recall_m: 0.9469
Epoch 3/50
63/63 [==============================] - 5s 77ms/step - loss: 0.3218 - recall_m: 0.8383 - val_loss: 0.1983 - val_recall_m: 0.9430
Epoch 4/50
63/63 [==============================] - 5s 77ms/step - loss: 0.2790 - recall_m: 0.8776 - val_loss: 0.2168 - val_recall_m: 0.9273
Epoch 5/50
63/63 [==============================] - 5s 76ms/step - loss: 0.2707 - recall_m: 0.8860 - val_loss: 0.2226 - val_recall_m: 0.9258
In [18]:
%matplotlib inline

plt.figure(figsize=(10,4))
plt.subplot(2,2,1)
plt.plot(res_history1.history['recall_m'])

plt.ylabel('Recall %')
plt.title('Training')
plt.subplot(2,2,2)
plt.plot(res_history1.history['val_recall_m'])
plt.title('Validation')

plt.subplot(2,2,3)
plt.plot(res_history1.history['loss'])
plt.ylabel('Training Loss')
plt.xlabel('epochs')

plt.subplot(2,2,4)
plt.plot(res_history1.history['val_loss'])
plt.xlabel('epochs')
Out[18]:
Text(0.5, 0, 'epochs')

2nd Variation of ResNet

With the following implementation, we will be using nadam as the optimizer

In [29]:
kfold_data = StratifiedKFold(n_splits=5, random_state=42, shuffle=True).split(X_data, target)
X_train, X_test, y_train, y_test = train_test_split(X_data, target, test_size= 0.20, random_state=42)

y_cat = to_categorical(target)
y_train_cat = to_categorical(y_train)
y_test_cat = to_categorical(y_test)

NUM_CLASSES = 2
In [30]:
%%time

input_holder = Input(shape=(h, w, 1))

# start with a conv layer
x = Conv2D(filters=32,
               input_shape = (h,w,1),
               kernel_size=(3,3),
               kernel_initializer='he_uniform', 
               kernel_regularizer=l2(l2_lambda),
               padding='same', 
               activation='relu', 
               data_format="channels_last")(input_holder)

x = MaxPooling2D(pool_size=(2, 2), data_format="channels_last")(x)

x = Conv2D(filters=32,
               kernel_size=(3,3),
               kernel_initializer='he_uniform', 
               kernel_regularizer=l2(l2_lambda),
               padding='same', 
               activation='relu', 
               data_format="channels_last")(x)

x_split = MaxPooling2D(pool_size=(2, 2), data_format="channels_last")(x)

x = Conv2D(filters=64,
               kernel_size=(1,1),
               kernel_initializer='he_uniform', 
               kernel_regularizer=l2(l2_lambda),
               padding='same', 
               activation='relu', 
               data_format="channels_last")(x_split)

x = Conv2D(filters=64,
               kernel_size=(3,3),
               kernel_initializer='he_uniform', 
               kernel_regularizer=l2(l2_lambda),
               padding='same', 
               activation='relu', 
               data_format="channels_last")(x)

x = Conv2D(filters=32,
               kernel_size=(1,1),
               kernel_initializer='he_uniform', 
               kernel_regularizer=l2(l2_lambda),
               padding='same', 
               activation='relu', 
               data_format="channels_last")(x)

# now add back in the split layer, x_split (residual added in)
x = Add()([x, x_split])
x = Activation("relu")(x)

x = MaxPooling2D(pool_size=(2, 2), data_format="channels_last")(x)

x = Flatten()(x)
x = Dropout(0.25)(x)
x = Dense(256)(x)
x = Activation("relu")(x)
x = Dropout(0.5)(x)
x = Dense(NUM_CLASSES)(x)
x = Activation('softmax')(x)

resnet2 = Model(inputs=input_holder,outputs=x)
CPU times: user 87.3 ms, sys: 4.05 ms, total: 91.3 ms
Wall time: 92.4 ms
In [31]:
resnet2.compile(loss='categorical_crossentropy', # 'categorical_crossentropy' 'mean_squared_error'
                optimizer='nadam', # 'adadelta' 'rmsprop'
                metrics=[recall_m])



cnn5 = []
for i, (train_data, test_data) in enumerate(kfold_data):
    res_history2 = resnet2.fit(X_data[train_data], y_cat[train_data], batch_size=16, 
                      epochs=50, verbose=1,
                      validation_data=(X_data[test_data],y_cat[test_data]),
                      callbacks=[EarlyStopping(monitor='val_loss', patience=4)]
                     )
    
    cnn5.append(res_history2)
Epoch 1/50
63/63 [==============================] - 6s 97ms/step - loss: 52.2721 - recall_m: 0.5452 - val_loss: 1.3917 - val_recall_m: 0.5991
Epoch 2/50
63/63 [==============================] - 6s 94ms/step - loss: 1.1065 - recall_m: 0.6183 - val_loss: 0.7065 - val_recall_m: 0.5888
Epoch 3/50
63/63 [==============================] - 6s 99ms/step - loss: 0.8516 - recall_m: 0.6216 - val_loss: 0.6936 - val_recall_m: 0.6158
Epoch 4/50
63/63 [==============================] - 6s 93ms/step - loss: 0.6893 - recall_m: 0.6321 - val_loss: 0.6821 - val_recall_m: 0.6314
Epoch 5/50
63/63 [==============================] - 6s 93ms/step - loss: 0.6491 - recall_m: 0.6462 - val_loss: 0.8122 - val_recall_m: 0.6374
Epoch 6/50
63/63 [==============================] - 6s 90ms/step - loss: 0.6982 - recall_m: 0.6476 - val_loss: 0.6747 - val_recall_m: 0.6449
Epoch 7/50
63/63 [==============================] - 6s 89ms/step - loss: 0.6453 - recall_m: 0.6385 - val_loss: 0.6646 - val_recall_m: 0.6548
Epoch 8/50
63/63 [==============================] - 6s 89ms/step - loss: 0.6327 - recall_m: 0.6575 - val_loss: 0.6720 - val_recall_m: 0.6300
Epoch 9/50
63/63 [==============================] - 6s 91ms/step - loss: 0.6221 - recall_m: 0.6563 - val_loss: 0.7033 - val_recall_m: 0.6509
Epoch 10/50
63/63 [==============================] - 6s 91ms/step - loss: 0.6073 - recall_m: 0.6532 - val_loss: 0.7016 - val_recall_m: 0.6435
Epoch 11/50
63/63 [==============================] - 6s 91ms/step - loss: 0.6033 - recall_m: 0.6641 - val_loss: 0.6180 - val_recall_m: 0.6222
Epoch 12/50
63/63 [==============================] - 6s 91ms/step - loss: 0.6087 - recall_m: 0.6534 - val_loss: 0.6275 - val_recall_m: 0.6605
Epoch 13/50
63/63 [==============================] - 6s 93ms/step - loss: 0.5793 - recall_m: 0.6579 - val_loss: 0.6514 - val_recall_m: 0.6158
Epoch 14/50
63/63 [==============================] - 6s 90ms/step - loss: 0.5743 - recall_m: 0.6742 - val_loss: 0.6326 - val_recall_m: 0.6779
Epoch 15/50
63/63 [==============================] - 6s 90ms/step - loss: 0.6941 - recall_m: 0.6573 - val_loss: 0.6144 - val_recall_m: 0.6470
Epoch 16/50
63/63 [==============================] - 6s 90ms/step - loss: 0.5921 - recall_m: 0.6823 - val_loss: 0.7391 - val_recall_m: 0.6683
Epoch 17/50
63/63 [==============================] - 6s 90ms/step - loss: 0.5679 - recall_m: 0.6571 - val_loss: 0.6354 - val_recall_m: 0.6488
Epoch 18/50
63/63 [==============================] - 6s 92ms/step - loss: 0.5492 - recall_m: 0.6790 - val_loss: 0.6422 - val_recall_m: 0.6467
Epoch 19/50
63/63 [==============================] - 6s 90ms/step - loss: 0.5617 - recall_m: 0.6726 - val_loss: 0.6477 - val_recall_m: 0.6584
Epoch 1/50
63/63 [==============================] - 6s 91ms/step - loss: 0.6188 - recall_m: 0.6780 - val_loss: 0.5000 - val_recall_m: 0.7248
Epoch 2/50
63/63 [==============================] - 6s 92ms/step - loss: 0.5661 - recall_m: 0.6714 - val_loss: 0.5173 - val_recall_m: 0.7170
Epoch 3/50
63/63 [==============================] - 6s 91ms/step - loss: 0.5904 - recall_m: 0.6629 - val_loss: 0.5379 - val_recall_m: 0.7013
Epoch 4/50
63/63 [==============================] - 6s 93ms/step - loss: 0.5438 - recall_m: 0.6802 - val_loss: 0.5371 - val_recall_m: 0.7092
Epoch 5/50
63/63 [==============================] - 6s 93ms/step - loss: 0.6087 - recall_m: 0.6891 - val_loss: 0.5512 - val_recall_m: 0.6857
Epoch 1/50
63/63 [==============================] - 6s 93ms/step - loss: 0.6604 - recall_m: 0.6688 - val_loss: 0.5034 - val_recall_m: 0.7053
Epoch 2/50
63/63 [==============================] - 6s 93ms/step - loss: 0.5462 - recall_m: 0.6887 - val_loss: 0.4723 - val_recall_m: 0.7834
Epoch 3/50
63/63 [==============================] - 6s 91ms/step - loss: 0.5564 - recall_m: 0.6909 - val_loss: 0.4635 - val_recall_m: 0.7812
Epoch 4/50
63/63 [==============================] - 6s 94ms/step - loss: 0.5276 - recall_m: 0.7220 - val_loss: 0.4716 - val_recall_m: 0.7404
Epoch 5/50
63/63 [==============================] - 6s 92ms/step - loss: 0.5606 - recall_m: 0.6960 - val_loss: 0.4835 - val_recall_m: 0.7404
Epoch 6/50
63/63 [==============================] - 6s 99ms/step - loss: 0.5496 - recall_m: 0.6911 - val_loss: 0.4702 - val_recall_m: 0.7504
Epoch 7/50
63/63 [==============================] - 7s 106ms/step - loss: 0.5238 - recall_m: 0.7298 - val_loss: 0.4694 - val_recall_m: 0.7638
Epoch 1/50
63/63 [==============================] - 7s 103ms/step - loss: 0.5504 - recall_m: 0.7254 - val_loss: 0.4318 - val_recall_m: 0.7703
Epoch 2/50
63/63 [==============================] - 6s 96ms/step - loss: 0.5219 - recall_m: 0.7091 - val_loss: 0.4138 - val_recall_m: 0.7680
Epoch 3/50
63/63 [==============================] - 7s 104ms/step - loss: 0.5186 - recall_m: 0.6981 - val_loss: 0.4013 - val_recall_m: 0.8133
Epoch 4/50
63/63 [==============================] - 6s 98ms/step - loss: 0.5168 - recall_m: 0.7119 - val_loss: 0.4210 - val_recall_m: 0.7922
Epoch 5/50
63/63 [==============================] - 7s 109ms/step - loss: 0.4958 - recall_m: 0.7200 - val_loss: 0.4032 - val_recall_m: 0.7797
Epoch 6/50
63/63 [==============================] - 7s 111ms/step - loss: 0.4887 - recall_m: 0.7362 - val_loss: 0.4129 - val_recall_m: 0.7781
Epoch 7/50
63/63 [==============================] - 7s 104ms/step - loss: 0.4723 - recall_m: 0.7319 - val_loss: 0.4570 - val_recall_m: 0.7625
Epoch 1/50
63/63 [==============================] - 6s 94ms/step - loss: 0.4925 - recall_m: 0.7416 - val_loss: 0.3853 - val_recall_m: 0.8453
Epoch 2/50
63/63 [==============================] - 6s 92ms/step - loss: 0.4744 - recall_m: 0.7328 - val_loss: 0.3715 - val_recall_m: 0.8359
Epoch 3/50
63/63 [==============================] - 6s 93ms/step - loss: 0.4591 - recall_m: 0.7561 - val_loss: 0.3440 - val_recall_m: 0.8430
Epoch 4/50
63/63 [==============================] - 6s 94ms/step - loss: 0.4628 - recall_m: 0.7462 - val_loss: 0.3905 - val_recall_m: 0.7977
Epoch 5/50
63/63 [==============================] - 6s 98ms/step - loss: 0.4528 - recall_m: 0.7461 - val_loss: 0.3628 - val_recall_m: 0.8258
Epoch 6/50
63/63 [==============================] - 6s 92ms/step - loss: 0.4449 - recall_m: 0.7471 - val_loss: 0.4018 - val_recall_m: 0.8000
Epoch 7/50
63/63 [==============================] - 6s 94ms/step - loss: 0.4233 - recall_m: 0.7685 - val_loss: 0.3763 - val_recall_m: 0.8477
In [33]:
plt.figure(figsize=(10,4))
plt.subplot(2,2,1)
plt.plot(res_history2.history['recall_m'])

plt.ylabel('Recall %')
plt.title('Training')
plt.subplot(2,2,2)
plt.plot(res_history2.history['val_recall_m'])
plt.title('Validation')

plt.subplot(2,2,3)
plt.plot(res_history2.history['loss'])
plt.ylabel('Training Loss')
plt.xlabel('epochs')

plt.subplot(2,2,4)
plt.plot(res_history2.history['val_loss'])
plt.xlabel('epochs')
Out[33]:
Text(0.5, 0, 'epochs')

Comparing between the two variations of ResNet Architecture

In [85]:
'First ResNet Architecture AUC & ROC'
y_pred_resnet = np.argmax(resnet1.predict(X_test), axis=1)
# np.argmax(alexnet.predict(X_test), axis=1)

#false positive and true positive rates using roc
fpr_resnet, tpr_resnet, thresholds_resnet = mt.roc_curve(np.argmax(y_test_cat, axis=1), y_pred_resnet)

#area under the curve
auc_resnet = auc(fpr_resnet, tpr_resnet)

'Second ResNet Architecture AUC & ROC'
y_pred_resnet2 = np.argmax(resnet2.predict(X_test), axis=1)

#false positive and true positive rates using roc
fpr_resnet1, tpr_resnet1, thresholds_resnet1 = mt.roc_curve(np.argmax(y_test_cat, axis=1), y_pred_resnet2)

#area under the curve
auc_resnet1 = auc(fpr_resnet1, tpr_resnet1)
In [35]:
plt.figure(figsize=(12,12))

#plot halfway line
plt.plot([0,1], [0,1], 'k--')

#plot for CNN
plt.plot(fpr_resnet, tpr_resnet,label='ResNet1  (area = {:.3f})'.format(auc_resnet))

#plot for CNN1
plt.plot(fpr_resnet1, tpr_resnet1,label='ResNet2  (area = {:.3f})'.format(auc_resnet1))

plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Resnet1 vs Resnet2')
plt.legend(loc='best')
plt.show()