Learning Computer Vision Week 4

Neural Networks aren’t that complicated…until you get into backpropagation.

Ugh.

That’s then I forced myself to learn multivariable calculus, and I actually kinda succeeded in it (nah not really, I was just a high schooler, and still am a high schooler). But of course, because of all the cool computers and libraries we have now, we don’t have to worry about all that dirty calculus, hooray!

Tuesday, October 1st

Happy October!

Today I dove in to the differences between performances with the ReLU activation functions vs the sigmoid activation function.

I worked with some code examples that tested this comparison, using the same Fully Connected Neural Network structure:

Sigmoid

class Net(nn.Module):
    def __init__(self, D_in, H1, H2, D_out):
        super(Net, self).__init__()
        self.linear1 = nn.Linear(D_in, H1)
        self.linear2 = nn.Linear(H1, H2)
        self.linear3 = nn.Linear(H2, D_out)
    
    # Prediction
    def forward(self,x):
        x = torch.sigmoid(self.linear1(x)) 
        x = torch.sigmoid(self.linear2(x))
        x = self.linear3(x)
        return x

ReLU

class NetRelu(nn.Module):
   
    def __init__(self, D_in, H1, H2, D_out):
        super(NetRelu, self).__init__()
        self.linear1 = nn.Linear(D_in, H1)
        self.linear2 = nn.Linear(H1, H2)
        self.linear3 = nn.Linear(H2, D_out)
    
    # Prediction
    def forward(self, x):
        x = torch.relu(self.linear1(x))  
        x = torch.relu(self.linear2(x))
        x = self.linear3(x)
        return x

D_in

This is the input size of the first/input layer.

H1 and H2

These are the output size of the first layer/input size of the second layer, and output size of the second layer/input size of the third layer, respectively.

D_out

This is the output size of the third/output layer.

Wednesday, October 2nd

I finished off the lab today, here are some snippets:

def train(model, criterion, train_loader, validation_loader, optimizer, epochs=100):
    i = 0
    useful_stuff = {'training_loss': [], 'validation_accuracy': []}  
    # Number of times we train on the entire training dataset
    for epoch in range(epochs):
        for i, (x, y) in enumerate(train_loader):
            optimizer.zero_grad()
            z = model(x.view(-1, 28 * 28))
            loss = criterion(z, y)
            loss.backward()
            optimizer.step()
            # Saves the loss
            useful_stuff['training_loss'].append(loss.data.item())

# Later in the code...
training_results = train(Net(28 * 28, 50, 50, 10), nn.CrossEntropyLoss(), torch.utils.data.DataLoader(dataset=train_dataset, batch_size=2000, shuffle=True), torch.utils.data.DataLoader(dataset=validation_dataset, batch_size=5000, shuffle=False), torch.optim.SGD(model.parameters(), lr=learning_rate), epochs=10)

for i, (x, y) in enumerate(train_loader)

This iterates though each batch (a part of the whole input) in the train_loader, which in this context is an object of PyTorch’s DataLoader class.

optimizer.zero_grad()

Resets the calculated gradient value; it adds up if it’s not reset.

z = model(x.view(-1, 28 * 28))

Makes a prediction on the image tensor by flattening it to a 1 by 28*28 tensor.

loss = criterion(z, y)

Calculate the loss between the prediction and actual class.

loss.backward()

Calculates the gradient value with respect to each weight and bias.

optimizer.step()

Updates the weight and bias according to calculated gradient value.

Thursday, October 3rd

Today, I discovered the concept of Receptive Fields, which is the name for the size of a region in the input that produces a pixel value in the activation map.

Another concept that I learned but didn’t quite explain fully is flattening layers; it’s where the input data (i.e. image) is taken, and then is transformed into a 1D form (array/list) where the top row (y-axis = 0) is “taken off”, and then the second row is placed right after the first, and then the third row, and so on.

Friday, October 4th

Today, I looked at an example of some CNN code.

class CNN(nn.Module):
    
    # Contructor
    def __init__(self, out_1=16, out_2=32):
        super(CNN, self).__init__()
        # The reason we start with 1 channel is because we have a single black and white image
        # Channel Width after this layer is 16
        self.cnn1 = nn.Conv2d(in_channels=1, out_channels=out_1, kernel_size=5, padding=2)
        # Channel Wifth after this layer is 8
        self.maxpool1=nn.MaxPool2d(kernel_size=2)
        
        # Channel Width after this layer is 8
        self.cnn2 = nn.Conv2d(in_channels=out_1, out_channels=out_2, kernel_size=5, stride=1, padding=2)
        # Channel Width after this layer is 4
        self.maxpool2=nn.MaxPool2d(kernel_size=2)
        # In total we have out_2 (32) channels which are each 4 * 4 in size based on the width calculation above. Channels are squares.
        # The output is a value for each class
        self.fc1 = nn.Linear(out_2 * 4 * 4, 10)
    
    # Other functions such as forward and activations...

Saturday, October 5th

I was pretty busy today with a tournament and HoCo, so I didn’t do much today; that aside, I learned what common CNN architectures there were.

CNNs can have any amount of layers, with any type of layer, and with any amount of neurons for each layer, so how do researchers and machine learning engineers determine the best way to make CNN with all of this customization at their disposal? Well, that’s where CNN architectures come in; CNN architectures are…well…architectures or “models” of CNNs that have been tried and true in terms of consistently achieving a certain goal for a model, whether it be having a high accuracy or low training time.

LeNet-5

The most successful use case of the LeNet-5 is the MNIST Dataset of handwritten digits, where it’s best in using normally grayscale images: its first layer is a convolutional layer, where a 5 by 5 filter with a stride 1 and results in a volume of 28 by 28 outputs. The next layer is a pooling layer with 14 by 14 outputs. It repeats itself with a filter and pooling layer until it gets to the fully connected layers where it flattens to create 120 neurons and another with 84 neurons while using a sigmoid activation function to produce an output.

AlexNet

AlexNet, invented in 2012, has about a 63.3% accuracy, which was better than any other CNN architecture at the time, where it takes an input of a picture with 227 by 227 pixels, has a convolution layer with 25 channels sized at 11 by 11 pixels, and then has another convolution layer with 256 channels sized at 5 by 5 pixels, then a pooling layer, a convolution layer with 384 channels with size 3 by 3 pixels, a pooling layer, a convolution layer with 384 channels of size 3×3, a convolution layer with 256 channels with size 3×3, a pooling layer, two fully-connected layers with 4096 neurons back-to-back, a fully-connected layer with 100 neurons, and then finally, a softmax function.

VGGNet

VGGNet is a much deeper architecture/model compared to the ones I’ve discussed so far:

The main reason why there was such an increase in the number of layers in architectures such as VGGNet, was that by reducing the size of the filters for convolutional layers and instead adding more layers with smaller filters, it greatly reduced the amount of parameters that the model has to handle, while also reducing the number of operations and math that the model had to do.

ResNet

Now we’re are going into the deep learning side of artificial intelligence, where the layers become more and more numerous and complex: this where the vanishing gradient problem begins to show up. The vanishing gradient problem is where, in backpropagation (where the model “learns”), the calculus (more specifically the chain rule) behind the “learning” does a lot of math and has to multiply many small numbers—weights—together, resulting in even smaller gradients (numbers) where the model can’t tell much from those microscopic numbers; these small numbers are often the result of weights being raised to large powers e.g. 0.2 to the 12th power.

One of these fixes is using the ReLU (rectified linear unit) activation function in place of sigmoid and/or tanh, but another solution is to use ResNet: ResNet stands for “Residual Network”, where “residual layers” or “skip connections” allow the gradient to bypass/skip certain layers, so in backpropagation, the gradients have multiple “shortcut” ways of getting to the neurons they need to go to.

Resources

Course I followed: