robin's secret blog

Introduction

Regularization is a weird thing. You’d think randomly killing node outputs would make a model less inteligent, but somehow it’s a viable strategy for improving performance on unseen data.

These are the notes for day 8 of my AI/ML grind. Today’s topic: regularization.

You can also checkout these notes using my ai-ml-diary repository.

Regularization

Regularization techniques penalize memorization (over-learning on examples) and help models generalize to unseen data. These techniques:

can increase or decrease training time;
decrease training accuracy but increase generalization;
work better for larger models with multiple hidden layers;
generally work better with large amounts of data.

Regularization techniques work by adding penalizations, constraints or modifications to the training process. There are many ways to regularize a model.

Node regularization

For node regularization, there is the dropout method. It works by randomly removing nodes while training a model. This:

prevents a single node from learning too much;
forces the model to have distributed representations;
makes the model less reliant on individual nodes and thus more stable.

Implementing this technique implies pretty minor modifications to the standard Neural Network architecture I’ve been using. Pytorch’s built-in dropout function can be easily added into an existing forward() method.

class DropoutModel(nn.Module):
    def forward(self, x):
        x = self.layers['input'](x)
        x = F.relu(x)
        x = F.dropout(x, p=self.dropout_rate)

        for i in range(self.n_layers):
            x = self.layers[f'hidden{i}'](x)
            x = F.relu(x)
            x = F.dropout(x, p=self.dropout_rate)

        x = self.layers['output'](x)
        return x

Loss regularization

For loss regularization, there are the L1 and L2 regularization methods. Both follow the same principle, which is that of adding extra penalties to the loss function.

Regular cost function: $$ 𝓙 = \frac{1}{n} \sum_{i=1}^n{𝓛(y_{prediction_i}, y_{expected_i})} $$

L1/L2 regularization cost function: $$ 𝓙 = \frac{1}{n} \sum_{i=1}^n{𝓛(y_{prediction_i}, y_{expected_i}) + λ * penalty(w)} $$

This techniques leads to lower weight values, which allows model learning to be smoother and more consistent.

Higher weight values are less desirable because they cause similar inputs to produce very different outputs, and are usually a result of models being overfit for their training sets.

The λ value controls the strength of the penalty. The equation for the penalty depends on which regularization technique we are using (L1 or L2).

L2 Regularization $$ penalty_{L2} = ||w||^{2}_{2} = \sum_{i}{w^{2}_{i}} $$

In L2 regularization, the penalty is calculated using the sum of all squared weights. This encorages small weights without driving values to zero. This technique is useful for generating denser solutions (denser, here, means that every node has a non-zero weight).

Implementing L2 regularization in Pytorch is pretty simple, there’s a parameter for it in the optimizer I’ve been using to far: weight_decay. This value sets the λ variable for the regularization.

class L2Model(nn.Module):
    def fit(self, data, labels, training_epochs, train_size, batch_size, l2lambda):
        # ...
        optimizer = torch.optim.SGD(self.parameters(), lr=0.02, weight_decay=l2lambda)

L1 Regularization $$ penalty_{L1} = ||w||_{1} = \sum_{i}{|w_{i}|} $$

In L1 regularization, the penalty is calculated using the sum of the absolute value of all weights. This can drive many weights to a value of zero. This technique is useful for generating sparser solutions (certain inputs will be treated as irrelevant/noise).

There are no buit-in functions in Pytorch for implementing L1 regularization, so it must be implemented manually. This is less complicated than it sounds.

class L1Model(nn.Module):
    def fit(self, data, labels, training_epochs, train_size, batch_size, l1lambda):
        # ...
        for epoch in range(training_epochs):
            self.train()
            batch_losses = torch.zeros(len(train_ldr))
            for i, batch in enumerate(train_ldr):
                y_hat = self.forward(batch[0])

                loss = loss_func(y_hat, batch[1])
                
                l1_penalization = torch.tensor(0.0, requires_grad=True)
                for p in self.parameters():
                    l1_penalization = l1_penalization + p.abs().sum()
                l1_penalization = l1_penalization * l1lambda
                
                loss = loss + l1_penalization
                # ...
        # ...

Data regularization

For data regularization, batch training is used. Batch training consists in dividing the training data into smaller samples (batches) and training the model iteratively on those batches. This works as a regularization method because it smooths learning by averaging the loss over many samples, which reduces overfitting.

The DataLoader() Pytorch function makes it trivial to implement batch training.

class RegModel(nn.Module):
    def fit(self, data, labels, training_epochs, train_size, batch_size):
        loss_func = nn.BCEWithLogitsLoss()
        optimizer = torch.optim.SGD(self.parameters(), lr=0.02)

        train_data, test_data, train_labels, test_labels = train_test_split(data, labels, train_size=train_size)
        train_ldr = DataLoader(TensorDataset(torch.tensor(train_data), torch.tensor(train_labels)), batch_size)

        train_losses = torch.zeros(training_epochs)
        train_accuracies = torch.zeros(training_epochs)
        test_accuracies = torch.zeros(training_epochs)

        for epoch in range(training_epochs):
            batch_losses = torch.zeros(len(train_ldr))
            for i, batch in enumerate(train_ldr):
                y_hat = self.forward(batch[0])

                loss = loss_func(y_hat, batch[1])
                batch_losses[i] = loss

                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

            train_losses[epoch] = torch.mean(batch_losses)
            train_accuracies[epoch] = torch.mean(((F.sigmoid(self.forward(train_data)) > 0.5) == train_labels).float())
            test_accuracies[epoch] = torch.mean(((F.sigmoid(self.forward(test_data)) > 0.5) == test_labels).float())

        return train_losses, train_accuracies, test_accuracies

Experiments