Logistic Regression

Logistic regression is a machine learning algorithm for classification. In this algorithm, the probabilities describing the possible outcomes of a single trial are modeled using a logistic function.

GitHub |

Google Colab |

MATLAB Live Script

Logistic regression makes a binary classification prediction based on the sigmoid function with n input features, $x_1 \ldots x_n$ .

$p = \frac{1}{1+e^{-(\beta_0+\beta_1x_1+...+\beta_nx_n)}}$

The $\beta_i$ are coefficients that can be determined from a stochastic gradient descent or other optimization on the training dataset. The desired classification is either a 0 or 1, so the probability calculated with the sigmoid function is rounded to the nearest value.

Advantages: Logistic regression is designed for this purpose (classification), and is most useful for understanding the influence of several independent variables on a single outcome variable.

Disadvantages: Assumes all predictors are independent of each other, and assumes data set is free of missing values.

from sklearn.linear_model import LogisticRegression
lr = LogisticRegression(solver='lbfgs')
lr.fit(XA,yA)
yP = lr.predict(XB)

[$[Get Code]]

Optical Character Recognition with Logistic Regression

Optical character recognition (OCR) is a method used to convert images of text into machine-readable text. It is typically used to process scanned documents and recognize the text in the images. Logistic regression is a type of machine learning model that can be used for classification tasks. In the context of OCR, logistic regression could be trained to recognize characters in images of text.

Here is an example of using logistic regression for OCR in Python:

# Import necessary libraries
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# Load the dataset of images of handwritten digits
digits = load_digits()

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(digits.data,
digits.target,
random_state=0)

# Create a logistic regression classifier
clf = LogisticRegression(max_iter=5000, random_state=0)

# Train the model using the training set
clf.fit(X_train, y_train)

# Evaluate the model's performance on the test set
accuracy = clf.score(X_test, y_test)
print("Accuracy: %0.2f" % accuracy)

[$[Get Code]]

In this example, we use the scikit-learn library to load the dataset of images of handwritten digits, split the dataset into training and testing sets, and train a logistic regression classifier. We then evaluate the model's performance on the test set by computing the accuracy, which is the proportion of test images that the model correctly identifies. Below is an additional example that shows a sample image from the test set.

Logistic Regression Programming

Using only Python code (not a machine learning package), create a Logistic Regression classifier. Use the blobs test code to evaluate the performance of the code with any number of features (n_features). First develop the code to test 2 features. Use the source code blocks as needed.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
import seaborn as sns

# Generate blobs dataset
features, label = make_blobs(n_samples=800, centers=2,\
n_features=2, random_state=12)
data = pd.DataFrame()
data['x1'] = features[:,0]
data['x2'] = features[:,1]
data['y'] = label

sns.pairplot(data,hue='y')

[$[Get Code]]

Split data into train (80%) and test (20%) sets.

Create a function to make a prediction with coefficients.

$p = \frac{1}{1+e^{-(\beta_0+\beta_1x_1+...+\beta_nx_n)}}$

The predicted $\hat y$ is $p$ rounded to the nearest 0 or 1 to produce a final predicted value.

Update coefficients for the intercept $\beta_0$ with

$\beta_0 = \beta_0 + l_{rate} (err) p (1-p)$

and for the feature coefficients $beta_{i+1}$ with

$\beta_{i+1} = \beta_{i+1} + l_{rate} (err) p (1-p) x_i$

where $l_{rate}$ is the learning rate (use 0.3), $err$ is the error (difference) between the measured label $y$ and the predicted label $p$ , and $x_i$ is the input feature. Continue for 100 epochs (iteration updates of $\beta$ ) and plot the loss function.

Test the logistic regression model on the remaining 20% the data not used for training.

As a final step, make the code general to accept any number of input features. Test with a random integer for n_features between 3 and 10.

Solutions

Logistic Regression Solution in Python

Google Colab

Resources

Logistic Regression on Wikipedia, retrieved 22 Feb 2021.
Brownlee, J., How To Implement Logistic Regression From Scratch in Python, Machine Learning Mastery, Posted 31 Oct 2016, Updated 11 Dec 2019.

✅ Knowledge Check

1. Which statement best describes the Logistic Regression?

A. It's an algorithm used primarily for regression problems.

B. It predicts the output based on a linear combination of input features.

C. Logistic Regression assumes all predictors are independent of each other.

D. It is most effective for multi-class classification problems.

2. In the context of Optical Character Recognition (OCR), what is the primary role of Logistic Regression?

A. To convert images of text into a digital format.

B. To train on images to recognize and classify text characters.

C. To improve the resolution of scanned documents.

D. To detect and remove noise from scanned documents.

Return to Classification Overview

Machine Learning for Engineers