Machine Learning for Engineers

Regression is a type of supervised learning that predicts continuous output values from a set of input features. The goal of regression is to build a model that can accurately predict the target variable for new, unseen data.

Regression Python Jupyter Notebook

Jupyter Notebook in Google Colab

Regression Overview MATLAB Live Script

There are many different algorithms that can be used for regression, and some of the most commonly used ones include:

1️⃣ Linear Regression: Linear regression is a simple and widely used approach to regression that models the relationship between the input and output variables as a linear function. Linear regression is easy to interpret and computationally efficient, but it may not perform well for nonlinear relationships between variables.

2️⃣ k-Nearest Neighbors: k-Nearest Neighbors (k-NN) is a non-parametric algorithm that predicts the target value for a new data point by finding the k closest data points in the training set and averaging their target values. k-NN is simple and easy to implement, but it can be computationally expensive for large datasets.

3️⃣ Support Vector Regressor: Support Vector Regression (SVR) is a variant of support vector machines that can be used for regression problems. It works by finding the hyperplane that maximally separates the data points while minimizing the error on the training data.

4️⃣ Gaussian Processes: Gaussian Processes (GP) is a probabilistic approach to regression that models the relationship between the input and output variables as a Gaussian distribution. It is a flexible method that can be used to model complex, nonlinear relationships between variables, and can be used for both regression and classification problems.

5️⃣ Neural Network Regressor: Neural networks are a powerful class of models that can be used for regression and classification tasks. They are highly flexible and can model complex, nonlinear relationships between variables, but can be computationally expensive and require large amounts of training data.

6️⃣ XGBoost Regressor: XGBoost is a popular algorithm for regression and classification that uses a gradient boosting approach to iteratively improve the performance of the model. It is highly accurate and can handle large datasets with many features, but can be computationally expensive.

Other regression algorithms include decision trees, random forests, and Bayesian regression, among others. The choice of algorithm depends on the specific problem and the characteristics of the data, and it is often useful to try multiple algorithms and compare the performance.

1D Regression Example

A linear regression has 100 sample points with one feature (X), one output label (y), and random variations in the data (noise). The scikit-learn.datasets method make_regression produces the sample data.

2D Regression Example

Another linear regression also has 100 sample points, but with two features (X). A 3-dimensional plot is used to view the relationship between the two features and the output.

Multiple Regression

Regression results, especially with 3+ features, can be viewed with a parity plot to show the performance of the regression fit. Separate train (80%) and test sets (20%) are split to fit and evaluate the performance on separate data.

Test Regressors

The lazypredict package tests many of the regression models in scikit-learn and returns a ranked list of the performance.