Regression Introduction
Main.MultipleRegression History
Hide minor edits - Show changes to output
Changed lines 5-6 from:
to:
(:html:)
<iframe width="560" height="315" src="https://www.youtube.com/embed/daavzLUZqZY" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
(:htmlend:)
<iframe width="560" height="315" src="https://www.youtube.com/embed/daavzLUZqZY" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
(:htmlend:)
Changed lines 186-188 from:
See [[Main/DynamicEstimation|Introduction to Dynamic Estimation]] Examples 3 and 4 for ''IMODE=5'' with differential equations and the [[https://gekko.readthedocs.io/en/latest/imode.html|Gekko IMODE Documentation]].
to:
See [[Main/DynamicEstimation|Introduction to Dynamic Estimation]] Examples 3 and 4 for ''IMODE=5'' with differential equations and the [[https://gekko.readthedocs.io/en/latest/imode.html|Gekko IMODE Documentation]].
%width=550px%Attach:regression_introduction.png
%width=550px%Attach:regression_introduction.png
Changed line 9 from:
* [[https://apmonitor.com/me575/index.php/Main/LinearMultivariateRegression|Linear Regression]]
to:
* [[https://apmonitor.com/pds/index.php/Main/LinearRegression|Linear Regression]]
Deleted line 12:
Changed line 185 from:
See [[Main/DynamicEstimation|Introduction to Dynamic Estimation]] Examples 3 and 4 for ''IMODE=5'' with differential equations.
to:
See [[Main/DynamicEstimation|Introduction to Dynamic Estimation]] Examples 3 and 4 for ''IMODE=5'' with differential equations and the [[https://gekko.readthedocs.io/en/latest/imode.html|Gekko IMODE Documentation]].
Changed line 152 from:
<th class='text-small'></th>
to:
<th class='text-small'>Mode (IMODE)</th>
Changed line 161 from:
<td class='text-left'>Non-Dynamic</td>
to:
<td class='text-left'><b>No Dynamics</b></td>
Changed line 168 from:
<td class='text-left'>Dynamic Simultaneous</td>
to:
<td class='text-left'><b>Dynamic<br>Simultaneous</b></td>
Changed line 175 from:
<td class='text-left'>Dynamic Sequential</td>
to:
<td class='text-left'><b>Dynamic<br>Sequential</b></td>
Changed line 205 from:
height: 320px;
to:
height: 120px;
Changed line 290 from:
width: 45%;
to:
width: 25%;
Changed lines 147-307 from:
to:
These examples do not have differential equations to describe dynamic systems. For problems with differential equations, use ''IMODE=5'' instead of ''IMODE=2''.
(:html:)
<table class='table-fill'>
<thead>
<th class='text-small'></th>
<th class='text-center'>Simulation</th>
<th class='text-center'>Estimation</th>
<th class='text-center'>Control</th>
</thead>
<tbody class='table-hover'>
<tr>
<td class='text-left'>Non-Dynamic</td>
<td class='text-left'>1=Steady-State (SS)</td>
<td class='text-left'>2=SS Regression (MPU)</td>
<td class='text-left'>3=SS Optimize (RTO)</td>
</tr>
<tr>
<td class='text-left'>Dynamic Simultaneous</td>
<td class='text-left'>4=Simulation (SIM)</td>
<td class='text-left'>5=Estimation (EST)</td>
<td class='text-left'>6=Control (CTL)</td>
</tr>
<tr>
<td class='text-left'>Dynamic Sequential</td>
<td class='text-left'>7=Simulation (SQS)</td>
<td class='text-left'>8=Estimation (EST)</td>
<td class='text-left'>9=Control (CTL)</td>
</tr>
</tbody>
</table>
(:htmlend:)
See [[Main/DynamicEstimation|Introduction to Dynamic Estimation]] Examples 3 and 4 for ''IMODE=5'' with differential equations.
(:html:)
<style type="text/css">
div.table-title {
display: block;
margin: auto;
max-width: 600px;
padding:5px;
width: 100%;
}
.table-title h3 {
color: #fafafa;
}
.table-fill {
background: white;
border-radius:3px;
border-collapse: collapse;
height: 320px;
margin: auto;
max-width: 600px;
padding:5px;
width: 100%;
box-shadow: 0 5px 10px rgba(0, 0, 0, 0.1);
animation: float 5s infinite;
}
th {
color:#D5DDE5;;
background:#1b1e24;
border-bottom:4px solid #9ea7af;
border-right: 1px solid #343a45;
text-align:left;
vertical-align:middle;
}
th:first-child {
border-top-left-radius:3px;
}
th:last-child {
border-top-right-radius:3px;
border-right:none;
}
tr {
border-top: 1px solid #C1C3D1;
border-bottom-: 1px solid #C1C3D1;
color:#666B85;
font-weight:normal;
}
tr:hover td {
border-top: 1px solid #22262e;
border-bottom: 1px solid #22262e;
}
tr:first-child {
border-top:none;
}
tr:last-child {
border-bottom:none;
}
tr:nth-child(odd) td {
background:#EEEEEE;
}
tr:last-child td:first-child {
border-bottom-left-radius:3px;
}
tr:last-child td:last-child {
border-bottom-right-radius:3px;
}
td {
background:#FFFFFF;
padding:5px;
text-align:left;
vertical-align:middle;
border-right: 1px solid #C1C3D1;
}
td:last-child {
border-right: 0px;
}
th.text-left {
text-align: left;
}
th.text-center {
text-align: center;
}
th.text-right {
text-align: right;
}
td.text-left {
text-align: left;
width: 45%;
}
td.text-center {
text-align: center;
width: 10%;
}
td.text-small {
text-align: center;
width: 10%;
}
td.text-right {
text-align: right;
}
</style>
(:htmlend:)
(:html:)
<table class='table-fill'>
<thead>
<th class='text-small'></th>
<th class='text-center'>Simulation</th>
<th class='text-center'>Estimation</th>
<th class='text-center'>Control</th>
</thead>
<tbody class='table-hover'>
<tr>
<td class='text-left'>Non-Dynamic</td>
<td class='text-left'>1=Steady-State (SS)</td>
<td class='text-left'>2=SS Regression (MPU)</td>
<td class='text-left'>3=SS Optimize (RTO)</td>
</tr>
<tr>
<td class='text-left'>Dynamic Simultaneous</td>
<td class='text-left'>4=Simulation (SIM)</td>
<td class='text-left'>5=Estimation (EST)</td>
<td class='text-left'>6=Control (CTL)</td>
</tr>
<tr>
<td class='text-left'>Dynamic Sequential</td>
<td class='text-left'>7=Simulation (SQS)</td>
<td class='text-left'>8=Estimation (EST)</td>
<td class='text-left'>9=Control (CTL)</td>
</tr>
</tbody>
</table>
(:htmlend:)
See [[Main/DynamicEstimation|Introduction to Dynamic Estimation]] Examples 3 and 4 for ''IMODE=5'' with differential equations.
(:html:)
<style type="text/css">
div.table-title {
display: block;
margin: auto;
max-width: 600px;
padding:5px;
width: 100%;
}
.table-title h3 {
color: #fafafa;
}
.table-fill {
background: white;
border-radius:3px;
border-collapse: collapse;
height: 320px;
margin: auto;
max-width: 600px;
padding:5px;
width: 100%;
box-shadow: 0 5px 10px rgba(0, 0, 0, 0.1);
animation: float 5s infinite;
}
th {
color:#D5DDE5;;
background:#1b1e24;
border-bottom:4px solid #9ea7af;
border-right: 1px solid #343a45;
text-align:left;
vertical-align:middle;
}
th:first-child {
border-top-left-radius:3px;
}
th:last-child {
border-top-right-radius:3px;
border-right:none;
}
tr {
border-top: 1px solid #C1C3D1;
border-bottom-: 1px solid #C1C3D1;
color:#666B85;
font-weight:normal;
}
tr:hover td {
border-top: 1px solid #22262e;
border-bottom: 1px solid #22262e;
}
tr:first-child {
border-top:none;
}
tr:last-child {
border-bottom:none;
}
tr:nth-child(odd) td {
background:#EEEEEE;
}
tr:last-child td:first-child {
border-bottom-left-radius:3px;
}
tr:last-child td:last-child {
border-bottom-right-radius:3px;
}
td {
background:#FFFFFF;
padding:5px;
text-align:left;
vertical-align:middle;
border-right: 1px solid #C1C3D1;
}
td:last-child {
border-right: 0px;
}
th.text-left {
text-align: left;
}
th.text-center {
text-align: center;
}
th.text-right {
text-align: right;
}
td.text-left {
text-align: left;
width: 45%;
}
td.text-center {
text-align: center;
width: 10%;
}
td.text-small {
text-align: center;
width: 10%;
}
td.text-right {
text-align: right;
}
</style>
(:htmlend:)
Changed line 147 from:
These examples do not have differential equations to describe dynamic systems. For problems with differential equations, use ''IMODE=5'' instead of ''IMODE=2''.
to:
These examples do not have differential equations to describe dynamic systems. For problems with differential equations, use ''IMODE=5'' instead of ''IMODE=2''. See [[Main/DynamicEstimation|Introduction to Dynamic Estimation]] Examples 3 and 4 for ''IMODE=5'' with differential equations.
Added lines 4-5:
%width=550px%Attach:regression_introduction.png
Changed lines 5-16 from:
Regression is used to train a model to predict a relationship between a dependent variable and one or more independent variables. Regression models can be linear or nonlinear, depending on the relationship between the dependent and independent variables. See me575/index.php/Main/LinearMultivariateRegression|Linear Regression]], [[https://apmonitor.com/me575/index.php/Main/NonlinearRegression|Nonlinear Regression]], and the [[https://apmonitor.com/pds|Machine Learning for Engineers]] course for additional regression methods and examples.
to:
Regression is used to train a model to predict a relationship between a dependent variable and one or more independent variables. Regression models can be linear or nonlinear, depending on the relationship between the dependent and independent variables. See the [[https://apmonitor.com/pds|Machine Learning for Engineers]] course for additional information and the following tutorials:
* [[https://apmonitor.com/me575/index.php/Main/LinearMultivariateRegression|Linear Regression]]
* [[https://apmonitor.com/me575/index.php/Main/NonlinearRegression|Nonlinear Regression]]
* [[https://apmonitor.com/pds/index.php/Main/GaussianProcessRegression|Gaussian Processes]]
* [[https://apmonitor.com/pds/index.php/Main/KNearestNeighborsRegression|k-Nearest Neighbors]]
* [[https://apmonitor.com/pds/index.php/Main/LinearRegression|Linear Regression]]
* [[https://apmonitor.com/pds/index.php/Main/MultilayerPerceptronNeuralNetwork|Neural Network Regressor]]
* [[https://apmonitor.com/pds/index.php/Main/SupportVectorRegressor|Support Vector Regressor]]
* [[https://apmonitor.com/pds/index.php/Main/XGBoostRegressor|XGBoost Regressor]]
This tutorial is with linear regression to demonstrate a simple example in Python Gekko.
* [[https://apmonitor.com/me575/index.php/Main/LinearMultivariateRegression|Linear Regression]]
* [[https://apmonitor.com/me575/index.php/Main/NonlinearRegression|Nonlinear Regression]]
* [[https://apmonitor.com/pds/index.php/Main/GaussianProcessRegression|Gaussian Processes]]
* [[https://apmonitor.com/pds/index.php/Main/KNearestNeighborsRegression|k-Nearest Neighbors]]
* [[https://apmonitor.com/pds/index.php/Main/LinearRegression|Linear Regression]]
* [[https://apmonitor.com/pds/index.php/Main/MultilayerPerceptronNeuralNetwork|Neural Network Regressor]]
* [[https://apmonitor.com/pds/index.php/Main/SupportVectorRegressor|Support Vector Regressor]]
* [[https://apmonitor.com/pds/index.php/Main/XGBoostRegressor|XGBoost Regressor]]
This tutorial is with linear regression to demonstrate a simple example in Python Gekko.
Added lines 1-134:
(:title Regression Introduction:)
(:keywords regression, multiple, multivariate, tutorial:)
(:description Multiple Linear Regression with Python Gekko with IMODE=2 and IMODE=3.:)
Regression is used to train a model to predict a relationship between a dependent variable and one or more independent variables. Regression models can be linear or nonlinear, depending on the relationship between the dependent and independent variables. See [[https://apmonitor.com/me575/index.php/Main/LinearMultivariateRegression|Linear Regression]], [[https://apmonitor.com/me575/index.php/Main/NonlinearRegression|Nonlinear Regression]], and the [[https://apmonitor.com/pds|Machine Learning for Engineers]] course for additional regression methods and examples. This tutorial is with linear regression to demonstrate a simple example in Python Gekko.
'''Example Multiple Linear Regression'''
Multiple linear regression models the relationship between a dependent variable and one or more independent variables. It is used when there are multiple independent variables that contribute to the prediction of the dependent variable. The goal of multiple linear regression is to find the best fit that minimizes the differences between the observed and predicted values of the dependent variable.
'''Objective:''' Perform multiple linear regression on sample data with two inputs.
Find unknown parameters ''c'_0_'''-''c'_2_''' to minimize the difference between measured ''ym'' and predicted ''yp'' subject to a constraint on the summation of ''c'_1_''' ''x'_1_'''.
'''Data'''
{$x_1 = [1,2,5,3,2,5,2]$}
{$x_2 = [5,6,7,2,1,3,2]$}
{$y_m = [3,2,3,5,6,7,8]$}
'''Linear Equation'''
{$y_p = c_0 + c_1 x_1 + c_2 x_2$}
'''Constraint'''
{$0 \le \sum_{i=1}^n c_1 x_{1,i} \le 10$}
'''Minimize Objective'''
{$\min_{c} \sum_{i=1}^n \left(y_{m,i}-y_{p,i}\right)^2$}
where ''n'' is the length of ''y'_m_''' and ''c'_0_'-c'_2_''' are adjusted to minimize the sum of the squared errors. Report the parameter values and display a plot of the results.
'''Solution'''
Python Gekko has a regression mode where the equations are written once and applied over all data rows. The ''vsum'' object creates a vertical summation over a column.
%width=550px%Attach:multiple_linear_regression_solution.png
(:toggle hide gekko2 button show="Show GEKKO IMODE=2 Solution":)
(:div id=gekko2:)
(:source lang=python:)
import numpy as np
from gekko import GEKKO
# load data
x1 = np.array([1,2,5,3,2,5,2])
x2 = np.array([5,6,7,2,1,3,2])
ym = np.array([3,2,3,5,6,7,8])
# model
m = GEKKO()
c = m.Array(m.FV,3)
for ci in c:
ci.STATUS=1
x1 = m.Param(value=x1)
x2 = m.Param(value=x2)
ymeas = m.Param(value=ym)
ypred = m.Var()
m.Equation(ypred == c[0] + c[1]*x1 + c[2]*x2)
# add constraint on sum(c[1]*x1) with vsum
v1 = m.Var(); m.Equation(v1==c[1]*x1)
con = m.Var(lb=0,ub=10); m.Equation(con==m.vsum(v1))
m.Minimize((ypred-ymeas)**2)
m.options.IMODE = 2
m.solve()
print('Final SSE Objective: ' + str(m.options.objfcnval))
print('Solution')
for i,ci in enumerate(c):
print(i,ci.value[0])
# plot solution
import matplotlib.pyplot as plt
plt.figure(figsize=(8,4))
plt.plot(ymeas,ypred,'ro')
plt.plot([0,10],[0,10],'k-')
plt.xlabel('Meas')
plt.ylabel('Pred')
plt.savefig('results.png',dpi=300)
plt.show()
(:sourceend:)
(:divend:)
It is also possible to write each equation individually for additional control over the regression form. The ''IMODE=3'' option is for optimization problems where each variable, objective term, and equation are written individually. ''IMODE=2'' is more efficient for large-scale problems.
(:toggle hide gekko3 button show="Show GEKKO IMODE=3 Solution":)
(:div id=gekko3:)
(:source lang=python:)
import numpy as np
from gekko import GEKKO
# load data
x1 = np.array([1,2,5,3,2,5,2])
x2 = np.array([5,6,7,2,1,3,2])
ym = np.array([3,2,3,5,6,7,8])
n = len(ym)
# model
m = GEKKO()
c = m.Array(m.FV,3)
for ci in c:
ci.STATUS=1
yp = m.Array(m.Var,n)
for i in range(n):
m.Equation(yp[i]==c[0]+c[1]*x1[i]+c[2]*x2[i])
m.Minimize((yp[i]-ym[i])**2)
# add constraint on sum(c[1]*x1)
s = m.Var(lb=0,ub=10); m.Equation(s==c[1]*sum(x1))
m.options.IMODE = 3
m.solve()
print('Final SSE Objective: ' + str(m.options.objfcnval))
print('Solution')
for i,ci in enumerate(c):
print(i,ci.value[0])
# plot solution
import matplotlib.pyplot as plt
plt.figure(figsize=(8,4))
ypv = [yp[i].value[0] for i in range(n)]
plt.plot(ym,ypv,'ro')
plt.plot([0,10],[0,10],'k-')
plt.xlabel('Meas')
plt.ylabel('Pred')
plt.savefig('results.png',dpi=300)
plt.show()
(:sourceend:)
(:divend:)
These examples do not have differential equations to describe dynamic systems. For problems with differential equations, use ''IMODE=5'' instead of ''IMODE=2''.
(:keywords regression, multiple, multivariate, tutorial:)
(:description Multiple Linear Regression with Python Gekko with IMODE=2 and IMODE=3.:)
Regression is used to train a model to predict a relationship between a dependent variable and one or more independent variables. Regression models can be linear or nonlinear, depending on the relationship between the dependent and independent variables. See [[https://apmonitor.com/me575/index.php/Main/LinearMultivariateRegression|Linear Regression]], [[https://apmonitor.com/me575/index.php/Main/NonlinearRegression|Nonlinear Regression]], and the [[https://apmonitor.com/pds|Machine Learning for Engineers]] course for additional regression methods and examples. This tutorial is with linear regression to demonstrate a simple example in Python Gekko.
'''Example Multiple Linear Regression'''
Multiple linear regression models the relationship between a dependent variable and one or more independent variables. It is used when there are multiple independent variables that contribute to the prediction of the dependent variable. The goal of multiple linear regression is to find the best fit that minimizes the differences between the observed and predicted values of the dependent variable.
'''Objective:''' Perform multiple linear regression on sample data with two inputs.
Find unknown parameters ''c'_0_'''-''c'_2_''' to minimize the difference between measured ''ym'' and predicted ''yp'' subject to a constraint on the summation of ''c'_1_''' ''x'_1_'''.
'''Data'''
{$x_1 = [1,2,5,3,2,5,2]$}
{$x_2 = [5,6,7,2,1,3,2]$}
{$y_m = [3,2,3,5,6,7,8]$}
'''Linear Equation'''
{$y_p = c_0 + c_1 x_1 + c_2 x_2$}
'''Constraint'''
{$0 \le \sum_{i=1}^n c_1 x_{1,i} \le 10$}
'''Minimize Objective'''
{$\min_{c} \sum_{i=1}^n \left(y_{m,i}-y_{p,i}\right)^2$}
where ''n'' is the length of ''y'_m_''' and ''c'_0_'-c'_2_''' are adjusted to minimize the sum of the squared errors. Report the parameter values and display a plot of the results.
'''Solution'''
Python Gekko has a regression mode where the equations are written once and applied over all data rows. The ''vsum'' object creates a vertical summation over a column.
%width=550px%Attach:multiple_linear_regression_solution.png
(:toggle hide gekko2 button show="Show GEKKO IMODE=2 Solution":)
(:div id=gekko2:)
(:source lang=python:)
import numpy as np
from gekko import GEKKO
# load data
x1 = np.array([1,2,5,3,2,5,2])
x2 = np.array([5,6,7,2,1,3,2])
ym = np.array([3,2,3,5,6,7,8])
# model
m = GEKKO()
c = m.Array(m.FV,3)
for ci in c:
ci.STATUS=1
x1 = m.Param(value=x1)
x2 = m.Param(value=x2)
ymeas = m.Param(value=ym)
ypred = m.Var()
m.Equation(ypred == c[0] + c[1]*x1 + c[2]*x2)
# add constraint on sum(c[1]*x1) with vsum
v1 = m.Var(); m.Equation(v1==c[1]*x1)
con = m.Var(lb=0,ub=10); m.Equation(con==m.vsum(v1))
m.Minimize((ypred-ymeas)**2)
m.options.IMODE = 2
m.solve()
print('Final SSE Objective: ' + str(m.options.objfcnval))
print('Solution')
for i,ci in enumerate(c):
print(i,ci.value[0])
# plot solution
import matplotlib.pyplot as plt
plt.figure(figsize=(8,4))
plt.plot(ymeas,ypred,'ro')
plt.plot([0,10],[0,10],'k-')
plt.xlabel('Meas')
plt.ylabel('Pred')
plt.savefig('results.png',dpi=300)
plt.show()
(:sourceend:)
(:divend:)
It is also possible to write each equation individually for additional control over the regression form. The ''IMODE=3'' option is for optimization problems where each variable, objective term, and equation are written individually. ''IMODE=2'' is more efficient for large-scale problems.
(:toggle hide gekko3 button show="Show GEKKO IMODE=3 Solution":)
(:div id=gekko3:)
(:source lang=python:)
import numpy as np
from gekko import GEKKO
# load data
x1 = np.array([1,2,5,3,2,5,2])
x2 = np.array([5,6,7,2,1,3,2])
ym = np.array([3,2,3,5,6,7,8])
n = len(ym)
# model
m = GEKKO()
c = m.Array(m.FV,3)
for ci in c:
ci.STATUS=1
yp = m.Array(m.Var,n)
for i in range(n):
m.Equation(yp[i]==c[0]+c[1]*x1[i]+c[2]*x2[i])
m.Minimize((yp[i]-ym[i])**2)
# add constraint on sum(c[1]*x1)
s = m.Var(lb=0,ub=10); m.Equation(s==c[1]*sum(x1))
m.options.IMODE = 3
m.solve()
print('Final SSE Objective: ' + str(m.options.objfcnval))
print('Solution')
for i,ci in enumerate(c):
print(i,ci.value[0])
# plot solution
import matplotlib.pyplot as plt
plt.figure(figsize=(8,4))
ypv = [yp[i].value[0] for i in range(n)]
plt.plot(ym,ypv,'ro')
plt.plot([0,10],[0,10],'k-')
plt.xlabel('Meas')
plt.ylabel('Pred')
plt.savefig('results.png',dpi=300)
plt.show()
(:sourceend:)
(:divend:)
These examples do not have differential equations to describe dynamic systems. For problems with differential equations, use ''IMODE=5'' instead of ''IMODE=2''.