Main.QuasiNewton History

Hide minor edits - Show changes to output

February 15, 2013, at 05:41 PM by 128.187.97.21 -
Added lines 35-38:

----

Attach:group50.png This assignment can be completed in groups of two. Additional guidelines on individual, collaborative, and group assignments are provided under the [[Main/CourseStandards | Expectations link]].
February 15, 2013, at 01:43 PM by 128.187.97.21 -
Changed line 14 from:
[[Attach:search_direction_worksheet.pdf|Search Direction Worksheet]]
to:
[[Attach:search_direction_homework.pdf|Search Direction Homework]]
February 11, 2013, at 04:41 PM by 128.187.97.21 -
Deleted lines 34-253:

----

!!!! Python Source: methods.py

(:html:)
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/1999/REC-html401-19991224/strict.dtd">
<html>
<head>
<META http-equiv=Content-Type content="text/html; charset=UTF-8">
<title>Exported from Notepad++</title>
<style type="text/css">
span {
font-family: 'Courier New';
font-size: 10pt;
color: #000000;
}
.sc0 {
}
.sc1 {
color: #008000;
}
.sc2 {
color: #FF0000;
}
.sc4 {
color: #808080;
}
.sc5 {
font-weight: bold;
color: #0000FF;
}
.sc9 {
color: #FF00FF;
}
.sc10 {
font-weight: bold;
color: #000080;
}
.sc11 {
}
.sc12 {
color: #008000;
}
</style>
</head>
<body>
<div style="float: left; white-space: pre; line-height: 1; background: #FFFFFF; "><span class="sc12">## Generate a contour plot</span><span class="sc0">
</span><span class="sc1"># Import some other libraries that we'll need</span><span class="sc0">
</span><span class="sc1"># matplotlib and numpy packages must also be installed</span><span class="sc0">
</span><span class="sc5">import</span><span class="sc0"> </span><span class="sc11">matplotlib</span><span class="sc0">
</span><span class="sc5">import</span><span class="sc0"> </span><span class="sc11">numpy</span><span class="sc0"> </span><span class="sc5">as</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc0">
</span><span class="sc5">import</span><span class="sc0"> </span><span class="sc11">matplotlib</span><span class="sc10">.</span><span class="sc11">pyplot</span><span class="sc0"> </span><span class="sc5">as</span><span class="sc0"> </span><span class="sc11">plt</span><span class="sc0">

</span><span class="sc1"># define objective function</span><span class="sc0">
</span><span class="sc5">def</span><span class="sc0"> </span><span class="sc9">f</span><span class="sc10">(</span><span class="sc11">x</span><span class="sc10">):</span><span class="sc0">
    </span><span class="sc11">x1</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]</span><span class="sc0">
    </span><span class="sc11">x2</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x</span><span class="sc10">[</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0">
    </span><span class="sc11">obj</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x1</span><span class="sc10">**</span><span class="sc2">2</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc0"> </span><span class="sc2">2.0</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">x1</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">x2</span><span class="sc0"> </span><span class="sc10">+</span><span class="sc0"> </span><span class="sc2">4</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">x2</span><span class="sc10">**</span><span class="sc2">2</span><span class="sc0">
    </span><span class="sc5">return</span><span class="sc0"> </span><span class="sc11">obj</span><span class="sc0">

</span><span class="sc1"># define objective gradient</span><span class="sc0">
</span><span class="sc5">def</span><span class="sc0"> </span><span class="sc9">dfdx</span><span class="sc10">(</span><span class="sc11">x</span><span class="sc10">):</span><span class="sc0">
    </span><span class="sc11">x1</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]</span><span class="sc0">
    </span><span class="sc11">x2</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x</span><span class="sc10">[</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0">
    </span><span class="sc11">grad</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc10">[]</span><span class="sc0">
    </span><span class="sc11">grad</span><span class="sc10">.</span><span class="sc11">append</span><span class="sc10">(</span><span class="sc2">2.0</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">x1</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc0"> </span><span class="sc2">2.0</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">x2</span><span class="sc10">)</span><span class="sc0">
    </span><span class="sc11">grad</span><span class="sc10">.</span><span class="sc11">append</span><span class="sc10">(-</span><span class="sc2">2.0</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">x1</span><span class="sc0"> </span><span class="sc10">+</span><span class="sc0"> </span><span class="sc2">8.0</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">x2</span><span class="sc10">)</span><span class="sc0">
    </span><span class="sc5">return</span><span class="sc0"> </span><span class="sc11">grad</span><span class="sc0">

</span><span class="sc1"># Exact 2nd derivatives (hessian)</span><span class="sc0">
</span><span class="sc11">H</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc10">[[</span><span class="sc2">2.0</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc2">2.0</span><span class="sc10">],[-</span><span class="sc2">2.0</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc2">8.0</span><span class="sc10">]]</span><span class="sc0">

</span><span class="sc1"># Start location</span><span class="sc0">
</span><span class="sc11">x_start</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc10">[-</span><span class="sc2">3.0</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc2">2.0</span><span class="sc10">]</span><span class="sc0">

</span><span class="sc1"># Design variables at mesh points</span><span class="sc0">
</span><span class="sc11">i1</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">arange</span><span class="sc10">(-</span><span class="sc2">4.0</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc2">4.0</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc2">0.1</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc11">i2</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">arange</span><span class="sc10">(-</span><span class="sc2">4.0</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc2">4.0</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc2">0.1</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc11">x1_mesh</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc11">x2_mesh</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">meshgrid</span><span class="sc10">(</span><span class="sc11">i1</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc11">i2</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc11">f_mesh</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x1_mesh</span><span class="sc10">**</span><span class="sc2">2</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc0"> </span><span class="sc2">2.0</span><span class="sc10">*</span><span class="sc11">x1_mesh</span><span class="sc10">*</span><span class="sc11">x2_mesh</span><span class="sc0"> </span><span class="sc10">+</span><span class="sc0"> </span><span class="sc2">4.0</span><span class="sc10">*</span><span class="sc11">x2_mesh</span><span class="sc10">**</span><span class="sc2">2</span><span class="sc0">

</span><span class="sc1"># Create a contour plot</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">figure</span><span class="sc10">()</span><span class="sc0">
</span><span class="sc1"># Specify contour lines</span><span class="sc0">
</span><span class="sc11">lines</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">range</span><span class="sc10">(</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">52</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc1"># Plot contours</span><span class="sc0">
</span><span class="sc11">CS</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">contour</span><span class="sc10">(</span><span class="sc11">x1_mesh</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc11">x2_mesh</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc11">f_mesh</span><span class="sc10">,</span><span class="sc11">lines</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc1"># Label contours</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">clabel</span><span class="sc10">(</span><span class="sc11">CS</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc11">inline</span><span class="sc10">=</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc11">fontsize</span><span class="sc10">=</span><span class="sc2">10</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc1"># Add some text to the plot</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">title</span><span class="sc10">(</span><span class="sc4">'f(x) = x1^2 - 2*x1*x2 + 4*x2^2'</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">xlabel</span><span class="sc10">(</span><span class="sc4">'x1'</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">ylabel</span><span class="sc10">(</span><span class="sc4">'x2'</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc1"># Show the plot</span><span class="sc0">
</span><span class="sc1">#plt.show()</span><span class="sc0">

</span><span class="sc12">##################################################</span><span class="sc0">
</span><span class="sc1"># Newton's method</span><span class="sc0">
</span><span class="sc12">##################################################</span><span class="sc0">
</span><span class="sc11">xn</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">xn</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x_start</span><span class="sc0">
</span><span class="sc1"># Get gradient at start location (df/dx or grad(f))</span><span class="sc0">
</span><span class="sc11">gn</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">dfdx</span><span class="sc10">(</span><span class="sc11">xn</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">])</span><span class="sc0">
</span><span class="sc1"># Compute search direction and magnitude (dx)</span><span class="sc0">
</span><span class="sc1">#  with dx = -inv(H) * grad</span><span class="sc0">
</span><span class="sc11">delta_xn</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">empty</span><span class="sc10">((</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">delta_xn</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">linalg</span><span class="sc10">.</span><span class="sc11">solve</span><span class="sc10">(</span><span class="sc11">H</span><span class="sc10">,</span><span class="sc11">gn</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc11">xn</span><span class="sc10">[</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">xn</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]+</span><span class="sc11">delta_xn</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">plot</span><span class="sc10">(</span><span class="sc11">xn</span><span class="sc10">[:,</span><span class="sc2">0</span><span class="sc10">],</span><span class="sc11">xn</span><span class="sc10">[:,</span><span class="sc2">1</span><span class="sc10">],</span><span class="sc4">'k-o'</span><span class="sc10">)</span><span class="sc0">

</span><span class="sc12">##################################################</span><span class="sc0">
</span><span class="sc1"># Steepest descent method</span><span class="sc0">
</span><span class="sc12">##################################################</span><span class="sc0">
</span><span class="sc1"># Number of iterations</span><span class="sc0">
</span><span class="sc11">n</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc2">8</span><span class="sc0">
</span><span class="sc1"># Use this alpha for every line search</span><span class="sc0">
</span><span class="sc11">alpha</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc2">0.15</span><span class="sc0">
</span><span class="sc1"># Initialize xs</span><span class="sc0">
</span><span class="sc11">xs</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc11">n</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">xs</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x_start</span><span class="sc0">
</span><span class="sc1"># Get gradient at start location (df/dx or grad(f))</span><span class="sc0">
</span><span class="sc5">for</span><span class="sc0"> </span><span class="sc11">i</span><span class="sc0"> </span><span class="sc5">in</span><span class="sc0"> </span><span class="sc11">range</span><span class="sc10">(</span><span class="sc11">n</span><span class="sc10">):</span><span class="sc0">
    </span><span class="sc11">gs</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">dfdx</span><span class="sc10">(</span><span class="sc11">xs</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">])</span><span class="sc0">
    </span><span class="sc1"># Compute search direction and magnitude (dx)</span><span class="sc0">
    </span><span class="sc1">#  with dx = - grad but no line searching</span><span class="sc0">
    </span><span class="sc11">xs</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">xs</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">alpha</span><span class="sc10">,</span><span class="sc11">dfdx</span><span class="sc10">(</span><span class="sc11">xs</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]))</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">plot</span><span class="sc10">(</span><span class="sc11">xs</span><span class="sc10">[:,</span><span class="sc2">0</span><span class="sc10">],</span><span class="sc11">xs</span><span class="sc10">[:,</span><span class="sc2">1</span><span class="sc10">],</span><span class="sc4">'g-o'</span><span class="sc10">)</span><span class="sc0">

</span><span class="sc12">##################################################</span><span class="sc0">
</span><span class="sc1"># Conjugate gradient method</span><span class="sc0">
</span><span class="sc12">##################################################</span><span class="sc0">
</span><span class="sc1"># Number of iterations</span><span class="sc0">
</span><span class="sc11">n</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc2">8</span><span class="sc0">
</span><span class="sc1"># Use this alpha for the first line search</span><span class="sc0">
</span><span class="sc11">alpha</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc2">0.15</span><span class="sc0">
</span><span class="sc11">neg</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc10">[[-</span><span class="sc2">1.0</span><span class="sc10">,</span><span class="sc2">0.0</span><span class="sc10">],[</span><span class="sc2">0.0</span><span class="sc10">,-</span><span class="sc2">1.0</span><span class="sc10">]]</span><span class="sc0">
</span><span class="sc1"># Initialize xc</span><span class="sc0">
</span><span class="sc11">xc</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc11">n</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">xc</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x_start</span><span class="sc0">
</span><span class="sc1"># Initialize delta_gc</span><span class="sc0">
</span><span class="sc11">delta_cg</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc11">n</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc1"># Initialize gc</span><span class="sc0">
</span><span class="sc11">gc</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc11">n</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc1"># Get gradient at start location (df/dx or grad(f))</span><span class="sc0">
</span><span class="sc5">for</span><span class="sc0"> </span><span class="sc11">i</span><span class="sc0"> </span><span class="sc5">in</span><span class="sc0"> </span><span class="sc11">range</span><span class="sc10">(</span><span class="sc11">n</span><span class="sc10">):</span><span class="sc0">
    </span><span class="sc11">gc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">dfdx</span><span class="sc10">(</span><span class="sc11">xc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">])</span><span class="sc0">
    </span><span class="sc1"># Compute search direction and magnitude (dx)</span><span class="sc0">
    </span><span class="sc1">#  with dx = - grad but no line searching</span><span class="sc0">
    </span><span class="sc5">if</span><span class="sc0"> </span><span class="sc11">i</span><span class="sc10">==</span><span class="sc2">0</span><span class="sc10">:</span><span class="sc0">
        </span><span class="sc11">beta</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc2">0</span><span class="sc0">
        </span><span class="sc11">delta_cg</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">alpha</span><span class="sc10">,</span><span class="sc11">dfdx</span><span class="sc10">(</span><span class="sc11">xc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]))</span><span class="sc0">
    </span><span class="sc5">else</span><span class="sc10">:</span><span class="sc0">
        </span><span class="sc11">beta</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">gc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">],</span><span class="sc11">gc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">])</span><span class="sc0"> </span><span class="sc10">/</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">gc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">-</span><span class="sc2">1</span><span class="sc10">],</span><span class="sc11">gc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">-</span><span class="sc2">1</span><span class="sc10">])</span><span class="sc0">
        </span><span class="sc11">delta_cg</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">alpha</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">neg</span><span class="sc10">,</span><span class="sc11">dfdx</span><span class="sc10">(</span><span class="sc11">xc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]))</span><span class="sc0"> \
            </span><span class="sc10">+</span><span class="sc0"> </span><span class="sc11">beta</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">delta_cg</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">-</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0">
    </span><span class="sc11">xc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">xc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">+</span><span class="sc0"> </span><span class="sc11">delta_cg</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">plot</span><span class="sc10">(</span><span class="sc11">xc</span><span class="sc10">[:,</span><span class="sc2">0</span><span class="sc10">],</span><span class="sc11">xc</span><span class="sc10">[:,</span><span class="sc2">1</span><span class="sc10">],</span><span class="sc4">'y-o'</span><span class="sc10">)</span><span class="sc0">

</span><span class="sc12">##################################################</span><span class="sc0">
</span><span class="sc1"># Quasi-Newton method</span><span class="sc0">
</span><span class="sc12">##################################################</span><span class="sc0">
</span><span class="sc1"># Number of iterations</span><span class="sc0">
</span><span class="sc11">n</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc2">8</span><span class="sc0">
</span><span class="sc1"># Use this alpha for every line search</span><span class="sc0">
</span><span class="sc11">alpha</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">linspace</span><span class="sc10">(</span><span class="sc2">0.1</span><span class="sc10">,</span><span class="sc2">1.0</span><span class="sc10">,</span><span class="sc11">n</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc1"># Initialize delta_xq and gamma</span><span class="sc0">
</span><span class="sc11">delta_xq</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">1</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">gamma</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">1</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part1</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part2</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part3</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part4</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part5</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part6</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">1</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part7</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">1</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part8</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part9</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc1"># Initialize xq</span><span class="sc0">
</span><span class="sc11">xq</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc11">n</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">xq</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x_start</span><span class="sc0">
</span><span class="sc1"># Initialize gradient storage</span><span class="sc0">
</span><span class="sc11">g</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc11">n</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">g</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">dfdx</span><span class="sc10">(</span><span class="sc11">xq</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">])</span><span class="sc0">
</span><span class="sc1"># Initialize hessian storage</span><span class="sc0">
</span><span class="sc11">h</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc11">n</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">h</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc10">[[</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc2">0.0</span><span class="sc10">],[</span><span class="sc2">0.0</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc2">1</span><span class="sc10">]]</span><span class="sc0">
</span><span class="sc5">for</span><span class="sc0"> </span><span class="sc11">i</span><span class="sc0"> </span><span class="sc5">in</span><span class="sc0"> </span><span class="sc11">range</span><span class="sc10">(</span><span class="sc11">n</span><span class="sc10">):</span><span class="sc0">
    </span><span class="sc1"># Compute search direction and magnitude (dx)</span><span class="sc0">
    </span><span class="sc1">#  with dx = -alpha * inv(h) * grad</span><span class="sc0">
    </span><span class="sc11">delta_xq</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">alpha</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">],</span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">linalg</span><span class="sc10">.</span><span class="sc11">solve</span><span class="sc10">(</span><span class="sc11">h</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">],</span><span class="sc11">g</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]))</span><span class="sc0">
    </span><span class="sc11">xq</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">xq</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">+</span><span class="sc0"> </span><span class="sc11">delta_xq</span><span class="sc0">

    </span><span class="sc1"># Get gradient update for next step</span><span class="sc0">
    </span><span class="sc11">g</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">dfdx</span><span class="sc10">(</span><span class="sc11">xq</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">])</span><span class="sc0">

    </span><span class="sc1"># Get hessian update for next step</span><span class="sc0">
    </span><span class="sc11">gamma</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">g</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">]-</span><span class="sc11">g</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0">
    </span><span class="sc11">part1</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">outer</span><span class="sc10">(</span><span class="sc11">gamma</span><span class="sc10">,</span><span class="sc11">gamma</span><span class="sc10">)</span><span class="sc0">
    </span><span class="sc11">part2</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">outer</span><span class="sc10">(</span><span class="sc11">gamma</span><span class="sc10">,</span><span class="sc11">delta_xq</span><span class="sc10">)</span><span class="sc0">
    </span><span class="sc11">part3</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">linalg</span><span class="sc10">.</span><span class="sc11">pinv</span><span class="sc10">(</span><span class="sc11">part2</span><span class="sc10">),</span><span class="sc11">part1</span><span class="sc10">)</span><span class="sc0">

    </span><span class="sc11">part4</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">outer</span><span class="sc10">(</span><span class="sc11">delta_xq</span><span class="sc10">,</span><span class="sc11">delta_xq</span><span class="sc10">)</span><span class="sc0">
    </span><span class="sc11">part5</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">h</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">],</span><span class="sc11">part4</span><span class="sc10">)</span><span class="sc0">
    </span><span class="sc11">part6</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">part5</span><span class="sc10">,</span><span class="sc11">h</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">])</span><span class="sc0">
    </span><span class="sc11">part7</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">delta_xq</span><span class="sc10">,</span><span class="sc11">h</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">])</span><span class="sc0">
    </span><span class="sc11">part8</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">part7</span><span class="sc10">,</span><span class="sc11">delta_xq</span><span class="sc10">)</span><span class="sc0">
    </span><span class="sc11">part9</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">part6</span><span class="sc10">,</span><span class="sc2">1</span><span class="sc10">/</span><span class="sc11">part8</span><span class="sc10">)</span><span class="sc0">
   
    </span><span class="sc11">h</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">h</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">+</span><span class="sc0"> </span><span class="sc11">part3</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc0"> </span><span class="sc11">part9</span><span class="sc0">

</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">plot</span><span class="sc10">(</span><span class="sc11">xq</span><span class="sc10">[:,</span><span class="sc2">0</span><span class="sc10">],</span><span class="sc11">xq</span><span class="sc10">[:,</span><span class="sc2">1</span><span class="sc10">],</span><span class="sc4">'r-o'</span><span class="sc10">)</span><span class="sc0">

</span><span class="sc1"># Save the figure as a PNG</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">savefig</span><span class="sc10">(</span><span class="sc4">'contour.png'</span><span class="sc10">)</span><span class="sc0">

</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">show</span><span class="sc10">()</span><span class="sc0">
</span></div></body>
</html>
(:htmlend:)
February 11, 2013, at 08:35 AM by 128.187.97.21 -
Added lines 37-39:

!!!! Python Source: methods.py

February 11, 2013, at 08:00 AM by 128.187.97.21 -
Added line 36:
----
February 11, 2013, at 07:59 AM by 128.187.97.21 -
Added lines 35-250:

(:html:)
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/1999/REC-html401-19991224/strict.dtd">
<html>
<head>
<META http-equiv=Content-Type content="text/html; charset=UTF-8">
<title>Exported from Notepad++</title>
<style type="text/css">
span {
font-family: 'Courier New';
font-size: 10pt;
color: #000000;
}
.sc0 {
}
.sc1 {
color: #008000;
}
.sc2 {
color: #FF0000;
}
.sc4 {
color: #808080;
}
.sc5 {
font-weight: bold;
color: #0000FF;
}
.sc9 {
color: #FF00FF;
}
.sc10 {
font-weight: bold;
color: #000080;
}
.sc11 {
}
.sc12 {
color: #008000;
}
</style>
</head>
<body>
<div style="float: left; white-space: pre; line-height: 1; background: #FFFFFF; "><span class="sc12">## Generate a contour plot</span><span class="sc0">
</span><span class="sc1"># Import some other libraries that we'll need</span><span class="sc0">
</span><span class="sc1"># matplotlib and numpy packages must also be installed</span><span class="sc0">
</span><span class="sc5">import</span><span class="sc0"> </span><span class="sc11">matplotlib</span><span class="sc0">
</span><span class="sc5">import</span><span class="sc0"> </span><span class="sc11">numpy</span><span class="sc0"> </span><span class="sc5">as</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc0">
</span><span class="sc5">import</span><span class="sc0"> </span><span class="sc11">matplotlib</span><span class="sc10">.</span><span class="sc11">pyplot</span><span class="sc0"> </span><span class="sc5">as</span><span class="sc0"> </span><span class="sc11">plt</span><span class="sc0">

</span><span class="sc1"># define objective function</span><span class="sc0">
</span><span class="sc5">def</span><span class="sc0"> </span><span class="sc9">f</span><span class="sc10">(</span><span class="sc11">x</span><span class="sc10">):</span><span class="sc0">
    </span><span class="sc11">x1</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]</span><span class="sc0">
    </span><span class="sc11">x2</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x</span><span class="sc10">[</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0">
    </span><span class="sc11">obj</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x1</span><span class="sc10">**</span><span class="sc2">2</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc0"> </span><span class="sc2">2.0</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">x1</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">x2</span><span class="sc0"> </span><span class="sc10">+</span><span class="sc0"> </span><span class="sc2">4</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">x2</span><span class="sc10">**</span><span class="sc2">2</span><span class="sc0">
    </span><span class="sc5">return</span><span class="sc0"> </span><span class="sc11">obj</span><span class="sc0">

</span><span class="sc1"># define objective gradient</span><span class="sc0">
</span><span class="sc5">def</span><span class="sc0"> </span><span class="sc9">dfdx</span><span class="sc10">(</span><span class="sc11">x</span><span class="sc10">):</span><span class="sc0">
    </span><span class="sc11">x1</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]</span><span class="sc0">
    </span><span class="sc11">x2</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x</span><span class="sc10">[</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0">
    </span><span class="sc11">grad</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc10">[]</span><span class="sc0">
    </span><span class="sc11">grad</span><span class="sc10">.</span><span class="sc11">append</span><span class="sc10">(</span><span class="sc2">2.0</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">x1</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc0"> </span><span class="sc2">2.0</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">x2</span><span class="sc10">)</span><span class="sc0">
    </span><span class="sc11">grad</span><span class="sc10">.</span><span class="sc11">append</span><span class="sc10">(-</span><span class="sc2">2.0</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">x1</span><span class="sc0"> </span><span class="sc10">+</span><span class="sc0"> </span><span class="sc2">8.0</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">x2</span><span class="sc10">)</span><span class="sc0">
    </span><span class="sc5">return</span><span class="sc0"> </span><span class="sc11">grad</span><span class="sc0">

</span><span class="sc1"># Exact 2nd derivatives (hessian)</span><span class="sc0">
</span><span class="sc11">H</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc10">[[</span><span class="sc2">2.0</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc2">2.0</span><span class="sc10">],[-</span><span class="sc2">2.0</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc2">8.0</span><span class="sc10">]]</span><span class="sc0">

</span><span class="sc1"># Start location</span><span class="sc0">
</span><span class="sc11">x_start</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc10">[-</span><span class="sc2">3.0</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc2">2.0</span><span class="sc10">]</span><span class="sc0">

</span><span class="sc1"># Design variables at mesh points</span><span class="sc0">
</span><span class="sc11">i1</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">arange</span><span class="sc10">(-</span><span class="sc2">4.0</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc2">4.0</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc2">0.1</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc11">i2</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">arange</span><span class="sc10">(-</span><span class="sc2">4.0</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc2">4.0</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc2">0.1</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc11">x1_mesh</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc11">x2_mesh</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">meshgrid</span><span class="sc10">(</span><span class="sc11">i1</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc11">i2</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc11">f_mesh</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x1_mesh</span><span class="sc10">**</span><span class="sc2">2</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc0"> </span><span class="sc2">2.0</span><span class="sc10">*</span><span class="sc11">x1_mesh</span><span class="sc10">*</span><span class="sc11">x2_mesh</span><span class="sc0"> </span><span class="sc10">+</span><span class="sc0"> </span><span class="sc2">4.0</span><span class="sc10">*</span><span class="sc11">x2_mesh</span><span class="sc10">**</span><span class="sc2">2</span><span class="sc0">

</span><span class="sc1"># Create a contour plot</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">figure</span><span class="sc10">()</span><span class="sc0">
</span><span class="sc1"># Specify contour lines</span><span class="sc0">
</span><span class="sc11">lines</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">range</span><span class="sc10">(</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">52</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc1"># Plot contours</span><span class="sc0">
</span><span class="sc11">CS</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">contour</span><span class="sc10">(</span><span class="sc11">x1_mesh</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc11">x2_mesh</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc11">f_mesh</span><span class="sc10">,</span><span class="sc11">lines</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc1"># Label contours</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">clabel</span><span class="sc10">(</span><span class="sc11">CS</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc11">inline</span><span class="sc10">=</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc11">fontsize</span><span class="sc10">=</span><span class="sc2">10</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc1"># Add some text to the plot</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">title</span><span class="sc10">(</span><span class="sc4">'f(x) = x1^2 - 2*x1*x2 + 4*x2^2'</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">xlabel</span><span class="sc10">(</span><span class="sc4">'x1'</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">ylabel</span><span class="sc10">(</span><span class="sc4">'x2'</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc1"># Show the plot</span><span class="sc0">
</span><span class="sc1">#plt.show()</span><span class="sc0">

</span><span class="sc12">##################################################</span><span class="sc0">
</span><span class="sc1"># Newton's method</span><span class="sc0">
</span><span class="sc12">##################################################</span><span class="sc0">
</span><span class="sc11">xn</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">xn</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x_start</span><span class="sc0">
</span><span class="sc1"># Get gradient at start location (df/dx or grad(f))</span><span class="sc0">
</span><span class="sc11">gn</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">dfdx</span><span class="sc10">(</span><span class="sc11">xn</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">])</span><span class="sc0">
</span><span class="sc1"># Compute search direction and magnitude (dx)</span><span class="sc0">
</span><span class="sc1">#  with dx = -inv(H) * grad</span><span class="sc0">
</span><span class="sc11">delta_xn</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">empty</span><span class="sc10">((</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">delta_xn</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">linalg</span><span class="sc10">.</span><span class="sc11">solve</span><span class="sc10">(</span><span class="sc11">H</span><span class="sc10">,</span><span class="sc11">gn</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc11">xn</span><span class="sc10">[</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">xn</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]+</span><span class="sc11">delta_xn</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">plot</span><span class="sc10">(</span><span class="sc11">xn</span><span class="sc10">[:,</span><span class="sc2">0</span><span class="sc10">],</span><span class="sc11">xn</span><span class="sc10">[:,</span><span class="sc2">1</span><span class="sc10">],</span><span class="sc4">'k-o'</span><span class="sc10">)</span><span class="sc0">

</span><span class="sc12">##################################################</span><span class="sc0">
</span><span class="sc1"># Steepest descent method</span><span class="sc0">
</span><span class="sc12">##################################################</span><span class="sc0">
</span><span class="sc1"># Number of iterations</span><span class="sc0">
</span><span class="sc11">n</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc2">8</span><span class="sc0">
</span><span class="sc1"># Use this alpha for every line search</span><span class="sc0">
</span><span class="sc11">alpha</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc2">0.15</span><span class="sc0">
</span><span class="sc1"># Initialize xs</span><span class="sc0">
</span><span class="sc11">xs</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc11">n</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">xs</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x_start</span><span class="sc0">
</span><span class="sc1"># Get gradient at start location (df/dx or grad(f))</span><span class="sc0">
</span><span class="sc5">for</span><span class="sc0"> </span><span class="sc11">i</span><span class="sc0"> </span><span class="sc5">in</span><span class="sc0"> </span><span class="sc11">range</span><span class="sc10">(</span><span class="sc11">n</span><span class="sc10">):</span><span class="sc0">
    </span><span class="sc11">gs</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">dfdx</span><span class="sc10">(</span><span class="sc11">xs</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">])</span><span class="sc0">
    </span><span class="sc1"># Compute search direction and magnitude (dx)</span><span class="sc0">
    </span><span class="sc1">#  with dx = - grad but no line searching</span><span class="sc0">
    </span><span class="sc11">xs</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">xs</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">alpha</span><span class="sc10">,</span><span class="sc11">dfdx</span><span class="sc10">(</span><span class="sc11">xs</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]))</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">plot</span><span class="sc10">(</span><span class="sc11">xs</span><span class="sc10">[:,</span><span class="sc2">0</span><span class="sc10">],</span><span class="sc11">xs</span><span class="sc10">[:,</span><span class="sc2">1</span><span class="sc10">],</span><span class="sc4">'g-o'</span><span class="sc10">)</span><span class="sc0">

</span><span class="sc12">##################################################</span><span class="sc0">
</span><span class="sc1"># Conjugate gradient method</span><span class="sc0">
</span><span class="sc12">##################################################</span><span class="sc0">
</span><span class="sc1"># Number of iterations</span><span class="sc0">
</span><span class="sc11">n</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc2">8</span><span class="sc0">
</span><span class="sc1"># Use this alpha for the first line search</span><span class="sc0">
</span><span class="sc11">alpha</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc2">0.15</span><span class="sc0">
</span><span class="sc11">neg</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc10">[[-</span><span class="sc2">1.0</span><span class="sc10">,</span><span class="sc2">0.0</span><span class="sc10">],[</span><span class="sc2">0.0</span><span class="sc10">,-</span><span class="sc2">1.0</span><span class="sc10">]]</span><span class="sc0">
</span><span class="sc1"># Initialize xc</span><span class="sc0">
</span><span class="sc11">xc</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc11">n</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">xc</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x_start</span><span class="sc0">
</span><span class="sc1"># Initialize delta_gc</span><span class="sc0">
</span><span class="sc11">delta_cg</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc11">n</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc1"># Initialize gc</span><span class="sc0">
</span><span class="sc11">gc</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc11">n</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc1"># Get gradient at start location (df/dx or grad(f))</span><span class="sc0">
</span><span class="sc5">for</span><span class="sc0"> </span><span class="sc11">i</span><span class="sc0"> </span><span class="sc5">in</span><span class="sc0"> </span><span class="sc11">range</span><span class="sc10">(</span><span class="sc11">n</span><span class="sc10">):</span><span class="sc0">
    </span><span class="sc11">gc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">dfdx</span><span class="sc10">(</span><span class="sc11">xc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">])</span><span class="sc0">
    </span><span class="sc1"># Compute search direction and magnitude (dx)</span><span class="sc0">
    </span><span class="sc1">#  with dx = - grad but no line searching</span><span class="sc0">
    </span><span class="sc5">if</span><span class="sc0"> </span><span class="sc11">i</span><span class="sc10">==</span><span class="sc2">0</span><span class="sc10">:</span><span class="sc0">
        </span><span class="sc11">beta</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc2">0</span><span class="sc0">
        </span><span class="sc11">delta_cg</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">alpha</span><span class="sc10">,</span><span class="sc11">dfdx</span><span class="sc10">(</span><span class="sc11">xc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]))</span><span class="sc0">
    </span><span class="sc5">else</span><span class="sc10">:</span><span class="sc0">
        </span><span class="sc11">beta</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">gc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">],</span><span class="sc11">gc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">])</span><span class="sc0"> </span><span class="sc10">/</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">gc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">-</span><span class="sc2">1</span><span class="sc10">],</span><span class="sc11">gc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">-</span><span class="sc2">1</span><span class="sc10">])</span><span class="sc0">
        </span><span class="sc11">delta_cg</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">alpha</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">neg</span><span class="sc10">,</span><span class="sc11">dfdx</span><span class="sc10">(</span><span class="sc11">xc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]))</span><span class="sc0"> \
            </span><span class="sc10">+</span><span class="sc0"> </span><span class="sc11">beta</span><span class="sc0"> </span><span class="sc10">*</span><span class="sc0"> </span><span class="sc11">delta_cg</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">-</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0">
    </span><span class="sc11">xc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">xc</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">+</span><span class="sc0"> </span><span class="sc11">delta_cg</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">plot</span><span class="sc10">(</span><span class="sc11">xc</span><span class="sc10">[:,</span><span class="sc2">0</span><span class="sc10">],</span><span class="sc11">xc</span><span class="sc10">[:,</span><span class="sc2">1</span><span class="sc10">],</span><span class="sc4">'y-o'</span><span class="sc10">)</span><span class="sc0">

</span><span class="sc12">##################################################</span><span class="sc0">
</span><span class="sc1"># Quasi-Newton method</span><span class="sc0">
</span><span class="sc12">##################################################</span><span class="sc0">
</span><span class="sc1"># Number of iterations</span><span class="sc0">
</span><span class="sc11">n</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc2">8</span><span class="sc0">
</span><span class="sc1"># Use this alpha for every line search</span><span class="sc0">
</span><span class="sc11">alpha</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">linspace</span><span class="sc10">(</span><span class="sc2">0.1</span><span class="sc10">,</span><span class="sc2">1.0</span><span class="sc10">,</span><span class="sc11">n</span><span class="sc10">)</span><span class="sc0">
</span><span class="sc1"># Initialize delta_xq and gamma</span><span class="sc0">
</span><span class="sc11">delta_xq</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">1</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">gamma</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">1</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part1</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part2</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part3</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part4</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part5</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part6</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">1</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part7</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">1</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part8</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">part9</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc1"># Initialize xq</span><span class="sc0">
</span><span class="sc11">xq</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc11">n</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">xq</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">x_start</span><span class="sc0">
</span><span class="sc1"># Initialize gradient storage</span><span class="sc0">
</span><span class="sc11">g</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc11">n</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">g</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">dfdx</span><span class="sc10">(</span><span class="sc11">xq</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">])</span><span class="sc0">
</span><span class="sc1"># Initialize hessian storage</span><span class="sc0">
</span><span class="sc11">h</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">zeros</span><span class="sc10">((</span><span class="sc11">n</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">,</span><span class="sc2">2</span><span class="sc10">))</span><span class="sc0">
</span><span class="sc11">h</span><span class="sc10">[</span><span class="sc2">0</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc10">[[</span><span class="sc2">1</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc2">0.0</span><span class="sc10">],[</span><span class="sc2">0.0</span><span class="sc10">,</span><span class="sc0"> </span><span class="sc2">1</span><span class="sc10">]]</span><span class="sc0">
</span><span class="sc5">for</span><span class="sc0"> </span><span class="sc11">i</span><span class="sc0"> </span><span class="sc5">in</span><span class="sc0"> </span><span class="sc11">range</span><span class="sc10">(</span><span class="sc11">n</span><span class="sc10">):</span><span class="sc0">
    </span><span class="sc1"># Compute search direction and magnitude (dx)</span><span class="sc0">
    </span><span class="sc1">#  with dx = -alpha * inv(h) * grad</span><span class="sc0">
    </span><span class="sc11">delta_xq</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">alpha</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">],</span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">linalg</span><span class="sc10">.</span><span class="sc11">solve</span><span class="sc10">(</span><span class="sc11">h</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">],</span><span class="sc11">g</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]))</span><span class="sc0">
    </span><span class="sc11">xq</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">xq</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">+</span><span class="sc0"> </span><span class="sc11">delta_xq</span><span class="sc0">

    </span><span class="sc1"># Get gradient update for next step</span><span class="sc0">
    </span><span class="sc11">g</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">dfdx</span><span class="sc10">(</span><span class="sc11">xq</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">])</span><span class="sc0">

    </span><span class="sc1"># Get hessian update for next step</span><span class="sc0">
    </span><span class="sc11">gamma</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">g</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">]-</span><span class="sc11">g</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0">
    </span><span class="sc11">part1</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">outer</span><span class="sc10">(</span><span class="sc11">gamma</span><span class="sc10">,</span><span class="sc11">gamma</span><span class="sc10">)</span><span class="sc0">
    </span><span class="sc11">part2</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">outer</span><span class="sc10">(</span><span class="sc11">gamma</span><span class="sc10">,</span><span class="sc11">delta_xq</span><span class="sc10">)</span><span class="sc0">
    </span><span class="sc11">part3</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">linalg</span><span class="sc10">.</span><span class="sc11">pinv</span><span class="sc10">(</span><span class="sc11">part2</span><span class="sc10">),</span><span class="sc11">part1</span><span class="sc10">)</span><span class="sc0">

    </span><span class="sc11">part4</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">outer</span><span class="sc10">(</span><span class="sc11">delta_xq</span><span class="sc10">,</span><span class="sc11">delta_xq</span><span class="sc10">)</span><span class="sc0">
    </span><span class="sc11">part5</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">h</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">],</span><span class="sc11">part4</span><span class="sc10">)</span><span class="sc0">
    </span><span class="sc11">part6</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">part5</span><span class="sc10">,</span><span class="sc11">h</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">])</span><span class="sc0">
    </span><span class="sc11">part7</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">delta_xq</span><span class="sc10">,</span><span class="sc11">h</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">])</span><span class="sc0">
    </span><span class="sc11">part8</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">part7</span><span class="sc10">,</span><span class="sc11">delta_xq</span><span class="sc10">)</span><span class="sc0">
    </span><span class="sc11">part9</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">np</span><span class="sc10">.</span><span class="sc11">dot</span><span class="sc10">(</span><span class="sc11">part6</span><span class="sc10">,</span><span class="sc2">1</span><span class="sc10">/</span><span class="sc11">part8</span><span class="sc10">)</span><span class="sc0">
   
    </span><span class="sc11">h</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">+</span><span class="sc2">1</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">=</span><span class="sc0"> </span><span class="sc11">h</span><span class="sc10">[</span><span class="sc11">i</span><span class="sc10">]</span><span class="sc0"> </span><span class="sc10">+</span><span class="sc0"> </span><span class="sc11">part3</span><span class="sc0"> </span><span class="sc10">-</span><span class="sc0"> </span><span class="sc11">part9</span><span class="sc0">

</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">plot</span><span class="sc10">(</span><span class="sc11">xq</span><span class="sc10">[:,</span><span class="sc2">0</span><span class="sc10">],</span><span class="sc11">xq</span><span class="sc10">[:,</span><span class="sc2">1</span><span class="sc10">],</span><span class="sc4">'r-o'</span><span class="sc10">)</span><span class="sc0">

</span><span class="sc1"># Save the figure as a PNG</span><span class="sc0">
</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">savefig</span><span class="sc10">(</span><span class="sc4">'contour.png'</span><span class="sc10">)</span><span class="sc0">

</span><span class="sc11">plt</span><span class="sc10">.</span><span class="sc11">show</span><span class="sc10">()</span><span class="sc0">
</span></div></body>
</html>
(:htmlend:)
February 11, 2013, at 07:49 AM by 128.187.97.21 -
Added lines 19-26:

----

!!!! MATLAB Source Code

* [[Attach:search_methods_matlab.zip|Methods for Obtaining Search Directions (search_methods_matlab.zip)]]

[[Attach:search_methods_matlab.zip|Attach:search_methods_matlab.png]]
February 07, 2013, at 10:40 AM by 128.187.97.21 -
Changed line 14 from:
[[Attach:search_direction_worksheet.png|Search Direction Worksheet]]
to:
[[Attach:search_direction_worksheet.pdf|Search Direction Worksheet]]
February 07, 2013, at 10:39 AM by 128.187.97.21 -
Added lines 14-15:
[[Attach:search_direction_worksheet.png|Search Direction Worksheet]]
Added lines 22-23:
!!!! Python Source Code
Deleted lines 26-31:

----

(:html:)
<iframe width="640" height="360" src="http://www.youtube.com/embed/mLYpktmbLZw?rel=0" frameborder="0" allowfullscreen></iframe>
(:htmlend:)
February 07, 2013, at 10:34 AM by 128.187.97.21 -
Added lines 1-47:
(:title Quasi Newton Methods in Optimization:)
(:keywords quasi-newton, optimization, nonlinear programming:)
(:description Exercise on Quasi-Newton approximations and code examples for solving simple problems.:)

!!!! Quasi-Newton Approximations

The following exercise demonstrates the use of Quasi-Newton methods, Newton's methods, and a Steepest Descent approach to unconstrained optimization. The following tutorial covers:

* Newton's method (exact 2nd derivatives)
* BFGS-Update method (approximate 2nd derivatives)
* Conjugate gradient method
* Steepest descent method

[[Attach:chap3_unconstrained.pdf|Chapter 3]] covers each of these methods and the theoretical background for each. The following exercise is a practical implementation of each method with simplified example code for instructional purposes. The examples do not perform line searching which will be covered in more detail later.

Attach:search_direction_contours.png

----

* [[Attach:search_direction_methods.zip|Methods for Obtaining Search Directions (search_direction_methods.zip)]]

[[Attach:search_direction_methods.zip|Attach:search_direction_methods.png]]

----

(:html:)
<iframe width="640" height="360" src="http://www.youtube.com/embed/mLYpktmbLZw?rel=0" frameborder="0" allowfullscreen></iframe>
(:htmlend:)

----

(:html:)
 <div id="disqus_thread"></div>
    <script type="text/javascript">
        /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
        var disqus_shortname = 'apmonitor'; // required: replace example with your forum shortname

        /* * * DON'T EDIT BELOW THIS LINE * * */
        (function() {
            var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
            dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
            (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
        })();
    </script>
    <noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>
    <a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a>
(:htmlend:)
GlossyBlue theme adapted by David Gilbert
Powered by PmWiki