Data Analysis with Python
Main.PythonDataAnalysis History
Show minor edits - Show changes to output
Changed line 44 from:
<iframe width="560" height="315" src="http://www.youtube.com/embed/pQv6zMlYJ0A" frameborder="0" allowfullscreen></iframe>
to:
<iframe width="560" height="315" src="https://www.youtube.com/embed/pQv6zMlYJ0A" frameborder="0" allowfullscreen></iframe>
Changed line 44 from:
<iframe width="560" height="315" src="https://www.youtube.com/embed/pQv6zMlYJ0A" frameborder="0" allowfullscreen></iframe>
to:
<iframe width="560" height="315" src="http://www.youtube.com/embed/pQv6zMlYJ0A" frameborder="0" allowfullscreen></iframe>
Changed lines 180-181 from:
url = 'https://apmonitor.com/che263/uploads/Main/goog.csv'
to:
url = 'http://apmonitor.com/che263/uploads/Main/goog.csv'
Changed line 211 from:
url = 'https://apmonitor.com/che263/uploads/Main/goog.csv'
to:
url = 'http://apmonitor.com/che263/uploads/Main/goog.csv'
Changed lines 5-9 from:
A common task for scientists and engineers is to analyze data from an external source that may be in a text or comma separated value (CSV) format. By importing the data into Python, data analysis such as statistics, trending, or calculations can be made to synthesize the information into relevant and actionable information. Tutorials below demonstrate how to import data (including online data), perform a basic analysis, trend the results, and export the results to another text file. Two examples are provided with Pandas and Numpy.
to:
A common task for scientists and engineers is to analyze data from an external source that may be in a text or comma separated value (CSV) format.
%width=550px%Attach:import_export_data.png
By importing the data into Python, data analysis such as statistics, trending, or calculations can be made to synthesize the information into relevant and actionable information. Tutorials below demonstrate how to import data (including online data), perform a basic analysis, trend the results, and export the results to another text file. Two examples are provided with Pandas and Numpy.
%width=550px%Attach:import_export_data.png
By importing the data into Python, data analysis such as statistics, trending, or calculations can be made to synthesize the information into relevant and actionable information. Tutorials below demonstrate how to import data (including online data), perform a basic analysis, trend the results, and export the results to another text file. Two examples are provided with Pandas and Numpy.
Added lines 8-9:
%width=30px%Attach:colab.png [[https://colab.research.google.com/github/APMonitor/learn_python/blob/master/Import_Export_Data.ipynb|Import and Export Data (Google Colab)]]
Changed lines 5-27 from:
to:
A common task for scientists and engineers is to analyze data from an external source that may be in a text or comma separated value (CSV) format. By importing the data into Python, data analysis such as statistics, trending, or calculations can be made to synthesize the information into relevant and actionable information. Tutorials below demonstrate how to import data (including online data), perform a basic analysis, trend the results, and export the results to another text file. Two examples are provided with Pandas and Numpy.
%width=30px%Attach:github.png [[https://github.com/APMonitor/learn_python/blob/master/Import_Export_Data.ipynb|Import and Export Data (Jupyter Notebook)]]
'''Pandas Import and Export Data'''
(:source lang=python:)
import pandas as pd
url = 'http://apmonitor.com/pdc/uploads/Main/tclab_data2.txt'
data = pd.read_csv(url)
data.to_csv('file.csv')
(:sourceend:)
'''Numpy Import and Export Data'''
(:source lang=python:)
import numpy as np
data = np.loadtxt('file.csv',delimiter=',',skiprows=1)
np.savetxt('file2.csv',data,delimiter=',',\
comments='',header='Index,Time,Q1,Q2,T1,T2')
(:sourceend:)
Additional script files with Python source code with sample data are below.
%width=30px%Attach:github.png [[https://github.com/APMonitor/learn_python/blob/master/Import_Export_Data.ipynb|Import and Export Data (Jupyter Notebook)]]
'''Pandas Import and Export Data'''
(:source lang=python:)
import pandas as pd
url = 'http://apmonitor.com/pdc/uploads/Main/tclab_data2.txt'
data = pd.read_csv(url)
data.to_csv('file.csv')
(:sourceend:)
'''Numpy Import and Export Data'''
(:source lang=python:)
import numpy as np
data = np.loadtxt('file.csv',delimiter=',',skiprows=1)
np.savetxt('file2.csv',data,delimiter=',',\
comments='',header='Index,Time,Q1,Q2,T1,T2')
(:sourceend:)
Additional script files with Python source code with sample data are below.
Deleted lines 237-255:
----
(:html:)
<div id="disqus_thread"></div>
<script type="text/javascript">
/* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
var disqus_shortname = 'apmonitor'; // required: replace example with your forum shortname
/* * * DON'T EDIT BELOW THIS LINE * * */
(function() {
var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
dsq.src = 'https://' + disqus_shortname + '.disqus.com/embed.js';
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
})();
</script>
<noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>
<a href="https://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a>
(:htmlend:)
Changed lines 95-96 from:
sensors = data_file.ix[:,'s1':'s4']
to:
sensors = data_file.loc[:, 's1':'s4']
Added line 99:
# or use: print(sensors.head(6))
Changed lines 110-114 from:
# column names
cn =
result.columns = cn
to:
result.columns.values[-1] = 'avg'
Changed lines 89-90 from:
to:
url='http://apmonitor.com/che263/uploads/Main/data_with_headers.txt'
data_file = pd.read_csv(url)
data_file = pd.read_csv(url)
Added lines 110-114:
# column names
cn = result.columns.values
cn[-1] = 'avg' # change last column
result.columns = cn
Changed line 72 from:
Attach:download.png [[Attach:data_file.txt|Data File]]
to:
Attach:download.png [[Attach:data_with_headers.txt|Data File with Headers]]
Added lines 71-72:
Attach:download.png [[Attach:data_file.txt|Data File]]
Changed line 165 from:
Once the data is imported, it can be analyzed with many different tools such as machine learning algorithms. Below is an example of using the data for analysis of correlation between open and close price of Google stock.
to:
Once the data is imported, it can be analyzed with many different tools such as machine learning algorithms. Below is an example of using the data for analysis of correlation between open and close price of Google publicly traded shares.
Added lines 167-168:
%width=500px%Attach:machine_learning_stock.png
Deleted lines 228-229:
%width=500px%Attach:machine_learning_stock.png
Added lines 164-228:
Once the data is imported, it can be analyzed with many different tools such as machine learning algorithms. Below is an example of using the data for analysis of correlation between open and close price of Google stock.
(:toggle hide online2 button show="Show Python Regression Analysis":)
(:div id=online2:)
(:source lang=python:)
from gekko import GEKKO
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
# Google stock
url = 'https://apmonitor.com/che263/uploads/Main/goog.csv'
# import data with pandas
data = pd.read_csv(url)
print(data['Close'][0:5])
print('min: '+str(min(data['Close'][0:20])))
print('max: '+str(max(data['Close'][0:20])))
# GEKKO model
m = GEKKO()
# input data
x = m.Param(value=np.array(data['Open']))
# parameters to optimize
a = m.FV()
a.STATUS=1
b = m.FV()
b.STATUS=1
c = m.FV()
c.STATUS=1
# variables
y = m.CV(value=np.array(data['Close']))
y.FSTATUS=1
# regression equation
m.Equation(y==b*m.exp(a*x)+c)
# regression mode
m.options.IMODE = 2
# optimize
m.options.solver = 1
m.solve(disp=True)
# print parameters
print('Optimized, a = ' + str(a.value[0]))
print('Optimized, b = ' + str(b.value[0]))
print('Optimized, c = ' + str(c.value[0]))
# plot data
plt.figure()
plt.plot(data['Open'],data['Close'],'ro',label='Stock Data')
plt.plot(x.value,y.value,'bx',label='Predicted')
plt.xlabel('Open Price')
plt.ylabel('Close Price')
plt.legend()
plt.show()
(:sourceend:)
(:divend:)
%width=500px%Attach:machine_learning_stock.png
Changed lines 145-152 from:
import wget
except:
# install wget if needed
import pip
pip.main(['install','wget'])
import wget
to:
Deleted line 146:
Changed lines 148-154 from:
filename = wget.download(url)
# rename file
from shutil import move
move(filename,stock.lower()+'.csv')
to:
Changed line 150 from:
data = pd.read_csv(stock+'.csv')
to:
data = pd.read_csv(url)
Deleted line 143:
Changed lines 154-155 from:
stock = 'GOOGL'
url = 'https://chart.finance.yahoo.com/table.csv?s='+stock
url = 'https://chart.finance.yahoo.com/table.csv?s='+stock
to:
stock = 'GOOG'
url = 'https://apmonitor.com/che263/uploads/Main/goog.csv'
#url = 'https://chart.finance.yahoo.com/table.csv?s='+stock
url = 'https://apmonitor.com/che263/uploads/Main/goog.csv'
#url = 'https://chart.finance.yahoo.com/table.csv?s='+stock
Changed lines 166-168 from:
print('min: '+str(min(data['Close'][0:30])))
print('max: '+str(max(data['Close'][0:30])))
print('max: '+str(max(data['Close'][0:
to:
print('min: '+str(min(data['Close'][0:20])))
print('max: '+str(max(data['Close'][0:20])))
print('max: '+str(max(data['Close'][0:20])))
Changed lines 171-172 from:
plt.plot(data['Open'][0:30])
plt.plot(data['Close'][0:30])
plt.plot(data['Close'][0:
to:
plt.plot(data['Open'][0:20])
plt.plot(data['Close'][0:20])
plt.plot(data['Close'][0:20])
Deleted lines 73-74:
Added lines 133-134:
Attach:download.png [[Attach:goog.csv|Google Stock Data File]]
Added lines 73-74:
Attach:download.png [[Attach:goog.csv|Google Data File]]
Changed line 16 from:
<iframe width="560" height="315" src="//www.youtube.com/embed/Tq6rCWPdXoQ" frameborder="0" allowfullscreen></iframe>
to:
<iframe width="560" height="315" src="https://www.youtube.com/embed/pQv6zMlYJ0A" frameborder="0" allowfullscreen></iframe>
Changed lines 5-7 from:
A common task for scientists and engineers is to analyze data from an external source that may be in a text or comma separated value
to:
A common task for scientists and engineers is to analyze data from an external source that may be in a text or comma separated value (CSV) format. By importing the data into Python, data analysis such as statistics, trending, or calculations can be made to synthesize the information into relevant and actionable information. Tutorials below demonstrate how to import data (including online data), perform a basic analysis, trend the results, and export the results to another text file. Two examples are provided with Numpy and Pandas. Script files of the Python source code with sample data are below.
Added lines 9-10:
----
Added lines 70-71:
----
Added lines 130-131:
----
Added lines 178-179:
----
Changed lines 134-135 from:
(:toggle hide pandas button show="Show Python Source":)
(:div id=pandas:)
(:div id=
to:
(:toggle hide online button show="Show Python Source":)
(:div id=online:)
(:div id=online:)
Added lines 15-18:
(:html:)
<iframe width="560" height="315" src="//www.youtube.com/embed/Tq6rCWPdXoQ" frameborder="0" allowfullscreen></iframe>
(:htmlend:)
<iframe width="560" height="315" src="//www.youtube.com/embed/Tq6rCWPdXoQ" frameborder="0" allowfullscreen></iframe>
(:htmlend:)
Added lines 68-71:
!!!! Import Data and Analyze with Pandas
Attach:download.png [[Attach:python_with_pandas.zip|Python Data Analysis Example Source Code (Pandas)]]
Attach:download.png [[Attach:python_with_pandas.zip|Python Data Analysis Example Source Code (Pandas)]]
Changed line 73 from:
<iframe width="560" height="315" src="//www.youtube.com/embed/Tq6rCWPdXoQ" frameborder="0" allowfullscreen></iframe>
to:
<iframe width="560" height="315" src="https://www.youtube.com/embed/FXhED53VZ50" frameborder="0" allowfullscreen></iframe>
Deleted lines 75-82:
Attach:download.png [[Attach:python_with_pandas.zip|Python Data Analysis Example Source Code (Pandas)]]
(:html:)
<iframe width="560" height="315" src="https://www.youtube.com/embed/FXhED53VZ50" frameborder="0" allowfullscreen></iframe>
(:htmlend:)
Added lines 134-135:
(:toggle hide pandas button show="Show Python Source":)
(:div id=pandas:)
(:div id=pandas:)
Added line 171:
(:divend:)
Changed line 15 from:
(:toggle hide numpy button show="Show Solution":)
to:
(:toggle hide numpy button show="Show Python (NumPy) Source":)
Changed line 70 from:
to:
Attach:download.png [[Attach:python_with_pandas.zip|Python Data Analysis Example Source Code (Pandas)]]
Changed lines 11-24 from:
<iframe width="560" height
(
!!!! Import Data and Analyze with Pandas
* [[Attach:python_with_pandas.zip|Python Data Analysis Example Source Code (Pandas)]]
(:html:)
<iframe width="560" height="315" src="https://www.youtube.com/embed/FXhED53VZ50" frameborder="0" allowfullscreen></iframe>
(:htmlend:)
to:
Attach:download.png [[Attach:python_data_analysis.zip|Python Data Analysis Example Source Code (Numpy)]]
%width=500px%Attach:analyze_numpy.png
(:toggle hide numpy button show="Show Solution":)
(:div id=numpy:)
%width=500px%Attach:analyze_numpy.png
(:toggle hide numpy button show="Show Solution":)
(:div id=numpy:)
Changed line 18 from:
# import Numpy, Pandas, and Matplotlib
to:
# Numpy (data import, manipulation, export)
Changed line 20 from:
to:
# Matplotlib (create trends)
Changed lines 24-27 from:
data_file = pd.read_csv('data_with_headers.csv')
# create time vector from imported data
time= data_file['time']
# create time vector from imported data
time
to:
data_file = np.genfromtxt('data_file.txt', delimiter=',')
# create time vector from imported data (starts from index 0)
time = data_file[:,0]
# create time vector from imported data (starts from index 0)
time = data_file[:,0]
Changed lines 29-30 from:
sensors = data_file.ix[:,'s1':'s4']
to:
sensors = data_file[:,1:5]
Changed lines 42-49 from:
result = pd.concat(
result
result.to
result.to
to:
# stack time and avg as column vectors
my_data = np.vstack((time,sensors.T,avg))
# transpose data
my_data = my_data.T
# save text file with comma delimiter
np.savetxt('export_from_python.txt',my_data,delimiter=',')
my_data = np.vstack((time,sensors.T,avg))
# transpose data
my_data = my_data.T
# save text file with comma delimiter
np.savetxt('export_from_python.txt',my_data,delimiter=',')
Changed lines 51-52 from:
plt.plot(time,sensors['s1'],'r-')
plt.plot(time,avg,'b.')
plt.plot(time,avg,'b.')
to:
plt.plot(time/60.0,sensors[:,1],'ro')
plt.plot(time/60.0,avg,'b.')
plt.plot(time/60.0,avg,'b.')
Changed lines 54-55 from:
plt.legend(['Sensor 2','Average'])
plt.xlabel('Time (sec)')
plt.xlabel('Time (
to:
plt.legend(['Sensor 2','Average Sensors 1-4'])
plt.xlabel('Time (min)')
plt.xlabel('Time (min)')
Changed line 59 from:
# show the figure on the screen
to:
# show the figure on the screen (pauses execution until closed)
Added lines 62-124:
(:divend:)
(:html:)
<iframe width="560" height="315" src="//www.youtube.com/embed/Tq6rCWPdXoQ" frameborder="0" allowfullscreen></iframe>
(:htmlend:)
!!!! Import Data and Analyze with Pandas
* [[Attach:python_with_pandas.zip|Python Data Analysis Example Source Code (Pandas)]]
(:html:)
<iframe width="560" height="315" src="https://www.youtube.com/embed/FXhED53VZ50" frameborder="0" allowfullscreen></iframe>
(:htmlend:)
(:toggle hide pandas button show="Show Python (Pandas) Source":)
(:div id=pandas:)
(:source lang=python:)
# import Numpy, Pandas, and Matplotlib
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# load the data file
data_file = pd.read_csv('data_with_headers.csv')
# create time vector from imported data
time = data_file['time']
# parse good sensor data from imported data
sensors = data_file.ix[:,'s1':'s4']
# display the first 6 sensor rows
print(sensors[0:6])
# adjust time to start at zero by subtracting the
# first element in the time vector (index = 0)
time = time - time[0]
# calculate the average of the sensor readings
avg = np.mean(sensors,1) # over the 2nd dimension
# export data
my_data = [time, sensors, avg]
result = pd.concat(my_data,axis=1)
result.to_csv('result.csv')
#result.to_excel('result.xlsx')
result.to_html('result.htm')
result.to_clipboard()
# generate a figure
plt.figure(1)
plt.plot(time,sensors['s1'],'r-')
plt.plot(time,avg,'b.')
# add text labels to the plot
plt.legend(['Sensor 2','Average'])
plt.xlabel('Time (sec)')
plt.ylabel('Sensor Values')
# save the figure as a PNG file
plt.savefig('my_Python_plot.png')
# show the figure on the screen
plt.show()
(:sourceend:)
(:divend:)
(:html:)
<iframe width="560" height="315" src="//www.youtube.com/embed/Tq6rCWPdXoQ" frameborder="0" allowfullscreen></iframe>
(:htmlend:)
!!!! Import Data and Analyze with Pandas
* [[Attach:python_with_pandas.zip|Python Data Analysis Example Source Code (Pandas)]]
(:html:)
<iframe width="560" height="315" src="https://www.youtube.com/embed/FXhED53VZ50" frameborder="0" allowfullscreen></iframe>
(:htmlend:)
(:toggle hide pandas button show="Show Python (Pandas) Source":)
(:div id=pandas:)
(:source lang=python:)
# import Numpy, Pandas, and Matplotlib
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# load the data file
data_file = pd.read_csv('data_with_headers.csv')
# create time vector from imported data
time = data_file['time']
# parse good sensor data from imported data
sensors = data_file.ix[:,'s1':'s4']
# display the first 6 sensor rows
print(sensors[0:6])
# adjust time to start at zero by subtracting the
# first element in the time vector (index = 0)
time = time - time[0]
# calculate the average of the sensor readings
avg = np.mean(sensors,1) # over the 2nd dimension
# export data
my_data = [time, sensors, avg]
result = pd.concat(my_data,axis=1)
result.to_csv('result.csv')
#result.to_excel('result.xlsx')
result.to_html('result.htm')
result.to_clipboard()
# generate a figure
plt.figure(1)
plt.plot(time,sensors['s1'],'r-')
plt.plot(time,avg,'b.')
# add text labels to the plot
plt.legend(['Sensor 2','Average'])
plt.xlabel('Time (sec)')
plt.ylabel('Sensor Values')
# save the figure as a PNG file
plt.savefig('my_Python_plot.png')
# show the figure on the screen
plt.show()
(:sourceend:)
(:divend:)
Changed line 72 from:
!!!! Import Data from an Internet Source
to:
!!!! Import Online Data and Analyze
Added lines 75-78:
(:html:)
<iframe width="560" height="315" src="https://www.youtube.com/embed/KzOEmMiPSjg" frameborder="0" allowfullscreen></iframe>
(:htmlend:)
Deleted lines 24-25:
Changed line 74 from:
to:
Below is an example of pulling data from an Internet source, such as financial information about a stock. The example shows how to request, parse, and display the financial data.
Added lines 71-110:
plt.show()
(:sourceend:)
!!!! Import Data from an Internet Source
(:source lang=python:)
import pandas as pd
from matplotlib.finance import *
import matplotlib.pyplot as plt
try:
import wget
except:
# install wget if needed
import pip
pip.main(['install','wget'])
import wget
# stock ticker symbol
stock = 'GOOGL'
url = 'https://chart.finance.yahoo.com/table.csv?s='+stock
filename = wget.download(url)
# rename file
from shutil import move
move(filename,stock.lower()+'.csv')
# import data with pandas
data = pd.read_csv(stock+'.csv')
print(data['Close'][0:5])
print('min: '+str(min(data['Close'][0:30])))
print('max: '+str(max(data['Close'][0:30])))
# plot data with pyplot
plt.figure()
plt.plot(data['Open'][0:30])
plt.plot(data['Close'][0:30])
plt.xlabel('days ago')
plt.ylabel('price')
(:sourceend:)
!!!! Import Data from an Internet Source
(:source lang=python:)
import pandas as pd
from matplotlib.finance import *
import matplotlib.pyplot as plt
try:
import wget
except:
# install wget if needed
import pip
pip.main(['install','wget'])
import wget
# stock ticker symbol
stock = 'GOOGL'
url = 'https://chart.finance.yahoo.com/table.csv?s='+stock
filename = wget.download(url)
# rename file
from shutil import move
move(filename,stock.lower()+'.csv')
# import data with pandas
data = pd.read_csv(stock+'.csv')
print(data['Close'][0:5])
print('min: '+str(min(data['Close'][0:30])))
print('max: '+str(max(data['Close'][0:30])))
# plot data with pyplot
plt.figure()
plt.plot(data['Open'][0:30])
plt.plot(data['Close'][0:30])
plt.xlabel('days ago')
plt.ylabel('price')
Changed lines 27-71 from:
to:
(:source lang=python:)
# import Numpy, Pandas, and Matplotlib
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# load the data file
data_file = pd.read_csv('data_with_headers.csv')
# create time vector from imported data
time = data_file['time']
# parse good sensor data from imported data
sensors = data_file.ix[:,'s1':'s4']
# display the first 6 sensor rows
print(sensors[0:6])
# adjust time to start at zero by subtracting the
# first element in the time vector (index = 0)
time = time - time[0]
# calculate the average of the sensor readings
avg = np.mean(sensors,1) # over the 2nd dimension
# export data
my_data = [time, sensors, avg]
result = pd.concat(my_data,axis=1)
result.to_csv('result.csv')
#result.to_excel('result.xlsx')
result.to_html('result.htm')
result.to_clipboard()
# generate a figure
plt.figure(1)
plt.plot(time,sensors['s1'],'r-')
plt.plot(time,avg,'b.')
# add text labels to the plot
plt.legend(['Sensor 2','Average'])
plt.xlabel('Time (sec)')
plt.ylabel('Sensor Values')
# save the figure as a PNG file
plt.savefig('my_Python_plot.png')
# show the figure on the screen
plt.show()
(:sourceend:)
# import Numpy, Pandas, and Matplotlib
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# load the data file
data_file = pd.read_csv('data_with_headers.csv')
# create time vector from imported data
time = data_file['time']
# parse good sensor data from imported data
sensors = data_file.ix[:,'s1':'s4']
# display the first 6 sensor rows
print(sensors[0:6])
# adjust time to start at zero by subtracting the
# first element in the time vector (index = 0)
time = time - time[0]
# calculate the average of the sensor readings
avg = np.mean(sensors,1) # over the 2nd dimension
# export data
my_data = [time, sensors, avg]
result = pd.concat(my_data,axis=1)
result.to_csv('result.csv')
#result.to_excel('result.xlsx')
result.to_html('result.htm')
result.to_clipboard()
# generate a figure
plt.figure(1)
plt.plot(time,sensors['s1'],'r-')
plt.plot(time,avg,'b.')
# add text labels to the plot
plt.legend(['Sensor 2','Average'])
plt.xlabel('Time (sec)')
plt.ylabel('Sensor Values')
# save the figure as a PNG file
plt.savefig('my_Python_plot.png')
# show the figure on the screen
plt.show()
(:sourceend:)
Changed line 41 from:
print sensors[0:6]
to:
print(sensors[0:6])
Changed line 14 from:
<iframe width="560" height="315" src="https://www.youtube.com/embed/FXhED53VZ50" frameborder="0" allowfullscreen></iframe>
to:
<iframe width="560" height="315" src="//www.youtube.com/embed/Tq6rCWPdXoQ" frameborder="0" allowfullscreen></iframe>
Changed line 22 from:
<iframe width="560" height="315" src="//www.youtube.com/embed/Tq6rCWPdXoQ" frameborder="0" allowfullscreen></iframe>
to:
<iframe width="560" height="315" src="https://www.youtube.com/embed/FXhED53VZ50" frameborder="0" allowfullscreen></iframe>
Changed line 14 from:
<iframe width="560" height="315" src="//www.youtube.com/embed/Tq6rCWPdXoQ" frameborder="0" allowfullscreen></iframe>
to:
<iframe width="560" height="315" src="https://www.youtube.com/embed/FXhED53VZ50" frameborder="0" allowfullscreen></iframe>
Changed line 7 from:
A common task for scientists and engineers is to analyze data from an external source that may be in a text or comma separated value (CSV) format. By importing the data into Python, data analysis such as statistics, trending, or calculations can be made to synthesize the information into relevant and actionable information. This tutorial demonstrates how to import data, perform a basic analysis, trend the results, and export the results to another text file. A script file of the Python source code with sample data is below.
to:
A common task for scientists and engineers is to analyze data from an external source that may be in a text or comma separated value (CSV) format. By importing the data into Python, data analysis such as statistics, trending, or calculations can be made to synthesize the information into relevant and actionable information. This tutorial demonstrates how to import data, perform a basic analysis, trend the results, and export the results to another text file. Two examples are provided with Numpy and Pandas. Script files of the Python source code with sample data are below.
Changed lines 11-12 from:
* [[Attach:python_data_analysis.zip|Python Data Analysis Example Source Code]]
to:
* [[Attach:python_data_analysis.zip|Python Data Analysis Example Source Code (Numpy)]]
Changed line 19 from:
* [[Attach:python_data_analysis.zip|Python Data Analysis Example Source Code]]
to:
* [[Attach:python_with_pandas.zip|Python Data Analysis Example Source Code (Pandas)]]
Added lines 9-10:
!!!! Import Data and Analyze with Numpy
Added lines 16-71:
!!!! Import Data and Analyze with Pandas
* [[Attach:python_data_analysis.zip|Python Data Analysis Example Source Code]]
(:html:)
<iframe width="560" height="315" src="//www.youtube.com/embed/Tq6rCWPdXoQ" frameborder="0" allowfullscreen></iframe>
(:htmlend:)
!!!! Source Code
# import Numpy, Pandas, and Matplotlib
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# load the data file
data_file = pd.read_csv('data_with_headers.csv')
# create time vector from imported data
time = data_file['time']
# parse good sensor data from imported data
sensors = data_file.ix[:,'s1':'s4']
# display the first 6 sensor rows
print sensors[0:6]
# adjust time to start at zero by subtracting the
# first element in the time vector (index = 0)
time = time - time[0]
# calculate the average of the sensor readings
avg = np.mean(sensors,1) # over the 2nd dimension
# export data
my_data = [time, sensors, avg]
result = pd.concat(my_data,axis=1)
result.to_csv('result.csv')
#result.to_excel('result.xlsx')
result.to_html('result.htm')
result.to_clipboard()
# generate a figure
plt.figure(1)
plt.plot(time,sensors['s1'],'r-')
plt.plot(time,avg,'b.')
# add text labels to the plot
plt.legend(['Sensor 2','Average'])
plt.xlabel('Time (sec)')
plt.ylabel('Sensor Values')
# save the figure as a PNG file
plt.savefig('my_Python_plot.png')
# show the figure on the screen
plt.show()
Changed line 12 from:
<iframe width="560" height="315" src="//www.youtube.com/embed/E56egH10RJA?list=PLLBUgWXdTBDi-E--rwBujaNkTejLNI6ap" frameborder="0" allowfullscreen></iframe>
to:
<iframe width="560" height="315" src="//www.youtube.com/embed/Tq6rCWPdXoQ" frameborder="0" allowfullscreen></iframe>
Added lines 7-10:
A common task for scientists and engineers is to analyze data from an external source that may be in a text or comma separated value (CSV) format. By importing the data into Python, data analysis such as statistics, trending, or calculations can be made to synthesize the information into relevant and actionable information. This tutorial demonstrates how to import data, perform a basic analysis, trend the results, and export the results to another text file. A script file of the Python source code with sample data is below.
* [[Attach:python_data_analysis.zip|Python Data Analysis Example Source Code]]
* [[Attach:python_data_analysis.zip|Python Data Analysis Example Source Code]]
Added lines 14-15:
This tutorial can also be completed with [[Main/ExcelDataAnalysis|Excel]] and [[Main/MatlabDataAnalysis|Matlab]]. Click on the appropriate link for additional information.
Added lines 1-28:
(:title Data Analysis with Python:)
(:keywords big data, data analysis, Python, numpy, spreadsheet, nonlinear, optimization, engineering optimization, university course:)
(:description Data Analysis with Python - Problem-Solving Techniques for Chemical Engineers at Brigham Young University:)
!!!!Python Data Analysis
(:html:)
<iframe width="560" height="315" src="//www.youtube.com/embed/E56egH10RJA?list=PLLBUgWXdTBDi-E--rwBujaNkTejLNI6ap" frameborder="0" allowfullscreen></iframe>
(:htmlend:)
----
(:html:)
<div id="disqus_thread"></div>
<script type="text/javascript">
/* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
var disqus_shortname = 'apmonitor'; // required: replace example with your forum shortname
/* * * DON'T EDIT BELOW THIS LINE * * */
(function() {
var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
dsq.src = 'https://' + disqus_shortname + '.disqus.com/embed.js';
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
})();
</script>
<noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>
<a href="https://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a>
(:htmlend:)
(:keywords big data, data analysis, Python, numpy, spreadsheet, nonlinear, optimization, engineering optimization, university course:)
(:description Data Analysis with Python - Problem-Solving Techniques for Chemical Engineers at Brigham Young University:)
!!!!Python Data Analysis
(:html:)
<iframe width="560" height="315" src="//www.youtube.com/embed/E56egH10RJA?list=PLLBUgWXdTBDi-E--rwBujaNkTejLNI6ap" frameborder="0" allowfullscreen></iframe>
(:htmlend:)
----
(:html:)
<div id="disqus_thread"></div>
<script type="text/javascript">
/* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
var disqus_shortname = 'apmonitor'; // required: replace example with your forum shortname
/* * * DON'T EDIT BELOW THIS LINE * * */
(function() {
var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
dsq.src = 'https://' + disqus_shortname + '.disqus.com/embed.js';
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
})();
</script>
<noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>
<a href="https://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a>
(:htmlend:)