前言
本次分析的主要内容涉及以下几个方面：
- UFO长啥样？
- UFO在哪些地方出现的次数较多？
- UFO在哪些年份出现的次数较多？
- 热力图同时显示哪些州和哪些年UFO出现次数最多
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
% matplotlib inline
plt.style.use('ggplot')
1 数据整理与清洗
df = pd.read_csv('nuforc_events.csv')
print(df.shape) # 查看数据的结构
print(df.head())
(110265, 13)
Event_Time Event_Date Year Month Day Hour Minute \
0 2017-04-20T14:15:00Z 2017-04-20 2017.0 4.0 20.0 14.0 15.0
1 2017-04-20T04:56:00Z 2017-04-20 2017.0 4.0 20.0 4.0 56.0
2 2017-04-19T23:55:00Z 2017-04-19 2017.0 4.0 19.0 23.0 55.0
3 2017-04-19T23:50:00Z 2017-04-19 2017.0 4.0 19.0 23.0 50.0
4 2017-04-19T23:29:00Z 2017-04-19 2017.0 4.0 19.0 23.0 29.0
City State Shape Duration \
0 Palmyra NJ Other 5 minutes
1 Bridgeview IL Light 20 seconds
2 Newton AL Triangle 5 seconds
3 Newton AL Triangle 5-6 minutes
4 Denver CO Light 1 hour
Summary \
0 I observed an aircraft that seemed to look odd.
1 Bridgeview, IL, blue light. ((anonymous report))
2 Silent triangle UFO.
3 My friend and I stepped outside hoping to catc...
4 Moved slow but made quick turns staying and ci...
Event_URL
0 http://www.nuforc.org/webreports/133/S133726.html
1 http://www.nuforc.org/webreports/133/S133720.html
2 http://www.nuforc.org/webreports/133/S133724.html
3 http://www.nuforc.org/webreports/133/S133723.html
4 http://www.nuforc.org/webreports/133/S133721.html
- 由于存在许多包含NaN的数据信息，在进行分析之前，先用dropna()方法去除包含NaN的行数
df_clean = df.dropna()
print(df_clean.shape) # 查看去除Nan后还有多少行
print(df_clean.head())
(95004, 13)
Event_Time Event_Date Year Month Day Hour Minute \
0 2017-04-20T14:15:00Z 2017-04-20 2017.0 4.0 20.0 14.0 15.0
1 2017-04-20T04:56:00Z 2017-04-20 2017.0 4.0 20.0 4.0 56.0
2 2017-04-19T23:55:00Z 2017-04-19 2017.0 4.0 19.0 23.0 55.0
3 2017-04-19T23:50:00Z 2017-04-19 2017.0 4.0 19.0 23.0 50.0
4 2017-04-19T23:29:00Z 2017-04-19 2017.0 4.0 19.0 23.0 29.0
City State Shape Duration \
0 Palmyra NJ Other 5 minutes
1 Bridgeview IL Light 20 seconds
2 Newton AL Triangle 5 seconds
3 Newton AL Triangle 5-6 minutes
4 Denver CO Light 1 hour
Summary \
0 I observed an aircraft that seemed to look odd.
1 Bridgeview, IL, blue light. ((anonymous report))
2 Silent triangle UFO.
3 My friend and I stepped outside hoping to catc...
4 Moved slow but made quick turns staying and ci...
Event_URL
0 http://www.nuforc.org/webreports/133/S133726.html
1 http://www.nuforc.org/webreports/133/S133720.html
2 http://www.nuforc.org/webreports/133/S133724.html
3 http://www.nuforc.org/webreports/133/S133723.html
4 http://www.nuforc.org/webreports/133/S133721.html
- 由于1900年以前的数据较少，这里选择1900年以后的数据来进行分析，如下：
df_clean = df_clean[df_clean['Year']>=1900] # 获取1900年以后的数据来进行分析
- 查看导入的每列数据的数据类型，通过运行结果，可以看到，“Event_Date”列并不是日期类型，因此要将之转换。
- 可以采用pd.to_datetime()方法来操作
df_clean.dtypes
Event_Time object
Event_Date object
Year float64
Month float64
Day float64
Hour float64
Minute float64
City object
State object
Shape object
Duration object
Summary object
Event_URL object
dtype: object
- 用pd.to_datetime()方法来将str格式的日期转换成日期类型
pd.to_datetime(df_clean['Event_Date']) # 1061-12-31年不能显示
# OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 1061-12-31 00:00:00
df_clean.dtypes
Event_Time object
Event_Date object
Year float64
Month float64
Day float64
Hour float64
Minute float64
City object
State object
Shape object
Duration object
Summary object
Event_URL object
dtype: object
2 UFO长啥样？
- 按UFO出现的形状类型来分析,统计不同类型的UFO出现的次数
s_shape = df_clean.groupby('Shape')['Event_Date'].count()
print(type(s_shape))
s_shape.sort_values(inplace=True)
s_shape
<class 'pandas.core.series.Series'>
Shape
Changed 1
Hexagon 1
Pyramid 1
Flare 1
Round 2
Crescent 2
Delta 7
Cross 287
Cone 383
Egg 842
Teardrop 866
Chevron 1187
Diamond 1405
Cylinder 1495
Rectangle 1620
Flash 1717
Cigar 2313
Changing 2378
Formation 3070
Oval 4332
Disk 5841
Sphere 6482
Other 6658
Unknown 6887
Fireball 7785
Triangle 9358
Circle 9818
Light 20254
Name: Event_Date, dtype: int64
剔除特殊情况
- 剔除出现次数少于10次的类型
- 剔除“Unknown”及“Other”类型
s_shape_normal = s_shape[s_shape.values>10]
s_shape_normal
Shape
Cross 287
Cone 383
Egg 842
Teardrop 866
Chevron 1187
Diamond 1405
Cylinder 1495
Rectangle 1620
Flash 1717
Cigar 2313
Changing 2378
Formation 3070
Oval 4332
Disk 5841
Sphere 6482
Other 6658
Unknown 6887
Fireball 7785
Triangle 9358
Circle 9818
Light 20254
Name: Event_Date, dtype: int64
s_shape_normal = s_shape_normal[s_shape_normal.index.isin(['Unknown', 'Other'])==False]
s_shape_normal
Shape
Cross 287
Cone 383
Egg 842
Teardrop 866
Chevron 1187
Diamond 1405
Cylinder 1495
Rectangle 1620
Flash 1717
Cigar 2313
Changing 2378
Formation 3070
Oval 4332
Disk 5841
Sphere 6482
Fireball 7785
Triangle 9358
Circle 9818
Light 20254
Name: Event_Date, dtype: int64
from matplotlib import font_manager as fm
from matplotlib import cm
labels = s_shape_normal.index
sizes = s_shape_normal.values
explode = (0.1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.1) # "explode" ， show the selected slice
fig, axes = plt.subplots(figsize=(10,5),ncols=2) # 设置绘图区域大小
ax1, ax2 = axes.ravel()
colors = cm.rainbow(np.arange(len(sizes))/len(sizes)) # colormaps: Paired, autumn, rainbow, gray,spring,Darks
patches, texts, autotexts = ax1.pie(sizes, labels=labels, autopct='%1.0f%%',explode=explode,
shadow=False, startangle=150, colors=colors, labeldistance=1.2,pctdistance=1.05, radius=0.95)
# labeldistance: 控制labels显示的位置
# pctdistance: 控制百分比显示的位置
# radius: 控制切片突出的距离
ax1.axis('equal')
# 重新设置字体大小
proptease = fm.FontProperties()
proptease.set_size('xx-small')
# font size include: ‘xx-small’,x-small’,'small’,'medium’,‘large’,‘x-large’,‘xx-large’ or number, e.g. '12'
plt.setp(autotexts, fontproperties=proptease)
plt.setp(texts, fontproperties=proptease)
ax1.set_title('Shapes', loc='center')
# ax2 只显示图例（legend）
ax2.axis('off')
ax2.legend(patches, labels, loc='center left', fontsize=9)
# plt.tight_layout()
# plt.savefig("pie_shape_ufo.png", bbox_inches='tight')
plt.savefig('ufo_shapes.jpg')
plt.show()
运行结果如下：
3 UFO在美国那些州（state）出现的次数比较多？
按”State”进行分组运算，统计ufo在各个州出现的次数
s_state = df_clean.groupby('State')['Event_Date'].count()
print(type(s_state))
s_state.head()
<class 'pandas.core.series.Series'>
State
AB 438
AK 472
AL 930
AR 791
AZ 3488
Name: Event_Date, dtype: int64
将分析得到的结果进行可视化显示，如下：
fig, ax1 = plt.subplots(figsize=(12,8))
width = 0.5
state = s_state.index
x_pos1 = np.arange(len(state))
y1 = s_state.values
ax1.bar(x_pos1, y1,color='#4F81BD',align='center', width=width, label='Amounts', linewidth=0)
ax1.set_title('Amount of reporting UFO events by State ')
ax1.set_xlim(-1, len(state))
ax1.set_xticks(x_pos1)
ax1.set_xticklabels(state, rotation = -90)
ax1.set_ylabel('Amount')
fig.savefig('ufo_state.jpg')
plt.show()
运行结果如下：
从上图可看出，ufo在加州（CA）出现的总次数明显比其他地方多，难道是ufo偏爱加州人民？
4 UFO在哪些年份出现的次数较多？
按”Year”进行分组运算，统计ufo在各个年份出现的次数
# df_clean['Year'].astype(int)
s_year = df_clean.groupby(df_clean['Year'].astype(int))['Event_Date'].count()
print(type(s_year))
s_year.head()
<class 'pandas.core.series.Series'>
Year
1905 1
1910 2
1920 1
1925 1
1929 1
Name: Event_Date, dtype: int64
将分析得到的结果进行可视化显示，如下：
fig, ax = plt.subplots(figsize=(12,20))
# fig, ax1 = plt.subplots(figsize=(12,8))
# fig, axes = plt.subplots(nrows=2, figsize=(12,8))
# fig, axes = plt.subplots(ncols=2, figsize=(18,4))
year = s_year.index
y_pos = np.arange(len(year))
x_value = s_year.values
ax.barh(y_pos, x_value,color='#4F81BD',align='center', label='Amounts', linewidth=0)
ax.set_title('Amount of reporting UFO events by Year ')
ax.set_ylim(-0.5, len(year)-0.5)
ax.set_yticks(y_pos)
ax.set_yticklabels(year, rotation = 0, fontsize=6)
ax.set_xlabel('Amount')
plt.savefig('ufo_year.jpg')
plt.show()
运行结果如下：
从上图可看出，近年来UFO出现的报告次数最多
5 1997年以后的UFO事件分析
- 通过上述分析可看出，1997年以前，报告发现UFO的事件相对较少，下面将针对1997年以后的情况进行分析
df_97 = df_clean[(df_clean['Year']>=1997)]
df_97['Year'] = df_97['Year'].astype(int)
# df_97.astype({'Year':int})
print(df_97.shape)
print(df_97.head())
(86041, 13)
Event_Time Event_Date Year Month Day Hour Minute \
0 2017-04-20T14:15:00Z 2017-04-20 2017 4.0 20.0 14.0 15.0
1 2017-04-20T04:56:00Z 2017-04-20 2017 4.0 20.0 4.0 56.0
2 2017-04-19T23:55:00Z 2017-04-19 2017 4.0 19.0 23.0 55.0
3 2017-04-19T23:50:00Z 2017-04-19 2017 4.0 19.0 23.0 50.0
4 2017-04-19T23:29:00Z 2017-04-19 2017 4.0 19.0 23.0 29.0
City State Shape Duration \
0 Palmyra NJ Other 5 minutes
1 Bridgeview IL Light 20 seconds
2 Newton AL Triangle 5 seconds
3 Newton AL Triangle 5-6 minutes
4 Denver CO Light 1 hour
Summary \
0 I observed an aircraft that seemed to look odd.
1 Bridgeview, IL, blue light. ((anonymous report))
2 Silent triangle UFO.
3 My friend and I stepped outside hoping to catc...
4 Moved slow but made quick turns staying and ci...
Event_URL
0 http://www.nuforc.org/webreports/133/S133726.html
1 http://www.nuforc.org/webreports/133/S133720.html
2 http://www.nuforc.org/webreports/133/S133724.html
3 http://www.nuforc.org/webreports/133/S133723.html
4 http://www.nuforc.org/webreports/133/S133721.html
将数据按”Year”和”State”进行分组运算，如下：
df_amount_year = df_97.groupby(['Year', 'State'])['Event_Date'].size().reset_index()
df_amount_year.columns = ['Year', 'State', 'Amount']
print(df_amount_year.head())
Year State Amount
0 1997 AB 6
1 1997 AK 5
2 1997 AL 8
3 1997 AR 10
4 1997 AZ 127
import seaborn as sns
df_pivot = df_amount_year.pivot_table(index='State', columns='Year', values='Amount')
f, ax = plt.subplots(figsize = (10, 15))
cmap = sns.cubehelix_palette(start = 1, rot = 3, gamma=0.8, as_cmap = True)
sns.heatmap(df_pivot, cmap = cmap, linewidths = 0.05, ax = ax)
ax.set_title('Amounts per State and Year since Year 1997')
ax.set_xlabel('Year')
ax.set_ylabel('State')
ax.set_xticklabels(ax.get_xticklabels(), rotation=90)
f.savefig('ufo_per_year_state.jpg')- 上图中，颜色越深的地方，表示UFO事件报告的次数越多。
更多精彩内容请关注公众号：
“Python数据之道”