Purpose
最近因为要买房子，扫过了各种信息，貌似lianjia上的数据还是靠点谱的(最起码房源图片没有太大的出入)，心血来潮想着做几个图表来显示下房屋的数据信息，顺便练练手。
需求分析
1从lianjia的网站上获取关键的房屋信息数据，然后按照自己的需求通过图表显示出来。
2每天从lianjia的网站上获取一次数据
3以上海地区为主(本人在上海)
4最终生成图表有：房屋交易总量，二手房均价，在售房源，近90天成交量，昨日带看次数
分析获取网站数据
1 数据源
数据的获取主要是从两个地方：
http://sh.lianjia.com/chengjiao/ //成交量数据统计获取
页面上的数据(下面显示的是没登录前的量，貌似登录之后会比这个量要多一点)：
http://sh.lianjia.com/ershoufang/ //二手房相关数据获取
页面上数据：
2 获取方法获取网页数据的话，首先想到的是scrapy，不过考虑到获取的数据不是很多很复杂，这里只用urllib.request来获取就可以了。后面因为使用到tornado的异步，所以会替换成httpclient.AsyncHTTPClient().fetch()。
3 使用urllib.request来获取相关数据。
首先，从网页上爬数据，使用obtain_page_data基础的函数：
def obtain_page_data(target_url):
with urllib.request.urlopen(target_url) as f:
data = f.read().decode('utf8')
return data
obtain_page_data()函数的话，主要是访问给定页面，然后返回页面的数据
然后，获取了数据之后，要按照需求来获取网页上的数据，主要是两大块：
1）房屋总成交量(http://sh.lianjia.com/chengjiao/)
定义函数get_total_dealed_house()，函数最终是返回页面上的总成交量，那么在调用obtain_page_data()获取页面的data后，分析下这个数据是在哪个位置。
那么看到数据一个div下，那么使用BeautifulSoup解析一下获取的html数据后，通过下面的命令来获取text数据：
dealed_house = soup_obj.html.body.find('div', {'class': 'list-head'}).text
找到了text内容之后通过正则表达式过滤掉非数字的字符，然后就获取到了这个数据，具体如下：
def get_total_dealed_house(target_url):
# 获取总的房屋成交量
page_data = obtain_page_data(target_url)
soup_obj = BeautifulSoup(page_data,"html.parser")
dealed_house = soup_obj.html.body.find('div', {'class': 'list-head'}).text
dealed_house_num = re.findall(r'\d+', dealed_house)[0] return int(dealed_house_num)
2）获取其他在线数据(http://sh.lianjia.com/ershoufang/)
类似的，要先分析自己要的数据在网页中的哪个位置，然后去获取，过滤，具体如下：
def get_online_data(target_url):
# 获取 城市挂牌均价，正在出售数量，90天内交易量，昨日看房次数
page_data = obtain_page_data(target_url)
soup_obj = BeautifulSoup(page_data, "html.parser")
online_data_str = soup_obj.html.body.find('div', {'class': 'secondcon'}).text
online_data = online_data_str.replace('\n', '')
avg_price, on_sale, _, sold_in_90, yesterday_check_num = re.findall(r'\d+', online_data) return {'avg_price':avg_price,'on_sale':on_sale,'sold_in_90':sold_in_90,'yesterday_check_num':yesterday_check_num}
3）数据整合/细分各区
使用shanghai_data_process()函数来整合一下1，2中获取的数据，另外lianjia网页上上海区域的数据其实是可以按照各个区来查询的，那么这里也做一下处理，如下：
def shanghai_data_process():
'''
获取上海各个区的数据
:return:
'''
chenjiao_page = "http://sh.lianjia.com/chengjiao/"
ershoufang_page = "http://sh.lianjia.com/ershoufang/"
sh_area_dict = {
"all":"",
"pudongxinqu": "pudongxinqu/",
"minhang": "minhang/",
"baoshan": "baoshan/",
"xuhui": "xuhui/",
"putuo": "putuo/",
"yangpu": "yangpu/",
"changning": "changning/",
"songjiang": "songjiang/",
"jiading": "jiading/",
"huangpu": "huangpu/",
"jingan": "jingan/",
"zhabei": "zhabei/",
"hongkou": "hongkou/",
"qingpu": "qingpu/",
"fengxian": "fengxian/",
"jinshan": "jinshan/",
"chongming": "chongming/",
"shanghaizhoubian": "shanghaizhoubian/",
}
dealed_house_num = get_total_dealed_house(chenjiao_page)
sh_online_data = {}
for key,value in sh_area_dict.items():
sh_online_data[key] = get_online_data(ershoufang_page+sh_area_dict[key])
print("dealed_house_num %s" %dealed_house_num)
for key,value in sh_online_data.items():
print(key,value)
4）整体代码以及输出效果
import urllib.request
import re
from bs4 import BeautifulSoup
import time def obtain_page_data(target_url):
with urllib.request.urlopen(target_url) as f:
data = f.read().decode('utf8')
return data def get_total_dealed_house(target_url):
# 获取总的房屋成交量
page_data = obtain_page_data(target_url)
soup_obj = BeautifulSoup(page_data,"html.parser")
dealed_house = soup_obj.html.body.find('div', {'class': 'list-head'}).text
dealed_house_num = re.findall(r'\d+', dealed_house)[0] return int(dealed_house_num) def get_online_data(target_url):
# 获取 城市挂牌均价，正在出售数量，90天内交易量，昨日看房次数
page_data = obtain_page_data(target_url)
soup_obj = BeautifulSoup(page_data, "html.parser")
online_data_str = soup_obj.html.body.find('div', {'class': 'secondcon'}).text
online_data = online_data_str.replace('\n', '')
avg_price, on_sale, _, sold_in_90, yesterday_check_num = re.findall(r'\d+', online_data) return {'avg_price':avg_price,'on_sale':on_sale,'sold_in_90':sold_in_90,'yesterday_check_num':yesterday_check_num} def shanghai_data_process():
'''
获取上海各个区的数据
:return:
'''
chenjiao_page = "http://sh.lianjia.com/chengjiao/"
ershoufang_page = "http://sh.lianjia.com/ershoufang/"
sh_area_dict = {
"all":"",
"pudongxinqu": "pudongxinqu/",
"minhang": "minhang/",
"baoshan": "baoshan/",
"xuhui": "xuhui/",
"putuo": "putuo/",
"yangpu": "yangpu/",
"changning": "changning/",
"songjiang": "songjiang/",
"jiading": "jiading/",
"huangpu": "huangpu/",
"jingan": "jingan/",
"zhabei": "zhabei/",
"hongkou": "hongkou/",
"qingpu": "qingpu/",
"fengxian": "fengxian/",
"jinshan": "jinshan/",
"chongming": "chongming/",
"shanghaizhoubian": "shanghaizhoubian/",
}
dealed_house_num = get_total_dealed_house(chenjiao_page)
sh_online_data = {}
for key,value in sh_area_dict.items():
sh_online_data[key] = get_online_data(ershoufang_page+sh_area_dict[key])
print("dealed_house_num %s" %dealed_house_num)
for key,value in sh_online_data.items():
print(key,value) def main():
start_time = time.time()
shanghai_data_process()
print("time cost: %s" % (time.time() - start_time)) if __name__=='__main__':
main()
初版源码collect_data.py
Result:
dealed_house_num 51691
zhabei {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
changning {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
baoshan {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
putuo {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
qingpu {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
jinshan {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
chongming {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
all {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
jingan {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
xuhui {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
songjiang {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
yangpu {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
pudongxinqu {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
shanghaizhoubian {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
minhang {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
hongkou {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
fengxian {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
jiading {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
huangpu {'yesterday_check_num': '', 'sold_in_90': '', 'avg_price': '', 'on_sale': ''}
time cost: 12.94211196899414
Result
移植到tornado上
1 为什么要使用tornado
tornado是一个小巧的异步的python框架，这里使用到它是因为在发送request获取网页数据(IO密集)其实可以使用异步来提高效率，特别是在后期访问量大的时候，使用tornado会提高效率。
2 移植上面初步获取数据功能到tornado上
这里的关键点有这么几个：
1）异步获取网页数据
使用httpclient.AsyncHTTPClient().fetch()来获取页面数据，配合使用gen.coroutine+yield来实现异步。
2）返回数据的时候要使用raise gen.Return(data)
3）初步改造后的版本以及运行结果如下：
import re
from bs4 import BeautifulSoup
import time
from tornado import httpclient,gen,ioloop @gen.coroutine
def obtain_page_data(target_url):
response = yield httpclient.AsyncHTTPClient().fetch(target_url)
data = response.body.decode('utf8')
print("start %s %s" %(target_url,time.time())) raise gen.Return(data) @gen.coroutine
def get_total_dealed_house(target_url):
# 获取总的房屋成交量
page_data = yield obtain_page_data(target_url)
soup_obj = BeautifulSoup(page_data,"html.parser")
dealed_house = soup_obj.html.body.find('div', {'class': 'list-head'}).text
dealed_house_num = re.findall(r'\d+', dealed_house)[0] raise gen.Return(int(dealed_house_num)) @gen.coroutine
def get_online_data(target_url):
# 获取 城市挂牌均价，正在出售数量，90天内交易量，昨日看房次数
page_data = yield obtain_page_data(target_url)
soup_obj = BeautifulSoup(page_data, "html.parser")
online_data_str = soup_obj.html.body.find('div', {'class': 'secondcon'}).text
online_data = online_data_str.replace('\n', '')
avg_price, on_sale, _, sold_in_90, yesterday_check_num = re.findall(r'\d+', online_data) raise gen.Return({'avg_price':avg_price,'on_sale':on_sale,'sold_in_90':sold_in_90,'yesterday_check_num':yesterday_check_num}) @gen.coroutine
def shanghai_data_process():
'''
获取上海各个区的数据
:return:
'''
start_time = time.time()
chenjiao_page = "http://sh.lianjia.com/chengjiao/"
ershoufang_page = "http://sh.lianjia.com/ershoufang/"
dealed_house_num = yield get_total_dealed_house(chenjiao_page)
sh_area_dict = {
"all": "",
"pudongxinqu": "pudongxinqu/",
"minhang": "minhang/",
"baoshan": "baoshan/",
"xuhui": "xuhui/",
"putuo": "putuo/",
"yangpu": "yangpu/",
"changning": "changning/",
"songjiang": "songjiang/",
"jiading": "jiading/",
"huangpu": "huangpu/",
"jingan": "jingan/",
"zhabei": "zhabei/",
"hongkou": "hongkou/",
"qingpu": "qingpu/",
"fengxian": "fengxian/",
"jinshan": "jinshan/",
"chongming": "chongming/",
"shanghaizhoubian": "shanghaizhoubian/",
}
sh_online_data = {}
for key,value in sh_area_dict.items():
sh_online_data[key] = yield get_online_data(ershoufang_page+sh_area_dict[key])
print("dealed_house_num %s" %dealed_house_num)
for key,value in sh_online_data.items():
print(key,value) print("tornado time cost: %s" %(time.time()-start_time) ) if __name__=='__main__':
io_loop = ioloop.IOLoop.current()
io_loop.run_sync(shanghai_data_process)
tornado初版
start http://sh.lianjia.com/chengjiao/ 1480320585.879013
start http://sh.lianjia.com/ershoufang/jinshan/ 1480320586.575354
start http://sh.lianjia.com/ershoufang/chongming/ 1480320587.017322
start http://sh.lianjia.com/ershoufang/yangpu/ 1480320587.515317
start http://sh.lianjia.com/ershoufang/hongkou/ 1480320588.051793
start http://sh.lianjia.com/ershoufang/fengxian/ 1480320588.593865
start http://sh.lianjia.com/ershoufang/jiading/ 1480320589.134367
start http://sh.lianjia.com/ershoufang/qingpu/ 1480320589.6134
start http://sh.lianjia.com/ershoufang/pudongxinqu/ 1480320590.215136
start http://sh.lianjia.com/ershoufang/putuo/ 1480320590.696576
start http://sh.lianjia.com/ershoufang/zhabei/ 1480320591.34218
start http://sh.lianjia.com/ershoufang/changning/ 1480320591.935762
start http://sh.lianjia.com/ershoufang/xuhui/ 1480320592.5159
start http://sh.lianjia.com/ershoufang/minhang/ 1480320593.096085
start http://sh.lianjia.com/ershoufang/songjiang/ 1480320593.749226
start http://sh.lianjia.com/ershoufang/ 1480320594.306287
start http://sh.lianjia.com/ershoufang/shanghaizhoubian/ 1480320594.807418
start http://sh.lianjia.com/ershoufang/huangpu/ 1480320595.2744
start http://sh.lianjia.com/ershoufang/jingan/ 1480320595.850909
start http://sh.lianjia.com/ershoufang/baoshan/ 1480320596.368479
dealed_house_num 51691
jinshan {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
yangpu {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
hongkou {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
fengxian {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
chongming {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
pudongxinqu {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
putuo {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
zhabei {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
changning {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
baoshan {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
xuhui {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
minhang {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
songjiang {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
all {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
shanghaizhoubian {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
jingan {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
jiading {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
qingpu {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
huangpu {'yesterday_check_num': '', 'on_sale': '', 'avg_price': '', 'sold_in_90': ''}
tornado time cost: 10.953541040420532
初版运行结果
存储数据到数据库中
这里我使用的是mysql数据库，那么在tornado中可以使用pymysql来连接数据库，并且我这里使用了sqlalchemy来完成程序中的DML。
sqlalchemy部分的内容详见这里。
1）表结构
这里需要的表不是很多，如下：
sh_area //上海区域表，存放上海各个区域
aaarticlea/png;base64,iVBORw0KGgoAAAANSUhEUgAAAcUAAABlCAIAAACzyG+JAAAAAXNSR0IArs4c6QAAAdVpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IlhNUCBDb3JlIDUuNC4wIj4KICAgPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICAgICAgPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIKICAgICAgICAgICAgeG1sbnM6dGlmZj0iaHR0cDovL25zLmFkb2JlLmNvbS90aWZmLzEuMC8iPgogICAgICAgICA8dGlmZjpDb21wcmVzc2lvbj4xPC90aWZmOkNvbXByZXNzaW9uPgogICAgICAgICA8dGlmZjpQaG90b21ldHJpY0ludGVycHJldGF0aW9uPjI8L3RpZmY6UGhvdG9tZXRyaWNJbnRlcnByZXRhdGlvbj4KICAgICAgICAgPHRpZmY6T3JpZW50YXRpb24+MTwvdGlmZjpPcmllbnRhdGlvbj4KICAgICAgPC9yZGY6RGVzY3JpcHRpb24+CiAgIDwvcmRmOlJERj4KPC94OnhtcG1ldGE+Cjl0tmoAAC5ESURBVHgB7V0HXBTH95+DE4E7mqCniCAgFrBj7xo1iD3GbtS/ii2J+osxthST2GLU2MXeEBWNJRoLaGJHBFGMigVEFI6jl+t1/7O71zj2uD3ujjtwh8+Hm3s78973fWd2bnZ29i0NQRBAJYoBigGKAYoBkxmwM1kDpYBigGKAYoBiAGWAGk+pfkAxQDFAMWAeBqjx1Dw8UlooBigGKAao8ZTqAxQDFAMUA+ZhQDmexmDJPCqJtET98c18mJZslxIdJS9LvHxqe/Ql8uXNUFJevGrhogyuzAyqKBUUAxQDtZoBzfx03LhxlvM0pMuwzo0ddp+LM3EzQVb8jcu371kOJ4FmRHhu29ZcnpzgECWiGKAYoBjQYoCulUezBeyM7PzSp4kpnQYOTL0X6xLYa1C3oCJ2RmYp0qFVAFoCET9OfNwkONSLWQfIRHdjr77M4/sFBPbs3sW5DjY6S/lXL1/i09x6dW2TVypr09wPVurQc0BjxyKw7y2qAUupTx+zAoPrMeqqBJV9yoUl586ekzk3lIgUWuUUybeuJr9g+3bsPrhriFJOCAmAYk76xSt37J1cevYf2JTlpqWkfFZPdWlJdsyNG8CtxZjhfezL19D+lp+V9uB+YpFQEdKld6dWvvghRMJNTHkb3KJR3PlYhXvDQWH9XR2gDgLwhNWhkoyXT/PLkLad2znStK1ReYoBigEbYwDuP4Xp1KlTeObEilkMVlB4x9YQZvioEQCEvBfKX8bsAIwgtkAOy3AenwbA+0mBCEEUKyZ08Q4MnjN3dt+WAdvvsjENokXd2nj5B08aNQBqaDJ4Dq4W/s9NOg38hotV33//Eir3mhTx9c2kV6jeSpKsZHj7QN/g9pNH9YI6w+Ysw8qK13zRmeEdOGvWdB93sOAPHD8hJCTprwNeAPQfNXHqhGFeAzWQKtgkqi7NbgutMry/mDrRHYD5269UqKUWiIawmOFjJsz6vwmw5Lojl/ADYk48AB4+vp79ho3u0azp9rvZCEIInrg6VDJ3aDMAgrLU3KkNUhmKAYoBW2IA4GDU4+nZ7yPC565Axz7fIQKEH+YH4tkiRFYywMdp7ZHHsPDGeSN7TFyI1RIObwZ+OfcMy0slUnRULHkFFze97r0rQ0fb8T2bh83FjqL/dMZTBJGnP73/7VczvN0ZTVv2Wbf1UCFfqi6snXn7zw4A/F8Vw+FEOntYED6eshMPAsC8n14gEAhSr28HIJSD1iaABIWjAuqNX7Ef11mYn6+tvHyeqDo2nm69+AKWjFw8ufOEReWraH9TlHF5UomILxCcXxfRPHwefkycmwAH5BV7YtGvcrFQLNMDnrg6rLRsRr+mzT5lS3B91H+KAYoBG2VAs34Kz3k81XNp7OjqBmh0e2Dv6IjJ7N0mzxq7PzpSIc6JjrnwxYwFmLTugEGdfxzdut/Q0f9bsqVUgi6Ncl69Ap4dO/q5AEAb1qsbrlDPf7uANt1/334g6dGdZi4fli/8vzMp+YQlS3OzvPv0CXJ3AIAe1r4fXiYn7TUAvME9Qvz9/ftNXs1gvMotgbeMCCABadGbt0XTp4zBK9bzglNVfYmoOla2W+dA+Nm1UxMeT6yvMpDzNn451d3BkeXvP3n1vvLLAg1nThmIVrRzcHSwJwavv/q6A/9mvLnaqI5ey9QBigGKAVtggGA8daZ7OrrprjBOiJhXdC3mRNSuV4xPpvQPwKDTFu168OHNsynhPa/FrP/fD/ug0IFRB8jhH5q4igo383ml6vVPEbfw2N7NowaEegd+4t46LC7+2cxujbB6uv8QGcLniXGdPKnyPju6kOgXns9RJh6P29YLrgUTQAI0tGxuMV9XL8F3oupYMYVCeSNNLtN7Ry3938O/XHt5L7OAy+FcWrtIrqqCKXB3omvWPgnBV1K9gPMhLS1dKNdrmsAVSkQxQDFQ7QwQjKeEGJy9u4wfXm/KrNWfz5jMVE69ZLfirrj7NJ85e16PNt5CCQ9WZLXoxCyJPXY2VS4s3HXooraqen6BoPC/vxLTysrK4E2lX5ZM+umPC90++zqvjH364K6B3ULsNQOOdj3QtEvf0uRr0fffiUqzd1yA6wloatFrhHvm5V1Rd6Eqfgnnzyu4nAASoLN6hwdu+nZVHk8ik/DOx17ANRD9J6pOVI5QJuaWwfmnuytDUJCxfF8MYRlcSAi+kuo/zBoQFNTsca6kEp3UIYoBigGrM6A7niIKOKph42VdOpwOIfbwiVR8nLOb9dnncMly1ozxKtCKoysiXJwcXDxY116BiK+mQ7lz4+67f583b0ywl7dfDq3cJS/dq92eHwfP6xbk5tY1RwKW/xaTlnpr2VfT67vgawoqrRU+PZp/uvfn8Gk9/Z3cfSQSBg1B8Tj7dD8evWXr/3rb02hMj0YHLz3H6hFAgmPcusizvoI7LJe6deq6rI9+WsGCWkBYnQanxnRskquwB3Staaa6Gp5p+emMId7ipk3qN6jftoGfP91O+ftAo0GS6SivqkQIXl91uBNAKCwFoKGHk+5mDJU+6pNigGLAJhigwXVdCARu54f/K9+CumfpmPUpHmlX92sNk4iAx5fIZC5u7tqzS25xvphWN+vqb+OPFL26stssjvLKSmh0R4ZzucFXLhWV8UQOjk4MJ/W+K2JI8KehtKSURqe7MJlaI1tFaPqqa5WUCV+kpunsR2W4sQJ8GwBEXlrGpTs4M5zgaq+BRACeqLqCn+bDDAqds+Vi5EIDGqnDFAMUA1ZlQDnlqXwklRU9Cx85I+5u4qFLr7UGUwic5sxkOldwwMWjPrwh9UGmkCvg5bh5EtMV7kHSTfZ1HD08yo2w+iBBuZs7gQZdjXo80i4mK3k5bvg4efnZYujQiKiN3wCavZsbGSuoPgLwRNXhltuZ338/+evZ2hioPMUAxYANMqCcnxpAJhM8/e+VW8Omfo08DJTUOlyWn52WL+4YjN+80jpAZSkGKAYoBmojA+TG09roOeUTxQDFAMWAeRnQvR9lXu2UNooBigGKgY+HAWo8/XjamvKUYoBiwLIMUOOpZfmltFMMUAx8PAwQj6dyfua3S9ZydfYE2R4rNQWndZmzQZasC8m61i3RGWqfR5ZgyUSdZEgmvh8lyXtYlzWNLU5tZHgbpYkgTapeU3Ca5KTJlW2QJetCsq51k9uTQEHt84jASWuLyJCs+8gNtyiXU8SVFb4DoORNehrfXuHp7ecIRKV8kWYbPIK4uNenyXhWFMoFBTUCJ8WSTrexbsNZ17olOkPt88gSLJmok5BkD6b6GSKtkV4n7tWZtXNYLJYPtvXdw6Mhi8XYEJ0Mg6Jq1UCz+++zrSusKTgplrR7Duw21m0461q3RGeofR5ZgiUTdRKSrDNy4l+Jr/eleQ8dWNM+iFN9bPt6v6bg1B5Tqj9vgyxZF5J1rVuiA9Q+jyzBkok6yZBMfD8KQWPNyYmPmQjKrNVrCk6zOm20MhtkybqQrGvd6PYjUaH2eUTC6eouQoZk4jGzjmerOzej6tv25BTSWVNwVnfLl7dngyxZF5J1rZdvHPN8q30emYcXs2ohQzLx9b5ZYVDKKAYoBigGPgoGiOenH4XrlJMUAxQDFANmZYAaT81KJ6WMYoBi4CNmgBpPP+LGp1ynGKAYMCsD1HhqVjopZRQDFAMfMQN6xlN50fLZ3xYp3yWql57UlKQEVYpPSBZp3r8pO7Dmm3tv4VuPKktRf3wzH6Yl2yu8BxWtlXj51PZo5dv39GohgfPlUw1IHOw7dpFehbXygGGWpC//S3me9gH3XlCc+yQltZLGJ2w4QqFeOg1DAvwCdnzyc+ULHhDx44SE15kcvQqNOkDSekJCfHz8s1eZ5F4yITm5d+PCrxeu3HXSKCywsHHUEWon4RHAOFSdrwmJT55rzldCnZRQhwFSJBNu85dmtwUt2WLCY2qhaEQzwPJuHoql9u2H5UjUh8RzPgm+8ChP/Z0wk3z3xsHVC0HgcEI7Z7+PCJuzjLCiRmgYp/ir8M4QYJsg+DJqZgcM6k87/9Jo+BhyBlmSZLVEu47/mxK0CZOOLQMgKIuwVTC6CBuOUKiXXYOQECQZwvAeKkJVSDZ93YPh0zruTYFehUYdIGkdMLv17gFfXxMS+tmbPEHlFl5f3Qbf7rv6jy1RJ/+uvGTFo/qomzciZNnR5IrlCSQkPEIk2bCVvX2UJ2zowPn6Wxi1YIR1AkC1UUSCZN3n93VG5Mq/2tmDGetPrv2ig3YxfhHn1XvO3A3HfJq5ackVybeuJr9g+3bsPrhrCC7v0HNAY8cisO+tVjEgF5acO3tO5twQvjdJW17VvMP2vx/CuvmPzzQYG/Ug6Ty+p7aInZFZinRoFYCqhb/biY+bBId6MRSPE595NXC6df2+Z8vOQ3q1UxklAK86VCs+aTQHJgC8jDXbog/9MM0e7RT26JULIk559LxZuw6MOjRufnZGkaxtCz8oJmw4QqEp7NAgjDoQhWLPykmLYwqv3L49sJknplC3OYhbk1nHVOuBn9y6fV6Snz54YP85SzbfOLyS0DrGUsp/t5Ib+LT6tE9vupPyBWL5WWkP7icWCRUhXXp3auWL1iXNJ5ybv3jPzs3LE6Y+TUqSMz0btfRvbIo7aF0agJ3/9z/vTOrSQK2qmJOZlsMP7RAMiRYW5z59nd22U0dFMbF1RMJNTHkb3KJR3PlYhXvDQWH9XR3sCdxUa//4MuhZY0oqysh4haWMD3m4nqzb/6xds6p/aOiV54UqzZK1U7v1mbQgIfnezLDWC7fEqOQVPuWlo3t0Wrx226Wja6dtPFDhcNUFCBycZZoBOv/e5Y6dw3KEqCQ35WLHrmOyxQogK5zUtZNv60H/PLg7tXf7pZFnMXukwVcdnQ3U5IHx8yMOb12fzdeK0igrnNA5NCVfAvGlXdsxesG66gbqSIv6fcHcvY8vxN0Ia46PAgTNQdyapmPFOgyzfuDaiM//uXCjDO0sBNaBgrdn85bjtxLy0lM2w3TmOmZZPK1Th8iYv+7eujQo2G/90b9RIWk+s+/f+WPT5icZ+Q/+PoqqjL2H6TTDvzdv0vD0Lhs9Ye3luVMGh2w4fgcgwtlDBsz/9QzdHuizLi1+3rVL/1ZtgredPrt5ccTRxFwAiNw0A8waq4J4Xk5iZosgolEtANPDwx9LoRPma6mSjG4Boh/k4BJ24kF4rX0/vUAgEKRe3w5AKEeqLJubdBr4aa733/6zA151viqGFyLS2cOCzHG9T2wIkZUM8HFae+QxPLxx3sgeExei5VCvwdpj6BXWv5FLQaNPuXKkEvBolRqRDLYm5viKo/HjQxrP3XQy+QS83sdWezB5PBu95n58YlnzsLlqd3UaDpcTCtVVymUMQsIs4mdVQPtxPLmyNnFzELZmOXvlv5C0ruqZWfePQULgAgixdUz34xMrmofP0zKjKOPypBIRXyA4vy5CechIPr8cEbLCjNf7mHUmkwkDHsHUYagS7dPz8I3u/vO/CPNsFvqhVKZ2oaJ1cW4CbJEVe2LRMnKxUAwLE7mpVlHLMiS6jUnX+5DcrzZfXze9Y4VfE/T5f3XKSXsNLyYH9whhYCIGg59bImN5EZguzc3y7tMnyB29KA9r329vvlqHuTP2bpNnjV0THbl0/E/RMRciTqarDfT/BF2O6DRwIJh7oVQGckmDV2uooRmF3GHZj4s7LN464bd+NuNCw+1R2zbPGbd085gd346DqPT0Jb2taRZHhFy0I8LAg3qs4z1ZAbR7vZy38cvpm46epbFYCC/Xt++8KiCBb1tH5NpnUhV06FbZe/3NxK4NtaVtRs5dPOv8pv1X99z74OOqeR+8HusNZ04ZiFa3c3CEp6mca7qb2mBqet7U631NdEv9TKBl/MLzOcrE43Hbag+mvFL1SikiQ/g8MX7ByZNWcodZvzHSRyZEzCu6FnMiatcrxidT+geo68lkqF2JUAD/Q+QGwKur1fwM9LT95xHDnNOX74xVeYN6b0dD/+eyUULKJa2G08gJhZrDRuYCu86ePHbL1pU7lyz860UBrKyvOfS1ppH2iIvff/gctGrp6aDXesVq6f8e/uXay3uZBVwO59LaRXLlUGs8nwSzjorWTJKIitLiYuOhipOHT+oO3gTW3Z3oqBd40uOm6vDH92nqeFqRMZlUIpOKJSIgkUvFYjHsSC16jXDPvLwr6i5cgOKXcP68otkFVc8vEBT+91diWlkZujzVtEvf0uRr0fffiUqzd1zQFKtoxXSJs3eX8cPrTZm1+vMZk5maX2WwdWeUHEgP79nN6h4KI8JUAt50DDanwY65YvmC/54+AgwcGnprKu5qskxQEBmNrQCqEOs0HC4mFKpqVOkTW8EcMfP7yZ3dFk6aWSbX2xz6WrNKVlWVeKLcvNzTR7b9tOnAjGljYfRg8p1BzC2Dv0TurgxBQcbyfeobBsbxSbNHXifeFgiFUpnWorYKXdU+3+aws7D0PouDKRV8M3QUrfWcwvcPM6OX/Hb8rlotGet63FTr+PgyxEscJFYKEEQ8uo3T8mPoEqR2mjushTaLxx9y4NG/o7f4einF4fPXa5WX7/lxPLy3jK9PwVWZvT9Pxsu1D2g6ZPZyrZJEWVI40Yq5j2IqbsxKPLQULuzeec9XqsYWmEJ8fVEAXoFn4t7icv3glfVs/cMgS5IcuHC8IvoJ6oikqF9TVwDasrHdb4fWTMGagzksrGfwMO31QZ2GwzkgFBLRYxASgjw5sxoEjsT2SyElGXd8AJj1xwmoS19z6LYmkVmljKR1zHOWT7ufth1XbwXUZ/1RzLLWwzS3EOR89pAOQYDJZADmiKE9g1VLq0bxmXTxSJsgdEvD0LnmOBGkbHQlS5OawyaO2bHQ3bddWiG6dSrhxCZ4IfmIrdwZVtG6JC8RgNZZai7g6arHzUq4r8GHSHQbQOweiZrEFfVLZRJhUVExT4CfIPrLIQi3tJjHF1ZWQn3MNJyR333W9NOZmhV4VBtIyBWUFhcLRVq9BkHIg1dDs6GMaSyVlZSQaTXj/DUNEmFz6LZmJYAsYJ3AmkJWUgI7PDpUaScb5FMbntF5PW4arcf2K5DoNgQLJJofMLPm7Os4eng4klHJdFXu4CNTuGplZEXPwkfOiLubeOjSa821PgLEcE+LzM61gS4A8uCrhseWa7m4ae8jtgmkOs1B3JoWQ6pjXa8dmr2bm25HgoVtkE+9LpA5oMdNMlVrXxk946mdy6TvpjprRhpbdbyqOOmuARu37XFr2NSvkYfGN7rn6UfJAQ3gQlntSlVlyYIsmBUScWtWgt6s1iuxU32Hap9H1ccdaUskSKbiSZNmkypIMUAxQDFQKQPmv79fqTnqIMUAxQDFQK1lgBpPa23TUo5RDFAMVDMD1HhazYRT5igGKAZqLQPE46mcn/ntkrVcs20ithR9NQWnpfwnp9cGWbIuJOtaJ9doxpWqfR4Z53+1lCZDMvH9KEnew7qsaWxxaiM8vF21wK2CkZqCswqumbGKDbJkXUjWtW7GllWrqn0eqV2znQwZknX3S3GLcjlFXFnhOwBK3qSn8e0Vnt5+jkBUyhdpntpFEBf3+jQZz4pCuaCgRuCkWNLpNtZtOOtat0RnqH0eWYIlE3USkuzBJNpYqfNUwpm1c2AsLx93dB+yh0dDFouxITr5xIpZOr8S+++zrSusKTgplrR7Duw21m0461q3RGeofR5ZgiUTdRKSrDNy4l+Jr/eleQ8dWNM+iFN9bPt6v6bg1B5Tqj9vgyxZF5J1rVuiA9Q+jyzBkok6yZBMfD8KQcOLyYmPmQjKrNVrCk6zOm20MhtkybqQrGvd6PYjUaH2eUTC6eouQoZk4jGzjmerOzejYLQ6G081Bad1abRBlqwLybrWLdEZap9HlmDJRJ1kSCa+3jfRMFWdYoBigGLgI2SAeH76ERJBuUwxQDFAMWAiA9R4aiKBVHWKAYoBigElA9R4SnUFigGKAYoB8zBAjadkeRQLuIVlfLKltcohMhEnr1BLQGUpBigGaicDesZTedHy2d8WkXrBqOzAmm/uvS2tSE/i5VPboy37Tj1gDpwVkRNJpF+N6rzuyFXtQ2Je8dMnTwp4ksqFCnH+gLZep5JztYtVa94wS9KX/6U8T/uAoxIU5z5JSVU2PiK+cf7E7t27458rj5oHuWFIACDixwnK9CaTg9sVl+TGx6PCxEfJJXz4OgUsIZKUhw9UZRNepmcr5fo+DFsnJkRclhefkCLF1PIL2PHJz+FLJAFQPE9+wC7Q+a0lFOoDZLLcsEcmmzBagd6RwWhNNlKBDMmEu/wREm9KUVUUz/kk+MKjPNVXzefZ7yPC5izTfLdEzhw4dXDNGxGy7GiyjjD30THg3iJLqFDKFYLIX+f7YG8YPHqfXZkQO3Zg1dTQsQt0dFbfV4MsSbJaol3W/00J+taspGPLAAjKQt97JPlleCcGK/CLicPgiwv3ntN992LVXTAICTWeDVH5+LXuEgpf6wX6z94gRRDOwxMQSedu3XwasQDD+9jFZygGKWdkaIc2QY3goQ6hobN/3m4AmEHregjBrOPMIMmQJe+h2NvQRCOagWUHE8sbJRSWL2LGbwY9Mt4W4YlgjBq9I4MxSixe1gg3SZCsZ35K7geBX8RJfvJi7oZjPZppXjEkF5acOX7o5LkrXBH2401OlUVLEeCEc5+HSfl5WTFR+2Iu3sYDacEZR2JyUm5eHif1aVJS0ssMzTTnwM6oEeMiGjuqHkaXlcZefPrrrisD/QDdrlIh5tj46RHpp6Of5KnmUxb1tgrKaTQH9B2zGWu2RcMP9KXGwB72jLK0mz9eTNp7+e7R6Itb5w74fv1GUlcsVQBAWIUG4Abo32JuJCSlJJ/Y9O/e75JzJDQ7Ogj85G58/Iesd0s+9fp19060Kp11Pin5xqlt8NCDpKQ9P35FqM8IoR5CUOsYM1AVDWbrKNvezh7QaLqnEqHQCAwWKJqflXYx5sSRI8eTUt8r1SPilKRkvhQ+vwO4+dlPX2XCjN4TQSGKO38i6ujRhGeq6npAkjzj8NrFnPSjhw4fP/nnu1zlZS4i4T5MTOGV5Z07GvXnX9fLJPgJqki+dXn/7v2xCc/xigXsjJSUJ8f2H0l9l332+KG4B29UcHRLotc6xpzvKj3GfxKP/yRGYljx5bnjYz4fCR/118zRZCXD2wf6BrefPKoXxGIj81MCnNjcB05wvpg6EeKfv/0KdOfVhZMTJ00MYIGWbQdMnDjx18hTKnKkE7rUW3ooQfVV/Ska1QJEP8hRf8cyREJF2UAvsP/mu/Ilq+ubwdZEC4Dx8yOAZ8ssnuzxCTg/bckWI49gxqt/iRzF+fbGdtU7vc0B2yAkaARDhdMreH8HdqdzKQW5SafV7/3e/s3EwLC5ajToIb/hum8TVR/Wzhi0rocQ1ATGDFSGsqQ0h7b48kOPtC0gCKGwfBEzfjPoEWpLNITFDB8zYdb/TYB9ft2RS6gMOxHuZaPzbDjjDhg8B2b0nAiiRd3aePm3nDH1c9gWm6Nvo9X1JJJnHKyd9NcBeJnXf9TEqROGeQ1ErcMk5sTD8CE+vp79ho3u0azp9rvZULbmi84M78BZs6b7uIMFf6DnJnwqn8EKCu/YGuIJHzUCgJD3QthZCUribpI+3zEQFf+RIBmdilQ5tRg16cyosZ+1dFDP0TJuR118onhVnNDc3Y4xPNjAr1iVDRtZsSJOgM19tp68vmBYqz317Q7cuwa+Cms+Ynz0iPFfjXzq9vnGNV900BhR8PLSi8JbeGskxuZojvWb1kvLKAR9/YytWm3lA7vNGH/r8uo9Z2arHC3LEwEG0wmbeDnURT8QdB5TrenO5Qv0t4ojm3cAVnCnZvVAKgCFL1auXCkvZh/afWLp4YuWQ1OREMvZqhbNDqfSOE516RKZYlhzl+9O/b1s6lD8RMCn2XDGjZ/IhCdCSWrslgc5d9LTewW49g6atmjjrq8n9tY3fJA84wAQrV605JMV+0+umQkZKCooUPKATvaLp66MXTN7EFBIRDL7nKSjK4+l3k9/176Rc+aE0FYDN6z4alxdO1rf0WMPzerA+uzgmXMnPmvKyC6W0rOjK5ZkGXW+m9AY2LliQn24GK99ipXmZnn36RPkDi/U6GHt+5mk2MyVy+HEdXfrHAgzXTs14fE0V+IKBUDk2j4BYFfXyQMUcDVljIeGiIRF6BWiDSeF3GHZj4sj/9haplCuYLjUcwAqzHTHehA7TbW2UW1+3Lp67tTpOFbowPjb//g4K83LpLLstMclPu3GDR9kOSQVCaHRyncMy9m2hGY5b+OXU90dHFn+/pNX77M3ZELnRMhJSwV+3bsEuMJ64UOGl77NKDUQb57EGSctevO2aPqUMTiWel7YHQklsIYzpwxEs3YOjg72OWmvAeAN7hHi7+/fb/JqBuNVbgm6+FTPpbGjqxtcfLEH9o7Y2+j1lYSFyZ7vqNUqJtXpUsXqutUQGcLniSHPUC9PWq2rbbpQSHxXqFpcLit/nuiyUre+T+OsVzlgMDr+ViUpuDnZYGyLxlWpW1114FjV/vOIYT+sX75TuWPBwckVcLJhv21AB/lZHwBwopv++2ukOyu3H53UpUG5Sp7Bv234zUHxg7xD8IIVkZciF5Y7ar4vFQlBECjT7RzmM2hZTen/Hv7l2ssnmQXtfD1vbvvfnCsizB76E2WH/U7msgW6CLR8dXJxBjyxBAEONCAUlgE6E2aMTbpnHG63GG6NQAOElk/uTnSNATTnF57/7m9szFQWTAfAme7p6Ka5eQMPEJYE2FCka11tT8tNtaxqGZPOD5lUIpOKJSIgkUvFYjEcnZp26VuafC36/jtRafaOC5eqhsnstSrirMQEzR55nXhbIBRKZerfX7teffrfvBWjXQsG3s7OZvNEIDcvJys7F7/1RiiEtdiP/noAgnqFaP/8aiuzmbwdc8XyBf89fQQYKKSWA0ay+Mlb952ViYp/WXuw/djeLPP1PJI+05AKdzXxn2k75q8/Lvl7z4bHOUIAZJysLE5+PkBEWVnsvGIeSeWGi5UnBJ0KgXdXbr+WSwRxN5+wvN3V508pOzsLT++zhFIlZkKhYaOWKSHmlsGR092VISjIWL5P3ZnRu49xV5NlgoLI6L+1LeucCA2ad3IvjN176hEi5e/asCMgNIChdl67GpYne8bRWb3DAzd9uyqPJ5FJeOdjL1TQpBS06DXCPfPyrqi7kFl+CefPK3rHFvIlcdU6buoDQFZecdUVlZBYeYWl5g5roW3m+EMOgsj3/jwZF7YPaDpk9nJi/eaSVhmnlB0CQGIOevfiUcyy4GHz1IiSLh5pE+QJXRg6VwNekPOQBRomcYTqYivmhWv5HsJGNxohhEIoXzNz0OgvDW3iUas2e8YgS5IceD9qRfQT1LKkqF9TeE3XFvfo4uE16J1/uG+peYdH74rNBs0gJGgJa6NTibBTaVLuoxj1/ShEwRvZquGERbvg3YZgrcYw3OsMWtdLiHzfTxNwU+7eLU7HpmHIRCO0zcPbs4m58OYPkVDjiJlzBj2CZyafPaRDEGAyGYA5YmjP4HBlnz+0ZgrmEXNYWM/KT4SYXcvxzsAKaHflRbl20XGHYGTQc8YVv08Z2k45jHSd9guuR5KXCEDrLOycUmv+O3qLr2pCEj5/PZT/uWL27OWnxbkJoPloESIaGQwS4F1UBKlYEu9L5M93tdFyGRIk64kvJWO3q/PJ1aq+P4pXVkKjOzKctefmWv3djFnTcBoFJPKHGa8ch/2x8jOjasHC8pLXfXtO3fXvzbYNLE8IITjTWBLySoVSORMupppxcmoaJEIvjRCaZl0s5AklCgbTpY695oLUCOuWKErSI0ReWsalOzgznMrF4uSWlsJVSoZTXYPQRHwu7AwMFzcH6LtM+CI1TX0Rh9dluLECfMuvzxhUCpDSklIane7CZFZOqFwqKuOJHBydDEIlX9IwOnUJEiSb8RRRmwVM14qrIZqjNTQ399eDVUNu79787vMHVatrC7WcmG5OtoDDZjDUdWLWraGM0Ozd3AjOTZfyS5CVMO3IcFFPCmQlL8cNHyd3KjeGhA6NiNr4TSUaiA7R3LAXLBEdKiezr+Po4aG2X+6QzhfyJXUqmvi1HBcaXXYuk76b6mzwFqCmgpVyNQWnlehRmrVBlqwLybrWLdEZrOER3avDs3fqLfSW8MrGdJIgWc/1vo05QsGhGKAYoBiwfQb036Kzfew1HyEVs6rmtyHlAcWAhgE94ymZSCoaJdbLWQunvHjVwkUZXBM32BLErMp89mD3/v3nL8UV85WbQHFySznvE+ITudhz1lBiXMwqEiylpjx88yFP1ZCKZ0kP0tiF2hGe0JBOT57j23QRqfDfv8/uP3Ik7nY8TyhV1TLmkwQkY9QZWdawdSq+lJGUfgzFDXcb9AlCokRiZwBRtWqXWQsnahfEs7HoQlV1WjdmFYKc2/8T7JaDRozr3zFg6c4EjWIF7/P2/vAQ/pw1LjciZhUJlnZ+O8Kz+ZBSGMEJQd7dPQQDNcUk5eBPPXv7NA/F08D56G4USeGg9oFefkFTv5jczLdRxPIoHI9x/0lAMk6hUaUNWqfiSxnF50dS2GC3QRA996PI/NrAkC2J//k0bfhv7BXg1mLM8D747SsYw+bB/cQioSKkS+9OrXyhJhgGJju/9GliSqeBA1PvxboE9hrULQizAMPAXE1+wfbt2H1wV7gf1CKpIDudLaC3DfJDtcv4SY9eNO/QydWBVhEnPA4D2ySmvA1u0SjufKzCveGgsP6uDqhbMATOxSt37J1cevYf2JSlfB5DWpIdc+NG5b5XolMnZpWcm75w1s/rjtxcNrUvnIBKVVNRqOFs5M/PpCwmyFCHsoJCGLNqcdPRT/I2tG9geJsLLF95mv3Tth3Hmq45EPfbnJ7Lv1w1ZMbPY0MbAhkbbqv5/c872k8ovYuPjstyyEx76utGBwpRdi63cs018igeX4qHBtw69MM0dcCtmh5fqka2RY0Cred6n4wPssJJXTs3COh6Ke7fOSP6LthxFaskntapQ2TMX3dvXRoU7Lf+KPrExfWda3t+Ou7k7o3B/j4Hzpwf3H30BzSUn2Tt1G59Ji1ISL43M6z1wi0xZGxWoUzmzT/b9Z2GP2uccedY57DJXGgcEOCEUmnx865d+rdqE7zt9NnNiyPgxmwofHTxYPNGzQ7/FRt74XDnKUvVGPp07m3Qd3064dD+9Fliq+691dryXjx4z+odzMhcuvS7bQdjFPCJZCzxOY8XLzl5IPo3nYd+GL7tOnkVPErlqDWYkqEz/XasW7Xhp0WXI/84kemw5fev1NrevEnD07tsbEEAslfAe/r2A1rAzrFxo/rqkrUqw0MDbh3euj6br7PDslZ5STljZgaIp+okZrZ4OLWtF19ADZGLJ3eesAhTpSjj8qQSEV8gOL8uojn2DAYMLB0+dwUa68x3iADhh/mhV8rsxIPwovJ+eoFAIEi9vh2AUA52sUmMR5+UBE6FICuAAfbd/QB1/DxrUNgs9MkKBCHACaXosxbwSaE9sWgRuVgoliGIcFRAvfEr9qMSBCnMz0c/ULuAjO+wLJFOqLx4gCeMc4iiwhMaHA8A31ahy5YubsYCw2fjz1NJvhne5otfjuFKyq8wSCZ2qrdMN0ycSp32JwmW0OJy7pi2TSCGxTvOKGtjbjKZTBaWOgxFn6hRiAsmd0afB2L5tJ4+b9GHQs0zY8paZD5IQiKjqgplDFrHHF9xNH58SOO5m04mqwIY1vB4fVVgiqqixYDBboMgJsxP4SlVMWSLnhg2RoWBwRSb7R/NqfH0iYMuRB1HRDnHzsSNmY49DqsHJ2a1XGAboD8Ejm64GvI6oZkKMatg+GEY8/H41Zvr1m88vem3izGnyxTgyenIzUnOaxaNgUM7PCwSwDcEYBjRf+aOWWXHXLdpMfDqt3KOMt4Pbmnv9TccLCVf2gUlNAfPqIcpb54lrlw0PvFKdN/xX9fW+Vttiy+FNyf135IMmDqe6oRswWPY3Mss4HI4l9YukqvOfr1hYPAzlcPh8bhtvUxYzK2Uo6mTP7t0+kJC7Kk0Zv/x3XxgWX04MTXlAtvgIepy0RA4uomk7wQ6UZEqZpVKa+PAznDOF9AQfUK6YUtfUFIEtw88f/IfyEnwdXV2bdoHyvs387jPUYUNxGJWtTFrzCo3j8aA4cIw3A70ZiGdvl78/ZVj695ef5hXpTv8Kr9t9xM++4gG3HJOX74zFkdZo+NL2S7RtQiZqeOpDhV6YtjolEK/GhsGhkAFaZFf7wl968aPnb16/NixLtiyJHmc8F0aJEPgGKETRa4bs8qrVSd/8HL3oTh4RR118Ihnp3YsBzDh551CLJVm3oZ1bqaX9GiovPtUbTGr3uawVYGTOHAqWpJ26/fdxwq4QomIe+bcWdDI193w+Eu6qWytoF25gFs1Or6UrVFbK/GYMp7S4NlFxyIYKuwBHQtW2PLTGUO8xU2b1G9Qv20DP3883DeCxifGhrG6dHi1itgDO0Bz9ul+PHrL1v/1tqfRmB6NDl56bkF+7d3Hfj42K7dw0sxxuBVCnPAQ9hYguiqeMl7Wbl3kWV/BHZZL3Tp1XdZHP8WkZH3XoxPVMWneguw/Tz/KxcNQApqTb2TM9n1zB9vZe646n75rzU9wmLKn13HEkpMT+sQ4w9lJfYv/8J4ToycvaGL2h4LLDY6om9+PDm2CJb8mfeFUlAbkUWuX1Hd1ruvk+kvM62N7f3GqPIgF6mtNS9g6GK0u2mm7T57fycsV8B2g565Bn+z7adjMQS3odRlrL2bs+BltI5hg/9+1cgTOUhO/JmdSCvQJseLUv9rLgNZyq1aWxMqrVunyWYWspKSYJ0C3KhpMMomwqAgWrupGzurCCW/DlBQXl3K5qhec6vHMGN+hit3f/9+i1X9q6xLyywqLikUSeB+ssiQrftUzuGtKLrl7QaawRIxCXlZaWlxUBN+cQXzcoNT8kAya1CpgmnWRgFtcUlp137WAmC1rmkdmg1G7FZEgudxsxDy/Gnpi2BAqt1YYGBSMMThhaVIhcIzTCSrGrHJ0dnF0JqSqnNDaMavsXFxhjNSPNNXg+FIfaYtVn9v2q1atIrRWJhJ3H9Tb0ZT1AEK95hbWFJzm9ts4fTbIknUhWde6cY1HrnTt84ic39VayiDJVHypam0PyhjFAMVALWbA5ueftZh7yjWKAYqB2sUANZ7WrvakvKEYoBiwHgPUeGo97inLFAMUA7WLAeLxVM7P/HbJWq7NP0hYU3Bat8/YIEvWhWRd65boDLXPI0uwZKJOMiQT34+S5D2sy5rGrur7TU3ETb56TcFJ3iNLlLRBlqwLybrWP5ImtoSb1tVJptvo7j/lFuVy4IPjhe/gg4Vv0tP49gpPbz9HICrlizRPwSCIi3t9moxnRaFcUFAjcFIs6XQb6zacda1bojPUPo8swZKJOglJ9mASxR3WeaLhzNo5MDabD/b6Vg+PhiwWY0N08okVs3R+GfbfZ1tXWFNwUixp9xzYbazbcNa1bonOUPs8sgRLJuokJFln5MS/El/vS/MeOrCmfRCn+sD47DacagpO61JogyxZF5J1rVuiM9Q+jyzBkok6yZBMfD8KQePsyYmPmQjKrNVrCk6zOm20MhtkybqQrGvd6PYjUaH2eUTC6eouQoZk4jGzjmerOzej6tv25BTSWVNwVnfLl7dngyxZF5J1rZdvHPN8q30emYcXs2ohQzLx9b5ZYVDKKAYoBigGPgoGiOenH4XrlJMUAxQDFANmZYAaT81KJ6WMYoBi4CNm4P8BInhaigc/g5MAAAAASUVORK5CYII=" alt="" />
sh_total_city_dealed //上海地区二手房总成交量
online_data //上海各区二手房数据
2) 使用sqlalchemy来初始化表
settings中设置的是数据库连接相关内容。
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
DB={
'connector':'mysql+pymysql://root:[email protected]:3306/devdb1',
'max_session':5
} engine = create_engine(DB['connector'], max_overflow= DB['max_session'], echo= False)
SessionCls = sessionmaker(bind=engine)
session = SessionCls()
settings.py
初始化脚本
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column,Integer,String,ForeignKey,DateTime import os,sys
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.append(BASE_DIR) from conf import settings Base = declarative_base() class SH_Area(Base):
__tablename__ = 'sh_area' # 表名
id = Column(Integer, primary_key=True)
name = Column(String(64)) class Online_Data(Base):
__tablename__ = 'online_data' # 表名
id = Column(Integer, primary_key=True)
sold_in_90 = Column(Integer)
avg_price = Column(Integer)
yesterday_check_num = Column(Integer)
on_sale = Column(Integer)
date = Column(DateTime)
belong_area = Column(Integer,ForeignKey('sh_area.id')) class SH_Total_city_dealed(Base):
__tablename__ = 'sh_total_city_dealed' # 表名
id = Column(Integer, primary_key=True)
dealed_house_num = Column(Integer)
date = Column(DateTime)
memo = Column(String(64),nullable=True) def db_init():
Base.metadata.create_all(settings.engine) # 创建表结构
for district in settings.sh_area_dict.keys():
item_obj = SH_Area(name = district)
settings.session.add(item_obj)
settings.session.commit() if __name__ == '__main__':
db_init()
database_init
图表绘制
1前端绘制
图表绘制的话，这里我使用的是Highcharts。图形比较美观，使用的时候只需要提供需要的数据即可。
我使用的是基础折线图，需要在前端引入几个js文件，如下：jquery.min.js，highcharts.js，exporting.js。然后添加一个div，使用id来标示这个div，样例中使用的是id="container"
官方js部分的代码如下：
$(function () {
$('#container').highcharts({
title: {
text: 'Monthly Average Temperature',
x: -20 //center
},
subtitle: {
text: 'Source: WorldClimate.com',
x: -20
},
xAxis: {
categories: ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun',
'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']
},
yAxis: {
title: {
text: 'Temperature (°C)'
},
plotLines: [{
value: 0,
width: 1,
color: '#808080'
}]
},
tooltip: {
valueSuffix: '°C'
},
legend: {
layout: 'vertical',
align: 'right',
verticalAlign: 'middle',
borderWidth: 0
},
series: [{
name: 'Tokyo',
data: [7.0, 6.9, 9.5, 14.5, 18.2, 21.5, 25.2, 26.5, 23.3, 18.3, 13.9, 9.6]
}, {
name: 'New York',
data: [-0.2, 0.8, 5.7, 11.3, 17.0, 22.0, 24.8, 24.1, 20.1, 14.1, 8.6, 2.5]
}, {
name: 'Berlin',
data: [-0.9, 0.6, 3.5, 8.4, 13.5, 17.0, 18.6, 17.9, 14.3, 9.0, 3.9, 1.0]
}, {
name: 'London',
data: [3.9, 4.2, 5.7, 8.5, 11.9, 15.2, 17.0, 16.6, 14.2, 10.3, 6.6, 4.8]
}]
});
});官方js
我的工作是在这个基础上，修改js内容来画出符合自己的图。
具体的参考github上代码中的修改，最后画出来的图是这样的。
2 后端获取数据并传输给前端
基本上前端表哥需要的数据是一维或者二维数组，比如横坐标时间数组[time1,time2,time3]，纵坐标数据数组[data1,data2,data3]这样子。
这里需要注意几点：
1）tornado后端返回数据，使用render()函数渲染到指定的页面即可。
2) js中使用{{ data_rendered }}来获取数据
3）后端传入前端的时间数据为timestamp时间戳，这里需要format一下显示，如下：
function formatDate(timestamp_v) {
var now = new Date(parseFloat(timestamp_v)*1000);
var year=now.getFullYear();
var month=now.getMonth()+1;
var date=now.getDate();
var hour=now.getHours();
var minute=now.getMinutes();
var second=now.getSeconds();
return year+"-"+month+"-"+date+" "+hour+":"+minute+":"+second;
};formatDate
4）注意js部分二维数组的定义处理
3 前端请求传给后端参数
因为需求中可以查询上海各个区的图表，那么可以设计访问地址为r'/view/(\w+)/(\w+)'，这样前面是city(比如sh,bj等)后面是具体的哪个区area。后端接收到这两个参数后去数据库中查找数据并返回。
最终成型
在数据库中有了数据之后，后面的内容就是前端后端数据的交互，在前端哪些地方绘制图表，需要什么数据，后端返回即可，最终主要的代码是这样的：
import re
from bs4 import BeautifulSoup
import datetime
import time
from tornado import httpclient,gen,ioloop,httpserver
from tornado import web
import tornado.options
import json import os,sys
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.append(BASE_DIR) from conf import settings
from database_init import Online_Data,SH_Total_city_dealed,SH_Area
from tornado.options import define,options define("port",default=8888,type=int) @gen.coroutine
def obtain_page_data(target_url):
response = yield httpclient.AsyncHTTPClient().fetch(target_url)
data = response.body.decode('utf8')
print("start %s %s" %(target_url,time.time())) raise gen.Return(data) @gen.coroutine
def get_total_dealed_house(target_url):
# 获取总的房屋成交量
page_data = yield obtain_page_data(target_url)
soup_obj = BeautifulSoup(page_data,"html.parser")
dealed_house = soup_obj.html.body.find('div', {'class': 'list-head'}).text
dealed_house_num = re.findall(r'\d+', dealed_house)[0] raise gen.Return(int(dealed_house_num)) @gen.coroutine
def get_online_data(target_url):
# 获取 城市挂牌均价，正在出售数量，90天内交易量，昨日看房次数
page_data = yield obtain_page_data(target_url)
soup_obj = BeautifulSoup(page_data, "html.parser")
online_data_str = soup_obj.html.body.find('div', {'class': 'secondcon'}).text
online_data = online_data_str.replace('\n', '')
avg_price, on_sale, _, sold_in_90, yesterday_check_num = re.findall(r'\d+', online_data) raise gen.Return({'avg_price':avg_price,'on_sale':on_sale,'sold_in_90':sold_in_90,'yesterday_check_num':yesterday_check_num}) @gen.coroutine
def shanghai_data_process():
'''
获取上海各个区的数据
:return:
'''
start_time = time.time()
chenjiao_page = "http://sh.lianjia.com/chengjiao/"
ershoufang_page = "http://sh.lianjia.com/ershoufang/"
dealed_house_num = yield get_total_dealed_house(chenjiao_page)
sh_online_data = {}
for key,value in settings.sh_area_dict.items():
sh_online_data[key] = yield get_online_data(ershoufang_page+settings.sh_area_dict[key])
print("dealed_house_num %s" %dealed_house_num)
for key,value in sh_online_data.items():
print(key,value) print("tornado time cost: %s" %(time.time()-start_time) ) #settings.session
update_date = datetime.datetime.now()
dealed_house_num_obj = SH_Total_city_dealed(dealed_house_num=dealed_house_num,
date = update_date)
settings.session.add(dealed_house_num_obj) for key,value in sh_online_data.items():
area_obj = settings.session.query(SH_Area).filter_by(name=key).first()
online_data_obj = Online_Data(sold_in_90 = value['sold_in_90'],
avg_price = value['avg_price'],
yesterday_check_num = value['yesterday_check_num'],
on_sale = value['on_sale'],
date = update_date,
belong_area = area_obj.id)
settings.session.add(online_data_obj)
settings.session.commit() class IndexHandler(web.RequestHandler):
def get(self, *args, **kwargs):
total_dealed_house_num = settings.session.query(SH_Total_city_dealed).all()
cata_list = []
data_list = []
for item in total_dealed_house_num:
cata_list.append(time.mktime(item.date.timetuple()))
data_list.append(item.dealed_house_num) area_id = settings.session.query(SH_Area).filter_by(name='all').first()
area_avg_price = settings.session.query(Online_Data).filter_by(belong_area = area_id.id).all()
area_date_list = []
area_data_list = []
area_on_sale_list = []
area_sold_in_90_list = []
area_yesterday_check_num = []
for item in area_avg_price:
area_date_list.append(time.mktime(item.date.timetuple()))
area_data_list.append(item.avg_price)
area_on_sale_list.append([time.mktime(item.date.timetuple()),item.on_sale])
area_sold_in_90_list.append(item.sold_in_90)
area_yesterday_check_num.append(item.yesterday_check_num)
self.render("index.html",cata_list=cata_list,
data_list=data_list,area_date_list = area_date_list,area_data_list = area_data_list,
area_on_sale_list = area_on_sale_list,area_sold_in_90_list=area_sold_in_90_list,
area_yesterday_check_num = area_yesterday_check_num,city="sh",area="all") class QueryHandler(web.RequestHandler):
def get(self,city,area): if city == "sh":
total_dealed_house_num = settings.session.query(SH_Total_city_dealed).all() cata_list = []
data_list = []
for item in total_dealed_house_num:
cata_list.append(time.mktime(item.date.timetuple()))
data_list.append(item.dealed_house_num) area_id = settings.session.query(SH_Area).filter_by(name=area).first()
area_avg_price = settings.session.query(Online_Data).filter_by(belong_area=area_id.id).all()
area_date_list = []
area_data_list = []
area_on_sale_list = []
area_sold_in_90_list = []
area_yesterday_check_num = []
for item in area_avg_price:
area_date_list.append(time.mktime(item.date.timetuple()))
area_data_list.append(item.avg_price)
area_on_sale_list.append([time.mktime(item.date.timetuple()), item.on_sale])
area_sold_in_90_list.append(item.sold_in_90)
area_yesterday_check_num.append(item.yesterday_check_num) self.render("index.html", cata_list=cata_list,
data_list=data_list, area_date_list=area_date_list, area_data_list=area_data_list,
area_on_sale_list=area_on_sale_list, area_sold_in_90_list=area_sold_in_90_list,
area_yesterday_check_num=area_yesterday_check_num,city=city,area=area)
else:
self.redirect("/") class MyApplication(web.Application):
def __init__(self):
handlers = [
(r'/',IndexHandler),
(r'/view/(\w+)/(\w+)',QueryHandler), ] settings = {
'static_path': os.path.join(os.path.dirname(os.path.dirname(__file__)), "static"),
'template_path': os.path.join(os.path.dirname(os.path.dirname(__file__)), "templates"),
} super(MyApplication,self).__init__(handlers,**settings) # ioloop.PeriodicCallback(f2s, 2000).start() if __name__=='__main__':
http_server = httpserver.HTTPServer(MyApplication())
http_server.listen(options.port)
ioloop.PeriodicCallback(shanghai_data_process,86400000).start() #毫秒 86400000
ioloop.IOLoop.instance().start()
data_collect
几点说明：
1 因为要定期去网页上获取数据，这里使用了ioloop.PeriodicCallback()函数来定时处理。
结合nginx部署
自己有一台AWS 的EC2虚机，操作系统是centos7，最后是要把程序放到上面去跑。
1 安装部署nginx
因为时间关系没有做过深入的研究，只是从网上翻了下几本的东西，如下：
1 使用wget下载nginx包(nginx-1.11.6.tar.gz)，并解压
2 进入nginx-1.11.6
3 ./configure
4 make
5 make install
配置文件修改/usr/local/nginx/conf/nginx.conf
reload nginx 使用 /usr/local/nginx/sbin/nginx -s reload
2 调整虚机的inbound 防火墙规则，我添加的是80端口(nginx配置文件中同样监听80端口)
1、登录到AWS console主界面
2、左侧INSTANCES-Instances
3、右侧group security
4、下面inbounds
5、edit
6、edit inbounds rules页面中自己添加规则
3 测试访问nginx
如果正常，会显示Welcome nginx的页面
4 运行tornadao代码后reload nginx
效果图以及代码
1 几个效果图如下：
2 代码放在github上
解决sqlalchemy session问题
在代码运行之后的几天发现，每隔大约半天的时间，程序虽然不会挂掉，但是在浏览器访问的时候会出现500 error。后台日志中也会报访问的错误。
仔细研究了下后台日志的报错，发现应该是浏览器使用旧的session信息来访问，但是session信息在程序中已经过期，所以导致错误。仔细审查了下代码，确实是在settings文件中初始化了一个session，然后后面所有的DB相关操作都用了这个session。显然是有问题的。
解决办法其实很简单，只要把数据库session的生命周期与http 每次request的生命周期放在一起即可。也就说在每次http request开始的时候初始化一个db session，然后在每次reqeust结束的时候close掉这个db session即可。可以参考下flask框架中这部分内容的介绍。
1 sqlalchemy部分
为了实现上述的说明，sqlalchemy 这边需要使用一个新的对象scoped_session，官方示例如下：
>>> from sqlalchemy.orm import scoped_session
>>> from sqlalchemy.orm import sessionmaker #创建session
>>> session_factory = sessionmaker(bind=some_engine)
>>> Session = scoped_session(session_factory) #关闭session
>>> Session.remove()
更多的说明参考这里。
2 tornado 部分
在RequestHandler中重写initialize()和on_finish()两个函数。initialize()函数中初始化db session，而在on_finish()的时候结束这个db session。BaseHandler是一个基础的handler，其他request handler 只需要继承 BaseHandler即可。
class BaseHandler(web.RequestHandler):
def initialize(self):
self.db_session = scoped_session(sessionmaker(bind=settings.engine))
self.db_query = self.db_session().query def on_finish(self):
self.db_session.remove()