问题描述
我在一个打开10个孩子的脚本中运行多线程".每个子脚本都会查询一个API,并将一些数据记录到json中.我记录了保存在另一个json中的每个json的日志,但是代码有时突然结束并破坏了json的完整性,因此无法重新打开它,例如:
I'm running 'multiple threads' in one script that open 10 children. Each child script queries an API and logs some data into a json. I make a log of each json saved in another json, however the code sometimes ends abruptly and destroys the integrity of the json so it can't be re-opened eg:
["2016_02_21_18_46_41", 1], ["2016_02_21_18_46_42", 1], ["2016_02_21_18_46_4
如何做到这一点,使其仅在完成后才将数据写入json?或者我该如何解决此问题?
How can make it so it only writes the data to the json if it's complete? or how can I alternatively fix this problem ?
为了连续性,如果我的父脚本在这里
For continuity, here if my parent script
from threading import Thread
import sys
sys.path.append('/python/loanrates/master')
names = ['BTS', 'ETH', 'CLAM', 'DOGE', 'FCT', 'MAID', 'STR', 'XMR', 'XRP', 'BTC' ]
threads = []
for name in names:
sys.path.append('/python/loanrates/'+name)
import Master
for name in names:
T = Thread(target=Master.main, args=(name,))
print T
threads.append(T)
for thread_ in threads:
thread_.start()
for thread_ in threads:
thread_.join()
这是我的孩子脚本
import arrow
import json
import pickle
import time
import urllib
def main(name):
date = arrow.utcnow().format('YYYY_MM_DD_HH_mm_ss')
previous_date = "2016_02_18_09_02_52"
previous_second = date[-2:]
count = 0
print name,'has started'
while True:
date = arrow.utcnow().format('YYYY_MM_DD_HH_mm_ss')
second = int(date[-2:])
url = 'https://poloniex.com/public?command=returnLoanOrders¤cy='+name
try:
response = urllib.urlopen(url)
data = json.loads(response.read())
except:
data = 'error'
print 'error wth name has occured, probably been blocked by polo'
time.sleep(10)
#open previous data
with open( 'D:/python/loanrates/'+name+'/'+previous_date+'.json', 'r') as f:
previous_data = json.load(f)
#open date log
with open( 'D:/python/loanrates/'+name+'/date_store.json', 'r') as f:
date_store = json.load(f)
#compare new to old data
# if new != old, new data is saved and that date recives a 1 in the 'date_store' dict,
# signifying theres is new data for that date
if previous_data != data and previous_second != second and second%10 == 0:
date_store.append((date,1))
with open( 'D:/python/loanrates/'+name+'/'+date+'.json', 'w') as f:
json.dump(data, f)
with open( 'D:/python/loanrates/'+name+'/date_store.json', 'w') as f:
json.dump(date_store, f)
previous_date = date
previous_second = second
count += 1
if count == 1000: print 'name has logged 1000'
# if new = old the new data hasn't changed and isn't saved, that date 'date_store' recieves a 0
# signifying that the previous date of value 1 can be subtituted for this date
elif previous_second != second and second%10 == 0:
date_store.append((date,0))
with open( 'D:/python/loanrates/'+name+'/date_store.json', 'w') as f:
json.dump(date_store, f)
previous_second = second
if __name__ == '__main__':
try:
main(name)
except KeyboardInterrupt:
print 'Interrupted'
try:
sys.exit(0)
except SystemExit:
os._exit(0)
except:
pass
(如您所见,只有一个子脚本,但是它处于for循环中)
(as you can see there is only one child script but it's in a for loop)
推荐答案
如果在数据未损坏的情况下成功加载备份文件时,try/except应该可以工作.然后,当尝试加载json时,在出现异常时加载备份文件
try/except should work if you keep a backup file on successful loading when the data is not corrupted. Then when trying to load json load the backup file when it goes to exception
这篇关于程序突然结束时如何维护json的完整性的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!