上次项目中用到<<PHP采集淘宝商品>>
此方法有一个缺点,就是执行效率问题.一个商品采集平均需要0.8秒.那10000个商品采集完需要2个半小时.
首先想到的解决办法是并发.
但是PHP不支持并发(这里是指通过PHP命令执行PHP文件,如果通过apache或nginx等做服务器是可以并发的,是并发访问,不能在程序中实现并发).
通过Shell把对php命令推到后台执行,以达到并发的效果.
整体思路:
1.在Shell中连接数据库,取出需要更新的产品
2.Shell中对数据进行循环,把商品id,price,url传递给PHP执行,将执行过程推到后台
3.每循环20条使程序暂停5秒,达到控制并发数的目的
4.php得到id,price,url参数后,通过URL进行采集,并返回现价
5.将现价和原数据库中的价格进行比较,如果有变化或下架则更新.
6.将执行结果写入日志文件.
Shell
#!/bin/sh
#updateprice.sh
j=
currcyline=;
#查询数据库
for i in `/usr/local/bin/mysql -uroot -pshop123 shop -e"SELECT id,price,url FROM s_goods WHERE url!='' AND keyid like 'taobao%' AND is_off_sale=0 ORDER BY id DESC "`
do
if [ $j -gt ]; then
#前3个循环分别为id,price,url这相当于表头,不需要进行操作,所以从第3开始.
line=$(($j%))
case $line in
)
currcyline=$(($currcyline+)) s=$(($currcyline%)) if [ $s -eq ]; then
sleep #每循环20次休息5秒,以此来控制避免产生过多的后台进行,使服务器压力过大或死机.
fi
id=$i;;
)
price=$i;; )
url=$i;
#echo id:${id} price:${price} url:${url} {
#调用php命令执行PHP文件.
re=`/usr/local/bin/php /var/www/9384shop/cron/goodsupdate.php ${id} ${price} "${url}"`
}&
#此&为推到后台执行, 关键
esac fi
j=$(($j+))
done
wait
#等待后台进行执行完成
PHP:
<?php
/*
============================================================================
Name : goodsupdate.php
Author : 风飘无痕
Version :
Copyright : Your copyright notice
Description : Collect taobao goods
============================================================================
*/ //$argv为获取命令中的参数
$s=microtime(1);
$id=$argv[1];
$oldprice=$argv[2];
$price=getPrice($argv[3]);
$t=microtime(1)-$s;
$r=array();
$r[]=date('Y-m-d H:i:s');
$r[]=$id;
$r[]=ceil($t*1000)/1000;
if($price=='soldout'){
$r[]="OutStock";
$con=mysql_connect('localhost','shop','shop123');
mysql_select_db("shop", $con);
mysql_query("UPDATE s_goods SET is_off_sale=1 WHERE id=".$id);
mysql_close($con);
}
elseif($price===false) $r[]= 'FALSE';
elseif($price==$oldprice) $r[]='EQUAL';
else{
$r[]="UPDATE";
$r[]=$oldprice;
$r[]=$price;
$con=mysql_connect('localhost','shop','shop123');
mysql_select_db("shop", $con);
mysql_query("UPDATE s_goods SET price=".$price." WHERE id=".$id);
mysql_close($con);
}
//以日志的形式保存执行过程
$h=fopen('/home/staff/www/9384shop/log/goodsUpdate.log','a+'); fputcsv($h,$r);
fclose($h); function getPrice($url,$time=1){
$des_url='';
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 6.2; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0');
curl_setopt($ch, CURLOPT_REFERER,'http://www.tmall.com/');//采集淘宝商品必须设置此项
curl_setopt($ch, CURLOPT_FOLLOWLOCATION,1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);//设置输出方式, 0为自动输出返回的内容, 1为返回输出的内容,但不自动输出.
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30); //timeout on connect
curl_setopt($ch, CURLOPT_TIMEOUT, 30); //timeout on response
curl_setopt($ch, CURLOPT_HEADER, 1);//是否输出头信息，0为不输出，非零则输出
curl_setopt($ch, CURLOPT_MAXREDIRS, 10 );
curl_setopt($ch, CURLOPT_URL, $url);
$content = curl_exec($ch);
curl_close($ch);
if(preg_match('/noitem\.htm/',$content)){
return 'soldout';
}elseif(preg_match("/'reservePrice'\s*:\s*'([\d\.]+?)',/",$content,$price)){
$price = (float)$price[1];
}elseif(preg_match('/price:([\d\.]+?),/',$content,$price)){
$price = (float)$price[1];
}
if(!$price){
preg_match('/id=(\d+)+/',$url,$temp);
$url2="http://mdskip.taobao.com/core/initItemDetail.htm?itemId=".$temp[1];
$ch = curl_init();
curl_setopt( $ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1" );
curl_setopt( $ch, CURLOPT_URL, $url2 );
curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, true );
curl_setopt( $ch, CURLOPT_ENCODING, "" );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
curl_setopt( $ch, CURLOPT_REFERER, 'http://www.tmall.com' );
curl_setopt( $ch, CURLOPT_AUTOREFERER, true );
curl_setopt( $ch, CURLOPT_SSL_VERIFYPEER, false );
curl_setopt( $ch, CURLOPT_CONNECTTIMEOUT, 10 );
curl_setopt( $ch, CURLOPT_TIMEOUT, 10 );
curl_setopt( $ch, CURLOPT_MAXREDIRS, 10 );
$price_content = curl_exec( $ch );
$response = curl_getinfo( $ch );
curl_close ( $ch );
$price_content=json_decode(iconv('gbk','utf-8',preg_replace('/(\d{10,}):/','"${1}":',$price_content)),true);
$priceinfo=$price_content['defaultModel']['itemPriceResultDO']['priceInfo'];
$price=array();
if(is_array($priceinfo)){
foreach ($priceinfo as $v){
$price[]=$v['price'];
if(is_array($v['promotionList'])){
foreach ($v['promotionList'] as $v2){
$price[]=$v2['extraPromPrice']?$v2['extraPromPrice']:$v2['price'];
}
}
if(is_array($v['suggestivePromotionList'])){
foreach ($v['suggestivePromotionList'] as $v2){
$price[]=$v2['extraPromPrice']?$v2['extraPromPrice']:$v2['price'];
}
}
}
}
$price=count($price)>0?min($price):false; }
if($price) return $price;
elseif($time<3) return getPrice($url,$time+1);//如果没有取到价格递归执行,最多执行3次.
else return false;
}
执行结果:
tail - goodsUpdate.log
"2014-03-21 13:45:34",,0.273,EQUAL
"2014-03-21 13:45:35",,5.883,EQUAL
"2014-03-21 13:45:35",,0.914,EQUAL
"2014-03-21 13:45:35",,0.923,EQUAL
"2014-03-21 13:45:35",,0.927,UPDATE,599.00,181.00
"2014-03-21 13:45:35",,0.908,EQUAL
"2014-03-21 13:45:35",,0.93,EQUAL
"2014-03-21 13:45:35",,0.933,EQUAL
"2014-03-21 13:45:35",,0.946,UPDATE,1547.00,1877.00
"2014-03-21 13:45:35",,0.947,EQUAL
此方法比只用PHP更新大大节约了时间,更新2万个商品大约是半小时.但是线程数非常不好控制