Swoole大数据量传输解决方案
问题：要不压缩的情况下，在Swoole TCP中传输大量数据length>1M
首先用node的mockjs生成一个大于1M的文件
var jsonfile = require('jsonfile')
var file = './data.json'
let fs = require("fs");
var Mock = require('mockjs')
function injectContentTofileUntil(fileSizeInMega) {
let obj = jsonfile.readFileSync(file)
console.log(obj)
if(!obj){
obj=[]
}
for (let index = 0; index < 1000; index++) {
obj.push({
name: Mock.Random.cname(),
country: Mock.Random.county(true),
})
}
jsonfile.writeFileSync(file, obj)
let fileSizeInMegabytes = getFileSizeInMega(file)
if (fileSizeInMegabytes >= fileSizeInMega) {
return true
} else {
return false
}
}
function getFileSizeInMega(filePath) {
let stats = fs.statSync(file)
let fileSizeInBytes = stats.size
let fileSizeInMegabytes = fileSizeInBytes / 1000000.0
return fileSizeInMegabytes
}
while (!injectContentTofileUntil(1)) {
console.log(getFileSizeInMega(file));
}
该文件为大于1M的无序文件，在不使用压缩的情况下传输该文件到server端。
1.正常起server和client传输
Server.php
error_reporting(0);
$client = new swoole_client(SWOOLE_SOCK_TCP, SWOOLE_SOCK_ASYNC);
$filePath = realpath(__DIR__ . '/./jstools/data.json');
$fileContent = file_get_contents($filePath);
$client->on("connect", function ($cli) {
global $fileContent;
$cli->send($fileContent);
});
$client->on("receive", function ($cli, $data) {
echo "received: {$data}\n";
});
$client->on("error", function ($cli) {
echo "connect failed\n";
});
$client->on("close", function ($cli) {
echo "connection close\n";
});
$client->connect("127.0.0.1", 8091, 0.5);
Client.php
$server = new swoole_server("0.0.0.0", 8081);
$server->on('connect', function ($server, $fd){
echo "connection open: {$fd}\n";
});
$server->on('receive', function ($server, $fd, $reactor_id, $data) {
echo "received:" . $data;
$server->close($fd);
file_put_contents('./response.json', $data);
});
$server->on('close', function ($server, $fd) {
echo "connection close: {$fd}\n";
});
$server->start();
执行命令
php server.php
php client.php
报错:
WARNING swReactor_write (ERROR 1201): data is too large, cannot exceed buffer size.
原因:
buffer不足溢出
2.修改buffer不足问题
设置client最大buffer:
"socketbuffersize"=>210241024,
报错:
2018-05-18 17:46:09 *6146.0 ERROR swWorker_discard_data (ERROR 1007): 1received the wrong data8180 bytes from socket#1
原因:
当swoole接收到满足buffer大小的内容时会自动结束传输，已经把数据给消费者，但是数据并没有传输完成，此时有几种方案解决该问题：
- 1.压缩
- 2.告诉swoole到达buffer大小时还没传输结束，本文只讨论此方案
3.实现数据传输切分
server设置：
true,"> 'package_eof' => "\r\n\r\n",
'open_eof_split' => true,
client设置：
2*1024*1024,"> 'package_eof' => "\r\n\r\n",
可以正常传输。
4.此方案优缺点：
优点：可以正常传输任意大小数据
缺点：需要手动给流增加eof
字段含义请自行查阅swoole文档。