我有一个庞大的数组,其中包含大约一千万个对象,每个对象具有11个不同的键和值对。
将数组内容写入csv的最佳方法是什么?
我尝试使用csv-writer和fast-csv-Reference
使用csv-writer
fastcsv
.write(final, { headers: true })
.pipe(ws);
使用快速csv
csvWriter.writeRecords(final)
.then(() => {
console.log('...Writing csv Done. Check CSV');
});
这两种方法都导致无效的字符串长度
示例对象如下所示
{ distance: 0.14,
a_id: 1923,
long: -122.234,
lat: 47.631,
DPlong: -122.234,
DPlat: 47.632,
class: 'secondary',
way_id: 2,
timestamp: '5-6-2017',
user: 'hello',
code: 'DS' }
解决问题的最佳方法是什么?任何帮助将不胜感激。谢谢。
最佳答案
您无法将整个数组转换为字符串,因为该字符串太长而无法处理js,而是必须逐块格式化它:
const tick = () => new Promise(res => setTimeout(res, 0));
const toRow = obj =>
obj.distance + "," +
obj.a_id + "," +
obj.long + "," +
obj.lat + "," +
obj.DPlong + "," +
obj.DPlat + "," +
obj.class + "," +
obj.way_id + "," +
obj.timestamp + "," +
obj.user + "," +
obj.code;
const formatChunk = array => array.map(toRow).join("\n") + "\n";
const size = 1000; // <- experiment with it
(async function() {
for(let i = 0; i < final.length; i += size) {
ws.write(formatChunk(final.slice(i, i + size)));
await tick();
}
ws.end();
})();