我有一个庞大的数组,其中包含大约一千万个对象,每个对象具有11个不同的键和值对。

将数组内容写入csv的最佳方法是什么?

我尝试使用csv-writer和fast-csv-Reference

使用csv-writer

fastcsv
  .write(final, { headers: true })
  .pipe(ws);

使用快速csv
csvWriter.writeRecords(final)
    .then(() => {
      console.log('...Writing csv Done. Check CSV');
    });

这两种方法都导致无效的字符串长度



示例对象如下所示
{ distance: 0.14,
a_id: 1923,
long: -122.234,
lat: 47.631,
DPlong: -122.234,
DPlat: 47.632,
class: 'secondary',
way_id: 2,
timestamp: '5-6-2017',
user: 'hello',
code: 'DS' }

解决问题的最佳方法是什么?任何帮助将不胜感激。谢谢。

最佳答案

您无法将整个数组转换为字符串,因为该字符串太长而无法处理js,而是必须逐块格式化它:

const tick = () => new Promise(res => setTimeout(res, 0));

const toRow = obj =>
  obj.distance + "," +
  obj.a_id + "," +
  obj.long + "," +
  obj.lat + "," +
  obj.DPlong + "," +
  obj.DPlat + "," +
  obj.class + "," +
  obj.way_id + "," +
  obj.timestamp + "," +
  obj.user + "," +
  obj.code;

const formatChunk = array => array.map(toRow).join("\n") + "\n";

const size = 1000; // <- experiment with it
(async function() {
  for(let i = 0; i < final.length; i += size) {
    ws.write(formatChunk(final.slice(i, i + size)));
    await tick();
  }
  ws.end();
})();

09-11 01:53
查看更多