rust - 在Rust中逐行读取大文件

This question already has answers here:

What's the de-facto way of reading and writing files in Rust 1.x?

(3个答案)



Is this the right way to read lines from file and split them into words in Rust?

(2个答案)



Read lines from file, iterate over each line and each character in that line

(2个答案)



How to combine reading a file line by line and iterating over each character in each line?

(1个答案)



Read a file and get an array of strings

(2个答案)


3年前关闭。




我的Rust程序旨在逐行读取非常大(最多几个GB)的简单文本文件。问题是该文件太大，无法一次读取，或者无法将所有行都传送到Vec<String>。

在Rust中处理该问题的惯用方式是什么？

最佳答案

您想使用buffered reader, BufRead ，特别是功能 BufReader.lines() :

use std::fs::File;
use std::io::{self, prelude::*, BufReader};

fn main() -> io::Result<()> {
    let file = File::open("foo.txt")?;
    let reader = BufReader::new(file);

    for line in reader.lines() {
        println!("{}", line?);
    }

    Ok(())
}

请注意，您是而不是返回了换行符，如文档中所述。

如果您不想为每行分配一个字符串，请参见下面的示例以重用同一缓冲区:

fn main() -> std::io::Result<()> {
    let mut reader = my_reader::BufReader::open("Cargo.toml")?;
    let mut buffer = String::new();

    while let Some(line) = reader.read_line(&mut buffer) {
        println!("{}", line?.trim());
    }

    Ok(())
}

mod my_reader {
    use std::{
        fs::File,
        io::{self, prelude::*},
    };

    pub struct BufReader {
        reader: io::BufReader<File>,
    }

    impl BufReader {
        pub fn open(path: impl AsRef<std::path::Path>) -> io::Result<Self> {
            let file = File::open(path)?;
            let reader = io::BufReader::new(file);

            Ok(Self { reader })
        }

        pub fn read_line<'buf>(
            &mut self,
            buffer: &'buf mut String,
        ) -> Option<io::Result<&'buf mut String>> {
            buffer.clear();

            self.reader
                .read_line(buffer)
                .map(|u| if u == 0 { None } else { Some(buffer) })
                .transpose()
        }
    }
}

Playground
或者，如果您更喜欢标准迭代器，则可以使用我无耻地采用from Reddit的Rc技巧:

fn main() -> std::io::Result<()> {
    for line in my_reader::BufReader::open("Cargo.toml")? {
        println!("{}", line?.trim());
    }

    Ok(())
}

mod my_reader {
    use std::{
        fs::File,
        io::{self, prelude::*},
        rc::Rc,
    };

    pub struct BufReader {
        reader: io::BufReader<File>,
        buf: Rc<String>,
    }

    fn new_buf() -> Rc<String> {
        Rc::new(String::with_capacity(1024)) // Tweakable capacity
    }

    impl BufReader {
        pub fn open(path: impl AsRef<std::path::Path>) -> io::Result<Self> {
            let file = File::open(path)?;
            let reader = io::BufReader::new(file);
            let buf = new_buf();

            Ok(Self { reader, buf })
        }
    }

    impl Iterator for BufReader {
        type Item = io::Result<Rc<String>>;

        fn next(&mut self) -> Option<Self::Item> {
            let buf = match Rc::get_mut(&mut self.buf) {
                Some(buf) => {
                    buf.clear();
                    buf
                }
                None => {
                    self.buf = new_buf();
                    Rc::make_mut(&mut self.buf)
                }
            };

            self.reader
                .read_line(buf)
                .map(|u| if u == 0 { None } else { Some(Rc::clone(&self.buf)) })
                .transpose()
        }
    }
}

Playground

关于rust - 在Rust中逐行读取大文件，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/57053099/