当长度已知时，从Lazy ByteString构造RequestBodyStream

问题描述

我正在尝试修改此 AWS S3上传代码处理长度已知的Lazy ByteString(这样就不会强制在内存中完整读取它-它是通过​​事先发送长度的网络来的).看来我必须定义一个 GivesPopper 在Lazy ByteString上起作用，将其转换为 RequestBodyStream .由于定义了GivesPopper复杂的方式，因此我不确定如何为Lazy ByteString编写它.将不胜感激如何编写它的指针. 此处是为从文件中读取而编写的:

I am trying to adapt this AWS S3 upload code to handle Lazy ByteString where length is already known (so that it is not forced to be read in its entirety in memory - it comes over the network where length is sent beforehand). It seems I have to define a GivesPopper function over Lazy ByteString to convert it to RequestBodyStream. Because of the convoluted way GivesPopper is defined, I am not sure how to write it for Lazy ByteString. Will appreciate pointers on how to write it. Here is how it is written for reading from the file:

let file ="test"
-- streams large file content, without buffering more than 10k in memory
let streamer sink = withFile file ReadMode $ \h -> sink $ S.hGet h 10240

如果我正确理解的话，上面代码中的

streamer属于GivesPopper ()类型.给定长度为len的Lazy ByteString，在其上编写GivesPopper函数的一种好方法是什么?我们一次只能读取一个块.

streamer in the code above is of type GivesPopper () if I understand it correctly.Given a Lazy ByteString with known length len, what would be a good way to write GivesPopper function over it? We can read one chunk at a time.

推荐答案

这是您要寻找的吗?

import qualified Data.ByteString as S
import qualified Data.ByteString.Lazy as L
import System.IO

file = "test"
-- original streamer for feeding a sink from a file
streamer :: (IO S.ByteString -> IO r) -> IO r
streamer sink = withFile file ReadMode $ \h -> sink $ S.hGet h 10240

-- feed a lazy ByteString to sink
lstreamer :: L.ByteString -> (IO S.ByteString -> IO r) -> IO r
lstreamer lbs sink = sink (return (L.toStrict lbs))

lstreamer类型检查，但可能不完全符合您的期望.每次接收器调用时，它仅返回相同的数据.另一方面，S.hGet h ...最终将返回空字符串.

lstreamer type checks but probably doesn't do exactly what you want it to do. It simply returns the same data every time the sink calls it. On the other hand S.hGet h ... will eventually return the empty string.

这是一个使用IORef跟踪我们是否应该开始返回空字符串的解决方案:

Here is a solution which uses an IORef to keep track of if we should start returning the empty string:

import Data.IORef

mklstream :: L.ByteString -> (IO S.ByteString -> IO r) -> IO r
mklstream lbs sink = do
  ref <- newIORef False
  let fetch :: IO S.ByteString
      fetch = do sent <- readIORef ref
                 writeIORef ref True
                 if sent
                   then return S.empty
                   else return (L.toStrict lbs)
  sink fetch

此处fetch是获取下一个块的动作.第一次调用它时，您将获得原始的惰性Bytestring(严格限制).后续调用将始终返回空字符串.

Here fetch is the action which gets the next chunk. The first time you call it you will get the original lazy Bytestring (strict-ified). Subsequent calls will always return the empty string.

更新

这里是一次分发少量硬币的方法:

Here's how to give out a small amount at a time:

mklstream :: L.ByteString -> (IO S.ByteString -> IO r) -> IO r
mklstream lbs sink = do
  ref <- newIORef (L.toChunks lbs)
  let fetch :: IO S.ByteString
      fetch = do chunks <- readIORef ref
                 case chunks of
                   [] -> return S.empty
                   (c:cs) -> do writeIORef ref cs
                                return c
  sink fetch

这篇关于当长度已知时，从Lazy ByteString构造RequestBodyStream的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！