Struct tokio_util::io::StreamReader

source ·
pub struct StreamReader<S, B> { /* private fields */ }
Expand description

Convert a Stream of byte chunks into an AsyncRead.

This type performs the inverse operation of ReaderStream.

This type also implements the AsyncBufRead trait, so you can use it to read a Stream of byte chunks line-by-line. See the examples below.

Example

use bytes::Bytes;
use tokio::io::{AsyncReadExt, Result};
use tokio_util::io::StreamReader;

// Create a stream from an iterator.
let stream = tokio_stream::iter(vec![
    Result::Ok(Bytes::from_static(&[0, 1, 2, 3])),
    Result::Ok(Bytes::from_static(&[4, 5, 6, 7])),
    Result::Ok(Bytes::from_static(&[8, 9, 10, 11])),
]);

// Convert it to an AsyncRead.
let mut read = StreamReader::new(stream);

// Read five bytes from the stream.
let mut buf = [0; 5];
read.read_exact(&mut buf).await?;
assert_eq!(buf, [0, 1, 2, 3, 4]);

// Read the rest of the current chunk.
assert_eq!(read.read(&mut buf).await?, 3);
assert_eq!(&buf[..3], [5, 6, 7]);

// Read the next chunk.
assert_eq!(read.read(&mut buf).await?, 4);
assert_eq!(&buf[..4], [8, 9, 10, 11]);

// We have now reached the end.
assert_eq!(read.read(&mut buf).await?, 0);

If the stream produces errors which are not std::io::Error, the errors can be converted using StreamExt to map each element.

use bytes::Bytes;
use tokio::io::AsyncReadExt;
use tokio_util::io::StreamReader;
use tokio_stream::StreamExt;

// Create a stream from an iterator, including an error.
let stream = tokio_stream::iter(vec![
    Result::Ok(Bytes::from_static(&[0, 1, 2, 3])),
    Result::Ok(Bytes::from_static(&[4, 5, 6, 7])),
    Result::Err("Something bad happened!")
]);

// Use StreamExt to map the stream and error to a std::io::Error
let stream = stream.map(|result| result.map_err(|err| {
    std::io::Error::new(std::io::ErrorKind::Other, err)
}));

// Convert it to an AsyncRead.
let mut read = StreamReader::new(stream);

// Read five bytes from the stream.
let mut buf = [0; 5];
read.read_exact(&mut buf).await?;
assert_eq!(buf, [0, 1, 2, 3, 4]);

// Read the rest of the current chunk.
assert_eq!(read.read(&mut buf).await?, 3);
assert_eq!(&buf[..3], [5, 6, 7]);

// Reading the next chunk will produce an error
let error = read.read(&mut buf).await.unwrap_err();
assert_eq!(error.kind(), std::io::ErrorKind::Other);
assert_eq!(error.into_inner().unwrap().to_string(), "Something bad happened!");

// We have now reached the end.
assert_eq!(read.read(&mut buf).await?, 0);

Using the AsyncBufRead impl, you can read a Stream of byte chunks line-by-line. Note that you will usually also need to convert the error type when doing this. See the second example for an explanation of how to do this.

use tokio::io::{Result, AsyncBufReadExt};
use tokio_util::io::StreamReader;

// Create a stream of byte chunks.
let stream = tokio_stream::iter(vec![
    Result::Ok(b"The first line.\n".as_slice()),
    Result::Ok(b"The second line.".as_slice()),
    Result::Ok(b"\nThe third".as_slice()),
    Result::Ok(b" line.\nThe fourth line.\nThe fifth line.\n".as_slice()),
]);

// Convert it to an AsyncRead.
let mut read = StreamReader::new(stream);

// Loop through the lines from the `StreamReader`.
let mut line = String::new();
let mut lines = Vec::new();
loop {
    line.clear();
    let len = read.read_line(&mut line).await?;
    if len == 0 { break; }
    lines.push(line.clone());
}

// Verify that we got the lines we expected.
assert_eq!(
    lines,
    vec![
        "The first line.\n",
        "The second line.\n",
        "The third line.\n",
        "The fourth line.\n",
        "The fifth line.\n",
    ]
);

Implementations§

source§

impl<S, B, E> StreamReader<S, B>where S: Stream<Item = Result<B, E>>, B: Buf, E: Into<Error>,

source

pub fn new(stream: S) -> Self

Convert a stream of byte chunks into an AsyncRead.

The item should be a Result with the ok variant being something that implements the Buf trait (e.g. Vec<u8> or Bytes). The error should be convertible into an io error.

source

pub fn into_inner_with_chunk(self) -> (S, Option<B>)

Consumes this StreamReader, returning a Tuple consisting of the underlying stream and an Option of the internal buffer, which is Some in case the buffer contains elements.

source§

impl<S, B> StreamReader<S, B>

source

pub fn get_ref(&self) -> &S

Gets a reference to the underlying stream.

It is inadvisable to directly read from the underlying stream.

source

pub fn get_mut(&mut self) -> &mut S

Gets a mutable reference to the underlying stream.

It is inadvisable to directly read from the underlying stream.

source

pub fn get_pin_mut(self: Pin<&mut Self>) -> Pin<&mut S>

Gets a pinned mutable reference to the underlying stream.

It is inadvisable to directly read from the underlying stream.

source

pub fn into_inner(self) -> S

Consumes this BufWriter, returning the underlying stream.

Note that any leftover data in the internal buffer is lost. If you additionally want access to the internal buffer use into_inner_with_chunk.

Trait Implementations§

source§

impl<S, B, E> AsyncBufRead for StreamReader<S, B>where S: Stream<Item = Result<B, E>>, B: Buf, E: Into<Error>,

source§

fn poll_fill_buf( self: Pin<&mut Self>, cx: &mut Context<'_> ) -> Poll<Result<&[u8]>>

Attempts to return the contents of the internal buffer, filling it with more data from the inner reader if it is empty. Read more
source§

fn consume(self: Pin<&mut Self>, amt: usize)

Tells this buffer that amt bytes have been consumed from the buffer, so they should no longer be returned in calls to poll_read. Read more
source§

impl<S, B, E> AsyncRead for StreamReader<S, B>where S: Stream<Item = Result<B, E>>, B: Buf, E: Into<Error>,

source§

fn poll_read( self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_> ) -> Poll<Result<()>>

Attempts to read from the AsyncRead into buf. Read more
source§

impl<S: Debug, B: Debug> Debug for StreamReader<S, B>

source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
source§

impl<S: Unpin, B> Unpin for StreamReader<S, B>

Auto Trait Implementations§

§

impl<S, B> RefUnwindSafe for StreamReader<S, B>where B: RefUnwindSafe, S: RefUnwindSafe,

§

impl<S, B> Send for StreamReader<S, B>where B: Send, S: Send,

§

impl<S, B> Sync for StreamReader<S, B>where B: Sync, S: Sync,

§

impl<S, B> UnwindSafe for StreamReader<S, B>where B: UnwindSafe, S: UnwindSafe,

Blanket Implementations§

source§

impl<T> Any for Twhere T: 'static + ?Sized,

source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
source§

impl<R> AsyncBufReadExt for Rwhere R: AsyncBufRead + ?Sized,

source§

fn read_until<'a>( &'a mut self, byte: u8, buf: &'a mut Vec<u8, Global> ) -> ReadUntil<'a, Self>where Self: Unpin,

Reads all bytes into buf until the delimiter byte or EOF is reached. Read more
source§

fn read_line<'a>(&'a mut self, buf: &'a mut String) -> ReadLine<'a, Self>where Self: Unpin,

Reads all bytes until a newline (the 0xA byte) is reached, and append them to the provided buffer. Read more
source§

fn split(self, byte: u8) -> Split<Self>where Self: Sized + Unpin,

Returns a stream of the contents of this reader split on the byte byte. Read more
source§

fn fill_buf(&mut self) -> FillBuf<'_, Self>where Self: Unpin,

Returns the contents of the internal buffer, filling it with more data from the inner reader if it is empty. Read more
source§

fn consume(&mut self, amt: usize)where Self: Unpin,

Tells this buffer that amt bytes have been consumed from the buffer, so they should no longer be returned in calls to read. Read more
source§

fn lines(self) -> Lines<Self>where Self: Sized,

Returns a stream over the lines of this reader. This method is the async equivalent to BufRead::lines. Read more
source§

impl<R> AsyncReadExt for Rwhere R: AsyncRead + ?Sized,

source§

fn chain<R>(self, next: R) -> Chain<Self, R>where Self: Sized, R: AsyncRead,

Creates a new AsyncRead instance that chains this stream with next. Read more
source§

fn read<'a>(&'a mut self, buf: &'a mut [u8]) -> Read<'a, Self>where Self: Unpin,

Pulls some bytes from this source into the specified buffer, returning how many bytes were read. Read more
source§

fn read_buf<B, 'a>(&'a mut self, buf: &'a mut B) -> ReadBuf<'a, Self, B>where Self: Sized + Unpin, B: BufMut,

Pulls some bytes from this source into the specified buffer, advancing the buffer’s internal cursor. Read more
source§

fn read_exact<'a>(&'a mut self, buf: &'a mut [u8]) -> ReadExact<'a, Self>where Self: Unpin,

Reads the exact number of bytes required to fill buf. Read more
source§

fn read_u8<'a>(&'a mut self) -> ReadU8<&'a mut Self>where Self: Unpin,

Reads an unsigned 8 bit integer from the underlying reader. Read more
source§

fn read_i8<'a>(&'a mut self) -> ReadI8<&'a mut Self>where Self: Unpin,

Reads a signed 8 bit integer from the underlying reader. Read more
source§

fn read_u16<'a>(&'a mut self) -> ReadU16<&'a mut Self>where Self: Unpin,

Reads an unsigned 16-bit integer in big-endian order from the underlying reader. Read more
source§

fn read_i16<'a>(&'a mut self) -> ReadI16<&'a mut Self>where Self: Unpin,

Reads a signed 16-bit integer in big-endian order from the underlying reader. Read more
source§

fn read_u32<'a>(&'a mut self) -> ReadU32<&'a mut Self>where Self: Unpin,

Reads an unsigned 32-bit integer in big-endian order from the underlying reader. Read more
source§

fn read_i32<'a>(&'a mut self) -> ReadI32<&'a mut Self>where Self: Unpin,

Reads a signed 32-bit integer in big-endian order from the underlying reader. Read more
source§

fn read_u64<'a>(&'a mut self) -> ReadU64<&'a mut Self>where Self: Unpin,

Reads an unsigned 64-bit integer in big-endian order from the underlying reader. Read more
source§

fn read_i64<'a>(&'a mut self) -> ReadI64<&'a mut Self>where Self: Unpin,

Reads an signed 64-bit integer in big-endian order from the underlying reader. Read more
source§

fn read_u128<'a>(&'a mut self) -> ReadU128<&'a mut Self>where Self: Unpin,

Reads an unsigned 128-bit integer in big-endian order from the underlying reader. Read more
source§

fn read_i128<'a>(&'a mut self) -> ReadI128<&'a mut Self>where Self: Unpin,

Reads an signed 128-bit integer in big-endian order from the underlying reader. Read more
source§

fn read_f32<'a>(&'a mut self) -> ReadF32<&'a mut Self>where Self: Unpin,

Reads an 32-bit floating point type in big-endian order from the underlying reader. Read more
source§

fn read_f64<'a>(&'a mut self) -> ReadF64<&'a mut Self>where Self: Unpin,

Reads an 64-bit floating point type in big-endian order from the underlying reader. Read more
source§

fn read_u16_le<'a>(&'a mut self) -> ReadU16Le<&'a mut Self>where Self: Unpin,

Reads an unsigned 16-bit integer in little-endian order from the underlying reader. Read more
source§

fn read_i16_le<'a>(&'a mut self) -> ReadI16Le<&'a mut Self>where Self: Unpin,

Reads a signed 16-bit integer in little-endian order from the underlying reader. Read more
source§

fn read_u32_le<'a>(&'a mut self) -> ReadU32Le<&'a mut Self>where Self: Unpin,

Reads an unsigned 32-bit integer in little-endian order from the underlying reader. Read more
source§

fn read_i32_le<'a>(&'a mut self) -> ReadI32Le<&'a mut Self>where Self: Unpin,

Reads a signed 32-bit integer in little-endian order from the underlying reader. Read more
source§

fn read_u64_le<'a>(&'a mut self) -> ReadU64Le<&'a mut Self>where Self: Unpin,

Reads an unsigned 64-bit integer in little-endian order from the underlying reader. Read more
source§

fn read_i64_le<'a>(&'a mut self) -> ReadI64Le<&'a mut Self>where Self: Unpin,

Reads an signed 64-bit integer in little-endian order from the underlying reader. Read more
source§

fn read_u128_le<'a>(&'a mut self) -> ReadU128Le<&'a mut Self>where Self: Unpin,

Reads an unsigned 128-bit integer in little-endian order from the underlying reader. Read more
source§

fn read_i128_le<'a>(&'a mut self) -> ReadI128Le<&'a mut Self>where Self: Unpin,

Reads an signed 128-bit integer in little-endian order from the underlying reader. Read more
source§

fn read_f32_le<'a>(&'a mut self) -> ReadF32Le<&'a mut Self>where Self: Unpin,

Reads an 32-bit floating point type in little-endian order from the underlying reader. Read more
source§

fn read_f64_le<'a>(&'a mut self) -> ReadF64Le<&'a mut Self>where Self: Unpin,

Reads an 64-bit floating point type in little-endian order from the underlying reader. Read more
source§

fn read_to_end<'a>( &'a mut self, buf: &'a mut Vec<u8, Global> ) -> ReadToEnd<'a, Self>where Self: Unpin,

Reads all bytes until EOF in this source, placing them into buf. Read more
source§

fn read_to_string<'a>( &'a mut self, dst: &'a mut String ) -> ReadToString<'a, Self>where Self: Unpin,

Reads all bytes until EOF in this source, appending them to buf. Read more
source§

fn take(self, limit: u64) -> Take<Self>where Self: Sized,

Creates an adaptor which reads at most limit bytes from it. Read more
source§

impl<T> Borrow<T> for Twhere T: ?Sized,

source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
source§

impl<T> BorrowMut<T> for Twhere T: ?Sized,

source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
source§

impl<T> From<T> for T

source§

fn from(t: T) -> T

Returns the argument unchanged.

source§

impl<T> Instrument for T

source§

fn instrument(self, span: Span) -> Instrumented<Self>

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more
source§

fn in_current_span(self) -> Instrumented<Self>

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
source§

impl<T, U> Into<U> for Twhere U: From<T>,

source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

source§

impl<T, U> TryFrom<U> for Twhere U: Into<T>,

§

type Error = Infallible

The type returned in the event of a conversion error.
source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
source§

impl<T, U> TryInto<U> for Twhere U: TryFrom<T>,

§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
source§

impl<T> WithSubscriber for T

source§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more
source§

fn with_current_subscriber(self) -> WithDispatch<Self>

Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more