1
Fork 0
Commit graph

113 commits

Author SHA1 Message Date
Guillaume Gomez
0d3bee8be0
Rollup merge of #85275 - CDirkx:memchr, r=m-ou-se
Move `std::memchr` to `sys_common`

`std::memchr` is a thin abstraction over the different `memchr` implementations in `sys`, along with documentation and tests. The module is only used internally by `std`, nothing is exported externally. Code like this is exactly what the `sys_common` module is for, so this PR moves it there.
2021-05-20 17:56:46 +02:00
Jorge Ferreira
d13745d99b
fix typo 2021-05-18 17:05:40 +01:00
Geoffroy Couprie
95ccdb11da add an example to explain std::io::Read::read returning 0 in some cases
the example focuses on Linux, but that should be enough to explain how
the behaviour can change
2021-05-14 18:06:31 +02:00
Christiaan Dirkx
5353c5c3fb Move std::memchr to sys_common 2021-05-14 03:54:46 +02:00
Ian Jackson
7ae852e349 io::Seek: Set tracking issue
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
2021-05-10 13:55:31 +01:00
Ian Jackson
3113b6bd69
Fix typo in doc
Co-authored-by: Mara Bos <m-ou.se@m-ou.se>
2021-05-10 13:50:56 +01:00
Ian Jackson
c3ca148ac0 io::Seek: Provide rewind()
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
2021-05-10 13:03:49 +01:00
Ian Jackson
74e0e45f3c io::Seek: Mention that seeking can fail due to buffer flush fail
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
2021-05-10 12:57:19 +01:00
Christiaan Dirkx
5b30662741 Rework at_exit to cleanup 2021-04-22 10:34:48 +02:00
bors
5662d9343f Auto merge of #80965 - camelid:rename-doc-spotlight, r=jyn514
Rename `#[doc(spotlight)]` to `#[doc(notable_trait)]`

Fixes #80936.

"spotlight" is not a very specific or self-explaining name.
Additionally, the dialog that it triggers is called "Notable traits".
So, "notable trait" is a better name.

* Rename `#[doc(spotlight)]` to `#[doc(notable_trait)]`
* Rename `#![feature(doc_spotlight)]` to `#![feature(doc_notable_trait)]`
* Update documentation
* Improve documentation

r? `@Manishearth`
2021-04-02 07:04:58 +00:00
Mara Bos
7c01e6c38a Derive Debug for io::Chain instead of manually implementing it.
The manual implementation has the same bounds, so I don't think there's
any reason for a manual implementation. The names used in the derive
implementation are even nicer (`first`/`second`) than the manual
implementation (`t`/`u`), and include the `done_first` field too.
2021-03-27 13:37:52 +01:00
Dylan DPC
a42e62fa0a
Rollup merge of #83353 - m-ou-se:io-error-avoid-alloc, r=nagisa
Add internal io::Error::new_const to avoid allocations.

This makes it possible to have a io::Error containing a message with zero allocations, and uses that everywhere to avoid the *three* allocations involved in `io::Error::new(kind, "message")`.

The function signature isn't perfect, because it needs a reference to the `&str`. So for now, this is just a `pub(crate)` function. Later, we'll be able to use `fn new_const<MSG: &'static str>(kind: ErrorKind)` to make that a bit better. (Then we'll also be able to use some ZST trickery if that would result in more efficient code.)

See https://github.com/rust-lang/rust/issues/83352
2021-03-24 01:52:29 +01:00
Mara Bos
7b71719faf Use io::Error::new_const everywhere to avoid allocations. 2021-03-21 20:22:38 +01:00
Jannis Harder
9dfda62763 Clarify docs for Read::read's return value 2021-03-18 22:52:46 +01:00
Camelid
34c6cee397 Rename #[doc(spotlight)] to #[doc(notable_trait)]
"spotlight" is not a very specific or self-explaining name.
Additionally, the dialog that it triggers is called "Notable traits".
So, "notable trait" is a better name.

* Rename `#[doc(spotlight)]` to `#[doc(notable_trait)]`
* Rename `#![feature(doc_spotlight)]` to `#![feature(doc_notable_trait)]`
* Update documentation
* Improve documentation
2021-03-15 13:59:54 -07:00
Mara
60138110d7
Rollup merge of #81136 - Xavientois:io_reader_size_hint, r=cramertj
Improved IO Bytes Size Hint

After trying to implement better `size_hint()` return values for `File` in [this PR](https://github.com/rust-lang/rust/pull/81044) and changing to implementing it for `BufReader` in [this PR](https://github.com/rust-lang/rust/pull/81052), I have arrived at this implementation that provides tighter bounds for the `Bytes` iterator of various readers including `BufReader`, `Empty`, and `Chain`.

Unfortunately, for `BufReader`, the size_hint only improves after calling `fill_buffer` due to it using the contents of the buffer for the hint. Nevertheless, the the tighter bounds  should result in better pre-allocation of space to handle the contents of the `Bytes` iterator.

Closes #81052
2021-03-05 10:57:17 +01:00
Joshua Nelson
9a75f4fed1 Convert primitives to use intra-doc links 2021-02-25 20:31:53 -05:00
Xavientois
fc9cd4a14b Fix formatting on mod 2021-01-31 08:34:42 -05:00
Xavientois
96255f82c9 Implement SizeHint trait for BufReader, Emtpy, and Chain 2021-01-31 08:34:42 -05:00
Xavientois
c8e0f8aaa3 Use fully qualified syntax to avoid dyn 2021-01-31 08:31:35 -05:00
Xavientois
7869371bf1 Remove unnecessary default keyword 2021-01-31 08:31:35 -05:00
Xavientois
93870c8d5f Remove stable annotation 2021-01-31 08:31:35 -05:00
Xavientois
265db94dc2 Fix formatting 2021-01-31 08:31:35 -05:00
Xavientois
421b40cd6a Add dyn for SizeHint cast 2021-01-31 08:31:35 -05:00
Xavientois
1190321b76 Remove exposing private trait 2021-01-31 08:31:35 -05:00
Xavientois
442de9ac45 Fix semicolon 2021-01-31 08:31:35 -05:00
Xavientois
7e56637c74 Add back lower_bound as memeber 2021-01-31 08:31:35 -05:00
Xavientois
eea99f491b Add default keyword for specialization 2021-01-31 08:31:34 -05:00
Xavientois
5f60a3048e Fix incorrect token 2021-01-31 08:31:34 -05:00
Xavientois
260a270f7c Move default to trait definition 2021-01-31 08:31:34 -05:00
Xavientois
11c49f6a2a Add missing generic 2021-01-31 08:31:34 -05:00
Xavientois
fa76db3104 Use helper trait to follow min_specialization rules 2021-01-31 08:31:34 -05:00
Xavientois
c3e47d974a Fix implementation to specialize 2021-01-31 08:31:34 -05:00
Xavientois
f45bdcce69 Implement size_hint for BufReader 2021-01-31 08:31:34 -05:00
Lukas Kalbertodt
8a18fb0f73
Stabilize Seek::stream_position & change feature of Seek::stream_len 2021-01-24 10:14:24 +01:00
Ben Kimock
4e27ed3af1 Add benchmark and fast path for BufReader::read_exact 2021-01-17 12:10:39 +10:00
Mara Bos
ce48709405
Rollup merge of #80895 - sfackler:read-to-end-ub, r=m-ou-se
Fix handling of malicious Readers in read_to_end

A malicious `Read` impl could return overly large values from `read`, which would result in the guard's drop impl setting the buffer's length to greater than its capacity! ~~To fix this, the drop impl now uses the safe `truncate` function instead of `set_len` which ensures that this will not happen. The result of calling the function will be nonsensical, but that's fine given the contract violation of the `Read` impl.~~

~~The `Guard` type is also used by `append_to_string` which does not pass untrusted values into the length field, so I've copied the guard type into each function and only modified the one used by `read_to_end`. We could just keep a single one and modify it, but it seems a bit cleaner to keep the guard code close to the functions and related specifically to them.~~

To fix this, we now assert that the returned length is not larger than the buffer passed to the method.

For reference, this bug has been present for ~2.5 years since 1.20: ecbb896b9e.

Closes #80894.
2021-01-14 18:00:11 +00:00
Mara Bos
9fc298ca89
Rollup merge of #80217 - camelid:io-read_to_string, r=m-ou-se
Add a `std::io::read_to_string` function

I recognize that you're usually supposed to open an issue first, but the
implementation is very small so it's okay if this is closed and it was 'wasted
work' :)

-----

The equivalent of `std::fs::read_to_string`, but generalized to all
`Read` impls.

As the documentation on `std::io::read_to_string` says, the advantage of
this function is that it means you don't have to create a variable first
and it provides more type safety since you can only get the buffer out
if there were no errors. If you use `Read::read_to_string`, you have to
remember to check whether the read succeeded because otherwise your
buffer will be empty.

It's friendlier to newcomers and better in most cases to use an explicit
return value instead of an out parameter.
2021-01-14 18:00:00 +00:00
Camelid
7463292015 Add docs on performance 2021-01-11 19:18:39 -08:00
Steven Fackler
e6c07b0628 clarify docs a bit 2021-01-11 17:16:44 -05:00
Steven Fackler
5cb830397e make check a bit more clear 2021-01-11 17:13:50 -05:00
Steven Fackler
a9ef7983a6 clean up control flow 2021-01-11 07:48:24 -05:00
Steven Fackler
ebe402dc9e Fix handling of malicious Readers in read_to_end 2021-01-11 07:27:03 -05:00
Camelid
0506789014 Remove many unnecessary manual link resolves from library
Now that #76934 has merged, we can remove a lot of these! E.g, this is
no longer necessary:

    [`Vec<T>`]: Vec
2020-12-31 11:54:32 -08:00
Camelid
588786a788 Add error docs 2020-12-30 11:44:03 -08:00
Camelid
4ee6d1bf54 Add description independent of Read::read_to_string 2020-12-30 11:35:17 -08:00
Camelid
1f9a8a1620 Add a std::io::read_to_string function
The equivalent of `std::fs::read_to_string`, but generalized to all
`Read` impls.

As the documentation on `std::io::read_to_string` says, the advantage of
this function is that it means you don't have to create a variable first
and it provides more type safety since you can only get the buffer out
if there were no errors. If you use `Read::read_to_string`, you have to
remember to check whether the read succeeded because otherwise your
buffer will be empty.

It's friendlier to newcomers and better in most cases to use an explicit
return value instead of an out parameter.
2020-12-19 21:46:40 -08:00
William Chargin
bdaa76cfde Fix typo in std::io::Write docs
These referred to a “`Write`er”—extra *e*. Presumably a copy-paste
holdover from “`Read`er”.

Test Plan:
Running ``git grep '`\?[Ww]rite`\?er'`` no longer finds any results.

wchargin-branch: io-write-docs
2020-11-17 15:32:23 -08:00
Mara Bos
11ce918c75
Rollup merge of #78714 - m-ou-se:simplify-local-streams, r=KodrAus
Simplify output capturing

This is a sequence of incremental improvements to the unstable/internal `set_panic` and `set_print` mechanism used by the `test` crate:

1. Remove the `LocalOutput` trait and use `Arc<Mutex<dyn Write>>` instead of `Box<dyn LocalOutput>`. In practice, all implementations of `LocalOutput` were just `Arc<Mutex<..>>`. This simplifies some logic and removes all custom `Sink` implementations such as `library/test/src/helpers/sink.rs`. Also removes a layer of indirection, as the outermost `Box` is now gone. It also means that locking now happens per `write_fmt`, not per individual `write` within. (So `"{} {}\n"` now results in one `lock()`, not four or more.)

2. Since in all cases the `dyn Write`s were just `Vec<u8>`s, replace the type with `Arc<Mutex<Vec<u8>>>`. This simplifies things more, as error handling and flushing can be removed now. This also removes the hack needed in the default panic handler to make this work with `::realstd`, as (unlike `Write`) `Vec<u8>` is from `alloc`, not `std`.

3. Replace the `RefCell`s by regular `Cell`s. The `RefCell`s were mostly used as `mem::replace(&mut *cell.borrow_mut(), something)`, which is just `Cell::replace`. This removes an unecessary bookkeeping and makes the code a bit easier to read.

4. Merge `set_panic` and `set_print` into a single `set_output_capture`. Neither the test crate nor rustc (the only users of this feature) have a use for using these separately. Merging them simplifies things even more. This uses a new function name and feature name, to make it clearer this is internal and not supposed to be used by other crates.

Might be easier to review per commit.
2020-11-16 17:26:27 +01:00
bors
30e49a9ead Auto merge of #75272 - the8472:spec-copy, r=KodrAus
specialize io::copy to use copy_file_range, splice or sendfile

Fixes #74426.
Also covers #60689 but only as an optimization instead of an official API.

The specialization only covers std-owned structs so it should avoid the problems with #71091

Currently linux-only but it should be generalizable to other unix systems that have sendfile/sosplice and similar.

There is a bit of optimization potential around the syscall count. Right now it may end up doing more syscalls than the naive copy loop when doing short (<8KiB) copies between file descriptors.

The test case executes the following:

```
[pid 103776] statx(3, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=17, ...}) = 0
[pid 103776] write(4, "wxyz", 4)        = 4
[pid 103776] write(4, "iklmn", 5)       = 5
[pid 103776] copy_file_range(3, NULL, 4, NULL, 5, 0) = 5

```

0-1 `stat` calls to identify the source file type. 0 if the type can be inferred from the struct from which the FD was extracted
𝖬 `write` to drain the `BufReader`/`BufWriter` wrappers. only happen when buffers are present. 𝖬 ≾ number of wrappers present. If there is a write buffer it may absorb the read buffer contents first so only result in a single write. Vectored writes would also be an option but that would require more invasive changes to `BufWriter`.
𝖭 `copy_file_range`/`splice`/`sendfile` until file size, EOF or the byte limit from `Take` is reached. This should generally be *much* more efficient than the read-write loop and also have other benefits such as DMA offload or extent sharing.

## Benchmarks

```

OLD

test io::tests::bench_file_to_file_copy         ... bench:      21,002 ns/iter (+/- 750) = 6240 MB/s    [ext4]
test io::tests::bench_file_to_file_copy         ... bench:      35,704 ns/iter (+/- 1,108) = 3671 MB/s  [btrfs]
test io::tests::bench_file_to_socket_copy       ... bench:      57,002 ns/iter (+/- 4,205) = 2299 MB/s
test io::tests::bench_socket_pipe_socket_copy   ... bench:     142,640 ns/iter (+/- 77,851) = 918 MB/s

NEW

test io::tests::bench_file_to_file_copy         ... bench:      14,745 ns/iter (+/- 519) = 8889 MB/s    [ext4]
test io::tests::bench_file_to_file_copy         ... bench:       6,128 ns/iter (+/- 227) = 21389 MB/s   [btrfs]
test io::tests::bench_file_to_socket_copy       ... bench:      13,767 ns/iter (+/- 3,767) = 9520 MB/s
test io::tests::bench_socket_pipe_socket_copy   ... bench:      26,471 ns/iter (+/- 6,412) = 4951 MB/s
```
2020-11-14 12:01:55 +00:00