Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Option unwrap() on None when using AsyncWriteExt #6933

Open
msftsecurityteam opened this issue Oct 24, 2024 · 5 comments
Open

Option unwrap() on None when using AsyncWriteExt #6933

msftsecurityteam opened this issue Oct 24, 2024 · 5 comments
Labels
A-tokio Area: The main tokio crate C-bug Category: This is a bug. M-fs Module: tokio/fs

Comments

@msftsecurityteam
Copy link

msftsecurityteam commented Oct 24, 2024 •
edited
Loading

Version
1.41.0 and 1.40.0

Platform
Linux rddlr 6.1.0-26-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.112-1 (2024-09-30) x86_64 GNU/Linux

Description
I am writing async code in which I am passing around a lazy_static!'ed Arc<Option<File>>. I am a new Rust developer so maybe some of what I am doing is incorrect, but this issue always happens at the end of the runtime of my code. My code does a lot of async HTTP requests (each of which is tokio::spawn()'ed and then each return value is worked on via a second set of tokio::spawn() threads. This is all done via a stream::iter(chunks).map(|| { ... }).buffer_unordered(N).for_each(|res| async { ... }).await, where chunks is a collection of Strings.

I have attached the full backtrace here:

thread 'tokio-runtime-worker' panicked at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/fs/file.rs:732:51:
called `Option::unwrap()` on a `None` value
stack backtrace:
   0:     0x56055c7aa4e5 - std::backtrace_rs::backtrace::libunwind::trace::h649ab3318d3445c5
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1:     0x56055c7aa4e5 - std::backtrace_rs::backtrace::trace_unsynchronized::hf4bb60c3387150c3
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x56055c7aa4e5 - std::sys::backtrace::_print_fmt::hd9186c800e44bd00
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:65:5
   3:     0x56055c7aa4e5 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h1b9dad2a88e955ff
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:40:26
   4:     0x56055c7cfc9b - core::fmt::rt::Argument::fmt::h351a7824f737a6a0
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/fmt/rt.rs:173:76
   5:     0x56055c7cfc9b - core::fmt::write::h4b5a1270214bc4a7
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/fmt/mod.rs:1182:21
   6:     0x56055c7a790f - std::io::Write::write_fmt::hd04af345a50c312d
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/io/mod.rs:1827:15
   7:     0x56055c7ab7e1 - std::sys::backtrace::BacktraceLock::print::h68d41b51481bce5c
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:43:9
   8:     0x56055c7ab7e1 - std::panicking::default_hook::{{closure}}::h96ab15e9936be7ed
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:269:22
   9:     0x56055c7ab4bc - std::panicking::default_hook::h3cacb9c27561ad33
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:296:9
  10:     0x56055c7abe41 - std::panicking::rust_panic_with_hook::hfe205f6954b2c97b
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:800:13
  11:     0x56055c7abc73 - std::panicking::begin_panic_handler::{{closure}}::h6cb44b3a50f28c44
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:667:13
  12:     0x56055c7aa9a9 - std::sys::backtrace::__rust_end_short_backtrace::hf1c1f2a92799bb0e
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:168:18
  13:     0x56055c7ab934 - rust_begin_unwind
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:665:5
  14:     0x56055c04b2d3 - core::panicking::panic_fmt::h3d8fc78294164da7
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:74:14
  15:     0x56055c04b35c - core::panicking::panic::hec978767ec2d35ff
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:148:5
  16:     0x56055c04b239 - core::option::unwrap_failed::hba6b08832f9ce30b
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/option.rs:2020:5
  17:     0x56055c6400af - core::option::Option<T>::unwrap::h2c408f65545b6fc1
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/option.rs:970:21
  18:     0x56055c6400af - <tokio::fs::file::File as tokio::io::async_write::AsyncWrite>::poll_write::hf4e8f8d18a0ce1bd
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/fs/file.rs:732:35
  19:     0x56055c062251 - <&mut T as tokio::io::async_write::AsyncWrite>::poll_write::hbad151a078d0bfd2
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/io/async_write.rs:192:13
  20:     0x56055c076255 - <tokio::io::util::write_all::WriteAll<W> as core::future::future::Future>::poll::h6d02d0ad4bf03094
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/io/util/write_all.rs:43:28
  21:     0x56055c08caca - rddlr::write_to_file::{{closure}}::h6c52804999bcbc84
                               at /home/m/riddler/rddlr/src/lib.rs:68:48
  22:     0x56055c08f4d4 - rddlr::update_db::{{closure}}::{{closure}}::hba4e3babc34d81bc
                               at /home/m/riddler/rddlr/src/lib.rs:96:42
  23:     0x56055c0abd3f - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hfcbe4afeae601794
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/core.rs:331:17
  24:     0x56055c0aab8d - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h4648bddbd7bef5bf
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/loom/std/unsafe_cell.rs:16:9
  25:     0x56055c0aab8d - tokio::runtime::task::core::Core<T,S>::poll::hccc119ba2c2a2f14
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/core.rs:320:13
  26:     0x56055c05170a - tokio::runtime::task::harness::poll_future::{{closure}}::h9b9cbc0f8693b90a
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/harness.rs:499:19
  27:     0x56055c09d140 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hb3303775436e64fc
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs:272:9
  28:     0x56055c0958d8 - std::panicking::try::do_call::h241a365f20480bad
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:557:40
  29:     0x56055c0a4edb - __rust_try
  30:     0x56055c0a2316 - std::panicking::try::heca9eac5abfb8096
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:521:19
  31:     0x56055c0a2316 - std::panic::catch_unwind::h151c0d8f1eb7e988
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs:350:14
  32:     0x56055c04f56f - tokio::runtime::task::harness::poll_future::h43a6182ab4b73fee
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/harness.rs:487:18
  33:     0x56055c0520e7 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h08431fd45c9a495f
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/harness.rs:209:27
  34:     0x56055c054dd3 - tokio::runtime::task::harness::Harness<T,S>::poll::h946a01863d743bf1
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/harness.rs:154:15
  35:     0x56055c07c3db - tokio::runtime::task::raw::poll::h737a55a06e0d483c
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/raw.rs:271:5
  36:     0x56055c6246c7 - tokio::runtime::task::raw::RawTask::poll::h5d15c4892734d6b0
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/raw.rs:201:18
  37:     0x56055c63ac52 - tokio::runtime::task::LocalNotified<S>::run::h6074739368ab5b61
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/mod.rs:435:9
  38:     0x56055c693be7 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h5b2b845e8538efc4
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/scheduler/multi_thread/worker.rs:659:17
  39:     0x56055c693404 - tokio::runtime::coop::with_budget::h6880d8e13f1bc2ab
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/coop.rs:107:5
  40:     0x56055c693404 - tokio::runtime::coop::budget::h8a6346b062152998
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/coop.rs:73:5
  41:     0x56055c693404 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h48cc5e988d5ab844
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/scheduler/multi_thread/worker.rs:595:9
  42:     0x56055c692ad3 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h1845136b12163b49
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/scheduler/multi_thread/worker.rs:546:24
  43:     0x56055c692739 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::h6458dd2a1c080743
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/scheduler/multi_thread/worker.rs:511:21
  44:     0x56055c641060 - tokio::runtime::context::scoped::Scoped<T>::set::h35ff1f539ee75ab1
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/context/scoped.rs:40:9
  45:     0x56055c62e73b - tokio::runtime::context::set_scheduler::{{closure}}::h3b2fc7981eb184c7
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/context.rs:180:26
  46:     0x56055c6508d0 - std::thread::local::LocalKey<T>::try_with::h916dab96c048bcec
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/local.rs:283:12
  47:     0x56055c64f8eb - std::thread::local::LocalKey<T>::with::h4d1a4e5c73044a4e
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/local.rs:260:9
  48:     0x56055c62e674 - tokio::runtime::context::set_scheduler::h1c36c4fce850766c
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/context.rs:180:9
  49:     0x56055c692644 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h57fd5603045879b8
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/scheduler/multi_thread/worker.rs:506:9
  50:     0x56055c69db20 - tokio::runtime::context::runtime::enter_runtime::hc10b0594d99eb384
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/context/runtime.rs:65:16
  51:     0x56055c6923e6 - tokio::runtime::scheduler::multi_thread::worker::run::hd8c2dae097f14897
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/scheduler/multi_thread/worker.rs:498:5
  52:     0x56055c69210b - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::hbf53b781f9169222
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/scheduler/multi_thread/worker.rs:464:45
  53:     0x56055c6624fe - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h13ce29aa647a1f74
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/blocking/task.rs:42:21
  54:     0x56055c6885d3 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::ha564cfb16c89726a
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/core.rs:331:17
  55:     0x56055c688081 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hc267351fc8f9d3f6
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/loom/std/unsafe_cell.rs:16:9
  56:     0x56055c688081 - tokio::runtime::task::core::Core<T,S>::poll::h4b0161cbdca6d7ae
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/core.rs:320:13
  57:     0x56055c6171ae - tokio::runtime::task::harness::poll_future::{{closure}}::hab31ce16ab6fb02d
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/harness.rs:499:19
  58:     0x56055c61db21 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h88b5b9f00ad79926
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs:272:9
  59:     0x56055c663749 - std::panicking::try::do_call::hed8ab69602e68b3c
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:557:40
  60:     0x56055c69e4bb - __rust_try
  61:     0x56055c69c5f6 - std::panicking::try::h7c9620815e3036a6
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:521:19
  62:     0x56055c69c5f6 - std::panic::catch_unwind::h2f21e58816f72ad5
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs:350:14
  63:     0x56055c616880 - tokio::runtime::task::harness::poll_future::h973c2f04a8e94a8e
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/harness.rs:487:18
  64:     0x56055c61474e - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h53116de9c9d8687d
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/harness.rs:209:27
  65:     0x56055c6143b7 - tokio::runtime::task::harness::Harness<T,S>::poll::hd2795e66e6a9ecd7
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/harness.rs:154:15
  66:     0x56055c62492d - tokio::runtime::task::raw::poll::h035ede83177f58ed
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/raw.rs:271:5
  67:     0x56055c6246c7 - tokio::runtime::task::raw::RawTask::poll::h5d15c4892734d6b0
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/raw.rs:201:18
  68:     0x56055c63ad17 - tokio::runtime::task::UnownedTask<S>::run::hccc8ffbfb8d3c54e
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/mod.rs:472:9
  69:     0x56055c6970f7 - tokio::runtime::blocking::pool::Task::run::h33d4ebb4b885b2c4
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/blocking/pool.rs:161:9
  70:     0x56055c699ede - tokio::runtime::blocking::pool::Inner::run::h392fb86cd584a4d3
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/blocking/pool.rs:511:17
  71:     0x56055c699bf4 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::hfdeff38519ea2830
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/blocking/pool.rs:469:13
  72:     0x56055c672116 - std::sys::backtrace::__rust_begin_short_backtrace::h8c793e533b1c36fd
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:152:18
  73:     0x56055c673ad2 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h58c065ad7e75361e
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/mod.rs:538:17
  74:     0x56055c61da22 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h54f71e4f943f91c8
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs:272:9
  75:     0x56055c663093 - std::panicking::try::do_call::h20b94d30cc8ce131
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:557:40
  76:     0x56055c67721b - __rust_try
  77:     0x56055c673716 - std::panicking::try::h71d8ca21fec86411
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:521:19
  78:     0x56055c673716 - std::panic::catch_unwind::h399adad2b13170a4
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs:350:14
  79:     0x56055c673716 - std::thread::Builder::spawn_unchecked_::{{closure}}::h22242bfe68727c34
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/mod.rs:537:30
  80:     0x56055c66794f - core::ops::function::FnOnce::call_once{{vtable.shim}}::h2e60a4cbf4c7a1d1
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/ops/function.rs:250:5
  81:     0x56055c7aea5b - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::ha1963004222e7822
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs:2070:9
  82:     0x56055c7aea5b - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h1086ced1f7c494c2
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs:2070:9
  83:     0x56055c7aea5b - std::sys::pal::unix::thread::Thread::new::thread_start::ha8af9c992ef0b208
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/pal/unix/thread.rs:108:17
  84:     0x7efdcf8a8144 - start_thread
                               at ./nptl/pthread_create.c:442:8
  85:     0x7efdcf9287dc - __GI___clone3
                               at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
  86:                0x0 - <unknown>
@msftsecurityteam msftsecurityteam added A-tokio Area: The main tokio crate C-bug Category: This is a bug. labels Oct 24, 2024
@msftsecurityteam
Copy link
Author

msftsecurityteam commented Oct 24, 2024 •
edited
Loading

Towards the end of my process_chunk() logic (this is the second of the two tokio::spawn()'s that I mentioned), I write to the shared filehandle that is guarded by a mutex, via another tokio::spawn thread. Adding this and I see ~25% performance boost, and it seems to work just fine because all of the data is processed, but this is when I begin to see these unwrap() on None errors at process shutdown. If I remove this third tokio::spawn(), I take a slight performance hit but it doesn't seem to be causing any unwrap() on None issues anymore.

@Darksonn Darksonn added the M-fs Module: tokio/fs label Oct 24, 2024
@Darksonn
Copy link
Contributor

This is happening during runtime shutdown?

@msftsecurityteam
Copy link
Author

I don't know if there is an official time for "runtime shutdown" but yes, it happens at the very end of my code after all of the data has been processed. I get like, 8 or 16 or whatever the number of threads is saying Option unwrap() on None panic

@msftsecurityteam
Copy link
Author

msftsecurityteam commented Oct 24, 2024 •
edited
Loading

I am updating the comment. I believe the second tokio::spawn() in the .for_each() was unnecessary. I pulled it out, keeping the tokio::spawn() just for the very end for the file I/O write_all() call, and I am still getting these unwrap() errors.

Code is more or less:

let _ = stream::iter(chunks).map(|chunk| { tokio::spawn(async move { fetch_with_retry(&chunk).await })}) .buffer_unordered(NUM_REQUESTS).for_each(|fetch_result| async { match fetch_result { Ok(Some(data)) => { if let Err(e) = process_chunk(&tag).await { eprintln!("process_chunk failed: {}", e); } }, Ok(None) => { eprintln!("fetch_with_retry returned no data"); }, Err(e) => { panic!("match fetch_with_retry fail: {:?}", e) } } }) .await;

Inside process_chunk it is a bunch of JSON parsing, with a tokio::spawn at the end to async write_all the results.

@Darksonn
Copy link
Contributor

Runtime shutdown starts once you return from the #[tokio::main] function.

Are you able to share a small complete example that triggers the error so I can try it myself?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-tokio Area: The main tokio crate C-bug Category: This is a bug. M-fs Module: tokio/fs
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Darksonn @msftsecurityteam