Feature Request: mechanism to control the number of the threads in the cpprestsdk's thread pool before startup #298

yxiang92128 · 2019-10-22T20:49:09Z

Currently it starts 40 threads in the pool and there is no way to configure that. Our application limits the number of threads due to resource constraints. And I am wondering if we can configure that when initialize_blob_client.

Thanks,

Yang

Jinming-Hu · 2019-10-28T03:57:57Z

Hi Yang, give this a try.

Apply this patch no_init_thpool.patch.txt, and in your program, #include <pplx/threadpool.h>, before calling any function of cpprestsdk/azure-storage-cpp, call crossplat::threadpool::initialize_with_threads(4) to change thread pool size.

Tell me if you come across crash or any other kinds of issues. Thanks

yxiang92128 · 2019-10-30T17:07:06Z

I will try that next week.

yxiang92128 · 2019-11-06T00:35:16Z

@JinmingHu-MSFT
Seems to be working. I will patch my v6.0.0 source with it.

Thanks. Yang

yxiang92128 · 2019-11-11T18:10:33Z

@JinmingHu-MSFT
unfortunately I have to backtrack my changes out. It seems to intermittently hang on some of the established client connections (over 8000 max # of connections) after repeat download operations in the same process. Don't know how to work around the hang. It's discovered during our stresstest.

Jinming-Hu · 2019-11-12T02:52:20Z

@yxiang92128 Well, the result is not terribly surprising. There've been hang issues in the SDK for a very long time, and some of them are never really fixed. It's more likely to trigger with fewer threads.

yxiang92128 · 2019-11-12T16:42:50Z

@JinmingHu-MSFT can you explain what other hang issues the C++ REST SDK might have so we can be aware when support issues arise?
Thanks.
Yang

Jinming-Hu · 2019-11-13T02:29:21Z

@yxiang92128 Well, not cpprest sdk, it's in our azure-storage sdk. At least as far as I know, when uploading a blob with max-execution-time set and parallelism > 1, there's small possibility that it will hang or crash.

Next time you come across this hang issue, you can tell me the related information (which API, max-execution-time set or not, parallelism), maybe it'll be helpful to fix it.

yxiang92128 · 2020-05-13T16:53:53Z

@JinmingHu-MSFT
Hi Jinming,
It seems like the 40 threads all made TLSv1 Client Hello exchange with the Azure backend and thus established socket connection with them. The effect is that these socket lingered in TIME_WAIT state and quickly depleted our socket pool and our application is very network centric so it does not bode well.

Would you think of any other workaround to inhibit this behavior so it would only start one thread on startup without causing it to hang?

Thanks for any hint.

Yang

kiril-kirov · 2020-08-25T15:01:19Z

Hi,

It's me again. I got some questions on this topic. Sorry for the long post in advance.

The number of threads can't be changed runtime, can it? I mean - once initialized with N threads, this can't be changed anymore (a process restart is required)? The same implies for stopping the threads.

Use case: our app works with different object stores and this can be switched at runtime. For this purpose we use different SDKs, as one might expect. So, my point is - once we connect to an Azure storage account and the threads are started, they can't be stopped anymore? Is there a workaround for this?

Looks like these 40 threads can actually maintain more TCP connections. I can see 128 TCP connections when sending 128 parallel GET requests for example (I'd assume PUT requests are similar?). My question is - are these 128 parallel requests really asynchronously processed (in parallel)?

If so, why so many threads? Or each thread can actually work with one TCP connection at a time (blocking/waiting for response) and connections are being served using round-robin/other algorithm?

Seems like the Windows implementation is completely different. How does the SDK behave in this case (it obviously does not start 40 additional threads, nor uses boost::asio)?

Sorry, I'm not that familiar with the win32. I tried figuring out, but could you give me a hint about where/what to look at? I saw the set_wastorage_ambient_delayed_scheduler, but I don't see any calls to this function.

Thanks,
Kiril

Jinming-Hu · 2020-08-25T15:14:32Z

The number of threads can't be changed runtime, can it? I mean - once initialized with N threads, this can't be changed anymore (a process restart is required)? The same implies for stopping the threads.

No, you cannot change the number of threads once the thread pool is initialized.

Use case: our app works with different object stores and this can be switched at runtime. For this purpose we use different SDKs, as one might expect. So, my point is - once we connect to an Azure storage account and the threads are started, they can't be stopped anymore? Is there a workaround for this?

Not that I know of. Unless you run storage sdk in another process and communicate via socket or pipe or something. But i guess you won't like this way.

Looks like these 40 threads can actually maintain more TCP connections. I can see 128 TCP connections when sending 128 parallel GET requests for example (I'd assume PUT requests are similar?). My question is - are these 128 parallel requests really asynchronously processed (in parallel)?

Yes, cpprestsdk is asynchronous.

If so, why so many threads? Or each thread can actually work with one TCP connection at a time (blocking/waiting for response) and connections are being served using round-robin/other algorithm?

This is pre-defined in cpprestsdk. Maybe (i'm not sure, i'm not developer of cpprest project) besides network requests, cpprest can also handle some cpu-intensive async tasks.

Seems like the Windows implementation is completely different. How does the SDK behave in this case (it obviously does not start 40 additional threads, nor uses boost::asio)?

Correct, cpprest only depends on boost on Linux. On Windows, it uses a built-in thread pool.

Sorry, I'm not that familiar with the win32. I tried figuring out, but could you give me a hint about where/what to look at? I saw the set_wastorage_ambient_delayed_scheduler, but I don't see any calls to this function.

That set_wastorage_ambient_delayed_scheduler function should be related to scheduling algorithm. You don't need to call it If you don't want to customize the scheduling algorithm.

kiril-kirov · 2020-08-26T08:20:15Z

Is the thread pool stateless then? I mean - switching between Azure storage accounts should be safe although the thread pool is not recreated, correct?

Jinming-Hu · 2020-08-26T08:38:31Z

Is the thread pool stateless then? I mean - switching between Azure storage accounts should be safe although the thread pool is not recreated, correct?

Yes, it is safe to switch accounts.

kiril-kirov · 2020-08-27T15:48:09Z

I applied the patch and it works pretty good on Linux. But it crashes on MacOS. Looks like a timing issue, as it doesn't happen every single time, but often enough.

I've attached a repro case + a MacOS crash report. Seems like an issue with http_client.

Stack from the crash report:

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_pthread.dylib       	0x00007fff6bbe65ad pthread_mutex_lock + 0
1   test                          	0x00000001010e4969 unsigned long boost::asio::detail::kqueue_reactor::cancel_timer<boost::asio::time_traits<boost::posix_time::ptime> >(boost::asio::detail::timer_queue<boost::asio::time_traits<boost::posix_time::ptime> >&, boost::asio::detail::timer_queue<boost::asio::time_traits<boost::posix_time::ptime> >::per_timer_data&, unsigned long) + 41 (kqueue_reactor.hpp:67)
2   test                          	0x00000001010e0190 boost::asio::basic_io_object<boost::asio::deadline_timer_service<boost::posix_time::ptime, boost::asio::time_traits<boost::posix_time::ptime> >, false>::~basic_io_object() + 64 (basic_io_object.hpp:130)
3   test                          	0x000000010138eeb5 std::__1::__shared_ptr_emplace<web::http::client::details::asio_connection_pool, std::__1::allocator<web::http::client::details::asio_connection_pool> >::__on_zero_shared() + 21 (memory:3865)
4   libc++.1.dylib                	0x00007fff68ce8232 std::__1::__shared_weak_count::__release_shared() + 40
5   test                          	0x000000010138e446 std::__1::__shared_ptr_emplace<web::http::client::details::asio_client, std::__1::allocator<web::http::client::details::asio_client> >::__on_zero_shared() + 38 (memory:3865)
6   libc++.1.dylib                	0x00007fff68ce8232 std::__1::__shared_weak_count::__release_shared() + 40
7   libc++.1.dylib                	0x00007fff68ce8232 std::__1::__shared_weak_count::__release_shared() + 40
8   test                          	0x00000001010e4d46 std::__1::__tree<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::shared_ptr<web::http::client::http_client> >, std::__1::__map_value_compare<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::shared_ptr<web::http::client::http_client> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::shared_ptr<web::http::client::http_client> > > >::destroy(std::__1::__tree_node<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::shared_ptr<web::http::client::http_client> >, void*>*) + 102 (__tree:1860)
9   libsystem_c.dylib             	0x00007fff6ba9013c __cxa_finalize_ranges + 319
10  libsystem_c.dylib             	0x00007fff6ba90412 exit + 55
11  libdyld.dylib                 	0x00007fff6b9e6cd0 start + 8

Please advise.

Thanks,
Kiril

test.txt
crash-report.txt

kiril-kirov · 2020-08-27T16:06:56Z

I'm wrong. I ran the test app on Linux multiple times (with a bash loop) and it broke as well.

Here's the Linux stack:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007fe413814859 in __GI_abort () at abort.c:79
#2  0x00007fe41387f3ee in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fe4139a9285 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3  0x00007fe41388747c in malloc_printerr (str=str@entry=0x7fe4139a743a "corrupted size vs. prev_size") at malloc.c:5347
#4  0x00007fe413887aeb in unlink_chunk (p=p@entry=0x7fe390000c10, av=0x7fe390000020) at malloc.c:1454
#5  0x00007fe413887c2f in malloc_consolidate (av=av@entry=0x7fe390000020) at malloc.c:4502
#6  0x00007fe413889160 in _int_free (av=0x7fe390000020, p=0x7fe39001ade0, have_lock=<optimized out>) at malloc.c:4400
#7  0x0000000000a38cad in CRYPTO_free ()
#8  0x0000000000a0b678 in SSL_CTX_free ()
#9  0x0000000000a0daed in SSL_free ()
#10 0x000000000087c009 in boost::asio::ssl::detail::stream_core::~stream_core (this=0x1acb8b8, __in_chrg=<optimized out>)
    at /home/build/toolset/boost/1.65.1/Source/boost/asio/ssl/detail/stream_core.hpp:59
#11 boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >&>::~stream (this=0x1acb8b0, __in_chrg=<optimized out>)
    at /home/build/toolset/boost/1.65.1/Source/boost/asio/ssl/stream.hpp:117
#12 std::default_delete<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >&> >::operator() (this=<optimized out>, 
    __ptr=0x1acb8b0) at /usr/include/c++/4.9/bits/unique_ptr.h:76
#13 std::unique_ptr<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >&>, std::default_delete<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >&> > >::~unique_ptr (this=0x1ac9868, __in_chrg=<optimized out>)
    at /usr/include/c++/4.9/bits/unique_ptr.h:236
#14 web::http::client::details::asio_connection::~asio_connection (this=0x1ac9820, __in_chrg=<optimized out>)
    at /home/build/toolset/casablanca/2.10.14/Source/Release/src/http/client/http_client_asio.cpp:160
#15 __gnu_cxx::new_allocator<web::http::client::details::asio_connection>::destroy<web::http::client::details::asio_connection> (this=<optimized out>, __p=0x1ac9820)
    at /usr/include/c++/4.9/ext/new_allocator.h:124
#16 std::allocator_traits<std::allocator<web::http::client::details::asio_connection> >::_S_destroy<web::http::client::details::asio_connection> (__p=0x1ac9820, __a=...)
    at /usr/include/c++/4.9/bits/alloc_traits.h:282
#17 std::allocator_traits<std::allocator<web::http::client::details::asio_connection> >::destroy<web::http::client::details::asio_connection> (__a=..., __p=0x1ac9820)
    at /usr/include/c++/4.9/bits/alloc_traits.h:411
#18 std::_Sp_counted_ptr_inplace<web::http::client::details::asio_connection, std::allocator<web::http::client::details::asio_connection>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x1ac9810)
    at /usr/include/c++/4.9/bits/shared_ptr_base.h:524
#19 0x00000000006702f4 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x1ac9810) at /usr/include/c++/4.9/bits/shared_ptr_base.h:149
#20 0x0000000000874046 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7fe3ec001718, __in_chrg=<optimized out>) at /usr/include/c++/4.9/bits/shared_ptr_base.h:666
#21 std::__shared_ptr<web::http::client::details::asio_connection, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7fe3ec001710, __in_chrg=<optimized out>)
    at /usr/include/c++/4.9/bits/shared_ptr_base.h:914
#22 std::shared_ptr<web::http::client::details::asio_connection>::~shared_ptr (this=0x7fe3ec001710, __in_chrg=<optimized out>) at /usr/include/c++/4.9/bits/shared_ptr.h:93
#23 std::_Destroy<std::shared_ptr<web::http::client::details::asio_connection> > (__pointer=0x7fe3ec001710) at /usr/include/c++/4.9/bits/stl_construct.h:93
#24 std::_Destroy_aux<false>::__destroy<std::shared_ptr<web::http::client::details::asio_connection>*> (__last=<optimized out>, __first=0x7fe3ec001710) at /usr/include/c++/4.9/bits/stl_construct.h:103
#25 std::_Destroy<std::shared_ptr<web::http::client::details::asio_connection>*> (__last=<optimized out>, __first=<optimized out>) at /usr/include/c++/4.9/bits/stl_construct.h:126
#26 std::_Destroy<std::shared_ptr<web::http::client::details::asio_connection>*, std::shared_ptr<web::http::client::details::asio_connection> > (__last=0x7fe3ec001770, __first=<optimized out>)
    at /usr/include/c++/4.9/bits/stl_construct.h:151
#27 std::vector<std::shared_ptr<web::http::client::details::asio_connection>, std::allocator<std::shared_ptr<web::http::client::details::asio_connection> > >::~vector (this=0x7fe398000de8, 
    __in_chrg=<optimized out>) at /usr/include/c++/4.9/bits/stl_vector.h:424
#28 web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection>::~connection_pool_stack (this=0x7fe398000de8, __in_chrg=<optimized out>)
    at /home/build/toolset/casablanca/2.10.14/Source/Release/src/http/client/../common/connection_pool_helpers.h:17
#29 std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> >::~pair (
    this=0x7fe398000de0, __in_chrg=<optimized out>) at /usr/include/c++/4.9/bits/stl_pair.h:96
#30 __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> > > >::destroy<std::pair<std::basic_string<char> const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> > > (
--Type <RET> for more, q to quit, c to continue without paging--
    this=<optimized out>, __p=0x7fe398000de0) at /usr/include/c++/4.9/ext/new_allocator.h:124
#31 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> > > > >::_S_destroy<std::pair<std::basic_string<char> const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> > > (
    __p=0x7fe398000de0, __a=...) at /usr/include/c++/4.9/bits/alloc_traits.h:282
#32 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> > > > >::destroy<std::pair<std::basic_string<char> const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> > > (
    __a=..., __p=0x7fe398000de0) at /usr/include/c++/4.9/bits/alloc_traits.h:411
#33 std::_Rb_tree<std::string, std::pair<std::string const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> >, std::_Select1st<std::pair<std::string const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> > >, std::less<std::string>, std::allocator<std::pair<std::string const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> > > >::_M_destroy_node (this=0x188f448, __p=0x7fe398000dc0) at /usr/include/c++/4.9/bits/stl_tree.h:436
#34 std::_Rb_tree<std::string, std::pair<std::string const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> >, std::_Select1st<std::pair<std::string const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> > >, std::less<std::string>, std::allocator<std::pair<std::string const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> > > >::_M_erase (this=this@entry=0x188f448, __x=0x7fe398000dc0) at /usr/include/c++/4.9/bits/stl_tree.h:1247
#35 0x000000000087beed in std::_Rb_tree<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> >, std::_Select1st<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> > >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> > > >::~_Rb_tree (
    this=0x188f448, __in_chrg=<optimized out>) at /usr/include/c++/4.9/bits/stl_tree.h:715
#36 std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection>, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, web::http::client::details::connection_pool_stack<web::http::client::details::asio_connection> > > >::~map (this=0x188f448, __in_chrg=<optimized out>) at /usr/include/c++/4.9/bits/stl_map.h:96
#37 web::http::client::details::asio_connection_pool::~asio_connection_pool (this=0x188f410, __in_chrg=<optimized out>)
    at /home/build/toolset/casablanca/2.10.14/Source/Release/src/http/client/http_client_asio.cpp:368
#38 __gnu_cxx::new_allocator<web::http::client::details::asio_connection_pool>::destroy<web::http::client::details::asio_connection_pool> (this=<optimized out>, __p=0x188f410)
    at /usr/include/c++/4.9/ext/new_allocator.h:124
#39 std::allocator_traits<std::allocator<web::http::client::details::asio_connection_pool> >::_S_destroy<web::http::client::details::asio_connection_pool> (__p=0x188f410, __a=...)
    at /usr/include/c++/4.9/bits/alloc_traits.h:282
#40 std::allocator_traits<std::allocator<web::http::client::details::asio_connection_pool> >::destroy<web::http::client::details::asio_connection_pool> (__a=..., __p=0x188f410)
    at /usr/include/c++/4.9/bits/alloc_traits.h:411
#41 std::_Sp_counted_ptr_inplace<web::http::client::details::asio_connection_pool, std::allocator<web::http::client::details::asio_connection_pool>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (
    this=0x188f400) at /usr/include/c++/4.9/bits/shared_ptr_base.h:524
#42 0x00000000006702f4 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x188f400) at /usr/include/c++/4.9/bits/shared_ptr_base.h:149
#43 0x0000000000876692 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x1890f08, __in_chrg=<optimized out>) at /usr/include/c++/4.9/bits/shared_ptr_base.h:666
#44 std::__shared_ptr<web::http::client::details::asio_connection_pool, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x1890f00, __in_chrg=<optimized out>)
    at /usr/include/c++/4.9/bits/shared_ptr_base.h:914
#45 std::shared_ptr<web::http::client::details::asio_connection_pool>::~shared_ptr (this=0x1890f00, __in_chrg=<optimized out>) at /usr/include/c++/4.9/bits/shared_ptr.h:93
#46 web::http::client::details::asio_client::~asio_client (this=0x1890cf0, __in_chrg=<optimized out>) at /home/build/toolset/casablanca/2.10.14/Source/Release/src/http/client/http_client_asio.cpp:466
#47 __gnu_cxx::new_allocator<web::http::client::details::asio_client>::destroy<web::http::client::details::asio_client> (this=<optimized out>, __p=0x1890cf0)
    at /usr/include/c++/4.9/ext/new_allocator.h:124
#48 std::allocator_traits<std::allocator<web::http::client::details::asio_client> >::_S_destroy<web::http::client::details::asio_client> (__p=0x1890cf0, __a=...)
    at /usr/include/c++/4.9/bits/alloc_traits.h:282
#49 std::allocator_traits<std::allocator<web::http::client::details::asio_client> >::destroy<web::http::client::details::asio_client> (__a=..., __p=0x1890cf0)
    at /usr/include/c++/4.9/bits/alloc_traits.h:411
#50 std::_Sp_counted_ptr_inplace<web::http::client::details::asio_client, std::allocator<web::http::client::details::asio_client>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x1890ce0)
    at /usr/include/c++/4.9/bits/shared_ptr_base.h:524
#51 0x00000000006702f4 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x1890ce0) at /usr/include/c++/4.9/bits/shared_ptr_base.h:149
#52 0x00000000006702f4 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x188f240) at /usr/include/c++/4.9/bits/shared_ptr_base.h:149
--Type <RET> for more, q to quit, c to continue without paging--
#53 0x00000000006702f4 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x188f350) at /usr/include/c++/4.9/bits/shared_ptr_base.h:149
#54 0x000000000078e776 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x1890820, __in_chrg=<optimized out>) at /usr/include/c++/4.9/bits/shared_ptr_base.h:666
#55 std::__shared_ptr<web::http::client::http_client, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x1890818, __in_chrg=<optimized out>) at /usr/include/c++/4.9/bits/shared_ptr_base.h:914
#56 std::shared_ptr<web::http::client::http_client>::~shared_ptr (this=0x1890818, __in_chrg=<optimized out>) at /usr/include/c++/4.9/bits/shared_ptr.h:93
#57 std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<web::http::client::http_client> >::~pair (this=0x1890810, __in_chrg=<optimized out>)
    at /usr/include/c++/4.9/bits/stl_pair.h:96
#58 __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<web::http::client::http_client> > > >::destroy<std::pair<std::basic_string<char> const, std::shared_ptr<web::http::client::http_client> > > (this=<optimized out>, __p=0x1890810) at /usr/include/c++/4.9/ext/new_allocator.h:124
#59 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<web::http::client::http_client> > > > >::_S_destroy<std::pair<std::basic_string<char> const, std::shared_ptr<web::http::client::http_client> > > (__p=0x1890810, __a=...) at /usr/include/c++/4.9/bits/alloc_traits.h:282
#60 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<web::http::client::http_client> > > > >::destroy<std::pair<std::basic_string<char> const, std::shared_ptr<web::http::client::http_client> > > (__a=..., __p=0x1890810) at /usr/include/c++/4.9/bits/alloc_traits.h:411
#61 std::_Rb_tree<std::string, std::pair<std::string const, std::shared_ptr<web::http::client::http_client> >, std::_Select1st<std::pair<std::string const, std::shared_ptr<web::http::client::http_client> > >, std::less<std::string>, std::allocator<std::pair<std::string const, std::shared_ptr<web::http::client::http_client> > > >::_M_destroy_node (
    this=0xd93de0 <azure::storage::core::http_client_reusable::s_http_clients>, __p=0x18907f0) at /usr/include/c++/4.9/bits/stl_tree.h:436
#62 std::_Rb_tree<std::string, std::pair<std::string const, std::shared_ptr<web::http::client::http_client> >, std::_Select1st<std::pair<std::string const, std::shared_ptr<web::http::client::http_client> > >, std::less<std::string>, std::allocator<std::pair<std::string const, std::shared_ptr<web::http::client::http_client> > > >::_M_erase (
    this=0xd93de0 <azure::storage::core::http_client_reusable::s_http_clients>, __x=0x18907f0) at /usr/include/c++/4.9/bits/stl_tree.h:1247
#63 0x00007fe413838a27 in __run_exit_handlers (status=0, listp=0x7fe4139da718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#64 0x00007fe413838be0 in __GI_exit (status=<optimized out>) at exit.c:139
#65 0x00007fe4138160ba in __libc_start_main (main=0x663b92 <main()>, argc=1, argv=0x7ffddaf93b68, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffddaf93b58)
    at ../csu/libc-start.c:342
#66 0x0000000000662f9e in _start ()

At least it's much more verbose that stack.
Maybe I need some cleanup step?

Jinming-Hu · 2020-08-28T03:44:46Z

@kiril-kirov I looked through the code again. I did find some issues in the implementation of http client reuse part. Because we're using a static std::map to store those http clinets. The construct & destruct order of static variables among multiple object files (source files) is undetermined. I think this leads to the crash problem during program finalizing. (You did encounter this crash when your program exited, didn't you?) I guess this only affects non-Windows OSes, because http client reuse isn't enabled for Windows.

Jinming-Hu · 2020-08-28T03:47:08Z

Maybe I need some cleanup step?

I don't think there's anything you can do. although I can think of something we storage sdk can do to properly clean up stuff so the program can elegantly exit. But i haven't tried it yet.

kiril-kirov · 2020-08-28T07:18:39Z

Yes, it's non-Windows only. Yes, it's on exit, as the stacks show.
I expected this is a problem with static variables and the order of their destruction. I hope there's an easy way for fast cleanup on exit. I'm also looking at the problem.

Jinming-Hu · 2020-08-28T07:26:57Z

@kiril-kirov Add a function to clean this std::map after all HTTP connections are finished. Call this function before main() exits. This may be able to fix this issue.

kiril-kirov · 2020-08-28T07:28:40Z

Just clearing the app? Or some graceful stop of those clients needs to be done?

Jinming-Hu · 2020-08-28T07:30:47Z

@kiril-kirov that's a map of shared_pointer, so I guess just clearing the map is fine.

kiril-kirov · 2020-08-28T09:17:27Z

It looks promising at the moment. Here's the whole diff, if someone's interested:

diff --git a/6.1.0/Source/Microsoft.WindowsAzure.Storage/includes/wascore/util.h b/6.1.0/Source/Microsoft.WindowsAzure.Storage/includes/wascore/util.h
index 5605bba..98ddc58 100644
--- a/6.1.0/Source/Microsoft.WindowsAzure.Storage/includes/wascore/util.h
+++ b/6.1.0/Source/Microsoft.WindowsAzure.Storage/includes/wascore/util.h
@@ -125,8 +125,12 @@ namespace azure { namespace storage { namespace core {
         WASTORAGE_API static std::shared_ptr<web::http::client::http_client> get_http_client(const web::uri& uri);
         WASTORAGE_API static std::shared_ptr<web::http::client::http_client> get_http_client(const web::uri& uri, const web::http::client::http_client_config& config);
 
+        WASTORAGE_API static void clean_up()
+        {
+            std::lock_guard<std::mutex> guard(s_mutex);
+            s_http_clients.clear();
+        }
     private:
-        static const boost::asio::io_service& s_service;
         WASTORAGE_API static std::map<utility::string_t, std::shared_ptr<web::http::client::http_client>> s_http_clients;
         WASTORAGE_API static std::mutex s_mutex;
     };
diff --git a/6.1.0/Source/Microsoft.WindowsAzure.Storage/src/util.cpp b/6.1.0/Source/Microsoft.WindowsAzure.Storage/src/util.cpp
index 2ea3f5a..301e127 100644
--- a/6.1.0/Source/Microsoft.WindowsAzure.Storage/src/util.cpp
+++ b/6.1.0/Source/Microsoft.WindowsAzure.Storage/src/util.cpp
@@ -500,7 +500,6 @@ namespace azure { namespace storage {  namespace core {
     }
 
 #ifndef _WIN32
-    const boost::asio::io_service& http_client_reusable::s_service = crossplat::threadpool::shared_instance().service();
     std::map<utility::string_t, std::shared_ptr<web::http::client::http_client>> http_client_reusable::s_http_clients;
     std::mutex http_client_reusable::s_mutex;

So, you need to call:

crossplat::threadpool::initialize_with_threads(threads); - only once during the lifetime of the process. Needs to be called before any other SDK API calls. Once created, the threadpool cannot be resized, nor destroyed.
azure::storage::core::http_client_reusable::clean_up(); - only once during the lifetime of the process. Needs to be called after all other SDK API calls. #include "wascore/util.h" is required.

jade2k11598 · 2021-03-11T23:08:49Z

@Jinming-Hu, did this feature ever make it to a branch that was released? I currently have a master version (Tue Feb 23 14:00:43 or dbdd3fa6ec40fb4a283cbc26f1f521d0761d8d33) and I attempted to call crossplat::threadpool::initialize_with_threads before attempting to call any azure::storage objects but I get this crash (indicating that crossplat::threadpool::initialize_with_threads is getting called twice):

terminate called after throwing an instance of 'std::runtime_error'
  what():  the cpprestsdk threadpool has already been initialized

I understand you provided a patch above, but I really don't want to maintain that (nor have I tried it with that patch) in a version of azure-storage-cpp.

Jinming-Hu · 2021-03-12T00:43:58Z

@jade2k11598 This feature is not officially supported and it's not stable enough so we don't want to maintain it.

jade2k11598 · 2021-03-12T05:05:43Z

@Jinming-Hu, it's really not practical for a utility to spawn a pool of 40 threads. My application just needs to upload files to Azure (just one of its many other features), but it shouldn't have to take 40 threads just to do that. This utility alone is spawning more threads, than the other features in my application.

It should be left to the user to determine how many threads should be utilized for this utility (usually optimized for the number of hardware cores running on the machine). Granted there should be a minimum of how many threads are required, but 40 is excessive. Not having that ability to dictate how many threads are dedicated to this utility, is a major drawback in design.

Jinming-Hu added the feature request label Oct 23, 2019

Jinming-Hu self-assigned this Oct 23, 2019

Jinming-Hu mentioned this issue Aug 20, 2020

All SDK threads eventually block forever in protocol::parse_extended_error #364

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature Request: mechanism to control the number of the threads in the cpprestsdk's thread pool before startup #298

Feature Request: mechanism to control the number of the threads in the cpprestsdk's thread pool before startup #298

yxiang92128 commented Oct 22, 2019

Jinming-Hu commented Oct 28, 2019

yxiang92128 commented Oct 30, 2019

yxiang92128 commented Nov 6, 2019

yxiang92128 commented Nov 11, 2019 •

edited

Loading

Jinming-Hu commented Nov 12, 2019

yxiang92128 commented Nov 12, 2019

Jinming-Hu commented Nov 13, 2019

yxiang92128 commented May 13, 2020

kiril-kirov commented Aug 25, 2020

Jinming-Hu commented Aug 25, 2020 •

edited

Loading

kiril-kirov commented Aug 26, 2020

Jinming-Hu commented Aug 26, 2020

kiril-kirov commented Aug 27, 2020

kiril-kirov commented Aug 27, 2020

Jinming-Hu commented Aug 28, 2020 •

edited

Loading

Jinming-Hu commented Aug 28, 2020

kiril-kirov commented Aug 28, 2020

Jinming-Hu commented Aug 28, 2020

kiril-kirov commented Aug 28, 2020

Jinming-Hu commented Aug 28, 2020

kiril-kirov commented Aug 28, 2020

jade2k11598 commented Mar 11, 2021

Jinming-Hu commented Mar 12, 2021

jade2k11598 commented Mar 12, 2021

Feature Request: mechanism to control the number of the threads in the cpprestsdk's thread pool before startup #298

Feature Request: mechanism to control the number of the threads in the cpprestsdk's thread pool before startup #298

Comments

yxiang92128 commented Oct 22, 2019

Jinming-Hu commented Oct 28, 2019

yxiang92128 commented Oct 30, 2019

yxiang92128 commented Nov 6, 2019

yxiang92128 commented Nov 11, 2019 • edited Loading

Jinming-Hu commented Nov 12, 2019

yxiang92128 commented Nov 12, 2019

Jinming-Hu commented Nov 13, 2019

yxiang92128 commented May 13, 2020

kiril-kirov commented Aug 25, 2020

Jinming-Hu commented Aug 25, 2020 • edited Loading

kiril-kirov commented Aug 26, 2020

Jinming-Hu commented Aug 26, 2020

kiril-kirov commented Aug 27, 2020

kiril-kirov commented Aug 27, 2020

Jinming-Hu commented Aug 28, 2020 • edited Loading

Jinming-Hu commented Aug 28, 2020

kiril-kirov commented Aug 28, 2020

Jinming-Hu commented Aug 28, 2020

kiril-kirov commented Aug 28, 2020

Jinming-Hu commented Aug 28, 2020

kiril-kirov commented Aug 28, 2020

jade2k11598 commented Mar 11, 2021

Jinming-Hu commented Mar 12, 2021

jade2k11598 commented Mar 12, 2021

yxiang92128 commented Nov 11, 2019 •

edited

Loading

Jinming-Hu commented Aug 25, 2020 •

edited

Loading

Jinming-Hu commented Aug 28, 2020 •

edited

Loading