Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PoC Couch Stats Resource Tracker for tracking process local resource usage #4812

Draft
wants to merge 29 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
be84279
PoC couch_stats local resource usage tracking
chewbranca Jul 18, 2022
e5e4820
Instrument more stats and rudimentary delta accumulation
chewbranca Oct 23, 2023
ecda8f3
Add more usage tracking and core functionality
chewbranca Oct 30, 2023
73863a7
Embed worker usage deltas in rexi:ping
chewbranca Oct 31, 2023
12e0267
WIP: core delta aggregation working
chewbranca Nov 9, 2023
84be4af
TMP: rewire rexi to fail on unexpected messages
chewbranca Nov 10, 2023
19224af
Add baseline group by and sort by aggregations
chewbranca Nov 14, 2023
cafa512
Add watchdog scanning for unmonitored processes
chewbranca Nov 14, 2023
996c385
Set dbname context properly
chewbranca Nov 15, 2023
1bbf7a9
Add unsafe_foldl for perf testing
chewbranca Nov 16, 2023
e70ba4d
Rework csrt context setting
chewbranca Nov 18, 2023
715b524
WIP: ugly but working HTTP API around groupings
chewbranca Nov 18, 2023
be8ab04
Cleanup debug info
chewbranca Nov 20, 2023
4c05a0f
Make CSRT toggle-able and rework delta sending
chewbranca Nov 20, 2023
eb20b5d
Remove debug exit clause
chewbranca Nov 20, 2023
f4a712c
Declare missing metrics
chewbranca Nov 21, 2023
c758cb7
WIP: relax record constraints...
chewbranca Nov 21, 2023
4f00910
Use Default in config:get/3 on ets table error
chewbranca Nov 30, 2023
887a7f2
Switch to delta extraction from messages
chewbranca Nov 30, 2023
1a1a584
Revert "Use Default in config:get/3 on ets table error"
chewbranca Nov 30, 2023
db1f874
Flush chttpd_db monitor refs on demonitor
chewbranca Dec 9, 2023
50495bf
TMP Hack around issue #4909
chewbranca Dec 18, 2023
9578773
Avoid mem3_rpc:rexi_call selective receive
chewbranca Dec 18, 2023
f77583a
Pass tests and toggle csrt
chewbranca Dec 19, 2023
ac17510
Handle delta in fabric rpc tests
chewbranca Dec 19, 2023
c500f0f
Conditionally log reports
chewbranca Dec 19, 2023
d7e9bd9
Do not persist doc size test config settings
chewbranca Jan 5, 2024
f68005a
Configurable logging and no more io_lib:format
chewbranca Jan 8, 2024
0371589
Limit resource usage output and fix keys
chewbranca Feb 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Avoid mem3_rpc:rexi_call selective receive
We need to potentially extract the usage delta from the incoming RPC
message. Rather than pattern match on all possible message formats that
could potentially include usage deltas, we instead utilize
rexi_util:extract_delta which matches against tuples ending in
`{delta, Delta}`, and separates that out from the underlying message.

The subtlety here is that receiving the message to extract the delta
changes the behavior as this was previously doing a selective receive
keyed off of the Ref, and then ignoring any other messages that arrived.
I don't know if the selective receive was intended, but I don't think
it's appropriate to leave unexpected messages floating around,
especially given things like issue #4909.

Instead of utilizing a selective receive, this switches to extracting
the message and delta like we need to do, and then in the event it finds
unexpected messages they're logged and skipped.

This selective receive was masking the lack of unlink on the linked
rexi_mon pid in fix #4906. I've also noticed some rpc responses arriving
late as well, but I haven't tracked that down, so let's log when it does
happen.
  • Loading branch information
chewbranca committed Dec 18, 2023
commit 957877347995e18175a18b3ed4ca15e193602430
1 change: 1 addition & 0 deletions src/couch/include/couch_db.hrl
Original file line number Diff line number Diff line change
@@ -127,6 +127,7 @@
meta = []
}).

-define(LOG_UNEXPECTED_MSG(Msg), couch_log:warning("[~p:~p:~p/~p]{~p[~p]} Unexpected message: ~w", [?MODULE, ?LINE, ?FUNCTION_NAME, ?FUNCTION_ARITY, self(), element(2, process_info(self(), message_queue_len)), Msg])).

-record(user_ctx, {
name=null,
34 changes: 24 additions & 10 deletions src/mem3/src/mem3_rpc.erl
Original file line number Diff line number Diff line change
@@ -378,20 +378,34 @@ rexi_call(Node, MFA, Timeout) ->
Mon = rexi_monitor:start([rexi_utils:server_pid(Node)]),
Ref = rexi:cast(Node, self(), MFA, [sync]),
try
receive
{Ref, {ok, Reply}} ->
Reply;
{Ref, Error} ->
erlang:error(Error);
{rexi_DOWN, Mon, _, Reason} ->
erlang:error({rexi_DOWN, {Node, Reason}})
after Timeout ->
erlang:error(timeout)
end
wait_message(Node, Ref, Mon, Timeout)
after
rexi_monitor:stop(Mon)
end.

wait_message(Node, Ref, Mon, Timeout) ->
receive
Msg ->
process_raw_message(Msg, Node, Ref, Mon, Timeout)
after Timeout ->
erlang:error(timeout)
end.

process_raw_message(Msg0, Node, Ref, Mon, Timeout) ->
{Msg, Delta} = rexi_utils:extract_delta(Msg0),
couch_stats_resource_tracker:accumulate_delta(Delta),
case Msg of
{Ref, {ok, Reply}} ->
Reply;
{Ref, Error} ->
erlang:error(Error);
{rexi_DOWN, Mon, _, Reason} ->
erlang:error({rexi_DOWN, {Node, Reason}});
Other ->
?LOG_UNEXPECTED_MSG(Other),
wait_message(Node, Ref, Mon, Timeout)
end.

get_or_create_db(DbName, Options) ->
mem3_util:get_or_create_db_int(DbName, Options).