[Consensus] Make consensus pipeline self-sending optional.

[JoshLind]

Description

This PR makes self-sending in the consensus pipeline optional, such that it does not occur for consensus observer (CO). This is prompted by a spurious error log for CO that self sends are failing (see: #14559), however, self-sends are not required for CO.

Testing Plan

Existing test infrastructure.

[trunk-io]

⏱️ 2h 31m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
forge-e2e-test / forge	37m	🟥 🟩
forge-compat-test / forge	34m	🟩 🟩
execution-performance / test-target-determinator	18m	🟩 🟩
test-target-determinator	16m	🟩 🟩
check	10m	🟩 🟩
rust-doc-tests	5m	🟩
rust-doc-tests	5m	🟩
general-lints	5m	🟩 🟩 🟩
rust-cargo-deny	5m	🟩 🟩 🟩
check-dynamic-deps	4m	🟩 🟩 🟩
indexer-grpc-e2e-tests / test-indexer-grpc-docker-compose	3m	🟩 🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
rust-move-tests	1m	🟩
semgrep/ci	1m	🟩 🟩 🟩
file_change_determinator	37s	🟩 🟩 🟩
file_change_determinator	29s	🟩 🟩 🟩
file_change_determinator	26s	🟩 🟩
execution-performance / single-node-performance	24s	🟩 🟩
permission-check	9s	🟩 🟩 🟩
permission-check	8s	🟩 🟩 🟩
permission-check	8s	🟩 🟩 🟩
permission-check	7s	🟩 🟩 🟩
permission-check	5s	🟩 🟩
determine-docker-build-metadata	4s	🟩 🟩
Backport PR	3s	🟥
permission-check	3s	🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
execution-performance / single-node-performance	11s	18m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[zekun000]

which messages trigger the errors? it might be useful to listen to this message to make epoch switch smoother

aptos-core/consensus/src/pipeline/buffer_manager.rs

Line 501 in 574fa48

.send_epoch_change(EpochChangeProof::new(vec![commit_proof], false))

currently we rely on upstream to send a message from a new epoch and call into state sync to enter new epoch but we should be able to avoid that

[github-actions]

✅ Forge suite realistic_env_max_load success on 4beb28710a8ca7e24bb90c7dfcce8e74aa3d0f35

two traffics test: inner traffic : committed: 13581.60 txn/s, submitted: 13582.84 txn/s, failed submission: 1.24 txn/s, expired: 1.24 txn/s, latency: 2930.02 ms, (p50: 2700 ms, p70: 3000, p90: 3300 ms, p99: 5200 ms), latency samples: 5164010
two traffics test : committed: 98.67 txn/s, submitted: 100.02 txn/s, failed submission: 1.35 txn/s, expired: 1.35 txn/s, latency: 2400.30 ms, (p50: 2300 ms, p70: 2400, p90: 2600 ms, p99: 8100 ms), latency samples: 1780
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.243, avg: 0.221", "QsPosToProposal: max: 0.578, avg: 0.385", "ConsensusProposalToOrdered: max: 0.330, avg: 0.306", "ConsensusOrderedToCommit: max: 0.442, avg: 0.423", "ConsensusProposalToCommit: max: 0.762, avg: 0.729"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.65s no progress at version 2878664 (avg 0.21s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 9.45s no progress at version 2877313 (avg 9.45s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on 70806ff543496aa6de7807feff49e7e1370efd20 ==> 4beb28710a8ca7e24bb90c7dfcce8e74aa3d0f35

Compatibility test results for 70806ff543496aa6de7807feff49e7e1370efd20 ==> 4beb28710a8ca7e24bb90c7dfcce8e74aa3d0f35 (PR)
1. Check liveness of validators at old version: 70806ff543496aa6de7807feff49e7e1370efd20
compatibility::simple-validator-upgrade::liveness-check : committed: 11662.43 txn/s, latency: 2426.54 ms, (p50: 2000 ms, p70: 2400, p90: 3100 ms, p99: 16200 ms), latency samples: 475760
2. Upgrading first Validator to new version: 4beb28710a8ca7e24bb90c7dfcce8e74aa3d0f35
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 8057.60 txn/s, latency: 3506.71 ms, (p50: 3800 ms, p70: 4000, p90: 4100 ms, p99: 4300 ms), latency samples: 147720
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 8042.35 txn/s, latency: 3933.44 ms, (p50: 4100 ms, p70: 4200, p90: 5700 ms, p99: 6200 ms), latency samples: 265580
3. Upgrading rest of first batch to new version: 4beb28710a8ca7e24bb90c7dfcce8e74aa3d0f35
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 6087.84 txn/s, latency: 4765.12 ms, (p50: 4000 ms, p70: 6400, p90: 6800 ms, p99: 7000 ms), latency samples: 112880
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7306.63 txn/s, latency: 4342.50 ms, (p50: 4400 ms, p70: 4700, p90: 6300 ms, p99: 6800 ms), latency samples: 243300
4. upgrading second batch to new version: 4beb28710a8ca7e24bb90c7dfcce8e74aa3d0f35
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 10373.29 txn/s, latency: 2373.20 ms, (p50: 2500 ms, p70: 2700, p90: 3100 ms, p99: 3800 ms), latency samples: 206060
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10458.12 txn/s, latency: 2936.65 ms, (p50: 2600 ms, p70: 2700, p90: 5900 ms, p99: 7700 ms), latency samples: 340040
5. check swarm health
Compatibility test for 70806ff543496aa6de7807feff49e7e1370efd20 ==> 4beb28710a8ca7e24bb90c7dfcce8e74aa3d0f35 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking