Investigate and fix CPU/Memory inefficiencies in core services

[Pyatakov]

Problem description

During recent internal findings, we observed unusual CPU and/or memory usage patterns in policy-service, guardian-service, and analytics-service. Additionally, testing (as described in this document) highlighted similar issues in worker-service. These anomalies suggest potential memory leaks or inefficient resource utilization, which could impact system stability and performance over time. To ensure optimal resource usage effectiveness, we need to investigate these issues and address any underlying problems.

Requirements

1. Conduct a detailed investigation into the CPU and memory usage of the affected services (policy-service, guardian-service, analytics-service, and worker-services).
2. Identify potential memory leaks, excessive CPU consumption, and inefficient resource usage.
3. Use profiling tools and logs to analyze trends over time.
4. Reproduce the issue in a controlled environment if necessary.
5. Provide a report detailing findings, root causes (if identified), and proposed solutions.
6. Implement fixes or optimizations based on findings.
7. Validate improvements through additional testing and monitoring.

Definition of done

• Comprehensive investigation completed, and a report summarizing findings is available.
• Identified memory leaks or CPU inefficiencies have been addressed with appropriate fixes.
• Improvements in resource usage effectiveness are verified through testing and monitoring.
• No significant regressions are introduced in the system.

Acceptance criteria

1. Profiling and monitoring data confirm the presence of CPU/memory inefficiencies.
2. Root causes of inefficiencies are identified and documented.
3. Fixes are implemented and verified in a test/staging environment.
4. Improvements in resource usage effectiveness are confirmed via benchmarks or stress tests.
5. No critical bugs or regressions introduced as a result of the changes.
6. Changes are reviewed and approved by relevant team members.