Implement retry logic for lab tests

[nuclearcat]

In situations of unreliable test results kernel maintainers often need to rerun the tests to ensure the results are correct. This can be due to a variety of reasons such as flaky tests, network issues, or other transient failures.
Previous attempts to implement retry logic have been unsuccessful due to the way the event logic implemented. Even if we spoof pubsub event and retry build node, the event logic will fetch
nodeid with real data (different from spoofed) and will fail to trigger test event.
This time we will put more effort to implement retry logic for lab tests.
Further information will be posted as follow-up comments.

[JenySadadia]

Hello,
There was already an issue created for it by @hardboprobot
kernelci/kernelci-api#509

[nuclearcat]

OK i will migrate this issue as comment there with details according to initial issue