Job Dispatcher

[adamdriscoll]

Summary of the new feature / enhancement

The PowerShell Universal job system is built on Hangfire. While hangfire provides a lot of great functionality for the platform, it has limitations we have tried to work around for too long and causes a lot of weird problems that are hard to track down when environments start to grow.

I'd like to introduce a job dispatcher service that provides an intelligent layer between the hangfire scheduler and job execution and PowerShell Universal.

The general concept is that all jobs will be put on a dispatch queue within hangfire. The Job Dispatcher service will be responsible for deciding how to deal with those jobs. This means that every job execution and schedule job goes on a single hangfire queue. Any online PowerShell Universal node can process an item on this queue and determine where the job should run.

For example:

• Default queue -> push job to default job
• Computer Group (All) -> push job to every computer on the queue, minus jobs in maintenance mode
• Computer Group (Any) - push to a single computer in the computer group
• Computer - Push to that computer's queue

Schedule jobs will work the same way. They will be queued in the dispatcher and then redirected to the proper entity for processing.

Each node will then subscribe to 3 queues.

• default queue
• dispatcher queue
• computer queue

This simplifies recurring schedules by only having a single recurring schedule in hangfire and the dispatcher will redirect it to the proper node at runtime. This reduces the complexity of the job state in PSU to avoid issues where the job has disappeared in hangfire but still queued in PSU.

This also lets us implement more complex logic to determine job priority and computer preference for jobs. We can implement some logic to do more than just a round robin for job execution and provide tools to dictate when and how to run jobs.

We also benefit by still using the hangfire scheduling and job execution system.

Proposed technical implementation details (optional)

No response

[adamdriscoll]

Thinking about it more, we can probably get rid of the default queue. We probably only need the dispatcher and computer queues.

[bt-ndollimount]

Well I had planned to stick with 5.3.0 for a while once it was released but you had to go and log this feature that I won't be able to resist installing 😆 This is a wonderful idea and I'm excited for its implementation!

[adamdriscoll]

We have the basic implementation of this running in 5.3 as an optional feature. It does everything that current scheduler does. I'll make sure this is documented on release so folks can play with it if they want.