Enhancement: Implement Cyclic Learning Rate and Step-wise Learning Rate Scheduler

[Franklalalala]

Background

Our current learning rate scheduler has limitations that impact training efficiency and flexibility, especially for large datasets:

1. It lacks support for Cyclic Learning Rate (CLR), which has shown promising fast-convergence ability, particularly in the initial training stages.
2. The current scheduler only functions on an epoch-scale, which may be inappropriate for extremely large datasets. (e.g. QH9)

Describe the solution you'd like

I propose the following enhancements:

1. Implement Cyclic Learning Rate (CLR) support
2. Add step-wise learning rate update capability

1. Cyclic Learning Rate (CLR) Support

• Add CLR support in the train_options/lr_scheduler field
• Implement common CLR policies (e.g., triangular, triangular2, exp_range)
• Add necessary arguments for CLR configuration (e.g., base_lr, max_lr, step_size)

2. Step-wise Learning Rate Updates

• Modify the learning rate scheduler to support updates on a per-step basis
• Add a new option in train_options/optimizer field to switch between epoch-wise and step-wise updates
• Implement necessary logic to track global steps and update learning rate accordingly

3. Unit-test

Additional Context

I will propose a PR as soon as possiable!