add draft_gpu_split option for spec decoding #254
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
just as I sent this I realized the sample config in the root of the project would need to have this option added too, I'm exclusively an in-line user so it escaped my mind entirely. |
The previous code overrode the existing gpu split and device idx values. This now sets an independent draft_gpu_split value and adjusts the gpu_devices check only if the draft_gpu_split array is larger than the gpu_split array. Draft gpu split is not Tensor Parallel, and defaults to gpu_split_auto if a split is not provided. Signed-off-by: kingbri <[email protected]>
|
@lucyknada please check the changes and see if they work. They run properly on my end. However, I only have one GPU. |
|
confirming it works! thanks @kingbri1 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
with a setup of 3090+3060; loading 32b (5bpw) and 1.5b or 0.5b draft (6bpw) first loads the draft model and then the model; making an otherwise 32b fitting onto the single 3090 (a lot more bandwidth) leak onto the 3060 with quite a few layers, negating all benefits of the draft model and even dropping performance even more (15-30% less tps); now pinning the draft model to another gpu (the 3060) entirely bypasses that and you get the same performance boost for spec decoding as before.
tabby_config.yml example of a draft_gpu_split:
I have tested it both with autosplit on the target model and manually splitting via gpu_split, the gpu-split syntax stays the same to be able to re-use the same code.
this is merely a kick-off PR to make it work, as I'm not sure if there's anything else that needs to be modified for this to be upstreamable and if there's some neat python tricks to not have to duplicate code in e.g. initializing gpu_device_list.