[qwen2 audio] remove redundant code and update docs #36282
Conversation
| @@ -851,7 +776,7 @@ def forward(self, audio_features): | |||
| class Qwen2AudioForConditionalGeneration(Qwen2AudioPreTrainedModel, GenerationMixin): | |||
| def __init__(self, config: Qwen2AudioConfig): | |||
| super().__init__(config) | |||
| self.audio_tower = AutoModel.from_config(config.audio_config) | |||
| self.audio_tower = AutoModel.from_config(config.audio_config) # Usually a `Qwen2AudioEncoder` instance | |||
There was a problem hiding this comment.
There were no references of Qwen2AudioEncoder usage in our codebase nor on the hub. However, upon closer inspection of checkpoints, we can see it is used here. Added a comment for clarification.
(btw, Qwen2AudioEncoder is untested 😢 )
| @@ -1259,78 +1184,5 @@ def forward( | |||
| attention_mask=attention_mask, | |||
| ) | |||
|
|
|||
| def prepare_inputs_for_generation( | |||
There was a problem hiding this comment.
Doesn't seem to be needed? No tests fail. (the only custom code is in L1280-L1285, and _merge_input_ids_with_audio_features is related to the legacy processing)
It has the old, outdated pattern, so we would have to rewrite it anyway. If we get issues, then we can add the new prepare_inputs_for_generation pattern, with the correction
| def _reorder_cache(self, *args, **kwargs): | ||
| return self.language_model._reorder_cache(*args, **kwargs) |
There was a problem hiding this comment.
Only used in models that use the legacy cache format (not the case here)
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
What does this PR do?
Carved from #36238: Removes redundant code and updates outdated docs in
qwen2_audio.In a nutshell, the attention layers were copied from
whisper. However, inqwen2_audio, the attention layers are exclusively used in the encoder, and thus never use cache. Removing cache-related code there prevents us from having to refactor every time we revisit caches, and results in more readable code.The following bits were also updated in this PR: