[Sharderformer] Support zbv in Sharderformer Policy #6150

duanjunwen · 2024-11-21T09:40:00Z

📌 Checklist before creating the PR

I have created an issue for this PR for traceability
The title follows the standard format: [doc/gemini/tensor/...]: A concise description
I have added relevant tags if possible for us to better distinguish different PRs
I have installed pre-commit: pip install pre-commit && pre-commit install

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge

e.g. fixed #1234, closed #1234, resolved #1234

📝 What does this PR do?

Summarize your work here.
if you have any plots/diagrams/screenshots/tables, please attach them here.

💥 Checklist before requesting a review

I have linked my PR to an issue (instruction)
My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
I have performed a self-review of my code
I have added thorough tests.
I have added docstrings for all the functions/methods I implemented

⭐️ Do you enjoy contributing to Colossal-AI?

🌝 Yes, I do.
🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

falcon,gptj,mistral,opt,qwen2,t5, vit, whisper

Col and Row.

base testcase;

for more information, see https://pre-commit.ci

colossalai/shardformer/layer/qkv_fused_linear.py

colossalai/shardformer/policies/gpt2.py

docs/source/en/features/zerobubble_pipeline_parallelism.md

colossalai/shardformer/layer/__init__.py

colossalai/shardformer/layer/qkv_fused_linear.py

docs/source/en/features/zerobubble_pipeline_parallelism.md

requirements/requirements.txt

tests/test_shardformer/test_layer/test_gpt2_qkv_fused_linear_1d.py

colossalai/shardformer/layer/_operation.py

docs/source/en/features/zerobubble_pipeline_parallelism.md

tests/test_shardformer/test_layer/test_gpt2_qkv_fused_linear_1d.py

docs/source/zh-Hans/features/zerobubble_pipeline_parallelism.md

[feat] Sharderformer support zbv

b31a052

duanjunwen requested a review from a team as a code owner November 21, 2024 09:40

[feat] support chatglm2, command, deepseek for zbv

5f89e7f

duanjunwen requested a review from ver217 November 21, 2024 10:58

duanjunwen added 8 commits November 22, 2024 05:38

[feat] support zbv in shardformer policy:

41e1972

falcon,gptj,mistral,opt,qwen2,t5, vit, whisper

Merge branch 'main' into feature/sharderformer_support_zbv

37a5a66

[feat] support GPT2FusedLinearConv1D

efffe6b

Merge branch 'main' into feature/sharderformer_support_zbv

2b94e00

[feat] support GPT2FusedLinear (without tp)

a84fc41

[fix] debug FusedConvLinear

014cc27

[shardfromer] support gpt2 policy for zbv, support GPT2FusedLinearConv

778d4df

Col and Row.

Merge branch 'main' into feature/sharderformer_support_zbv

8cb74e7

duanjunwen force-pushed the feature/sharderformer_support_zbv branch from ba7fc35 to 8cb74e7 Compare December 10, 2024 07:09

duanjunwen added 4 commits December 10, 2024 15:30

[Shardformer] support FusedLinear1D base for zbv

d168b73

[shardformer] support zbv in FusedLinear1D base, Col, Row

01a9cb3

[shardformer] support zbv in blip2 and sam policy

fc77b24

[shardformer] fix bug incorrect number of gradients; add fusedLinear

70b0ae1

base testcase;

duanjunwen force-pushed the feature/sharderformer_support_zbv branch from 3fd2402 to 70b0ae1 Compare December 10, 2024 08:50

[fix] fix incorrect number of gradients ;

37b670e

duanjunwen force-pushed the feature/sharderformer_support_zbv branch from 44b5786 to 37b670e Compare December 10, 2024 11:26

pre-commit-ci bot and others added 5 commits December 10, 2024 11:27

[pre-commit.ci] auto fixes from pre-commit.com hooks

94bb9ec

for more information, see https://pre-commit.ci

[Shardformer] add en doc for zbv;

dee1878

[fix] fix typo in Model compatibility table

83e670e

[fix] fix API Reference typo

2a55566

[Shardformer] add zh-Han doc for zbv

5430eb0

ver217 reviewed Dec 18, 2024

View reviewed changes

[fix] fix Linear name; update en & zh doc

25da23d

duanjunwen requested a review from ver217 December 18, 2024 06:02

duanjunwen added 2 commits December 18, 2024 14:32

[fix] fix shardformer doc import err

fd5bd33

[fix] fix shardconfig import in doc

c749a7c

duanjunwen added 11 commits December 23, 2024 13:40

[fix] fix pyramid, zope version

541664a

Merge branch 'main' into feature/sharderformer_support_zbv

ed76d69

[fix] try fix workflow

e592884

[fix] try import ShardConfig in yml

3b0669a

[fix] fix workflow

1cd60a0

[fix] fix workflow

573d5ce

[fix] fix workflow

938bf6d

[fix] fix workflow

90d1d53

Merge branch 'main' into feature/sharderformer_support_zbv

9b7940f

[fix] fix ci

aab6275

[fix] fix zbv doc

63b7db5

ver217 reviewed Dec 24, 2024

View reviewed changes

duanjunwen added 4 commits December 24, 2024 14:25

[fix] fix param for qkv linear, gpt2fused linear; fix requirments;

7fb23a5

[fix] fix policy use fused_linear

f0a8d78

[fix] fix weight grad none, err caused by weight ptr change

ff316c9

[fix] fix comm in WeightGradStore

f52c36e

duanjunwen requested a review from ver217 December 24, 2024 11:01

[fix] fix WeightGradStore pop param

feca06e

ver217 reviewed Dec 25, 2024

View reviewed changes

colossalai/shardformer/layer/_operation.py Outdated Show resolved Hide resolved

docs/source/en/features/zerobubble_pipeline_parallelism.md Outdated Show resolved Hide resolved

tests/test_shardformer/test_layer/test_gpt2_qkv_fused_linear_1d.py Outdated Show resolved Hide resolved

duanjunwen added 3 commits December 25, 2024 13:38

[fix] remove useless param in doc; fix gpt2 qkv test;

d74071a

[shardformer] simplify execute_w_pass_grad_accum;

c0b6fbc

[fix] rm useless comments

130b50c

duanjunwen requested a review from ver217 December 25, 2024 06:09

ver217 reviewed Dec 25, 2024

View reviewed changes

docs/source/zh-Hans/features/zerobubble_pipeline_parallelism.md Outdated Show resolved Hide resolved

duanjunwen added 2 commits December 25, 2024 17:12

[shardformer] simplify execute_w_pass_grad_accum & execute_w_pass

c4df1cc

[shardformer] Run meaningful doc test

52a3b88

duanjunwen requested a review from ver217 December 25, 2024 09:18

[shadformer] fix doc test cmd;

ee6bba9

ver217 approved these changes Jan 2, 2025

View reviewed changes

ver217 merged commit a9bedc7 into hpcaitech:main Jan 2, 2025
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Sharderformer] Support zbv in Sharderformer Policy #6150

[Sharderformer] Support zbv in Sharderformer Policy #6150

duanjunwen commented Nov 21, 2024

[Sharderformer] Support zbv in Sharderformer Policy #6150

[Sharderformer] Support zbv in Sharderformer Policy #6150

Conversation

duanjunwen commented Nov 21, 2024

📌 Checklist before creating the PR

🚨 Issue number

📝 What does this PR do?

💥 Checklist before requesting a review

⭐️ Do you enjoy contributing to Colossal-AI?