colin
/
Witllm
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

更新 Readme.md

This commit is contained in:
colin 2023-12-31 17:42:21 +08:00
parent 1579e0b8f2
commit bde8f71a7f
1 changed files with 3 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
Readme.md
View File

@ -107,8 +107,9 @@ Linear(intermediate_parallel) no bias -> [6, 1, 4096]
| expand expand -> [6, 1, 32, 128] | expand expand -> [6, 1, 32, 128]
\ / | \ / |
┏---- dot | ┏---- dot |
┃ softmax / ┃ += attention_mask /
attention┃ \ / attention┃ softmax /
┃ \ /
┗---- dot -> [1, 32, 6, 128] -> [6, 1, 4096] ┗---- dot -> [1, 32, 6, 128] -> [6, 1, 4096]
Linear -> [6, 1, 4096] Linear -> [6, 1, 4096]