diff options
Diffstat (limited to 'docs/outreach')
| -rw-r--r-- | docs/outreach/EPT_TECHNICAL_SUMMARY.pdf | bin | 0 -> 24634 bytes | |||
| -rw-r--r-- | docs/outreach/EPT_TECHNICAL_SUMMARY.tex | 11 | ||||
| -rw-r--r-- | docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf | bin | 29672 -> 29813 bytes |
3 files changed, 7 insertions, 4 deletions
diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY.pdf b/docs/outreach/EPT_TECHNICAL_SUMMARY.pdf Binary files differnew file mode 100644 index 0000000..90c86d1 --- /dev/null +++ b/docs/outreach/EPT_TECHNICAL_SUMMARY.pdf diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY.tex b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex index 4ac89fc..ebf4102 100644 --- a/docs/outreach/EPT_TECHNICAL_SUMMARY.tex +++ b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex @@ -45,14 +45,17 @@ backprop seeds at 72M. Corpus is FineWeb-Edu with a 32k vocabulary. 11M & 4.194 & 4.197 & $-0.4\%$ (EP ahead)\\ 18M & 3.945 & 3.933 & $+1.2\%$\\ 36M & 3.649 & 3.648 & $+0.1\%$\\ -72M & rerun in progress & 3.288 & tracks the twins at matched progress\\ -135M & rerun queued & 3.090 & \\ +72M & 3.400 & 3.397 & $+0.4\%$\\ +135M & rerun in progress & 3.469 & level at matched progress\\ \bottomrule \end{tabular} \end{center} -Differences at the first three sizes are smaller than the spread between seeds, so EP and -backpropagation are statistically indistinguishable there. Wall clock cost of EP training in +All values are tail window means over the last tenth of training, which is the statistic we quote +everywhere. Every difference above is smaller than or comparable to the spread between backprop +seeds, which is 0.003 in cross entropy at 72M, so EP and backpropagation are statistically +indistinguishable at these sizes. The 135M entry compares both arms at the same point in an +unfinished run and carries one seed per arm, so it is provisional. Wall clock cost of EP training in simulation is about twice backpropagation. To our knowledge the 135M model is the largest language model trained from scratch with no backpropagation at any layer. diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf b/docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf Binary files differindex 19de900..d764699 100644 --- a/docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf +++ b/docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf |