diff options
Diffstat (limited to 'docs/outreach/EPT_TECHNICAL_SUMMARY.tex')
| -rw-r--r-- | docs/outreach/EPT_TECHNICAL_SUMMARY.tex | 12 |
1 files changed, 9 insertions, 3 deletions
diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY.tex b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex index ebf4102..8552afb 100644 --- a/docs/outreach/EPT_TECHNICAL_SUMMARY.tex +++ b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex @@ -55,9 +55,15 @@ All values are tail window means over the last tenth of training, which is the s everywhere. Every difference above is smaller than or comparable to the spread between backprop seeds, which is 0.003 in cross entropy at 72M, so EP and backpropagation are statistically indistinguishable at these sizes. The 135M entry compares both arms at the same point in an -unfinished run and carries one seed per arm, so it is provisional. Wall clock cost of EP training in -simulation is about twice backpropagation. To our knowledge the 135M model is the largest language -model trained from scratch with no backpropagation at any layer. +unfinished run and carries one seed per arm, so it is provisional. To our knowledge the 135M model is +the largest language model trained from scratch with no backpropagation at any layer. + +Training cost is 2.7 times backpropagation in wall clock, measured on the same GPU with the same +model and batch and with all other load suspended, and 3 times in analytic FLOPs. We quote the +measured ratio against backprop rather than a relaxation-iteration count, because iteration counts in +this literature measure different operations: the closest comparable work updates layers +asynchronously, so one of its iterations moves information about one layer, while one of ours +transmits through the whole stack. That work reports 6.7 to 12 times backprop in wall clock. \section{A result from this week, and why it matters for how we work} Between 72M and 135M we had been measuring a quality loss that grew with model width, resisted every |