There does not seem to be any consensus in the community about when to stop pre-training or how to interpret the loss coming from BERT's self-supervision. We take the approach of BERT's original authors and evaluated the model performance on downstream tasks
. As tasks we gathered the following German datasets:
- germEval18Fine: Macro f1 score for multiclass sentiment classification
- germEval18coarse: Macro f1 score for binary sentiment classification
- germEval14: Seq f1 score for NER (file names deuutf.*)
- CONLL03: Seq f1 score for NER
- 10kGNAD: Accuracy for document classification
Even without thorough hyperparameter tuning
, we observed quite stable learning especially for our German model. Multiple restarts with different seeds produced quite similar results.
While outperforming on 4 out of 5 task we wondered why the German BERT model did not outperform on CONLL03-de. So we compared English BERT with multilingual on CONLL03-en and found them to perform similar as well.
We further evaluated different points during the 9 days of pre-training and were astonished how fast the model converges to the maximally reachable performance. We ran all 5 downstream tasks on 7 different model checkpoints - starting with a completely untrained model. Most checkpoints are taken from early training where we expected most performance changes. Surprisingly, even a completely random BERT can be trained with small downstream datasets from scratch and nearly reach top performance (blue line, GermEval 2018 Coarse task, 795 kB trainset size).