# Improving performance results for BERT

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fimproving-performance-results-for-bert%2F2121%3E%5C
**Category:** 🤗Transformers\
**Created:** [November 19, 2020, 10:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fimproving-performance-results-for-bert%2F2121 "2020-11-19T10:53:26Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Stimmot](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@Stimmot](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FStimmot)%5C
**Post date:** [November 19, 2020, 10:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fimproving-performance-results-for-bert%2F2121%2F1 "2020-11-19T10:53:26Z")

</div>

I’m using the bert-base-german-cased model to perform token classification with custom NER labels on a dataset of German court documents. I have 11 labels in total (including the O label), which are however not tagged in BIO form. I’m letting the model train and evaluate on an NVidia GeForce GTX Titan X.

But despite the good ressources and the model, which was actually pretrained on German judicial documents, the results are rather lacking.

```auto
precision recall f1-score support

                              Date 0.87 0.99 0.93 407
                   Schadensbetrag 0.77 0.58 0.66 112
                            Delikt 0.59 0.50 0.54 44
                    Gestaendnis_ja 0.60 0.71 0.65 21
                    Vorstrafe_nein 0.00 0.00 0.00 6
Strafe_Gesamtfreiheitsstrafe_Dauer 0.76 0.91 0.83 35
          Strafe_Gesamtsatz_Betrag 0.42 0.52 0.46 25
           Strafe_Gesamtsatz_Dauer 0.52 0.82 0.64 28
                 Strafe_Tatbestand 0.30 0.29 0.30 283

                        micro avg 0.65 0.68 0.66 961
                        macro avg 0.54 0.59 0.56 961
                     weighted avg 0.64 0.68 0.66 961

```

What could be some steps to improve these results?  
Perhaps it’s the low data count for some of the labels, or that the labels often are not single tokens but text spans of multiple tokens?

I would be glad for every hint of some more experienced users. I can also share data or other files, if they are relevant.

This is my config file:

```auto
{
    "data_dir": "./Data",
    "labels": "./Data/labels.txt",
    "model_name_or_path": "bert-base-german-cased",
    "output_dir": "./Data/Models",
    "task_type": "NER",
    "max_seq_length": 180,
    "num_train_epochs": 6,
    "per_device_train_batch_size": 48,
    "seed": 7,
    "fp16": true,
    "do_train": true,
    "do_predict": true,
    "do_eval": true
}

```

---

<div class="post-metadata">

**Author:** ![Stimmot](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@Stimmot](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FStimmot)%5C
**Post date:** [November 25, 2020, 1:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fimproving-performance-results-for-bert%2F2121%2F2 "2020-11-25T13:52:53Z")

</div>

Anyone who could help on this topic?

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvblagoje)%5C
**Post date:** [November 25, 2020, 2:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fimproving-performance-results-for-bert%2F2121%2F3 "2020-11-25T14:27:24Z")

</div>

As you suggest I’d start with exploration of your dataset. See how many examples of each tag/token you have and if rebalancing improves your scores.