Update tokenizer_config.json
#14
by
awni
- opened
This fixes the tokenizer to work with Transformers V5 otherwise you get the incorrect normalization.
See https://github.com/ml-explore/mlx-lm/issues/725 and https://github.com/huggingface/transformers/issues/43122 for relevant discussion.