Tokenizer configuration – MLX
Summary The issue at hand is related to incorrect tokenization when loading a model on MLX and using its tokenizer. This is due to an incorrect regex pattern in the tokenizer configuration. To resolve this, it’s essential to set the fix_mistral_regex flag to True when loading the tokenizer. Root Cause The root cause of this … Read more