Set tokenizer "model_max_length" property to 8192
#39
by NohTow - opened
Somehow composer exported model_max_length tokenizer property to a very huge value instead of 8192.
This breaks the tokenizer.model_max_length call that some pipelines rely on.
As we corrected max_pos_embeddings, I suggest we also fix this for consistency, although this is not an hard limit.
See this issue for more information.
bwarner changed pull request status to merged