Hugging Face's logo

answerdotai
/
ModernBERT-base

Fill-Mask
Transformers
PyTorch
ONNX
Safetensors
English
modernbert
masked-lm
long-context
Model card Files
xet
 Community
85

Training with transformers API

#75
by Padajno - opened
Discussion
Padajno

Hi,
in my experiments, I was able to train ModernBERT by relying on the Transformers library (specifically, using a modified run_mlm.py script from the transformers github: https://github.com/huggingface/transformers/blob/v4.49.0/examples/pytorch/language-modeling/run_mlm.py).
But if I understand correctly, this approach does not utilize the local/global attention ModernBERT implements?
wilfoderek

Hey! could you share your colab ?
wilfoderek

Hi,
in my experiments, I was able to train ModernBERT by relying on the Transformers library (specifically, using a modified run_mlm.py script from the transformers github: https://github.com/huggingface/transformers/blob/v4.49.0/examples/pytorch/language-modeling/run_mlm.py).
But if I understand correctly, this approach does not utilize the local/global attention ModernBERT implements?

It uses RoPE.

Sign up or log in to comment