mbart50 & m2m models now use huggingface transformers
The mbart50 & m2m models required in version 1 the fairseq library. This caused several issues: fairseq cannot be used on Windows, multi-processing did not work with fairseq models, loading and using the models were quite complicated.
With this release, the fairseq dependency is removed and mbart50 / m2m models are loaded with huggingface transformers version >= 4.4.0
From a user perspective, no changes should be visible. But from a developer perspective, this simplifies the architecture of EasyNMT and allows new futures more easily be integrated.
Saving models
Models can now be saved to disc by calling:
model.save(output_path)Models can be loaded from disc by calling:
model = EasyNMT(output_path)Loadings models from huggingface model hub
Loading of any Huggingface Translation Model is now simple. Simply pass the name or the model path to the following code:
from easynmt import EasyNMT, models
article = """EasyNMT is an open source library for state-of-the-art neural machine translation. Installation is simple using
pip or pre-build docker images. EasyNMT provides access to various neural machine translation models. It can translate
sentences and documents of any length. Further, it includes code to automatically detect the language of a text."""
model = EasyNMT(translator=models.AutoModel('facebook/mbart-large-en-ro'))
print(model.translate(article, source_lang='en_XX', target_lang='ro_RO'))This loads the facebook/mbart-large-en-ro model from the model hub.
Note: Models might use different language codes, e.g. the mbart model uses 'en_XX' instead of 'en' and 'ro_RO' instead of 'ro'. To make the language code consistent, you can pass a lang_map:
from easynmt import EasyNMT, models
article = """EasyNMT is an open source library for state-of-the-art neural machine translation. Installation is simple using
pip or pre-build docker images. EasyNMT provides access to various neural machine translation models. It can translate
sentences and documents of any length. Further, it includes code to automatically detect the language of a text."""
output_path = 'output/mbart-large-en-ro'
model = EasyNMT(translator=models.AutoModel('facebook/mbart-large-en-ro', lang_map={'en': 'en_XX', 'ro': 'ro_RO'}))
#Save the model to disc
model.save(output_path)
# Load the model from disc
model = EasyNMT(output_path)
print(model.translate(article, target_lang='ro'))