mpnet (masked and permuted pre-training for language understanding)

MPNet introduces a novel self-supervised objective named Masked and Permuted Language Modeling, which improves upon BERT by allowing bidirectional contexts in a more flexible and effective manner.