spanbert

SpanBERT is an improvement over BERT and involves pre-training on span boundaries rather than individual tokens which improves performance on tasks like question answering and coreference resolution.