bigbird

BigBird, developed by Google, is designed to handle longer context. It combines sparse attention mechanisms with the original transformer model to efficiently handle sequences of length up to 8K tokens.