RoFormer paper explained and implemented in JAX

In this article, we will go through the RoFormer paper, which introduced rotary positional embedding for transformer architecture and positional encodings. Also, we will implement it using the JAX deep learning framework

RoFormer: Enhanced Transformer with Rotary Position Embedding