Transformer-based language models such as Bert suffer from a high complexity in open domain question answering. In this paper, we change their architecture to allow a more efficient management of computations. The resulting variants are competitive with original models and allow a significant speedup in both GPU and CPU.