Mixture Of Experts: Old But Gold

Through the news, I've recently become aware of an interesting large language model (LLM) model architecture, Mixture of Experts (MoE), a concept which was actually established in a 1991 paper but has only recently come to prominence.

In MoE, each separate model is specialised, or expert, in one or more domains, subject to their training. During inference, depending on the nature of the particular prompt and the suitability of the particular model(s), only a subset of the overall models are called-upon. This reportedly improves computational efficiency and scalability.

MoE has advanced significantly from 2010 onwards, including scaling the concept up to a 100B+ parameter LSTM applied to natural language processing tasks in 2017. Historically, there have been a number of hurdles to overcome in order to realise MoE's full potential. For instance, due to the branching nature of MoE, such models have not been particularly suited to computation on graphical processing