Silicon Sonnets

In-Context Language Learning: Arhitectures and Algorithms

January 25, 2024
Silicon Sonnets
In-Context Language Learning: Arhitectures and Algorithms
Show Notes

This study explores in-context learning (ICL) of neural language models on formal languages, showing Transformers outperform other models and identifying 'n-gram heads' as key to their success. It suggests architectural changes to improve ICL in models, with potential applications in natural language modeling.