Silicon Sonnets

Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs

January 29, 2024
Silicon Sonnets
Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs
Show Notes

This research compares unsupervised fine-tuning and retrieval-augmented generation (RAG) for injecting knowledge into large language models (LLMs). RAG consistently outperforms fine-tuning for both existing and new knowledge, suggesting that LLMs struggle to learn new facts through unsupervised fine-tuning alone.