Combining RAG with Continued Pretraining

For leaning purposes, I ran an experiment where I trained a model on a new domain using continued pretraining. Then to make it more flexible, I added a RAG step to inject dynamic data to augment the stable training data. The specific example is training qwen 3.5 4B on a fictional subway system to w

中文全文翻译正在补齐，可先查看下方英文原文。

For leaning purposes, I ran an experiment where I trained a model on a new domain using continued pretraining. Then to make it more flexible, I added a RAG step to inject dynamic data to augment the stable training data. The specific example is training qwen 3.5 4B on a fictional subway system to w