Boosting Model Generalization with DoGE

Ever wondered how the data used to train large language models (LLMs) affects their ability to understand new information? Turns out, it's a big deal! The variety and mix of this data can make or break an LLM's performance. Currently, many LLMs rely on guesswork and trial and error to tweak how much different types of data influence the training process. Enter DoGE—a method designed to tackle this problem head-on. DoGE stands for Domain reweighting with Generalization Estimation. It's a smart two-step process that first trains a smaller 'proxy' model to figure out the best way to balance different data types during training. Once that's done, it uses these 'domain weights' to train a larger, more powerful model.By doing this, DoGE helps LLMs generalize better to new tasks and data they weren't trained on. This is crucial for improving things like predicting text or solving reasoning problems. In their tests, DoGE showed impressive results, beating baseline methods hands down on six different tasks using the SlimPajama dataset. What's more, DoGE can even identify how different data types relate to each other, which helps when dealing with tasks that the model hasn't seen before. This means DoGE can consistently perform better on these unseen tasks, making it a game-changer in the world of language models.