Discovering Versatile Language Models: The Domain-General Lottery

Ever thought about how language models learn from multiple domains? Well, turns out, not all parameters in these models behave the same way. Some act differently depending on the domain, while others stay consistent. This can lead to both strengths and weaknesses. Imagine you're trying to build a language model that's really good at understanding texts from various sources. You'd want it to be consistent, right? That's where domain-general parameters come in. They're the ones that stay the same regardless of the domain. So, how do we find these versatile parameters? Researchers have come up with an interesting method: they call it 'doge tickets'. It's like a lottery where they pick out the most consistent parameters.To play this lottery, they've created a special score. This score helps them figure out how consistent a parameter is across different domains. The higher the score, the more likely a parameter is to be versatile. They tested this on some well-known datasets, like Amazon, Mnli, and OntoNotes. Guess what? The 'doge tickets' did really well. They improved the model's ability to understand texts from outside the original domains. But it gets better. Their analysis also showed that these domain-general parameters aren't just a lucky find. They're real and they make a big difference in how well the model can adapt to new tasks.