AI Text Still Stands Out Even After Tweaks

Computer programs that write text still look different from what people write. They are used in social science studies to act like humans, but we have no good way to see if they really match.

The new test called CHAT mixes several clues to see how close the computer text is to real conversation. It looks at how often the text is flagged as machine made, how close its meaning matches, and what style or topics it shows, to see how humanlike the AI language really is.

We tested nine open models using five ways to adjust them, such as fine tuning, style prompts, or pulling in context, on posts from X, Bluesky and Reddit. Our results show that even after tweaking, the AI words still stand out, especially in how they feel and express emotion. Models that have been taught with instructions usually do worse than plain ones, and making them bigger does not clearly make them sound more human. We also see a trade‑off: pushing for a more human feel often pushes the meaning farther from what real human replies say.

So the CHAT tool gives researchers a way to check AI talk, but it also shows that current models can't fully copy real conversation. This pushes us to think about better ways to measure and improve AI language.