The punctuation told us which threads would explode

About 15 years ago, I was running moderation for a community with around a million monthly active users.

Threads were getting out of hand. We needed to work out which ones were likely to turn toxic, and we couldn't review everything manually. This was before today's AI moderation tools.

We already had a history of reports, so we went back through it and asked a simple question: what did the problem cases have in common?

Some of the answers looked almost silly. Posts with periods in the title were flagged 80% more often. Threads that started without a capital letter caused more trouble. Certain word combinations kept appearing in threads that needed intervention.

None of those details told us much by itself. Together, more than 100 of them gave us a surprisingly accurate scoring system.

The score gave our moderators somewhere to start. They could work through the threads most likely to need attention instead of taking an arbitrary pile in order. It wasn't perfect, but it helped us catch most problems before they exploded.

A proxy can still be useful

We're taught to validate the obvious things. Check the email format. Verify the phone number. Make sure required fields aren't empty. Those checks tell you whether the data looks valid, but they couldn't tell us which conversation was about to go wrong.

We had to use proxies instead. Punctuation and capitalization had no obvious causal relationship with a toxic thread. They were clues, learned from thousands of earlier cases, that helped us decide where limited human attention should go.

That distinction mattered. The system had a narrow job: put a smaller set of threads in front of a moderator quickly enough for a person to help. The moderator still read the conversation and made the call.

I've since found more strange correlations in other kinds of data. Some were useful and some were better left alone. I wrote about one of those decisions here.

I still like this way of approaching validation. When you can't inspect everything, look at the failures you already have. The useful warning may be hiding in a detail you would normally throw away.