Using punctuation to prioritize the moderation queue
About 15 years ago, I was running moderation for a community with around a million monthly active users.
Threads were getting out of hand. We needed to work out which ones were likely to turn toxic, and we couldn't review everything manually. This was before today's AI moderation tools.
We already had a history of reports, so we went back through it and asked a simple question: what did the problem cases have in common?
Some of the answers looked almost silly. Posts with periods in the title were flagged 80% more often. Threads that started without a capital letter caused more trouble. Certain word combinations kept appearing in threads that needed intervention.
None of those details told us much by itself. Together, more than 100 of them gave us a surprisingly accurate scoring system.
The score gave our moderators somewhere to start. They could work through the threads most likely to need attention instead of taking an arbitrary pile in order. It wasn't perfect, but it helped us catch most problems before they got out of hand.
A proxy can still be useful
We're taught to validate the obvious things. Check the email format. Verify the phone number. Make sure required fields aren't empty. Those checks tell you whether the data looks valid, but they couldn't tell us which conversation was about to go wrong.
We had to use proxies instead. Punctuation and capitalization had no obvious causal relationship with a toxic thread. They were clues, learned from thousands of earlier cases, that helped us decide where limited human attention should go.
That distinction mattered. The system had a narrow job: put a smaller set of threads in front of a moderator quickly enough for a person to help. The moderator still read the conversation and made the call.
I've since found more strange correlations in other kinds of data. Some were useful and some were better left alone. I wrote about one of those decisions here.
The reports were useful for more than resolving individual complaints. Looking back through them gave us a way to decide which threads to read next, using details we'd previously ignored.