Measuring the mood of a live chat is harder than it looks

ChatMood reads the chat of a live broadcast and answers a question every presenter asks themselves: is the audience enjoying this? Nobody on air can read a thousand messages a minute and talk at the same time.

I assumed it was a model problem. It was not.

Chat looks nothing like a product review

Sentiment analysis grew up on well-behaved text. Shop reviews, satisfaction surveys, form answers. Full sentences, a stated subject, one opinion per text.

Broadcast chat is the opposite. Messages run three words long, half of them are emotes, and irony is the default rather than the exception. The word “clap” can be praise or mockery, and what decides it sits in whatever happened ten seconds earlier, never in the message itself. Then there is repetition: a joke lands, three hundred people send the same line in twenty seconds, and a naive classifier counts three hundred votes where there is one.

Where the time actually went

Three problems took more time than the model did.

The first was collapsing repetition. Identical messages inside a short window become one weighted event rather than N independent opinions.

The second was holding the stream. A chat spike cannot take down the dashboard or delay the reading, because what counts is the trend of the minute and it has to arrive while that minute still matters.

The third was showing without lying. A 0 to 100 score implies a precision that is not there. The dashboard shows direction and intensity over time, which is enough for somebody to decide whether to change the subject.

What I carry into client work

When someone asks me for sentiment analysis, my first question is not which model. It is what decision the measurement will support, who takes that decision, and with how many seconds of warning.

If the answer is a monthly report, almost any approach will do. If it is somebody deciding live, the problem stops being text classification and becomes delivering a readable signal inside a short window. At that point the engineering around the model weighs more than the model.

Need this solved at your company?

A twenty minute call is usually enough to tell whether it makes sense, and you talk straight to the engineer who builds it.

Talk to the studio ↗︎