People determined to spread toxic messages online have taken to masking their words to bypass automated moderation filters.
Author
- Johnny Chan
Lecturer, Business School, University of Auckland, Waipapa Taumata Rau
A user might replace letters with numbers or symbols, for example, writing "Y0u're st00pid" instead of "You're stupid".
Another tactic involves combining words, such as "IdiotFace". Doing this masks the harmful intent from systems that look for individual toxic words.
Similarly, harmful terms can be altered with spaces or additional characters, such as "h a t e " or "h@te", effectively slipping through keyword-based filters.
While the intent remains harmful, traditional moderation tools often overlook such messages. This leaves users - particularly vulnerable groups - exposed to their negative impact.
To address this, we have developed a novel pre-processing technique designed to help moderation tools more effectively handle the subtle complexities of hidden toxicity.
An intelligent assistant
Our tool works in conjunction with existing moderation. It acts as an intelligent assistant, preparing content for deeper and more accurate evaluation by restructuring and refining input text.
By addressing common tricks users employ to disguise harmful intent, it ensures moderation systems are more effective. The tool performs three key functions.
It first simplifies the text. Irrelevant elements, such as excessive punctuation or extraneous characters, are removed to make text straightforward and ready for evaluation.
It then standardises what is written. Variations in spelling, phrasing and grammar are resolved. This includes interpreting deliberate misspellings ("h8te" for "hate").
Finally, it looks for patterns. Recurring strategies such as breaking up toxic words ("I d i o t"), or embedding them within benign phrases, are identified and normalised to reveal the underlying intent.
These steps can break apart compound words like "IdiotFace" or normalise modified phrases like "Y0u're st00pid". This makes harmful content visible to traditional filters.
Importantly, our work is not about reinventing the wheel but ensuring the existing wheel functions as effectively as it should, even when faced with disguised toxic messages.
Catching subtle forms of toxicity
The applications of this tool extend across a wide range of online environments. For social media platforms, it enhances the ability to detect harmful messages, creating a safer space for users. This is particularly important for protecting younger audiences, who may be more vulnerable to online abuse.
By catching subtle forms of toxicity, the tool helps to prevent harmful behaviours like bullying from persisting unchecked.
Businesses can also use this technology to safeguard their online presence. Negative campaigns or covert attacks on brands often employ subtle and disguised messaging to avoid detection. By processing such content before it is moderated, the tool ensures that businesses can respond swiftly to any reputational threats.
Additionally, policymakers and organisations that monitor public discourse can benefit from this system. Hidden toxicity, particularly in polarised discussions, can undermine efforts to maintain constructive dialogue.
The tool provides a more robust way for identifying problematic content and ensuring that debates remain respectful and productive.
Better moderation
Our tool marks an important advance in content moderation. By addressing the limitations of traditional keyword-based filters, it offers a practical solution to the persistent issue of hidden toxicity.
Importantly, it demonstrates how small but focused improvements can make a big difference in creating safer and more inclusive online environments. As digital communication continues to evolve, tools like ours will play an increasingly vital role in protecting users and fostering positive interactions.
While this research addresses the challenges of detecting hidden toxicity within text, the journey is far from over.
Future advances will likely delve deeper into the complexities of context-analysing how meaning shifts depending on conversational dynamics, cultural nuances and intent.
By building on this foundation, the next generation of content moderation systems could uncover not just what is being said but also the circumstances in which it is said, paving the way for safer and more inclusive online spaces.
Johnny Chan receives funding from Netsafe New Zealand.
