Facebook’s Rosetta AI can read all the memes

There are so many memes on Facebook and Instagram that the company has enlisted its artificial intelligence to help understand them.

In a blog post, Facebook developers say they have created a dedicated AI tool, called Rosetta, to read the text that appears in memes (and other images and video frames) that are shared to Facebook and Instagram.

At face value, understanding memes might not seem like the most important problem for AI to solve. But Facebook’s researchers point out that the technology, which is designed to recognize depicted text in a wide variety of languages, has many practical uses.

For example, Rosetta can read photographed text in menus and street signs, as well as words appearing on clothing and product labels. So while Rosetta isn’t dedicated to memes, their prevalence on Facebook and Instagram will undoubtedly make them a major use case, especially in Facebook’s detection of offensive material.

“Understanding the text that appears on images is important for improving experiences, such as a more relevant photo search or the incorporation of text into screen readers that make Facebook more accessible for the visually impaired,” Facebook explains, adding that reading text in images is important in identifying “inappropriate or harmful content and keep our community safe.”

According to Facebook, the system is able to process more than a billion images a day.

The process essentially boils down to two steps: Rosetta scans images for text, and then uses text recognition to identify what the text actually says. Once the text has been transcribed, the system interprets what the text could mean.

Through Rosetta, Facebook is able to improve its image search as well as the systems that determine the types of images that may appear in your News Feed. It also helps the company automatically detect and remove hate speech that it may not have been able to identify previously.

In the future, Facebook says it could apply the same technology toward understanding text that appears in video as well, though that requires a more complex system.