Facebook, Instagram, and Twitter use algorithmic processing for image analysis and sharing. The platforms try to understand the content of these images, and they algorithmically detect the persons and objects within these images at the pixel level. Benjamin N. Jacobsen’s research, Regimes of recognition on algorithmic media, examines how algorithmic media shapes and controls how people see the social world. He defines the concept of regimes of recognition as the algorithmic tools, techniques, and practices that social media platform use to decide recognizability within their platforms. 

The platforms use algorithms to crop images to generate engaging content. Applicable to Jacobsen’s research, the algorithm systems learn to detect and weight certain features in images rather than others. In other words, the algorithms actively generate what is recognized. He agrees with researcher Louise Amoore, who believes social platforms decide what images are interesting and recognizable. Amoore states, “they actively generate recognizability as such so that they decide what or who is recognizable as a target of interest in an occluded landscape.”

Twitter’s detection algorithm

In September 2020, consumers complained that Twitter’s image-cropping algorithm favored white faces over black faces. One example showed that the algorithms consistently cropped out images of former president Barack Obama. Experiments with stock photo models, cartoon characters, and even white and black dogs resulted in similar detection biases. Eventually, Twitter’s chief design officer, Dantley Davis, agreed there was a racial bias in the algorithm. He added, “the algorithm is not explicitly racist since it does not make its decision based on particular faces but rather on the contrasts that are calculated from the pixel values of the image.”

According to Twitter engineers Lucas Theis and Zehan Wang, the company first used facial recognition to auto-crop images. However, it often awkwardly cropped faces not centered within the picture. Twitter then moved to use deep saliency prediction networks to find the “most interesting part” of the image. Saliency maps use predefined areas of images to evaluate the pixel contrasts between different image regions. These data-trained neural networks and other algorithms find and then crop around the most engaging area of the image. Unfortunately, as the algorithms decide what is uploaded, they also determine what is nonrecognizable and set conditions for who is visible on the platform.

Deciding recognizability

The algorithm disregards socio-cultural contents and context in its processing of images. Regimes of recognition can easily make certain people non recognizable — focusing on specific features of an image and ignoring others. Algorithmic media platforms produce rules for what is visible and what is not visible.

Jacobsen questions the shaping, organizing, and automating parameters of cropping images. He sees decisions of inclusion and exclusion as a fundamental political question. Saliency detection algorithms rest on the assumption that some things deserve attention and others do not. Further, Twitter’s role in cropping people’s images (algorithmically) decides what elements in images are redundant. This begs a larger question as to why social media platforms need to edit people’s uploaded images at all?

Social media platforms algorithmically shape and organize people’s factors of attention. Twitter offers a prime example of the algorithmic processes in how it cropped out out black faces because it did not recognize the image. This sends a signal that that what is non-recognizable is unimportant. Further research is needed to examine the role (and potential biases) of algorithmic cropping, techniques, and results of these tools on other social platforms.