Now suppose the map-maker is handed a list of past crimes and where they took place, and asked to shade in where the next ones are likely to happen. Police then patrol the shaded areas.
More crime is recorded there simply because more officers are there to see it. Areas outside the shading appear to have less crime, because fewer people are looking. When the next map is drawn, the same areas are shaded even darker.
So the crime figures are not a full list of crimes. They are partly a record of where police have chosen to look.
The problem is as old as policing, but AI may be speeding it up.
Sarah Brayne, a sociologist at the University of Texas at Austin, spent time inside the Los Angeles Police Department studying how it uses big data. In her research paper, published in the American Sociological Review, she found that the force increasingly used data to predict crime, rather than only to respond to it.
She also found that the data fed into these systems was skewed. Crimes in public places were more likely to be seen and recorded, and police focus was disproportionately on black neighbourhoods. Once this goes into the system, she writes, “the predictions appear impartial”.
The models, she says, create “a feedback loop in which they not only predict events such as crime or police contact, but also contribute to their future occurrence.”
But that is a view challenged by Dr Dimitrios Kollias, an associate professor in multimodal artificial intelligence at the Queen Mary University of London (QMUL). He says that the technology has moved on to take the bias of historical data into account.
“The old-map analogy is useful for explaining how historical bias can be inherited, but it has its limits,” he told BBC News.
“Modern AI systems do not simply retrieve or copy their training examples; they can combine learned information in new ways and generate outputs and solutions that were not explicitly present in those examples.”
But those new insights pose a separate risk, Kollias adds.
“As AI systems become more capable of generating novel decisions and acting with greater autonomy, we need to ensure that their behaviour remains safe, reliable and under meaningful human control,” he said.
