Using LLMs in Data Journalism Can Be Trustworthy, If These Five Elements Are In Your Methodology

AI may bode ill for the journalism industry as a whole, but for us data journalists, it opens up a new method for answering important questions in ways readers should trust.

LLM classification — and its cousin, extraction — sits in the Venn diagram overlap: an AI tool that helps us do more valuable, credible work, and that we can convince our readers to trust.

So often, data journalists deal with big, unwieldy piles of text — or worse, images and video. Maybe you have a spreadsheet full of police report narratives and you want to find those that involve a pursuit — to count them, chart them over time, and research them more deeply. Or you have 73,000 tweets from a loquacious trillionaire and you want to know how often he posts about politics.

If you wanted to classify those tweets by hand in 2022, you’d have had two options:

1. Define some keywords to catch the political tweets — but accept you’d fail to catch some rows with misspellings, circumlocutions, or keywords you hadn’t thought of.
2. Read every single tweet, create a new column in Excel, and for each row, write the correct category in it — perhaps assigning some poor intern to spend summer day after boring summer day categorizing tweets.

LLM classification workflows are a new option for us. It is a lot like option #2: you still wind up with a new column, containing a category for each row. But it’s filled in by a chatbot, not you.

Think of it like asking ChatGPT “Is this tweet about politics or not? ‘Vote for @realDonaldTrump if you want humanity to be a spacefaring civilization!‘“ for each row in your spreadsheet of tweets.

(If you want to learn the technical details of how to do this, I recommend Ben Welsh’s First LLM Classifier guide.)

But just asking ChatGPT about each row can’t be where you stop. Our audience doesn’t come to us to hear what ChatGPT has to say. In this era of fragmentation, they aren’t coming to us unless they already trust us.

Journalism is about what humans can see — us, our sources, or perhaps a census enumerator. That’s why this Wall Street Journal article where humans saw worrying signs of former president Joe Biden’s advancing age was credible, and why everyone mocked this Washington Times article that began “ChatGPT could not find one example in American history of a third-place candidate surging days after an election to overtake second place” and “SuperGrok estimates those odds at less than 1 in trillions.”

Because our audience already trusts us, they trust that we’d do a good job if we read every tweet in the spreadsheet. There is no angelic .xlsx in Plato’s world of forms with the one true right answer to whether each tweet is about politics — we are the ones who determine truth. It is not in heaven; when we disagree with the model’s conclusion, we are right, and it is wrong. (And if neither you nor your expert sources can determine if the model’s answer is right or wrong, you shouldn’t trust the model.)

So, you need to prove that ChatGPT reaches about the same conclusion as you would have, if you had painstakingly classified all 73,000 tweets or 5,000 arrest records.

The way you do this is to painstakingly classify a random subset of your data, (ideally a few hundred, but at least several dozen), then measure whether your AI is giving the same answer as you, calculating an accuracy rate. If the accuracy rate isn’t great, try another model or adjust your prompt until it is.

But it’s not enough to have done a good job. If we didn’t read the whole dataset, we have to prove to the audience why we can make claims about it — in our methodology.

When I write a methodology, I’m writing it for my friend Aaron, who is a professor in a specialized quantitative field. Aaron thinks data journalism is neat and his statistics chops are, frankly, better than mine. But Aaron mostly works with data about his own specialty; what fascinates him are the data acquisition hurdles and the analytical trade-offs involved in analyzing messier, timelier, newsier — and often novel — datasets.

If you think of someone who mostly works with one-row-per-person datasets of categorical and numeric data, 73,000 rows of natural language — or better, 90,000 frames of TV sports broadcasts — raises big questions about how to extract real meaning from such a pile of crap. It’s genuinely interesting to him, just like our news articles are (hopefully) interesting to our broader audience.

Here’s my opinion on how to fascinate him while also convincing him that I’m still worthy of his trust. Methodologies for AI workflows should include these ingredients:

1. What was the underlying dataset?
2. What model did you use?
3. How did you define whatever you’re classifying? (This definition is likely in the prompt — but you also used this definition implicitly when generating your evaluation set)
4. Size of the evaluation set
5. What accuracy rate did you get from comparing the AI’s answers on the evaluation set to your own? And what accuracy rate applies to the numbers in the story? (If different due to manual fixes)

I implemented this first in a recent article: “Elon Musk’s near-daily online posts about race are turning off some fans.“

With all of the ingredients, we can use LLM classification and extraction workflows to open up new possibilities for complex analyses — ones that audiences will trust, and that will be worthy of that trust.