Chat the Leg
Journalists worried that ChatGPT will eliminate their jobs can rest easy.
The incredibly powerful technology may write plays like Shakespeare and pass the legal bar exam, but it does a pathetic job at reporting, which is the heartbeat of journalism. I enlisted the highly touted technology and master of the Internet to be a digital leg man in a search for public records to find out if Wake County officials fairly assessed my home for tax purposes, a simple journalistic task. Did tax officials in Wake County North Carolina charge me the same property taxes as they did on a house across the street with the same square footage?
photo courtesy Chicago Tribune
To answer these questions, I created a digital pal I nicknamed Chat the Leg for a job that Mike Royko, the late and legendary Chicago newspaper columnist, used to give to young reporters he hired as leg men to do grunt work for the five columns he wrote with flair each week for Chicago newspapers.
I also got some help from Hanke Gratteau, who worked as Royko’s leg at the now closed Chicago Daily News, the Chicago Sun-Times and the Chicago Tribune newspapers. To give Chat the Leg some guidance, I asked Hanke what a leg man did for the famed columnist? She said everything from interviewing sources angry at Mike for past columns to poring over property tax records involved in Cook County property tax sales, a corrupt system and favorite Royko target. “We’d go through all of the tips he got to determine which ones he wanted to pursue, things like that,” she recalled. “A lot of what I did was just reporting, you know, looking up lawsuits or property tax records.”
So, I tasked Chat the Leg to ask ChatGPT about property tax records in Wake County, North Carolina. My original idea was to see if the highly touted technology would produce records that revealed patterns of inequity, a reporting assignment that would have been as basic to Gratteau as a press pass.
We started simple. Since we tapped ChatGPT 3.5, a version that lacks data beyond September 2021, I told Chat the Leg to stick to 2020. The first question?
Chat the Leg: What is the 2020 Wake County property tax valuation for the residence located at 111 Woodglen Drive in Cary, North Carolina, (my home)?
ChatGTP: I’m sorry, but as an AI language model, I don’t have access to up-to-date property tax information for specific addresses or locations. However, you can find the Wake County property tax valuation for the residence located at 111 Woodglen Drive in Cary, North Carolina by visiting the Wake County Revenue Department’s website or by contacting them directly. The website is wakegov.com/tax. You can find the phone number and other information for the Revenue Department on their website as well. They should be able to provide you with the most accurate and up-to-date information on the property tax valuation for this specific address.
Really? Come on! I could have done that myself. Besides, I’m asking for records from 2020, hardly "up to date." With ChatGPT’s infinite wisdom, it surely can do better than that! Try again.
Chat the Leg: Okay. ChatGPT, why can’t you provide access to Wake County North Carolina property tax records that are public information from 2020 and available on the Internet?
ChatGPT: Something went wrong. If this issue persists, please contact us through our help center at help.openai.com.
Undeterred, our man upped the pressure.
Chat the Leg: Let’s make it simple, ChatGPT. How many residents of Wake County North Carolina appealed their property tax valuations in 2020?
At first, ChatGPT spit out its “Something went wrong message” again. But Chat the Leg persisted and got this gem:
ChatGPT: I apologize for the confusion in my previous response. As an AI language model, I do not have access to the exact number of residents of Wake County, North Carolina, who appealed their property tax assessments in 2020. However, according to a news article published by WRAL.com, more than 15,000 Wake County property owners appealed their property tax assessments in 2020, which was a significant increase from the previous year.
Are you kidding me? ChatGPT referred my leg man to a news article? That’s like sending me to a Chevy dealer for a Tesla! When I went to the help line for OpenAI, the godfather of ChatGPT, I got a response that the Internet address was not recognized. Chat the Leg pressed on with several other questions about property taxes, but the answers were always the same “I don’t know”.
ChatGPT’s lame responses to the property tax inquiries are a small window into the faults and potential of the technology. I’d bet Mike Royko would have demanded answers to tough questions like: How many appeals involved homes worth more than $500,000? How many less than $500,000? Which group was more successful? What lawyers did the successful candidate hire? Unlike Chicago, government officials involved in the computation of tax bills in Wake County are non-partisan. Nonetheless, Mike would have no doubt asked Gratteau to see if the assessor or the tax man had any ties to the successful lawyers like they do in Chicago.
I'd bet that Open AI and its powerful technology could obtain the level of detail Chat the Leg sought if the company wanted to. The reach of the large language model that powers ChatGPT is awesome. But some details of Wake County property tax details are not public record, a huge deficit in government transparency. When you are less transparent than Cook County, Illinois, you have a problem. A cautious OpenAI may be sitting on information I can get on the Internet because it fears crossing a line that might generate criticism. Or it may not have the information because getting it would not be cost effective. The system is only as good as the data in its vast libraries and assembling it is costly.
Could ChatGTP be more interested in helping the public plan cruises than answering questions of legitimate public interest? Or maybe I’m just not as good at asking questions as Royko, a distinct possibility. The quality of the responses from Chat GTP depends on the skills of the person generating prompts for information.
Even a leg man as adept as Gratteau couldn’t get answers to all the questions Chat the Leg posed from a single document without more legwork involving more than one set of questions. But the prospective story that Chat the Leg worked on is not an impossible job for a skilled reporter, particularly one as gifted as Gratteau.
The exercise also raises some interesting questions in my mind about public information policies going forward in the new world inhabited by powerful sources such as ChatGPT. Public officials are quite adept at frustrating Freedom of Information requests by reporters and members of the public. Can they be the roadblock they’ve often been now that ChatGPT is on the scene?
Even though Chat the Leg came up short on property tax records, the technology has enormous potential as a tool that could help revive journalism, particularly local journalism which is truly in a crisis. Creating tools that would help journalists get answers to basic questions involving public policy seems to me to be much more valuable than crafting prose that sounds likes Shakespeare. If the technological powerhouses behind OpenAI want to infuse public good into their technology, they should consider an idea I first heard from AI expert Richard Boyd, the CEO of Ultisim in Chapel Hill, N.C.
He proposes creating a data warehouse of public information that reporters, aided by technological tools like ChatGPT, could tap for stories. “Can you image what reporters could do with that,” Boyd asked me? But that’s another story.
As for now, journalists who believe in reporting rather than repeating don’t have as much to fear from ChatGPT than they do from the hedge funds and newspaper chains that are closing local newspapers at a rate of two a week. As Gratteau said after hearing about my disappointing experience with Chat the Leg: “That wouldn’t cut it with Mike.”
—James O’Shea
James O’Shea is a longtime Chicago author and journalist who now lives in North Carolina. He is the author of several books and is the former editor of the Los Angeles Times and managing editor of the Chicago Tribune. Follow Jim’s Five W’s Substack here.