Late last year, Toronto Star senior politics reporter David Rider questioned how Ontario mayors were using newly expanded powers to build homes. Answering it meant examining thousands of municipal decisions scattered across hundreds of websites, each with its own systems and standards.
At the Toronto Star, a Canadian daily newspaper serving the Greater Toronto Area, the prospect of reviewing that material by human labour alone put the investigation beyond practical reach. Instead, a small team of journalists used AI-assisted analysis to examine an unstructured set of 4,242 decisions from 216 municipalities. The resulting queryable dataset helped them check findings, identify patterns and produce an investigation, a searchable database and follow-up stories.
Ontario’s provincial government introduced “strong mayor” powers in 2022, which gave designated mayors greater authority over municipal budgets, senior staffing and certain council decisions. The province presented the changes as a way to accelerate housing construction.
Building the workflow
The Star’s team included Rider, along with Kate Allen, AI reporting lead, and Nathan Pilla, AI tools and data lead. They wrote code to scrape the decisions posted online by the 216 municipalities granted the powers and manually downloaded others that couldn’t be scraped. They also wrote an extensive set of instructions that defined categories and questions their custom-built AI analysis tool had to answer for each decision. These included whether the mayor had actually used the powers, which part of the legislation was invoked and what the decision was about.
“That editorial judgment was encoded in the instructions, which were paired with a commercial large language model engine,” says Grant Ellis, executive editor. “Human labelling entered the process later, as an independent check on the model’s output rather than as an input to it.”
Building verification into the workflow
Beyond classification, the tool generated a summary of each decision and produced its reasoning for use at the verification stage, Ellis explains.
The AI flagged 75 decisions as directly housing-related, some 2% of all decisions, but the Star did not trust the results immediately.
A panel of journalists independently reviewed 121 decisions, about 3% of the total. The sample was randomly selected, with deliberate over-weighting of substantive uses of strong mayor powers and housing-related outcomes, where an error would have been most consequential, Ellis explains.
Three journalists classified each decision independently, answering four questions per document, for 484 answers in total; disagreements were discussed until they reached a single agreed answer, Ellis says. The final tally was 81 decisions out of 4,242, still about 2%.
“One of the key discoveries through this project is that a well-managed use of an AI was actually significantly more accurate at classifying large sets of totally different documents,” he says. “Perhaps we could have done this project more manually but there would have also been a significant risk of shifting human judgement, both between journalists and from single journalists through times or states of mind.”
From 4,242 documents to a focused story
The AI-assisted workflow gave the Star journalists a classified, cleaned and queryable dataset to identify patterns across municipalities.
“This was a giant filtering process,” Ellis says. “By amplifying human judgement with AI, we were able to set a standard, validate that standard and then scale it in a big way.”
The breadth of the dataset opened possibilities beyond the published story. The team could have examined other uses of strong mayor powers, including personnel changes. “The possibilities to find things in that very defined haystack become completely different than it would have been before,” Ellis says. But the team stuck to their housing angle because they knew it resonated with their readers.
“It’d be very easy in a project like this to merely go, ‘Look how impressive this is and how many different things you can learn from this,’” he says. “At some point, that’s not a story anymore and nobody’s going to pay attention to it. So, what is the thing, what is the core?”
Transparency to their audience
In an abundance of caution, The Star held the series days before publication. “We held a few meetings just to discuss what we could have missed, how best to ensure complete transparency, etc.,” Ellis says. “Closer to the planned publication date we felt we needed an extra level of shoe leather reporting to call many municipalities and double check whether any rulings might have slipped through. It didn’t meaningfully change the result, but it was still worth a bit of extra sweat.”
The Star also published a detailed methodology article for readers. “We knew that transparency was going to be critical,” Ellis says. “I don’t think anyone in Canada had used AI in journalism quite this way before, but we also knew throughout that what we were doing was demonstrably valuable and very explainable.”
In the end, they opted for maximal disclosure. Ellis says the team was proud to stand behind the journalism and happy to have readers scrutinize the work. The feedback was very positive.
A reusable tool for future investigations
The Star’s strong mayors project required a major investment in time, particularly in the beginning stages of designing and refining the system. “Although this project took so long that the time savings over human scrutiny alone could be debated, much of our time was spent devising and refining the verification system that will make future projects much quicker,” he says.
The Star is now working toward a reusable document-analysis tool Ellis describes as a filtering system that doesn’t have file size limits, applies editorial thinking established in advance and gives journalists a way to query documents, FOI releases and other large datasets. He expects it will be useful for future FOI responses which can often be in unpredictable formats and volumes.
He says newsrooms need simple, secure tools they can understand. That clarity allows journalists to trace each stage of the analysis and maintain control when working with large collections of documents.
“I do think this will change a lot of the way that we do things,” he says. “I think this will be a very popular tool… we do a lot of FOIs and when we get them back, you know, they could be in any shape or format. So having a kind of a standardized way to query large data sets, query FOI dumps, anything that comes our way, will be very handy.”
Taking bigger swings
Ellis sees the project as evidence that AI can help newsrooms expand their journalism or pursue investigations that they once would have considered impractical. He frames it as taking bigger swings and getting better skills.
“Journalism itself cannot be a wholly automated process because, in my belief, it’s a human endeavor no matter how that human is augmented,” Ellis says.
One of the mistakes digital media companies make is putting a lot of investment into big projects and then treating the work as disposable, or a one-off. Ellis argues newsrooms ought to treat major projects as sources of learning.
“When we really feel like something was a big learning experience, we have to make sure that we really drill into what we did right, what we did wrong, what we would have done differently, what we’ll do differently next time and make sure we scale that learning.”
For the Star, the project has yielded practical lessons. AI can help a newsroom search through much larger corpuses of data, but journalists still decide what matters and what becomes a story. “AI’s effect on journalism isn’t all roses, but it should be emboldening us to dream at least a little bit bigger about the stories we can take on,” Ellis says.
