World
Academic libraries are fighting off swarms of data-scraping bots — that are slowing or crashing sites meant for students, researchers and the public — as AI companies continue hunting for content to feed their large language models.
One school says it was overwhelmed by more than 11,000 searches on Finnish music in just one day
Kevin Maimann · CBC News
· Posted: Nov 05, 2026 4:00 AM EST | Last Updated: November 5
Listen to this article
Estimated 6 minutes
The audio version of this article is generated by AI-based technology. Mispronunciations can occur. We are working with our partners to continually review and improve the results.
Students work on laptops in the Pedagogical Library at the Free University of Berlin on Sept. 20, 2011. The online resources of academic libraries around the world are increasingly being disrupted by AI bots hunting for content. (Sean Gallup/Getty Images)
Academic libraries are fighting off swarms of data-scraping bots — that are slowing or crashing sites meant for students, researchers and the public — as AI companies continue hunting for content to feed their large language models.
University libraries say entire collections are being knocked offline by persistent streams of unruly bots trying to access their archives, which include academic, historical and government documents.
"They're just out there on the internet, just bombarding us with trying to grab up as much material as possible," Jill Strykowski, cataloguing and government documents lead at San Jose State University, told CBC News.
The bots are used by both AI and third-party companies to scour websites and retrieve data, often to feed LLMs like Claude, Gemini and ChatGPT.
Libraries, museums and other cultural heritage institutions are prime targets because they have quality, vetted information that is generally not hidden behind paywalls or logins.
The bots are "causing an incredible amount of work and headache for every institution that does digitization work and every institution that is trying to make information openly available," Strykowski says.
Researchers found such reports started appearing in late 2024 across North America, Europe and Oceania and experts tell CBC News the problem continues to get worse.
LISTEN | Even smarter, with even fewer guardrails:
The Sunday Magazine23:41Inside the global race toward an AI that might kill us all
And new challenges keep arising, says Dean Seeman, director of collections and digital infrastructure at University of Victoria Libraries.
"Sometimes the bot traffic gets so bad that it causes the sites to go down completely for a little while," he said.
Seeman says this creates a "real problem" for public access.
Regular people using library websites are now, effectively, in competition with infinite traffic from bot farms, says Peter Lewis, Canada Research Chair in Trustworthy Artificial Intelligence at Ontario Tech University.
"This is another instance of large, private companies making use of common-good public interest institutions in a way that's very extractive and just assumes that that stuff is available and is free — and then using it to be able to produce highly commercial models that then they can go and sell back to people," he said.
Workers enter data from an old document belonging to the McMillan Memorial Library during a digitization exercise at the Eastland Library in Nairobi on March 24, 2022. Libraries and other cultural heritage institutions are prime targets for AI bots. (Patrick Meinhardt/AFP/Getty Images)
Good bots and bad bots
Eric Hellman, executive director of Project Gutenberg — a U.S.-based volunteer organization that digitizes and archives cultural works and distributes eBooks — wrote in a blog last year that these bots, which he characterizes as poorly programmed "monstrosities," are "destroying open access" on the internet.
He tells CBC News that "everything's gotten worse" since he wrote that.
Hellman says the problem comes down to good bots and bad bots.
He explains that search engines like Google, and archiving sites like Internet Archive, use bots that follow robots.txt — voluntary sets of instructions on websites that tell them what they can and can't do, so they don't disrupt the site's functioning while gathering information.
AI scraping bots, on the other hand, sometimes ignore these rules and overwhelm websites — in some cases requesting thousands of documents a second and downloading them in every available format.
WATCH | Young AI researchers in Ontario:
This is what young people studying artificial intelligence are thinking — as the future of AI grabs headlines
September 14|
Duration
2:16
Developments in AI, and companies and developers warning about the future of the technology, are grabbing headlines again. But there are young people in Windsor studying the technology. The CBC's Jacob Barker caught up with them to hear their perspective.
As an example, the University of North Carolina at Chapel Hill reported last year that it received more than 11,000 searches for Finnish music from thousands of different internet addresses on the same day. The university said its library catalogue was periodically shutting out students, faculty and staff due to bot traffic.
And because one company or bot farm can work from thousands of IP addresses, Hellman says the old fix of blocking problematic IPs is no longer useful.
Michael Weinberg, co-director of GLAM-E, a collaborative effort focused on open-access policies for cultural institutions, is working to protect public access to information. (Submitted by Michael Weinberg)
He says Project Gutenberg is happy for its materials to be used as AI training data and even offers a one-click download of its entire library of books, "but the bots are too stupid to use it."
"Our material is almost all public domain material. It's exactly the sort of material that they should be training on," he said. "We just wish they would do it in a respectful way that doesn't take down websites."
Open access a 'huge benefit'
Some libraries have set up CAPTCHA challenges or other verification systems to weed out bots.
The problem is, those measures can also weed out the good bots, says Michael Weinberg, co-director of GLAM-E, a collaborative academic initiative at NYU Law School and the University of Warwick focused on open-access policies for cultural institutions.
Some libraries have also begun to require login credentials, a controversial move that those who spoke to CBC News say they want to avoid because it puts barriers on human access to things they believe should be easily accessible.
And as the scraping bots get more advanced, experts say they could also learn to work around these strategies.
The Anthropic logo is displayed at during the Dreamforce technology summit in San Francisco, on Sept. 17. AI firms have broadly argued that scraping publicly available data is permissible under fair use and in the public's best interest. (Carlos Barria/Reuters)
Major AI firms have argued that scraping publicly available data to train their models is permissible under fair use and in the public's best interest.
But faced with widespread criticism and copyright lawsuits, Anthropic, OpenAI, Google and Microsoft expressed support last week for honouring a single, standardized opt-out signal in a website's robots.txt file that could block their web crawlers from harvesting data.
Compliance would be voluntary, however, and the agreement leaves out problematic bots from third-party companies and smaller AI startups.
Weinberg says the digitization of valuable information has been a "fantastic" development for humanity — giving researchers, teachers and curious citizens easy access to resources that, 20 years ago, "would have required driving for flying or a special permission to get access."
Organizations like GLAM-E are trying to figure out long-term solutions to preserve that access for humans and helpful bots alike.
Right now, the bots create "a disincentive for libraries and other institutions to make parts of their collection available online," Weinberg said. "I think it's been a huge benefit to society that libraries and museums and other cultural institutions have put a huge amount of work into making their collections available."
People look at their laptops and in the main reading room of the New York Public Library on Oct. 5, 2016, in New York City. Increased bot traffic means regular people using library websites are effectively competing with infinite traffic from bot farms, says one expert. (Dominick Reuter/AFP/Getty Images)
ABOUT THE AUTHOR
Kevin Maimann is a senior writer for CBC News based in Edmonton. He has covered a wide range of topics for publications including VICE, the Toronto Star, Xtra Magazine and the Edmonton Journal. You can reach Kevin by email at kevin.maimann@cbc.ca.
Instagram Engineering
Shopify Engineering