Regulation and reporting needed, or we’re at risk of letting critical incidents go unnoticed, warns report
Posted on 3rd July 2024
The UK must establish a system to record the misuse and malfunctions of artificial intelligence, or else ministers’ risk being unaware of critical incidents involving the technology, a report warns.
The Centre for Long-Term Resilience (CLTR), a think tank dedicated to addressing unforeseen crises and extreme risks, advises that the next government should develop a mechanism for logging AI-related incidents in public services, along with the creation of a central hub to compile any episodes across the country.
Drawing parallels to the Air Accidents Investigation Branch (AAIB) in aviation, the CLTR emphasises that a robust incident reporting regime is crucial for the safe integration of AI technologies. The report highlights the necessity of this system by referencing a database from the Organisation for Economic Co-operation and Development (OECD), which has catalogued 10,000 AI ‘safety incidents’ since 2014. These incidents range from physical harm to economic, reputational, and psychological damages.
Examples from the OECD’s database include a deepfake video of Labour leader Keir Starmer appearing abusive towards staff, a misrepresentation by Google’s Gemini model depicting German WWII soldiers as people of colour, issues with self-driving cars, and a chatbot encouraging a man to assassinate the late Queen.
Tommy Shaffer Shane, a policy manager at CLTR and the report’s author, states, ‘Incident reporting has been transformative in managing risks in sectors like aviation and medicine, but it’s largely missing in AI regulation.
‘This absence leaves the UK government blind to emerging AI issues and hampers its response capabilities’.
Given the lack of a dedicated regulator for advanced AI technologies, including chatbots and image generators, many AI incidents may fall outside the purview of existing UK watchdogs. Labour has already committed to implementing binding regulations for leading AI companies.
The FCA has already noted that there is ‘no rush’ to introduce detailed regulations to police the use of artificial intelligence (AI) in the financial services sector, with FCA CEO Nikhil Rathi noting that:
‘We are not at the moment inclined to just jump in and write lots of detailed rules. When you do that, almost by the time you’ve written the rules, they’re out of date’.
‘We are going to have to think about this very differently to other forms of technological development, or financial services evolution’.
Conversely to the FCA, the CLTR has stressed that more stringent regulations and reporting systems are desperately needed, and that a functional incident reporting system would allow for the rapid identification of AI malfunctions and facilitate not only anticipation of similar future incidents, but also swift responses to severe incidents and identify early signs of large-scale harm.
The report also points out that the Department for Science, Innovation and Technology (DSIT) could miss out on real-time insights into AI system abuses, such as disinformation campaigns, the development of bioweapons, biases in AI, or public service misuse. For instance, the Netherlands’ deployment of an AI system to combat benefits fraud caused financial distress for thousands of families.
According to the report, ‘DSIT should prioritise finding out about novel harms not through the news, but through a proven process of incident reporting’.
To address these challenges, CLTR, funded primarily by Estonian tech entrepreneur Jaan Tallinn, recommends three immediate actions:
- Establishing a government system for reporting AI incidents in public services.
- Tasking UK regulators with identifying gaps in AI incident reporting.
- Considering a pilot AI incident database to gather information from existing bodies such as the AAIB, the Information Commissioner’s Office, and the Medicines and Healthcare products Regulatory Agency (MHRA).
CLTR suggests that the new reporting system could build on the current algorithmic transparency standard, which encourages departments and police authorities to disclose their use of AI.
In May, the UK, alongside nine other countries and the EU, signed a statement on AI safety cooperation, including the monitoring of ‘AI harms and safety incidents.’
An effective incident reporting system would also support DSIT’s Central AI Risk Function (CAIRF), which is tasked with evaluating and reporting on risks associated with AI.