U.K. agency releases tools to test AI model safety

The U.K. Safety Institute, the U.K.’s recently established AI safety body, has released a toolset designed to “strengthen AI safety” by making it easier for industry, research organizations and academia to develop AI evaluations. 

Called Inspect, the toolset — which is available under an open source license, specifically an MIT License — aims to assess certain capabilities of AI models, including models’ core knowledge and ability to reason, and generate a score based on the results. 

In a press release announcing the news on Friday, the Safety Institute claimed that Inspect marks “the first time that an AI safety testing platform which has been spearheaded by a state-backed body has been released for wider use.”

“Successful collaboration on AI safety testing means having a shared, accessible approach to evaluations, and we hope Inspect can be a building block,” Safety Institute chair Ian Hogarth said in a statement. “We hope to see the global AI community using Inspect to not only carry out their own model safety tests, but to help adapt and build upon the open source platform so we can produce high-quality evaluations across the board.”

As we’ve written about before, AI benchmarks are hard — not least of which because the most sophisticated AI models today are black boxes whose infrastructure, training data and other key details are details are kept under wraps by the businesses creating them. So how does Inspect tackle the challenge? By being extensible and extendable to new testing techniques, mainly. 

Inspect is made up of three basic components: data sets, solvers and scorers. Data sets provide samples for evaluation tests. Solvers do the work of carrying out the tests. And scorers evaluate the work of solvers and aggregate scores from the tests into metrics.  

Inspect’s built-in components can be augmented via third-party packages written in Python. 

In a post on X, Deborah Raj, a research fellow at Mozilla and noted AI ethicist, called Inspect a “testament to the power of public investment in open source tooling for AI accountability.”