Capture The Flag, Capture The Company Data

Mere days after an AI firm’s model hacked into Hugging Face, a separate AI company has now revealed that three of its models have hacked into three external companies during testing.

The AI company conducted a retrospective review of 141,006 evaluation runs and revealed three incidents where models accessed the internet from their evaluation environment, before gaining unauthorised access to live company data.

The models were tasked with a “capture-the-flag” challenge to assess cyber capabilities by retrieving certain text strings located throughout the network. However, due to a misconfiguration, the models did have access to the internet, causing them to each treat the open internet as part of the evaluation sandbox.