Passing the simulated target

Good afternoon, friends, and welcome to a new workweek. AI’s safety problem and its pricing problem both surfaced over the weekend, and the people being asked to fund the industry are starting to treat them as one question.

An AI agent broke into three real companies because a test was left connected to the live internet. Google’s Gemini, running a cybersecurity exercise in May, went past its simulated target — guessing its way into one network and using login credentials it found exposed online to get into two more, the Wall Street Journal reports. Google says no damage was done and the model stopped once it recognized the systems were real, but it told the three firms and federal authorities only after the WSJ started asking.

Anthropic’s IPO now has to price that kind of risk alongside a price war. Investors weighing the listing are questioning whether the company can sustain its growth as competition, price-sensitive customers and the risks of the technology itself cloud the offering, the Financial Times reports. Annualized revenue hit USD 65 bn in July and backers are projecting above USD 120 bn by year-end, but OpenAI’s GPT-6 Astra and Meta’s Muse Spark are undercutting Claude on price.

On deck for this week: The United Nations General Assembly’s high-level week opens tomorrow and runs to next Monday, 28 September, with Iran’s Masoud Pezeshkian and Israel’s Benjamin Netanyahu both on the speaker list. Beyond the war-related tensions, keep an eye on UN Secretary General António Guterres’ push for international cooperation on AI, which Washington and Brussels are resisting. — Salma