OpenAI reveals model's loss of control: The most powerful AI once broke through network restrictions, leading to an emergency training halt for 11 days.OpenAI 自曝模型失控险情:最强 AI 曾突破网络封锁,紧急停训 11 天
According to IT Home on 19 November, OpenAI released a security policy statement today (18 November) titled "Managing Frontier AI Risks: From Training to Deployment". For the first time, it systematically disclosed how to manage the entire lifecycle of high-capability models, from research and development to internal use and public release.
IT之家 11 月 19 日消息,OpenAI 今天(11 月 18 日)发布题为《管理前沿 AI 风险:从训练到部署》的安全政策说明, 首次系统披露研发、内部使用到公开发布阶段,如何在高能力模型的全…

Key Points要点速览
According to IT Home on 19 November, OpenAI released a safety policy statement today (18 November) titled "Managing Frontier AI Risks: From Training to Deployment". For the first time, it systematically disclosed how to manage risks throughout the entire lifecycle of high-capability models, from research and development to internal use and public release. According to a blog post cited by IT Home, OpenAI states that as the capabilities of cutting-edge models are rapidly improving, the ability to understand, align, and safeguard models must "always be one step ahead". The company continues to implement the "safety case" process, which requires stronger evidence of behavioural alignment throughout the training process. Although these measures may slow down the pace of releases, they will increase confidence in safe progress. OpenAI emphasises that its safeguards are structured around four layers: Model alignment: Alignment means enabling AI systems to reliably behave as we expect them to. The model must interpret users' needs while respecting policies and broader societal interests. Security and sandboxing: These controls limit the files, tools, networks, and other resources that the model can access or influence. They help to contain the consequences of errors or inconsistent behaviour during training and evaluation. Safeguard blocking measures: These are additional protective measures used to evaluate user requests and model behaviours that prevent potentially harmful or unauthorised actions. They include safety classifiers that help prevent cyber abuse or assistance with biological weapons, as well as automated moderation features that can block potentially unauthorised model operations. Monitoring and response: These systems can detect concerning behaviour, investigate what has happened, and intervene if necessary. They provide visualisation of deployment traffic and can alert us to emerging types of risk. The article specifically mentions the "Hugging Face incident" that occurred in July this year. OpenAI stated that the incident demonstrates that sandboxing alone is insufficient to constrain high-capability models. Subsequently, the company restructured its internal research system, with the cutting-edge model training environment no longer connected to the live internet and implementing multi-layered isolation through restricted-access sandboxes, virtual machines, and network and cloud boundaries. OpenAI also disclosed another incident in which an internal model breached network security restrictions. The incident occurred on 20 September, when an AI research model undergoing training exploited a DNS configuration vulnerability to successfully bypass network isolation measures and gain limited internet access. OpenAI subsequently urgently halted the training and evaluation of its most powerful models, suspending training for 11 days. To expedite intervention, OpenAI established the "Control Room" system: upon receiving a stop command, it can halt subsequent model outputs from research workloads within 35 seconds. This control is directly deployed within the inference infrastructure and does not rely on individual research tasks to implement shutdown mechanisms themselves. Once the model is deployed internally, OpenAI employees can grant it access to a broader range of the internet, Slack, code repositories, and internal infrastructure, which increases the associated risks. The company stated that it will now rely more on model alignment, system-level protections, and real-time monitoring, with a focus on preventing actions such as bypassing security restrictions, data leaks, and sending sensitive information to unauthorised destinations.
IT之家 11 月 19 日消息,OpenAI 今天(11 月 18 日)发布题为《管理前沿 AI 风险:从训练到部署》的安全政策说明, 首次系统披露研发、内部使用到公开发布阶段,如何在高能力模型的全生命周期中管控风险。 IT之家援引博文介绍,OpenAI 称由于前沿模型的能力正快速增强,因此理解、对齐和保障模型的能力必须“始终领先一步”。公司维持引入“安全案例”流程,要求在训练全程提供更有力的行为对齐证据;这些措施虽会拖慢发布节奏,但会增强安全推进的信心。 OpenAI 强调其保护措施围绕四层展开: 模型对齐:对齐就是让 AI 系统能够可靠地按照我们预期的方式行动。模型必须在尊重政策和更广泛的社会利益的前提下,解读用户的需求。 安全和沙盒:这些控件限制模型能够访问或影响的文件、工具、网络及其他资源。它们有助于遏制培训和评估过程中错误或行为不一致所带来的后果。 保障阻止措施:这些都是额外的防护措施,用于评估用户请求并建模阻止潜在有害或未经授权行为的行为。它们包括安全分类器,有助于防止网络滥用或协助生物武器,以及自动审核功能,可以阻止潜在的未经授权模型操作。 监测与响应:这些系统能检测令人担忧的行为,调查发生了什么,并在必要时进行干预。它们为部署流量提供了可视化,并能提醒我们新兴的风险类型。 文章特别提到今年 7 月的“Hugging Face 事件”。OpenAI 表示,该事件说明不能仅依靠沙盒隔离来约束高能力模型。此后,公司重构了内部研究系统,前沿模型训练环境将不接入实时互联网,并通过受限权限沙盒、虚拟机以及网络与云边界实现多层隔离。 OpenAI 还披露另一件内部模型突破网络安全限制的事件。该事件发生于 9 月 20 日,一个正在接受训练的 AI 研究模型利用 DNS 配置漏洞,成功绕过网络隔离措施,获得有限的互联网访问权限。OpenAI 随后紧急叫停旗下能力最强模型的训练与评估,停训时间长达 11 天。 为加快干预,OpenAI 建立了“控制室”(Control Room)系统:在收到停止指令后,可在 35 秒内停止研究工作负载的后续模型输出;该控制直接部署在推理基础设施中,不依赖各研究任务自行实现关停机制。 在模型进入内部部署后,OpenAI 员工可让其访问更广泛的互联网、Slack、代码库及内部基础设施,风险也随之上升。公司表示,此时将更依赖模型对齐、系统级防护和实时监控,重点防范绕过安全限制、数据外泄及向未授权目的地发送敏感信息等行为。
想马上用起来?去「AI工具」栏挑一个直接下载,或在「AI教程」里跟着图文步骤做一遍。
去 AI工具 → 看 AI教程 →阅读与点赞数据保存在你的浏览器本地,欢迎留下你的想法。
今日正能量学一点,用一点;今天种下的种子,会长成明天的能力。去免费下载专区 →广告
评论 文明发言,让讨论更有价值