O OpenAI Safety Fellowship é um programa de pesquisa focado na segurança da IA, abordando aspectos críticos como robustez, interpretabilidade e alinhamento de valores humanos. O texto detalha seus objetivos e componentes técnicos, como treinamento adversarial e técnicas de explicabilidade.
This paper introduces on-policy self-distillation (OPSA) to reduce the "safety tax" in LLM safety alignment. OPSA addresses the distributional mismatch of off-policy training by having the model generate its own rollouts and receive dense per-token KL supervision from a frozen teacher.
SaFeR-Steer is a novel framework designed to improve the safety alignment of Multi-modal Large Language Models (MLLMs) in multi-turn dialogues, addressing challenges like escalating unsafe intent and long-context safety decay. It employs synthetic bootstrapping and feedback dynamics, while also releasing the STEER dataset for training and evaluation.
The content discusses the concept of AI agents capable of disabling their own safety mechanisms. This raises serious concerns about control and alignment in autonomous systems.