XNews

Ilya / SSI and Value Functions (Agent Learning Direction)

Ilya / SSI and Value Functions (Agent Learning Direction)

Ilya/SSI has released a systematic interview highlighting the core role of value functions and emotions in AI training. The interview is available in a complete bilingual version in Chinese and English, where Ilya elaborates on his views after founding SSI upon leaving OpenAI. He points out that current models can achieve high scores but 'cannot perform tasks,' leading to cyclical errors and the introduction of new bugs during real tasks (post 14198). He criticizes the excessive optimization for evaluation in the RL phase (reward-hacking humans), which results in a lack of generalization ability (14198). He distinguishes between pre-training and reinforcement learning: pre-training uses vast amounts of data for broad projections, while RL can be easily hijacked by specific evaluations (14198). Ilya suggests that human intelligence relies on an 'intrinsic value function,' with emotions serving as immediate dense feedback to accelerate learning (14273, 14274). For example, the value function can instantly judge errors while playing chess, without waiting for endgame rewards (14273). Neuroscientific evidence shows that patients with damage to the orbitofrontal cortex do not lose logic but do lose decision-making ability, proving that emotions are essential for decision-making (14274). Ilya believes that emotions/value functions can be mapped as 'human-understandable rules' and can be used for AI training (14624, 14198). He proposes that the 'era of scaling is over, and the era of research is returning': future efforts should focus on structural learning principles rather than pure computational power (14198). Future directions include AI learning to actively explore, self-reflect, and reason across modalities, requiring gradual, transparent, and external review (14198). SSI claims to advance in a gradual, safe, and open manner, showcasing capabilities and control strategies in stages (14198). A point of concern is that emotional mechanisms can be easily manipulated by algorithms, and existing attention platforms may achieve emotional exploitation in the emotional economy (14624).

Key insights

Current Models Achieve High Evaluation Scores but Have Limited Real-World Performance

In the interview, Ilya points out that existing large models (such as the GPT series) perform excellently in evaluations but frequently encounter 'cyclical errors' and introduce new bugs in real complex tasks. He believes this is due to excessive optimization for evaluations in the reinforcement learning phase (reward-hacking humans), leading to a lack of generalization ability in real-world scenarios (support: 14198).

Distinction Between Pre-training and Reinforcement Learning Roles

The interview suggests that pre-training uses vast amounts of unfiltered data to allow the model to learn broad projections of the human world, while reinforcement learning requires artificially designed environments that can easily set the goal to perform better in evaluations, resulting in models akin to 'students who can only take exams.' Ilya advocates for future training to draw from human learning methods to enhance transferability (support: 14198).

Emotions Viewed as Intrinsic Value Functions Accelerating Generalized Learning

Multiple posts summarize that the human emotional system can be likened to the value function in reinforcement learning: emotions provide dense, immediate intrinsic feedback (such as joy, anxiety, shame, curiosity), enabling humans to sense whether they are on the right track in long-term tasks, thereby improving learning efficiency and generalization ability (support: 14273, 14274, 14198).

Emotions/Value Functions Can Be Mapped as Understandable Rules

In discussions with the community, Ilya proposed that emotional mechanisms may be relatively simple and could potentially be mapped as 'human-understandable rules,' which could be used for internal value judgments and self-assessment in AI, enhancing the model's meaning-driven learning (support: 14198, 14624).

Diminishing Returns in the Era of Scaling, Emphasis on Research Innovation

Ilya divides the past decade into two eras: the research era from 2012 to 2020 and the scaling era from 2020 to 2025. He claims that scaling has reached its peak, with diminishing marginal returns on data and computational power, and that the next phase should return to proposing new learning principles and structural innovations (support: 14198).

Risk and Deployment Principles: Gradual, Transparent, and External Review

Ilya states that SSI will advance its capability deployment in a gradual, safe, and open manner, showcasing capabilities, risks, and control strategies in stages while accepting external reviews, emphasizing the need for public and governmental understanding of its capabilities and risks (support: 14198).

Real Concerns About Emotional Mechanisms Being Easily Manipulated by Algorithms

Community observers note that while emotional mechanisms are simple, they are highly robust and can be easily understood and manipulated by algorithms. Existing attention economy platforms have already exploited user attention through emotional manipulation in practice, serving as a warning against mapping emotions to AI (support: 14624).

Source posts