XNews

2025 LLM Training Paradigm & Karpathy Yearly Review

2025 LLM Training Paradigm & Karpathy Yearly Review

Karpathy published a 2025 LLM year-in-review identifying six paradigm shifts; other experts reflected on training, capabilities and developer workflows. • 2025 review introduced RLVR (Reinforcement Learning from Verifiable Rewards). • RLVR trains models in environments with objective answers (math, code). • Karpathy: LLM intelligence is "jagged"—high in verifiable domains, brittle elsewhere. • New LLM app layer (Cursor-style) organizes context, orchestrates models, and exposes autonomy sliders. • Claude Code demonstrated local agents running on users' machines. • "Vibe coding": natural-language-driven rapid programming became mainstream. • Nano Banana signals an LLM GUI era with integrated image/animation outputs. • Armin (Sentry founder) reported shifting from hand-coding to AI-driven toolchains (Claude Code, Amp, Pi). • Salvatore Sanfilippo noted: LLMs show internal representations, chain-of-thought value, and RL-based self-improvement. • Jack Rae / compression view: training is analogous to lossless compression; example numbers cited for Llama 65B compressing 5.6TB to ~400GB (14x). • Practical engineering issues listed: new version control for prompts, PR/CR gaps with AI, observability, and open-source vetting of AI-generated code.

Key insights

Karpathy: RLVR and six paradigm shifts define 2025

Karpathy's 2025 LLM review identifies six major shifts: (1) RLVR (Reinforcement Learning from Verifiable Rewards) becomes a standard training stage for tasks with objective answers; (2) LLM intelligence is uneven ('jagged intelligence') with rapid gains in verifiable domains; (3) an LLM application layer (Cursor-style) emerges to organize context and orchestrate models; (4) local agents like Claude Code run on users' machines and integrate private data; (5) 'vibe coding' enables natural-language-first programming; (6) early signs of LLM graphical UIs (Nano Banana) indicate a move beyond text-only interfaces. Karpathy emphasized compute reallocation toward RL phases and runtime 'thinking knobs'.

Salvatore Sanfilippo: LLMs are stronger, chain-of-thought matters, RL opens new paths

Salvatore (Redis founder) reported that the 'random parrot' critique is waning given LLM success on professional exams and internal concept representations. He highlighted chain-of-thought (CoT) as an underestimated breakthrough that samples internal representations and combines with RL to find solution paths. He argued that verifiable rewards let models self-improve (e.g., program execution, theorem proving) and predicted this will drive future ability gains. He noted programmer attitudes shifted: many now treat LLMs as either conversational colleagues or autonomous coding agents.

Practitioner report: AI-enabled career shifts and toolchains

An engineering practitioner account described 2025 as a year of professional transition: leaving established companies to found AI-driven startups, reframing roles from coder to 'engineering manager of virtual interns', and adopting new toolchains (moving from Cursor to Claude Code, plus Amp and Pi). The author reported building an AI-driven SDK generator rather than outsourcing, noting historical software-engineering heuristics degrade in the new agent era and that developers increasingly choose tools by 'vibe'.

Training-as-compression perspective: models as lossless compressors

A theoretical framing equates LLM training to lossless compression: a 65B Llama trained on 1.4T tokens can compress 5.6TB of text to ~400GB (14×), outperforming traditional compressors. Under this view, better compression implies discovering more concise, generalizing representations; scaling improves compression and thus generalization. The framing claims compression is robust to test-set leakage and suggests directions (architectures, multimodal fusion, tool use) that reduce 'post-compression' data size.

Nano Banana and image-text fusion improve on-image text and GUI signaling

Reports attribute Nano Banana and multimodal 'token-in token-out' models with strong gains in on-image text rendering and instruction following. The technical note contrasts diffusion pixel-space denoising models (which treat text as pixels) with native multimodal transformers that generate cross-modal tokens, leading to lower character-error rates in complex Chinese text in images and coherent multimodal outputs. This was framed as evidence of an emergent LLM 'graphical interface' trend.

Engineering gaps: versioning, code review, observability need redesign

Practitioners highlighted four infrastructure gaps caused by agent-driven development: Git cannot record agent internal deliberations or prompts; pull-request/code-review processes are human-centric and fail to reflect human–AI local collaboration; observability tools must surface AI-driven analysis (e.g., deep SQL/eBPF insights); and open-source ecosystems contain unvetted AI-generated code, requiring new social contracts and contribution norms.

Source posts