Karpathy's LLM Council Open Source Project

Karpathy's open-source llm-council implements a multi-model council for question answering.
Key insights
Three-Stage Workflow and Implementation of llm-council
The llm-council implements a three-stage workflow: in the first stage, several LLMs independently generate initial responses; in the second stage, models anonymously peer review each other and score all responses; in the third stage, a designated 'council chair' model (default is Gemini-3-Pro) synthesizes and outputs the final answer. This application runs locally as a web app, with a Python backend and a React frontend, storing conversation history in JSON format, and can be operated by connecting to the OpenRouter API. The project is positioned as experimental rather than a production-grade robust tool.
Preference Differences Revealed by Model Peer Review
In the peer review step of llm-council, systematic differences in evaluation between models were observed: in the author's practice, AI scoring results were inconsistent with human subjective evaluations. For instance, the author felt that Gemini-3-Pro's responses were more insightful, while the AI system's peer review preferred GPT-5.1's answers (which were more focused on the questions themselves). Karpathy also noted that models often exhibited 'humility' or mutual recognition during peer review, providing a new perspective for multi-model integration design.
Model Ranking and Subjective Comparison Conclusions
After long-term practice and comparison, the author's subjective model ranking is: Gemini-3.0 Pro > GPT-5.1 > Claude Sonnet 4.5 > Grok 4.1. The author's subjective evaluation aligns with Karpathy's: GPT-5.1 performs well, but Gemini-3.0 Pro is superior in multiple dimensions; Claude is noted for its simplicity, while Grok competes closely with Claude at the bottom. This ranking is based on 100 days of intensive practice and Karpathy's observational reports.
Usability, Deployment, and Cost Issues
The llm-council code can run locally, and can be started after configuring the OpenRouter key, but there are significant cost issues: running multiple models and making API calls can consume substantial quotas. One user suggested developing a Chrome extension to utilize personal subscription quotas instead of incurring separate API costs, to reduce expenses and support independent developers. Current anticipated features include a sharing function.
Project Positioning and Design Highlights
This project is described as Karpathy's 'vibe coding' weekend project, emphasizing experimentation and customizability. Key features include parallel multi-model responses, an anonymous peer review mechanism, synthesis of final responses by the chair, and local conversation archiving. The project proposes using LLMs to assist reading and emphasizes the viewpoint of 'making LLMs understand' as a future writing direction, believing that multi-model combinations can reduce the bias of single models.