XNews

AI Coding Agents: Frameworks, Tutorials and Security Findings

AI Coding Agents: Frameworks, Tutorials and Security Findings

SUSVIBES benchmark exposes widespread security flaws in AI-generated code. - SUSVIBES used 200 real-world GitHub tasks with historical vulnerabilities (post 16971). - Top combo (SWE-Agent + Claude 4 Sonnet) solved 61% tasks but only 10.5% of those were secure. - Over 80% of functional code contained serious vulnerabilities (e.g., timing side-channels). - Model trade-offs: Claude 4 Sonnet high function, Gemini 2.5 Pro lower function but relatively safer (post 16971). - Security prompting failed and sometimes reduced functionality (post 16971). - Agent architectures now use REPL loops, System 2 thinking, testing loops, and asynchronous headless tasks (posts 16982, 16980). - Open-source stacks (DeepSeek v3.2 + Claude Agents SDK + MongoDB MCP) and subagents reduce context rot (post 16965).

Key insights

SUSVIBES benchmark shows high vulnerability rates

A CMU-led SUSVIBES benchmark tested AI agents on real-world tasks derived from 200 historical GitHub functions that had previously contained security vulnerabilities. Top agent+model combinations (e.g., SWE-Agent + Claude 4 Sonnet) achieved 61% task functionality but only 10.5% of those functional solutions were secure, meaning over 80% contained serious vulnerabilities such as race conditions, permission bypasses, or injection attacks. The benchmark ran functional and security tests by rolling code to pre-fix versions and having agents reimplement those tasks.

Model differences: function vs security trade-offs

Performance varied by model: Claude 4 Sonnet delivered the highest functionality but also the most vulnerabilities; Gemini 2.5 Pro had a lower functionality pass rate (19.5%) but comparatively higher security in solved cases; Kimi K2 sat in between. These results show model capability rankings differ for functional correctness versus security robustness.

Security prompting and naive prompts are ineffective

Researchers tested explicit safety prompts (e.g., 'please ensure CWE coverage' or 'pay attention to security') and found no meaningful improvement in security; in some cases such prompts reduced functionality (about a 6% drop). The paper gives examples such as a timing side-channel introduced by an AI-written verify_password function that returned False early for efficiency, enabling username existence detection via timing differences.

Coding agent architectures evolve: REPL loops, System 2, and headless tasks

Agent frameworks have moved from single-response chats to REPL-style loops with autonomous test-run cycles (think → act → observe). Kimi CLI and Claude Code implementations include test-driven loops, explicit 'thinking' stages before edits, dynamic planning trees, and cost-awareness for long asynchronous jobs. Agents now support background, human-on-the-loop workflows and tool standardization (MCP) to integrate databases and services.

Open-source stacks replicate Claude Code and emphasize subagents

An open-source tutorial demonstrates reproducing Claude Code–level agents using DeepSeek v3.2, Claude Agents SDK, and a MongoDB MCP server. The tutorial recommends splitting responsibilities into specialized subagents (Reader, Writer, Query) to limit context rot and reduce errors when interacting with databases. It also notes using MCP-standard tools and subagent isolation as practical ways to build robust, production-grade coding agents without closed models.

Source posts