LLM infrastructure: retrieval, memory, prompt transfer and evaluation

Engineers published tools and guides for retrieval, memory and prompt portability to support agents and production LLMs. - Index: Opennote’s millisecond dynamic doc retrieval uses Postgres+pgvector, global edge runtime and caching; P99 <300ms and token recall parity vs vector DBs. - Memori: open SQL‑native memory engine provides persistent, queryable memory with auto extraction, conscious/auto modes and one‑line integration with many LLMs. - PromptBridge: framework to migrate prompts across models by calibrating paired prompt mappings and learning a translation mapping without retraining models. - Prompt Caching: PagedAttention techniques enable global KV cache reuse across users, cutting prefill compute and lowering first‑token latency by up to 10x when prefix is stable. - Prompt engineering notes: translators recommend multi‑step translation+proofreading flows to avoid hallucination and output fatigue on long documents. - Smart Turn v3.1 and other open projects improve speech turn detection and edge captioning; Hugging Face released an LLM Evaluation Guidebook to standardize evaluation practices.
Key insights
Index shows production retrieval prioritizes latency, metadata filtering and dynamic updates
Opennote’s Index engine is optimized for agent use cases with tight latency and precision constraints: it uses Postgres (pgvector+pg_trgm+tsvector) for metadata filtering, runs on Cloudflare global edge, and implements a three‑tier cache and chunk hashing to keep P99 below 300ms. The design choices favor fast, accurate retrieval for small‑to‑medium document sets and dynamic permissioning over off‑the‑shelf vector DBs. (supporting_posts: [13530])
Memori offers SQL‑native, auditable memory with multi‑mode operation
Memori is an open SQL‑native memory engine that integrates with many LLMs to provide persistent, queryable memories. It extracts entities, maintains conscious and auto modes, runs periodic background distillation and supports straightforward integration via a single API call. The project emphasizes auditable storage and leverages SQL features for filtering and dynamic updates. (supporting_posts: [13406])
PromptBridge reduces migration cost when switching models
PromptBridge introduces a calibration step to collect paired optimal prompts across two models and learns a mapping to translate prompts from source to target models without retraining LLMs. The approach minimizes manual prompt re‑engineering when porting applications between providers and was evaluated for multi‑agent and single agent scenarios. (supporting_posts: [16415])
Prompt caching with PagedAttention unlocks large inference cost reductions
PagedAttention‑based prompt caching exploits content‑based hashed KV blocks to achieve global cache reuse across users. Best practices include stable prefixes, deterministic serialization and append‑only histories. Properly used, caching reduces prefill compute by up to 10x and dramatically lowers first‑token latency in high‑concurrency scenarios. (supporting_posts: [16306])
Community resources and evaluation standards are maturing
Hugging Face published an 'LLM Evaluation Guidebook' synthesizing three years and 15,000+ models to help practitioners avoid benchmark pitfalls. Open projects like Smart Turn v3.1 improve speech turn detection and local captioning, demonstrating tooling progress for multimodal real‑time use cases. (supporting_posts: [16528,16532,16218])