AI Insights · Oct 22, 2026 · Partially verified
Answering questions from papers is nearly solved. Citing them is not.
Simple agents over a good paper-search service already score about 85% on literature benchmarks. The open problems are citing sources accurately and combining them well.
Claim
On literature-understanding benchmarks, commercial and open agents built as a thin loop over a good paper search API already score about 85%. Search-style benchmarks like BrowseComp are near saturation. What remains hard is synthesis against expert rubrics (best agents under 50%) and citation faithfulness, where frontier chat models hallucinate most citations without retrieval.
Evidence
- AstaBench (Ai2, 2025-08-26, updated 2025-10-27): 2,400+ problems, 57 agents. Asta Scholar QA, Elicit and SciSpace Deep Review all score about 85%+ on ScholarQA-CS2; Asta Paper Finder doubles ReAct on PaperFindingBench. Best overall agent 53.0%. Data analysis is hardest (under 34%).
- OpenScholar (Ai2 and UW, Nature 2026-02-04): RAG over 45M open-access papers with an 8B model beats GPT-4o by about 5% and PaperQA2 by about 7% on ScholarQABench correctness and matches expert citation accuracy. GPT-4o hallucinated citations 78% to 90% of the time.
- DeepResearch Bench II (v3 2026-09-10): 132 tasks, 9,430 binary rubrics; strongest models satisfy under 50%.
- BrowseComp: launch Deep Research 51.5%; 2026 aggregator snapshots put top single models at 90% to 92%, with a 10-agent Opus 5 configuration at 93.6%. Aggregator figures. UNVERIFIED against the system card.
- LitSearch (EMNLP 2024, 597 queries): dense retrievers beat BM25 by 24.8 recall@5 points and LLM reranking adds 4.4. No 2025 to 2026 agent leaderboard found.
- Undermind is the purest iterative citation-chasing product and claims 85% recall at top-20 versus 50% or less for frontier agents with web search. Vendor benchmark. UNVERIFIED. Elicit (structured screening into columns), Consensus (evidence meter over 220M papers) and Scite (1.6B classified citations) occupy other niches.
Open questions
- Whether any public leaderboard scores agents on a private paper corpus rather than the open web.
Sources
- https://allenai.org/blog/astabench — blog, Sun Oct 26
- https://allenai.org/blog/openscholar — blog, Tue Feb 03
- https://arxiv.org/abs/2601.08536 — paper, Wed Sep 09
- https://arxiv.org/abs/2407.18940 — paper, Thu Jul 25
- https://www.undermind.ai/ — docs, Wed Oct 21
AI-assisted research: gathered and summarised with AI from the sources above, each claim checked against them.