llmwiki: A Literature Library for Research Agents

Every experiment I run now starts with the same question to an agent: what do we already know about this? The answer comes from llmwiki, a private git repository of Markdown files that my coding agents keep for me. Every paper, blog post or codebase they have read in depth has a folder there, with a citation and a reading note, filed under a research topic.

This post explains how it works, for classmates who want the same thing. The wiki stays private. The two skills that build and read it are public, and the last section shows how to install them.

Why agents need a library

A coding agent such as Claude Code or Codex can find and read a paper in minutes. What it learns is gone when the session ends, and a long session summarizes its own history to make room, dropping details on the way. Next week another session reads the same paper again, or answers from memory. A model’s memory of papers is good enough to suggest leads and bad enough to invent a citation, a number or a result.

The fix is old: write the reading down, somewhere every later session can open, in a form that says what was checked. Andrej Karpathy described the general pattern as an LLM wiki: the agent, not the person, maintains a Markdown wiki built from raw sources. llmwiki narrows it to research literature and adds rules about evidence.

One folder per reference

The wiki is organized by research topic. Inside a topic, each reference gets a folder named by a short slug, the short name plus the year:

wiki/research/<topic>/
  index.md              what the topic knows, and its references
  threads/              ideas and drafts
  assets/<slug>/        one reference, e.g. example-2025
    citation.bib        the citation; its key is the folder name
    note.md             the reading note, written only after reading
    paper-pdf/          the PDF as downloaded
    paper-tex/          the LaTeX source, when there is one
    github-repo/        the code, when there is any

Four rules keep it trustworthy:

• Each reference exists once. A slug appears once in the whole wiki. A paper that matters to two topics lives in one and is linked from the other.
• No note means not read. Downloading a paper is not reading it. note.md appears only after an agent has read the source, and it keeps four things apart: what the paper claims, what the code shows, what we reproduced, and what the agent makes of it. An unread reference is only its citation, linked from the index.
• The citation records whether code exists. Its code field holds the repository URL, or none once someone checked that there is none. A missing field means nobody has checked.
• Git is the log. Every ingestion is a commit, and the commit message says what came in.

The originals sit next to the note. When a question needs a detail the note skipped, the agent opens the PDF, the TeX or the code instead of guessing.

Two skills: collect and query

A skill is a folder of instructions, a SKILL.md with its supporting files, that an agent loads when a task calls for it. Claude Code, Codex and pi can all load skills. llmwiki uses two.

llmwiki-collect brings references in, in five steps:

1. Search. Bundled scripts query arXiv, DBLP, OpenAlex, OpenReview, Semantic Scholar and Crossref in parallel. They come from Microsoft’s ResearchStudio under the MIT license. The agent can also use its own web search.
2. Filter. Drop arXiv-only papers that have no code and no author from a major company or a top lab, and work the topic would not cite. Every dropped candidate is reported with the reason. A paper I name myself is flagged instead, and stays unless I agree to drop it.
3. Verify. Check the metadata against the paper, the publisher, the repository or a scholarly index. In the skill’s words: “Model recall supplies leads, not verification.”
4. Archive and read. Save the PDF, the TeX source and the code, then read and write the note.
5. File. Add the reference to the topic index and commit.

The same skill also validates the archive. A short script, lint.py, checks the rules above: one citation per folder keyed by its name, each slug once, every reference reachable from its topic index, no broken links. It only reports; fixes need my approval.

llmwiki-query answers questions and never changes the wiki. It reads the topic index, the notes or the originals, whichever the question needs. Answers cite the references they use, keep the sources’ claims apart from interpretation, and say where evidence is missing. That is where collect takes over.

How I use it

• Before an experiment. “What does the wiki say about this? Which baselines and benchmarks do people use?” The answer comes with references and their notes, and the gaps go to collect.
• When choosing a direction. Before one of my agents picked a benchmark, it surveyed what the field uses now and filed what it read.
• When writing. Citations are ready in citation.bib, and each claim in a draft can be traced to a note and a page.
• Across agents. My research runs on a team of AI agents. Every one of them can read llmwiki, so a paper one agent has read is available to all of them.

One boundary matters: the wiki holds other people’s work. Our own results stay in each project’s notes. Mixing the two would blur which claims are published and checked and which are ours and still in progress.

Set up your own

Copy the prompt at the top of this post into your agent, or install by hand. For Claude Code:

git clone https://github.com/zedong-peng/llmwiki-skill.git
mkdir -p ~/.claude/skills
ln -s "$PWD/llmwiki-skill/skills/llmwiki-query" ~/.claude/skills/
ln -s "$PWD/llmwiki-skill/skills/llmwiki-collect" ~/.claude/skills/

For other agents, the repository’s README says where skills go. Then create a git repository for the wiki and keep it private: the archived PDFs belong to their publishers. Ask in plain words:

Use llmwiki-collect to ingest https://arxiv.org/abs/<id> into <wiki path> under the topic <topic>.
Use llmwiki-query: what does <wiki path> say about <question>?

The search scripts need a few Python packages: pip install -r skills/llmwiki-collect/paper-search/requirements.txt. API keys for Semantic Scholar, OpenAlex and OpenReview are optional and are read from environment variables. Keep them out of the wiki.

Limits

• A note is only as good as the reading behind it. Agents misread too, so each note says what was read (sections, pages, files) and what was not, and you can check.
• The filter is a judgment call. It drops arXiv-only work with no code and no known lab, which sometimes drops something good. Name a paper yourself and it stays.