# llm-wiki

**Repository Path**: computerhong/llm-wiki

## Basic Information

- **Project Name**: llm-wiki
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-05-21
- **Last Updated**: 2026-06-04

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# LLM-Wiki

> A local, LLM-maintained personal knowledge base. Drop documents in, watch an LLM compile them into a living, interlinked Obsidian wiki you can search, query, and export as slides.

[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Local-first](https://img.shields.io/badge/runs-100%25_local-green.svg)](#)

Built on the pattern Andrej Karpathy described in his [LLM Wiki gist](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f): instead of retrieving from raw documents at query time (classic RAG), an LLM incrementally **compiles** your sources into a structured, cross-linked markdown wiki. The wiki is a **persistent, compounding artifact** — cross-references are already there, contradictions have already been flagged, synthesis already reflects everything you've read.

You never write the wiki yourself. The LLM does all the grunt work: summarizing, cross-referencing, filing, bookkeeping. You bring the sources and ask the questions.

Runs 100% locally via Ollama. No API keys, no cloud, no data leaving your machine.

## What it does

```bash
# Drop files in (PDFs, markdown, HTML, DOCX, text)
wiki add ~/Documents/papers --recursive

# Watch the LLM read them and build an interlinked wiki
wiki ingest

# Ask questions — it searches the compiled wiki and cites its sources
wiki query "what's the main argument about X?"

# Health-check the knowledge base
wiki lint --fix

# Start the web UI
wiki serve

# Browse the whole thing in Obsidian
open wiki/
```

## Features

### Core capabilities
- **Incremental ingest** — drop a file, run `wiki ingest`, get cross-linked wiki pages
- **Structured extraction** — LLM identifies entities (people, orgs, models), concepts, and key takeaways per source
- **Smart merging** — re-ingesting related sources updates existing pages instead of overwriting them, preserving provenance
- **Cited synthesis** — queries return markdown answers with `[[wikilinks]]` pointing to supporting pages
- **Write-back** — save good answers as new `synthesis/` pages with `--save-as`
- **Wiki linting** — health checks for broken links, orphans, malformed frontmatter, with auto-fix
- **Auto-reindex** — search index refreshes automatically after ingest and lint

### Web UI (`wiki serve`)
A FastAPI web interface with SSE streaming for real-time progress:

- **Obsidian 管理** — unified dashboard combining:
  - D3 force-directed knowledge graph (color-coded by page type)
  - Markdown source reader (opens in new tab, no download)
  - Real-time ingest monitor with live progress
  - Lint report with one-click auto-fix
  - Persistent job history (survives tab close and server restart)
  - Marp PPT generation — select wiki pages, generate slides, export as PDF via marp-cli
- **Sources** — list, inspect, delete, or re-ingest sources
- **Query** — chat-style interface with streaming synthesis, scope toggle, save-as-synthesis
- **Dashboard** — project stats and recent activity

### Supported input formats
`.pdf` · `.md` · `.html` · `.docx` · `.txt`

### Obsidian integration
The `wiki/` folder is a ready-made Obsidian vault:
- Color-coded graph view (sources, entities, concepts, synthesis each get their own color)
- YAML frontmatter compatible with Dataview plugin
- All cross-references as native `[[wikilinks]]`

## Architecture

```
raw/ ──────► LLM Agent ──────► wiki/
(documents)   (Ollama)        (markdown,
                               auto-linked)
                  │                │
                  ▼                ▼
            schema/          Obsidian
            AGENTS.md        graph view
```

- **`raw/`** — source documents. Immutable. The agent reads but never modifies.
- **`wiki/`** — LLM-maintained markdown. One folder per page type (`sources/`, `entities/`, `concepts/`, `synthesis/`). Open this in Obsidian.
- **`schema/AGENTS.md`** — conventions file. Tells the LLM how to format pages, when to merge vs create, how to cite.
- **`output/`** — generated Marp slide decks and PDFs.
- **`.wiki/`** — internal state: SQLite ingest history, search index, config. Git-ignored.

## Stack

| Layer | Component |
|---|---|
| LLM | [Ollama](https://ollama.com) + Qwen via local model |
| Web | FastAPI + uvicorn + vanilla JS |
| Graph | D3.js v7 force-directed |
| Slides | [Marp](https://marp.app/) CLI |
| CLI | [Typer](https://typer.tiangolo.com) + [Rich](https://rich.readthedocs.io) |
| Parsers | pypdf, python-docx, beautifulsoup4, lxml |
| Vault | [Obsidian](https://obsidian.md) |

No cloud services. No API keys.

## Requirements

- **Python 3.11+**
- **Ollama** with a model pulled (e.g. `ollama pull qwen3:14b`)
- **Node.js 18+** (for Marp CLI slide export)
- **~10GB free disk space** for models
- **Obsidian** (optional for browsing)

## Installation

```bash
git clone https://gitee.com/computerhong/llm-wiki.git
cd llm-wiki
uv venv && source .venv/bin/activate
uv pip install -e .

# Pull the LLM
ollama pull qwen3:14b

# Install Marp CLI for PPT/PDF export
npm install -g @marp-team/marp-cli

wiki --help
```

## Quick start

```bash
# 1. Create a wiki
mkdir my-wiki && cd my-wiki
wiki init

# 2. Add source documents
wiki add ~/Documents/papers --recursive

# 3. Run ingest
wiki ingest

# 4. Start the web UI
wiki serve --port 24311
# Open http://localhost:24311/obsidian

# 5. Or browse in Obsidian
open wiki/
```

## Commands

| Command | Purpose |
|---|---|
| `wiki init [path]` | Scaffold a new wiki project |
| `wiki add <file-or-folder> [-r]` | Copy sources into `raw/` and register for ingest |
| `wiki ingest [source_id]` | Run the LLM ingest pipeline |
| `wiki query "<question>"` | Search + synthesize a cited answer |
| `wiki lint [--fix]` | Health-check the wiki |
| `wiki serve [--port N]` | Launch the web UI |
| `wiki status` | Show project stats and backend health |

## License

MIT — see [LICENSE](LICENSE).

---

*"The tedious part of maintaining a knowledge base is not the reading or the thinking — it's the bookkeeping. LLMs don't get bored, don't forget to update a cross-reference."*
— Karpathy