# httpcached

**Repository Path**: ghink/httpcached

## Basic Information

- **Project Name**: httpcached
- **Description**: Cache HTTP data simply
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-06-12
- **Last Updated**: 2026-07-12

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# httpcached

A local **HTTP reverse-proxy cache** for file-style resources — Linux package
repos (apt / yum / apk), the npm registry, PyPI, container blobs, and any other
resource served over plain HTTP. It cuts load on upstream mirrors, eliminates
wasted re-downloads, and dramatically speeds up repeated/CI builds.

Built on GoFiber v3 (fasthttp), zap and viper.

## How it works

```
client ──HTTP──> GoFiber (fasthttp)  ──>  cache handler
   1. Host selects a site; longest-prefix path match selects an upstream
   2. Rules (ext / path / host / upstream) resolve a Policy (ttl, min_hits, ...)
   3. lookup:
        fresh hit         → stream from disk            X-Cache: HIT
        stale + validators → conditional GET (304/200)   X-Cache: REVALIDATED
        cold (< min_hits)  → pass-through, not cached     X-Cache: PASS
        admitted miss      → single-flight fetch + tee    X-Cache: MISS
```

Storage: bodies live in sharded blob files (`blobs/ab/cd/<sha256>`); metadata is
a JSON `.meta` sidecar next to each blob. Writes are atomic (temp file → fsync →
rename); a crash can only leave an orphan blob, which is cleaned on the next
startup scan.

## Quick start

```sh
go build -o httpcached .
./httpcached            # reads ./config.yaml, listens on server.port
```

Point a client at it by Host + path prefix. With the sample config:

```sh
# apt: /etc/apt/sources.list ->
#   deb http://mirror.local:8000/ubuntu jammy main
# (resolve mirror.local to the cache host, or use the cache IP directly)

# npm:
npm config set registry http://mirror.local:8000/npm/

# pip:
pip install --index-url http://mirror.local:8000/pypi/simple some-pkg

# raw check:
curl -H 'Host: mirror.local' http://127.0.0.1:8000/ubuntu/dists/jammy/InRelease -D-
```

Every response carries an `X-Cache: HIT|MISS|REVALIDATED|PASS|STALE` header.
`GET /_cache/stats` returns JSON counters (hits, misses, hit ratio, disk usage…).

## Configuration (`config.yaml`)

Send `SIGHUP` to hot-reload the `cache` section (rules / sites / upstream pool)
without dropping the cache.

```yaml
cache:
  storage:
    dir: "./cache_data"      # blobs/ + tmp/ live here
    max_size: "50GB"         # soft cap; LFU/LRU eviction above it (0 = unlimited)
    max_age: "720h"          # hard retention cap
    caretaker_interval: "5m"
  upstream:                  # shared origin connection pool / timeouts
    max_conns_per_host: 256
    max_idle_conns_per_host: 64
    idle_conn_timeout: "90s"
    dial_timeout: "10s"
    response_header_timeout: "30s"
    follow_redirects: true
    # user_agent: "httpcached/1.0.0"   # override UA sent to origins (empty = forward client UA)
  admission:
    window_size: 100000      # max keys tracked for popularity
    default_min_hits: 2
  defaults:                  # fall-back policy; rules inherit unset fields
    cache: true
    ttl: "24h"
    revalidate: true
    min_hits: 2
    stale_if_error: true
  rules:                     # first match wins
    - name: immutable-artifacts
      match: { ext: [deb, rpm, apk, whl, tgz, jar] }
      policy: { immutable: true, ttl: "720h", min_hits: 1 }
    - name: repo-metadata
      match: { path_regex: '(InRelease|Packages(\.\w+)?|repomd\.xml|.*\.json)$' }
      policy: { ttl: "5m", revalidate: true, min_hits: 1 }
    - name: no-cache-dynamic
      match: { path_regex: '(/token|/v2/auth)' }
      policy: { cache: false }
  sites:                     # Host -> site; path prefix -> upstream
    - hosts: ["mirror.local", "127.0.0.1"]
      routes:
        - { path: "/ubuntu", upstream: "http://archive.ubuntu.com/ubuntu" }
        - { path: "/npm",    upstream: "https://registry.npmjs.org" }
        - { path: "/pypi",   upstream: "https://files.pythonhosted.org" }
    - hosts: ["npm.local"]
      routes:
        - { path: "/", upstream: "https://registry.npmjs.org" }
```

### Policy fields

| field | meaning |
|---|---|
| `cache` | `false` ⇒ never cache (pure pass-through) |
| `ttl` | freshness lifetime |
| `immutable` | never revalidate; serve until evicted |
| `revalidate` | when stale, do a conditional GET (`If-None-Match` / `If-Modified-Since`) |
| `min_hits` | requests before an object is admitted to disk (`1` = cache immediately) |
| `respect_origin` | fold the origin's `Cache-Control` / `Expires` into the TTL |
| `stale_if_error` | serve a stale copy if the origin is unreachable during revalidation |
| `ignore_query` | drop the query string from the cache key |
| `max_object_size` | objects larger than this are streamed through but not cached |

## Behaviour notes

- Only `GET` is cached; other methods and no-cache policies are proxied verbatim.
- `Accept-Encoding` is not forwarded, so cached bodies are identity-encoded and
  safe to replay to any client.
- Range requests are served from a fully-cached object (`206`); a range for an
  uncached object is passed through (the cache is never populated from a partial
  response).
- Hop-by-hop headers are stripped both ways. Proxy-chain headers from a front
  nginx/CDN (`X-Forwarded-*`, `Forwarded`, `X-Real-IP`, `Via`) are also stripped
  before contacting the origin — forwarding them makes some CDNs "correct" the
  scheme with a redirect loop.
- Origin fetch failures are logged (with the URL and, for redirect loops, the
  full hop chain) in addition to the 502 body returned to the client.

## Limitations

- No Docker Registry v2 token-auth handshake (generic HTTP caching only — works
  for repos/registries that don't require a per-request token).
- Multi-range requests are served as their first range.
- Admission uses an approximate LRU frequency filter (good enough to separate
  cold from hot).

## Development

```sh
go test ./... -race      # unit + integration (single-flight, admission, range, persistence)
```