Archivepanel Web Preservation System

Features

Everything a snapshot needs to still be worth something later.

Archivepanel is deliberately narrow: it captures pages faithfully, files them privately, keeps them unchanged, and refuses to trap them. What follows is what that means in practice.

Capture fidelity

One file that still renders when the site is gone.

A capture drives a real browser, waits for the page to settle, and writes a single document with every stylesheet, image and font embedded as a data URI and unused CSS pruned away. Scripting is stripped, and a content policy is embedded in the saved file so an opened archive cannot reach the network even if something were left behind. The browser runs at a 1440×900 viewport; video and audio are not captured.

Why one file

A saved page that scatters assets across a folder is a snapshot with moving parts. Move it, mail it, put it in evidence storage, and half of it stops resolving. One document has nothing left to lose.

Why a real browser

Faithfully embedding a modern page — responsive image sets, shadow DOM, custom properties — is a hard problem. Archivepanel drives headless Chromium through SingleFile rather than pretending a hand-rolled inliner is equivalent.

Why the preview matters

The preview image is a screenshot of the saved file, never of the live page. A capture that lost its styling therefore looks wrong in your archive list on day one, instead of looking fine until someone opens it in year three.

Why the banner goes first

A cookie banner is the usual reason a capture records nothing useful: it covers the page, and nearly all of them lock scrolling while they are up, so the lazily loaded content below the fold is never fetched either.

How the banner is handled

Where a known consent platform is recognised, its own accept control is pressed — which is also the only thing that reveals a page genuinely gated behind consent, rather than merely covered by it — and its container is dropped from the saved document.

Why no guessing

Every selector names one platform's own control. There is deliberately no "looks like a banner" pass: a heuristic that hides whatever floats and mentions cookies eventually takes a piece of the page with it. A banner left standing is a worse archive; a missing masthead is a wrong one. The record states that consent was auto-accepted, so the snapshot is honest about how it was taken.

Stored files

Five artifacts per capture, each with a job.

FileProduced fromPurpose
.html The live page The archive itself. Self-contained, script-free, offline-renderable.
.jpg The saved file Verification and recognition. Shown on the record and beside each row in the archive list.
.txt The saved file Reading, quoting, grepping, diffing. Served from the dashboard itself, since it is text we generated rather than captured markup.
.warc.gz All three Interchange. One download carrying the document, the text rendering and the screenshot together.
.json All of the above Audit and inventory. A size, a SHA-256 digest and a content type for every stored file, plus the addresses, times, viewport and tool versions behind the capture.
Four of the five are companions, not the archive. If the preview, the text, the WARC or the manifest cannot be produced, the capture is still kept and the record says so. Nothing is discarded because an accessory did not render.

Text snapshots

The page, in a form you can actually work with.

Every capture is also converted to structured plain text — from the bytes just saved, not from the live page, so the text and the archive can never disagree about what the page said.

It keeps the structure worth keeping and drops what carries no readable text: inline SVG, hidden elements, template bodies, and the data URIs that would otherwise print megabytes of base64 mid-sentence.

What survives the conversion

  • Headings underlined
  • Lists bulleted
  • Blockquotes prefixed
  • Tables drawn as an ASCII grid
  • Link targets printed beside their text
  • A provenance header carrying the title, the resolved address and the capture time

Reading it back

The text opens in the dashboard with a find box that matches as you type, highlights every occurrence and steps between them with Enter and Shift+Enter. It runs entirely in your browser over text the page already holds — no request per keystroke, and what you search an archived document for is never sent anywhere. Snapshots are shown up to 512 KB, with the whole file a download away.

Finding things again

An archive you cannot search is a drawer.

Three ways in, each answering a different question: which snapshot, which site, and which passage.

The archive list

Which snapshot

Title, address and host are searched a word at a time — every word you type must appear somewhere, in any order, so nos italie finds the article about Italy on nos.nl. Filter by status, and sort by newest, oldest, largest, title or site.

Sites

Which site

Every snapshot you hold, grouped by host — the view to use when you want the history of one place rather than one page, and the way to see how often a site has been captured and when it last was.

Search inside a site

Which passage

Searching within a site reads the stored text of that site's snapshots and shows the passage that matched, so a page can be found by something it said rather than by its title.

The text search is bounded, and says when it was. It reads the newest 200 snapshots of the site, within a few seconds, and tells you when it did not cover everything rather than implying it did. There is no index of your archive contents anywhere — nothing to build, backfill, invalidate on delete or carry in every nightly backup — which is affordable only because you have already said which site you mean.

Scheduled captures

Two kinds of schedule, four cadences.

A page schedule captures its own address, one archive per run. A sitemap schedule reads its address as an XML sitemap and captures the pages that sitemap lists — so one schedule covers a whole section of a site, and keeps covering it as the site adds pages.

Cadence

Hourly, daily, weekly or monthly. A run is skipped while the previous one is still draining, so a slow site cannot pile work on itself.

Sitemap handling

Both sitemaps.org document types are understood — a urlset listing pages, and a sitemapindex pointing at further sitemaps, followed one level down. The gzipped form is decompressed transparently, detected by content rather than by file extension, because servers are inconsistent about both.

Bounds on a sitemap run

BoundValueWhy
Pages per run20 default, 100 maxSet per schedule when it is created.
Sitemap files read12An index pointing at hundreds of children costs twelve requests, not hundreds.
Document size12 MBAfter decompression.
Whole traversal45 sA sitemap that will not resolve does not hold a worker.
Outstanding captures5A run is skipped rather than queued behind itself.
Failure is reported once per run, not once per page. A sitemap that cannot be read raises one alert and one email. Dozens of messages for one event is not a notice, it is a flood.

WARC export

A snapshot taken here is not trapped here.

Every capture is packaged as a WARC file (ISO 28500, WARC 1.1) alongside the document. The same bytes open in pywb, replay through ReplayWeb.page, and satisfy an institution that accepts nothing else.

It is offered as a download from the record page rather than as a view — it is a file for other tools, not something a browser renders. Packaging happens after the document is saved and costs no second visit to the site.

Its metadata record is the JSON manifest itself rather than a summary of it, so one download carries the tool versions, the viewport, the timings, what the page declared about itself, how its host resolved, and a digest for every file the capture produced. Two accounts of one capture are two things that can come to disagree.

Records in the file

RecordContents
warcinfoSoftware, format version, user agent, and the page list replay tools need
responseThe saved document, as an HTTP response block
metadataThe JSON manifest in full, byte for byte, tied to the document it describes
conversionThe plain-text rendering
resourceThe preview image
The response headers are reconstructed, and the file says so. A capture drives a browser and returns one document with its subresources already inlined; the individual HTTP exchanges that built the page are never observed. The container states only what is genuinely known, and the metadata record notes in so many words that the origin response was not seen — rather than leaving a later reader to assume a crawler produced it.

Progress and notifications

A capture takes minutes. You should not have to sit there.

Live status

Status, stage and an elapsed-time progress bar update in place while a capture runs, and the record page refreshes itself once the capture settles. Polling backs off when nothing is running and stops entirely in a background tab.

Notifications

When a capture is stored, fails, or is stopped by a storage limit, a notification goes to the tray in the masthead with an unread count — and arrives as a toast if you happen to be looking at the time.

A capture you no longer want can be stopped. Anything queued or running can be halted from the progress card, the archive list or the record page. Stopping kills the browser immediately and writes nothing to storage, so an abandoned capture costs no allowance. It is not a failure: it raises no alert, sends no email, and shows in grey rather than red — though the activity log still records it, which is where a question about a missing snapshot is answered.

Security and privacy

Two threats, handled separately.

A service that fetches addresses on request and later renders the result in your browser has two distinct problems: it must not be usable as a way into our network, and a captured page must not be able to act inside your account.

Outbound — what we will fetch
  • Addresses resolving to loopback, RFC1918, link-local, carrier-grade NAT or unique-local space are refused.
  • Every redirect hop is screened again, not just the address you typed.
  • Addresses named by a sitemap get the identical check before they are fetched or queued.
  • Capture work runs off the request path, under a fixed worker pool and a memory cap.
Inbound — what an archive may do
  • Captured markup is served under a policy that pins it to an opaque origin, with scripts and forms refused.
  • The saved file carries an embedded policy of its own, so the archive cannot reach the network wherever it is opened.
  • Scripts are stripped at capture and every asset is inlined, so an archive needs no network access to display.
  • Preview images are proxied the same way rather than served from storage.

Account security

  • Two-step sign-in — password, then a six-digit emailed code. Only the code's digest is stored, five attempts per challenge.
  • argon2id password hashing at 64 MB, unique salt per hash.
  • Session cookies hold a 256-bit token; only its digest is stored.
  • Proof-of-work on sign-in and registration, with single-use solutions.
  • Signed-in sessions are listed in settings, so a session that is not you can be revoked without changing anything else.
  • CSRF protection on every state-changing request.
  • Unknown addresses still incur the password delay, so response timing does not disclose which accounts exist.
  • Every archive query is scoped to the owning account.
  • Deletion requires your password and is rate limited.
  • Leaving is self-service — empty the account of every snapshot and schedule, or close it outright, both from settings and both behind your password.

Storage and limits

An allowance you can see, enforced where it can be honest.

Each account has a snapshot storage allowance — 50 MB by default. Usage counts every file a capture stores, since all of them occupy the bucket. Your current usage is on the dashboard, not buried in a settings page.

  • Queuing is refused up front when an account is already full — a courtesy, since a page's size is unknown until it has been captured.
  • The binding check is immediately before upload. A capture that would cross the line is failed with an explanation and never written.
  • Scheduled captures are skipped while an account is over its limit and resume on their own once space is freed. The schedule is not disabled behind your back.
  • Need more room? Allowances are set per account — write to us.

Start with one page.

Create an account, paste an address, and see the snapshot — document, preview, text, WARC and manifest — land in your archive.