What is a WARC file, and do you need one?

Sooner or later someone asks you for a WARC. A library taking a deposit, a research partner, a litigation support team, or an archiving policy written by someone who has since left. You have a PDF of the page, perhaps a screenshot, and a file extension you have never opened.

This article explains what a WARC file is, what it records that other formats don't, how to open and check one with free tools, and when you can safely do without.

The short answer

WARC (Web ARChive) is the standard container format of web archiving. One WARC file holds any number of records, each a short block of text headers followed by some content: usually a web server's response, with a note of the address it came from and when. The specification describes it as a convention for "concatenating multiple resource records (data objects), each consisting of a set of simple text headers and an arbitrary data block into one long file".

It is not a document format. You cannot double-click a WARC and read it. You open it in replay software, which rebuilds the page from the stored responses.

Where the format came from

WARC grew out of ARC, the format the Internet Archive has used to store its crawls since 1996. WARC extends it so that one file can also carry metadata, note duplicate content, and hold the results of later transformations alongside the original.

WARC 1.0 was published as ISO 28500:2009. The current version, WARC 1.1, was prepared with a task force of the International Internet Preservation Consortium (IIPC) and published in August 2017 as ISO 28500:2017. The format matters because that is where the collections are. The Library of Congress stores its web archives as WARC files, with ARC for some older collections, and replay tools are built to read it.

What is inside a WARC file

Every record starts with a version line and a set of named fields. Here is the example response record header from the WARC 1.1 specification:

WARC/1.1
WARC-Type: response
WARC-Target-URI: http://www.archive.org/images/logoc.jpg
WARC-Warcinfo-ID: <urn:uuid:d7ae5c10-e6b3-4d27-967d-34780c58ba39>
WARC-Date: 2016-09-19T17:20:24Z
WARC-Block-Digest: sha1:UZY6ND6CCHXETFVJD2MSS7ZENMWF7KQ2
WARC-Payload-Digest: sha1:CCHXETFVJD2MUZY6ND6SS7ZENMWF7KQ2
WARC-IP-Address: 207.241.233.58
WARC-Record-ID: <urn:uuid:92283950-ef2f-4d72-b224-f54c6ec90bb0>
Content-Type: application/http;msgtype=response
WARC-Identified-Payload-Type: image/jpeg
Content-Length: 1902

After the headers comes the content itself, in this case the server's full HTTP response: status line, response headers and the image.

Four fields are required in every record: WARC-Record-ID, a unique identifier; Content-Length; WARC-Type; and WARC-Date, which the specification defines as the moment "that data capture for record creation began", in UTC. WARC-Target-URI is the address that was fetched. The two digest fields are optional fingerprints of the record's content, so a later reader can tell whether it has changed.

The record types you are most likely to meet:

• warcinfo describes the file itself: usually the tool that made it and how.
• request and response hold the request sent and the reply received over the network, paired by a WARC-Concurrent-To field.
• resource holds content "without full protocol response information", for example a file retrieved where the protocol details were discarded.
• metadata carries descriptive information about another record.
• revisit notes that content already archived was seen again, so an unchanged image isn't stored twice.
• conversion holds a version of another record's content made by an archival process, such as a format migration.

A single page usually takes many records: one for the HTML, and one for each stylesheet, script, font and image it loaded. Files are normally compressed record by record with gzip and named .warc.gz, which lets tools jump straight to one record without unpacking the rest.

Not every WARC records the same thing

If you rely on a WARC as a record, this matters. The specification says a response record for a web page "should contain the full HTTP response received over the network, including headers". A crawler such as Heritrix, or wget with its --warc-file option, records its exchanges with the server as it makes them.

Some tools work differently: they save the page only after a browser has assembled it. The individual network exchanges were never recorded, so any response headers in the file have been rebuilt afterwards. That is a legitimate WARC, and it replays well. But it is a record of the finished page, not of the traffic.

Neither is better in every case. A crawler WARC shows what the server sent; it can miss content that only appears once scripts run. A browser WARC shows what a reader saw, but its headers are not evidence of what the server said.

Before relying on a WARC, open its warcinfo record and read what made it. A well-made file says so. If it doesn't, ask whoever produced it.

How to open and check a WARC file

All of these tools are free and open source.

1. To look at it, load the file into ReplayWeb.page. It runs in your browser with no server behind it, reads .warc, .warc.gz and WACZ files among others, and shows the archived pages as they were captured.

2. To host a collection, use pywb, which gives your own files Wayback Machine-style replay. After pip install pywb:

   wb-manager init my-web-archive
   wb-manager add my-web-archive capture.warc.gz
   wayback
   

   The pages are then at http://localhost:8080/my-web-archive/<url>/.

3. To see what is inside, use warcio (pip install warcio). This lists every record with its type and address:

   warcio index capture.warc.gz -f warc-type,warc-target-uri,content-length
   

4. To check it hasn't been altered, run warcio check -v capture.warc.gz. It recomputes the payload and block digests where the file includes them, and exits with an error if any record fails.

A digest check tells you the file is internally consistent. It doesn't tell you when the file was made: anyone rewriting a WARC can write fresh digests. For that you need an independent timestamp over the file, which we explain in how to verify an RFC 3161 timestamp with OpenSSL.

Do you need one?

You need WARC if you are depositing web material with a library or archive, feeding captures into existing web archive tools, or sharing collections with researchers who use replay software. It is also the format to keep when you capture many pages from one site and want them replayed together, links and all.

You probably don't need it to read, quote or show a page to someone else. A WARC needs software to open, and a judge, a client or a regulator is unlikely to have it. For that, a self-contained HTML file or a PDF opens anywhere, and a plain-text copy is what you actually quote. Our guide on how to preserve a web page as evidence covers what that copy has to record.

The practical answer for most people is to keep both: a readable copy for people, and a WARC for tools and institutions.

WACZ, from the Webrecorder project, packages WARC files, an index and a list of pages into one standard ZIP file. The WARCs inside are ordinary WARC files.

How Archivepanel handles it

Each Archivepanel capture stores a WARC 1.1 file (.warc.gz) alongside the self-contained HTML page, a preview image, a plain-text version and a JSON manifest. One WARC download carries the page, its text and its preview together, and it opens in pywb and ReplayWeb.page.

Our captures are made by a real browser, so the WARC is the second kind described above. Its response headers are reconstructed, and the file says so in its own metadata, so nobody later mistakes it for a crawler's record of the network traffic. The HTML file itself carries a DigiCert RFC 3161 timestamp, which you can check with OpenSSL without an account.

WARC downloads and whole-account exports are available on every plan, including the free one. The export page lists what each download contains.

Sources

• IIPC, The WARC Format 1.1, including the record types, required fields and the example record quoted above.
• IIPC, The WARC file format: preparing next steps, on the publication of WARC 1.1 as ISO 28500:2017 in August 2017.
• Library of Congress, How are the web archives stored?
• Webrecorder, ReplayWeb.page user guide, WACZ specification 1.1.1 and warcio.
• pywb documentation: usage.
• GNU Wget manual, WARC options.
• Heritrix documentation.