Sometimes one page is not enough. You need every page of a regulator's guidance, every article in a supplier's help centre, or every press release a company has published, as they all stood on one day. Capturing them one by one is slow, and you cannot be sure you found them all.
The usual answer is a crawler that starts at one page and follows links. It finds pages that are linked, misses pages that are not, and wanders into menus, tag pages and other sections you did not want. There is a better starting point: the site's own list of its pages.
Why start from the sitemap
A sitemap is an XML file a site publishes to tell search engines which addresses it has. The format is set out in the Sitemaps protocol. Each entry gives a page's address in a <loc> tag, and may give the date it last changed in <lastmod>.
For archiving, that gives you three things a link-following crawl does not:
- A defined list. You capture what the site says it has, not whatever the links happened to lead to.
- A record of scope. Keep the list and you can show exactly which pages you set out to capture, and which you got.
- Change dates. Where
<lastmod>is filled in honestly, you can see which pages changed since your last pass.
Step 1: find the sitemap
Try these in order.
- Read the site's robots.txt. Open
https://example.org/robots.txtand look for a line such asSitemap: https://example.org/sitemap.xml. The protocol lets a site name its sitemap there, and the line can appear anywhere in the file. - Try the usual address. Many sites put it at
/sitemap.xmlin the root. Google recommends the root, though a sitemap can live anywhere on the site. - Look for a section's own sitemap. Large sites often split theirs, so a news or guidance section may have a file of its own.
What you open will be one of two kinds. A urlset lists pages. A sitemapindex lists further sitemaps, often one per section or per month, and you follow it to the one you need. Either may be gzipped, ending in .xml.gz.
Step 2: read what it lists
The protocol caps a single sitemap at 50,000 addresses and 50 MB uncompressed. Bigger sites must split their list across several files under an index, which can help you: the split often follows the site's own sections.
Only <loc> is required. Treat the optional tags with care:
<lastmod>should be the date the page last changed, not the date the sitemap was generated. Not every site gets this right. Google says it uses the value only if it is "consistently and verifiably" accurate, which tells you it often is not.<changefreq>and<priority>are hints for crawlers. Google ignores both. They tell you nothing about what the page says.
Step 3: narrow it to the section you need
Extract the addresses and keep only the ones under the section's path. On macOS or Linux, this pulls the list from a sitemap and keeps the guidance pages:
curl -s https://example.org/sitemap.xml \
| grep -o '<loc>[^<]*</loc>' \
| sed -e 's/<loc>//' -e 's/<\/loc>//' -e 's/&/\&/g' \
| grep '^https://example.org/guidance/' \
> urls.txt
For a gzipped file, add | gunzip after the curl line. The & step matters: the protocol requires characters such as & to be escaped inside the file, so an address with a query string comes out wrong without it.
Then:
- Count the lines (
wc -l urls.txt) before you capture anything. A guidance section of 40 pages and one of 4,000 need different plans. - Save the sitemap file itself, with the date and time you fetched it. It is your evidence of what the site listed on that day.
- Keep
urls.txt. It is the list you will check your captures against.
Step 4: capture every page on the list
How you capture depends on how many pages there are and what the record is for.
A few dozen pages. Capture each one with whatever tool you already use for single pages. It is slow but you see every page as it is saved.
More than that, with free tools. GNU Wget reads a list of addresses from a file and can write a WARC file as it goes:
wget -i urls.txt -p -E -H -k --wait=2 --random-wait --warc-file=guidance
-i reads the list, -p fetches the images and stylesheets each page needs to display, -H lets it fetch those from other hosts such as a CDN, -E and -k make the saved copies open locally, and --wait with --random-wait spaces out the requests so you do not load the site heavily. Wget follows what it finds in HTML and CSS. A page whose text is assembled by scripts in the browser can come out thin or empty, so open a sample before trusting the run.
Browsertrix Crawler, from Webrecorder, drives a real browser instead. It can read a sitemap itself (--sitemap), filter its entries by date (--sitemapFromDate), or take your list as a seed file (--seedFile), and it can package the result as a WACZ file. If you are not sure which container you need, we explain WARC in what is a WARC file.
Whatever the tool, space out your requests. The standard for robots.txt, RFC 9309, says its rules are not a form of access authorisation. They lock nothing away, but a site that asks crawlers to keep out of a path is still telling you something, and it is worth knowing before you capture there.
Step 5: check the result against the list
A run that finished is not a run that worked. Before you rely on it:
- Compare counts. The number of pages captured should match the lines in
urls.txt. List the gaps and capture them again. - Open a sample. Check a handful of pages from different parts of the list, and look for empty pages, cookie banners covering the text, and error pages saved as if they were content.
- Write down what you did. The date and time, the sitemap address, the tool and its version, and any pages that failed. A short note now saves an awkward answer later.
- Hash the files. Record a SHA-256 hash of the sitemap, the list and each capture, so you can later show none of them changed. We cover what that does and does not prove in what a SHA-256 hash proves.
What a sitemap will not give you
The sitemap is the site's list, and the site chose what to put on it.
- Pages left off. A page can be live and linked but missing from the sitemap. If completeness matters, spot-check the section's menus and index pages against your list.
- Pages behind a login. A sitemap lists public addresses. Anything that needs an account is out of reach of these methods.
- Deleted pages. A page removed from the site usually disappears from the sitemap too, without a trace. Only your earlier copy shows it existed.
- Reliable dates. As above,
<lastmod>is the site's claim, not proof of when a page changed.
Doing it again
The second pass is where a sitemap pays off. Save each run's list under its date, sort each one (sort -o urls.txt urls.txt), and compare two of them:
comm -3 urls-2026-10-06.txt urls-2026-11-06.txt
The output shows addresses that appeared or disappeared between the two dates, which is often the first sign that a section was reorganised or a page withdrawn. For how often to repeat the capture, see how often should you capture a web page.
Where Archivepanel fits
On the paid plan, Archivepanel can run this on a schedule. A sitemap schedule reads an XML sitemap, follows a sitemap index one level down, handles gzipped files, and captures the pages listed, hourly, daily, weekly or monthly. Each page is saved by a real Chromium browser as a self-contained HTML file with a preview, a text version, a WARC file and a manifest listing a SHA-256 digest for each file, and each capture gets an RFC 3161 timestamp from DigiCert. Any two captures of a page can be compared as text. See scheduled captures.
The limits are worth knowing before you start. A run captures 20 pages by default and at most 100, and reads at most 12 sitemap files, so a larger section needs a narrower sitemap, such as one child file from an index. Pages behind a login are not captured. The free plan, with 100 MB of storage, captures single pages by hand.
Sources
- Sitemaps.org, Sitemaps XML format.
- Google Search Central, Build and submit a sitemap.
- IETF, RFC 9309: Robots Exclusion Protocol, September 2022.
- GNU Project, GNU Wget manual.
- Webrecorder, Browsertrix Crawler: CLI options.