Augmentations to Common Crawl

1. Introduction

This site contains a copy of my augmented index files for CC-MAIN-2019-35. This index contains all of the original index, with one additional field, lastmod, in about 18% of the entries, giving the value of the Last-Modified header as a POSIX-format timestamp, enabling much finer-grained longitudinal study of the corresponding web resources. The filename, offset and length fields in the augmented index are unchanged, and so can be used for retrieval from the original WARC files.

The format of the Common Crawl's index files is described in this announcement.

2. Contents

• My paper, presented at WebSci24, describing the augmented index and its uses
• The top-level index file
• The directory containing the individual gzipped index files themselves, with names of the form cdx-00nnn.gz, for nnn in 000–299
• WebSci 24 conference slides

3. Efficient access to Common Crawl using Amazon S3

The University of Edinburgh's Edinburgh International Data Facility (EIDF) hosts a copy of the augmented index in an Amazon S3 server. It supports open access to the index via unsigned requests to (range-restricted) s3: URIs, for example using the Amazon aws Command Line Interface.

The best way to understand how this works, once you've read how the index itself works in the paper, section 2.1, is to work through an example of using the augmented index to access an individual Common Crawl retrieval record using a timestamp.

4. Licence and citation

The paper and data contained herein are Copyright © 2024 Henry S. Thompson CC-BY-SA

Please cite information from here as follows:

• For the paper   

  Henry S. Thompson. 2024. "Improved methodology for longitudinal Web analytics using Common Crawl". In ACM Web Science Conference (Websci ’24), May 21–24, 2024, Stuttgart, Germany. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3614419.3644018

• For the data   

  Henry S. Thompson. 2024. Augmented index for Common Crawl August 2019, with Last-Modified timestamps. https://markup.co.uk/ccrawl/. Retrieved ...

5. Acknowledgements

Without the vision of those responsible for Common Crawl and the generosity of Amazon in hosting it this work could never have happened.

Access to the Cirrus UK National Tier-2 HPC Service at the Edinburgh Parallel Computing Centre used to produce the augmented index was supported by EPSRC and UKRI HPC Access awards to Henry S. Thompson.

Thanks to Sebastian Nagel of Common Crawl for many prompt and helpful replies to many emails over the years, and to Greg Lindahl of Common Crawl and Tom Morris for more recent help with consistency problems in the index and the challenges of increasing load on the Common Crawl servers.