Mercurial > hg > cc > pub

diff index.xml @ 1:d6f13dda3a11

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
As sent to Lindahl and Nagel
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Mon, 15 Apr 2024 15:25:50 +0100
parents
children 268fe5fd117f
line wrap: on
line diff
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/index.xml	Mon Apr 15 15:25:50 2024 +0100
@@ -0,0 +1,54 @@
+<?xml version='1.0'?>
+<?xml-stylesheet type="text/xsl" href="../../../lib/xml/doc.xsl" ?>
+<!DOCTYPE doc SYSTEM "../../../lib/xml/doc.dtd" >
+<doc>
+ <head>
+  <title>Augmentations to Common Crawl</title>
+  <author>Henry S. Thompson</author>
+  <date>15 Apr 2024</date>
+ </head>
+ <body>
+  <div>
+   <title>Introduction</title>
+   <p>This site contains a preliminary publication of my augmented <link href="https://commoncrawl.org/blog/announcing-the-common-crawl-index">index files</link>
+for <link href="https://commoncrawl.org/blog/august-2019-crawl-archive-now-available">CC-MAIN-2019-35</link>.  This index contains one additional field, <code>lastmod</code>, in about 18% of the entries, giving the value of the <code>Last-Modified</code> header as a POSIX-format timestamp, enabling much finer-grained longitudinal study of the corresponding web resources.  The filename, offset and length fields in the augmented index are unchanged, and so can be used for retrieval from the original WARC files.</p>
+  </div>
+  <div>
+   <title>Contents</title>
+   <list>
+    <item>My <link href="Thompson_WebSci24.pdf">forthcoming paper</link> describing
+the augmented index and its uses</item>
+    <item>The <link href="CC-MAIN-2019-35/cdx/warc/cluster.idx">top-level index file</link></item>
+    <item><link href="CC-MAIN-2019-35/cdx/warc/idx/">The directory containing
+the individual gzipped index files themselves</link>, with names of the form
+<code>cdx-00nnn.gz</code>, for <code>nnn</code> in <code>000&ndash;299</code></item>
+   </list>
+  </div>
+  <div>
+   <title>Licence and citation</title>
+   <p>The paper and data contained herein are Copyright © 2024 Henry S. Thompson <link href="http://creativecommons.org/licenses/by-sa/3.0/deed.en">CC-BY-SA</link></p>
+   <p>Please cite information from here as follows:</p>
+   <list type="1defn">
+    <item term="For the paper"><display>Henry S. Thompson. 2024. "Improved methodology for longitudinal Web
+analytics using Common Crawl". In <emph>ACM Web Science Conference (Websci ’24)</emph>,
+May 21–24, 2024, Stuttgart, Germany. ACM, New York, NY, USA, 11 pages.
+<link href="...">[coming soon]</link>
</display><!--https://doi.org/10.1145/3614419.3644018--></item>
+    <item term="For the data"><display>Henry S. Thompson. 2024. <emph>Augmented index
+for Common Crawl August 2019, with Last-Modified timestamps</emph>.
+<link href="https://markup.co.uk/ccrawl/">https://markup.co.uk/ccrawl/</link>. Retrieved ...</display></item>
+   </list>
+  </div>
+  <div>
+   <title>Acknowledgements</title>
+   <p>Without the vision of those responsible for Common Crawl and the
+generosity of Amazon in hosting it this work could never have happened.</p>
+   <p>Access to the <link href="http://www.cirrus.ac.uk">Cirrus UK National Tier-2 HPC Service</link> at the Edinburgh
+Parallel Computing Centre used to produce the augmented index was supported
+by EPSRC and UKRI HPC Access awards to Henry S. Thompson.</p>
+   <p>Thanks to Sebastian Nagel of Common Crawl for many prompt and helpful
+replies to many emails over the years, and to Greg Lindahl of Common
+Crawl and Tom Morris for more recent help with consistency problems in the index
+and the challenges of increasing load on the Common Crawl servers.</p>
+  </div>
+ </body>
+</doc>