|
33
|
1 <?xml version='1.0'?>
|
|
|
2 <?xml-stylesheet type="text/xsl" href="../../../lib/xml/doc.xsl" ?>
|
|
|
3 <!DOCTYPE doc SYSTEM "../../../lib/xml/doc.dtd" >
|
|
|
4 <doc xmlns:x="http://www.w3.org/1999/xhtml">
|
|
|
5 <head>
|
|
|
6 <title>LURID3: Longitudinal studies of the World Wide Web<x:br/>
|
|
|
7 <span style="font-size:80%">UKRI reference APP39557</span></title>
|
|
|
8 <author>Henry S. Thompson</author>
|
|
|
9 <date>22 Apr 2024</date>
|
|
|
10 </head>
|
|
|
11 <body>
|
|
|
12 <div>
|
|
|
13 <title>Vision</title>
|
|
|
14 <div>
|
|
|
15 <title>Motivation</title>
|
|
|
16 <p>Empirical evidence of how use of the Web has changed in the past provides crucial input to decisions about its future. Creative uses of the mechanisms the Web provides expand its potential, but also sometimes put it at risk, so it’s worrying that there’s surprisingly little empirical evidence available to guide standardization and planning more generally. Which aspects of the Web’s functionality are widely used? Hardly ever used? How is this changing over time?</p>
|
|
|
17 <p>The kind of evidence needed to answer such questions is hard to come by.
|
|
|
18 The proposed research builds on our previous work in this area [Thompson and
|
|
|
19 Tong 2018], [Thompson 2024], taking advantage of the computational resource Cirrus provides to validate and expand our work on the Common Crawl web archive.</p>
|
|
|
20 <p>Common Crawl (CC) is a very-large-scale web archive, containing
|
|
|
21 petabytes of data from more than 65 monthly/bi-monthly archives, totalling over
|
|
|
22 100 billion web pages. Collection began in 2008 with annual archives,
|
|
|
23 expanding steadily to the point that since 2017 archives were collected monthly
|
|
|
24 until 2023, since when it's been bi-monthly. Recent archives contain over 3x10^9 pages, about 50 Terabytes (compressed). Together with Edinburgh colleagues we have created local copies of 8 months of CC in a petabyte store attached to Cirrus. For our purposes it is important to note that the overlap between any two archives as measured by Jaccard similarity of page checksums is less than .02 [9].</p>
|
|
|
25 <p>The proposed work will build on results from our just-completed project
|
|
|
26 (<name>LURID2: Assessing the validity of Common Crawl</name>, EPSRC Access to
|
|
|
27 HPC Award from 2022–12 to 2023–04) on the Common Crawl web
|
|
|
28 archive (CC), in the course of which we accomplished almost all of our four main objectives.</p>
|
|
|
29 </div>
|
|
|
30 </div>
|
|
|
31 <div>
|
|
|
32 <title>Approach</title>
|
|
|
33 </div>
|
|
|
34 </body>
|
|
|
35 </doc>
|
