Mercurial > hg > cc > pub
comparison eidf125_example.sh @ 5:e265fcc42974
s3 use
| author | Henry S. Thompson <ht@inf.ed.ac.uk> |
|---|---|
| date | Thu, 23 May 2024 15:00:40 +0100 |
| parents | |
| children |
comparison
equal
deleted
inserted
replaced
| 4:268fe5fd117f | 5:e265fcc42974 |
|---|---|
| 1 # Illustrates the use of eidf125 data to find a particular archived page from CC-MAIN-2019-35 | |
| 2 # Cut-and-paste a bit at a time so you can see the outputs | |
| 3 aws s3 ls s3://eidf125-cc-main-2019-35-augmented-index --no-sign-request --endpoint-url https://s3.eidf.ac.uk/ | |
| 4 aws s3 cp s3://eidf125-cc-main-2019-35-augmented-index/cluster.idx --no-sign-request --endpoint-url https://s3.eidf.ac.uk/ . | |
| 5 grep -E -n '^uk,ac,ed' cluster.idx |wc -l | |
| 6 grep -E -n '^uk,ac,ed,inf' cluster.idx |wc -l | |
| 7 grep -E -n '^uk,ac,ed,inf' cluster.idx | |
| 8 aws s3api get-object --bucket eidf125-cc-main-2019-35-augmented-index --key idx/cdx-00289.gz --range bytes=88951112-$((88951112 + 181085 - 1)) --no-sign-request --endpoint-url https://s3.eidf.ac.uk/ block.gz | |
| 9 ls -l block.gz | |
| 10 zcat block.gz|less | |
| 11 zgrep ')/people/staff/.*lastmod' block.gz | sed 's/ .*lastmod/ /' | sort -k2r | |
| 12 date --date=@1533112224 | |
| 13 zgrep Plotkin block.gz | |
| 14 curl --range 847252902-$((847252902 + 3179 - 1)) -o plotkin.gz https://data.commoncrawl.org/crawl-data/CC-MAIN-2019-35/segments/1566027313803.9/warc/CC-MAIN-20190818104019-20190818130019-00233.warc.gz | |
| 15 zcat plotkin.gz |less | |
| 16 # If you have the relevant credentials, this will also work: | |
| 17 # aws s3api get-object --bucket commoncrawl --key crawl-data/CC-MAIN-2019-35/segments/1566027313803.9/warc/CC-MAIN-20190818104019-20190818130019-00233.warc.gz --range bytes=847252902-$((847252902 + 3179 - 1)) plotkin.gz |