Mercurial > hg > cc > cirrus_home
changeset 19:c858a4d4bd4f
copied from valhalla/bin
| author | Henry S. Thompson <ht@inf.ed.ac.uk> |
|---|---|
| date | Mon, 16 Mar 2020 15:57:23 +0000 |
| parents | 6662a353379a |
| children | 5371599c0cfa |
| files | bin/extract.sh |
| diffstat | 1 files changed, 11 insertions(+), 0 deletions(-) [+] |
line wrap: on
line diff
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/bin/extract.sh Mon Mar 16 15:57:23 2020 +0000 @@ -0,0 +1,11 @@ +#!/bin/bash +me=$$ +SHARED=/home/shared/ht +cd $SHARED/data/$(hostname) +mkdir -p logs +while read id +do + echo starting ${id} $(date) >> logs/${me}_log + unpigz -dp 1 -c /data/common_crawl/CC-MAIN-2019-35/CC-MAIN-${id}.warc.gz|$SHARED/bin/warc.sh ${id} application/pdf 2>> logs/${me}_log + echo finished ${id} $(date) >> logs/${me}_log +done