Mercurial > hg > cc > cirrus_home
comparison bin/doC2S.sh @ 148:f0bee28995f1
do the work for cdx2sql
| author | Henry S. Thompson <ht@inf.ed.ac.uk> |
|---|---|
| date | Mon, 25 Oct 2021 15:05:46 +0000 |
| parents | |
| children | 4e1364b8c89a |
comparison
equal
deleted
inserted
replaced
| 147:11d973ecff4e | 148:f0bee28995f1 |
|---|---|
| 1 #!/usr/bin/bash | |
| 2 # Usage: doC2S.sh node task cc resdir workd i | |
| 3 node=$1 | |
| 4 task=$2 | |
| 5 cc=$3 | |
| 6 resdir=$4 | |
| 7 workd=$5 | |
| 8 i=$6 | |
| 9 | |
| 10 echo "> $node.$task: $i" | |
| 11 rm -f $workd/cdx$i.db | |
| 12 cdx2sql.py /beegfs/common_crawl/$cc/cdx/warc $i 2>$workd/cdx$i.errs | \ | |
| 13 sqlite3 $workd/cdx$i.db '.read results/cdx.sql' '.mode tabs' '.import /dev/stdin props' '.quit' 2>$workd/cdx$i.log | |
| 14 echo "< $node.$task: $i"; | |
| 15 | |
| 16 |