Mercurial > hg > cc > azure
view master/wecu/run_sac.sh @ 57:ac1a20e627a9
from lukasz git repo 2020-05-26 (see ~/src/wecu), then editted,
sac not quite working yet
| author | Henry S. Thompson <ht@markup.co.uk> |
|---|---|
| date | Wed, 27 May 2020 20:54:34 +0000 |
| parents | |
| children |
line wrap: on
line source
#!/bin/bash cores=$1 hosts=$2 wd=$3 shift shift shift rm -f allout parallel -v \ --sshloginfile $hosts \ --retries 3 \ --transferfile $(which sac_mapper.py|sed 's/sac_/.\/sac_/') \ --will-cite \ --jobs $cores \ --workdir $wd \ -a input_paths \ "hostname 1>&2 ; export PYTHONIOENCODING=utf-8; curl -s -N https://commoncrawl.s3.amazonaws.com/{} | unpigz -dp 1 -c | tee >(wc -l 1>&2) | ./sac_mapper.py $* 2>&1" | tee -a allout | grep -v 'Authorized uses only' | \ sac_reducer.py "$*"