Mercurial > hg > cc > azure

view master/src/wecu/run.sh @ 68:1f04bce6ead7 default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
use basefile instead of transferfile, and remove cleanup: belt and braces wrt lossage of sac_schemes.py in 15% of 1000_k3, this as used in a_2
author Henry S. Thompson <ht@markup.co.uk>
date Thu, 04 Jun 2020 20:44:44 +0000
parents a3edba8dab11
children
line wrap: on
line source

cores=`cat cores.txt`

time parallel \
    --sshloginfile hosts \
    --transferfile mapper.py \
    --transferfile reducer.py \
    --will-cite \
    --retries 3 \
    --jobs $cores \
    --workdir $PWD \
    -a input_paths  \
    'curl -s -N "https://commoncrawl.s3.amazonaws.com/{}" | unpigz -dp 1 -c | ./mapper.py' | \
    sort | \
    ./reducer.py