Mercurial > hg > cc > cirrus_work

annotate bin/_atest3.sh @ 378:e44d214699be plus

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
color ls
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Thu, 11 Jun 2026 21:09:38 +0100
parents b3ba45af45c9
children a00e888e4df3
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
377
b3ba45af45c9 prep for exercise 4
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
1 #!/usr/bin/bash
b3ba45af45c9 prep for exercise 4
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
2 # Usage: _atest3.sh j n
b3ba45af45c9 prep for exercise 4
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
3 # Process the jth group of cdx files taken n at a time, doing 10 at a time
b3ba45af45c9 prep for exercise 4
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
4 j=$1
b3ba45af45c9 prep for exercise 4
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
5 n=$2
b3ba45af45c9 prep for exercise 4
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
6 sleep 5
b3ba45af45c9 prep for exercise 4
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
7 h_s=$((n * j))
b3ba45af45c9 prep for exercise 4
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
8 h_e=$(( ( ( n * (j + 1) ) - 1 ) ))
b3ba45af45c9 prep for exercise 4
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
9 echo _a passed j=$j n=$n, processing "[$h_s,$h_e]"
b3ba45af45c9 prep for exercise 4
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
10 #seq --format "%03g" $h_s $h_e | parallel -j 10 echo '$(date)' cdx-00'{}.gz' ' {#} \| sus' #| uniq_merge.py
b3ba45af45c9 prep for exercise 4
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
11 cd /work/dc007/dc007/hst/results/CC-MAIN-2025-33/warc_lmhx/idx
b3ba45af45c9 prep for exercise 4
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
12 seq --format "%03g" $h_s $h_e | parallel -j 10 'uz cdx-00{}.gz | /work/dc007/dc007/shared/bin/cdx2tsv.sh languages | egrep -v "NA|," | sus' | uniq_merge.py