cc/cirrus_work: bin/ex4_1.sh annotate

annotate bin/ex4_1.sh @ 385:615efac7fc09 plus

working for ex4a

author	Henry S. Thompson <ht@inf.ed.ac.uk>
date	Mon, 15 Jun 2026 23:31:47 +0100
parents
children

rev	line source
385 615efac7fc09 working for ex4a Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	1 #!/usr/bin/bash
615efac7fc09 working for ex4a Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	2 # Usage: ex4_1.sh j n
615efac7fc09 working for ex4a Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	3 # Process the jth group of cdx files taken n at a time
615efac7fc09 working for ex4a Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	4 # with 10 parallel threads at a time
615efac7fc09 working for ex4a Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	5 j=$1
615efac7fc09 working for ex4a Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	6 n=$2
615efac7fc09 working for ex4a Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	7 c_s=$((n * j))
615efac7fc09 working for ex4a Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	8 c_e=$(( ( ( n * (j + 1) ) - 1 ) ))
615efac7fc09 working for ex4a Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	9 echo ex4_1 passed j=$j n=$n, processing "[$c_s,$c_e]"
615efac7fc09 working for ex4a Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	10 cd /work/dc007/dc007/hst/results/CC-MAIN-2025-33/warc_lmhx/idx
615efac7fc09 working for ex4a Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	11 seq --format "%03g" $c_s $c_e \| parallel -j 10 \
615efac7fc09 working for ex4a Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	12 /work/dc007/dc007/hst/bin/ex4_2.sh '{}' \| uniq_merge.py
615efac7fc09 working for ex4a Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	13
615efac7fc09 working for ex4a Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	14