Mercurial > hg > cc > azure
annotate workers/bin/count1.sh @ 10:2fbefb8d1a9e
wrun.sh: usage catchup
invoke.sh: force terminal allocation on workers
test1.sh: support control of number of worker processes are spawned,
support -t to turn off random delay at startup
count1.sh: actual do the counting in subprocs to avoid disk contention
author | Henry S. Thompson <ht@markup.co.uk> |
---|---|
date | Mon, 08 Oct 2018 13:17:23 +0000 |
parents | 5db6015689a2 |
children | 36b5d379909a |
rev | line source |
---|---|
8
5db6015689a2
slightly updated, 4 W only, versions of old test4.sh and subcount.sh
Henry S. Thompson <ht@markup.co.uk>
parents:
diff
changeset
|
1 #!/bin/bash |
5db6015689a2
slightly updated, 4 W only, versions of old test4.sh and subcount.sh
Henry S. Thompson <ht@markup.co.uk>
parents:
diff
changeset
|
2 echo "# $ID" |
10 | 3 jq '.Envelope|.["WARC-Header-Metadata"]["WARC-Target-URI"]'|cut -f 1 -d ':'|awk '{c[$1]+=1} END {for (k in c) {print k, c[k]}}' |
8
5db6015689a2
slightly updated, 4 W only, versions of old test4.sh and subcount.sh
Henry S. Thompson <ht@markup.co.uk>
parents:
diff
changeset
|
4 |
5db6015689a2
slightly updated, 4 W only, versions of old test4.sh and subcount.sh
Henry S. Thompson <ht@markup.co.uk>
parents:
diff
changeset
|
5 |
5db6015689a2
slightly updated, 4 W only, versions of old test4.sh and subcount.sh
Henry S. Thompson <ht@markup.co.uk>
parents:
diff
changeset
|
6 |
5db6015689a2
slightly updated, 4 W only, versions of old test4.sh and subcount.sh
Henry S. Thompson <ht@markup.co.uk>
parents:
diff
changeset
|
7 |