Mercurial > hg > cc > cirrus_work

annotate bin/do_idx.sh @ 115:0b1e6e134aca

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
robotstxt and crawldiagnostics get free ride, get rid of DFQ and xq, big simplification and refactor as a result, fix bug in date stream eof handling
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Tue, 26 Sep 2023 17:42:57 +0100
parents a0ea1e4a714d
children 5b952d16838c
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
101
e2e64c3d763e bug4 fixed, but that created a new, earlier bug
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
1 #!/bin/bash
110
a0ea1e4a714d pass in debug flag(s) to merge_date.py
Henry Thompson <ht@markup.co.uk>
parents: 105
diff changeset
2 while [ "$1" = "-d" ]
a0ea1e4a714d pass in debug flag(s) to merge_date.py
Henry Thompson <ht@markup.co.uk>
parents: 105
diff changeset
3 do
a0ea1e4a714d pass in debug flag(s) to merge_date.py
Henry Thompson <ht@markup.co.uk>
parents: 105
diff changeset
4 shift
a0ea1e4a714d pass in debug flag(s) to merge_date.py
Henry Thompson <ht@markup.co.uk>
parents: 105
diff changeset
5 debug="$debug -d"
a0ea1e4a714d pass in debug flag(s) to merge_date.py
Henry Thompson <ht@markup.co.uk>
parents: 105
diff changeset
6 done
101
e2e64c3d763e bug4 fixed, but that created a new, earlier bug
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
7 export res="$1"
e2e64c3d763e bug4 fixed, but that created a new, earlier bug
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
8 orig="$2"
105
9403c02d5034 switch to gzip -7 to get comparable compressed cdx block size
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 103
diff changeset
9 # igzip was faster, but produced bigger files, so went to gzip one step
9403c02d5034 switch to gzip -7 to get comparable compressed cdx block size
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 103
diff changeset
10 # smaller than default (-6), which produces slightly _smaller_ blocks.
110
a0ea1e4a714d pass in debug flag(s) to merge_date.py
Henry Thompson <ht@markup.co.uk>
parents: 105
diff changeset
11 merge_date.py $debug <(LC_ALL=C sort -m -k1,2 -s $res/ks_[0-9]*.tsv) $orig $res/idx |\
101
e2e64c3d763e bug4 fixed, but that created a new, earlier bug
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
12 parallel -j 10 'echo {#} {} >$res/merge_{#}.log
e2e64c3d763e bug4 fixed, but that created a new, earlier bug
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
13 echo $(date) {#} {}
e2e64c3d763e bug4 fixed, but that created a new, earlier bug
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
14 export res
105
9403c02d5034 switch to gzip -7 to get comparable compressed cdx block size
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 103
diff changeset
15 split -l 3000 --filter="gzip -c -7 --keep | \
101
e2e64c3d763e bug4 fixed, but that created a new, earlier bug
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
16 tee >(wc -c >> \
e2e64c3d763e bug4 fixed, but that created a new, earlier bug
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
17 $res/merge_{#}.log)" \
e2e64c3d763e bug4 fixed, but that created a new, earlier bug
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
18 {} > {}.gz && \
e2e64c3d763e bug4 fixed, but that created a new, earlier bug
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
19 rm {}'