Mercurial > hg > cc > work

changeset 88:d622b1b5e60d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
shuffle a bit, plan
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Fri, 02 May 2025 13:10:21 +0100
parents 90b9c202922c
children 367dca1afea4
files lurid3/notes.txt
diffstat 1 files changed, 596 insertions(+), 233 deletions(-) [+]
line wrap: on
line diff
--- a/lurid3/notes.txt	Wed Apr 23 18:23:35 2025 +0100
+++ b/lurid3/notes.txt	Fri May 02 13:10:21 2025 +0100
@@ -357,6 +357,234 @@
   >: shuf /tmp/hst/headers.txt > src/test/org/commoncrawl/util/headers_mixed.txt
 
 Won't quite work :-(
+==========[end of that line of work?==========
+At some point the following was explored:
+=======<Overtaken, probably>============================
+Try it with the existing _per segment_ index we have for 2019-35
+
+Assuming we have to key on segment / file and offset, as reconstructing the
+proper index key is such a pain / buggy / is going to change with the year.
+
+Stay with segment 49
+
+  >: uz cdx.gz |wc -l
+ 29,870,307
+
+  >: time uz cdx.gz|egrep -ao ' "length": "[0-9]*", "offset": "[0-9]*"' |wc
+  29,870,307 119,481,228 1,241,098,122
+             = 4 * 29,870,307
+
+So no bogons, not _too_ surprising :-)
+
+Bad news is it's a _big_ file:
+
+  >: ls -lh cdx.gz
+  -rw-r--r-- 1 hst dc007 2.0G Mar 18  2021 cdx.gz
+
+So not viable to paste offset as a key and then sort on command line,
+or to load it in to python and do the work there...
+
+Do it per warc file and then merge?
+
+  >: time uz cdx.gz |fgrep -a warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | sort -n -t\" -k28,28 > /tmp/hst/558.warc.cdx
+
+  real  0m23.494s
+  user  0m14.541s
+  sys   0m9.158s
+
+  >: wc -l /tmp/hst/558.warc.cdx
+  53432 /tmp/hst/558.warc.cdx
+
+  >: echo $((600 * 53432))
+  32,059,200
+
+So, 600 of those, plus approx. same again for extracting, that pbly
+_is_ doable in python, not more than 10 hours total, assuming internal
+sort and external merge is not too expensive...
+
+For each segment, suppose we pull out 60 groups of 10 target files
+  >: time uz cdx.gz |egrep -a warc/CC-MAIN-2019[^-]*-2019[^-]*-0000..warc.gz > /tmp/hst/0000.warc.cdx
+
+  real  0m42.129s
+  user  0m35.147s
+  sys   0m9.140s
+  >: wc -l /tmp/hst/0000.warc.cdx
+  533150
+
+Key it with offset and sort:
+
+  >: time egrep -ao ' "length": "[0-9]*", "offset": "[0-9]*"' /tmp/hst/0000.warc.cdx | cut -f 5 -d ' ' | tr -d \"  > /tmp/hst/0000_offsets
+
+  real  0m5.578s
+  user  0m5.593s
+  sys   0m0.265s
+
+  >: time paste /tmp/hst/0000_offsets /tmp/hst/0000.warc.cdx |sort -nk1,1 | cut -f 2 > /tmp/hst/0000_sorted.warc.cdx
+
+  real  0m4.185s
+  user  0m2.001s
+  sys   0m1.334s
+
+  >: time seq 0 9 | parallel -j 10 "~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-*-*-0000'{}'.warc.gz > /tmp/hst/lm_0000'{}'.tsv"
+
+  real  0m24.610s
+  user  2m54.146s
+  sys   0m10.226s
+
+  >: head /tmp/hst/lm_00000.tsv
+  9398  16432     Mon, 19 Aug 2019 02:44:15 GMT
+  20796 26748     Tue, 16 Jul 2019 04:39:09 GMT
+  4648  340633    Fri, 07 Dec 2018 09:05:59 GMT
+  3465  357109    Sun, 18 Aug 2019 11:48:23 GMT
+  7450  914189    Mon, 19 Aug 2019 02:50:08 GMT
+  ...
+  sing<3956>: fgrep '"length": "9398", "offset": "16432"' /tmp/hst/0000_sorted.warc.cdx
+  com,roommeme,0401a)/index.phtml?channel=&op=&p=140&put=show&r2= 20190819024416 {"url": "http://0401a.roommeme.com/index.phtml?PUT=SHOW&R2=&OP=&P=140&CHANNEL=", "mime": "text/html", "mime-detected": "text/html", "status": "200", "digest": "5DNDVX5HQBOOBHISSCOI4UBVMUL63L36", "length": "9398", "offset": "16432", "filename": "crawl-data/CC-MAIN-2019-35/segments/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00000.warc.gz", "charset": "Big5", "languages": "zho"}
+
+bingo
+
+So, the python code is pretty straightfoward: open the 10 individual
+lm-*.tsv outputs into an array, initialise a 10-elt array with the
+first line of each and another with its offset, record the
+fileno(s) of the lowest offset, then iterate
+
+  read cdx lines and write unchanged until offset = lowest
+  merge line from fileno and output
+  remove fileno from list of matches
+  read and store a new line for fileno [handle EOF]
+  if list of matches is empty, redo setting of lowest
+
+Resort the result by actual key
+
+Meanwhile, get a whole test set:
+sbatch --output=slurm_aug_cdx_49_10-599-out --time=01:00:00 --ntasks=10 -c 36 --exclusive $HOME/bin/runme.sh -m 00 59 $PWD -t 18 -b 'export resdir=CC-MAIN-2019-35/aug_cdx/49
+export DEC=$xarg' "export PYTHONPATH=./lib/python/cc:$PYTHONPATH
+seq 0 9 | parallel -j 10 \"~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-*-*-00\${DEC}'{}'.warc.gz > \$resdir/00\${DEC}'{}'.tsv\""
+
+Actually finished 360 in the hour.
+
+Leaving
+
+sbatch --output=slurm_aug_cdx_49_360-599-out --time=01:00:00 --ntasks=10 -c 36 --exclusive $HOME/bin/runme.sh -m 36 59 $PWD -t 18 -b 'export resdir=CC-MAIN-2019-35/aug_cdx/49
+export DEC=$xarg' "export PYTHONPATH=./lib/python/cc:$PYTHONPATH
+seq 0 9 | parallel -j 10 \"~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-*-*-00\${DEC}'{}'.warc.gz > \$resdir/00\${DEC}'{}'.tsv\""
+
+But something is wrong, the number of jobs is all wrong:
+  
+  5>: fgrep -c parallel slurm_aug_cdx_49_0-359-out
+  741
+  sing<4046>: ls -lt CC-MAIN-2019-35/aug_cdx/49/|wc -l
+  372
+
+Every file is being produced twice.
+
+Took me a while to figure out my own code :-(
+
+  >: sbatch --output=slurm_aug_cdx_49_360-599-out --time=01:00:00 --ntasks=10 -c 36 --exclusive $HOME/bin/runme.sh -m 49 49 $PWD -t 18 -b 'export resdir=CC-MAIN-2019-35/aug_cdx/$xarg
+  export SEG=$xarg
+  share_by_task.sh -f "%03g\n" -s 360 599 $n $task > /tmp/hst_$task' -i 'cat /tmp/hst_$task' 'export PYTHONPATH=./lib/python/cc:$PYTHONPATH
+   ~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/*.$SEG/orig/warc/CC-MAIN-*-*-00${arg}.warc.gz > $resdir/00${arg}.tsv'
+
+Oops, only 560, not 600
+
+Took 3.5 minutes for 200, so call it 10 for 560, so do 6 more in an
+hour:
+
+  >: sbatch --output=slurm_aug_cdx_50-55_out --time=01:00:00 --ntasks=10 -c 36 --exclusive $HOME/bin/runme.sh -m 50 55 $PWD -t 18 -b 'export resdir=CC-MAIN-2019-35/aug_cdx/$xarg
+mkdir -p $resdir
+> export SEG=$xarg
+share_by_task.sh -f "%03g\n" -s 360 599 $n $task > /tmp/hst_$task' -i 'cat /tmp/hst_$task' 'export PYTHONPATH=./lib/python/cc:$PYTHONPATH
+ ~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/*.$SEG/orig/warc/CC-MAIN-*-*-00${arg}.warc.gz > $resdir/00${arg}.tsv'
+  
+  >: tail slurm_aug_cdx_50-55_out
+  ...
+  Wed Oct 9 22:25:47 BST 2024 Finished 55
+  >: head -1 slurm_aug_cdx_50-55_out
+  Wed Oct 9 21:29:43 BST
+               56:04
+
+  >: du -s CC-MAIN-2019-35/aug_cdx
+  1,902,916
+  
+Not bad, so order 20MB for the whole thing
+
+Next step, compare to my existing cdx with timestamp
+
+First check looks about right:
+
+  [cd .../warc_lmhx]
+  >: seq --format='%03g' 0 299 > /tmp/hst/cdx_nums
+  >: parallel -j 20 -a /tmp/hst/cdx_nums 'uz idx/cdx-00{}.gz | egrep -o "\"filename\": \"crawl-data/CC-MAIN-2019-35/segments/[^.]*[.]50.*\"lastmod\":" | sed "s/^.*-00//;s/^\(...\).*/\1/"| sus > /tmp/hst/checkseg_50_{}'
+
+  [cd .../aug_cdx/50]
+  >: wc -l 00123.tsv
+  9333
+  >: egrep -h '123$' /tmp/hst/checkseg_50_??? | acut 1 | btot
+  9300
+  >: wc -l 00400.tsv
+  9477 00400.tsv
+  >: egrep -h '400$' /tmp/hst/checkseg_50_??? | acut 1 | btot
+  9439
+
+Difference is presumable the bogus timestamps aren't in the augmented
+cdx as shipped.
+
+Note that the following 'bad' kind of timestamp is fixed before
+sort_date.py does its thing:
+
+   ... sort_date.sh <(uz $arg/*00???.warc.gz | '"fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/')"' >$arg/ks.tsv
+  
+
+  >: egrep -c '[^ ]GMT$' 50/00123.tsv
+  22
+  >: egrep -c '[^ ]GMT$' 50/00400.tsv
+  14
+
+  >: PYTHONPATH=~/.local/lib/python3.9/site-packages:$PYTHONPATH sort_date.sh <(uz ../warc_lmhx/50/*00123.warc.gz | fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2> /tmp/hst/123_errs | wc -l
+  9300
+  >: fgrep -c Invalid /tmp/hst/123_errs
+  33
+  >: PYTHONPATH=~/.local/lib/python3.9/site-packages:$PYTHONPATH sort_date.sh <(uz ../warc_lmhx/50/*00400.warc.gz | fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2> /tmp/hst/400_errs | wc -l
+  9439
+  >: fgrep -c Invalid /tmp/hst/400_errs
+  38
+
+All good.
+
+But
+  >: seq --format='%03g' 0 559 > /tmp/hst/warc_nums
+  >: xx () {
+  r=$(diff -bw
+   <(echo $((
+    $(sort_date.sh <(uz ../warc_lmhx/50/*00$1.warc.gz |
+      fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2>/tmp/hst/ec_$1 |wc -l)
+       +
+    $(fgrep -c Invalid /tmp/hst/ec_$1))))
+  <(wc -l < 50/00$1.tsv))
+ if [ "$r" ]
+ then printf "%s:\n%s\n" $2 "$r"
+ fi
+ }
+  >: parallel -j 20 -a /tmp/hst/warc_nums xx '{}'  '$(({#} - 1))'  | tee /tmp/hst/aug_bugs
+  >: fgrep -c 1c1 /tmp/hst/aug_bugs
+  77
+  >: wc -l < /tmp/hst/aug_bugs
+  385
+  >: echo $((77 * 5))
+  385
+
+OK, there are a few other error messages from date conversion
+   >: xx () { r=$(diff -bw <(echo $(($(sort_date.sh <(uz ../warc_lmhx/50/*00$1.warc.gz | fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2>/tmp/hst/ec_$1 |wc -l) + $(egrep -c 'Invalid|must be in|out of range' /tmp/hst/ec_$1)))) <(wc -l < 50/00$1.tsv)); if [ "$r" ]; then  printf "%s:\n%s\n" $2 "$r"; fi; }
+  >: parallel -j 20 -a /tmp/hst/warc_nums xx '{}'  '$(({#} - 1))'  | tee /tmp/hst/aug_bugs2
+   [nothing]
+
+So, I think we can believe we're OK
+But 7 is better than 1:
+  >: xx () { r=$(diff -bw <(echo $(($(sort_date.sh <(uz ../warc_lmhx/$3/*00$1.warc.gz | fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2>/tmp/hst/ec_$1 |wc -l) + $(egrep -c 'Invalid|must be in|out of range' /tmp/hst/ec_$1)))) <(wc -l < $3/00$1.tsv)); if [ "$r" ]; then  printf "%s:\n%s\n" $2 "$r"; fi; }
+  >: for s in 49 {51..55}; do parallel -j 20 -a /tmp/hst/warc_nums xx '{}'  '$(({#} - 1))' $s  | tee /tmp/hst/aug_bugs_$s; done
+  [nothing]
+===========</Overtaken>==============
+
 How do We reconstruct the Warc filename, offset and length from the
 original index?
 
@@ -594,6 +822,8 @@
 Tried using FileIO.readinto([a fixed buffer]), but didn't immediately
 work. Abandoned because I still don't understand how zlib.decompress
 works at all...
+============Abandon the use of ix.py in order to work through index
+files using ix.py to find the corresponding headers and find LM there or not=========
 
 Time to convert unpackz to a library which takes a callback
 alternative to an output file -- Done
@@ -2831,9 +3061,10 @@
   ./rerun/4/a_w2c_errs    75      43572 Apr 2 12:48
 Next steps:
  Check robots.txt for LM headers: Yes
- Fetched robotstxt for 2019-35
+ Fetched robotstxt for 2019-35 Done
  Ran warc2cdb on 2019-35 robotstxt, crawldiagnostics (after fixing
-  bug) and warc:
+  bug (NOT CLEAR why I added " need to fill this in some how" to the
+       lmh.cdb_in value for revisit records or what it means!) and warc:
   >: time ~/lib/python/cc/w2c.py 2019-35 0 . crawldiagnostics 2> 0/crawldiagnostics/w2c_errs
 
   ./0/crawldiagnostics/lmh.cdb_in
@@ -2921,14 +3152,6 @@
 	1 year 641778 is out of range
 All good
 
- Fetch c_d and, if needed r_t for 2023-40 - DONE
- 
- Remove all w2c_errs and ldb.cdb_in from *** downwards
- Run a by-subparts batch of warc2cdb for warc, r_t and c_d for 2019-35
- Do whatever checks are possible on the results
- Fill in r_t? and c_d ldb.cdb_in for 0 1 2 4 11 12 15
- Take stock: more 2023-40, or back to 2019-35 to add c_d (and r_t)?
-
 This didn't work:
   >: sbatch --output=slurm_1-10x5.out --time=02:00:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh -m 1 10 $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh
 j=$((i - 1))
@@ -3094,7 +3317,7 @@
   112
   112
   >: headtail -1 slurm_0-9x5w.out
-  Wed Apr 23 17:00:25 BST 2025 Launching 5 nodes in /mnt/lustre/e1000/home/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx for runme /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx -d -t 3 -i cat ranges export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh
+  Wed Apr 23 17:00:25 BST 2025 Launching 5 nodes ...
   ===
   Wed Apr 23 17:49:15 BST 2025 Finished
   >: python3 -c 'import sys,datetime;(s,d1,d2)=sys.argv[1:4];dd2=datetime.datetime.strptime(d2[4:22],"%b %d %I:%M:%S %p");dd1=datetime.datetime.strptime(d1[4:19],"%b %d %H:%M:%S");d=(dd2-dd1).seconds;h=d/3600;print(a:=int(s)/50.0,"%d:%0.2d:%0.2d"%(h,(d:=(d%3600))/60,d % 60))' $(fgrep -ch beegfs x?/?/warc/w* | btot) 'Wed Apr 23 17:00:25 BST 2025' "Wed Apr 23 05:49:15 PM"
@@ -3103,231 +3326,371 @@
 All good
 
 Nothing but the same utf-8 mismatches in e.g. segment 5:
-  >: diff -bw <(cut -f 1 5/ks_errs.tsv) <(cat x{0..4}/5/warc/w2c_errs | egrep -v 'beegfs|records|headers' |cut -f 1|sed "s/^b'\(.*\)'$/\1/")
-================
-
-Try it with the existing _per segment_ index we have for 2019-35
-
-Assuming we have to key on segment / file and offset, as reconstructing the
-proper index key is such a pain / buggy / is going to change with the year.
-
-Stay with segment 49
-
-  >: uz cdx.gz |wc -l
- 29,870,307
-
-  >: time uz cdx.gz|egrep -ao ' "length": "[0-9]*", "offset": "[0-9]*"' |wc
-  29,870,307 119,481,228 1,241,098,122
-             = 4 * 29,870,307
-
-So no bogons, not _too_ surprising :-)
-
-Bad news is it's a _big_ file:
-
-  >: ls -lh cdx.gz
-  -rw-r--r-- 1 hst dc007 2.0G Mar 18  2021 cdx.gz
-
-So not viable to paste offset as a key and then sort on command line,
-or to load it in to python and do the work there...
-
-Do it per warc file and then merge?
-
-  >: time uz cdx.gz |fgrep -a warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | sort -n -t\" -k28,28 > /tmp/hst/558.warc.cdx
-
-  real  0m23.494s
-  user  0m14.541s
-  sys   0m9.158s
-
-  >: wc -l /tmp/hst/558.warc.cdx
-  53432 /tmp/hst/558.warc.cdx
-
-  >: echo $((600 * 53432))
-  32,059,200
-
-So, 600 of those, plus approx. same again for extracting, that pbly
-_is_ doable in python, not more than 10 hours total, assuming internal
-sort and external merge is not too expensive...
-
-For each segment, suppose we pull out 60 groups of 10 target files
-  >: time uz cdx.gz |egrep -a warc/CC-MAIN-2019[^-]*-2019[^-]*-0000..warc.gz > /tmp/hst/0000.warc.cdx
-
-  real  0m42.129s
-  user  0m35.147s
-  sys   0m9.140s
-  >: wc -l /tmp/hst/0000.warc.cdx
-  533150
-
-Key it with offset and sort:
-
-  >: time egrep -ao ' "length": "[0-9]*", "offset": "[0-9]*"' /tmp/hst/0000.warc.cdx | cut -f 5 -d ' ' | tr -d \"  > /tmp/hst/0000_offsets
-
-  real  0m5.578s
-  user  0m5.593s
-  sys   0m0.265s
-
-  >: time paste /tmp/hst/0000_offsets /tmp/hst/0000.warc.cdx |sort -nk1,1 | cut -f 2 > /tmp/hst/0000_sorted.warc.cdx
-
-  real  0m4.185s
-  user  0m2.001s
-  sys   0m1.334s
-
-  >: time seq 0 9 | parallel -j 10 "~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-*-*-0000'{}'.warc.gz > /tmp/hst/lm_0000'{}'.tsv"
-
-  real  0m24.610s
-  user  2m54.146s
-  sys   0m10.226s
-
-  >: head /tmp/hst/lm_00000.tsv
-  9398  16432     Mon, 19 Aug 2019 02:44:15 GMT
-  20796 26748     Tue, 16 Jul 2019 04:39:09 GMT
-  4648  340633    Fri, 07 Dec 2018 09:05:59 GMT
-  3465  357109    Sun, 18 Aug 2019 11:48:23 GMT
-  7450  914189    Mon, 19 Aug 2019 02:50:08 GMT
+  >: diff -bw <(cut -f 1 5/ks_errs.tsv) <(cat x{0..4}/5/warc/w2c_errs
+| egrep -v 'beegfs|records|headers' |cut -f 1|sed "s/^b'\(.*\)'$/\1/")
+
+And lmh.cdb_in is good:
+  >: diff -bw <(cut -f2- 5/ks.tsv| fgrep $'\t'| sed 's/  //;s/\.0$//'|sort -k1,1) <(sed 's/^\+[0-9,]*://;s/->\([0-9-]*\)$/ \1/' x{0..4}/5/warc/lmh.cdb_in | sort -k1,1)| tee /tmp/hst/5diff
+  >: cat /tmp/hst/5diff
+  0a1,5
+  >
+  >
+  >
+  >
+  >
+
+Next decile:
+
+  >: sbatch --output=slurm_10-19x5w.out --time=01:20:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh
+j=$((i - 1)) ; tp=$((task + 10))
+mkdir -p ./x$j/$tp/warc
+~/lib/python/cc/w2c.py 2019-35 $tp ./x$j warc "$arg"  2> ./x$j/$tp/warc/w2c_errs'
+  >: head slurm_10-19x5w.out
+  Thu Apr 24 10:34:12 BST 2025 Launching 5 nodes
+  >: tail -1 slurm_10-19x5w.out
+  Thu Apr 24 11:24:13 BST 2025 Finished
+  >: wprog.sh 'Thu Apr 24 10:38:13 AM BST 2025' 'Thu Apr 24 11:24:13 AM'
+  112.00 0:46:00 146.09
+
+Checked 15/.../w2c_errs vs 15/ks_errs.tsv, and likewise lmh.cdb_in, ok
+
+Fill in r_t and c_d
+  >: for ty in robotstxt crawldiagnostics; do sbatch --output=slurm_10-19x5${ty:1}.out --time=00:20:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh
+  j=$((i - 1)) ; tp=$((task + 10)) ; ty='$ty'
+  mkdir -p ./x$j/$tp/$ty
+  ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg"  2>./x$j/$tp/$ty/w2c_errs'; done
+  >: headtail -1 slurm_10-19x5obotstxt.out
+  Thu Apr 24 12:39:46 BST 2025 Launching 5 nodes ...
+  ===
+  Thu Apr 24 12:40:33 BST 2025 Finished
+  >: headtail -1 slurm_10-19x5rawldiagnostics.out
+  Thu Apr 24 12:38:47 BST 2025 Launching 5 nodes ...
+  ===
+  Thu Apr 24 12:41:22 BST 2025 Finished
+  >: wprog.sh 'Thu Apr 24 12:39:46 PM BST 2025' 'Thu Apr 24 12:40:33 PM' robotstxt
+  112.00 0:00:47 8578.72
+  >: wprog.sh 'Thu Apr 24 12:38:47 PM BST 2025' 'Thu Apr 24 12:41:22 PM' crawldiagnostics
+
+Oops:
+  >: lss x?/1?/robotstxt/w2c_errs
+  -rw-r--r-- 1 hst dc007 28753 Apr 24 12:40 x1/10/robotstxt/w2c_errs
+  -rw-r--r-- 1 hst dc007 28008 Apr 24 12:40 x2/10/robotstxt/w2c_errs
+  -rw-r--r-- 1 hst dc007 27974 Apr 24 12:40 x0/10/robotstxt/w2c_errs
+  -rw-r--r-- 1 hst dc007 27613 Apr 24 12:40 x4/10/robotstxt/w2c_errs
+  -rw-r--r-- 1 hst dc007 26894 Apr 24 12:40 x3/10/robotstxt/w2c_errs
+  -rw-r--r-- 1 hst dc007 13328 Apr 24 12:40 x1/11/robotstxt/w2c_errs
+  -rw-r--r-- 1 hst dc007 13328 Apr 24 12:40 x1/12/robotstxt/w2c_errs
+  -rw-r--r-- 1 hst dc007 13328 Apr 24 12:40 x1/13/robotstxt/w2c_errs
+  -rw-r--r-- 1 hst dc007 13328 Apr 24 12:40 x1/14/robotstxt/w2c_errs
+  ....
+  >: ls -d /beegfs/common_crawl/CC-MAIN-2019-35/*/orig/robotstxt/|wc -l
+  11
+
+But c_d looks ok
+
+  >: ls -d /beegfs/common_crawl/CC-MAIN-2019-35/*/orig/crawldiagnostics/|wc -l
+  100
+  >: fgrep -c records x?/19/{robotstxt,crawldiagnostics}/w2c_errs
+  x0/19/robotstxt/w2c_errs:0
+  x1/19/robotstxt/w2c_errs:0
+  x2/19/robotstxt/w2c_errs:0
+  x3/19/robotstxt/w2c_errs:0
+  x4/19/robotstxt/w2c_errs:0
+  x0/19/crawldiagnostics/w2c_errs:112
+  x1/19/crawldiagnostics/w2c_errs:112
+  x2/19/crawldiagnostics/w2c_errs:112
+  x3/19/crawldiagnostics/w2c_errs:112
+  x4/19/crawldiagnostics/w2c_errs:112
+
+Try all three for the next decile:
+  >: for ty in warc robotstxt crawldiagnostics; do sbatch --output=slurm_20-29x5${ty::1}.out --time=01:00:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh
+  j=$((i - 1)) ; tp=$((task + 20)) ; ty='$ty'
+  mkdir -p ./x$j/$tp/$ty
+  ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg"  2> ./x$j/$tp/$ty/w2c_errs'; done
+
+Started this prematurely before discovering the robotstxt were missing
+-- weirdly warc didn't seem to run...
+c_d ran fine:
+  >: headtail -1 slurm_20-29x5c.out 
+  Thu Apr 24 13:09:47 BST 2025 Launching 5 nodes ...
+  ===
+  Thu Apr 24 13:12:32 BST 2025 Finished
+  >: find . -type f -mtime -1 -name w2c_errs | xargs ls -lt | head
+  -rw-r--r-- 1 hst dc007  31051 Apr 24 13:12 ./x2/23/crawldiagnostics/w2c_errs
+  -rw-r--r-- 1 hst dc007  30541 Apr 24 13:12 ./x0/23/crawldiagnostics/w2c_errs
+  -rw-r--r-- 1 hst dc007  31139 Apr 24 13:12 ./x3/23/crawldiagnostics/w2c_errs
+  -rw-r--r-- 1 hst dc007  29927 Apr 24 13:12 ./x3/25/crawldiagnostics/w2c_errs
+  >: fgrep -c records x?/29/cra*/w2*
+  x0/29/crawldiagnostics/w2c_errs:112
+  x1/29/crawldiagnostics/w2c_errs:112
+  x2/29/crawldiagnostics/w2c_errs:112
+  x3/29/crawldiagnostics/w2c_errs:112
+  x4/29/crawldiagnostics/w2c_errs:112
+
+Fetching the missing r_t:
+  seq 11 99 | /usr/bin/parallel -j 16 time ./getrt.aws CC-MAIN-2019-35 '{}'
   ...
-  sing<3956>: fgrep '"length": "9398", "offset": "16432"' /tmp/hst/0000_sorted.warc.cdx
-  com,roommeme,0401a)/index.phtml?channel=&op=&p=140&put=show&r2= 20190819024416 {"url": "http://0401a.roommeme.com/index.phtml?PUT=SHOW&R2=&OP=&P=140&CHANNEL=", "mime": "text/html", "mime-detected": "text/html", "status": "200", "digest": "5DNDVX5HQBOOBHISSCOI4UBVMUL63L36", "length": "9398", "offset": "16432", "filename": "crawl-data/CC-MAIN-2019-35/segments/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00000.warc.gz", "charset": "Big5", "languages": "zho"}
-
-bingo
-
-So, the python code is pretty straightfoward: open the 10 individual
-lm-*.tsv outputs into an array, initialise a 10-elt array with the
-first line of each and another with its offset, record the
-fileno(s) of the lowest offset, then iterate
-
-  read cdx lines and write unchanged until offset = lowest
-  merge line from fileno and output
-  remove fileno from list of matches
-  read and store a new line for fileno [handle EOF]
-  if list of matches is empty, redo setting of lowest
-
-Resort the result by actual key
-
-Meanwhile, get a whole test set:
-sbatch --output=slurm_aug_cdx_49_10-599-out --time=01:00:00 --ntasks=10 -c 36 --exclusive $HOME/bin/runme.sh -m 00 59 $PWD -t 18 -b 'export resdir=CC-MAIN-2019-35/aug_cdx/49
-export DEC=$xarg' "export PYTHONPATH=./lib/python/cc:$PYTHONPATH
-seq 0 9 | parallel -j 10 \"~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-*-*-00\${DEC}'{}'.warc.gz > \$resdir/00\${DEC}'{}'.tsv\""
-
-Actually finished 360 in the hour.
-
-Leaving
-
-sbatch --output=slurm_aug_cdx_49_360-599-out --time=01:00:00 --ntasks=10 -c 36 --exclusive $HOME/bin/runme.sh -m 36 59 $PWD -t 18 -b 'export resdir=CC-MAIN-2019-35/aug_cdx/49
-export DEC=$xarg' "export PYTHONPATH=./lib/python/cc:$PYTHONPATH
-seq 0 9 | parallel -j 10 \"~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-*-*-00\${DEC}'{}'.warc.gz > \$resdir/00\${DEC}'{}'.tsv\""
-
-But something is wrong, the number of jobs is all wrong:
-  
-  5>: fgrep -c parallel slurm_aug_cdx_49_0-359-out
-  741
-  sing<4046>: ls -lt CC-MAIN-2019-35/aug_cdx/49/|wc -l
-  372
-
-Every file is being produced twice.
-
-Took me a while to figure out my own code :-(
-
-  >: sbatch --output=slurm_aug_cdx_49_360-599-out --time=01:00:00 --ntasks=10 -c 36 --exclusive $HOME/bin/runme.sh -m 49 49 $PWD -t 18 -b 'export resdir=CC-MAIN-2019-35/aug_cdx/$xarg
-  export SEG=$xarg
-  share_by_task.sh -f "%03g\n" -s 360 599 $n $task > /tmp/hst_$task' -i 'cat /tmp/hst_$task' 'export PYTHONPATH=./lib/python/cc:$PYTHONPATH
-   ~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/*.$SEG/orig/warc/CC-MAIN-*-*-00${arg}.warc.gz > $resdir/00${arg}.tsv'
-
-Oops, only 560, not 600
-
-Took 3.5 minutes for 200, so call it 10 for 560, so do 6 more in an
-hour:
-
-  >: sbatch --output=slurm_aug_cdx_50-55_out --time=01:00:00 --ntasks=10 -c 36 --exclusive $HOME/bin/runme.sh -m 50 55 $PWD -t 18 -b 'export resdir=CC-MAIN-2019-35/aug_cdx/$xarg
-mkdir -p $resdir
-> export SEG=$xarg
-share_by_task.sh -f "%03g\n" -s 360 599 $n $task > /tmp/hst_$task' -i 'cat /tmp/hst_$task' 'export PYTHONPATH=./lib/python/cc:$PYTHONPATH
- ~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/*.$SEG/orig/warc/CC-MAIN-*-*-00${arg}.warc.gz > $resdir/00${arg}.tsv'
-  
-  >: tail slurm_aug_cdx_50-55_out
+  real    9m38.739s
+  user    23m8.073s
+  sys     17m36.368s
+
+  real    9m39.145s
+  user    23m13.342s
+  sys     17m45.764s
   ...
-  Wed Oct 9 22:25:47 BST 2024 Finished 55
-  >: head -1 slurm_aug_cdx_50-55_out
-  Wed Oct 9 21:29:43 BST
-               56:04
-
-  >: du -s CC-MAIN-2019-35/aug_cdx
-  1,902,916
-  
-Not bad, so order 20MB for the whole thing
-
-Next step, compare to my existing cdx with timestamp
-
-First check looks about right:
-
-  [cd .../warc_lmhx]
-  >: seq --format='%03g' 0 299 > /tmp/hst/cdx_nums
-  >: parallel -j 20 -a /tmp/hst/cdx_nums 'uz idx/cdx-00{}.gz | egrep -o "\"filename\": \"crawl-data/CC-MAIN-2019-35/segments/[^.]*[.]50.*\"lastmod\":" | sed "s/^.*-00//;s/^\(...\).*/\1/"| sus > /tmp/hst/checkseg_50_{}'
-
-  [cd .../aug_cdx/50]
-  >: wc -l 00123.tsv
-  9333
-  >: egrep -h '123$' /tmp/hst/checkseg_50_??? | acut 1 | btot
-  9300
-  >: wc -l 00400.tsv
-  9477 00400.tsv
-  >: egrep -h '400$' /tmp/hst/checkseg_50_??? | acut 1 | btot
-  9439
-
-Difference is presumable the bogus timestamps aren't in the augmented
-cdx as shipped.
-
-Note that the following 'bad' kind of timestamp is fixed before
-sort_date.py does its thing:
-
-   ... sort_date.sh <(uz $arg/*00???.warc.gz | '"fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/')"' >$arg/ks.tsv
-  
-
-  >: egrep -c '[^ ]GMT$' 50/00123.tsv
-  22
-  >: egrep -c '[^ ]GMT$' 50/00400.tsv
-  14
-
-  >: PYTHONPATH=~/.local/lib/python3.9/site-packages:$PYTHONPATH sort_date.sh <(uz ../warc_lmhx/50/*00123.warc.gz | fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2> /tmp/hst/123_errs | wc -l
-  9300
-  >: fgrep -c Invalid /tmp/hst/123_errs
-  33
-  >: PYTHONPATH=~/.local/lib/python3.9/site-packages:$PYTHONPATH sort_date.sh <(uz ../warc_lmhx/50/*00400.warc.gz | fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2> /tmp/hst/400_errs | wc -l
-  9439
-  >: fgrep -c Invalid /tmp/hst/400_errs
-  38
+
+Something weird:
+  >: wc -l < rerrlog
+  56017
+  >: cat rerrlog |cut -f 9 -d \- |sus
+    101 00415.warc.gz
+    101 00416.warc.gz
+    101 00417.warc.gz
+    101 00418.warc.gz
+    101 00419.warc.gz
+    101 00420.warc.gz
+    101 00421.warc.gz
+    101 00422.warc.gz
+    101 00423.warc.gz
+    101 00424.warc.gz
+    101 00425.warc.gz
+    101 00426.warc.gz
+    101 00427.warc.gz
+    101 00428.warc.gz
+    101 00429.warc.gz
+    101 00430.warc.gz
+    101 00431.warc.gz
+    100 00000.warc.gz
+
+I had indeed started a sequential version of fetching 11..99, which
+got up as far as 415 (I thought).  Maybe a few stragglers came in
+later?  No, they were _logged_ before they were _fetched_, so they
+were logged again by the parallel version, fetched again successfully
+and so dupicated in the log.
+
+And indeed the log begins with 11 x 560 = 6160 lines for segs 0--10,
+then 431 lines for seg 11, then, after a gap of 25 minutes, lines for
+a mixture of segments, eventually picking up segment 11 again at 415.
 
 All good.
 
-But
-  >: seq --format='%03g' 0 559 > /tmp/hst/warc_nums
-  >: xx () {
-  r=$(diff -bw
-   <(echo $((
-    $(sort_date.sh <(uz ../warc_lmhx/50/*00$1.warc.gz |
-      fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2>/tmp/hst/ec_$1 |wc -l)
-       +
-    $(fgrep -c Invalid /tmp/hst/ec_$1))))
-  <(wc -l < 50/00$1.tsv))
- if [ "$r" ]
- then printf "%s:\n%s\n" $2 "$r"
- fi
- }
-  >: parallel -j 20 -a /tmp/hst/warc_nums xx '{}'  '$(({#} - 1))'  | tee /tmp/hst/aug_bugs
-  >: fgrep -c 1c1 /tmp/hst/aug_bugs
-  77
-  sing<4318>: wc -l < /tmp/hst/aug_bugs
-  385
-  sing<4319>: echo $((77 * 5))
-  385
-
-OK, there are a few other error messages from date conversion
-   >: xx () { r=$(diff -bw <(echo $(($(sort_date.sh <(uz ../warc_lmhx/50/*00$1.warc.gz | fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2>/tmp/hst/ec_$1 |wc -l) + $(egrep -c 'Invalid|must be in|out of range' /tmp/hst/ec_$1)))) <(wc -l < 50/00$1.tsv)); if [ "$r" ]; then  printf "%s:\n%s\n" $2 "$r"; fi; }
-sing<4337>: parallel -j 20 -a /tmp/hst/warc_nums xx '{}'  '$(({#} - 1))'  | tee /tmp/hst/aug_bugs2
-   [nothing]
-
-So, I think we can believe we're OK
-But 7 is better than 1:
-  >: xx () { r=$(diff -bw <(echo $(($(sort_date.sh <(uz ../warc_lmhx/$3/*00$1.warc.gz | fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2>/tmp/hst/ec_$1 |wc -l) + $(egrep -c 'Invalid|must be in|out of range' /tmp/hst/ec_$1)))) <(wc -l < $3/00$1.tsv)); if [ "$r" ]; then  printf "%s:\n%s\n" $2 "$r"; fi; }
-  >: for s in 49 {51..55}; do parallel -j 20 -a /tmp/hst/warc_nums xx '{}'  '$(({#} - 1))' $s  | tee /tmp/hst/aug_bugs_$s; done
+warc did run, just several hours later:
+>: headtail -1 slurm_20-29x5w.out 
+Thu Apr 24 15:25:02 BST 2025 Launching 5 nodes ...
+===
+Thu Apr 24 16:13:10 BST 2025 Finished
+
+Better check of error files:
+  >: diff -bw <(cut -f 1 25/ks_errs.tsv|python3 -c 'import sys
+  while c:=sys.stdin.buffer.read(1):
+   if (i:=ord(c)) > 127:
+    sys.stdout.buffer.write(b"\\x%02x"%i)
+   else:
+    sys.stdout.buffer.write(c)
+  ') <(cat x{0..4}/25/warc/w2c_errs| egrep -v 'beegfs|records|headers' |cut -f 1|sed "s/^b.\(.*\).$/\1/")
   [nothing]
-
-Next step: ?
+  >: diff -bw <(cut -f2- 25/ks.tsv| fgrep $'\t'| sed 's/ //;s/\.0$//'|sort -k1,1) <(sed 's/^\+[0-9,]*://;s/->\([0-9-]*\)$/ \1/' x{0..4}/25/warc/lmh.cdb_in | sort -k1,1)
+  0a1,5
+  >
+  >
+  >
+  >
+  >
+
+rerun robots
+  >: for ty in robotstxt; do sbatch --output=slurm_10-29x5${ty::1}.out --time=00:10:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh             
+  j=$((i - 1)) ; tp=$((task + 10)) ; ty='$ty'
+  mkdir -p ./x$j/$tp/$ty
+  ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg"  2> ./x$j/$tp/$ty/w2c_errs'; done
+That didn't do 20, only 10, d'oh
+  >: for ty in robotstxt; do sbatch --output=slurm_20-29x5${ty::1}.out --time=00:10:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh
+j=$((i - 1)) ; tp=$((task + 20)) ; ty='$ty'
+mkdir -p ./x$j/$tp/$ty
+~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg"  2> ./x$j/$tp/$ty/w2c_errs'; done
+
+Looks OK
+
+Next decile
+  >: for ty in warc crawldiagnostics robotstxt; do sbatch --output=slurm_30-39x5${ty::1}.out --time=01:20:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh
+j=$((i - 1)) ; tp=$((task + 30)) ; ty='$ty'
+mkdir -p ./x$j/$tp/$ty
+~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg"  2> ./x$j/$tp/$ty/w2c_errs'; done
+  Submitted batch job 7394434
+  Submitted batch job 7394435
+  Submitted batch job 7394436
+  >: head -1 slurm_30-39x5w.out 
+  Mon Apr 28 11:10:45 BST 2025
+
+Looks like maybe some disk contention...
+>: head -1 slurm_30-39x5w.out 
+  >: date
+  Mon Apr 28 11:20:45 AM BST 2025
+  >: fgrep -c records x4/3?/*/w2c_errs
+  x4/30/crawldiagnostics/w2c_errs:20
+  x4/30/robotstxt/w2c_errs:58
+  x4/30/warc/w2c_errs:2
+  x4/31/crawldiagnostics/w2c_errs:11
+  x4/31/robotstxt/w2c_errs:64
+  x4/31/warc/w2c_errs:1
+  x4/32/crawldiagnostics/w2c_errs:16
+  x4/32/robotstxt/w2c_errs:63
+  x4/32/warc/w2c_errs:4
+  x4/33/crawldiagnostics/w2c_errs:21
+  x4/33/robotstxt/w2c_errs:59
+  x4/33/warc/w2c_errs:0
+  x4/34/crawldiagnostics/w2c_errs:16
+  x4/34/robotstxt/w2c_errs:63
+  x4/34/warc/w2c_errs:0
+  x4/35/crawldiagnostics/w2c_errs:19
+  x4/35/robotstxt/w2c_errs:64
+  x4/35/warc/w2c_errs:1
+  x4/36/crawldiagnostics/w2c_errs:16
+  x4/36/robotstxt/w2c_errs:60
+  x4/36/warc/w2c_errs:4
+  x4/37/crawldiagnostics/w2c_errs:21
+  x4/37/robotstxt/w2c_errs:66
+  x4/37/warc/w2c_errs:1
+  x4/38/crawldiagnostics/w2c_errs:16
+  x4/38/robotstxt/w2c_errs:65
+  x4/38/warc/w2c_errs:0
+  x4/39/crawldiagnostics/w2c_errs:13
+  x4/39/robotstxt/w2c_errs:61
+  x4/39/warc/w2c_errs:0
+
+Yes -- after 10 minutes only r_t is even half-way done...
+We'll see if things speed up as the r_t and c_d finish
+
+  >: headtail -1 slurm_30-39x5r.out
+  Mon Apr 28 11:10:45 BST 2025 Launching 5 nodes ...
+  ===
+  Mon Apr 28 11:29:30 BST 2025 Finished
+  >: headtail -1 slurm_30-39x5c.out
+  Mon Apr 28 11:10:45 BST 2025 Launching 5 nodes ...
+  ===
+  Mon Apr 28 11:49:56 BST 2025 Finished
+
+Maybe a bit, but not enough:
+  >: headtail -1 slurm_30-39x5w.out
+  Mon Apr 28 11:10:45 BST 2025 Launching 5 nodes ...
+  ===
+  slurmstepd: error: *** STEP 7394434.0 ON r1i1n34 CANCELLED AT 2025-04-28T12:30:43 DUE TO TIME LIMIT ***
+
+Reran w. only warc, but looks like still too slow:
+  >: wprog.sh '3?' 'Mon Apr 28 13:34:05 BST 2025'
+  35.36 0:30:29 69.60
+
+Wait a bit longer, doesn't need to speed up much...
+Not even close, in the end:
+  >: wprog.sh '3?' 'Mon Apr 28 13:34:05 BST 2025'
+  74.48 1:19:07 56.48
+Reran w. 02:20
+
+Bother, one glitch, segment 35 is short:
+  >: fgrep -c records x0/35/warc/w2c_errs
+  x0/35/warc/w2c_errs:112
+  x1/35/warc/w2c_errs:112
+  x2/35/warc/w2c_errs:112
+  x3/35/warc/w2c_errs:81
+  x4/35/warc/w2c_errs:112
+
+Maybe we have enough info to fix this:
+  >: egrep -ho ' [^ ]* win,' x?/34/warc/w* | cut -f 2 -d ' ' | btot
+  5258229
+  >: egrep -ho ' [^ ]* win,' x?/35/warc/w* | cut -f 2 -d ' ' | btot
+  4971117
+  >: tail -14 x3/35/warc/w2c_errs
+  /beegfs/common_crawl/CC-MAIN-2019-35/1566027323246.35/orig/warc/CC-MAIN-20190825084751-20190825110751-00417.warc.gz
+  Traceback (most recent call last):
+    File "/work/dc007/dc007/hst/lib/python/cc/w2c.py", line 4, in <module>
+      sys.exit(warc2cdb.main(*sys.argv[1:]))
+	       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+    File "warc2cdb.py", line 105, in lmh.warc2cdb.main
+      with open((outfile_name:="%s/%s/%s/lmh.cdb_in"%(outdir, segment, subdir)),'wb') as OUT:
+    File "warc2cdb.py", line 110, in lmh.warc2cdb.main
+      warc.warc(infile_name,LMHline,[warc.RESP, warc.REVISIT],parts=3)
+    File "warc.py", line 125, in warc.warc
+      _out=callback(wtype,bufView[start_2:eo2+2],2)
+    File "warc2cdb.py", line 81, in lmh.warc2cdb.LMHline
+      l = len(lmi)
+  TypeError: object of type 'int' has no len()
+Excellent -- we should have all of 34, (almost) none of 35
+  >: head -1 x3/35/warc/w2c_errs
+  /beegfs/common_crawl/CC-MAIN-2019-35/1566027323246.35/orig/warc/CC-MAIN-20190825084751-20190825110751-00336.warc.gz
+  >: echo $(((417 - 336) +1))
+  82
+  >: echo $((5258229 / 560))
+  9389
+  >: python3 -c "print(5258229 * ((82.0 + (4 * 112)) / 560))"
+  4976538.16
+Close enough...
+Probably not worth it unless we hit a big set of partial results
+
+More deciles:
+  >: for ty in warc; do sbatch --output=slurm_40-49x5${ty::1}.out --time=02:30:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh                                           
+j=$((i - 1)) ; tp=$((task + 40)) ; ty='$ty'
+mkdir -p ./x$j/$tp/$ty                                                          ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg"  2> ./x$j/$tp/$ty/w2c_errs'; done
+  >: for ty in robotstxt crawldiagnostics; do sbatch --output=slurm_40-49x5${ty::1}.out --time=00:15:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh                  
+j=$((i - 1)) ; tp=$((task + 40)) ; ty='$ty'
+mkdir -p ./x$j/$tp/$ty                                                          ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg"  2> ./x$j/$tp/$ty/w2c_errs'; done
+  >: wprog.sh '4?' 'Tue Apr 29 09:43:06 BST 2025' robotstxt
+  112.00 3:00:11 ...
+  >: wprog.sh '4?' 'Tue Apr 29 09:43:06 BST 2025' crawldiagnostics
+  112.00 3:00:20 ...
+  >: headtail -1 slurm_40-49x5w.out
+  Tue Apr 29 09:43:06 BST 2025 ...
+  ===
+  Tue Apr 29 10:34:50 BST 2025 Finished
+  >: wprog.sh '4?' 'Tue Apr 29 09:43:06 BST 2025' 'Tue Apr 29 10:34:50 AM'
+  112.00 0:51:44 129.90
+Back to more-or-less full speed...
+
+  >: for ty in warc; do sbatch --output=slurm_50-59x5${ty::1}.out --time=02:00:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh
+                                           
+j=$((i - 1)) ; tp=$((task + 50)) ; ty='$ty'
+mkdir -p ./x$j/$tp/$ty                                                          ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg"  2> ./x$j/$tp/$ty/w2c_errs'; done
+  >: wprog.sh '5?' 'Tue Apr 29 12:31:20 BST 2025' 'Tue Apr 29 01:21:39 PM'
+  112.00 0:50:19 133.55
+  >: for ty in robotstxt crawldiagnostics; do sbatch --output=slurm_50-59x5${ty::1}.out --time=00:15:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh
+j=$((i - 1)) ; tp=$((task + 50)) ; ty='$ty'
+mkdir -p ./x$j/$tp/$ty                                                          ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg"  2> ./x$j/$tp/$ty/w2c_errs'; done
+  >: wprog.sh '5?' 'Tue Apr 29 12:31:56 BST 2025' 'Tue Apr 29 12:36:40 PM' crawldiagnostics
+  112.00 0:04:44 1419.72
+  >: wprog.sh '5?' 'Tue Apr 29 12:31:54 BST 2025' 'Tue Apr 29 12:33:21 PM' robotstxt
+  112.00 0:01:27 4634.48
+
+  >: for ty in warc; do sbatch --output=slurm_60-69x5${ty::1}.out --time=02:00:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh
+
+  j=$((i - 1)) ; tp=$((task + 60)) ; ty='$ty'
+  mkdir -p ./x$j/$tp/$ty                                                          ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg"  2> ./x$j/$tp/$ty/w2c_errs'; done
+
+  >: for ty in robotstxt crawldiagnostics; do sbatch --output=slurm_60-69x5${ty::1}.out --time=00:15:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh
+  j=$((i - 1)) ; tp=$((task + 60)) ; ty='$ty'
+  mkdir -p ./x$j/$tp/$ty
+  ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg"  2> ./x$j/$tp/$ty/w2c_errs'; done
+
+Continuing w/o logging to do the rest of the segments, checking as
+above for consistency with existing results...
+Done.
+
+Just rerun x3/35/ 336,447 Done and checked
+
+Next steps:
+ Fetch c_d and, if needed r_t for 2023-40 - DONE
+ Run a by-subparts batch of warc2cdb for warc, r_t and c_d for 2019-35: DONE
+ Build N lmh.cdb for 2019-35, _including_ r_t and c_d
+  Might there be key collisions?  Try a test...
+
+Next steps, cont'd:
+ Remove all w2c_errs and ldb.cdb_in from *** downwards of 2023-40
+ Do whatever checks are possible on the results
+ Fill in r_t? and c_d ldb.cdb_in for 0 1 2 4 11 12 15
+  >: egrep -o '^[^-]*->' x1/27/warc/lmh.cdb_in |sus | head
+      1 +100,10:20190822110255http://legacy.abcquant.com/index.php?option=com_content&view=article&id=158&Itemid=208->
+      ...
+  >: egrep -ho '^[^-]*->' x1/27/*/lmh.cdb_in |sus | head
+      1 +100,10:20190822110255http://legacy.abcquant.com/index.php?option=com_content&view=article&id=158&Itemid=208->
+      ...
+  >: egrep -ho '^[^-]*->' x?/27/*/lmh.cdb_in |sus | head
+      1 +100,10:20190822110255http://legacy.abcquant.com/index.php?option=com_content&view=article&id=158&Itemid=208->
+      ...
+