Mercurial > hg > cc > work

changeset 86:855665c79545

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Wed, 23 Apr 2025 17:03:32 +0100
parents 19e2136f595b
children 90b9c202922c
files lurid3/notes.txt
diffstat 1 files changed, 167 insertions(+), 2 deletions(-) [+]
line wrap: on
line diff
--- a/lurid3/notes.txt	Fri Apr 18 19:18:52 2025 +0100
+++ b/lurid3/notes.txt	Wed Apr 23 17:03:32 2025 +0100
@@ -2874,6 +2874,12 @@
     5308243 total
   >: wc -l 0/ks.tsv
   5308240 0/ks.tsv
+  >: diff -bw <(cut -f2- 0/ks.tsv| fgrep $'\t'| sed 's/  //;s/\.0$//'|sort -k1,1) <(sed 's/^\+[0-9,]*://;s/->\([0-9-]*\)$/ \1/' x{0..2}/0/warc/lmh.cdb_in | sort -k1,1)| tee /tmp/hst/0diff
+  >: cat /tmp/hst/0diff
+  0a1,3
+  >
+  >
+  >
 
   >: wc -l 0/ks_errs.tsv
   21462 0/ks_errs.tsv
@@ -2915,12 +2921,171 @@
 	1 year 641778 is out of range
 All good
 
- Fetch c_d and, if needed r_t for 2023-40
+ Fetch c_d and, if needed r_t for 2023-40 - DONE
+ 
  Remove all w2c_errs and ldb.cdb_in from *** downwards
- Run a parallel full segment job for 1 2 11 12
+ Run a by-subparts batch of warc2cdb for warc, r_t and c_d for 2019-35
  Do whatever checks are possible on the results
  Fill in r_t? and c_d ldb.cdb_in for 0 1 2 4 11 12 15
  Take stock: more 2023-40, or back to 2019-35 to add c_d (and r_t)?
+
+This didn't work:
+  >: sbatch --output=slurm_1-10x5.out --time=02:00:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh -m 1 10 $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh
+j=$((i - 1))
+mkdir -p ./x$j/$xarg/warc
+~/lib/python/cc/w2c.py 2019-35 $xarg ./x$j warc "$arg"  2> ./x$j/$xarg/warc/w2c_errs'
+
+so reverted to not using -m, and slightly modified versions of
+(_)runme.sh, and this:
+
+  >: sbatch --output=slurm_1-10x5.out --time=01:20:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh
+j=$((i - 1))
+mkdir -p ./x$j/$task/warc
+~/lib/python/cc/w2c.py 2019-35 $task ./x$j warc "$arg"  2> ./x$j/$task/warc/w2c_errs'
+  >: head -1 slurm_1-10x5.out
+  Wed Apr 23 10:55:08 BST 2025 Launching 5 nodes
+is working, I think:
+  >: fgrep -ch beegfs x?/?/warc/w* | sort -nr | { sed -u 3q ;echo === ; tail -3 ; }
+  61
+  61
+  60
+  ===
+  58
+  58
+  58
+  >: date
+  Wed Apr 23 11:22:20 AM BST 2025
+
+So half-way in 27 minutes, so should finish in about an hour...
+Better:
+  >: tail -1 slurm_1-10x5.out
+  Wed Apr 23 11:45:19 BST 2025 Finished
+But:
+  >: fgrep -h beegfs x?/*/warc/w* | cut -f 5,8 -d / | tr ./ - | wc -l
+  5544
+E.g.
+  >: fgrep -c beegfs x0/7/warc/w2c_errs
+  109
+  >: fgrep -h beegfs x0/7/warc/w* | cut -f 5,8 -d / | tr ./ - | cut -f 2,7 -d - | less -N
+      1 7-00000
+    ...
+    107 7-00106
+    108 7-00107
+    109 7-00108
+  (END)
+So just a premature termination? Yes:
+  >: tail -15 slurm_1-10x5.out
+  ./x2/3/warc/lmh.cdb_in
+  Wed Apr 23 11:45:17 AM BST 2025 end 3 3
+  /work/dc007/dc007/hst/bin/_runme.sh: line 101: rallel: command not found
+  /work/dc007/dc007/hst/bin/_runme.sh: line 102: syntax error near unexpected token `fi'
+  /work/dc007/dc007/hst/bin/_runme.sh: line 102: `   fi'
+  srun: error: r1i1n1: task 3: Exited with exit code 2
+  srun: launch/slurm: _step_signal: Terminating StepId=7388166.0
+  slurmstepd: error: *** STEP 7388166.0 ON r1i0n10 CANCELLED AT 2025-04-23T11:45:18 ***
+
+Bizarre -- some kind of file-system glitch?  File looks OK now...
+
+Try this, just to see if we can trigger it more easily...
+  >: sbatch --output=slurm_0-9x5rtcd.out --time=01:20:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh
+  j=$((i - 1))
+  mkdir -p ./x$j/$task/robotstxt ; mkdir -p ./x$j/$task/crawldiagnostics
+  ~/lib/python/cc/w2c.py 2019-35 $task ./x$j robotstxt "$arg"  2> ./x$j/$task/robotstxt/w2c_errs
+  ~/lib/python/cc/w2c.py 2019-35 $task ./x$j crawldiagnostics "$arg"  2> ./x$j/$task/crawldiagnostics/w2c_errs'
+  
+  >: fgrep -ch beegfs x?/?/robotstxt/w* | sort -nr | headtail -3
+  112
+  112
+  112
+  ===
+  112
+  112
+  112
+  >: fgrep -ch beegfs x?/?/crawldiagnostics/w* | sort -nr | headtail -3
+  112
+  112
+  112
+  ===
+  112
+  112
+  112
+
+A quick check suggests it's at least starting out OK
+  
+  >: head -4274 0/ks_errs.tsv | cut -f 1 | diff - <(egrep -v 'beegfs|records|headers' x0/0/warc/w2c_errs|cut -f 1|sed "s/^b'\(.*\)'$/\1/") | headtail -18
+  46,47c46,47
+  < Чт, 18 Июл 2019 12:38:01
+  < Пт, 14 Июн 2019 10:03:52
+  ---
+  > \xd0\xa7\xd1\x82, 18 \xd0\x98\xd1\x8e\xd0\xbb 2019 12:38:01
+  > \xd0\x9f\xd1\x82, 14 \xd0\x98\xd1\x8e\xd0\xbd 2019 10:03:52
+  70,71c70,71
+  < Пт, 14 Июн 2019 11:46:10
+  < Вт, 11 Июн 2019 14:11:46
+  ---
+  > \xd0\x9f\xd1\x82, 14 \xd0\x98\xd1\x8e\xd0\xbd 2019 11:46:10
+  > \xd0\x92\xd1\x82, 11 \xd0\x98\xd1\x8e\xd0\xbd 2019 14:11:46
+  173c173
+  < Пт, 14 Июн 2019 08:48:59
+  ---
+  > \xd0\x9f\xd1\x82, 14 \xd0\x98\xd1\x8e\xd0\xbd 2019 08:48:59
+  185c185
+  < {ts '2019-08-17 18:56:12'}
+  ===
+  4086,4087c4086,4087
+  < 土, 17  8月 2019 23:42:13 GMT
+  < 土, 17  8月 2019 23:50:00 GMT
+  ---
+  > \xe5\x9c\x9f, 17  8\xe6\x9c\x88 2019 23:42:13 GMT
+  > \xe5\x9c\x9f, 17  8\xe6\x9c\x88 2019 23:50:00 GMT
+  4141c4141
+  < Сб, 13 Июл 2019 00:21:19
+  ---
+  > \xd0\xa1\xd0\xb1, 13 \xd0\x98\xd1\x8e\xd0\xbb 2019 00:21:19
+  4208c4208
+  < {ts '2019-08-17 19:48:43'}
+  ---
+  > b"{ts '2019-08-17 19:48:43'}"
+  4261c4261
+  < 土, 17  8月 2019 23:08:06 GMT
+  ---
+  > \xe5\x9c\x9f, 17  8\xe6\x9c\x88 2019 23:08:06 GMT
+
+And finishing?  Yes, allowing for the fact that segment 0 was cut a
+tiny bit short:
+  >: tail -4307 0/ks_errs.tsv | head -4259 |cut -f 1 | diff - <(egrep -v 'beegfs|records|headers' x4/0/warc/w2c_errs|cut -f 1|sed "s/^b'\(.*\)'$/\1/") | headtail -12
+  60c60
+  < 土, 17  8月 2019 23:15:01 GMT
+  ---
+  > \xe5\x9c\x9f, 17  8\xe6\x9c\x88 2019 23:15:01 GMT
+  87c87
+  < {ts '2019-08-17 19:17:39'}
+  ---
+  > b"{ts '2019-08-17 19:17:39'}"
+  120c120
+  < Пт, 07 Дек 2018 08:14:56
+  ---
+  > \xd0\x9f\xd1\x82, 07 \xd0\x94\xd0\xb5\xd0\xba 2018 08:14:56
+  ===
+  4207,4208c4207,4208
+  < 土, 17  8月 2019 22:34:41 GMT
+  < 土, 17  8月 2019 22:33:05 GMT
+  ---
+  > \xe5\x9c\x9f, 17  8\xe6\x9c\x88 2019 22:34:41 GMT
+  > \xe5\x9c\x9f, 17  8\xe6\x9c\x88 2019 22:33:05 GMT
+  4255,4259d4254
+  < 8/17/2019 7:17:15 PM GMT
+  < 08/18/19, 18 37 2019 00:%i:1566081424 GMT
+  < 08/18/19, 18 28 2019 01:%i:1566084517 GMT
+  <
+  <
+
+Similarly for segment 9
+
+
+So try the real thing again
+  Wed Apr 23 17:00:25 BST 2025 Launching 5 nodes
+
 ================
 
 Try it with the existing _per segment_ index we have for 2019-35