Mercurial > hg > cc > work

changeset 87:90b9c202922c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
successful redo 2019-35 segs 0-9
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Wed, 23 Apr 2025 18:23:35 +0100
parents 855665c79545
children d622b1b5e60d
files lurid3/notes.txt
diffstat 1 files changed, 19 insertions(+), 1 deletions(-) [+]
line wrap: on
line diff
--- a/lurid3/notes.txt	Wed Apr 23 17:03:32 2025 +0100
+++ b/lurid3/notes.txt	Wed Apr 23 18:23:35 2025 +0100
@@ -3085,7 +3085,25 @@
 
 So try the real thing again
   Wed Apr 23 17:00:25 BST 2025 Launching 5 nodes
-
+  >: fgrep -ch beegfs x?/?/warc/w* | sort -nr | headtail -3
+  112
+  112
+  112
+  ===
+  112
+  112
+  112
+  >: headtail -1 slurm_0-9x5w.out
+  Wed Apr 23 17:00:25 BST 2025 Launching 5 nodes in /mnt/lustre/e1000/home/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx for runme /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx -d -t 3 -i cat ranges export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh
+  ===
+  Wed Apr 23 17:49:15 BST 2025 Finished
+  >: python3 -c 'import sys,datetime;(s,d1,d2)=sys.argv[1:4];dd2=datetime.datetime.strptime(d2[4:22],"%b %d %I:%M:%S %p");dd1=datetime.datetime.strptime(d1[4:19],"%b %d %H:%M:%S");d=(dd2-dd1).seconds;h=d/3600;print(a:=int(s)/50.0,"%d:%0.2d:%0.2d"%(h,(d:=(d%3600))/60,d % 60))' $(fgrep -ch beegfs x?/?/warc/w* | btot) 'Wed Apr 23 17:00:25 BST 2025' "Wed Apr 23 05:49:15 PM"
+  112.0 0:48:50
+
+All good
+
+Nothing but the same utf-8 mismatches in e.g. segment 5:
+  >: diff -bw <(cut -f 1 5/ks_errs.tsv) <(cat x{0..4}/5/warc/w2c_errs | egrep -v 'beegfs|records|headers' |cut -f 1|sed "s/^b'\(.*\)'$/\1/")
 ================
 
 Try it with the existing _per segment_ index we have for 2019-35