Mercurial > hg > cc > work
changeset 84:5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
| author | Henry S. Thompson <ht@inf.ed.ac.uk> |
|---|---|
| date | Mon, 14 Apr 2025 15:22:32 +0100 |
| parents | d41cef3d4582 |
| children | 19e2136f595b |
| files | lurid3/notes.txt |
| diffstat | 1 files changed, 359 insertions(+), 1 deletions(-) [+] |
line wrap: on
line diff
--- a/lurid3/notes.txt Wed Apr 09 21:53:55 2025 +0100 +++ b/lurid3/notes.txt Mon Apr 14 15:22:32 2025 +0100 @@ -1837,7 +1837,7 @@ OK, rerun in the right place: >: cd ~/results/CC-MAIN-2023-40/warc_lmhx - >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 15 . 2> 15/w2c_errs &/work/dc007/dc007/hst/lib/python/cc:/work/dc007/dc007/hst/lib/python/cc/lmh + >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 15 . 2> 15/w2c_errs & real 210m54.183s user 152m34.172s @@ -2384,7 +2384,365 @@ x4/11/w2c_errs:180 >: tail -1 slurm_*11n* Wed Apr 9 21:39:53 BST 2025 Finished + Need some sanity checks before gearing up production... + + >: head -1402 11/w2c_errs | diff -bw - <(fgrep -v records x1/11/w2c_errs) + [nothing] + +Argh. Missing a few hits at the end of each file: + >: python3 -m w2c 2023-40 11 plus 000,002 2> plus/11/w2c_err + >: grep records plus/11/w2c_errs + 115308 records, max record: 1058293, max header: 20289 + 114429 records, max record: 1058293, max header: 20289 + 114729 records, max record: 1058293, max header: 20289 + >: uz /beegfs/common_crawl/CC-MAIN-2023-40/1695233506559.11/orig/warc/CC-MAIN-20230924023050-20230924053050-00000.warc.gz |egrep -ac '^WARC-Type: ' + 115408 + +Last record from each buffer-load? No, the entire last buffer-full. +Fixed. + + >: python3 -m w2c 2023-40 11 plus 000,002 2> plus/11/w2c_errs + plus/11/lmh.cdb_in + >: grep records plus/11/w2c_errs + 115408 records, max record: 1058293, max header: 20289 + 114508 records, max record: 1058293, max header: 20289 + 114865 records, max record: 1058293, max header: 20289 + >: head -12 11/w2c_errs |diff -bw - plus/11/w2c_errs + 2a3,4 + > 115408 records, max record: 1058293, max header: 20289 + > 6186 LM headers, 6185 win, 1 lose, 0 non-ASCII URIs + 4a7,8 + > 114508 records, max record: 1058293, max header: 20289 + > 6149 LM headers, 6148 win, 1 lose, 1 non-ASCII URIs + 12c16,17 + < /beegfs/common_crawl/CC-MAIN-2023-40/1695233506559.11/orig/warc/CC-MAIN-20230924023050-20230924053050-00003.warc.gz + --- + > 114865 records, max record: 1058293, max header: 20289 + > 6192 LM headers, 6186 win, 6 lose, 1 non-ASCII URIs + +Retry the 5-by test: + + >: fgrep -c beegfs x?/11/w* + x0/11/w2c_errs:180 + x1/11/w2c_errs:180 + x2/11/w2c_errs:180 + x3/11/w2c_errs:180 + x4/11/w2c_errs:180 + >: head -1 slurm_*11n* + Thu Apr 10 15:35:12 BST 2025 Launching 3 nodes ... + >: tail -1 slurm_*11n* + Thu Apr 10 16:35:48 BST 2025 Finished + +A minute or two slower. + +Error logs still look OK: + + >: head -1402 11/w2c_errs | diff -bw - <(fgrep -v records x1/11/w2c_errs) + [nothing] + +Right number of records for 00000 + >: grep records x1/11/w2c_errs |head -3 + 115408 records, max record: 1058293, max header: 20289 + 114508 records, max record: 1058293, max header: 20289 + 114865 records, max record: 1058293, max header: 20289 + >: grep records x1/11/w2c_errs |tail -1 + 115216 records, max record: 1079547, max header: 40116 + >: for i in 0 2 3 4; do grep records x$i/11/w2c_errs |tail -1 ; done + 115654 records, max record: 1077742, max header: 40071 + 115978 records, max record: 1077845, max header: 16008 + 115483 records, max record: 1079052, max header: 25052 + 114757 records, max record: 1079623, max header: 40083 + >: diff -bw 11/lmh.cdb_in x1/11/lmh.cdb_in |egrep -v '^<' + 1130232,1174166c1130232 + >: for i in {0..4}; do grep 'LM headers' x$i/11/w2c_errs|cut -f 4 -d ' ' |btot ; done| cut -f 4 -d ' ' |tee >(cat 1>&2) | btot + 1132831 + 1130231 + 1131595 + 1131756 + 1131479 + 5657892 + +Belt and braces -- rerun all the existing full 900 cases except 11 + >: for d in 0 15 1 surplus/4; do mkdir -p tentative/$d; mv $d/{w2c_errs,lmh.cdb_in} tentative/$d; done + >: for d in 0; do time ~/lib/python/cc/w2c.py 2023-40 $d . 2> $d/w2c_errs; done & + Thu Apr 10 10:50:37 PM BST 2025 + + real 279m24.973s + user 176m3.292s + sys 18m21.907s + ./0/lmh.cdb_in + + >: date; for d in 1 2 4 15; do { ~/lib/python/cc/w2c.py 2023-40 $d . 2> $d/w2c_errs ; echo $d finished $(date); } & done + ./1/lmh.cdb_in + 1 finished Fri Apr 11 03:30:01 AM BST 2025 + ./4/lmh.cdb_in + 4 finished Fri Apr 11 03:31:44 AM BST 2025 + ./2/lmh.cdb_in + 2 finished Fri Apr 11 03:35:17 AM BST 2025 + ./15/lmh.cdb_in + 15 finished Fri Apr 11 03:38:49 AM BST 2025 + + >: cut -f 2 <(sed 's/value or format .*/value or format/' 0/w2c_errs)|egrep -v 'warc.gz|LM headers|records,'| sus + 4381 Invalid date value or format + 1 year 4263994 is out of range + 1 year 4278897 is out of range + 1 year 4355304 is out of range + 1 year 4361928 is out of range + 1 year 4370444 is out of range + 1 year 4430056 is out of range + 1 year 4433368 is out of range + 1 year 53310 is out of range + >: cut -f 2 0/ks_errs.tsv | sus + 3188 cannot unpack non-iterable NoneType object + 1193 list index out of range + 1 year 4263994 is out of range + 1 year 4278897 is out of range + 1 year 4355304 is out of range + 1 year 4361928 is out of range + 1 year 4370444 is out of range + 1 year 4430056 is out of range + 1 year 4433368 is out of range + 1 year 53310 is out of range +Same for 4 and 15 + + >: cd 0 + >: wc -l ks.tsv + 6414380 ks.tsv + >: wc -l lmh.cdb_in + 6414381 lmh.cdb_in + >: diff -bw <(cut -f2- ks.tsv| sed 's/ //;s/\.0$//'|sort -k1,1) <(sed 's/^\+[0-9,]*://;s/->\([0-9]*\)$/ \1/' lmh.cdb_in | sort -k1,1)|head + + >: ls -l {tentative/,}{0,4,15}/lmh.cdb_in tentative/surplus/4/lmh.cdb_in + -rw-r--r-- 1 hst dc007 719413017 Apr 11 03:24 0/lmh.cdb_in + -rw-r--r-- 1 hst dc007 719413017 Mar 24 16:52 tentative/0/lmh.cdb_in + -rw-r--r-- 1 hst dc007 603244719 Apr 11 03:38 15/lmh.cdb_in + -rw-r--r-- 1 hst dc007 603244719 Mar 19 22:56 tentative/15/lmh.cdb_in + -rw-r--r-- 1 hst dc007 704148582 Apr 11 03:31 4/lmh.cdb_in + -rw-r--r-- 1 hst dc007 704148582 Mar 27 02:12 tentative/surplus/4/lmh.cdb_in + +But: + + >: diff -bw <(cut -f2- ks.tsv| sed 's/ //;s/\.0$//'|sort -k1,1) <(sed 's/^\+[0-9,]*://;s/->\([0-9]*\)$/ \1/' lmh.cdb_in | sort -k1,1)|head + 14634c14635 + < 20230930082240https://www.levnepletaciprize.cz/shop/p\u0159\u00EDze-pro-miminka/elen-baby?error=404 1696062160 + --- + > 20230930082240https://www.levnepletaciprize.cz/shop/p%C5%99%C3%ADze-pro-miminka/elen-baby?error=404 1696062160 + 25306c25307 + < 20230930082317https://zums.ac.ir/content/68251/\u0647\u0632\u06CC\u0646\u0647-\u0627\u0639\u0645\u0627\u0644-\u062C\u0631\u0627\u062D\u06CC-\u06F1\u06F4\u06F0\u06F1 1696058653 + --- + > 20230930082317https://zums.ac.ir/content/68251/%D9%87%D8%B2%DB%8C%D9%86%D9%87-%D8%A7%D8%B9%D9%85%D8%A7%D9%84-%D8%AC%D8%B1%D8%A7%D8%AD%DB%8C-%DB%B1%DB%B4%DB%B0%DB%B1 1696058653 + +Oops! + + : fgrep -c \\u0 */ks.tsv + 0/ks.tsv:1062 + 12/ks.tsv:679 + 15/ks.tsv:634 + 36/ks.tsv:1311 + 4/ks.tsv:770 + 56/ks.tsv:1194 + 68/ks.tsv:846 + best_two_by_nl1/ks.tsv:1873 + +Even worse: + >: cd ../../CC-MAIN-2019-35/warc_lmhx + >: fgrep -c \\u0 ?/ks.tsv + 0/ks.tsv:3 + 1/ks.tsv:4 + 2/ks.tsv:4 + 3/ks.tsv:2 + 4/ks.tsv:0 + 5/ks.tsv:2 + 6/ks.tsv:1 + 7/ks.tsv:1 + 8/ks.tsv:8 + 9/ks.tsv:4 + +Or maybe not? + >: fgrep \\u0 cluster.idx + at,wko,news)/news/system/400.html?error=400&errorrequest=/pdf/gen?url=https:/news.wko.at/news/oesterreich/wirtschaftsbarometer_wirtschaftslage.html?shorturl=wirtschaftsbarometerat&key=98c90396b77ff930d6df30ce0f7f7d9c&pdfoptions={"header-html":"\/news\/templates\/header_html.php","footer-html":"https:\/\/news.wko.at\/news\/templates\/footer_html.php?url=https://news.wko.at/news/oesterreich/wirtschaftsbarometer_wirtschaftslage.html?shorturl=wirtschaftsbarometerat","print-media-type":"","title":"wirtschaftsbarometer:+wk\u00d6-konjunkturumfrage"} 20190826093246 cdx-00001.gz 849440852 286441 8844 + >: fgrep -c \\u0 /beegfs/common_crawl/CC-MAIN-2019-35/cdx/cluster.idx + 1 + sing<3771>: fgrep \\u0 /beegfs/common_crawl/CC-MAIN-2019-35/cdx/cluster.idx + at,wko,news)/news/system/400.html?error=400&errorrequest=/pdf/gen?url=https:/news.wko.at/news/oesterreich/wirtschaftsbarometer_wirtschaftslage.html?shorturl=wirtschaftsbarometerat&key=98c90396b77ff930d6df30ce0f7f7d9c&pdfoptions={"header-html":"\/news\/templates\/header_html.php","footer-html":"https:\/\/news.wko.at\/news\/templates\/footer_html.php?url=https://news.wko.at/news/oesterreich/wirtschaftsbarometer_wirtschaftslage.html?shorturl=wirtschaftsbarometerat","print-media-type":"","title":"wirtschaftsbarometer:+wk\u00d6-konjunkturumfrage"} 20190826093246 cdx-00001.gz 844682060 287090 8844 + +Try using the codecs hack from sort_date.py + +What's the best segment to test on... + + >: pwd + /beegfs/common_crawl/CC-MAIN-2023-40/cdx/warc + >: uz cdx-00001.gz |egrep -a '\\u[0-9a-z][0-9a-z]'|egrep -o '"filename": "crawl-data.*' | cut -f 4 -d / | cut -f 2 -d . | sus + 79 91 + 72 80 + 63 52 + 62 78 + 42 56 + 40 15 + ... + +No, those are mostly (all for seg 15) crawl-diagnostics +Ah, but, sigh, c-d/.../warc.gz do _have_ timestamps + +>: uz cdx-0001?.gz |egrep -a '\\u[0-9a-z][0-9a-z]'|egrep -o '"filename": "crawl-data.*/warc/' | cut -f 4 -d / | cut -f 2 -d . | sus + 9 57 + 6 97 + 4 49 + 4 6 + 3 11 + ... + + >: for f in cdx-0001?.gz; do echo -n "$f "; egrep -a '\\u[0-9a-z][0-9a-z]' <(uz $f)|egrep '"filename": "crawl-data.*/warc/' | fgrep -c '.11/warc'; done + cdx-00010.gz 0 + ... + cdx-00017.gz 3 + + >: for f in cdx-00017.gz; do egrep -a '\\u[0-9a-z][0-9a-z]' <(uz $f)|egrep '"filename": "crawl-data.*/warc/' | fgrep '.11/warc'; done + cn,kbstore,auth)/index.php/apireguser/flag/8?code=jdtong&code=jdtong&data={"logo":"http:\/\/jdtong.kbstore.cn\/public\/images\/logo-1.png","logo_url":"http:\/\/jdtong.kbstore.cn\/admin?flag=8","title":"\u8857\u5e97\u901a" [...] "filename": "crawl-data/CC-MAIN-2023-40/segments/1695233506559.11/warc/CC-MAIN-20230924023050-20230924053050-00003.warc.gz", "charset": "UTF-8", "languages": "zho"} + cn,kbstore,auth)/index.php/apireguser/flag/8?code=jdtong&code=jdtong&data={"logo":"http:\/\/jdtong.kbstore.cn\/public\/images\/logo-1.png","logo_url":"http:\/\/jiediantong.cn\/admin?flag=8","title":"\u8857\u5e97\u901a" ... "filename": "crawl-data/CC-MAIN-2023-40/segments/1695233506559.11/warc/CC-MAIN-20230924023050-20230924053050-00225.warc.gz", "charset": "UTF-8", "languages": "zho"} + cn,kbstore,auth)/index.php/apireguser/flag/8?code=kbcms&code=kbcms&data={"logo":"http:\/\/demo.cms.vip100.ltd\/public\/images\/logo.png","logo_url":"http:\/\/demo.cms.vip100.ltd\/admin?flag=8","title":"\u4eba\u4eba\u7ad9cms" [...] "filename": "crawl-data/CC-MAIN-2023-40/segments/1695233506559.11/warc/CC-MAIN-20230924023050-20230924053050-00537.warc.gz", "charset": "UTF-8", "languages": "zho"} + +Red herring chase: none of those have timestamps. + +So how can this be true: + >: egrep -c ' [^0] non-ASCII' {0,1,2,4,15}/w2* + 0/w2c_errs:774 + 1/w2c_errs:683 + 2/w2c_errs:685 + 4/w2c_errs:731 + 15/w2c_errs:643 + +Because e.g. Korean appears non-uniformly in index files: + + >: egrep '^kr,' cluster.idx |head -3 + kr,07e,m)/service/agreement 20230924093051 cdx-00221.gz 816253375 185571 1005693 + kr,114n,bit)/sys/login.php?amp;auto_no=568&returnurl=/sys/login.php?auto_no=568&returnurl=http://bit.114n.kr/sys/board.php?bo_id=m71 20230923204454 cdx-00221.gz 816438946 182780 1005694 + kr,1bang)/posts/online-shopping/13076 20231003101132 cdx-00221.gz 816621726184260 1005695 + + >: uz cdx-00221.gz | egrep -c '\\u[0-9a-f]' + 9575 + >: uz cdx-00010.gz | egrep -c '\\u[0-9a-f]' + 746 + + >: ~/lib/python/cc/w2c.py 2023-40 15 . 000 |& less + >: diff -bw 15/lmh.cdb_in ~/results/CC-MAIN-2023-40/warc_lmhx/15/lmh.cdb_in + 98c98 + < +99,10:20230922035528http://cn.coii.kr/board/\u5546\u54C1\u8D2D\u4E70\u540E\u8BB0/4/?link_product_no=19597->1695354928 + --- + > +117,10:20230922035528http://cn.coii.kr/board/%E5%95%86%E5%93%81%E8%B4%AD%E4%B9%B0%E5%90%8E%E8%AE%B0/4/?link_product_no=19597->1695354928 + +Try the whole thing: + + >: sbatch --output=slurm_w2c_15f.out --time=02:00:00 --ntasks=5 -c 2 $HOME/bin/runme.sh $PWD -d -t 2 -i 'cat 11n.pats |share_by_task.sh $n $task' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh + mkdir -p ./fixu/x$task/15 + ~/lib/python/cc/w2c.py 2023-40 15 ./fixu/x$task "$arg" 2> ./fixu/x$task/15/w2c_errs' + Sun Apr 13 03:45:46 PM BST 2025 end 4 + Sun Apr 13 03:45:46 PM BST 2025 task 5.4 on r1i0n20:2.1 end + ./fixu/x3/15/lmh.cdb_in + Sun Apr 13 03:45:50 PM BST 2025 end 3 + Sun Apr 13 03:45:50 PM BST 2025 task 5.3 on r1i0n20:2.1 end + ./fixu/x2/15/lmh.cdb_in + Sun Apr 13 03:46:04 PM BST 2025 end 2 + Sun Apr 13 03:46:04 PM BST 2025 task 5.2 on r1i0n20:2.1 end + ./fixu/x1/15/lmh.cdb_in + Sun Apr 13 03:46:41 PM BST 2025 end 1 + Sun Apr 13 03:46:41 PM BST 2025 task 5.1 on r1i0n3:2.0 end + ./fixu/x0/15/lmh.cdb_in + Sun Apr 13 03:52:49 PM BST 2025 end 0 + Sun Apr 13 03:52:49 PM BST 2025 task 5.0 on r1i0n3:2.0 end + Sun Apr 13 15:52:49 BST 2025 Finished + >: head -1 slurm_w2c_15f.out + Sun Apr 13 14:59:54 BST 2025 Launching 2 nodes in /mnt/lustre/e1000/home/dc007/dc007/hst/results/CC-MAIN-2023-40/warc_lmhx for runme /work/dc007/dc007/hst/results/CC-MAIN-2023-40/warc_lmhx -d -t 2 -i cat 11n.pats |share_by_task.sh $n $task export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh + +~50:10 elapsed: Faster! + + >: diff -bw <(cut -f2- 15/ks.tsv| sed 's/ //;s/\.0$//'|sort -k1,1) <(sed 's/^\+[0-9,]*://;s/->\([0-9-]*\)$/ \1/' fixu/x{0..4}/15/lmh.cdb_in | sort -k1,1)| tee /tmp/hst/udiff | less + >: cat /tmp/hst/udiff + 0a1,5 + > + > + > + > + > + +And correct. + +Another one: + + >: sbatch --output=slurm_w2c_4f.out --time=01:30:00 --ntasks=5 -c 2 $HOME/bin/runme.sh $PWD -d -t 2 -i 'cat 11n.pats |share_by_task.sh $n $task' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh + mkdir -p ./fixu/x$task/4 + ~/lib/python/cc/w2c.py 2023-40 4 ./fixu/x$task "$arg" 2> ./fixu/x$task/4/w2c_errs' + Submitted batch job 7381438 + cirrus-login1<6096>: fgrep -c beegfs fixu/x?/4/w* | cut -f 2 -d : | btot + 900 + >: head -1 slurm_w2c_4f.out + Mon Apr 14 08:55:08 BST 2025 Launching 3 nodes in /mnt/lustre/e1000/home/dc007/dc007/hst/results/CC-MAIN-2023-40/warc_lmhx for runme /work/dc007/dc007/hst/results/CC-MAIN-2023-40/warc_lmhx -d -t 2 -i cat 11n.pats |share_by_task.sh $n $task export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh + >: tail -1 slurm_w2c_4f.out + Mon Apr 14 09:41:51 BST 2025 Finished + +46:43 elapsed + + >: diff -bw <(cut -f2- 4/ks.tsv| sed 's/ //;s/\.0$//'|sort -k1,1) <(sed 's/^\+[0-9,]*://;s/->\([0-9-]*\)$/ \1/' fixu/x{0..4}/4/lmh.cdb_in | sort -k1,1)| tee /tmp/hst/udiff | less + >: cat /tmp/hst/udiff + 0a1,5 + > + > + > + > + > + +And errors: + >: for d in 4 15; do ls -ld $d; cut -f 2 <(sed 's/value or format .*/value or format/' fixu/x?/$d/w2c_errs)|egrep -v 'warc.gz|LM headers|records,'| sus ; cut -f 2 $d/ks_errs.tsv | sus ; done + drwxr-sr-x 2 hst dc007 110592 Apr 10 22:50 4 + 6107 Invalid date value or format + 8 year 539500 is out of range + 7 year 539498 is out of range + 2 year 55722 is out of range + 1 year 53337 is out of range + 1 year 539499 is out of range + 1 year 54618 is out of range + 1 year 54752 is out of range + 4605 cannot unpack non-iterable NoneType object + 1502 list index out of range + 8 year 539500 is out of range + 7 year 539498 is out of range + 2 year 55722 is out of range + 1 year 53337 is out of range + 1 year 539499 is out of range + 1 year 54618 is out of range + 1 year 54752 is out of range + drwxr-sr-x 2 hst dc007 98304 Apr 10 22:50 15 + 8090 Invalid date value or format + 113 year 641471 is out of range + 20 year 641474 is out of range + 9 hour must be in 0..23 + 9 year 642435 is out of range + 3 year 642437 is out of range + 1 year 4262575 is out of range + 1 year 641769 is out of range + 1 year 642434 is out of range + 5420 cannot unpack non-iterable NoneType object + 2670 list index out of range + 113 year 641471 is out of range + 20 year 641474 is out of range + 9 hour must be in 0..23 + 9 year 642435 is out of range + 3 year 642437 is out of range + 1 year 4262575 is out of range + 1 year 641769 is out of range + 1 year 642434 is out of range + +Those were both with warc2cdb.py. Try a 3rd run for timing after +cythonize + + >: head -1 slurm_w2c_0f.out + Mon Apr 14 14:15:55 BST 2025 Launching 1 nodes in /mnt/lustre/e1000/home/dc007/dc007/hst/results/CC-MAIN-2023-40/warc_lmhx for runme /work/dc007/dc007/hst/results/CC-MAIN-2023-40/warc_lmhx -d -t 2 -i cat 11n.pats |share_by_task.sh $n $task export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh + >: tail -1 slurm_w2c_0f.out + Mon Apr 14 15:03:41 BST 2025 Finished + +47:46 elapsed, mid-day weekday, all on one node... +Both cross-checks succeed ================ Try it with the existing _per segment_ index we have for 2019-35
