Mercurial > hg > cc > work
annotate lurid3/notes.txt @ 88:d622b1b5e60d
shuffle a bit, plan
| author | Henry S. Thompson <ht@inf.ed.ac.uk> |
|---|---|
| date | Fri, 02 May 2025 13:10:21 +0100 |
| parents | 90b9c202922c |
| children | 367dca1afea4 |
| rev | line source |
|---|---|
| 40 | 1 See old_notes.txt for all older notes on Common Crawl dataprocessing, |
| 2 starting from Azure via Turing and then LURID and LURID2. | |
| 3 | |
| 4 Installed /beegfs/common_crawl/CC-MAIN-2024-33/cdx | |
| 5 >: cd results/CC-MAIN-2024-33/cdx/ | |
| 6 >: cut -f 2 counts.tsv | btot | |
| 7 2,793,986,828 | |
| 8 | |
| 9 State of play wrt data -- see status.xlsx | |
| 10 | |
|
41
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
11 [in trying to tabulate the date ranges of the crawls, I found that the |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
12 WARC timestamp is sometimes bogus: |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
13 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
14 >: fgrep ' 2009' CC-MAIN-2018-34/cdx/cluster.idx |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
15 net,tyredeyes)/robots.txt 20090201191318 cdx-00230.gz 160573468 198277 920675 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
16 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
17 >: zgrep '^net,tyredeyes)/robots.txt' CC-MAIN-2018-34/cdx/warc/cdx-00230.gz |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
18 net,tyredeyes)/robots.txt 20090201191318 {"url": "http://tyredeyes.net/robots.txt", "mime": "text/html", "mime-detected": "text/html", "status": "301", "digest": "QH732FYSV7UM34JYWVYMB7EZGR2CYM6B", "length": "582", "offset": "1224614", "filename": "crawl-data/CC-MAIN-2018-34/segments/1534221215075.58/robotstxt/CC-MAIN-20180819090604-20180819110604-00558.warc.gz"} |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
19 net,tyredeyes)/robots.txt 20090201191319 {"url": "http://www.tyredeyes.net/robots.txt", "mime": "text/plain", "mime-detected": "text/plain", "status": "200", "digest": "PSX5IZU4B4SIXGNDKXCVFH75Q27VHUTJ", "length": "549", "offset": "2069841", "filename": "crawl-data/CC-MAIN-2018-34/segments/1534221215075.58/robotstxt/CC-MAIN-20180819090604-20180819110604-00485.warc.gz"} |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
20 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
21 This happens in 2019-35 as well :-( |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
22 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
23 >: fgrep ' 20181023' CC-MAIN-2019-35/cdx/cluster.idx |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
24 com,gyshbsh)/robots.txt 20181023022000 cdx-00078.gz 356340085 162332 315406 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
25 >: zgrep ' 20181023' CC-MAIN-2019-35/cdx/warc/cdx-00078.gz |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
26 com,gyshbsh)/robots.txt 20181023022000 {"url": "http://gyshbsh.com/robots.txt", "mime": "text/html", "mime-detected": "text/html", "status": "301", "digest": "3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ", "length": "529", "offset": "638892", "filename": "crawl-data/CC-MAIN-2019-35/segments/1566027315618.73/robotstxt/CC-MAIN-20190820200701-20190820222701-00120.warc.gz"} |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
27 ... |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
28 |
| 57 | 29 Full search: |
| 30 >: find CC*/cdx -type f -name cluster.idx > /tmp/hst/clus | |
| 31 >: cat /tmp/hst/clus | while read c; do printf '%s\t%s\n' $c $(cut -f 1 -d ' ' $c | fgrep -vc ${c:8:4}); done | |
| 32 CC-MAIN-2013-20/cdx/cluster.idx 0 | |
| 33 CC-MAIN-2014-35/cdx/cluster.idx 0 | |
| 34 CC-MAIN-2015-35/cdx/cluster.idx 0 | |
| 35 CC-MAIN-2016-30/cdx/cluster.idx 0 | |
| 36 CC-MAIN-2017-30/cdx/cluster.idx 0 | |
| 37 CC-MAIN-2018-30/cdx/warc/cluster.idx 0 | |
| 38 CC-MAIN-2018-34/cdx/cluster.idx 36 | |
| 39 CC-MAIN-2019-18/cdx/warc/cluster.idx 3 | |
| 40 CC-MAIN-2019-35/cdx/cluster.idx 1 | |
| 41 CC-MAIN-2020-34/cdx/cluster.idx 0 | |
| 42 CC-MAIN-2021-25/cdx/cluster.idx 0 | |
| 43 CC-MAIN-2021-31/cdx/cluster.idx 0 | |
| 44 CC-MAIN-2021-49/cdx/cluster.idx 0 | |
| 45 CC-MAIN-2022-21/cdx/warc/cluster.idx 0 | |
| 46 CC-MAIN-2022-33/cdx/warc/cluster.idx 0 | |
| 47 CC-MAIN-2022-40/cdx/warc/cluster.idx 0 | |
| 48 CC-MAIN-2022-49/cdx/warc/cluster.idx 0 | |
| 49 CC-MAIN-2023-40/cdx/warc/cluster.idx 0 | |
| 50 CC-MAIN-2023-50/cdx/warc/cluster.idx 0 | |
| 51 CC-MAIN-2024-33/cdx/warc/cluster.idx 0 | |
| 52 Emailed this info to Sebastian Nagel 2024-12-17 | |
| 53 | |
|
41
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
54 Tabulate all the date ranges for the WARC files we have |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
55 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
56 >: for d in {2017-30,2019-35,2020-34,2021-25,2023-40,2023-50}; do printf "%s\t" $d; (ls CC-MAIN-$d/*.{?,??}/orig/warc | fgrep .gz | cut -f 3,4 -d - | sort -u |tee /dev/fd/3 | head -1 ) 3> >( tail -1 ) | tr '\n' '\t'; echo; done | cut -f 1,2,4 -d - | sed 's/-20/ 20/;s/.$//' | tr ' ' '\t' > dates.tsv |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
57 >: for d in {2018-30,2018-34}; do printf "%s\t" $d; (ls CC-MAIN-$d/{*.?,*.??} | fgrep warc.gz | cut -f 3,4 -d - | sort -u |tee /dev/fd/3 | { sleep 10 ; head -1 ; } ) 3> >( tail -1 ) | tr '\n' '\t'; echo; done >> dates.tsv |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
58 >: for d in 2019-18; do printf "%s\t" $d; (ls CC-MAIN-$d/{*.?,*.??} | fgrep warc.gz | cut -f 3,4 -d - | sort -u | head -1); done |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
59 2019-18 20190418101243-20190418122248 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
60 >: for d in 2019-18; do printf "%s\t" $d; (ls CC-MAIN-$d/{*.?,*.??} | fgrep warc.gz | cut -f 3,4 -d - | sort -u | tail -1); done |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
61 2019-18 20190426153423-20190426175423 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
62 >: echo 2019-18 20190418101243-20190418122248 20190426153423-20190426175423 >> dates.tsv |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
63 >: pwd |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
64 /beegfs/common_crawl/CC-MAIN-2016-30/cdx/warc |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
65 >: echo {000..299} | tr ' ' '\n' | parallel -j 10 'uz cdx-00{}.gz | cut -f 2 -d " " | sort -u > /tmp/hst/{}' |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
66 >: sort -mu /tmp/hst/??? > /tmp/hst/all |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
67 >: wc -l /tmp/hst/all |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
68 679686 /tmp/hst/all |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
69 >: head -1 /tmp/hst/all |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
70 20160723090435 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
71 >: tail -1 /tmp/hst/all |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
72 20160731110639 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
73 >: cd ../../.. |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
74 >: echo 2016-30 20160723090435 20160731110639 >> dates.tsv |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
75 tweaked and sorted in xemacs: |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
76 2016-30 20160723090435 20160731110639 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
77 2017-30 20170720121902 20170729132938 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
78 2018-30 20180715183800 20180723184955 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
79 2018-34 20180814062251 20180822085454 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
80 2019-18 20190418101243 20190426175423 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
81 2019-35 20190817102624 20190826111356 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
82 2020-34 20200803083123 20200815214756 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
83 2021-25 20210612103920 20210625145905 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
84 2023-40 20230921073711 20231005042006 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
85 2023-50 20231128083443 20231212000408 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
86 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
87 Added to status.xlsx in shortened form, with number of days |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
88 8 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
89 9 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
90 8 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
91 8 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
92 8 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
93 9 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
94 12 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
95 13 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
96 15 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
97 15 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
98 |
|
42
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
99 Fill a gap by downloading 2022-33 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
100 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
101 >: for s in 0; do ~/bin/getcc_multi.aws CC-MAIN-2022-33 $s 5; done > /tmp/hst/get_22-33_0.log & |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
102 130 minutes... |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
103 >: for s in 1; do ~/bin/getcc_multi.aws CC-MAIN-2022-33 $s 10; done > /tmp/hst/get_22-33_1.log & |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
104 59 minutes |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
105 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
106 Another day to get to a quarter? |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
107 >: for s in {2..23}; do ~/bin/getcc_multi.aws CC-MAIN-2022-33 $s 10; done > /tmp/hst/get_22-33_2-23.log & |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
108 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
109 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
110 And finally 2015-35 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
111 Fetched in just 2 chunks, 0-9 and 10-99, e.g. |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
112 >: for s in {10..99}; do ~/bin/getcc_multi.aws CC-MAIN-2015-35 $s 10; done > /tmp/hst/get_15-35_10-99.log & |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
113 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
114 Much smaller. |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
115 Compare 2023-40, with 900 files per segment: |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
116 >: lss */orig/warc/*-0023?.* | cut -f 5 -d ' ' | stats |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
117 n = 1000 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
118 min = 1.14775e+09 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
119 max = 1.26702e+09 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
120 sum = 1.20192e+12 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
121 mean = 1.20192e+09 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
122 sd = 2.26049e+07 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
123 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
124 with 2015-35, with 353 files per segment |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
125 >: lss */orig/warc/*-0023?-* | cut -f 5 -d ' ' | stats |
| 43 | 126 n = 1000 |
| 127 min = 1.66471e+08 | |
|
42
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
128 max = 9.6322e+08 |
| 43 | 129 sum = 9.19222e+11 |
| 130 mean = 9.19222e+08 | |
| 131 sd = 8.20542e+07 | |
|
42
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
132 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
133 The min files all come from segment 1440644060633.7, whose files are |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
134 _all_ small: |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
135 >: uz *00123-*.gz | wc -l |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
136 12,759,931 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
137 Compare to 1440644060103.8 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
138 >: zcat *00123-*.gz | wc -l |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
139 75,806,738 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
140 Mystery |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
141 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
142 Also faster |
| 43 | 143 Compare 2022-33: |
|
42
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
144 >: fgrep -h BST /tmp/hst/get_22-33_{2-23,24-49,50-99}.log | cut -f 1-7 -d ' ' | while read s; do if read e; then echo $((($(date --date="$e" +%s) - $(date --date="$s" +%s)) / 60)); fi; done | stats n min max mean sd |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
145 98 19 256 75.1 25.2 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
146 with 2015-35: |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
147 >: fgrep -h BST /tmp/hst/get_15-35_{0-9,10-99}.log | cut -f 1-7 -d ' ' | while read s; do if read e; then echo $((($(date --date="$e" +%s) - $(date --date="$s" +%s)) / 60)); fi; done | stats n min max mean sd |
| 43 | 148 100 15 40 32.6 2.9 |
| 149 | |
| 150 >: echo {000..299} | tr ' ' '\n' | parallel -j 10 'uz cdx-00{}.gz | cut -f 2 -d " " | sort -u > /tmp/hst/2015_{}' & | |
| 151 >: sort --parallel=10 -mu /tmp/hst/2015_??? > /tmp/hst/2015_all | |
| 152 >: head -1 /tmp/hst/2015_all | |
| 153 20150827191534 | |
| 154 >: tail -1 /tmp/hst/2015_all | |
| 155 20150905180914 | |
| 156 >: wc -l /tmp/hst/2015_all | |
| 157 698128 /tmp/hst/2015_all | |
| 158 | |
| 159 What about wet files -- do they include text from pdfs? What about | |
| 160 truncated pdfs? | |
| 161 | |
| 162 >: time for s in 0; do ~/bin/getcc_wet_multi.aws CC-MAIN-2019-35 $s 10; done > /tmp/hst/get_wet_19-35_0.log & | |
| 163 real 26m3.049s | |
| 164 user 0m1.225s | |
| 165 sys 0m1.310s | |
| 166 | |
| 167 In the segment 0 cdx file (!) we find 3747 probable truncations: | |
| 168 >: zgrep -a '"mime-detected": "application/pdf", ' cdx.gz > /tmp/hst/2019-35_seg0_pdf.idx | |
| 169 >: wc -l /tmp/hst/2019-35_seg0_pdf.idx | |
| 170 42345 /tmp/hst/2019-35_seg0_pdf.idx | |
| 171 >: egrep -a '"length": "10....."' /tmp/hst/2019-35_seg0_pdf.idx > /tmp/hst/2019-35_seg0_long_pdf.idx & | |
| 172 >: wc -l < /tmp/hst/2019-35_seg0_long_pdf.idx | |
| 173 3747 | |
| 174 Of which 70 are in file 0: | |
| 175 >: egrep -a '.-00000\.' /tmp/hst/2019-35_seg0_pdf.idx > /tmp/hst/2019-35_seg0_file0_pdf.idx | |
| 176 >: wc -l /tmp/hst/2019-35_seg0_file0_pdf.idx | |
| 177 70 /tmp/hst/2019-35_seg0_file0_pdf.idx | |
| 178 | |
| 179 In segment 0 file 0 we find 70 application/pdf Content-Type headers: | |
| 180 >: ix.py -h -w -x </tmp/hst/2019-35_seg0_file0_pdf.idx |egrep '^(WARC-Target-URI:|Content-Length:) '|cut -f 2 -d ' ' |tr -d '\r'|while read l1; do read uri; read l2; printf '%s\t%s\t%s\n' $l1 $l2 "$uri"; done > ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | |
| 181 >: wc -l < ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | |
| 182 70 | |
| 183 >: head -3 ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | |
| 184 | |
| 185 | |
| 186 Of which 14 are truncated: | |
| 187 >: fgrep -c 1048576 ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | |
| 188 14 | |
| 189 | |
| 190 E.g. | |
| 191 >: fgrep 1048576 ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | head -3 | |
| 192 1049051 1048576 https://en.heks.ch/sites/default/files/documents/2017-09/HEKS_EPER_Mission_Statement_2016_e.pdf | |
| 193 1049469 1048576 https://bmcmicrobiol.biomedcentral.com/track/pdf/10.1186/s12866-017-0951-4 | |
| 194 1048824 1048576 https://citydocs.fcgov.com/?action=cao-cases&cmd=convert&docid=3332339 | |
| 195 | |
| 196 Are any of the pdfs in the corresponding wet file? | |
| 197 | |
| 198 Yes, 2: | |
| 199 >: cut -f 3 ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | fgrep -af - <(uz 1566027313501.0/orig/wet/*-00000.warc.wet.gz) | |
| 200 WARC-Target-URI: http://bdds.deux-sevres.com/recherche/simple/Editeur/2/Belfond/vignette?format=pdf | |
| 201 WARC-Target-URI: https://museum.wrap.gov.tw/GetFile4.ashx?Serial=201609200919D00 | |
| 202 | |
| 203 Is it in fact corresponding? | |
| 204 >: diff -bw <(uz 1566027313501.0/orig/warc/*-00000.warc.gz | egrep -a '^WARC-Target-URI: ' | uniq | head -1000) <(uz 1566027313501.0/orig/wet/*-00000.warc.wet.gz | egrep -a '^WARC-Target-URI: ' | head -1000)|egrep -c '^<' | |
| 205 19 | |
| 206 | |
| 207 So, yes, mostly. .2% are missing | |
| 208 | |
| 209 Just checking the search: | |
| 210 >: cut -f 3 ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | fgrep -af - <(uz 1566027313501.0/orig/warc/*-00000.warc.gz) | wc -l | |
| 211 210 | |
| 212 Correct | |
| 213 | |
| 214 So, what pdfs make it into the WET: | |
| 215 >: cut -f 3 ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | fgrep -af - <(uz 1566027313501.0/orig/wet/*-00000.warc.wet.gz) > ~/results/CC-MAIN-2019-35/s0_file0_pdf.txt | |
| 216 >: wc -l < ~/results/CC-MAIN-2019-35/s0_file0_pdf.txt | |
| 217 2 | |
| 218 >: cut -f 2 -d ' ' ~/results/CC-MAIN-2019-35/s0_file0_pdf.txt | tr -d '\r' | fgrep -f - ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | |
| 219 11588 10913 http://bdds.deux-sevres.com/recherche/simple/Editeur/2/Belfond/vignette?format=pdf | |
| 220 1048979 1048576 https://museum.wrap.gov.tw/GetFile4.ashx?Serial=201609200919D005 | |
| 221 | |
| 222 Here's the short one: | |
| 223 WARC/1.0 | |
| 224 WARC-Type: response | |
| 225 WARC-Date: 2019-08-17T22:40:17Z | |
| 226 WARC-Record-ID: <urn:uuid:ea98167b-c42a-4233-b57e-994aa627e38a> | |
| 227 Content-Length: 11588 | |
| 228 Content-Type: application/http; msgtype=response | |
| 229 WARC-Warcinfo-ID: <urn:uuid:f689f8d0-24f3-4824-9a38-4f3fee422a4e> | |
| 230 WARC-Concurrent-To: <urn:uuid:2d51c956-0012-4d78-affc-8f57fe9d2e15> | |
| 231 WARC-IP-Address: 92.175.114.24 | |
| 232 WARC-Target-URI: http://bdds.deux-sevres.com/recherche/simple/Editeur/2/Belfond/vignette?format=pdf | |
| 233 WARC-Payload-Digest: sha1:7VVIUDQ4Q6XKNOAURYU4VTMRSZNPHDQA | |
| 234 WARC-Block-Digest: sha1:OSTWXLV772XNHS22T4UBSCSJAAXM2J6T | |
| 235 WARC-Identified-Payload-Type: application/pdf | |
| 236 | |
| 237 HTTP/1.1 200 OK | |
| 238 Cache-Control: must-revalidate, post-check=0, pre-check=0,no-cache | |
| 239 Pragma: public,no-cache | |
| 240 Content-Type: application/pdf",text/html; charset=utf-8 | |
| 241 X-Crawler-Content-Encoding: gzip | |
| 242 Expires: 0 | |
| 243 Server: | |
| 244 X-Powered-By: | |
| 245 Set-Cookie: 166d74d734106ba68b20ea303011f622=301619e3fe31ecb98c8473f0ff5f35a2; path=/ | |
| 246 Content-Disposition: attachment; filename="Mdiathque dpartementale des Deux-Svres - Rsultats de la recherche Belfond.pdf" | |
| 247 Content-Transfer-Encoding: binary | |
| 248 P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM" | |
| 249 X-Content-Encoded-By: | |
| 250 X-Powered-By: | |
| 251 Date: Sat, 17 Aug 2019 22:40:16 GMT | |
| 252 X-Crawler-Content-Length: 5448 | |
| 253 Content-Length: 10913 | |
| 254 | |
| 255 %PDF-1.7 | |
| 256 %<E2><E3><CF><D3> | |
| 257 7 0 obj | |
| 258 << /Type /Page /Parent 1 0 R /LastModified (D:20190818004016+02'00') /Resources 2 | |
| 259 0 R /MediaBox [0.000000 0.000000 595.276000 841.890000] /CropBox [0.000000 0.000 | |
| 260 000 595.276000 841.890000] /BleedBox [0.000000 0.000000 595.276000 841.890000] /T | |
| 261 rimBox [0.000000 0.000000 595.276000 841.890000] /ArtBox [0.000000 0.000000 595.2 | |
| 262 76000 841.890000] /Contents 8 0 R /Rotate 0 /Group << /Type /Group /S /Transparen | |
| 263 cy /CS /DeviceRGB >> /PZ 1 >> | |
| 264 endobj | |
| 265 8 0 obj | |
| 266 | |
| 267 >: uz 1566027313501.0/orig/warc/*-00000.warc.gz|tail -n +1823434 | tail -n +24 | head -c 20000 > ~/results/CC-MAIN-2019-35/mediatheque.pdf | |
| 268 >: ps2ascii mediatheque.pdf | |
| 269 Médiathèque départementale des Deux-Sèvres - Résultats de la recherche Belfond | |
| 270 | |
| 271 Médiathèque départementale des Deux-Sèvres - Résultats de | |
| 272 la recherche Belfond | |
| 273 A charge de revanche | |
| 274 Titre : | |
| 275 Auteur : Grippando, James (1958-....) | |
| 276 ... | |
| 277 etc., three pages, no errors | |
| 278 | |
| 279 >: uz 1566027313501.0/orig/warc/*-00000.warc.gz|fgrep -an https://museum.wrap.gov.tw/GetFile4.ashx | |
| 280 38896837:WARC-Target-URI: https://museum.wrap.gov.tw/GetFile4.ashx?Serial=201609200919D005 | |
| 281 38896858:WARC-Target-URI: https://museum.wrap.gov.tw/GetFile4.ashx?Serial=201609200919D005 | |
| 282 38904590:WARC-Target-URI: https://museum.wrap.gov.tw/GetFile4.ashx?Serial=201609200919D005 | |
| 283 >: uz 1566027313501.0/orig/warc/*-00000.warc.gz|tail -n +38896858 | egrep -an '^%%EOF' | |
| 284 27:%%EOF | |
| 285 1114658:%%EOF | |
| 286 1313299:%%EOF | |
| 287 | |
| 288 Hunh? | |
| 289 | |
| 290 >: uz 1566027313501.0/orig/warc/*-00000.warc.gz|tail -n +38896858 | egrep -an '^(%%EOF|WARC)' | head -30 | |
| 291 1:WARC-Target-URI: https://museum.wrap.gov.tw/GetFile4.ashx?Serial=201609200919D005 | |
| 292 2:WARC-Payload-Digest: sha1:SZ53DQQHENC7DDN7GQ5IS7VMEPAXAMBE | |
| 293 3:WARC-Block-Digest: sha1:QTKJA6A7445Z7264K2YAFBUUM2OYH2T2 | |
| 294 4:WARC-Truncated: length | |
| 295 5:WARC-Identified-Payload-Type: application/pdf | |
| 296 27:%%EOF | |
| 297 7725:WARC/1.0 | |
| 298 7726:WARC-Type: metadata | |
| 299 7727:WARC-Date: 2019-08-17T22:59:14Z | |
| 300 7728:WARC-Record-ID: <urn:uuid:77df2747-e567-45d3-8646-3069ae9a9f25> | |
| 301 7731:WARC-Warcinfo-ID: <urn:uuid:f689f8d0-24f3-4824-9a38-4f3fee422a4e> | |
| 302 7732:WARC-Concurrent-To: <urn:uuid:eceb4adc-d81e-4497-82fe-eea61ce171f4> | |
| 303 7733:WARC-Target-URI: https://museum.wrap.gov.tw/GetFile4.ashx?Serial=201609200919D005 | |
| 304 7739:WARC/1.0 | |
| 305 | |
| 306 OK, so indeed truncated after 7700 lines or so... | |
| 307 >: uz 1566027313501.0/orig/warc/*-00000.warc.gz|tail -n +38896858 | tail -n +21 | head -c 1048576 > ~/results/CC-MAIN-2019-35/museum.pdf | |
| 308 >: ps2ascii ~/results/CC-MAIN-2019-35/museum.pdf | |
| 309 **** Error: An error occurred while reading an XREF table. | |
| 310 **** The file has been damaged. | |
| 311 Look in big_pdf? | |
|
44
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
312 |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
313 ====Modify the original CC indexer to write new indices including lastmod===== |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
314 Looks like WarcRecordWriter.write, in |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
315 src/nutch-cc/src/java/org/commoncrawl/util/WarcRecordWriter, is what |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
316 needs to be editted to include LastModified date |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
317 |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
318 To rebuild nutch-cc, particularly to recompile jar files after editting |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
319 anything |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
320 |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
321 >: cd $HHOME/src/nutch-cc |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
322 >: ant |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
323 |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
324 Fixed deprecation bug in WarcCdxWriter.java |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
325 |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
326 Modified src/java/org/commoncrawl/util/WarcCdxWriter.java |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
327 to include lastmod |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
328 |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
329 Can run just one test, which should allow testing this: |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
330 |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
331 >: ant test-core -Dtestcase='TestWarcRecordWriter' |
| 45 | 332 |
| 333 Logic is tricky, and there's no easy way in | |
| 334 | |
| 335 Basically, tools/WarcExport.java is launches a hadoop job based on a | |
| 336 hadoop-runnable WarcExport instance. Hadoop will in due course call | |
| 337 ExportReducer.reduce, which will create an instance of WarcCapture | |
| 338 "for each page capture", and call ExportMapper.context.write with that instance (via | |
| 339 some configuration magic with the hadoop job Context). That in turn | |
| 340 uses (more magic) WarcOutputFormat.getRecordWriter, which | |
| 341 (finally!) calls a previously created WarcRecordWriter | |
| 342 instance.write(the capture). | |
| 343 | |
| 344 So to fake a test case, I need to build | |
| 345 1) a WarcRecordWriter instance | |
| 346 2) a WarcCapture instance | |
| 347 and then invoke 1.write(2) | |
| 348 | |
| 349 Got that working, although still can't figure out where in the normal | |
| 350 flow the metadata entry for Response.CONTENT_TYPE gets set. | |
| 351 | |
| 352 Now, add a test that takes a stream of WARC Response extracts and | |
| 353 rewrites their index entries | |
| 354 | |
| 355 >: head -8804 <(uz /beegfs/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00150.gz)|tail -10| ix.py -h -w -x > /tmp/hst/headers.txt | |
| 356 >: cp /tmp/hst/headers.txt src/test/org/commoncrawl/util/ | |
| 357 >: shuf /tmp/hst/headers.txt > src/test/org/commoncrawl/util/headers_mixed.txt | |
| 358 | |
| 359 Won't quite work :-( | |
| 88 | 360 ==========[end of that line of work?========== |
| 361 At some point the following was explored: | |
| 362 =======<Overtaken, probably>============================ | |
| 363 Try it with the existing _per segment_ index we have for 2019-35 | |
| 364 | |
| 365 Assuming we have to key on segment / file and offset, as reconstructing the | |
| 366 proper index key is such a pain / buggy / is going to change with the year. | |
| 367 | |
| 368 Stay with segment 49 | |
| 369 | |
| 370 >: uz cdx.gz |wc -l | |
| 371 29,870,307 | |
| 372 | |
| 373 >: time uz cdx.gz|egrep -ao ' "length": "[0-9]*", "offset": "[0-9]*"' |wc | |
| 374 29,870,307 119,481,228 1,241,098,122 | |
| 375 = 4 * 29,870,307 | |
| 376 | |
| 377 So no bogons, not _too_ surprising :-) | |
| 378 | |
| 379 Bad news is it's a _big_ file: | |
| 380 | |
| 381 >: ls -lh cdx.gz | |
| 382 -rw-r--r-- 1 hst dc007 2.0G Mar 18 2021 cdx.gz | |
| 383 | |
| 384 So not viable to paste offset as a key and then sort on command line, | |
| 385 or to load it in to python and do the work there... | |
| 386 | |
| 387 Do it per warc file and then merge? | |
| 388 | |
| 389 >: time uz cdx.gz |fgrep -a warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | sort -n -t\" -k28,28 > /tmp/hst/558.warc.cdx | |
| 390 | |
| 391 real 0m23.494s | |
| 392 user 0m14.541s | |
| 393 sys 0m9.158s | |
| 394 | |
| 395 >: wc -l /tmp/hst/558.warc.cdx | |
| 396 53432 /tmp/hst/558.warc.cdx | |
| 397 | |
| 398 >: echo $((600 * 53432)) | |
| 399 32,059,200 | |
| 400 | |
| 401 So, 600 of those, plus approx. same again for extracting, that pbly | |
| 402 _is_ doable in python, not more than 10 hours total, assuming internal | |
| 403 sort and external merge is not too expensive... | |
| 404 | |
| 405 For each segment, suppose we pull out 60 groups of 10 target files | |
| 406 >: time uz cdx.gz |egrep -a warc/CC-MAIN-2019[^-]*-2019[^-]*-0000..warc.gz > /tmp/hst/0000.warc.cdx | |
| 407 | |
| 408 real 0m42.129s | |
| 409 user 0m35.147s | |
| 410 sys 0m9.140s | |
| 411 >: wc -l /tmp/hst/0000.warc.cdx | |
| 412 533150 | |
| 413 | |
| 414 Key it with offset and sort: | |
| 415 | |
| 416 >: time egrep -ao ' "length": "[0-9]*", "offset": "[0-9]*"' /tmp/hst/0000.warc.cdx | cut -f 5 -d ' ' | tr -d \" > /tmp/hst/0000_offsets | |
| 417 | |
| 418 real 0m5.578s | |
| 419 user 0m5.593s | |
| 420 sys 0m0.265s | |
| 421 | |
| 422 >: time paste /tmp/hst/0000_offsets /tmp/hst/0000.warc.cdx |sort -nk1,1 | cut -f 2 > /tmp/hst/0000_sorted.warc.cdx | |
| 423 | |
| 424 real 0m4.185s | |
| 425 user 0m2.001s | |
| 426 sys 0m1.334s | |
| 427 | |
| 428 >: time seq 0 9 | parallel -j 10 "~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-*-*-0000'{}'.warc.gz > /tmp/hst/lm_0000'{}'.tsv" | |
| 429 | |
| 430 real 0m24.610s | |
| 431 user 2m54.146s | |
| 432 sys 0m10.226s | |
| 433 | |
| 434 >: head /tmp/hst/lm_00000.tsv | |
| 435 9398 16432 Mon, 19 Aug 2019 02:44:15 GMT | |
| 436 20796 26748 Tue, 16 Jul 2019 04:39:09 GMT | |
| 437 4648 340633 Fri, 07 Dec 2018 09:05:59 GMT | |
| 438 3465 357109 Sun, 18 Aug 2019 11:48:23 GMT | |
| 439 7450 914189 Mon, 19 Aug 2019 02:50:08 GMT | |
| 440 ... | |
| 441 sing<3956>: fgrep '"length": "9398", "offset": "16432"' /tmp/hst/0000_sorted.warc.cdx | |
| 442 com,roommeme,0401a)/index.phtml?channel=&op=&p=140&put=show&r2= 20190819024416 {"url": "http://0401a.roommeme.com/index.phtml?PUT=SHOW&R2=&OP=&P=140&CHANNEL=", "mime": "text/html", "mime-detected": "text/html", "status": "200", "digest": "5DNDVX5HQBOOBHISSCOI4UBVMUL63L36", "length": "9398", "offset": "16432", "filename": "crawl-data/CC-MAIN-2019-35/segments/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00000.warc.gz", "charset": "Big5", "languages": "zho"} | |
| 443 | |
| 444 bingo | |
| 445 | |
| 446 So, the python code is pretty straightfoward: open the 10 individual | |
| 447 lm-*.tsv outputs into an array, initialise a 10-elt array with the | |
| 448 first line of each and another with its offset, record the | |
| 449 fileno(s) of the lowest offset, then iterate | |
| 450 | |
| 451 read cdx lines and write unchanged until offset = lowest | |
| 452 merge line from fileno and output | |
| 453 remove fileno from list of matches | |
| 454 read and store a new line for fileno [handle EOF] | |
| 455 if list of matches is empty, redo setting of lowest | |
| 456 | |
| 457 Resort the result by actual key | |
| 458 | |
| 459 Meanwhile, get a whole test set: | |
| 460 sbatch --output=slurm_aug_cdx_49_10-599-out --time=01:00:00 --ntasks=10 -c 36 --exclusive $HOME/bin/runme.sh -m 00 59 $PWD -t 18 -b 'export resdir=CC-MAIN-2019-35/aug_cdx/49 | |
| 461 export DEC=$xarg' "export PYTHONPATH=./lib/python/cc:$PYTHONPATH | |
| 462 seq 0 9 | parallel -j 10 \"~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-*-*-00\${DEC}'{}'.warc.gz > \$resdir/00\${DEC}'{}'.tsv\"" | |
| 463 | |
| 464 Actually finished 360 in the hour. | |
| 465 | |
| 466 Leaving | |
| 467 | |
| 468 sbatch --output=slurm_aug_cdx_49_360-599-out --time=01:00:00 --ntasks=10 -c 36 --exclusive $HOME/bin/runme.sh -m 36 59 $PWD -t 18 -b 'export resdir=CC-MAIN-2019-35/aug_cdx/49 | |
| 469 export DEC=$xarg' "export PYTHONPATH=./lib/python/cc:$PYTHONPATH | |
| 470 seq 0 9 | parallel -j 10 \"~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-*-*-00\${DEC}'{}'.warc.gz > \$resdir/00\${DEC}'{}'.tsv\"" | |
| 471 | |
| 472 But something is wrong, the number of jobs is all wrong: | |
| 473 | |
| 474 5>: fgrep -c parallel slurm_aug_cdx_49_0-359-out | |
| 475 741 | |
| 476 sing<4046>: ls -lt CC-MAIN-2019-35/aug_cdx/49/|wc -l | |
| 477 372 | |
| 478 | |
| 479 Every file is being produced twice. | |
| 480 | |
| 481 Took me a while to figure out my own code :-( | |
| 482 | |
| 483 >: sbatch --output=slurm_aug_cdx_49_360-599-out --time=01:00:00 --ntasks=10 -c 36 --exclusive $HOME/bin/runme.sh -m 49 49 $PWD -t 18 -b 'export resdir=CC-MAIN-2019-35/aug_cdx/$xarg | |
| 484 export SEG=$xarg | |
| 485 share_by_task.sh -f "%03g\n" -s 360 599 $n $task > /tmp/hst_$task' -i 'cat /tmp/hst_$task' 'export PYTHONPATH=./lib/python/cc:$PYTHONPATH | |
| 486 ~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/*.$SEG/orig/warc/CC-MAIN-*-*-00${arg}.warc.gz > $resdir/00${arg}.tsv' | |
| 487 | |
| 488 Oops, only 560, not 600 | |
| 489 | |
| 490 Took 3.5 minutes for 200, so call it 10 for 560, so do 6 more in an | |
| 491 hour: | |
| 492 | |
| 493 >: sbatch --output=slurm_aug_cdx_50-55_out --time=01:00:00 --ntasks=10 -c 36 --exclusive $HOME/bin/runme.sh -m 50 55 $PWD -t 18 -b 'export resdir=CC-MAIN-2019-35/aug_cdx/$xarg | |
| 494 mkdir -p $resdir | |
| 495 > export SEG=$xarg | |
| 496 share_by_task.sh -f "%03g\n" -s 360 599 $n $task > /tmp/hst_$task' -i 'cat /tmp/hst_$task' 'export PYTHONPATH=./lib/python/cc:$PYTHONPATH | |
| 497 ~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/*.$SEG/orig/warc/CC-MAIN-*-*-00${arg}.warc.gz > $resdir/00${arg}.tsv' | |
| 498 | |
| 499 >: tail slurm_aug_cdx_50-55_out | |
| 500 ... | |
| 501 Wed Oct 9 22:25:47 BST 2024 Finished 55 | |
| 502 >: head -1 slurm_aug_cdx_50-55_out | |
| 503 Wed Oct 9 21:29:43 BST | |
| 504 56:04 | |
| 505 | |
| 506 >: du -s CC-MAIN-2019-35/aug_cdx | |
| 507 1,902,916 | |
| 508 | |
| 509 Not bad, so order 20MB for the whole thing | |
| 510 | |
| 511 Next step, compare to my existing cdx with timestamp | |
| 512 | |
| 513 First check looks about right: | |
| 514 | |
| 515 [cd .../warc_lmhx] | |
| 516 >: seq --format='%03g' 0 299 > /tmp/hst/cdx_nums | |
| 517 >: parallel -j 20 -a /tmp/hst/cdx_nums 'uz idx/cdx-00{}.gz | egrep -o "\"filename\": \"crawl-data/CC-MAIN-2019-35/segments/[^.]*[.]50.*\"lastmod\":" | sed "s/^.*-00//;s/^\(...\).*/\1/"| sus > /tmp/hst/checkseg_50_{}' | |
| 518 | |
| 519 [cd .../aug_cdx/50] | |
| 520 >: wc -l 00123.tsv | |
| 521 9333 | |
| 522 >: egrep -h '123$' /tmp/hst/checkseg_50_??? | acut 1 | btot | |
| 523 9300 | |
| 524 >: wc -l 00400.tsv | |
| 525 9477 00400.tsv | |
| 526 >: egrep -h '400$' /tmp/hst/checkseg_50_??? | acut 1 | btot | |
| 527 9439 | |
| 528 | |
| 529 Difference is presumable the bogus timestamps aren't in the augmented | |
| 530 cdx as shipped. | |
| 531 | |
| 532 Note that the following 'bad' kind of timestamp is fixed before | |
| 533 sort_date.py does its thing: | |
| 534 | |
| 535 ... sort_date.sh <(uz $arg/*00???.warc.gz | '"fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/')"' >$arg/ks.tsv | |
| 536 | |
| 537 | |
| 538 >: egrep -c '[^ ]GMT$' 50/00123.tsv | |
| 539 22 | |
| 540 >: egrep -c '[^ ]GMT$' 50/00400.tsv | |
| 541 14 | |
| 542 | |
| 543 >: PYTHONPATH=~/.local/lib/python3.9/site-packages:$PYTHONPATH sort_date.sh <(uz ../warc_lmhx/50/*00123.warc.gz | fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2> /tmp/hst/123_errs | wc -l | |
| 544 9300 | |
| 545 >: fgrep -c Invalid /tmp/hst/123_errs | |
| 546 33 | |
| 547 >: PYTHONPATH=~/.local/lib/python3.9/site-packages:$PYTHONPATH sort_date.sh <(uz ../warc_lmhx/50/*00400.warc.gz | fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2> /tmp/hst/400_errs | wc -l | |
| 548 9439 | |
| 549 >: fgrep -c Invalid /tmp/hst/400_errs | |
| 550 38 | |
| 551 | |
| 552 All good. | |
| 553 | |
| 554 But | |
| 555 >: seq --format='%03g' 0 559 > /tmp/hst/warc_nums | |
| 556 >: xx () { | |
| 557 r=$(diff -bw | |
| 558 <(echo $(( | |
| 559 $(sort_date.sh <(uz ../warc_lmhx/50/*00$1.warc.gz | | |
| 560 fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2>/tmp/hst/ec_$1 |wc -l) | |
| 561 + | |
| 562 $(fgrep -c Invalid /tmp/hst/ec_$1)))) | |
| 563 <(wc -l < 50/00$1.tsv)) | |
| 564 if [ "$r" ] | |
| 565 then printf "%s:\n%s\n" $2 "$r" | |
| 566 fi | |
| 567 } | |
| 568 >: parallel -j 20 -a /tmp/hst/warc_nums xx '{}' '$(({#} - 1))' | tee /tmp/hst/aug_bugs | |
| 569 >: fgrep -c 1c1 /tmp/hst/aug_bugs | |
| 570 77 | |
| 571 >: wc -l < /tmp/hst/aug_bugs | |
| 572 385 | |
| 573 >: echo $((77 * 5)) | |
| 574 385 | |
| 575 | |
| 576 OK, there are a few other error messages from date conversion | |
| 577 >: xx () { r=$(diff -bw <(echo $(($(sort_date.sh <(uz ../warc_lmhx/50/*00$1.warc.gz | fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2>/tmp/hst/ec_$1 |wc -l) + $(egrep -c 'Invalid|must be in|out of range' /tmp/hst/ec_$1)))) <(wc -l < 50/00$1.tsv)); if [ "$r" ]; then printf "%s:\n%s\n" $2 "$r"; fi; } | |
| 578 >: parallel -j 20 -a /tmp/hst/warc_nums xx '{}' '$(({#} - 1))' | tee /tmp/hst/aug_bugs2 | |
| 579 [nothing] | |
| 580 | |
| 581 So, I think we can believe we're OK | |
| 582 But 7 is better than 1: | |
| 583 >: xx () { r=$(diff -bw <(echo $(($(sort_date.sh <(uz ../warc_lmhx/$3/*00$1.warc.gz | fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2>/tmp/hst/ec_$1 |wc -l) + $(egrep -c 'Invalid|must be in|out of range' /tmp/hst/ec_$1)))) <(wc -l < $3/00$1.tsv)); if [ "$r" ]; then printf "%s:\n%s\n" $2 "$r"; fi; } | |
| 584 >: for s in 49 {51..55}; do parallel -j 20 -a /tmp/hst/warc_nums xx '{}' '$(({#} - 1))' $s | tee /tmp/hst/aug_bugs_$s; done | |
| 585 [nothing] | |
| 586 ===========</Overtaken>============== | |
| 587 | |
| 45 | 588 How do We reconstruct the Warc filename, offset and length from the |
| 589 original index? | |
| 590 | |
| 591 Well, we can find a .warc.gz records! | |
| 592 Thanks to https://stackoverflow.com/a/37042747/2595465 | |
| 593 | |
| 594 >: ~/lib/python/unpackz.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz > /tmp/hst/recs.txt | |
| 595 | |
| 596 Nearly working, got 1/3rd of the way through a single WARC and then failed: | |
| 597 | |
| 598 >: n=0 && ~/lib/python/unpackz.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz 2>/tmp/hst/tt.txt|while read o l; do echo $((n+=1)); echo $o $l >> /tmp/hst/r3a; ix.py $l $o CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz| wc -l; done | |
| 599 ... | |
| 600 20 | |
| 601 10215 | |
| 602 CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | |
| 603 Process fail: Compressed file ended before the end-of-stream marker was reached, input: | |
| 604 length=10762, offset=60784640, file=/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | |
| 605 | |
| 606 >: head -10217 /tmp/hst/r3a | tail -4 | |
| 607 60784173 467 | |
| 608 60784640 10762 | |
| 609 60795402 463 | |
| 610 60795865 460 | |
| 611 >: ix.py 467 60784173 CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz|fgrep Target | |
| 612 WARC-Target-URI: http://drycarerestoration.co/corner-furniture-piece/unique-corner-decoration-pieces-or-corner-furniture-pieces-corner-corner-furniture-piece-corner-furniture-pieces-bedroom/ | |
| 613 | |
| 614 >: zcat /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/cdx/warc/cdx.gz | |
| 615 ... | |
| 616 co,drycarerestoration)/corner-furniture-piece/unique-corner-decoration-pieces-or-corner-furniture-pieces-corner-corner-furniture-piece-corner-furniture-pieces-bedroom 20190819020224 {"url": "http://drycarerestoration.co/corner-furniture-piece/unique-corner-decoration-pieces-or-corner-furniture-pieces-corner-corner-furniture-piece-corner-furniture-pieces-bedroom/", "mime": "text/html", "mime-detected": "text/html", "status": "200", "digest": "DTKGJL45XQDXUS7PTXPYR6POMPLG46RZ", "length": "2570", "offset": "60784640", "filename": "crawl-data/CC-MAIN-2019-35/segments/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz", "charset": "UTF-8", "languages": "eng"} | |
| 617 >: ix.py 2570 60784640 CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz|less | |
| 618 >: echo $((10762 - 2570)) | |
| 619 8192 | |
| 620 | |
| 621 Ah, the error I was dreading :-( I _think_ this happens when an | |
| 622 individual record ends exactly on a 8K boundary. | |
| 623 | |
| 624 Yes: | |
| 625 | |
| 626 >: echo $((60784640 % 8192)) | |
| 627 0 | |
| 628 | |
| 46 | 629 Even with buffer 1MB: |
| 630 21 | |
| 631 160245 | |
| 632 CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | |
| 633 Process fail: Compressed file ended before the end-of-stream marker was reached, input: | |
| 634 length=8415, offset=1059033915, file=/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | |
| 635 0 | |
| 636 160246 | |
| 637 | |
| 638 >: tail -60 /tmp/hst/r3b|head -20 | |
| 639 1059013061 423 | |
| 640 1059013484 7218 | |
| 641 1059020702 425 | |
| 642 1059021127 424 | |
| 643 1059021551 11471 | |
| 644 1059033022 426 | |
| 645 1059033448g 467 | |
| 646 1059033915 8415 | |
| 647 | |
| 648 Argh. This is at the _same_ point (before 51 fails before EOF). Ah, | |
| 649 maybe that's the point -- this is the last read before EOF, and it's | |
| 650 not a full buffer! | |
| 651 | |
| 652 >: ix.py 467 1059033448 CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz|less | |
| 653 ... | |
| 654 WARC-Target-URI: https://zowiecarrpsychicmedium.com/tag/oracle/ | |
| 655 | |
| 656 Reran with more instrumentation, took at least all day: | |
| 657 | |
| 658 >: n=0 && ~/lib/python/unpackz.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz 2> /tmp/hst/r3e_err.txt | while read o l; do | |
| 659 echo $((n+=1)); echo $o $l >> /tmp/hst/r3e_val; ix.py $l $o CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | wc -l; | |
| 660 done > /tmp/hst/r3e_log 2>&1 | |
| 661 >: wc -l /tmp/hst/r3e_err.txt | |
| 662 160296 /tmp/hst/r3e_err.txt | |
| 663 >: tail -60 /tmp/hst/r3e_err.txt|cat -n | grep -C2 True\ True | |
| 664 7 b 28738 28738 28312 426 False False | |
| 665 8 b 28312 28312 27845 467 False False | |
| 666 9 b 27845 378162 369747 8415 True True < this is the first hit the last | |
| 667 (partial) block | |
| 668 10 b 369747 369747 369312 435 False True | |
| 669 11 b 369312 369312 368878 434 False True | |
| 670 | |
| 671 >: tail -55 /tmp/hst/r3e_val | head -3 | |
| 672 1059033022 426 | |
| 673 1059033448 467 | |
| 674 1059033915 8415 | |
| 675 >: dd ibs=1 skip=1059033022 count=426 if=/beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz of=/dev/stdout | uz -t | |
| 676 ... | |
| 677 426 bytes copied, 0.00468243 s, 91.0 kB/s | |
| 678 sing<3411>: dd ibs=1 skip=1059033448 count=467 if=/beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz of=/dev/stdout | uz -t | |
| 679 ... | |
| 680 467 bytes copied, 0.00382692 s, 122 kB/s | |
| 681 sing<3412>: dd ibs=1 skip=1059033915 count=8415 if=/beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz of=/dev/stdout | uz -t | |
| 682 igzip: Error (null) does not contain a complete gzip file | |
| 683 ... | |
| 684 8415 bytes (8.4 kB, 8.2 KiB) copied, 0.00968889 s, 869 kB/s | |
| 685 | |
| 686 So, tried one change to use the actually size rather than BUFSIZE at | |
| 687 one point, seems to work now: | |
| 688 | |
| 689 >: time ~/lib/python/unpackz.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz 2> /tmp/hst/r3f_err.txt | tee /tmp/hst/r3f_val | while read l o; do printf '%s\t%s\t%s\n' $l $o 'CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz'; | |
| 690 done 2>&1 | tee /tmp/hst/r3f_log | ix.py -w | egrep -c '^WARC/1\.0' | |
| 691 160296 | |
| 692 real 3m48.393s | |
| 693 user 0m47.997s | |
| 694 sys 0m26.641s | |
| 695 | |
| 696 >: tail /tmp/hst/r3f_val | |
| 697 10851 1059370472 | |
| 698 475 1059381323 | |
| 699 444 1059381798 | |
| 700 22437 1059382242 | |
| 701 447 1059404679 | |
| 702 506 1059405126 | |
| 703 15183 1059405632 | |
| 704 471 1059420815 | |
| 705 457 1059421286 | |
| 706 17754 1059421743 | |
| 707 | |
| 708 >: wc -l /tmp/hst/*_val | |
| 709 171 /tmp/hst/r3d_val | |
| 710 160297 /tmp/hst/r3e_val | |
| 711 160296 /tmp/hst/r3f_val | |
| 712 320764 total | |
| 713 >: uz /tmp/hst/head.warc.gz |egrep -c '^WARC/1\.0.$' | |
| 714 171 | |
| 715 >: tail -n 3 /tmp/hst/*_val | |
| 716 ==> /tmp/hst/r3d_val <== | |
| 717 454 1351795 | |
| 718 414 1352249 | |
| 719 0 1352663 [so the 171 above is bogus, and we're missing one] | |
| 720 | |
| 721 ==> /tmp/hst/r3e_val <== | |
| 722 1059393441 457 | |
| 723 1059393898 17754 | |
| 724 0 [likewise bogus, so see below] | |
| 725 | |
| 726 ==> /tmp/hst/r3f_val <== | |
| 727 471 1059420815 | |
| 728 457 1059421286 | |
| 729 17754 1059421743 [better, but still one missing] | |
| 730 >: uz /tmp/hst/head.warc.gz |egrep '^WARC-Type: ' | tee >(wc -l 1>&2) | tail -4 | |
| 731 WARC-Type: response | |
| 732 WARC-Type: metadata | |
| 733 WARC-Type: request | |
| 734 WARC-Type: response [missing] | |
| 735 171 | |
| 736 >: ls -lt /tmp/hst/*_val | |
| 737 -rw-r--r-- 1 hst dc007 1977 Sep 29 09:27 /tmp/hst/r3d_val | |
| 738 -rw-r--r-- 1 hst dc007 2319237 Sep 28 14:28 /tmp/hst/r3f_val | |
| 739 -rw-r--r-- 1 hst dc007 2319238 Sep 27 19:41 /tmp/hst/r3e_val | |
| 740 >: ls -l ~/lib/python/unpackz.py | |
| 741 -rwxr-xr-x 1 hst dc007 1821 Sep 28 15:13 .../dc007/hst/lib/python/unpackz.py | |
| 742 So e and f are stale, rerun | |
| 743 >: time ~/lib/python/unpackz.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz 2>/tmp/hst/r3f_err.txt| tee /tmp/hst/r3f_val|while read l o; do printf '%s\t%s\t%s\n' $l $o 'CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz' ;done |& tee /tmp/hst/r3f_log |ix.py -w |egrep '^WARC-Type: ' | tail -4 & | |
| 744 >: Reading length, offset, filename tab-delimited triples from stdin... | |
| 745 WARC-Type: response | |
| 746 WARC-Type: metadata | |
| 747 WARC-Type: request | |
| 748 WARC-Type: response | |
| 749 | |
| 750 real 3m49.760s | |
| 751 user 0m47.180s | |
| 752 sys 0m32.218s | |
| 753 So missing the final metadata... | |
| 754 Back to head.warc.gz, with debug info | |
| 755 | |
| 756 >: n=0 && ~/lib/python/unpackz.py /tmp/hst/head.warc.gz 2>/tmp/hst/ttd.txt|while read l o; do echo $((n+=1)); echo $l $o >> /tmp/hst/r3d_val; dd ibs=1 skip=$o count=$l if=/tmp/hst/head.warc.gz of=/dev/stdout 2>/tmp/hst/r3d_ido| uz -t ; done >/tmp/hst/r3d_log 2>&1 | |
| 757 >: tail -2 /tmp/hst/r3d_log | |
| 758 171 | |
| 759 igzip: Error invalid gzip header found for file (null) | |
| 760 >: tail -n 3 /tmp/hst/ttd.txt /tmp/hst/r3d_val | |
| 761 ==> /tmp/hst/ttd.txt <== | |
| 762 b 9697 9697 9243 454 False True | |
| 763 b 9243 9243 8829 414 False True | |
| 764 n 8829 | |
| 765 | |
| 766 ==> /tmp/hst/r3d_val <== | |
| 767 454 1351795 | |
| 768 414 1352249 | |
| 769 0 1352663 | |
| 770 | |
| 771 >: cat -n /tmp/hst/r3f_val | head -172 | tail -4 | |
| 772 169 454 1351795 | |
| 773 170 414 1352249 | |
| 774 171 8829 1352663 | |
| 775 172 446 1361492 | |
| 776 | |
| 777 Fixed, maybe | |
| 778 | |
| 779 >: tail -n 3 /tmp/hst/r3d_log /tmp/hst/r3d_val | |
| 780 ==> /tmp/hst/r3d_log <== | |
| 781 169 | |
| 782 170 | |
| 783 171 | |
| 784 | |
| 785 ==> /tmp/hst/r3d_val <== | |
| 786 454 1351795 | |
| 787 414 1352249 | |
| 788 8829 1352663 | |
| 789 | |
| 790 Yes! | |
| 791 | |
| 792 >: time ~/lib/python/unpackz.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz 2>/tmp/hst/r3f_err| tee /tmp/hst/r3f_val|while read l o; do printf '%s\t%s\t%s\n' $l $o 'CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz' ;done |& tee /tmp/hst/r3f_log |ix.py -w |egrep '^WARC-Type: ' | tail -4 | |
| 793 Reading length, offset, filename tab-delimited triples from stdin... | |
| 794 WARC-Type: metadata | |
| 795 WARC-Type: request | |
| 796 WARC-Type: response | |
| 797 WARC-Type: metadata | |
| 798 | |
| 799 real 3m26.042s | |
| 800 user 0m44.167s | |
| 801 sys 0m24.716s | |
| 802 >: tail -n 3 /tmp/hst/r3f* | |
| 803 ==> /tmp/hst/r3f_err <== | |
| 804 | |
| 805 ==> /tmp/hst/r3f_val <== | |
| 806 457 1059421286 | |
| 807 17754 1059421743 | |
| 808 425 1059439497 | |
| 809 | |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
810 Doubling buffer size doesn't speed up |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
811 >: time ~/lib/python/unpackz.py -b $((2 * 1024 * 1024)) /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz 2>/tmp/hst/r3g_err| tee /tmp/hst/r3g_val|while read l o; do printf '%s\t%s\t%s\n' $l $o 'CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz' ;done |& tee /tmp/hst/r3g_log |ix.py -w |egrep '^WARC-Type: ' | tail -4 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
812 Reading length, offset, filename tab-delimited triples from stdin... |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
813 WARC-Type: metadata |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
814 WARC-Type: request |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
815 WARC-Type: response |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
816 WARC-Type: metadata |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
817 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
818 real 3m34.519s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
819 user 0m52.312s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
820 sys 0m24.875s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
821 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
822 Tried using FileIO.readinto([a fixed buffer]), but didn't immediately |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
823 work. Abandoned because I still don't understand how zlib.decompress |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
824 works at all... |
| 88 | 825 ============Abandon the use of ix.py in order to work through index |
| 826 files using ix.py to find the corresponding headers and find LM there or not========= | |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
827 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
828 Time to convert unpackz to a library which takes a callback |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
829 alternative to an output file -- Done |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
830 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
831 W/o using callback, timing and structure for what we need for |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
832 re-indexing task looks encouraging: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
833 >: time ~/lib/python/cc/unpackz.py -o /dev/stdout /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz |egrep -aA20 '^WARC-Type: response' | cut -f 1 -d ' ' | egrep -a '^WARC-' |sus | tee >(wc -l 1>&2) |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
834 52468 WARC-Block-Digest: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
835 52468 WARC-Concurrent-To: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
836 52468 WARC-Date: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
837 52468 WARC-Identified-Payload-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
838 52468 WARC-IP-Address: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
839 52468 WARC-Payload-Digest: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
840 52468 WARC-Record-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
841 52468 WARC-Target-URI: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
842 52468 WARC-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
843 52468 WARC-Warcinfo-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
844 236 WARC-Truncated: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
845 11 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
846 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
847 real 0m20.308s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
848 user 0m19.720s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
849 sys 0m4.505s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
850 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
851 Whole thing, with no pre-filtering: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
852 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
853 >: time ~/lib/python/cc/unpackz.py -o /dev/stdout /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | cut -f 1 -d ' ' | egrep -a '^(WARC-|Content-|Last-Modified)' |sus | tee >(wc -l 1>&2) |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
854 211794 Content-Length: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
855 211162 Content-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
856 159323 WARC-Target-URI: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
857 159311 WARC-Warcinfo-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
858 159301 WARC-Record-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
859 159299 WARC-Date: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
860 159297 WARC-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
861 105901 WARC-Concurrent-To: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
862 105896 WARC-IP-Address: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
863 52484 WARC-Block-Digest: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
864 52484 WARC-Identified-Payload-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
865 52482 WARC-Payload-Digest: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
866 9239 Last-Modified: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
867 3941 Content-Language: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
868 2262 Content-Security-Policy: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
869 642 Content-language: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
870 326 Content-Security-Policy-Report-Only: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
871 238 WARC-Truncated: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
872 114 Content-Disposition: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
873 352 Content-*: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
874 1 WARC-Filename: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
875 42 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
876 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
877 real 0m30.896s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
878 user 0m37.335s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
879 sys 0m7.542s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
880 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
881 First 51 after WARC-Type: response |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
882 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
883 >: time ~/lib/python/cc/unpackz.py -o /dev/stdout /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz |egrep -aA50 '^WARC-Type: response' | cut -f 1 -d ' ' | egrep -a '^(WARC-|Content-|Last-Modified)' |sus | tee >(wc -l 1>&2) |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
884 106775 Content-Length: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
885 106485 Content-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
886 55215 WARC-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
887 55123 WARC-Date: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
888 54988 WARC-Record-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
889 54551 WARC-Warcinfo-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
890 54246 WARC-Target-URI: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
891 54025 WARC-Concurrent-To: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
892 52806 WARC-IP-Address: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
893 52468 WARC-Block-Digest: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
894 52468 WARC-Identified-Payload-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
895 52468 WARC-Payload-Digest: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
896 9230 Last-Modified: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
897 3938 Content-Language: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
898 2261 Content-Security-Policy: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
899 639 Content-language: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
900 324 Content-Security-Policy-Report-Only: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
901 236 WARC-Truncated: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
902 114 Content-Disposition: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
903 342 Content-*: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
904 41 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
905 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
906 real 0m21.483s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
907 user 0m22.372s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
908 sys 0m5.400s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
909 |
| 58 | 910 So, not worth the risk, let's try python: cdx_extras implements a |
| 911 callback for unpackz that outputs the LM header if it's there | |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
912 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
913 >: time ~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz|wc -l |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
914 9238 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
915 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
916 real 0m25.426s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
917 user 0m23.201s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
918 sys 0m0.711s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
919 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
920 Looks good, but why 9238 instead of 9239??? |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
921 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
922 >: ~/lib/python/cc/unpackz.py -o /dev/stdout /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | egrep -a '^Last-Modified: ' > /tmp/hst/lmo.tsv |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
923 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
924 Argh. Serious bug in unpackz, wasn't handline cross-buffer-boundary |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
925 records correctly. Fixed. Redoing the above... |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
926 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
927 No pre-filter: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
928 >: uz /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz|egrep -c '^WARC/1\.0.$' |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
929 160297 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
930 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
931 >: time ~/lib/python/cc/unpackz.py -o /dev/stdout /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | cut -f 1 -d ' ' | egrep -a '^(WARC-|Content-|Last-Modified)' |sus | tee >(wc -l 1>&2) |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
932 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
933 213719 Content-Length: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
934 213088 Content-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
935 160297 WARC-Date: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
936 160297 WARC-Record-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
937 160297 WARC-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
938 160296 WARC-Target-URI: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
939 160296 WARC-Warcinfo-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
940 106864 WARC-Concurrent-To: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
941 106864 WARC-IP-Address: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
942 53432 WARC-Block-Digest: [consistent with 106297 == (3 * 53432) + 1] |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
943 53432 WARC-Identified-Payload-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
944 53432 WARC-Payload-Digest: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
945 9430 Last-Modified: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
946 4006 Content-Language: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
947 2325 Content-Security-Policy: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
948 653 Content-language: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
949 331 Content-Security-Policy-Report-Only: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
950 298 WARC-Truncated: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
951 128 Content-Disposition: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
952 83 Content-Location: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
953 67 Content-type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
954 51 Content-MD5: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
955 45 Content-Script-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
956 42 Content-Style-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
957 31 Content-Transfer-Encoding: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
958 13 Content-disposition: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
959 8 Content-Md5: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
960 5 Content-Description: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
961 5 Content-script-type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
962 5 Content-style-type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
963 3 Content-transfer-encoding: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
964 2 Content-Encoding-handler: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
965 1 Content-DocumentTitle: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
966 1 Content-Hash: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
967 1 Content-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
968 1 Content-Legth: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
969 1 Content-length: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
970 1 Content-Range: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
971 1 Content-Secure-Policy: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
972 1 Content-security-policy: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
973 1 Content-Type-Options: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
974 1 WARC-Filename: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
975 42 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
976 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
977 real 0m28.876s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
978 user 0m35.703s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
979 sys 0m6.976s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
980 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
981 >: ~/lib/python/cc/unpackz.py -o /dev/stdout /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | egrep -a '^Last-Modified: ' > /tmp/hst/lmo.tsv |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
982 >: wc -l /tmp/hst/lmo.tsv |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
983 9430 /tmp/hst/lmo.tsv |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
984 >: time ~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz > /tmp/hst/lm.tsv |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
985 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
986 real 0m17.191s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
987 user 0m15.739s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
988 sys 0m0.594s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
989 >: wc -l /tmp/hst/lm.tsv |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
990 9423 /tmp/hst/lm.tsv |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
991 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
992 >: diff <(sed 's/^Last-Modified: //' /tmp/hst/lmo.tsv | tr -d '\r') <(cut -f 3 /tmp/hst/lm.tsv) |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
993 853d852 |
| 58 | 994 < Mon, 19 Aug 2019 01:46:49 GMT [in XML comment at very end of xHTML] |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
995 4058d4056 |
| 58 | 996 < Tue, 03 Nov 2015 21:31:18 GMT<br /> [in an HTML table] |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
997 4405d4402 |
| 58 | 998 < Mon, 19 Aug 2019 01:54:52 GMT [double lm] |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
999 5237,5238d5233 |
| 58 | 1000 < 3 [bogus extension lines to preceding LM] |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
1001 < Asia/Amman |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
1002 7009d7003 |
| 58 | 1003 < Mon, 19 Aug 2019 02:34:20 GMT [in XML comment at very end of xHTML] |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
1004 9198d9191 |
| 58 | 1005 < Mon, 19 Aug 2019 02:14:49 GMT [in XML comment at very end of xHTML] |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
1006 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
1007 All good. The only implausable case is |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
1008 < Mon, 19 Aug 2019 01:54:52 GMT |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
1009 which turns out to be a case of two Last-Modified headers in the same |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
1010 the same response record's HTTP headers. RFCs 2616 and 7230 rule it |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
1011 out but neither specifies a recovery, so first-wins is as good as |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
1012 anything, and indeed 6797 specifies that. |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
1013 |
| 48 | 1014 Start looking at how we do the merge of cdx_extras.py with existing index |
| 1015 | |
| 58 | 1016 ====2024-12-19==== |
| 1017 | |
| 1018 The above test shows 17.6% of entries have an LM value | |
| 1019 For a 3 billion entry dataset, than means 530 million LM entries, call | |
| 1020 this n. | |
| 1021 | |
| 1022 Sizes? For a 10% error rate, we need m bits = -n * ln(.1) / ln(2)^2 | |
| 1023 | |
| 1024 (- (/ (* n (log .1)) (* (log 2)(log 2))) = 2,535,559,358 =~ 320MB | |
| 1025 | |
| 1026 That's too much :-) Per segment, that becomes possible? | |
| 1027 25,355,594 bits =~ 3.2MB | |
| 1028 | |
| 1029 But maybe it's _not_ too much. One of the python implementations I | |
| 1030 saw uses mmap: | |
| 1031 | |
| 1032 https://github.com/prashnts/pybloomfiltermmap3 | |
| 1033 | |
| 1034 Build a Bloom filter with all the URIs whose entries have LM value | |
| 1035 _and_ a python hashtable mapping from URI to LM and offset (is that | |
| 1036 enough for deduping?) | |
| 1037 Rewrite one index file at a time | |
| 1038 Probe with each URI, if positive | |
| 1039 look up in hashtable and use if found | |
| 1040 | |
| 1041 >: wc -l ks*.tsv | |
| 1042 52369734 ks_0-9.tsv | |
| 1043 52489306 ks_10-19.tsv | |
| 1044 52381115 ks_20-29.tsv | |
| 1045 52438862 ks_30-39.tsv | |
| 1046 52512044 ks_40-49.tsv | |
| 1047 52476964 ks_50-59.tsv | |
| 1048 52317116 ks_60-69.tsv | |
| 1049 52200680 ks_70-79.tsv | |
| 1050 52382426 ks_80-89.tsv | |
| 1051 52295136 ks_90-99.tsv | |
| 1052 523863383 total | |
| 68 | 1053 Later saved the above in kslines.tsv |
| 58 | 1054 |
| 1055 >>> from pybloomfilter import BloomFilter | |
| 1056 >>> f=BloomFilter(523863383,0.1,'/tmp/hst/uris.bloom') | |
| 1057 >>> def bff(f,fn): | |
| 1058 ... with open(fn) as uf: | |
| 1059 ... while (l:=uf.readline()): | |
| 1060 ... f.add(l.split('\t')[2]) | |
| 1061 ... | |
| 1062 >>> timeit.timeit("bff(f,'/dev/null')",number=1,globals=globals()) | |
| 1063 0.00012309104204177856 | |
| 1064 >>> timeit.timeit("bff(f,'/work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv')",number=1,globals=globals()) | |
| 1065 77.57737312093377 | |
| 1066 >>> 'http://71.43.189.10/dermorph' in f | |
| 1067 False | |
| 1068 >>> 'http://71.43.189.10/dermorph/' in f | |
| 1069 True | |
| 1070 >>> timeit.timeit("'http://71.43.189.10/dermorph/' in f",number=100000,globals=globals()) | |
| 1071 0.02377822808921337 | |
| 1072 >>> timeit.timeit("'http://71.43.189.10/dermorph' in f",number=100000,globals=globals()) | |
| 1073 0.019318239763379097 | |
| 1074 | |
| 1075 _That's_ encouraging... | |
| 1076 Be sure to f.close() | |
| 1077 Use BloomFilter.open for an existing bloom file | |
| 1078 Copying a file from /tmp to work/... still gives good (quick) lookup, | |
| 1079 but _creating and filling_ a file on work/... takes ... I stopped | |
| 1080 waiting after an hour or so. | |
| 59 | 1081 |
| 1082 How much bigger is .05 false positive? | |
| 1083 Less than expected: | |
| 1084 >: ls -l /tmp/hst | |
| 1085 -rwxr-xr-x 1 hst dc007 408301988 Jan 1 16:52 uris_20.bloom | |
| 1086 -rwxr-xr-x 1 hst dc007 313830100 Jan 1 15:04 uris.bloom | |
| 1087 And still same (?) fill time: | |
| 1088 >>> g=BloomFilter(523863383,0.05,'/tmp/hst/uris_20.bloom') | |
| 1089 >>> T=timeit.Timer("bff(g,'/work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv')",globals=globals()) | |
| 1090 >>> T.repeat(3,number=1) | |
| 1091 [89.64385064691305, 90.9979057777673, 83.9632708914578] | |
| 1092 Build a test harness wrt the python dict I'm going to need... | |
| 1093 Can't immediately find a way to optimise a dict to have umpty millions | |
| 1094 of entries... | |
| 1095 >: cat /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv|~/lib/python/cc/lmh/test.py -n 1000000 -r 5 | |
| 1096 1000002 | |
| 1097 1000002 | |
| 1098 1000002 | |
| 1099 1000002 | |
| 1100 1000002 | |
| 1101 [1.229693355038762, 1.3374222852289677, 1.3509841952472925, 1.080365838482976, 1.1893387716263533] | |
| 1102 Full as-it-were segment: | |
| 1103 >: ~/lib/python/cc/lmh/test.py -n 6000000 -r 5 -f /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv | |
| 1104 6000002 | |
| 1105 6000002 | |
| 1106 6000002 | |
| 1107 6000002 | |
| 1108 6000002 | |
| 1109 [7.250897390767932, 7.237801244482398, 7.239673590287566, 7.32976414449513, 7.23588689416647] | |
| 1110 Full 10th of the data: | |
| 1111 >: ~/lib/python/cc/lmh/test.py -r 3 -f /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv | |
| 1112 52369734 | |
| 1113 52369734 | |
| 1114 52369734 | |
| 1115 [69.63967163302004, 69.09140252694488, 66.49750975705683] | |
| 1116 That's tolerable. | |
| 60 | 1117 >: ~/lib/python/cc/lmh/test_hash.py -r 3 -f /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv |
| 59 | 1118 52369734 |
| 1119 52369734 | |
| 1120 52369734 | |
| 1121 [64.51177835091949, 71.6610240675509, 67.74966451153159] | |
| 1122 [0.0034751780331134796, 0.0034532323479652405, 0.0033454522490501404] | |
| 1123 Last line is 100000 lookups. | |
| 60 | 1124 |
| 1125 So, try a test: | |
| 1126 >: time ~/lib/python/cc/lmh/test_hash.py -r 1 -p results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.pickle -f /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv | |
| 1127 52369734 | |
| 1128 [70.98342595621943] | |
| 1129 [0.0037928372621536255] | |
| 1130 | |
| 1131 real 1m51.456s | |
| 1132 user 1m32.901s | |
| 1133 sys 0m17.937s | |
| 1134 >: ls -lh results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.pickle | |
| 1135 -rw-r--r-- 1 hst dc007 5.5G Jan 2 12:19 results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.pickle | |
| 1136 cdx_out.write(b' ') | |
| 1137 cdx_out.write(b' ') | |
| 1138 >: time ~/lib/python/cc/lmh/test_lookup1.py | |
| 1139 52369734 | |
| 1140 1076046 130318 | |
| 1141 | |
| 1142 real 1m52.668s | |
| 1143 user 1m40.751s | |
| 1144 sys 0m9.610s | |
| 1145 | |
| 1146 Not bad. 1.5 minutes per file, plus 10 x 20 secs or so for the | |
| 1147 unpickles =~ 453 minutes == 8 hours. | |
| 1148 | |
| 1149 Try pre-filter with the Bloom filter. | |
|
61
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1150 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1151 Rebuild with byte values: |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1152 >>> from pybloomfilter import BloomFilter |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1153 >>> g=BloomFilter(523863383,0.05,'/tmp/hst/uris_20.bloom') |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1154 >>> def bff(f,fn): |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1155 ... with open(fn,'rb') as uf: |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1156 ... for l in uf: |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1157 ... f.add(l.split(b'\t')[2]) |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1158 ... |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1159 >>> timeit.timeit("bff(g,'/work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv')",number=1,globals=globals()) |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1160 80.17189159989357 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1161 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1162 >: time ~/lib/python/cc/lmh/test_lookup2.py |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1163 52369734 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1164 1076046 entries, 130318 given lastmod, 78 false positives |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1165 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1166 real 1m49.567s |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1167 user 1m38.818s |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1168 sys 0m8.668s |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1169 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1170 Not worth it :-(. |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1171 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1172 Build the rest: |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1173 >: time ~/lib/python/cc/lmh/test_hash.py -r 1 -p results/CC-MAIN-2019-35/warc_lmhx/ks_10-19.pickle -f /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_10-19.tsv |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1174 52489306 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1175 [54.296645537018776] |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1176 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1177 real 1m32.529s |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1178 user 1m16.956s |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1179 sys 0m14.267s |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1180 >: seq 2 9 | parallel -j 8 'time ~/lib/python/cc/lmh/test_hash.py -r 1 -p results/CC-MAIN-2019-35/warc_lmhx/ks_{}0-{}9.pickle -f /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_{}0-{}9.tsv' |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1181 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
1182 Slightly slower when running in parallel, but done |
|
62
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1183 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1184 Try using cdb file, has to be one seg per file because of 4GB limit of |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1185 cdb C-code |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1186 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1187 >: ~/lib/python/cc/lmh/ks2cdb.py -f <(head -5236974 results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv) -c results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb_in |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1188 5236974 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1189 11.358113696798682 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1190 >: time cdbmake results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb results/CC-MAIN-2019-35/warc_lmhx/ks_0.tmp < results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb_in |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1191 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1192 real 0m4.981s |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1193 user 0m2.865s |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1194 sys 0m1.886s |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1195 >: ls -lh results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb_in |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1196 -rw-r--r-- 1 hst dc007 574M Jan 3 11:34 results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb_in |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1197 >: ls -lh results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1198 -rw-r--r-- 1 hst dc007 642M Jan 3 11:34 results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1199 >: ls -lh results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.{tsv,pickle} |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1200 -rw-r--r-- 1 hst dc007 5.4G Jan 2 13:41 results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.pickle |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1201 -rw-r--r-- 1 hst dc007 8.8G Oct 3 2023 results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1202 >: cdbdump <results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb | head |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1203 +52,10:20190817225913http://85.163.4.0:9999/faces/login.jsp->1566082752 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1204 +58,10:20190817232933https://1.1.1.1/?source=https://www.tuppu.fi->1560198539 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1205 +34,10:20190822010151http://163.30.193.1/->1256804738 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1206 +64,10:20190825182530http://45.70.224.1:20026/index.php?tipo=minhaconta->1565613324 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1207 +32,10:20190818210532http://50.63.77.1/->1366940302 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1208 +41,10:20190825182606http://119.145.170.10:2160/->1497240281 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1209 +43,10:20190825142846http://71.43.189.10/dermorph/->1564555978 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1210 +53,10:20190825143020http://71.43.189.10/dermorph/about.html->1564555936 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1211 +55,10:20190825142857http://71.43.189.10/dermorph/contact.html->1564748606 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1212 +54,10:20190825142842http://71.43.189.10/dermorph/guided.html->1563634912 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1213 sing<4795>: cdbget 20190825142846http://71.43.189.10/dermorph/ <results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1214 1564555978 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1215 >: cdbstats <results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1216 records 5226422 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1217 d0 3920023 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1218 d1 748992 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1219 d2 269375 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1220 d3 124011 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1221 d4 64304 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1222 d5 36623 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1223 d6 21765 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1224 d7 13737 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1225 d8 8712 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1226 d9 5866 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1227 >9 13014 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1228 >: cdbtest <results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1229 found: 5226422 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1230 different record: 0 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1231 bad length: 0 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1232 not found: 0 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1233 untested: 10552 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1234 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1235 All good, _but_ |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1236 >>> r=cdblib.Reader.from_file_path('results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb') |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1237 >>> r.get(b'xyzzy') |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1238 >>> r.get(b'20190825142846http://71.43.189.10/dermorph/') |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1239 b'1564555978' |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1240 >>> from timeit import Timer |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1241 >>> t=Timer("r.get(b'20190825142846http://71.43.189.10/dermorph/')", |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1242 globals=globals()) |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1243 >>> t.timeit(100000) |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1244 0.30162674374878407 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1245 >>> t.timeit(100000) |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1246 0.30807616002857685 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1247 >>> t.timeit(100000) |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1248 0.30170613154768944 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1249 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1250 So, 100 x slower than a dict :-(. Just checking that... |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1251 >>> t = Timer("b'20190825142846http://71.43.189.10/dermorph/' in d",globals={'d':d}) |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1252 >>> r=cdblib.Reader.from_file_path('results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb') |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1253 >>> r.get(b'20190825142846http://71.43.189.10/dermorph/') |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1254 b'1564555978' |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1255 >>> s = Timer("r.get(b'20190825142846http://71.43.189.10/dermorph/')", |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1256 ... globals=globals()) |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1257 >>> (t.repeat(5,100000),s.repeat(5,100000)) |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1258 [0.005662968382239342, 0.005780909210443497, 0.005478940904140472, 0.005713008344173431, 0.005547545850276947] |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1259 [0.30250774696469307, 0.303345350548625, 0.3002819549292326, 0.30161340720951557, 0.30262864381074905]) |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1260 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1261 so, forget cdb |
|
63
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1262 Hmm... |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1263 >: time cdbtest <results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1264 found: 5226422 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1265 different record: 0 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1266 bad length: 0 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1267 not found: 0 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1268 untested: 10552 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1269 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1270 real 0m34.151s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1271 user 0m7.513s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1272 sys 0m5.123s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1273 (/ 34.1 5226422) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1274 6.524540115589595e-06 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1275 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1276 Added a bunch of type hints, tried cython |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1277 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1278 >: cython --embed test_lookup1.py |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1279 ...[warnings] |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1280 >: python3 -m autotyping --aggressive test_lookup1.py |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1281 Calculating full-repo metadata... |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1282 ... |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1283 - 0 warnings were generated. |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1284 >: cc -I /usr/include/python3.11 -L /usr/lib/python3.11/config-3.11-x86_64-linux-gnu/ test_lookup1.c -l python3.11 -o test_lookup1 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1285 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1286 >: time ~/lib/python/cc/lmh/test_lookup1.py |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1287 52369734 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1288 1076046 entries, 130318 given lastmod |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1289 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1290 real 1m45.305s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1291 user 1m34.265s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1292 sys 0m8.988s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1293 sing<4969>: time ~/lib/python/cc/lmh/test_lookup1 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1294 52369734 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1295 1076046 entries, 130318 given lastmod |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1296 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1297 real 2m5.626s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1298 user 1m54.493s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1299 sys 0m9.169s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1300 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1301 Compiled in profiling |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1302 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1303 >: python3 /work/dc007/dc007/hst/lib/python/cc/lmh/test_pl.py |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1304 /work/dc007/dc007/hst/lib/python/cc/lmh/test_lookup1.py |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1305 52369734 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1306 1076046 entries, 130318 given lastmod |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1307 Mon Jan 6 13:56:04 2025 Profile.prof |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1308 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1309 152694273 function calls (152694246 primitive calls) in 169.605 seconds |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1310 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1311 Ordered by: internal time |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1312 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1313 ncalls tottime percall cumtime percall filename:lineno(function) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1314 1 30.001 30.001 30.001 30.001 {built-in method _pickle.load} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1315 11195226 29.825 0.000 68.952 0.000 igzip.py:196(write) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1316 11195226 24.403 0.000 24.403 0.000 {method 'compress' of 'isal_zlib.Compress' objects} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1317 1 23.919 23.919 167.278 167.278 test_lookup1.py:9(mainp) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1318 10543636 14.072 0.000 14.072 0.000 {method 'search' of 're.Pattern' objects} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1319 10543637 11.911 0.000 11.911 0.000 {method 'readline' of '_io.BufferedReader' objects} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1320 21738863 8.646 0.000 12.583 0.000 _compression.py:12(_check_not_closed) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1321 10543637 8.431 0.000 26.779 0.000 gzip.py:397(readline) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1322 21738868 3.937 0.000 3.937 0.000 gzip.py:323(closed) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1323 10543636 3.524 0.000 3.524 0.000 {method 'split' of 'bytes' objects} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1324 11195226 2.951 0.000 2.951 0.000 {built-in method isal.isal_zlib.crc32} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1325 11195236 2.814 0.000 2.814 0.000 {method 'write' of '_io.BufferedWriter' objects} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1326 1 2.327 2.327 169.605 169.605 <string>:1(<module>) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1327 11195324 1.414 0.000 1.414 0.000 {built-in method builtins.isinstance} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1328 11064997/11064989 1.399 0.000 1.399 0.000 {built-in method builtins.len} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1329 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1330 >: python3 /work/dc007/dc007/hst/lib/python/cc/lmh/test_pl.py |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1331 /work/dc007/dc007/hst/lib/python/cc/lmh/test_lookup1.cpython-311-x86_64-linux-gnu.so |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1332 52369734 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1333 1076046 entries, 130318 given lastmod |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1334 Mon Jan 6 14:50:59 2025 Profile.prof |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1335 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1336 131606993 function calls (131606966 primitive calls) in 164.213 seconds |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1337 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1338 Ordered by: internal time |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1339 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1340 ncalls tottime percall cumtime percall filename:lineno(function) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1341 1 67.658 67.658 164.213 164.213 test_lookup1.py:9(mainp) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1342 11195226 29.944 0.000 69.788 0.000 igzip.py:196(write) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1343 11195226 25.274 0.000 25.274 0.000 {method 'compress' of 'isal_zlib.Compress' objects} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1344 10543637 12.200 0.000 12.200 0.000 {method 'readline' of '_io.BufferedReader' objects} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1345 21738863 8.530 0.000 12.394 0.000 _compression.py:12(_check_not_closed) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1346 10543637 8.205 0.000 26.723 0.000 gzip.py:397(readline) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1347 21738868 3.863 0.000 3.863 0.000 gzip.py:323(closed) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1348 11195226 3.058 0.000 3.058 0.000 {built-in method isal.isal_zlib.crc32} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1349 11195236 2.784 0.000 2.784 0.000 {method 'write' of '_io.BufferedWriter' objects} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1350 11064996/11064988 1.343 0.000 1.343 0.000 {built-in method builtins.len} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1351 11195324 1.309 0.000 1.309 0.000 {built-in method builtins.isinstance} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1352 2 0.040 0.020 0.040 0.020 {built-in method io.open} |
| 64 | 1353 |
| 1354 After a long diversion to get cython access to C extensions working, | |
| 1355 now have a setup that works, in lib/python/cc/lmh: | |
| 1356 | |
| 1357 >: python3 setup.py build_ext -i | |
| 1358 >: python3 -c 'import db' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb xyzzy | |
| 1359 testing... | |
| 1360 0 | |
| 1361 tested | |
| 1362 >: python3 -c 'import db' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ | |
| 1363 testing... | |
| 1364 1 | |
| 1365 tested | |
| 1366 >: python3 -c 'import db' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 | |
| 1367 testing... | |
| 1368 1 | |
| 1369 tested | |
| 1370 1.8913232665508986 | |
| 1371 >: python3 -c 'import db' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb xyzzy 10000000 | |
| 1372 testing... | |
| 1373 0 | |
| 1374 tested | |
| 1375 1.31287781894207 | |
| 1376 sing<3792>: python3 -c 'import db' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb xyzzy 10000000 | |
| 1377 testing... | |
| 1378 0 | |
| 1379 tested | |
| 1380 1.3081446290016174 | |
| 1381 So, maybe that's hopeful | |
| 1382 | |
| 1383 to get the values out: | |
| 1384 https://candide-guevara.github.io/cs_related/2018/02/28/cython-memory-mgt.html | |
| 1385 mmap->memoryview, maybe? | |
| 1386 or | |
| 1387 cdef char[::1] mview = <char[:size:1]>(bp) | |
| 1388 https://stackoverflow.com/questions/50228544/how-do-i-read-a-c-char-array-into-a-python-bytearray-with-cython | |
| 1389 or, if that doesn't work, have to copy: | |
| 1390 https://stackoverflow.com/questions/50228544/how-do-i-read-a-c-char-array-into-a-python-bytearray-with-cython | |
| 1391 | |
| 1392 Not quite there, but, maybe, nearly: | |
| 1393 >: python3 -c 'import db' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 | |
| 1394 testing... | |
| 1395 2488 10 # offset and length? | |
| 1396 tested | |
| 1397 1.9035462848842144 | |
| 1398 >: dd ibs=1 skip=2488 count=10 if=~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb of=/dev/stdout | |
| 1399 1564555978 | |
| 1400 >: cdbget 20190825142846http://71.43.189.10/dermorph/ <~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb | |
| 1401 1564555978 | |
| 65 | 1402 |
| 1403 OK, definitely worth trying: | |
| 1404 >: python3 -c 'import db' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 | |
| 1405 testing... | |
| 1406 2488 10 | |
| 1407 1564555978 | |
| 1408 tested | |
| 1409 1.800725992769003 | |
| 1410 A long struggle to get the module structure better, finally sorted | |
| 1411 cdb.pxd exposes cdb.h from cdb-0.75 (slightly updated) | |
| 1412 db.{pyx, pxd} define an interface cython class to hold a cdb.Cdb instance | |
| 1413 nndb.pyx does a stress test of lookup in a cdb file from 1% of the | |
| 1414 timestamps | |
| 1415 >: python3 -c 'import ndb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 | |
| 1416 testing... /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb b'20190825142846http://71.43.189.10/dermorph/' x 10000000 | |
| 1417 mv 672917132 cdb 140602889433984 | |
| 1418 1 2488 10 | |
| 1419 1564555978 | |
| 1420 tested | |
| 1421 2.055266048759222 | |
|
66
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1422 Oops, that was ndb, and nndb doesn't work! |
| 65 | 1423 |
| 1424 Things to try next: | |
| 67 | 1425 1) Build a bigger .cdb w. as close to 4GB as possible Done |
| 1426 2) Shift to a shared library for cdb-0.75 Done | |
| 65 | 1427 3) Get rid of the single fixed Cdb struct instance and malloc it as |
| 67 | 1428 required Done |
| 1429 3a) Remove debugging output and recompile everything Done | |
| 65 | 1430 4) Build and test the real harness to process .cdx files using .cdb |
|
66
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1431 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1432 Try 50% more, e.g. approx. 1.5 segments |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1433 >: python3 -c "print(1.5 * 5236974)" |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1434 7855461.0 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1435 >: time ~/lib/python/cc/lmh/ks2cdb.py -f <(head -7855461 results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv) -c results/CC-MAIN-2019-35/warc_lmhx/ks_0.5.cdb_in7855461 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1436 real 0m14.585s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1437 user 0m13.909s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1438 sys 0m3.308s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1439 >: time cdbmake results/CC-MAIN-2019-35/warc_lmhx/ks_0.5.cdb results/CC-MAIN-2019-35/warc_lmhx/ks_0.5.tmp < results/CC-MAIN-2019-35/warc_lmhx/ks_0.5.cdb_in |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1440 real 0m6.075s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1441 user 0m3.682s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1442 sys 0m2.337s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1443 >: ls -lh results/CC-MAIN-2019-35/warc_lmhx/ks_0.5.cdb |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1444 -rw-r--r-- 1 hst dc007 991M Jan 27 15:00 results/CC-MAIN-2019-35/warc_lmhx/ks_0.5.cdb |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1445 >: python3 -c 'import ndb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1446 testing... /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb b'20190825142846http://71.43.189.10/dermorph/' x 10000000 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1447 mv 672917132 cdb 140324514468736 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1448 1 2488 10 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1449 1564555978 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1450 tested |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1451 2.016317328438163 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1452 >: python3 -c 'import ndb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.5.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1453 testing... /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0.5.cdb b'20190825142846http://71.43.189.10/dermorph/' x 10000000 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1454 mv 1039033960 cdb 140207982297984 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1455 1 2488 10 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1456 1564555978 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1457 tested |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1458 2.0484518501907587 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1459 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1460 So, that's OK, but still, would need 67 hash files == 265GB |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1461 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1462 Wait, isn't it 4GB max??? |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1463 Build ks_0-9.60.cdb, (as in 60% of 0-9, 31M lines ): |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1464 >: time ~/lib/python/cc/lmh/ks2cdb.py -f <(head -$((4 * 7855461)) ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv) -c ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb_in |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1465 31421844 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1466 real 0m58.676s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1467 user 0m56.312s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1468 sys 0m12.306s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1469 >: cdbmake ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.tmp < ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb_in |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1470 >: ls -lh ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1471 -rw-r--r-- 1 hst dc007 3.8G Jan 27 15:26 /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1472 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1473 cdbget works: |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1474 >: cdb-0.75/cdbget 20190818122159https://m.europapress.es/navarra/noticia-gobierno-navarra-gamesa-acuerdan-necesidad-reindustrializar-planta-alsasua-20100525100333.html <~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1475 1566130319 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1476 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1477 But cdbtest and cdbstats die with that as input. Looks like it's when int |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1478 overflows to -2MB |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1479 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1480 Careful review of cdb.c/h to change int to uint32 whenever an offset |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1481 in the mmap is stored finally got odb working, in due course should |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1482 try to fix and fork all of cdb |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1483 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1484 Still crashing with 0-9.60 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1485 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1486 Ah, a further int pblm -- editted seek-pos to use uint32, now |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1487 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1488 >: cdb-0.75/cdbtest <~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1489 found: 31403644 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1490 different record: 0 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1491 bad length: 0 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1492 not found: 0 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1493 untested: 18200 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1494 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1495 Finally got a version of nndb (which cimports db as well as cdb) to |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1496 work by moving _all_ uses of _c_cdb into db. Still comparable in |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1497 speed to the mixed version in e.g. odb: |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1498 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1499 >: python3 -c 'import odb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1500 testing... |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1501 mv 672917132 cdb 140317635072864 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1502 2488 10 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1503 1564555978 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1504 tested |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1505 2.1426388323307037 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1506 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1507 >: python3 -c 'import nndb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1508 testing... /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb b'20190825142846http://71.43.189.10/dermorph/' x 10000000 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1509 1 2488 10 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1510 1564555978 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1511 tested |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1512 2.0193762965500355 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1513 2.702429808676243 10000000 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1514 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1515 Interesting that just adding a counter to the test loop slows it down |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1516 so much: |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1517 'cfind(probe)', |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1518 vs |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1519 '(X:=X+1) if cfind(probe)==1 else None', |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1520 setup = 'global X' |
| 67 | 1521 |
| 1522 For this test, size of db doesn't matter: | |
| 1523 >: python3 -c 'import nndb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 | |
| 1524 testing... /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb b'20190825142846http://71.43.189.10/dermorph/' x 10000000 | |
| 1525 1 2488 10 | |
| 1526 1564555978 | |
| 1527 tested | |
| 1528 1.9914793614298105 | |
| 1529 2.711402891203761 10000000 | |
| 1530 | |
| 1531 >: python3 -c 'import nndb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 | |
| 1532 testing... /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb b'20190825142846http://71.43.189.10/dermorph/' x 10000000 | |
| 1533 1 2488 10 | |
| 1534 1564555978 | |
| 1535 tested | |
| 1536 2.015953341498971 | |
| 1537 2.798953704535961 10000000 | |
| 1538 | |
| 1539 Now using non-static Cdb, it's slower??? Cirrus load is low :-(. | |
| 1540 | |
| 1541 >: python3 -c 'import nndb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 | |
| 1542 testing... /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb b'20190825142846http://71.43.189.10/dermorph/' x 10000000 | |
| 1543 1 2488 10 | |
| 1544 1564555978 | |
| 1545 tested | |
| 1546 2.474294847997953 | |
| 1547 3.2440936170023633 10000000 | |
| 1548 | |
| 68 | 1549 [Forget that, it's back to more or less the same timing as before, |
| 1550 mystery | |
| 1551 >: python3 -c 'import nndb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 | |
| 1552 testing... /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb b'20190825142846http://71.43.189.10/dermorph/' x 10000000 | |
| 1553 1 2488 10 | |
| 1554 1564555978 | |
| 1555 tested | |
| 1556 2.0566618730081245 | |
| 1557 2.6779559339920525 10000000 | |
| 1558 ] | |
| 1559 | |
| 1560 And it works: | |
| 67 | 1561 |
| 1562 >: python3 -c 'import get2' cdb/rts-tmp/sv.cdb cdb/rts-tmp/12.cdb two discard/tcp | |
| 1563 two cdb/rts-tmp/sv.cdb missing | |
| 1564 discard/tcp cdb/rts-tmp/sv.cdb 9 | |
| 1565 two cdb/rts-tmp/12.cdb Goodbye | |
| 1566 discard/tcp cdb/rts-tmp/12.cdb missing | |
| 1567 | |
| 68 | 1568 Spent a morning getting profiling working, but only from/for apps |
| 1569 (cdbget, cdbtest) within the CDB distro. Had to change _exit(0) to | |
| 1570 exit(0) to get the profile data (mcount) actually written out. | |
| 1571 | |
| 1572 And guess what, by far the most time is spent in byte_copy. Noah is | |
| 1573 still correctly predicting things! | |
| 1574 | |
| 1575 Each sample counts as 0.01 seconds. | |
| 1576 % cumulative self self total | |
| 1577 time seconds seconds calls ms/call ms/call name | |
| 1578 41.10 1.54 1.54 119212725 0.00 0.00 byte_copy | |
| 1579 13.21 2.04 0.50 479500623 0.00 0.00 cdb_hashadd | |
| 1580 8.14 2.34 0.31 83427967 0.00 0.00 get | |
| 1581 7.74 2.63 0.29 5226422 0.00 0.00 cdb_findnext | |
| 1582 6.94 2.89 0.26 5226422 0.00 0.00 cdb_hash | |
| 1583 5.60 3.10 0.21 83489614 0.00 0.00 buffer_get | |
| 1584 4.27 3.26 0.16 17427786 0.00 0.00 byte_diff | |
| 1585 3.20 3.38 0.12 main | |
| 1586 2.13 3.46 0.08 5226422 0.00 0.00 seek_cur | |
| 1587 2.00 3.54 0.08 5226422 0.00 0.00 cdb_find | |
| 1588 1.47 3.59 0.06 35723096 0.00 0.00 cdb_read | |
| 1589 1.33 3.64 0.05 10474460 0.00 0.00 getnum | |
| 1590 | |
| 1591 And | |
| 1592 ---------------------------------------------- | |
| 1593 0.00 0.00 15/119212725 buffer_put [19] | |
| 1594 0.46 0.00 35723096/119212725 cdb_read [7] | |
| 1595 1.08 0.00 83489614/119212725 buffer_get [5] | |
| 1596 [4] 41.1 1.54 0.00 119212725 byte_copy [4] | |
| 1597 | |
| 1598 Not clear why this has slowed down 20--25% since Cirrus reboot... | |
| 1599 Obvs. could try increasing buffer size which might well help... | |
| 1600 | |
| 1601 Well, changing to 64K didn't help. | |
| 1602 it's giving the same results... | |
| 1603 | |
| 1604 Anyway, let's do a real experiment: | |
| 1605 | |
| 1606 | |
| 1607 >: seq 1 9 | parallel -j 9 'echo {} $(wc -l < ks_{}0-{}9.tsv)' |sort -k1n,1 >> ks_lines.tsv | |
| 1608 >: rename -n 's/\.(.*)/-\1.tsv/' ks.?? | |
| 1609 >: wc -l < ks.00 | |
| 1610 32468548 | |
| 1611 >: wc -l < ../ks_0-9.60.cdb_in | |
| 1612 31421845 | |
| 1613 >: cdbmake ks-00.cdb ks-00.tmp < ks-00.cdb_in | |
| 1614 >: ls -lh ks-00.cdb_in | |
| 1615 -rw-r--r-- 1 hst dc007 3.5G Feb 10 14:46 ks-00.cdb_in | |
| 1616 | |
| 1617 Try more? | |
| 1618 | |
| 1619 >: split -d -n l/1/15 /tmp/hst/ks.tsv > ks-00_15.tsv | |
| 1620 >: ~/lib/python/cc/lmh/ks2cdb.py -f ks-00_15.tsv -c ks-00_15.cdb_in | |
| 1621 34382190 | |
| 1622 >: cdbmake ks-00_15.cdb ks-00_15.tmp < ks-00_15.cdb_in | |
| 1623 cdbmake: fatal: unable to create ks-00_15.tmp: out of memory | |
| 1624 | |
| 1625 OK, 16 it is. | |
| 1626 >: seq -f "%02g" 1 15 | parallel -j 8 '~/lib/python/cc/lmh/ks2cdb.py -f ks-{}.tsv -c ks-{}.cdb_in && cdbmake ks-{}.cdb ks-{}.tmp < ks -{}.cdb_in' | |
| 1627 33058467 | |
| 1628 /usr/bin/bash: line 1: ks: No such file or directory [oops] | |
| 1629 32925450 | |
| 1630 32194266 | |
| 1631 32611527 | |
| 1632 33059551 | |
| 1633 32235932 | |
| 1634 32466927 | |
| 1635 33341399 | |
| 1636 >: seq -f "%02g" 9 15 | parallel -j 7 '~/lib/python/cc/lmh/ks2cdb.py -f ks-{}.tsv -c ks-{}.cdb_in' | |
| 1637 32282702 | |
| 1638 32852577 | |
| 1639 33016202 | |
| 1640 32294662 | |
| 1641 32584480 | |
| 1642 33069019 | |
| 1643 33401674 | |
| 1644 >: seq -f "%02g" 1 15 | parallel -j 8 'cdbmake ks-{}.cdb ks-{}.tmp < ks-{}.cdb_in' | |
| 1645 >: ls -lh *.cdb | |
| 1646 -rw-r--r-- 1 hst dc007 3.9G Feb 10 14:47 ks-00.cdb | |
| 1647 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:21 ks-01.cdb | |
| 1648 -rw-r--r-- 1 hst dc007 4.0G Feb 10 15:21 ks-02.cdb | |
| 1649 -rw-r--r-- 1 hst dc007 4.0G Feb 10 15:21 ks-03.cdb | |
| 1650 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:21 ks-04.cdb | |
| 1651 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:21 ks-05.cdb | |
| 1652 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:21 ks-06.cdb | |
| 1653 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:21 ks-07.cdb | |
| 1654 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:21 ks-08.cdb | |
| 1655 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:21 ks-09.cdb | |
| 1656 -rw-r--r-- 1 hst dc007 4.0G Feb 10 15:22 ks-10.cdb | |
| 1657 -rw-r--r-- 1 hst dc007 4.0G Feb 10 15:21 ks-11.cdb | |
| 1658 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:22 ks-12.cdb | |
| 1659 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:22 ks-13.cdb | |
| 1660 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:22 ks-14.cdb | |
| 1661 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:22 ks-15.cdb | |
| 69 | 1662 >: time cdbtest < ks-00.cdb |
| 1663 found: 32449485 | |
| 1664 different record: 0 | |
| 1665 bad length: 0 | |
| 1666 not found: 0 | |
| 1667 untested: 19063 | |
| 68 | 1668 |
| 69 | 1669 real 2m42.674s |
| 1670 user 0m46.146s | |
| 1671 sys 0m29.479s | |
| 1672 >: ls *.cdb | fgrep -v 00 | parallel -j 8 'cdbtest < {} | egrep -v " 0$"' |& tee /tmp/hst/cdbtest.out & | |
| 1673 found: 32588023 | |
| 1674 untested: 23504 | |
| 1675 found: 32180038 | |
| 1676 untested: 14228 | |
| 1677 found: 33045196 | |
| 1678 untested: 13271 | |
| 1679 found: 32216571 | |
| 1680 untested: 19361 | |
| 1681 found: 33329508 | |
| 1682 untested: 11891 | |
| 1683 found: 32449830 | |
| 1684 untested: 17097 | |
| 1685 found: 32913005 | |
| 1686 untested: 12445 | |
| 1687 found: 33046940 | |
| 1688 untested: 12611 | |
| 1689 found: 33056259 | |
| 1690 untested: 12760 | |
| 1691 found: 32277255 | |
| 1692 untested: 17407 | |
| 1693 found: 32270904 | |
| 1694 untested: 11798 | |
| 1695 found: 32838272 | |
| 1696 untested: 14305 | |
| 1697 found: 32560341 | |
| 1698 untested: 24139 | |
| 1699 found: 33389520 | |
| 1700 untested: 12154 | |
| 1701 found: 33002021 | |
| 1702 untested: 14181 | |
| 1703 | |
| 1704 All good. | |
| 1705 Loaded all 16 in to a python image w/o difficulty | |
| 1706 Converted CCdb shim to use cpdef for most methods w/o apparent loss of | |
| 1707 speed. | |
| 1708 | |
| 1709 But, sigh, need to be on segment boundaries so I can tell by the | |
| 1710 filename in an original index entry which cdb to look in. | |
| 1711 >: echo ?/ks_lines.txt ??/ks_lines.txt | xargs cat | tr ' ' '\n' |python3 -c 'M = 33000000 | |
| 1712 import sys | |
| 1713 n = 0 | |
| 1714 for i in range(100): | |
| 1715 j = int(sys.stdin.readline()) | |
| 1716 if n+j > M: | |
| 1717 print(i-1, n, sep = "\t") | |
| 1718 n = j | |
| 1719 else: | |
| 1720 n += j | |
| 1721 ' > ks_divs.tsv | |
| 1722 >: cat ks_divs.tsv | |
| 1723 5 31296954 | |
| 1724 11 31587394 | |
| 1725 17 31428928 | |
| 1726 23 31622785 | |
| 1727 29 31304094 | |
| 1728 35 31467923 | |
| 1729 41 31502409 | |
| 1730 47 31456463 | |
| 1731 53 31462669 | |
| 1732 59 31538406 | |
| 1733 65 31355147 | |
| 1734 71 31281451 | |
| 1735 77 31429609 | |
| 1736 83 31389542 | |
| 1737 89 31444473 | |
| 1738 95 31385531 | |
| 1739 >: wc -l ks_divs.tsv | |
| 1740 16 ks_divs.tsv | |
| 1741 | |
| 1742 So, has to be 17 :-( | |
| 70 | 1743 |
| 1744 >: cut -f 1 ../ks_divs.tsv | { i=0; while read j; do printf "%s\t%s\n" $i $j; i=$((j+1)); done ; printf "%s\t%s\n" $i 99 ; } | parallel --colsep "\t" 'echo cat ../\{{1}..{2}\}/ks.tsv \> ks_{1}-{2}.tsv \&' | |
| 1745 [couldn't make this work as written, hence the echo, followed by copy-paste] | |
| 1746 >: cut -f 1 ../ks_divs.tsv | { i=0; while read j; do printf "ks_%s-%s\n" $i $j; i=$((j+1)); done ; printf "ks_%s-%s\n" $i 99; } | parallel -j 9 '~/lib/python/cc/lmh/ks2cdb.py -f {}.tsv -c {}.cdb_in' & | |
| 1747 >: cut -f 1 ../ks_divs.tsv | { i=0; while read j; do printf "ks_%s-%s\n" $i $j; i=$((j+1)); done ; printf "ks_%s-%s\n" $i 99; } | parallel -j 9 'cdbmake {}.cdb {}.tmp <{}.cdb_in' & | |
| 1748 >: cut -f 1 ../ks_divs.tsv | { i=0; while read j; do printf "ks_%s-%s\n" $i $j; i=$((j+1)); done ; printf "ks_%s-%s\n" $i 99; } | parallel -j 9 'cdb < {}.cdb' > cdbtest.out & | |
|
71
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1749 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1750 Finished deveopment of test_lookup3, now renamed as test_cdb.py. |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1751 Runs, but _very_ slowly, only 117,666 lines output in 30 minutes. |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1752 top shows it's in state I for Idle: |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1753 1238723 hst 20 0 62.3g 89664 43840 I 2.0 0.0 0:01.81 test_cdb |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1754 No better on a compute node. |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1755 Maybe it's thrashing? |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1756 Tried |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1757 >: cythonize -i test_cdb.py |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1758 >: PYTHONPATH=~/lib/python/cc/lmh python3 -c 'import test_cdb |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1759 test_cdb.mainp() |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1760 ' ks_%d-%d.cdb |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1761 No better, AFAICS |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1762 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1763 Try not using python isal/g(un)zipping? That didn't help either (see |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1764 test_cdbp.py, even a version which only tests entries from segment 0) |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1765 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1766 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1767 1546945 hst 20 0 62.3g 50180 41104 I 1.0 0.0 0:00.98 python3 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1768 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1769 But the suppied test code, which processes 31 million keys, is |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1770 averaging |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1771 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1772 Whereas |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1773 >: time cdbtest < ks_0-5.cdb |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1774 found: 31,281,173 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1775 untested: 15781 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1776 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1777 real 2m45.149s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1778 user 0m48.747s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1779 sys 0m31.113s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1780 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1781 31M in 165 seconds (2.75 minutes) == 5.27e-06 (5 microsec???) per key |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1782 compared to nndb result of 2.67 _seconds_ for 10,000,000 (identical) probes |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1783 i.e. 2.67e-7 per probe. |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1784 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1785 Something weird just happened: |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1786 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1787 1777850 hst 20 0 11.2g 334892 325768 R 91.8 0.1 0:43.41 python3 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1788 1777851 hst 20 0 5192 2900 608 S 23.9 0.0 0:10.28 igzip |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1789 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1790 test_cdbp.mainp() |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1791 ' ks_%d-%d.cdb 3 0 > ../cdx-00101 2>/tmp/hst/three |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1792 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1793 real 0m54.498s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1794 user 0m44.699s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1795 sys 0m17.472s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1796 sing<4015>: fgrep -c lastmod ../cdx-00101 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1797 20203 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1798 sing<4016>: date |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1799 Wed Feb 26 05:59:06 PM GMT 2025 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1800 sing<4017>: ls -l date |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1801 ls: cannot access 'date': No such file or directory |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1802 sing<4018>: ls -l ../cdx-00101 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1803 -rw-r--r-- 1 hst dc007 7044044593 Feb 26 17:58 ../cdx-00101 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1804 sing<4019>: ls -l /tmp/hst/three |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1805 -rw-r--r-- 1 hst dc007 0 Feb 26 17:57 /tmp/hst/three |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1806 _Not_ because of adding more: |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1807 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1808 test_cdbp.mainp() |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1809 ' ks_%d-%d.cdb 1 0 > ../cdx-00101 2>/tmp/hst/one |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1810 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1811 real 0m52.195s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1812 user 0m42.983s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1813 sys 0m17.266s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1814 sing<4021>: fgrep -c lastmod ../cdx-00101 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1815 20203 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1816 sing<4022>: ls -l ../cdx-00101 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1817 -rw-r--r-- 1 hst dc007 7044044593 Feb 26 18:04 ../cdx-00101 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1818 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1819 Sometimes fast, sometimes not? |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1820 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1821 test_cdbp.mainp() |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1822 ' ks_%d-%d.cdb 0 1 0 1 > ../cdx-00101 2>/tmp/hst/one |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1823 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1824 real 2m50.546s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1825 user 0m45.288s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1826 sys 0m20.332s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1827 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz | cat|python3 -c 'import test_cdbp |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1828 test_cdbp.mainp() |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1829 ' ks_%d-%d.cdb 0 1 0 1 > ../cdx-00101x 2>/tmp/hst/onex |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1830 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1831 real 0m49.305s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1832 user 0m41.800s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1833 sys 0m22.880s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1834 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1835 I thought having the 'cat' in the pipeline was making the difference, |
| 75 | 1836 but no, just as fast w/o. Something very odd. |
| 1837 Next day, losing again: | |
| 1838 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp | |
| 1839 test_cdbp.mainp() | |
| 1840 ' ks_%d-%d.cdb 0 1 0 1 > ../cdx-00101x 2>/tmp/hst/onex | |
| 1841 | |
| 1842 real 2m37.951s | |
| 1843 user 0m45.490s | |
| 1844 sys 0m20.874s | |
| 1845 Try a full 17th anyway: | |
| 1846 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp | |
| 1847 test_cdbp.mainp() | |
| 1848 ' ks_%d-%d.cdb 0 1 0 6 > ../cdx-00101 2>/tmp/hst/six | |
| 1849 | |
| 1850 real 2m56.073s | |
| 1851 user 0m45.699s | |
| 1852 sys 0m19.964s | |
| 1853 Not significantly slower... | |
| 1854 >: fgrep -c lastmod ../cdx-00101 | |
| 1855 111967 | |
| 1856 >: wc -l ../cdx-00101 | |
| 1857 14681149 | |
| 1858 Try 2/17ths | |
| 1859 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp | |
| 1860 test_cdbp.mainp() | |
| 1861 ' ks_%d-%d.cdb 0 2 0 12 > ../cdx-00101 2>/tmp/hst/twelve | |
| 1862 real 4m19.232s | |
| 1863 user 0m50.538s | |
| 1864 sys 0m27.702s | |
| 1865 >: fgrep -c lastmod ../cdx-00101 | |
| 1866 221468 | |
| 1867 At least approx. linear... | |
| 1868 Try a compute node... | |
| 1869 No better | |
| 1870 Try chaining | |
| 1871 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00102.gz | python3 -c 'import test_cdbp | |
| 1872 test_cdbp.mainp() | |
| 1873 ' ks_%d-%d.cdb 0 2 0 12 | python3 -c 'import test_cdbp | |
| 1874 test_cdbp.mainp() | |
| 1875 ' ks_%d-%d.cdb 2 4 12 24 > ../cdx-00102 2> /tmp/hst/twentyfour | |
| 1876 [11 minutes] | |
| 1877 Sigh | |
| 1878 Compute node: | |
| 1879 >: time { uz /beegfs/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00103.gz|python3 -c 'import test_cdbp | |
| 1880 test_cdbp.mainp() | |
| 1881 ' ks_%d-%d.cdb 0 2 0 12 |python3 -c 'import test_cdbp | |
| 1882 test_cdbp.mainp() | |
| 1883 ' ks_%d-%d.cdb 2 4 12 24 |python3 -c 'import test_cdbp | |
| 1884 test_cdbp.mainp() | |
| 1885 ' ks_%d-%d.cdb 4 6 24 36 ; } > ../cdx-00103 2> /tmp/hst/36 & | |
| 1886 Ah, maybe I just hadn't cythonised after last edit... | |
| 1887 OK, yes, working! | |
| 1888 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND | |
| 1889 3675590 hst 20 0 3965368 519288 506468 R 86.9 0.2 0:28.18 python3 | |
| 1890 3675589 hst 20 0 3922944 507340 502332 S 84.3 0.2 0:25.53 python3 | |
| 1891 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp | |
| 1892 test_cdbp.mainp() | |
| 1893 ' ks_%d-%d.cdb 0 1 0 6 | python3 -c 'import test_cdbp | |
| 1894 test_cdbp.mainp() | |
| 1895 ' ks_%d-%d.cdb 1 2 6 12 > ../cdx-00101 | |
| 1896 14681147 entries, 111963 given lastmod | |
| 1897 14681147 entries, 109503 given lastmod | |
| 1898 | |
| 1899 real 0m57.837s | |
| 1900 user 1m23.062s | |
| 1901 sys 0m26.550s | |
| 1902 But with three chained it slowed a bit, top never showed more than 40% | |
| 1903 for the first in the chain: | |
| 1904 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp | |
| 1905 test_cdbp.mainp() | |
| 1906 ' ks_%d-%d.cdb 0 1 0 6 | python3 -c 'import test_cdbp | |
| 1907 test_cdbp.mainp() | |
| 1908 ' ks_%d-%d.cdb 1 2 6 12 | python3 -c 'import test_cdbp;test_cdbp.mainp() | |
| 1909 ' ks_%d-%d.cdb 2 3 12 18 > ../cdx-00101 > ../cdx-00101 | |
| 1910 14681147 entries, 111963 given lastmod | |
| 1911 14681147 entries, 109503 given lastmod | |
| 1912 14681147 entries, 111299 given lastmod | |
| 1913 | |
| 1914 real 2m39.481s | |
| 1915 user 2m8.402s | |
| 1916 sys 0m33.503s | |
| 1917 Try compute node again... | |
| 1918 Same behaviour -- 3 is bad, 2 is OK | |
| 1919 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND | |
| 1920 2163622 hst 20 0 3961280 291836 281560 S 10.9 0.1 0:15.66 python3 | |
| 1921 2163621 hst 20 0 3922948 289100 278796 I 10.6 0.1 0:15.38 python3 | |
| 1922 2163623 hst 20 0 3945880 306352 289356 S 10.3 0.1 0:15.50 python3 | |
|
71
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1923 |
|
72
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1924 OK, after many false starts, it's fairly simple: if a cdb file is |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1925 _not_ in the local cache for whereever /work is coming from, mmap |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1926 only accesses bits in 4MB or 8BM or something else relatively small. |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1927 But wc -l ...cdb a) runs in a few seconds and b) gets the whole thing |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1928 in the cache. So, always do that first, and a full 17-step pipeline |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1929 runs nearly as fast as a single step: |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1930 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1931 >: time uz /beegfs/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz | ~/bin/cdb_chain.sh 17 >/tmp/hst/cdb-00101 2>/tmp/hst/ch.out & |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1932 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1933 real 1m5.802s |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1934 user 1m27.873s |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1935 sys 0m34.349s |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1936 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1937 And it matches the earlier version: |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1938 >: fgrep -c lastmod /tmp/hst/cdb-00101 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1939 1864371 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1940 >: ls -l /tmp/hst/cdb-00101 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1941 -rw-r--r-- 1 hst dc007 7090034893 Mar 6 01:24 /tmp/hst/cdb-00101 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1942 >: uz idx/cdx-00101.gz | fgrep -c lastmod |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1943 1864371 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1944 >: uz idx/cdx-00101.gz | wc |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1945 14681147 314748509 7090034893 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1946 >: uz idx/cdx-00101.gz | cmp - /tmp/hst/cdb-00101 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1947 >: echo $? |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1948 0 |
|
73
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1949 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1950 OK, now to fill up 2023-40 so we can do a real trial. |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1951 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1952 Cythonised lmh.py and warc.py, still running about 14 seconds per file |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1953 Don't actually need these yet, as 2023-40 has a full set of lmh |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1954 outputs, but not many ks.tsv: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1955 >: ls */ks.tsv |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1956 0/ks.tsv 15/ks.tsv 4/ks.tsv 68/ks.tsv |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1957 12/ks.tsv 36/ks.tsv 56/ks.tsv best_two_by_nl1/ks.tsv |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1958 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1959 Note, that to populate cdbs, we don't need most of the logic in |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1960 sort_date.py, because all the sorting and merging isn't needed! |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1961 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1962 Hmm. In fact, just go straight to cdb_in.txt |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1963 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1964 >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 15 0 /tmp/hst 2>/tmp/hst/15/errs |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1965 /tmp/hst/15/lmh.cdb_in |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1966 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1967 real 223m28.517s |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1968 user 168m34.751s |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1969 sys 12m6.020s |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1970 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1971 This compares to 273 minutes for 806 files in the cleanup of segment |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1972 95, see end of old_notes. Best full segment was 96, i.e. 279 for |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1973 900. And those figures are for lmh.py only |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1974 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1975 Are results the same? |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1976 Old: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1977 >: wc -l *.tsv |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1978 8247 ks_errs.tsv |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1979 5324616 ks.tsv |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1980 New: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1981 >: wc -l * |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1982 9147 errs |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1983 5324617 lmh.cdb_in |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1984 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1985 I _think_ that's good -- there's one line per input file added to the |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1986 error output now, and one blank line at the end of the cdb_in file. |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1987 Old: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1988 >: cut -f 2- ks_errs.tsv |sus |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1989 5420 cannot unpack non-iterable NoneType object |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1990 2670 list index out of range |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1991 113 year 641471 is out of range |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1992 20 year 641474 is out of range |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1993 9 hour must be in 0..23 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1994 9 year 642435 is out of range |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1995 3 year 642437 is out of range |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1996 1 year 4262575 is out of range |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1997 1 year 641769 is out of range |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1998 1 year 642434 is out of range |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1999 New: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2000 >: cut -f 2- <(fgrep -v beegfs /tmp/hst/15/errs) | cut -f 1-5 -d ' ' |sus |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2001 8090 Invalid date value or format |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2002 113 year 641471 is out of |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2003 20 year 641474 is out of |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2004 9 hour must be in 0..23 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2005 9 year 642435 is out of |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2006 3 year 642437 is out of |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2007 1 year 4262575 is out of |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2008 1 year 641769 is out of |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2009 1 year 642434 is out of |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2010 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2011 But, failed to fix the case where the URI in the original WARC is not |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2012 %-encoded: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2013 >: diff -bw <(cut -f2- ks.tsv| sed 's/ //;s/\.0$//'|sort -k1,1) <(sed 's/^\+[0-9,]*://;s/->\([0-9]*\)$/ \1/' /tmp/hst/15/lmh.cdb_in_nopercent | sort -k1,1)|head |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2014 4360c4361 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2015 < 20230922034234https://www.hansaton.com/zh-hk/\u6D88\u8CBB\u8005.html 1695354153 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2016 --- |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2017 > 20230922034234https://www.hansaton.com/zh-hk/消費者.html 1695354153 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2018 6574c6575 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2019 < 20230922034248https://blog.arabtherapy.com/\u0639\u0644\u0627\u062C-\u0627\u0644\u062E\u062C\u0644-\u0627\u0644\u0627\u062C\u062A\u0645\u0627\u0639\u064A/ 1695354168 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2020 --- |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2021 > 20230922034248https://blog.arabtherapy.com/علاج-الخجل-الاجتماعي/ 1695354168 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2022 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2023 Could try fixing that at lookup time |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2024 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2025 Fixed in warc2cdb, seems OK for timing: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2026 /tmp/hst/15/lmh.cdb_in |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2027 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2028 real 224m32.815s |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2029 user 169m25.216s |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2030 sys 12m7.867s |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2031 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2032 Lengths are right, difference with ks looks better, try the real |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2033 thing: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2034 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2035 >: cd /beegfs/common_crawl/CC-MAIN-2023-40/cdx/warc |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2036 >: ls | parallel -j 10 "uz '{}' |cut -f 2,4,18 -d ' '|fgrep .15/warc/CC-MAIN | cut -f 1,2 -d ' ' | sort > '/tmp/hst/15/orig_{#}.cdx'" & |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2037 >: sort -m *.cdx | sponge all.cdx |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2038 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2039 Nope: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2040 >: diff <(sed 's/->[0-9]*$/",/' test.cdb_in) all.cdx |less |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2041 >: fgrep -n dryelf.com test.cdb_in all.cdx |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2042 test.cdb_in:6710:20230922034248 "http://www.dryelf.com/files/DryCookieInstall.exe", |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2043 test.cdb_in:1600193:20230922043436 "http://www.dryelf.com/files/DryLottoInstall.exe", |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2044 all.cdx:12190045:20230922044250 "http://www.dryelf.com/downloads.asp?Sort=0&Cat=3", |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2045 all.cdx:25401353:20230922053834 "http://www.dryelf.com/downloads.asp?Sort=2&Cat=0", |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
2046 |
|
74
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2047 Stupid -- mime-detected sometimes has a space in it... |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2048 |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2049 >: ls *.gz | parallel -j 10 "uz '{}' |fgrep .15/warc/CC-MAIN | cut -f 2,4 -d ' ' | sort > '/tmp/hst/15/orig_{#}.cdx'" |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2050 >: sort -m *.cdx | sponge all.cdx |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2051 >: wc -l all.cdx |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2052 34742533 |
|
78
45b45810aa12
cleaned up/filled in after merge
Henry Thompson <ht@markup.co.uk>
parents:
76
diff
changeset
|
2053 >: sed -E 's/^\+[0-9,]*:(.{14,14})/\1 "/;s/->[0-9]{10,10}$/",/' /tmp/hst/15/lmh.cdb_in | sort -k1,1 > test.cdb_in |
|
45b45810aa12
cleaned up/filled in after merge
Henry Thompson <ht@markup.co.uk>
parents:
76
diff
changeset
|
2054 |
|
74
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2055 Diff looks plausible, and a few known cases are correct now: |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2056 >: fgrep -n '20230922034234 "https://www.hansaton.com/zh-hk/' test.cdb_in |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2057 4361:20230922034234 "https://www.hansaton.com/zh-hk/%E6%B6%88%E8%B2%BB%E8%80%85.html", |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2058 >: fgrep -n '20230922034234 "https://www.hansaton.com/zh-hk/' all.cdx |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2059 27516:20230922034234 "https://www.hansaton.com/zh-hk/%E6%B6%88%E8%B2%BB%E8%80%85.html", |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2060 >: fgrep dryelf.com test.cdb_in all.cdx |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2061 test.cdb_in:20230922034248 "http://www.dryelf.com/files/DryCookieInstall.exe", |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2062 test.cdb_in:20230922043436 "http://www.dryelf.com/files/DryLottoInstall.exe", |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2063 all.cdx:20230922034248 "http://www.dryelf.com/files/DryCookieInstall.exe", |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2064 all.cdx:20230922043436 "http://www.dryelf.com/files/DryLottoInstall.exe", |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2065 all.cdx:20230922044250 "http://www.dryelf.com/downloads.asp?Sort=0&Cat=3", |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
2066 all.cdx:20230922053834 "http://www.dryelf.com/downloads.asp?Sort=2&Cat=0", |
| 76 | 2067 |
|
79
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
2068 OK, rerun in the right place: |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
2069 >: cd ~/results/CC-MAIN-2023-40/warc_lmhx |
|
84
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2070 >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 15 . 2> 15/w2c_errs & |
|
79
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
2071 |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
2072 real 210m54.183s |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
2073 user 152m34.172s |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
2074 sys 12m25.122s |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
2075 |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
2076 Ran a test of 5 jobs on a single compute node, did 78 files in 18 |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
2077 minutes. Try 15, for grins |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
2078 >: export PYTHONPATH=/work/dc007/dc007/hst/lib/python/cc:/work/dc007/dc007/hst/lib/python/cc/lmh |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
2079 >: cd $W/results/CC-MAIN-2023-40/warc_lmhx |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
2080 >: { date ; seq 0 14 | parallel "python3 -c 'import sys,warc2cdb; |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
2081 sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 '{}' . 2> '{}'/w2c_errs" ; date ; } & |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
2082 Thu Mar 20 12:05:50 AM GMT 2025 |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
2083 After an hour, we're only at about 160 per process, and the job only |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
2084 has 3:30 to run, so I'm going to kill 8-14. |
|
80
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2085 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2086 0-7 didn't finish: |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2087 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2088 reran the who |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2089 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2090 >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 0 . 2> 0/w2c_errsle of 0 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2091 ./0/lmh.cdb_in |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2092 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2093 real 224m28.784s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2094 user 169m21.796s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2095 sys 12m25.583s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2096 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2097 Update warc2cdb to accept a rangm patter, use it to try getting the |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2098 rest of segment 1: |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2099 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2100 >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 1 . '{685..899}' 2>> 1/w2c_errs & |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2101 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2102 Oops, complete fail, forgot that the earlier version used glob.glob |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2103 which does _not_ produce a sorted list of files, so need to refetch |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2104 all of 1 as well: |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2105 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2106 >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 1 . 2> 1/w2c_errs |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2107 ./1/lmh.cdb_in |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2108 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2109 real 210m56.444s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2110 user 154m29.608s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2111 sys 11m49.656s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2112 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2113 Here and subsequently, explicity sort the input file list. |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2114 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2115 Compute the missing bits, use to fill the gaps: |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2116 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2117 >: fgrep beegfs 2/w2c_errs |cut -f 8 -d / | tr -d \' | cut -f 5 -d - | cut -f 1 -d .|sort | diff - <(seq -f '%05g' 0 899)|~/lib/python/cc/lmh/hack.py > /tmp/hst/2.pat |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2118 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2119 >: echo python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 2 surplus $(cat /tmp/hst/2.pat) 2> surplus/2/w2c_errs |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2120 time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 2 surplus '{017,024,027,031,{034..035},038,040,043,046,{051..052},059,061,064,084,092,103,105,{108..109},111,114,117,119,{122..123},135,137,139,144,{146..147},150,153,162,186,195,197,206,208,211,{224..225},227,243,251,253,{264..265},269,285,{292..293},296,304,{307..308},{316..317},328,337,341,344,{360..361},373,380,{386..387},392,397,406,{418..419},426,439,445,454,456,459,462,465,467,473,480,{482..483},{486..487},489,{491..492},499,501,506,512,521,523,528,540,{545..546},550,555,562,568,573,{577..578},584,586,588,593,598,{607..608},621,625,{628..629},637,639,{645..647},649,{656..657},659,662,666,669,671,673,{678..679},683,{688..689},{693..694},702,704,706,{709..710},714,730,{740..741},{745..747},{750..751},754,758,763,767,771,{785..787},791,{794..795},800,804,809,{824..825},831,833,{841..842},846,{854..855},862,864,{875..878},880,883,890,{894..896}}' 2> surplus/2/w2c_errs & |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2121 surplus/2/lmh.cdb_in |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2122 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2123 real 47m35.809s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2124 user 35m10.748s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2125 sys 2m54.211s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2126 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2127 >: fgrep -h beegfs {,surplus/}2/w2c_errs | wc -l |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2128 900 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2129 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2130 Checked that the last 8 of the ones that _were_ logged from the |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2131 original aborted run of segment 2 are complete: |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2132 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2133 >: fgrep beegfs 2/w2c_errs |tail -8 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2134 /beegfs/common_crawl/CC-MAIN-2023-40/...-00030.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2135 /beegfs/common_crawl/CC-MAIN-2023-40/...-00756.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2136 /beegfs/common_crawl/CC-MAIN-2023-40/...-00249.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2137 /beegfs/common_crawl/CC-MAIN-2023-40/...-00301.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2138 /beegfs/common_crawl/CC-MAIN-2023-40/...-00158.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2139 /beegfs/common_crawl/CC-MAIN-2023-40/...-00009.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2140 /beegfs/common_crawl/CC-MAIN-2023-40/...-00294.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2141 /beegfs/common_crawl/CC-MAIN-2023-40/...-00576.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2142 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2143 >: python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 2 surplus '{030,756,249,301,158,009,294,576}' 2> surplus/2x/w2c_errs |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2144 >: fgrep -n beegfs 2/w2c_errs |tail -8 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2145 5765:/beegfs/common_crawl/CC-MAIN-2023-40/1695233506658.2/orig/warc/CC-MAIN-20230924155422-20230924185422-00030.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2146 ... |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2147 5821:/beegfs/common_crawl/CC-MAIN-2023-40/1695233506658.2/orig/warc/CC-MAIN-20230924155422-20230924185422-00576.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2148 >: wc -l 2/w2c_errs |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2149 5839 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2150 >: echo $((5839 - 5765)) |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2151 74 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2152 sing<4424>: tail -75 2/w2c_errs|sort |diff -bw - <(sort surplus/2x/w2c_errs) |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2153 [nothing] |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2154 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2155 Something is wrong? |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2156 >: wc -l 0/lmh.cdb_in |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2157 6414381 0/lmh.cdb_in |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2158 >: seq 0 8 | parallel -j 9 'for i in {00..99}; do uz 0/*00{}$i.warc.gz | egrep -c " .* "; done | btot | tee /dev/stderr' | btot |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2159 713980 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2160 714754 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2161 712334 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2162 712640 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2163 713943 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2164 714063 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2165 712428 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2166 712630 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2167 711997 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2168 6418769 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2169 >: echo $((6418769 - 6414381)) |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2170 4388 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2171 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2172 >: seq 0 8 | parallel -j 9 'for i in {00..99}; do uz 15/*00{}$i.warc.gz | egrep -c " .* "; done | btot | tee /dev/stderr' | btot |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2173 591136 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2174 591189 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2175 591105 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2176 593544 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2177 593214 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2178 594861 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2179 592556 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2180 592249 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2181 593009 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2182 5332863 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2183 >: echo $((5332863 - 5324617)) |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2184 8246 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2185 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2186 Ah, maybe OK: |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2187 wc -l < 0/ks_errs.tsv |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2188 4389 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2189 >: cut -f 2 0/ks_errs.tsv | sus |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2190 3188 cannot unpack non-iterable NoneType object |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2191 1193 list index out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2192 1 year 4263994 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2193 1 year 4278897 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2194 1 year 4355304 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2195 1 year 4361928 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2196 1 year 4370444 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2197 1 year 4430056 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2198 1 year 4433368 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2199 1 year 53310 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2200 >: cut -f 2 <(sed 's/value or format .*/value or format/' w2c_errs)|fgrep -v warc.gz| sus |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2201 4381 Invalid date value or format |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2202 1 year 4263994 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2203 1 year 4278897 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2204 1 year 4355304 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2205 1 year 4361928 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2206 1 year 4370444 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2207 1 year 4430056 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2208 1 year 4433368 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2209 1 year 53310 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2210 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2211 And last line of lmh.cdb_in is a blank EOF indicator |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2212 So |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2213 >: echo $((6418769 - 6414380)) |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2214 4389 |
| 81 | 2215 >: fgrep -v warc.gz w2c_errs |wc -l |
| 2216 4389 | |
|
80
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2217 So lengths of lmh.cdb_in match, given that ks.tsv has string dates and |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2218 lmh.cdb_in has dates successfully converted to numeric timestamps, and |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2219 non-conversion counts match. |
| 81 | 2220 |
| 2221 Same for 15... | |
| 2222 >: cut -f 2 ks_errs.tsv | sus | |
| 2223 5420 cannot unpack non-iterable NoneType object | |
| 2224 2670 list index out of range | |
| 2225 113 year 641471 is out of range | |
| 2226 20 year 641474 is out of range | |
| 2227 9 hour must be in 0..23 | |
| 2228 9 year 642435 is out of range | |
| 2229 3 year 642437 is out of range | |
| 2230 1 year 4262575 is out of range | |
| 2231 1 year 641769 is out of range | |
| 2232 1 year 642434 is out of range | |
| 2233 >: cut -f 2 <(sed 's/value or format .*/value or format/' w2c_errs)|fgrep -v warc.gz| sus | |
| 2234 8090 Invalid date value or format | |
| 2235 113 year 641471 is out of range | |
| 2236 20 year 641474 is out of range | |
| 2237 9 hour must be in 0..23 | |
| 2238 9 year 642435 is out of range | |
| 2239 3 year 642437 is out of range | |
| 2240 1 year 4262575 is out of range | |
| 2241 1 year 641769 is out of range | |
| 2242 1 year 642434 is out of range | |
| 2243 >: wc -l ks_errs.tsv | |
| 2244 8247 ks_errs.tsv | |
|
80
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2245 >: fgrep -v warc.gz w2c_errs |wc -l |
| 81 | 2246 8247 |
| 2247 >: wc -l< lmh.cdb_in | |
| 2248 5324617 | |
| 2249 >: seq 0 8 | parallel -j 9 'for i in {00..99}; do uz *00{}$i.warc.gz | fgrep -c " "; done | btot | tee /dev/stderr' | btot | |
| 2250 593009 | |
| 2251 591105 | |
| 2252 593544 | |
| 2253 591136 | |
| 2254 591189 | |
| 2255 594861 | |
| 2256 593214 | |
| 2257 592556 | |
| 2258 592249 | |
| 2259 5332863 | |
| 2260 >: echo $((5332863 - 5324616)) | |
| 2261 8247 | |
| 2262 | |
| 2263 And for 1 | |
| 2264 | |
| 2265 >: seq 0 8 | parallel -j 9 'for i in {00..99}; do uz *00{}$i.warc.gz | fgrep -c " "; done | btot | tee /dev/stderr' | btot | |
| 2266 685262 | |
| 2267 686242 | |
| 2268 684822 | |
| 2269 684268 | |
| 2270 684967 | |
| 2271 684771 | |
| 2272 684520 | |
| 2273 685418 | |
| 2274 686420 | |
| 2275 6166690 | |
| 2276 >: wc -l < lmh.cdb_in | |
| 2277 6160222 | |
| 2278 >: echo $((6166690 - 6160221)) | |
| 2279 6469 | |
| 2280 >: fgrep -v warc.gz w2c_errs |wc -l | |
| 2281 6469 | |
| 2282 | |
| 2283 And for 2? | |
| 2284 | |
| 2285 >: wc -l {,surplus/}2/lmh.cdb_in | |
| 2286 4670389 2/lmh.cdb_in | |
| 2287 52530 surplus/2/lmh.cdb_in | |
| 2288 4722919 total | |
| 2289 sing<4564>: seq 0 8 | parallel -j 9 'for i in {00..99}; do uz 2/*00{}$i.warc.gz | fgrep -c " "; done | btot | tee /dev/stderr' | btot | |
| 2290 659931 | |
| 2291 658839 | |
| 2292 659020 | |
| 2293 660117 | |
| 2294 659896 | |
| 2295 659771 | |
| 2296 659374 | |
| 2297 660426 | |
| 2298 660040 | |
| 2299 5937414 | |
| 2300 | |
| 2301 So surplus/2 is short? | |
| 2302 | |
| 2303 Maybe, but _2_ is short??? | |
| 2304 | |
| 2305 >: fgrep beegfs 2/w2c_errs |cut -f 8 -d / | tr -d \' | cut -f 5 -d - | cut -f 1 -d .|sort | parallel -j 9 'uz 2/*{}.warc.gz | fgrep -c " " | tee /dev/stderr' | btot | |
| 2306 4678728 | |
| 2307 >: echo $((4678728 - 4670388)) | |
| 2308 8340 | |
| 2309 >: fgrep -vh warc.gz 2/w2c_errs |wc -l | |
| 2310 5130 | |
| 2311 | |
| 2312 And so is 4??? | |
| 2313 >: fgrep beegfs 4/w2c_errs |cut -f 8 -d / | tr -d \' | cut -f 5 -d - | cut -f 1 -d .|wc -l | |
| 2314 716 | |
| 2315 >: fgrep beegfs 4/w2c_errs |cut -f 8 -d / | tr -d \' | cut -f 5 -d - | cut -f 1 -d .|sort | parallel -j 9 'uz 4/*{}.warc.gz | fgrep -c " "' | btot | |
| 2316 5003586 | |
| 2317 sing<4577>: fgrep -vh warc.gz 4/w2c_errs |wc -l | |
| 2318 4890 | |
| 2319 sing<4578>: wc -l < 4/lmh.cdb_in | |
| 2320 4975964 | |
| 2321 sing<4579>: echo $((5003586 - 4975963)) | |
| 2322 27623 | |
| 2323 | |
| 2324 Mystery. Just have to redo 2--14. | |
| 2325 Cirrus slurm doesn't allow forcing more than one node, e.g. with | |
| 2326 --nodes=2-7, says you should use --exclusive :-( | |
| 2327 | |
| 2328 So, just force it by hand | |
| 2329 | |
| 2330 >: sbatch --output=slurm_w2c_2-7.out --time=04:00:00 --ntasks=7 -c 3 $HOME/bin/runme.sh $PWD -t 3 -i 'share_by_task.sh -s 2 7 $n $task' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh | |
| 2331 echo $PYTHONPATH | |
| 2332 mkdir -p rerun/$arg | |
| 2333 python3 -c "import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))" 2023-40 $arg ./rerun 2> rerun/$arg/w2c_errs' | |
| 2334 | |
| 2335 >: sbatch --output=slurm_w2c_8-14.out --time=04:00:00 --ntasks=7 -c 3 $HOME/bin/runme.sh $PWD -t 3 -i 'share_by_task.sh -s 8 14 $n $task' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh | |
| 2336 echo $PYTHONPATH | |
| 2337 mkdir -p rerun/$arg | |
| 2338 python3 -c "import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))" 2023-40 $arg ./rerun 2> rerun/$arg/w2c_errs' | |
| 2339 | |
| 2340 >: python3 -c "print((900.0/(626/6))*35)" | |
| 2341 301.91693290734827 | |
| 2342 | |
| 2343 Going too slowly, killed the 2nd one after 39 minutes with less than | |
| 2344 110 files processed by the first in 35 minutes, hope it might now just | |
| 2345 squeak in under 240 minutes ... | |
| 2346 | |
| 2347 >: scancel --signal=HUP 7330184 | |
| 2348 >: python3 -c "print((900.0/(925/6))*45.5)" | |
| 2349 265.62162162162167 | |
| 2350 | |
| 2351 >: python3 -c "print((900.0/(1058/6))*52.25)" | |
| 2352 266.68241965973533 | |
| 2353 | |
| 2354 Nope: | |
| 2355 >: fgrep -ch beegfs rerun/{2..7}/w2c_errs | |
| 2356 680 | |
| 2357 804 | |
| 2358 825 | |
| 2359 816 | |
| 2360 825 | |
| 2361 809 | |
| 2362 >: cd rerun | |
| 2363 sing<4601>: for i in {2..7}; do fgrep beegfs $i/w2c_errs |cut -f 8 -d / | tr -d \' | cut -f 5 -d - | cut -f 1 -d .|sort | diff - <(seq -f '%05g' 0 899)|~/lib/python/cc/lmh/hack.py > $i.pat ; done | |
| 2364 sing<4602>: less 2.pat | |
| 2365 sing<4603>: fgrep -c warc.gz 4/w2c_errs | |
| 2366 825 | |
| 2367 sing<4604>: cat 2.pat | |
| 2368 {{680..899}} | |
| 2369 | |
| 2370 less than 10% speed-up after cancelling the 2nd job: | |
| 2371 >: python3 -c "print((900.0/(4759/6))*240)" | |
| 2372 272.32611893254887 | |
| 2373 | |
| 2374 Rerun 4 separately to have one last go at debugging the problem with | |
| 2375 partial runs: | |
| 2376 | |
| 2377 >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 4 ./surplus 2> surplus/4/w2c_errs & | |
| 2378 ./surplus/4/lmh.cdb_in | |
| 2379 | |
| 2380 real 229m55.344s | |
| 2381 user 164m5.765s | |
| 2382 sys 11m24.657s | |
| 2383 | |
| 2384 Is 4/rerun/w2c_errs the same as surplus/4/w2c_errs through file 00824? | |
| 2385 Yes: | |
| 2386 >: fgrep -c beegfs {rerun,surplus}/4/w2c_errs | |
| 2387 rerun/4/w2c_errs:825 | |
| 2388 surplus/4/w2c_errs:900 | |
| 2389 >: diff -bw <(fgrep beegfs rerun/4/w2c_errs) <(fgrep beegfs surplus/4/w2c_errs)|head -2 | |
| 2390 825a826,900 | |
| 2391 > /beegfs/common_crawl/CC-MAIN-2023-40/1695233511021.4/orig/warc/CC-MAIN-20231002200740-20231002230740-00825.warc.gz | |
| 2392 >: diff -bw <(fgrep beegfs rerun/4/w2c_errs) <(fgrep beegfs surplus/4/w2c_errs)|tail -1 | |
| 2393 > /beegfs/common_crawl/CC-MAIN-2023-40/1695233511021.4/orig/warc/CC-MAIN-20231002200740-20231002230740-00899.warc.gz | |
| 82 | 2394 |
| 2395 Trashed the above rerun by mistake | |
| 2396 | |
| 2397 Successfully ran the balance anyway just to check... | |
| 2398 >: time bash -c "python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 4 rerun '{825..899}' 2> rerun/4/a_w2c_errs | |
| 2399 >: diff -bw <(fgrep beegfs rerun/4/a_w2c_errs) <(fgrep beegfs surplus/4/w2c_errs)|less | |
| 2400 0a1,825 | |
| 2401 > /beegfs/common_crawl/CC-MAIN-2023-40/1695233511021.4/orig/warc/CC-MAIN-20231002200740-20231002230740-00000.warc.gz | |
| 2402 | |
| 2403 Check 4: | |
| 2404 | |
| 2405 >: cut -f 2 4/ks_errs.tsv | sus | |
| 2406 4605 cannot unpack non-iterable NoneType object | |
| 2407 1502 list index out of range | |
| 2408 8 year 539500 is out of range | |
| 2409 7 year 539498 is out of range | |
| 2410 2 year 55722 is out of range | |
| 2411 1 year 53337 is out of range | |
| 2412 1 year 539499 is out of range | |
| 2413 1 year 54618 is out of range | |
| 2414 1 year 54752 is out of range | |
| 2415 >: cut -f 2 <(sed 's/value or format .*/value or format/' surplus/4/w2c_errs)|fgrep -v warc.gz| sus | |
| 2416 6107 Invalid date value or format | |
| 2417 8 year 539500 is out of range | |
| 2418 7 year 539498 is out of range | |
| 2419 2 year 55722 is out of range | |
| 2420 1 Growing buffer 2113089 > 2097152 [! from where???] | |
| 2421 1 year 53337 is out of range | |
| 2422 1 year 539499 is out of range | |
| 2423 1 year 54618 is out of range | |
| 2424 1 year 54752 is out of range | |
| 2425 >: wc -l 4/ks_errs.tsv | |
| 2426 6128 4/ks_errs.tsv | |
| 2427 >: fgrep -v warc.gz surplus/4/w2c_errs |wc -l | |
| 2428 6129 | |
| 2429 That's the 'Growing buffer ???' | |
| 2430 | |
| 2431 >: wc -l surplus/4/lmh.cdb_in | |
| 2432 6281718 surplus/4/lmh.cdb_in | |
| 2433 >: seq 0 8 | parallel -j 9 'for i in {00..99}; do uz 4/*00{}$i.warc.gz | fgrep -c " "; done | btot | tee /dev/stderr' | btot | |
| 2434 697990 | |
| 2435 698141 | |
| 2436 697561 | |
| 2437 699427 | |
| 2438 698286 | |
| 2439 699690 | |
| 2440 699133 | |
| 2441 698055 | |
| 2442 699562 | |
| 2443 6287845 | |
| 2444 >: echo $((6287845 - 6281717)) | |
| 2445 6128 | |
| 2446 OK | |
| 2447 | |
| 2448 Where do we stand? | |
| 2449 >: ff w2c_errs | xargs -n 1 bash -c 'printf "%s\t%s\t%s\n" "$0" $(fgrep -c beegfs/ $0) "$(ls -l $0 | tr -s \ | cut -f 5,6,7,8 -d \ )"' |sort -k2nr,2 2>&1 | |
| 2450 ./0/w2c_errs 900 392410 Mar 24 16:52 | |
| 2451 ./15/w2c_errs 900 609663 Mar 19 22:56 | |
| 2452 ./1/w2c_errs 900 510538 Mar 25 19:02 | |
| 2453 ./surplus/4/w2c_errs 900 504252 Mar 27 02:12 | |
| 2454 ./rerun/4/w2c_errs 825 460572 Mar 26 18:22 | |
| 2455 ./rerun/6/w2c_errs 825 482598 Mar 26 18:22 | |
| 2456 ./rerun/5/w2c_errs 816 487423 Mar 26 18:22 | |
| 2457 ./rerun/7/w2c_errs 809 461695 Mar 26 18:22 | |
| 2458 ./rerun/3/w2c_errs 804 467730 Mar 26 18:22 | |
| 2459 ./rerun/2/w2c_errs 680 409265 Mar 26 18:22 | |
|
83
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2460 ./4/w2c_errs 716 403837 Mar 20 03:33 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2461 ./6/w2c_errs 712 414801 Mar 20 03:33 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2462 ./2/w2c_errs 709 428774 Mar 20 03:33 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2463 ./7/w2c_errs 704 405495 Mar 20 03:33 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2464 ./5/w2c_errs 703 420718 Mar 20 03:33 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2465 ./3/w2c_errs 696 402308 Mar 20 03:33 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2466 ./surplus/2/w2c_errs 191 111807 Mar 24 15:16 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2467 ./8/w2c_errs 173 90504 Mar 20 01:09 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2468 ./11/w2c_errs 170 92570 Mar 20 01:09 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2469 ./9/w2c_errs 168 93293 Mar 20 01:09 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2470 ./12/w2c_errs 166 107497 Mar 20 01:09 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2471 ./13/w2c_errs 166 102016 Mar 20 01:09 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2472 ./14/w2c_errs 166 71934 Mar 20 01:09 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2473 ./10/w2c_errs 162 86168 Mar 20 01:09 |
| 82 | 2474 ./rerun/9/w2c_errs 124 61805 Mar 26 15:01 |
| 2475 ./rerun/11/w2c_errs 122 65295 Mar 26 15:01 | |
| 2476 ./rerun/12/w2c_errs 121 84178 Mar 26 15:01 | |
| 2477 ./rerun/13/w2c_errs 121 65592 Mar 26 15:01 | |
| 2478 ./rerun/14/w2c_errs 121 54378 Mar 26 15:01 | |
| 2479 ./rerun/10/w2c_errs 120 62900 Mar 26 15:01 | |
| 2480 ./rerun/8/w2c_errs 107 53748 Mar 26 15:01 | |
| 2481 ./rerun/4/a_w2c_errs 75 43572 Apr 2 12:48 | |
| 2482 ./surplus/2x/w2c_errs 8 5534 Mar 24 19:10 | |
| 2483 | |
|
83
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2484 Do the extras for 2,3,5,6,7 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2485 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2486 >: sbatch --output=slurm_w2c_plus.out --time=04:00:00 --ntasks=5 -c 2 $HOME/bin/runme.sh $PWD -t 2 -i 'echo 2 3 5 6 7 | xargs -n 1 echo|share_by_task.sh $n $task' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2487 echo $PYTHONPATH |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2488 mkdir -p plus/$arg |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2489 python3 -c "import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))" 2023-40 $arg ./plus "$(cat $arg.pat)" 2> plus/$arg/w2c_errs' |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2490 >: fgrep -c beegfs plus/*/w* |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2491 plus/2/w2c_errs:220 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2492 plus/3/w2c_errs:96 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2493 plus/5/w2c_errs:84 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2494 plus/6/w2c_errs:75 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2495 plus/7/w2c_errs:91 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2496 >: ls -lt plus/*/w* |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2497 -rw-r--r-- 1 hst dc007 131136 Apr 3 12:06 plus/2/w2c_errs |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2498 -rw-r--r-- 1 hst dc007 52260 Apr 3 11:39 plus/3/w2c_errs |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2499 -rw-r--r-- 1 hst dc007 54141 Apr 3 11:36 plus/7/w2c_errs |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2500 -rw-r--r-- 1 hst dc007 49769 Apr 3 11:35 plus/5/w2c_errs |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2501 -rw-r--r-- 1 hst dc007 40819 Apr 3 11:33 plus/6/w2c_errs |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2502 (reverse-i-search)`': ^C |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2503 >: cat [23567].pat |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2504 {680..899}{804..899}{816..899}{825..899}{809..899} |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2505 >: head -1 slurm_w2c_plus.out |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2506 Thu Apr 3 11:12:52 BST 2025 ... |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2507 >: tail -1 slurm_w2c_plus.out |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2508 Thu Apr 3 12:07:02 BST 2025 Finished |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2509 Looks good... |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2510 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2511 Duration 54 minutes =~ 244 per hour for segment 2. Interesting, |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2512 because 3 different nodes were used... |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2513 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2514 Launch another 5 full runs... |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2515 >: sbatch --output=slurm_w2c_8-12.out --time=05:00:00 --ntasks=5 -c 2 $HOME/bin/runme.sh $PWD -t 2 -i 'share_by_task.sh -s 8 12 $n $task' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2516 echo $PYTHONPATH |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2517 mkdir -p $arg |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2518 python3 -c "import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))" 2023-40 $arg . 2> $arg/w2c_errs' |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2519 >: head -1 slurm_w2c_8-12.out |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2520 Thu Apr 3 13:13:49 BST 2025 ... |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2521 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2522 There's a bug in warc.py, if, approximately, the first line of a WARC |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2523 record is the last line in the buffer, see |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2524 >: python3 test_warc.py 1 /beegfs/common_crawl/CC-MAIN-2023-40/1695233506559.11/orig/warc/CC-MAIN-20230924023050-20230924053050-00186.warc.gz 1 | egrep -ac '^WARC-Type: response' |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2525 Traceback (most recent call last): |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2526 File "/work/dc007/dc007/hst/lib/python/cc/test_warc.py", line 25, in <module> |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2527 warc.warc(sys.argv[1],showme,[b'response','warcinfo','request','metadata'],parts=int(sys.argv[2]),debug=debug) |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2528 File "warc.py", line 141, in warc.warc |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2529 raise ValueError("Done but need more! %s + %s > %s in %s"%(bp, |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2530 ValueError: Done but need more! 1050941 + 278 > 1050538 in /beegfs/common_crawl/CC-MAIN-2023-40/1695233506559.11/orig/warc/CC-MAIN-20230924023050-20230924053050-00186.warc.gz |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2531 38660 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2532 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2533 After thrashing around for a few days trying to be clever (see |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2534 abandoned revisions 287 and 288), try a bigger buffer and a trivial |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2535 refill strategy: always have enough room for the largest possible |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2536 record before trying to read one. |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2537 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2538 Get some baseline timings (cythonized) before trying this: |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2539 >: time python3 test_warc.py 1 /beegfs/common_crawl/CC-MAIN-2023-40/1695233506559.11/orig/warc/CC-MAIN-20230924023050-20230924053050-00186.warc.gz 1 |wc -l |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2540 Traceback (most recent call last): |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2541 ... |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2542 1199124 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2543 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2544 real 0m14.601s |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2545 user 0m9.006s |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2546 sys 0m1.158s |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2547 >: time python3 test_warc.py 1 /beegfs/common_crawl/CC-MAIN-2023-40/1695233506559.11/orig/warc/CC-MAIN-20230924023050-20230924053050-00187.warc.gz 1 |wc -l |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2548 1192992 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2549 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2550 real 0m15.055s |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2551 user 0m9.140s |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2552 sys 0m1.207s |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2553 >: time python3 test_warc.py 1 /beegfs/common_crawl/CC-MAIN-2023-40/1695233506559.11/orig/warc/CC-MAIN-20230924023050-20230924053050-00187.warc.gz 1 >/dev/null |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2554 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2555 real 0m14.873s |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2556 user 0m9.358s |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2557 sys 0m1.060s |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2558 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2559 Working, including with problem case, but no faster :-( |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2560 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2561 >: time python3 test_warc.py 1 /beegfs/common_crawl/CC-MAIN-2023-40/1695233506559.11/orig/warc/CC-MAIN-20230924023050-20230924053050-00186.warc.gz 1 | wc -l |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2562 115981 records, max record: 1075305, max header: 4726 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2563 1199124 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2564 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2565 real 0m15.324s |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2566 user 0m10.308s |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2567 sys 0m1.153s |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2568 >: time python3 test_warc.py 1 /beegfs/common_crawl/CC-MAIN-2023-40/1695233506559.11/orig/warc/CC-MAIN-20230924023050-20230924053050-00187.warc.gz 1 | wc -l |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2569 115381 records, max record: 1056240, max header: 3849 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2570 1192992 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2571 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2572 real 0m14.939s |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2573 user 0m10.399s |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2574 sys 0m1.135s |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2575 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2576 Try to do 1 segment using multiple nodes: |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2577 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2578 >: cat 11n.pats |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2579 {000..179} |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2580 {180..359} |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2581 {360..539} |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2582 {540..719} |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2583 {720..899} |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2584 >: sbatch --output=slurm_w2c_11n.out --time=02:00:00 --ntasks=5 -c 2 $HOME/bin/runme.sh $PWD -d -t 2 -i 'cat 11n.pats |share_by_task.sh $n $task' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2585 mkdir -p ./x$task/11 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2586 ~/lib/python/cc/w2c.py 2023-40 11 ./x$task "$arg" 2> ./x$task/11/w2c_errs' |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2587 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2588 Found one more bug, gave up on .., now seems to be working: |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2589 >: cat 11n.pats |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2590 000,179 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2591 180,359 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2592 360,539 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2593 540,719 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2594 720,899 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2595 >: head -1 slurm_*11n* |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2596 Wed Apr 9 20:40:09 BST 2025 Launching 3 nodes in /mnt/lustre/e1000/home/dc007/dc007/hst/results/CC-MAIN-2023-40/warc_lmhx for runme /work/dc007/dc007/hst/results/CC-MAIN-2023-40/warc_lmhx -t 2 -i cat 11n.pats |share_by_task.sh $n $task export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2597 >: date |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2598 Wed Apr 9 20:50:13 BST 2025 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2599 cirrus-login1<6025>: fgrep -c beegfs x?/11/w* |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2600 x0/11/w2c_errs:31 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2601 x1/11/w2c_errs:31 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2602 x2/11/w2c_errs:31 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2603 x3/11/w2c_errs:33 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2604 x4/11/w2c_errs:33 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2605 But not much faster, 4:40 expected runtime for a whole segment of 900 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2606 files at that rate: |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2607 (/ 900.0 (* 32 6)) -> 4.69 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2608 And so it proved: |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2609 >: fgrep -c beegfs x?/11/w* |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2610 x0/11/w2c_errs:180 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2611 x1/11/w2c_errs:180 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2612 x2/11/w2c_errs:180 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2613 x3/11/w2c_errs:180 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2614 x4/11/w2c_errs:180 |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2615 >: tail -1 slurm_*11n* |
|
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2616 Wed Apr 9 21:39:53 BST 2025 Finished |
|
84
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2617 |
|
83
d41cef3d4582
new simpler warc.py did a whole segment, divided in 5 equal parts
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
82
diff
changeset
|
2618 Need some sanity checks before gearing up production... |
|
84
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2619 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2620 >: head -1402 11/w2c_errs | diff -bw - <(fgrep -v records x1/11/w2c_errs) |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2621 [nothing] |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2622 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2623 Argh. Missing a few hits at the end of each file: |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2624 >: python3 -m w2c 2023-40 11 plus 000,002 2> plus/11/w2c_err |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2625 >: grep records plus/11/w2c_errs |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2626 115308 records, max record: 1058293, max header: 20289 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2627 114429 records, max record: 1058293, max header: 20289 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2628 114729 records, max record: 1058293, max header: 20289 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2629 >: uz /beegfs/common_crawl/CC-MAIN-2023-40/1695233506559.11/orig/warc/CC-MAIN-20230924023050-20230924053050-00000.warc.gz |egrep -ac '^WARC-Type: ' |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2630 115408 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2631 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2632 Last record from each buffer-load? No, the entire last buffer-full. |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2633 Fixed. |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2634 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2635 >: python3 -m w2c 2023-40 11 plus 000,002 2> plus/11/w2c_errs |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2636 plus/11/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2637 >: grep records plus/11/w2c_errs |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2638 115408 records, max record: 1058293, max header: 20289 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2639 114508 records, max record: 1058293, max header: 20289 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2640 114865 records, max record: 1058293, max header: 20289 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2641 >: head -12 11/w2c_errs |diff -bw - plus/11/w2c_errs |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2642 2a3,4 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2643 > 115408 records, max record: 1058293, max header: 20289 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2644 > 6186 LM headers, 6185 win, 1 lose, 0 non-ASCII URIs |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2645 4a7,8 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2646 > 114508 records, max record: 1058293, max header: 20289 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2647 > 6149 LM headers, 6148 win, 1 lose, 1 non-ASCII URIs |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2648 12c16,17 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2649 < /beegfs/common_crawl/CC-MAIN-2023-40/1695233506559.11/orig/warc/CC-MAIN-20230924023050-20230924053050-00003.warc.gz |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2650 --- |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2651 > 114865 records, max record: 1058293, max header: 20289 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2652 > 6192 LM headers, 6186 win, 6 lose, 1 non-ASCII URIs |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2653 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2654 Retry the 5-by test: |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2655 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2656 >: fgrep -c beegfs x?/11/w* |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2657 x0/11/w2c_errs:180 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2658 x1/11/w2c_errs:180 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2659 x2/11/w2c_errs:180 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2660 x3/11/w2c_errs:180 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2661 x4/11/w2c_errs:180 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2662 >: head -1 slurm_*11n* |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2663 Thu Apr 10 15:35:12 BST 2025 Launching 3 nodes ... |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2664 >: tail -1 slurm_*11n* |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2665 Thu Apr 10 16:35:48 BST 2025 Finished |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2666 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2667 A minute or two slower. |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2668 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2669 Error logs still look OK: |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2670 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2671 >: head -1402 11/w2c_errs | diff -bw - <(fgrep -v records x1/11/w2c_errs) |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2672 [nothing] |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2673 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2674 Right number of records for 00000 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2675 >: grep records x1/11/w2c_errs |head -3 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2676 115408 records, max record: 1058293, max header: 20289 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2677 114508 records, max record: 1058293, max header: 20289 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2678 114865 records, max record: 1058293, max header: 20289 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2679 >: grep records x1/11/w2c_errs |tail -1 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2680 115216 records, max record: 1079547, max header: 40116 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2681 >: for i in 0 2 3 4; do grep records x$i/11/w2c_errs |tail -1 ; done |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2682 115654 records, max record: 1077742, max header: 40071 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2683 115978 records, max record: 1077845, max header: 16008 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2684 115483 records, max record: 1079052, max header: 25052 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2685 114757 records, max record: 1079623, max header: 40083 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2686 >: diff -bw 11/lmh.cdb_in x1/11/lmh.cdb_in |egrep -v '^<' |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2687 1130232,1174166c1130232 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2688 >: for i in {0..4}; do grep 'LM headers' x$i/11/w2c_errs|cut -f 4 -d ' ' |btot ; done| cut -f 4 -d ' ' |tee >(cat 1>&2) | btot |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2689 1132831 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2690 1130231 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2691 1131595 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2692 1131756 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2693 1131479 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2694 5657892 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2695 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2696 Belt and braces -- rerun all the existing full 900 cases except 11 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2697 >: for d in 0 15 1 surplus/4; do mkdir -p tentative/$d; mv $d/{w2c_errs,lmh.cdb_in} tentative/$d; done |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2698 >: for d in 0; do time ~/lib/python/cc/w2c.py 2023-40 $d . 2> $d/w2c_errs; done & |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2699 Thu Apr 10 10:50:37 PM BST 2025 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2700 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2701 real 279m24.973s |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2702 user 176m3.292s |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2703 sys 18m21.907s |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2704 ./0/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2705 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2706 >: date; for d in 1 2 4 15; do { ~/lib/python/cc/w2c.py 2023-40 $d . 2> $d/w2c_errs ; echo $d finished $(date); } & done |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2707 ./1/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2708 1 finished Fri Apr 11 03:30:01 AM BST 2025 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2709 ./4/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2710 4 finished Fri Apr 11 03:31:44 AM BST 2025 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2711 ./2/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2712 2 finished Fri Apr 11 03:35:17 AM BST 2025 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2713 ./15/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2714 15 finished Fri Apr 11 03:38:49 AM BST 2025 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2715 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2716 >: cut -f 2 <(sed 's/value or format .*/value or format/' 0/w2c_errs)|egrep -v 'warc.gz|LM headers|records,'| sus |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2717 4381 Invalid date value or format |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2718 1 year 4263994 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2719 1 year 4278897 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2720 1 year 4355304 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2721 1 year 4361928 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2722 1 year 4370444 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2723 1 year 4430056 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2724 1 year 4433368 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2725 1 year 53310 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2726 >: cut -f 2 0/ks_errs.tsv | sus |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2727 3188 cannot unpack non-iterable NoneType object |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2728 1193 list index out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2729 1 year 4263994 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2730 1 year 4278897 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2731 1 year 4355304 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2732 1 year 4361928 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2733 1 year 4370444 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2734 1 year 4430056 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2735 1 year 4433368 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2736 1 year 53310 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2737 Same for 4 and 15 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2738 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2739 >: cd 0 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2740 >: wc -l ks.tsv |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2741 6414380 ks.tsv |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2742 >: wc -l lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2743 6414381 lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2744 >: diff -bw <(cut -f2- ks.tsv| sed 's/ //;s/\.0$//'|sort -k1,1) <(sed 's/^\+[0-9,]*://;s/->\([0-9]*\)$/ \1/' lmh.cdb_in | sort -k1,1)|head |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2745 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2746 >: ls -l {tentative/,}{0,4,15}/lmh.cdb_in tentative/surplus/4/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2747 -rw-r--r-- 1 hst dc007 719413017 Apr 11 03:24 0/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2748 -rw-r--r-- 1 hst dc007 719413017 Mar 24 16:52 tentative/0/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2749 -rw-r--r-- 1 hst dc007 603244719 Apr 11 03:38 15/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2750 -rw-r--r-- 1 hst dc007 603244719 Mar 19 22:56 tentative/15/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2751 -rw-r--r-- 1 hst dc007 704148582 Apr 11 03:31 4/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2752 -rw-r--r-- 1 hst dc007 704148582 Mar 27 02:12 tentative/surplus/4/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2753 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2754 But: |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2755 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2756 >: diff -bw <(cut -f2- ks.tsv| sed 's/ //;s/\.0$//'|sort -k1,1) <(sed 's/^\+[0-9,]*://;s/->\([0-9]*\)$/ \1/' lmh.cdb_in | sort -k1,1)|head |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2757 14634c14635 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2758 < 20230930082240https://www.levnepletaciprize.cz/shop/p\u0159\u00EDze-pro-miminka/elen-baby?error=404 1696062160 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2759 --- |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2760 > 20230930082240https://www.levnepletaciprize.cz/shop/p%C5%99%C3%ADze-pro-miminka/elen-baby?error=404 1696062160 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2761 25306c25307 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2762 < 20230930082317https://zums.ac.ir/content/68251/\u0647\u0632\u06CC\u0646\u0647-\u0627\u0639\u0645\u0627\u0644-\u062C\u0631\u0627\u062D\u06CC-\u06F1\u06F4\u06F0\u06F1 1696058653 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2763 --- |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2764 > 20230930082317https://zums.ac.ir/content/68251/%D9%87%D8%B2%DB%8C%D9%86%D9%87-%D8%A7%D8%B9%D9%85%D8%A7%D9%84-%D8%AC%D8%B1%D8%A7%D8%AD%DB%8C-%DB%B1%DB%B4%DB%B0%DB%B1 1696058653 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2765 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2766 Oops! |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2767 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2768 : fgrep -c \\u0 */ks.tsv |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2769 0/ks.tsv:1062 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2770 12/ks.tsv:679 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2771 15/ks.tsv:634 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2772 36/ks.tsv:1311 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2773 4/ks.tsv:770 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2774 56/ks.tsv:1194 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2775 68/ks.tsv:846 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2776 best_two_by_nl1/ks.tsv:1873 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2777 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2778 Even worse: |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2779 >: cd ../../CC-MAIN-2019-35/warc_lmhx |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2780 >: fgrep -c \\u0 ?/ks.tsv |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2781 0/ks.tsv:3 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2782 1/ks.tsv:4 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2783 2/ks.tsv:4 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2784 3/ks.tsv:2 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2785 4/ks.tsv:0 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2786 5/ks.tsv:2 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2787 6/ks.tsv:1 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2788 7/ks.tsv:1 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2789 8/ks.tsv:8 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2790 9/ks.tsv:4 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2791 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2792 Or maybe not? |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2793 >: fgrep \\u0 cluster.idx |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2794 at,wko,news)/news/system/400.html?error=400&errorrequest=/pdf/gen?url=https:/news.wko.at/news/oesterreich/wirtschaftsbarometer_wirtschaftslage.html?shorturl=wirtschaftsbarometerat&key=98c90396b77ff930d6df30ce0f7f7d9c&pdfoptions={"header-html":"\/news\/templates\/header_html.php","footer-html":"https:\/\/news.wko.at\/news\/templates\/footer_html.php?url=https://news.wko.at/news/oesterreich/wirtschaftsbarometer_wirtschaftslage.html?shorturl=wirtschaftsbarometerat","print-media-type":"","title":"wirtschaftsbarometer:+wk\u00d6-konjunkturumfrage"} 20190826093246 cdx-00001.gz 849440852 286441 8844 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2795 >: fgrep -c \\u0 /beegfs/common_crawl/CC-MAIN-2019-35/cdx/cluster.idx |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2796 1 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2797 sing<3771>: fgrep \\u0 /beegfs/common_crawl/CC-MAIN-2019-35/cdx/cluster.idx |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2798 at,wko,news)/news/system/400.html?error=400&errorrequest=/pdf/gen?url=https:/news.wko.at/news/oesterreich/wirtschaftsbarometer_wirtschaftslage.html?shorturl=wirtschaftsbarometerat&key=98c90396b77ff930d6df30ce0f7f7d9c&pdfoptions={"header-html":"\/news\/templates\/header_html.php","footer-html":"https:\/\/news.wko.at\/news\/templates\/footer_html.php?url=https://news.wko.at/news/oesterreich/wirtschaftsbarometer_wirtschaftslage.html?shorturl=wirtschaftsbarometerat","print-media-type":"","title":"wirtschaftsbarometer:+wk\u00d6-konjunkturumfrage"} 20190826093246 cdx-00001.gz 844682060 287090 8844 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2799 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2800 Try using the codecs hack from sort_date.py |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2801 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2802 What's the best segment to test on... |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2803 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2804 >: pwd |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2805 /beegfs/common_crawl/CC-MAIN-2023-40/cdx/warc |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2806 >: uz cdx-00001.gz |egrep -a '\\u[0-9a-z][0-9a-z]'|egrep -o '"filename": "crawl-data.*' | cut -f 4 -d / | cut -f 2 -d . | sus |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2807 79 91 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2808 72 80 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2809 63 52 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2810 62 78 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2811 42 56 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2812 40 15 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2813 ... |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2814 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2815 No, those are mostly (all for seg 15) crawl-diagnostics |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2816 Ah, but, sigh, c-d/.../warc.gz do _have_ timestamps |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2817 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2818 >: uz cdx-0001?.gz |egrep -a '\\u[0-9a-z][0-9a-z]'|egrep -o '"filename": "crawl-data.*/warc/' | cut -f 4 -d / | cut -f 2 -d . | sus |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2819 9 57 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2820 6 97 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2821 4 49 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2822 4 6 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2823 3 11 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2824 ... |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2825 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2826 >: for f in cdx-0001?.gz; do echo -n "$f "; egrep -a '\\u[0-9a-z][0-9a-z]' <(uz $f)|egrep '"filename": "crawl-data.*/warc/' | fgrep -c '.11/warc'; done |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2827 cdx-00010.gz 0 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2828 ... |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2829 cdx-00017.gz 3 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2830 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2831 >: for f in cdx-00017.gz; do egrep -a '\\u[0-9a-z][0-9a-z]' <(uz $f)|egrep '"filename": "crawl-data.*/warc/' | fgrep '.11/warc'; done |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2832 cn,kbstore,auth)/index.php/apireguser/flag/8?code=jdtong&code=jdtong&data={"logo":"http:\/\/jdtong.kbstore.cn\/public\/images\/logo-1.png","logo_url":"http:\/\/jdtong.kbstore.cn\/admin?flag=8","title":"\u8857\u5e97\u901a" [...] "filename": "crawl-data/CC-MAIN-2023-40/segments/1695233506559.11/warc/CC-MAIN-20230924023050-20230924053050-00003.warc.gz", "charset": "UTF-8", "languages": "zho"} |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2833 cn,kbstore,auth)/index.php/apireguser/flag/8?code=jdtong&code=jdtong&data={"logo":"http:\/\/jdtong.kbstore.cn\/public\/images\/logo-1.png","logo_url":"http:\/\/jiediantong.cn\/admin?flag=8","title":"\u8857\u5e97\u901a" ... "filename": "crawl-data/CC-MAIN-2023-40/segments/1695233506559.11/warc/CC-MAIN-20230924023050-20230924053050-00225.warc.gz", "charset": "UTF-8", "languages": "zho"} |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2834 cn,kbstore,auth)/index.php/apireguser/flag/8?code=kbcms&code=kbcms&data={"logo":"http:\/\/demo.cms.vip100.ltd\/public\/images\/logo.png","logo_url":"http:\/\/demo.cms.vip100.ltd\/admin?flag=8","title":"\u4eba\u4eba\u7ad9cms" [...] "filename": "crawl-data/CC-MAIN-2023-40/segments/1695233506559.11/warc/CC-MAIN-20230924023050-20230924053050-00537.warc.gz", "charset": "UTF-8", "languages": "zho"} |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2835 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2836 Red herring chase: none of those have timestamps. |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2837 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2838 So how can this be true: |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2839 >: egrep -c ' [^0] non-ASCII' {0,1,2,4,15}/w2* |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2840 0/w2c_errs:774 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2841 1/w2c_errs:683 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2842 2/w2c_errs:685 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2843 4/w2c_errs:731 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2844 15/w2c_errs:643 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2845 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2846 Because e.g. Korean appears non-uniformly in index files: |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2847 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2848 >: egrep '^kr,' cluster.idx |head -3 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2849 kr,07e,m)/service/agreement 20230924093051 cdx-00221.gz 816253375 185571 1005693 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2850 kr,114n,bit)/sys/login.php?amp;auto_no=568&returnurl=/sys/login.php?auto_no=568&returnurl=http://bit.114n.kr/sys/board.php?bo_id=m71 20230923204454 cdx-00221.gz 816438946 182780 1005694 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2851 kr,1bang)/posts/online-shopping/13076 20231003101132 cdx-00221.gz 816621726184260 1005695 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2852 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2853 >: uz cdx-00221.gz | egrep -c '\\u[0-9a-f]' |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2854 9575 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2855 >: uz cdx-00010.gz | egrep -c '\\u[0-9a-f]' |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2856 746 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2857 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2858 >: ~/lib/python/cc/w2c.py 2023-40 15 . 000 |& less |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2859 >: diff -bw 15/lmh.cdb_in ~/results/CC-MAIN-2023-40/warc_lmhx/15/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2860 98c98 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2861 < +99,10:20230922035528http://cn.coii.kr/board/\u5546\u54C1\u8D2D\u4E70\u540E\u8BB0/4/?link_product_no=19597->1695354928 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2862 --- |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2863 > +117,10:20230922035528http://cn.coii.kr/board/%E5%95%86%E5%93%81%E8%B4%AD%E4%B9%B0%E5%90%8E%E8%AE%B0/4/?link_product_no=19597->1695354928 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2864 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2865 Try the whole thing: |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2866 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2867 >: sbatch --output=slurm_w2c_15f.out --time=02:00:00 --ntasks=5 -c 2 $HOME/bin/runme.sh $PWD -d -t 2 -i 'cat 11n.pats |share_by_task.sh $n $task' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2868 mkdir -p ./fixu/x$task/15 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2869 ~/lib/python/cc/w2c.py 2023-40 15 ./fixu/x$task "$arg" 2> ./fixu/x$task/15/w2c_errs' |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2870 Sun Apr 13 03:45:46 PM BST 2025 end 4 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2871 Sun Apr 13 03:45:46 PM BST 2025 task 5.4 on r1i0n20:2.1 end |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2872 ./fixu/x3/15/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2873 Sun Apr 13 03:45:50 PM BST 2025 end 3 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2874 Sun Apr 13 03:45:50 PM BST 2025 task 5.3 on r1i0n20:2.1 end |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2875 ./fixu/x2/15/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2876 Sun Apr 13 03:46:04 PM BST 2025 end 2 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2877 Sun Apr 13 03:46:04 PM BST 2025 task 5.2 on r1i0n20:2.1 end |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2878 ./fixu/x1/15/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2879 Sun Apr 13 03:46:41 PM BST 2025 end 1 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2880 Sun Apr 13 03:46:41 PM BST 2025 task 5.1 on r1i0n3:2.0 end |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2881 ./fixu/x0/15/lmh.cdb_in |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2882 Sun Apr 13 03:52:49 PM BST 2025 end 0 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2883 Sun Apr 13 03:52:49 PM BST 2025 task 5.0 on r1i0n3:2.0 end |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2884 Sun Apr 13 15:52:49 BST 2025 Finished |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2885 >: head -1 slurm_w2c_15f.out |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2886 Sun Apr 13 14:59:54 BST 2025 Launching 2 nodes in /mnt/lustre/e1000/home/dc007/dc007/hst/results/CC-MAIN-2023-40/warc_lmhx for runme /work/dc007/dc007/hst/results/CC-MAIN-2023-40/warc_lmhx -d -t 2 -i cat 11n.pats |share_by_task.sh $n $task export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2887 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2888 ~50:10 elapsed: Faster! |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2889 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2890 >: diff -bw <(cut -f2- 15/ks.tsv| sed 's/ //;s/\.0$//'|sort -k1,1) <(sed 's/^\+[0-9,]*://;s/->\([0-9-]*\)$/ \1/' fixu/x{0..4}/15/lmh.cdb_in | sort -k1,1)| tee /tmp/hst/udiff | less |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2891 >: cat /tmp/hst/udiff |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2892 0a1,5 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2893 > |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2894 > |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2895 > |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2896 > |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2897 > |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2898 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2899 And correct. |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2900 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2901 Another one: |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2902 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2903 >: sbatch --output=slurm_w2c_4f.out --time=01:30:00 --ntasks=5 -c 2 $HOME/bin/runme.sh $PWD -d -t 2 -i 'cat 11n.pats |share_by_task.sh $n $task' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2904 mkdir -p ./fixu/x$task/4 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2905 ~/lib/python/cc/w2c.py 2023-40 4 ./fixu/x$task "$arg" 2> ./fixu/x$task/4/w2c_errs' |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2906 Submitted batch job 7381438 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2907 cirrus-login1<6096>: fgrep -c beegfs fixu/x?/4/w* | cut -f 2 -d : | btot |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2908 900 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2909 >: head -1 slurm_w2c_4f.out |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2910 Mon Apr 14 08:55:08 BST 2025 Launching 3 nodes in /mnt/lustre/e1000/home/dc007/dc007/hst/results/CC-MAIN-2023-40/warc_lmhx for runme /work/dc007/dc007/hst/results/CC-MAIN-2023-40/warc_lmhx -d -t 2 -i cat 11n.pats |share_by_task.sh $n $task export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2911 >: tail -1 slurm_w2c_4f.out |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2912 Mon Apr 14 09:41:51 BST 2025 Finished |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2913 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2914 46:43 elapsed |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2915 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2916 >: diff -bw <(cut -f2- 4/ks.tsv| sed 's/ //;s/\.0$//'|sort -k1,1) <(sed 's/^\+[0-9,]*://;s/->\([0-9-]*\)$/ \1/' fixu/x{0..4}/4/lmh.cdb_in | sort -k1,1)| tee /tmp/hst/udiff | less |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2917 >: cat /tmp/hst/udiff |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2918 0a1,5 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2919 > |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2920 > |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2921 > |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2922 > |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2923 > |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2924 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2925 And errors: |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2926 >: for d in 4 15; do ls -ld $d; cut -f 2 <(sed 's/value or format .*/value or format/' fixu/x?/$d/w2c_errs)|egrep -v 'warc.gz|LM headers|records,'| sus ; cut -f 2 $d/ks_errs.tsv | sus ; done |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2927 drwxr-sr-x 2 hst dc007 110592 Apr 10 22:50 4 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2928 6107 Invalid date value or format |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2929 8 year 539500 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2930 7 year 539498 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2931 2 year 55722 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2932 1 year 53337 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2933 1 year 539499 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2934 1 year 54618 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2935 1 year 54752 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2936 4605 cannot unpack non-iterable NoneType object |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2937 1502 list index out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2938 8 year 539500 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2939 7 year 539498 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2940 2 year 55722 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2941 1 year 53337 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2942 1 year 539499 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2943 1 year 54618 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2944 1 year 54752 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2945 drwxr-sr-x 2 hst dc007 98304 Apr 10 22:50 15 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2946 8090 Invalid date value or format |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2947 113 year 641471 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2948 20 year 641474 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2949 9 hour must be in 0..23 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2950 9 year 642435 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2951 3 year 642437 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2952 1 year 4262575 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2953 1 year 641769 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2954 1 year 642434 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2955 5420 cannot unpack non-iterable NoneType object |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2956 2670 list index out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2957 113 year 641471 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2958 20 year 641474 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2959 9 hour must be in 0..23 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2960 9 year 642435 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2961 3 year 642437 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2962 1 year 4262575 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2963 1 year 641769 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2964 1 year 642434 is out of range |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2965 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2966 Those were both with warc2cdb.py. Try a 3rd run for timing after |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2967 cythonize |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2968 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2969 >: head -1 slurm_w2c_0f.out |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2970 Mon Apr 14 14:15:55 BST 2025 Launching 1 nodes in /mnt/lustre/e1000/home/dc007/dc007/hst/results/CC-MAIN-2023-40/warc_lmhx for runme /work/dc007/dc007/hst/results/CC-MAIN-2023-40/warc_lmhx -d -t 2 -i cat 11n.pats |share_by_task.sh $n $task export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2971 >: tail -1 slurm_w2c_0f.out |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2972 Mon Apr 14 15:03:41 BST 2025 Finished |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2973 |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2974 47:46 elapsed, mid-day weekday, all on one node... |
|
5aa8b75aa9f0
Confirmed new approach matches ks{,_errs} for the three cases we have: 0, 4 and 15
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
83
diff
changeset
|
2975 Both cross-checks succeed |
|
85
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2976 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2977 Where do we stand _now_? |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2978 >: ff w2c_errs | xargs -n 1 bash -c 'printf "%s\t%s\t%s\n" "$0" $(fgrep -c beegfs/ $0) "$(ls -l $0 | tr -s \ | cut -f 5,6,7,8 -d \ )"' |fgrep Apr | sort -k5nr,5 2>&1 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2979 ./fixu/x0/0/w2c_errs 180 97561 Apr 14 15:02 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2980 ./fixu/x0/4/w2c_errs 180 119330 Apr 14 09:41 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2981 ./fixu/x1/0/w2c_errs 180 92775 Apr 14 15:03 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2982 ./fixu/x1/4/w2c_errs 180 121103 Apr 14 09:39 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2983 ./fixu/x2/0/w2c_errs 180 99136 Apr 14 15:02 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2984 ./fixu/x2/4/w2c_errs 180 122217 Apr 14 09:40 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2985 ./fixu/x3/0/w2c_errs 180 98527 Apr 14 15:03 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2986 ./fixu/x3/4/w2c_errs 180 117299 Apr 14 09:41 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2987 ./fixu/x4/0/w2c_errs 180 97644 Apr 14 15:02 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2988 ./fixu/x4/4/w2c_errs 180 120875 Apr 14 09:41 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2989 ./fixu/x0/15/w2c_errs 180 141002 Apr 13 15:52 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2990 ./fixu/x1/15/w2c_errs 180 142454 Apr 13 15:46 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2991 ./fixu/x2/15/w2c_errs 180 139796 Apr 13 15:46 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2992 ./fixu/x3/15/w2c_errs 180 142030 Apr 13 15:45 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2993 ./fixu/x4/15/w2c_errs 180 141123 Apr 13 15:45 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2994 The above show 0, 4, 15 run in parts, confirmed to match 2024 results |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2995 exactly except for extra log lines from warc and warc2cdb in w2c_errs. |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2996 ./0/w2c_errs 900 485975 Apr 11 03:24 *** |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2997 ./15/w2c_errs 900 706412 Apr 11 03:38 *** |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2998 ./1/w2c_errs 900 607072 Apr 11 03:30 *** |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
2999 ./2/w2c_errs 900 637082 Apr 11 03:35 *** |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3000 ./4/w2c_errs 900 601209 Apr 11 03:31 *** |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3001 Concatenating errs for 0,4,15 match except for warc max counts after the first |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3002 180, which makes sense: |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3003 >: diff <(cat fixu/x{1,2,3,4,0}/0/w*) 0/w2c_errs|head |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3004 1361c1361 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3005 < 114802 records, max record: 1056834, max header: 4513 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3006 --- |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3007 > 114802 records, max record: 1075973, max header: 10302 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3008 1368c1368 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3009 < 115126 records, max record: 1058589, max header: 5466 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3010 --- |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3011 > 115126 records, max record: 1075973, max header: 10302 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3012 1377c1377 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3013 < 114040 records, max record: 1058589, max header: 7318 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3014 >: head -1360 0/w2c_errs |tail -4 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3015 b'' Invalid date value or format "" |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3016 115030 records, max record: 1075973, max header: 10302 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3017 7148 LM headers, 7146 win, 2 lose, 2 non-ASCII URIs |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3018 /beegfs/common_crawl/CC-MAIN-2023-40/1695233510671.0/orig/warc/CC-MAIN-20230930082033-20230930112033-00179.warc.gz |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3019 However, these runs must have been done before the \u fix |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3020 >: diff <(cat fixu/x{1,2,3,4,0}/4/l*) 4/lmh.cdb_in|egrep -c '^[0-9].*a' |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3021 0 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3022 >: diff <(cat fixu/x{1,2,3,4,0}/4/l*) 4/lmh.cdb_in|egrep -c '^[0-9].*d' |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3023 4 [these are the blank lines at the end of the first 4 parts] |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3024 >: diff <(cat fixu/x{1,2,3,4,0}/4/l*) 4/lmh.cdb_in|egrep -c '^[0-9].*c' |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3025 1435 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3026 >: diff <(cat fixu/x{1,2,3,4,0}/4/l*) 4/lmh.cdb_in|egrep -c '^<' |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3027 1495 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3028 >: diff <(cat fixu/x{1,2,3,4,0}/4/l*) 4/lmh.cdb_in|egrep -c '^>' |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3029 1491 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3030 >: diff <(cat fixu/x{1,2,3,4,0}/4/l*) 4/lmh.cdb_in|egrep -ic '[\\]u' |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3031 1491 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3032 >: diff <(cat fixu/x{1,2,3,4,0}/4/l*) 4/lmh.cdb_in|egrep -ic '^<.*[\\]u' |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3033 1491 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3034 Similarly for 0, 15 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3035 ./plus/11/w2c_errs 3 1295 Apr 10 15:36 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3036 ./x0/11/w2c_errs 180 119337 Apr 10 16:31 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3037 ./x1/11/w2c_errs 180 118047 Apr 10 16:31 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3038 ./x2/11/w2c_errs 180 109509 Apr 10 16:35 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3039 ./x3/11/w2c_errs 180 118879 Apr 10 16:27 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3040 ./x4/11/w2c_errs 180 116116 Apr 10 16:30 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3041 That batch was done to test the fix to warc.py for the last buffer |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3042 load being missed, but _not_ the \u fix: |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3043 >: egrep -ic '[\\]u' x?/11/lmh.cdb_in |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3044 x0/11/lmh.cdb_in:0 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3045 x1/11/lmh.cdb_in:0 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3046 x2/11/lmh.cdb_in:0 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3047 x3/11/lmh.cdb_in:0 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3048 x4/11/lmh.cdb_in:0 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3049 So the runs marked *** above must have been done before that fix and are _not valid_. |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3050 And everything else below here is too early to be any use |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3051 ./10/w2c_errs 900 473108 Apr 3 17:10 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3052 ./11/w2c_errs 188 103194 Apr 3 14:03 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3053 ./12/w2c_errs 900 605957 Apr 3 17:09 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3054 ./8/w2c_errs 900 468236 Apr 3 17:01 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3055 ./9/w2c_errs 900 453757 Apr 3 17:03 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3056 ./plus/2/w2c_errs 220 131136 Apr 3 12:06 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3057 ./plus/3/w2c_errs 96 52260 Apr 3 11:39 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3058 ./plus/5/w2c_errs 84 49769 Apr 3 11:35 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3059 ./plus/6/w2c_errs 75 40819 Apr 3 11:33 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3060 ./plus/7/w2c_errs 91 54141 Apr 3 11:36 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3061 ./rerun/4/a_w2c_errs 75 43572 Apr 2 12:48 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3062 Next steps: |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3063 Check robots.txt for LM headers: Yes |
| 88 | 3064 Fetched robotstxt for 2019-35 Done |
|
85
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3065 Ran warc2cdb on 2019-35 robotstxt, crawldiagnostics (after fixing |
| 88 | 3066 bug (NOT CLEAR why I added " need to fill this in some how" to the |
| 3067 lmh.cdb_in value for revisit records or what it means!) and warc: | |
|
85
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3068 >: time ~/lib/python/cc/w2c.py 2019-35 0 . crawldiagnostics 2> 0/crawldiagnostics/w2c_errs |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3069 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3070 ./0/crawldiagnostics/lmh.cdb_in |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3071 real 7m13.989s |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3072 user 6m24.400s |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3073 sys 0m15.013s |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3074 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3075 >: time ~/lib/python/cc/w2c.py 2019-35 0 . robotstxt 2> 0/robotstxt/w2c_errs |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3076 ./0/robotstxt/lmh.cdb_in |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3077 real 1m11.277s |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3078 user 0m57.372s |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3079 sys 0m3.942s |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3080 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3081 >: time ~/lib/python/cc/w2c.py 2019-35 0 ./x0 warc 000,185 2> x0/0/warc/w2c_errs & |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3082 >: less x0/0/warc/w2c_errs |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3083 >: time ~/lib/python/cc/w2c.py 2019-35 0 ./x1 warc 186,371 2> x1/0/warc/w2c_errs & |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3084 >: time ~/lib/python/cc/w2c.py 2019-35 0 ./x2 warc 372,559 2> x2/0/warc/w2c_errs & |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3085 ./x0/0/warc/lmh.cdb_in |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3086 real 54m59.239s |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3087 user 34m34.656s |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3088 sys 2m39.790s |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3089 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3090 ./x1/0/warc/lmh.cdb_in |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3091 real 55m15.233s |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3092 user 34m54.959s |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3093 sys 2m42.281s |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3094 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3095 ./x2/0/warc/lmh.cdb_in |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3096 real 55m48.226s |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3097 user 35m14.087s |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3098 sys 2m45.434s |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3099 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3100 Cross-check warc output: |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3101 >: wc -l x?/0/warc/l* |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3102 1763032 x0/0/warc/lmh.cdb_in |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3103 1763752 x1/0/warc/lmh.cdb_in |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3104 1781459 x2/0/warc/lmh.cdb_in |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3105 5308243 total |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3106 >: wc -l 0/ks.tsv |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3107 5308240 0/ks.tsv |
|
86
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3108 >: diff -bw <(cut -f2- 0/ks.tsv| fgrep $'\t'| sed 's/ //;s/\.0$//'|sort -k1,1) <(sed 's/^\+[0-9,]*://;s/->\([0-9-]*\)$/ \1/' x{0..2}/0/warc/lmh.cdb_in | sort -k1,1)| tee /tmp/hst/0diff |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3109 >: cat /tmp/hst/0diff |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3110 0a1,3 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3111 > |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3112 > |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3113 > |
|
85
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3114 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3115 >: wc -l 0/ks_errs.tsv |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3116 21462 0/ks_errs.tsv |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3117 >: wc -l x?/0/warc/w* |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3118 7621 x0/0/warc/w2c_errs |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3119 7768 x1/0/warc/w2c_errs |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3120 7753 x2/0/warc/w2c_errs |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3121 23142 total |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3122 >: echo $(((23142 - 21462))) |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3123 1680 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3124 >: echo $((560*3)) |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3125 1680 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3126 >: cut -f 2 0/ks_errs.tsv | sus |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3127 14749 cannot unpack non-iterable NoneType object |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3128 6604 list index out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3129 26 year 641480 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3130 24 year 641788 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3131 22 year 641785 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3132 21 year 641787 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3133 5 hour must be in 0..23 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3134 4 year 641791 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3135 3 year 641474 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3136 2 year 641484 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3137 1 year 640818 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3138 1 year 641778 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3139 >: echo $((14749 + 6604)) |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3140 21353 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3141 >: cut -f 2 <(sed 's/value or format .*/value or format/' ./x?/0/warc/w2c_errs)|egrep -v 'warc.gz|LM headers|records,'| sus |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3142 21353 Invalid date value or format |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3143 26 year 641480 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3144 24 year 641788 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3145 22 year 641785 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3146 21 year 641787 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3147 5 hour must be in 0..23 |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3148 4 year 641791 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3149 3 year 641474 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3150 2 year 641484 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3151 1 year 640818 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3152 1 year 641778 is out of range |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3153 All good |
|
19e2136f595b
switch to robotstxt crawldiagnostics,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
84
diff
changeset
|
3154 |
|
86
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3155 This didn't work: |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3156 >: sbatch --output=slurm_1-10x5.out --time=02:00:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh -m 1 10 $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3157 j=$((i - 1)) |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3158 mkdir -p ./x$j/$xarg/warc |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3159 ~/lib/python/cc/w2c.py 2019-35 $xarg ./x$j warc "$arg" 2> ./x$j/$xarg/warc/w2c_errs' |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3160 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3161 so reverted to not using -m, and slightly modified versions of |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3162 (_)runme.sh, and this: |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3163 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3164 >: sbatch --output=slurm_1-10x5.out --time=01:20:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3165 j=$((i - 1)) |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3166 mkdir -p ./x$j/$task/warc |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3167 ~/lib/python/cc/w2c.py 2019-35 $task ./x$j warc "$arg" 2> ./x$j/$task/warc/w2c_errs' |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3168 >: head -1 slurm_1-10x5.out |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3169 Wed Apr 23 10:55:08 BST 2025 Launching 5 nodes |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3170 is working, I think: |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3171 >: fgrep -ch beegfs x?/?/warc/w* | sort -nr | { sed -u 3q ;echo === ; tail -3 ; } |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3172 61 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3173 61 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3174 60 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3175 === |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3176 58 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3177 58 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3178 58 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3179 >: date |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3180 Wed Apr 23 11:22:20 AM BST 2025 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3181 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3182 So half-way in 27 minutes, so should finish in about an hour... |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3183 Better: |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3184 >: tail -1 slurm_1-10x5.out |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3185 Wed Apr 23 11:45:19 BST 2025 Finished |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3186 But: |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3187 >: fgrep -h beegfs x?/*/warc/w* | cut -f 5,8 -d / | tr ./ - | wc -l |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3188 5544 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3189 E.g. |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3190 >: fgrep -c beegfs x0/7/warc/w2c_errs |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3191 109 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3192 >: fgrep -h beegfs x0/7/warc/w* | cut -f 5,8 -d / | tr ./ - | cut -f 2,7 -d - | less -N |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3193 1 7-00000 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3194 ... |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3195 107 7-00106 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3196 108 7-00107 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3197 109 7-00108 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3198 (END) |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3199 So just a premature termination? Yes: |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3200 >: tail -15 slurm_1-10x5.out |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3201 ./x2/3/warc/lmh.cdb_in |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3202 Wed Apr 23 11:45:17 AM BST 2025 end 3 3 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3203 /work/dc007/dc007/hst/bin/_runme.sh: line 101: rallel: command not found |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3204 /work/dc007/dc007/hst/bin/_runme.sh: line 102: syntax error near unexpected token `fi' |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3205 /work/dc007/dc007/hst/bin/_runme.sh: line 102: ` fi' |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3206 srun: error: r1i1n1: task 3: Exited with exit code 2 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3207 srun: launch/slurm: _step_signal: Terminating StepId=7388166.0 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3208 slurmstepd: error: *** STEP 7388166.0 ON r1i0n10 CANCELLED AT 2025-04-23T11:45:18 *** |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3209 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3210 Bizarre -- some kind of file-system glitch? File looks OK now... |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3211 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3212 Try this, just to see if we can trigger it more easily... |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3213 >: sbatch --output=slurm_0-9x5rtcd.out --time=01:20:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3214 j=$((i - 1)) |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3215 mkdir -p ./x$j/$task/robotstxt ; mkdir -p ./x$j/$task/crawldiagnostics |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3216 ~/lib/python/cc/w2c.py 2019-35 $task ./x$j robotstxt "$arg" 2> ./x$j/$task/robotstxt/w2c_errs |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3217 ~/lib/python/cc/w2c.py 2019-35 $task ./x$j crawldiagnostics "$arg" 2> ./x$j/$task/crawldiagnostics/w2c_errs' |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3218 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3219 >: fgrep -ch beegfs x?/?/robotstxt/w* | sort -nr | headtail -3 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3220 112 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3221 112 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3222 112 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3223 === |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3224 112 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3225 112 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3226 112 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3227 >: fgrep -ch beegfs x?/?/crawldiagnostics/w* | sort -nr | headtail -3 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3228 112 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3229 112 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3230 112 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3231 === |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3232 112 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3233 112 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3234 112 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3235 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3236 A quick check suggests it's at least starting out OK |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3237 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3238 >: head -4274 0/ks_errs.tsv | cut -f 1 | diff - <(egrep -v 'beegfs|records|headers' x0/0/warc/w2c_errs|cut -f 1|sed "s/^b'\(.*\)'$/\1/") | headtail -18 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3239 46,47c46,47 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3240 < Чт, 18 Июл 2019 12:38:01 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3241 < Пт, 14 Июн 2019 10:03:52 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3242 --- |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3243 > \xd0\xa7\xd1\x82, 18 \xd0\x98\xd1\x8e\xd0\xbb 2019 12:38:01 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3244 > \xd0\x9f\xd1\x82, 14 \xd0\x98\xd1\x8e\xd0\xbd 2019 10:03:52 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3245 70,71c70,71 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3246 < Пт, 14 Июн 2019 11:46:10 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3247 < Вт, 11 Июн 2019 14:11:46 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3248 --- |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3249 > \xd0\x9f\xd1\x82, 14 \xd0\x98\xd1\x8e\xd0\xbd 2019 11:46:10 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3250 > \xd0\x92\xd1\x82, 11 \xd0\x98\xd1\x8e\xd0\xbd 2019 14:11:46 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3251 173c173 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3252 < Пт, 14 Июн 2019 08:48:59 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3253 --- |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3254 > \xd0\x9f\xd1\x82, 14 \xd0\x98\xd1\x8e\xd0\xbd 2019 08:48:59 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3255 185c185 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3256 < {ts '2019-08-17 18:56:12'} |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3257 === |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3258 4086,4087c4086,4087 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3259 < 土, 17 8月 2019 23:42:13 GMT |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3260 < 土, 17 8月 2019 23:50:00 GMT |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3261 --- |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3262 > \xe5\x9c\x9f, 17 8\xe6\x9c\x88 2019 23:42:13 GMT |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3263 > \xe5\x9c\x9f, 17 8\xe6\x9c\x88 2019 23:50:00 GMT |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3264 4141c4141 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3265 < Сб, 13 Июл 2019 00:21:19 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3266 --- |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3267 > \xd0\xa1\xd0\xb1, 13 \xd0\x98\xd1\x8e\xd0\xbb 2019 00:21:19 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3268 4208c4208 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3269 < {ts '2019-08-17 19:48:43'} |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3270 --- |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3271 > b"{ts '2019-08-17 19:48:43'}" |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3272 4261c4261 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3273 < 土, 17 8月 2019 23:08:06 GMT |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3274 --- |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3275 > \xe5\x9c\x9f, 17 8\xe6\x9c\x88 2019 23:08:06 GMT |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3276 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3277 And finishing? Yes, allowing for the fact that segment 0 was cut a |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3278 tiny bit short: |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3279 >: tail -4307 0/ks_errs.tsv | head -4259 |cut -f 1 | diff - <(egrep -v 'beegfs|records|headers' x4/0/warc/w2c_errs|cut -f 1|sed "s/^b'\(.*\)'$/\1/") | headtail -12 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3280 60c60 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3281 < 土, 17 8月 2019 23:15:01 GMT |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3282 --- |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3283 > \xe5\x9c\x9f, 17 8\xe6\x9c\x88 2019 23:15:01 GMT |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3284 87c87 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3285 < {ts '2019-08-17 19:17:39'} |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3286 --- |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3287 > b"{ts '2019-08-17 19:17:39'}" |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3288 120c120 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3289 < Пт, 07 Дек 2018 08:14:56 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3290 --- |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3291 > \xd0\x9f\xd1\x82, 07 \xd0\x94\xd0\xb5\xd0\xba 2018 08:14:56 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3292 === |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3293 4207,4208c4207,4208 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3294 < 土, 17 8月 2019 22:34:41 GMT |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3295 < 土, 17 8月 2019 22:33:05 GMT |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3296 --- |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3297 > \xe5\x9c\x9f, 17 8\xe6\x9c\x88 2019 22:34:41 GMT |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3298 > \xe5\x9c\x9f, 17 8\xe6\x9c\x88 2019 22:33:05 GMT |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3299 4255,4259d4254 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3300 < 8/17/2019 7:17:15 PM GMT |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3301 < 08/18/19, 18 37 2019 00:%i:1566081424 GMT |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3302 < 08/18/19, 18 28 2019 01:%i:1566084517 GMT |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3303 < |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3304 < |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3305 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3306 Similarly for segment 9 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3307 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3308 |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3309 So try the real thing again |
|
855665c79545
trying to (re)do 2019-35 segs 0-9, having done robotstxt and crawldiagnostics
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
85
diff
changeset
|
3310 Wed Apr 23 17:00:25 BST 2025 Launching 5 nodes |
|
87
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3311 >: fgrep -ch beegfs x?/?/warc/w* | sort -nr | headtail -3 |
|
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3312 112 |
|
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3313 112 |
|
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3314 112 |
|
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3315 === |
|
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3316 112 |
|
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3317 112 |
|
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3318 112 |
|
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3319 >: headtail -1 slurm_0-9x5w.out |
| 88 | 3320 Wed Apr 23 17:00:25 BST 2025 Launching 5 nodes ... |
|
87
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3321 === |
|
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3322 Wed Apr 23 17:49:15 BST 2025 Finished |
|
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3323 >: python3 -c 'import sys,datetime;(s,d1,d2)=sys.argv[1:4];dd2=datetime.datetime.strptime(d2[4:22],"%b %d %I:%M:%S %p");dd1=datetime.datetime.strptime(d1[4:19],"%b %d %H:%M:%S");d=(dd2-dd1).seconds;h=d/3600;print(a:=int(s)/50.0,"%d:%0.2d:%0.2d"%(h,(d:=(d%3600))/60,d % 60))' $(fgrep -ch beegfs x?/?/warc/w* | btot) 'Wed Apr 23 17:00:25 BST 2025' "Wed Apr 23 05:49:15 PM" |
|
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3324 112.0 0:48:50 |
|
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3325 |
|
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3326 All good |
|
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3327 |
|
90b9c202922c
successful redo 2019-35 segs 0-9
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
86
diff
changeset
|
3328 Nothing but the same utf-8 mismatches in e.g. segment 5: |
| 88 | 3329 >: diff -bw <(cut -f 1 5/ks_errs.tsv) <(cat x{0..4}/5/warc/w2c_errs |
| 3330 | egrep -v 'beegfs|records|headers' |cut -f 1|sed "s/^b'\(.*\)'$/\1/") | |
| 3331 | |
| 3332 And lmh.cdb_in is good: | |
| 3333 >: diff -bw <(cut -f2- 5/ks.tsv| fgrep $'\t'| sed 's/ //;s/\.0$//'|sort -k1,1) <(sed 's/^\+[0-9,]*://;s/->\([0-9-]*\)$/ \1/' x{0..4}/5/warc/lmh.cdb_in | sort -k1,1)| tee /tmp/hst/5diff | |
| 3334 >: cat /tmp/hst/5diff | |
| 3335 0a1,5 | |
| 3336 > | |
| 3337 > | |
| 3338 > | |
| 3339 > | |
| 3340 > | |
| 3341 | |
| 3342 Next decile: | |
| 3343 | |
| 3344 >: sbatch --output=slurm_10-19x5w.out --time=01:20:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh | |
| 3345 j=$((i - 1)) ; tp=$((task + 10)) | |
| 3346 mkdir -p ./x$j/$tp/warc | |
| 3347 ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j warc "$arg" 2> ./x$j/$tp/warc/w2c_errs' | |
| 3348 >: head slurm_10-19x5w.out | |
| 3349 Thu Apr 24 10:34:12 BST 2025 Launching 5 nodes | |
| 3350 >: tail -1 slurm_10-19x5w.out | |
| 3351 Thu Apr 24 11:24:13 BST 2025 Finished | |
| 3352 >: wprog.sh 'Thu Apr 24 10:38:13 AM BST 2025' 'Thu Apr 24 11:24:13 AM' | |
| 3353 112.00 0:46:00 146.09 | |
| 3354 | |
| 3355 Checked 15/.../w2c_errs vs 15/ks_errs.tsv, and likewise lmh.cdb_in, ok | |
| 3356 | |
| 3357 Fill in r_t and c_d | |
| 3358 >: for ty in robotstxt crawldiagnostics; do sbatch --output=slurm_10-19x5${ty:1}.out --time=00:20:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh | |
| 3359 j=$((i - 1)) ; tp=$((task + 10)) ; ty='$ty' | |
| 3360 mkdir -p ./x$j/$tp/$ty | |
| 3361 ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg" 2>./x$j/$tp/$ty/w2c_errs'; done | |
| 3362 >: headtail -1 slurm_10-19x5obotstxt.out | |
| 3363 Thu Apr 24 12:39:46 BST 2025 Launching 5 nodes ... | |
| 3364 === | |
| 3365 Thu Apr 24 12:40:33 BST 2025 Finished | |
| 3366 >: headtail -1 slurm_10-19x5rawldiagnostics.out | |
| 3367 Thu Apr 24 12:38:47 BST 2025 Launching 5 nodes ... | |
| 3368 === | |
| 3369 Thu Apr 24 12:41:22 BST 2025 Finished | |
| 3370 >: wprog.sh 'Thu Apr 24 12:39:46 PM BST 2025' 'Thu Apr 24 12:40:33 PM' robotstxt | |
| 3371 112.00 0:00:47 8578.72 | |
| 3372 >: wprog.sh 'Thu Apr 24 12:38:47 PM BST 2025' 'Thu Apr 24 12:41:22 PM' crawldiagnostics | |
| 3373 | |
| 3374 Oops: | |
| 3375 >: lss x?/1?/robotstxt/w2c_errs | |
| 3376 -rw-r--r-- 1 hst dc007 28753 Apr 24 12:40 x1/10/robotstxt/w2c_errs | |
| 3377 -rw-r--r-- 1 hst dc007 28008 Apr 24 12:40 x2/10/robotstxt/w2c_errs | |
| 3378 -rw-r--r-- 1 hst dc007 27974 Apr 24 12:40 x0/10/robotstxt/w2c_errs | |
| 3379 -rw-r--r-- 1 hst dc007 27613 Apr 24 12:40 x4/10/robotstxt/w2c_errs | |
| 3380 -rw-r--r-- 1 hst dc007 26894 Apr 24 12:40 x3/10/robotstxt/w2c_errs | |
| 3381 -rw-r--r-- 1 hst dc007 13328 Apr 24 12:40 x1/11/robotstxt/w2c_errs | |
| 3382 -rw-r--r-- 1 hst dc007 13328 Apr 24 12:40 x1/12/robotstxt/w2c_errs | |
| 3383 -rw-r--r-- 1 hst dc007 13328 Apr 24 12:40 x1/13/robotstxt/w2c_errs | |
| 3384 -rw-r--r-- 1 hst dc007 13328 Apr 24 12:40 x1/14/robotstxt/w2c_errs | |
| 3385 .... | |
| 3386 >: ls -d /beegfs/common_crawl/CC-MAIN-2019-35/*/orig/robotstxt/|wc -l | |
| 3387 11 | |
| 3388 | |
| 3389 But c_d looks ok | |
| 3390 | |
| 3391 >: ls -d /beegfs/common_crawl/CC-MAIN-2019-35/*/orig/crawldiagnostics/|wc -l | |
| 3392 100 | |
| 3393 >: fgrep -c records x?/19/{robotstxt,crawldiagnostics}/w2c_errs | |
| 3394 x0/19/robotstxt/w2c_errs:0 | |
| 3395 x1/19/robotstxt/w2c_errs:0 | |
| 3396 x2/19/robotstxt/w2c_errs:0 | |
| 3397 x3/19/robotstxt/w2c_errs:0 | |
| 3398 x4/19/robotstxt/w2c_errs:0 | |
| 3399 x0/19/crawldiagnostics/w2c_errs:112 | |
| 3400 x1/19/crawldiagnostics/w2c_errs:112 | |
| 3401 x2/19/crawldiagnostics/w2c_errs:112 | |
| 3402 x3/19/crawldiagnostics/w2c_errs:112 | |
| 3403 x4/19/crawldiagnostics/w2c_errs:112 | |
| 3404 | |
| 3405 Try all three for the next decile: | |
| 3406 >: for ty in warc robotstxt crawldiagnostics; do sbatch --output=slurm_20-29x5${ty::1}.out --time=01:00:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh | |
| 3407 j=$((i - 1)) ; tp=$((task + 20)) ; ty='$ty' | |
| 3408 mkdir -p ./x$j/$tp/$ty | |
| 3409 ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg" 2> ./x$j/$tp/$ty/w2c_errs'; done | |
| 3410 | |
| 3411 Started this prematurely before discovering the robotstxt were missing | |
| 3412 -- weirdly warc didn't seem to run... | |
| 3413 c_d ran fine: | |
| 3414 >: headtail -1 slurm_20-29x5c.out | |
| 3415 Thu Apr 24 13:09:47 BST 2025 Launching 5 nodes ... | |
| 3416 === | |
| 3417 Thu Apr 24 13:12:32 BST 2025 Finished | |
| 3418 >: find . -type f -mtime -1 -name w2c_errs | xargs ls -lt | head | |
| 3419 -rw-r--r-- 1 hst dc007 31051 Apr 24 13:12 ./x2/23/crawldiagnostics/w2c_errs | |
| 3420 -rw-r--r-- 1 hst dc007 30541 Apr 24 13:12 ./x0/23/crawldiagnostics/w2c_errs | |
| 3421 -rw-r--r-- 1 hst dc007 31139 Apr 24 13:12 ./x3/23/crawldiagnostics/w2c_errs | |
| 3422 -rw-r--r-- 1 hst dc007 29927 Apr 24 13:12 ./x3/25/crawldiagnostics/w2c_errs | |
| 3423 >: fgrep -c records x?/29/cra*/w2* | |
| 3424 x0/29/crawldiagnostics/w2c_errs:112 | |
| 3425 x1/29/crawldiagnostics/w2c_errs:112 | |
| 3426 x2/29/crawldiagnostics/w2c_errs:112 | |
| 3427 x3/29/crawldiagnostics/w2c_errs:112 | |
| 3428 x4/29/crawldiagnostics/w2c_errs:112 | |
| 3429 | |
| 3430 Fetching the missing r_t: | |
| 3431 seq 11 99 | /usr/bin/parallel -j 16 time ./getrt.aws CC-MAIN-2019-35 '{}' | |
|
49
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
3432 ... |
| 88 | 3433 real 9m38.739s |
| 3434 user 23m8.073s | |
| 3435 sys 17m36.368s | |
| 3436 | |
| 3437 real 9m39.145s | |
| 3438 user 23m13.342s | |
| 3439 sys 17m45.764s | |
|
51
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
3440 ... |
| 88 | 3441 |
| 3442 Something weird: | |
| 3443 >: wc -l < rerrlog | |
| 3444 56017 | |
| 3445 >: cat rerrlog |cut -f 9 -d \- |sus | |
| 3446 101 00415.warc.gz | |
| 3447 101 00416.warc.gz | |
| 3448 101 00417.warc.gz | |
| 3449 101 00418.warc.gz | |
| 3450 101 00419.warc.gz | |
| 3451 101 00420.warc.gz | |
| 3452 101 00421.warc.gz | |
| 3453 101 00422.warc.gz | |
| 3454 101 00423.warc.gz | |
| 3455 101 00424.warc.gz | |
| 3456 101 00425.warc.gz | |
| 3457 101 00426.warc.gz | |
| 3458 101 00427.warc.gz | |
| 3459 101 00428.warc.gz | |
| 3460 101 00429.warc.gz | |
| 3461 101 00430.warc.gz | |
| 3462 101 00431.warc.gz | |
| 3463 100 00000.warc.gz | |
| 3464 | |
| 3465 I had indeed started a sequential version of fetching 11..99, which | |
| 3466 got up as far as 415 (I thought). Maybe a few stragglers came in | |
| 3467 later? No, they were _logged_ before they were _fetched_, so they | |
| 3468 were logged again by the parallel version, fetched again successfully | |
| 3469 and so dupicated in the log. | |
| 3470 | |
| 3471 And indeed the log begins with 11 x 560 = 6160 lines for segs 0--10, | |
| 3472 then 431 lines for seg 11, then, after a gap of 25 minutes, lines for | |
| 3473 a mixture of segments, eventually picking up segment 11 again at 415. | |
|
53
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
3474 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
3475 All good. |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
3476 |
| 88 | 3477 warc did run, just several hours later: |
| 3478 >: headtail -1 slurm_20-29x5w.out | |
| 3479 Thu Apr 24 15:25:02 BST 2025 Launching 5 nodes ... | |
| 3480 === | |
| 3481 Thu Apr 24 16:13:10 BST 2025 Finished | |
| 3482 | |
| 3483 Better check of error files: | |
| 3484 >: diff -bw <(cut -f 1 25/ks_errs.tsv|python3 -c 'import sys | |
| 3485 while c:=sys.stdin.buffer.read(1): | |
| 3486 if (i:=ord(c)) > 127: | |
| 3487 sys.stdout.buffer.write(b"\\x%02x"%i) | |
| 3488 else: | |
| 3489 sys.stdout.buffer.write(c) | |
| 3490 ') <(cat x{0..4}/25/warc/w2c_errs| egrep -v 'beegfs|records|headers' |cut -f 1|sed "s/^b.\(.*\).$/\1/") | |
|
53
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
3491 [nothing] |
| 88 | 3492 >: diff -bw <(cut -f2- 25/ks.tsv| fgrep $'\t'| sed 's/ //;s/\.0$//'|sort -k1,1) <(sed 's/^\+[0-9,]*://;s/->\([0-9-]*\)$/ \1/' x{0..4}/25/warc/lmh.cdb_in | sort -k1,1) |
| 3493 0a1,5 | |
| 3494 > | |
| 3495 > | |
| 3496 > | |
| 3497 > | |
| 3498 > | |
| 3499 | |
| 3500 rerun robots | |
| 3501 >: for ty in robotstxt; do sbatch --output=slurm_10-29x5${ty::1}.out --time=00:10:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh | |
| 3502 j=$((i - 1)) ; tp=$((task + 10)) ; ty='$ty' | |
| 3503 mkdir -p ./x$j/$tp/$ty | |
| 3504 ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg" 2> ./x$j/$tp/$ty/w2c_errs'; done | |
| 3505 That didn't do 20, only 10, d'oh | |
| 3506 >: for ty in robotstxt; do sbatch --output=slurm_20-29x5${ty::1}.out --time=00:10:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh | |
| 3507 j=$((i - 1)) ; tp=$((task + 20)) ; ty='$ty' | |
| 3508 mkdir -p ./x$j/$tp/$ty | |
| 3509 ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg" 2> ./x$j/$tp/$ty/w2c_errs'; done | |
| 3510 | |
| 3511 Looks OK | |
| 3512 | |
| 3513 Next decile | |
| 3514 >: for ty in warc crawldiagnostics robotstxt; do sbatch --output=slurm_30-39x5${ty::1}.out --time=01:20:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh | |
| 3515 j=$((i - 1)) ; tp=$((task + 30)) ; ty='$ty' | |
| 3516 mkdir -p ./x$j/$tp/$ty | |
| 3517 ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg" 2> ./x$j/$tp/$ty/w2c_errs'; done | |
| 3518 Submitted batch job 7394434 | |
| 3519 Submitted batch job 7394435 | |
| 3520 Submitted batch job 7394436 | |
| 3521 >: head -1 slurm_30-39x5w.out | |
| 3522 Mon Apr 28 11:10:45 BST 2025 | |
| 3523 | |
| 3524 Looks like maybe some disk contention... | |
| 3525 >: head -1 slurm_30-39x5w.out | |
| 3526 >: date | |
| 3527 Mon Apr 28 11:20:45 AM BST 2025 | |
| 3528 >: fgrep -c records x4/3?/*/w2c_errs | |
| 3529 x4/30/crawldiagnostics/w2c_errs:20 | |
| 3530 x4/30/robotstxt/w2c_errs:58 | |
| 3531 x4/30/warc/w2c_errs:2 | |
| 3532 x4/31/crawldiagnostics/w2c_errs:11 | |
| 3533 x4/31/robotstxt/w2c_errs:64 | |
| 3534 x4/31/warc/w2c_errs:1 | |
| 3535 x4/32/crawldiagnostics/w2c_errs:16 | |
| 3536 x4/32/robotstxt/w2c_errs:63 | |
| 3537 x4/32/warc/w2c_errs:4 | |
| 3538 x4/33/crawldiagnostics/w2c_errs:21 | |
| 3539 x4/33/robotstxt/w2c_errs:59 | |
| 3540 x4/33/warc/w2c_errs:0 | |
| 3541 x4/34/crawldiagnostics/w2c_errs:16 | |
| 3542 x4/34/robotstxt/w2c_errs:63 | |
| 3543 x4/34/warc/w2c_errs:0 | |
| 3544 x4/35/crawldiagnostics/w2c_errs:19 | |
| 3545 x4/35/robotstxt/w2c_errs:64 | |
| 3546 x4/35/warc/w2c_errs:1 | |
| 3547 x4/36/crawldiagnostics/w2c_errs:16 | |
| 3548 x4/36/robotstxt/w2c_errs:60 | |
| 3549 x4/36/warc/w2c_errs:4 | |
| 3550 x4/37/crawldiagnostics/w2c_errs:21 | |
| 3551 x4/37/robotstxt/w2c_errs:66 | |
| 3552 x4/37/warc/w2c_errs:1 | |
| 3553 x4/38/crawldiagnostics/w2c_errs:16 | |
| 3554 x4/38/robotstxt/w2c_errs:65 | |
| 3555 x4/38/warc/w2c_errs:0 | |
| 3556 x4/39/crawldiagnostics/w2c_errs:13 | |
| 3557 x4/39/robotstxt/w2c_errs:61 | |
| 3558 x4/39/warc/w2c_errs:0 | |
| 3559 | |
| 3560 Yes -- after 10 minutes only r_t is even half-way done... | |
| 3561 We'll see if things speed up as the r_t and c_d finish | |
| 3562 | |
| 3563 >: headtail -1 slurm_30-39x5r.out | |
| 3564 Mon Apr 28 11:10:45 BST 2025 Launching 5 nodes ... | |
| 3565 === | |
| 3566 Mon Apr 28 11:29:30 BST 2025 Finished | |
| 3567 >: headtail -1 slurm_30-39x5c.out | |
| 3568 Mon Apr 28 11:10:45 BST 2025 Launching 5 nodes ... | |
| 3569 === | |
| 3570 Mon Apr 28 11:49:56 BST 2025 Finished | |
| 3571 | |
| 3572 Maybe a bit, but not enough: | |
| 3573 >: headtail -1 slurm_30-39x5w.out | |
| 3574 Mon Apr 28 11:10:45 BST 2025 Launching 5 nodes ... | |
| 3575 === | |
| 3576 slurmstepd: error: *** STEP 7394434.0 ON r1i1n34 CANCELLED AT 2025-04-28T12:30:43 DUE TO TIME LIMIT *** | |
| 3577 | |
| 3578 Reran w. only warc, but looks like still too slow: | |
| 3579 >: wprog.sh '3?' 'Mon Apr 28 13:34:05 BST 2025' | |
| 3580 35.36 0:30:29 69.60 | |
| 3581 | |
| 3582 Wait a bit longer, doesn't need to speed up much... | |
| 3583 Not even close, in the end: | |
| 3584 >: wprog.sh '3?' 'Mon Apr 28 13:34:05 BST 2025' | |
| 3585 74.48 1:19:07 56.48 | |
| 3586 Reran w. 02:20 | |
| 3587 | |
| 3588 Bother, one glitch, segment 35 is short: | |
| 3589 >: fgrep -c records x0/35/warc/w2c_errs | |
| 3590 x0/35/warc/w2c_errs:112 | |
| 3591 x1/35/warc/w2c_errs:112 | |
| 3592 x2/35/warc/w2c_errs:112 | |
| 3593 x3/35/warc/w2c_errs:81 | |
| 3594 x4/35/warc/w2c_errs:112 | |
| 3595 | |
| 3596 Maybe we have enough info to fix this: | |
| 3597 >: egrep -ho ' [^ ]* win,' x?/34/warc/w* | cut -f 2 -d ' ' | btot | |
| 3598 5258229 | |
| 3599 >: egrep -ho ' [^ ]* win,' x?/35/warc/w* | cut -f 2 -d ' ' | btot | |
| 3600 4971117 | |
| 3601 >: tail -14 x3/35/warc/w2c_errs | |
| 3602 /beegfs/common_crawl/CC-MAIN-2019-35/1566027323246.35/orig/warc/CC-MAIN-20190825084751-20190825110751-00417.warc.gz | |
| 3603 Traceback (most recent call last): | |
| 3604 File "/work/dc007/dc007/hst/lib/python/cc/w2c.py", line 4, in <module> | |
| 3605 sys.exit(warc2cdb.main(*sys.argv[1:])) | |
| 3606 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 3607 File "warc2cdb.py", line 105, in lmh.warc2cdb.main | |
| 3608 with open((outfile_name:="%s/%s/%s/lmh.cdb_in"%(outdir, segment, subdir)),'wb') as OUT: | |
| 3609 File "warc2cdb.py", line 110, in lmh.warc2cdb.main | |
| 3610 warc.warc(infile_name,LMHline,[warc.RESP, warc.REVISIT],parts=3) | |
| 3611 File "warc.py", line 125, in warc.warc | |
| 3612 _out=callback(wtype,bufView[start_2:eo2+2],2) | |
| 3613 File "warc2cdb.py", line 81, in lmh.warc2cdb.LMHline | |
| 3614 l = len(lmi) | |
| 3615 TypeError: object of type 'int' has no len() | |
| 3616 Excellent -- we should have all of 34, (almost) none of 35 | |
| 3617 >: head -1 x3/35/warc/w2c_errs | |
| 3618 /beegfs/common_crawl/CC-MAIN-2019-35/1566027323246.35/orig/warc/CC-MAIN-20190825084751-20190825110751-00336.warc.gz | |
| 3619 >: echo $(((417 - 336) +1)) | |
| 3620 82 | |
| 3621 >: echo $((5258229 / 560)) | |
| 3622 9389 | |
| 3623 >: python3 -c "print(5258229 * ((82.0 + (4 * 112)) / 560))" | |
| 3624 4976538.16 | |
| 3625 Close enough... | |
| 3626 Probably not worth it unless we hit a big set of partial results | |
| 3627 | |
| 3628 More deciles: | |
| 3629 >: for ty in warc; do sbatch --output=slurm_40-49x5${ty::1}.out --time=02:30:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh | |
| 3630 j=$((i - 1)) ; tp=$((task + 40)) ; ty='$ty' | |
| 3631 mkdir -p ./x$j/$tp/$ty ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg" 2> ./x$j/$tp/$ty/w2c_errs'; done | |
| 3632 >: for ty in robotstxt crawldiagnostics; do sbatch --output=slurm_40-49x5${ty::1}.out --time=00:15:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh | |
| 3633 j=$((i - 1)) ; tp=$((task + 40)) ; ty='$ty' | |
| 3634 mkdir -p ./x$j/$tp/$ty ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg" 2> ./x$j/$tp/$ty/w2c_errs'; done | |
| 3635 >: wprog.sh '4?' 'Tue Apr 29 09:43:06 BST 2025' robotstxt | |
| 3636 112.00 3:00:11 ... | |
| 3637 >: wprog.sh '4?' 'Tue Apr 29 09:43:06 BST 2025' crawldiagnostics | |
| 3638 112.00 3:00:20 ... | |
| 3639 >: headtail -1 slurm_40-49x5w.out | |
| 3640 Tue Apr 29 09:43:06 BST 2025 ... | |
| 3641 === | |
| 3642 Tue Apr 29 10:34:50 BST 2025 Finished | |
| 3643 >: wprog.sh '4?' 'Tue Apr 29 09:43:06 BST 2025' 'Tue Apr 29 10:34:50 AM' | |
| 3644 112.00 0:51:44 129.90 | |
| 3645 Back to more-or-less full speed... | |
| 3646 | |
| 3647 >: for ty in warc; do sbatch --output=slurm_50-59x5${ty::1}.out --time=02:00:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh | |
| 3648 | |
| 3649 j=$((i - 1)) ; tp=$((task + 50)) ; ty='$ty' | |
| 3650 mkdir -p ./x$j/$tp/$ty ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg" 2> ./x$j/$tp/$ty/w2c_errs'; done | |
| 3651 >: wprog.sh '5?' 'Tue Apr 29 12:31:20 BST 2025' 'Tue Apr 29 01:21:39 PM' | |
| 3652 112.00 0:50:19 133.55 | |
| 3653 >: for ty in robotstxt crawldiagnostics; do sbatch --output=slurm_50-59x5${ty::1}.out --time=00:15:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh | |
| 3654 j=$((i - 1)) ; tp=$((task + 50)) ; ty='$ty' | |
| 3655 mkdir -p ./x$j/$tp/$ty ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg" 2> ./x$j/$tp/$ty/w2c_errs'; done | |
| 3656 >: wprog.sh '5?' 'Tue Apr 29 12:31:56 BST 2025' 'Tue Apr 29 12:36:40 PM' crawldiagnostics | |
| 3657 112.00 0:04:44 1419.72 | |
| 3658 >: wprog.sh '5?' 'Tue Apr 29 12:31:54 BST 2025' 'Tue Apr 29 12:33:21 PM' robotstxt | |
| 3659 112.00 0:01:27 4634.48 | |
| 3660 | |
| 3661 >: for ty in warc; do sbatch --output=slurm_60-69x5${ty::1}.out --time=02:00:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh | |
| 3662 | |
| 3663 j=$((i - 1)) ; tp=$((task + 60)) ; ty='$ty' | |
| 3664 mkdir -p ./x$j/$tp/$ty ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg" 2> ./x$j/$tp/$ty/w2c_errs'; done | |
| 3665 | |
| 3666 >: for ty in robotstxt crawldiagnostics; do sbatch --output=slurm_60-69x5${ty::1}.out --time=00:15:00 --ntasks=10 -c 15 --exclusive $HOME/bin/runme.sh $PWD -d -t 3 -i 'cat ranges' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh | |
| 3667 j=$((i - 1)) ; tp=$((task + 60)) ; ty='$ty' | |
| 3668 mkdir -p ./x$j/$tp/$ty | |
| 3669 ~/lib/python/cc/w2c.py 2019-35 $tp ./x$j $ty "$arg" 2> ./x$j/$tp/$ty/w2c_errs'; done | |
| 3670 | |
| 3671 Continuing w/o logging to do the rest of the segments, checking as | |
| 3672 above for consistency with existing results... | |
| 3673 Done. | |
| 3674 | |
| 3675 Just rerun x3/35/ 336,447 Done and checked | |
| 3676 | |
| 3677 Next steps: | |
| 3678 Fetch c_d and, if needed r_t for 2023-40 - DONE | |
| 3679 Run a by-subparts batch of warc2cdb for warc, r_t and c_d for 2019-35: DONE | |
| 3680 Build N lmh.cdb for 2019-35, _including_ r_t and c_d | |
| 3681 Might there be key collisions? Try a test... | |
| 3682 | |
| 3683 Next steps, cont'd: | |
| 3684 Remove all w2c_errs and ldb.cdb_in from *** downwards of 2023-40 | |
| 3685 Do whatever checks are possible on the results | |
| 3686 Fill in r_t? and c_d ldb.cdb_in for 0 1 2 4 11 12 15 | |
| 3687 >: egrep -o '^[^-]*->' x1/27/warc/lmh.cdb_in |sus | head | |
| 3688 1 +100,10:20190822110255http://legacy.abcquant.com/index.php?option=com_content&view=article&id=158&Itemid=208-> | |
| 3689 ... | |
| 3690 >: egrep -ho '^[^-]*->' x1/27/*/lmh.cdb_in |sus | head | |
| 3691 1 +100,10:20190822110255http://legacy.abcquant.com/index.php?option=com_content&view=article&id=158&Itemid=208-> | |
| 3692 ... | |
| 3693 >: egrep -ho '^[^-]*->' x?/27/*/lmh.cdb_in |sus | head | |
| 3694 1 +100,10:20190822110255http://legacy.abcquant.com/index.php?option=com_content&view=article&id=158&Itemid=208-> | |
| 3695 ... | |
| 3696 |