Mercurial > hg > cc > work
annotate lurid3/notes.txt @ 81:b589ae992146
x
| author | Henry S. Thompson <ht@inf.ed.ac.uk> |
|---|---|
| date | Mon, 31 Mar 2025 12:41:58 +0100 |
| parents | 02fb801ac3c1 |
| children | 9fdeb9b3e58c |
| rev | line source |
|---|---|
| 40 | 1 See old_notes.txt for all older notes on Common Crawl dataprocessing, |
| 2 starting from Azure via Turing and then LURID and LURID2. | |
| 3 | |
| 4 Installed /beegfs/common_crawl/CC-MAIN-2024-33/cdx | |
| 5 >: cd results/CC-MAIN-2024-33/cdx/ | |
| 6 >: cut -f 2 counts.tsv | btot | |
| 7 2,793,986,828 | |
| 8 | |
| 9 State of play wrt data -- see status.xlsx | |
| 10 | |
|
41
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
11 [in trying to tabulate the date ranges of the crawls, I found that the |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
12 WARC timestamp is sometimes bogus: |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
13 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
14 >: fgrep ' 2009' CC-MAIN-2018-34/cdx/cluster.idx |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
15 net,tyredeyes)/robots.txt 20090201191318 cdx-00230.gz 160573468 198277 920675 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
16 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
17 >: zgrep '^net,tyredeyes)/robots.txt' CC-MAIN-2018-34/cdx/warc/cdx-00230.gz |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
18 net,tyredeyes)/robots.txt 20090201191318 {"url": "http://tyredeyes.net/robots.txt", "mime": "text/html", "mime-detected": "text/html", "status": "301", "digest": "QH732FYSV7UM34JYWVYMB7EZGR2CYM6B", "length": "582", "offset": "1224614", "filename": "crawl-data/CC-MAIN-2018-34/segments/1534221215075.58/robotstxt/CC-MAIN-20180819090604-20180819110604-00558.warc.gz"} |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
19 net,tyredeyes)/robots.txt 20090201191319 {"url": "http://www.tyredeyes.net/robots.txt", "mime": "text/plain", "mime-detected": "text/plain", "status": "200", "digest": "PSX5IZU4B4SIXGNDKXCVFH75Q27VHUTJ", "length": "549", "offset": "2069841", "filename": "crawl-data/CC-MAIN-2018-34/segments/1534221215075.58/robotstxt/CC-MAIN-20180819090604-20180819110604-00485.warc.gz"} |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
20 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
21 This happens in 2019-35 as well :-( |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
22 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
23 >: fgrep ' 20181023' CC-MAIN-2019-35/cdx/cluster.idx |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
24 com,gyshbsh)/robots.txt 20181023022000 cdx-00078.gz 356340085 162332 315406 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
25 >: zgrep ' 20181023' CC-MAIN-2019-35/cdx/warc/cdx-00078.gz |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
26 com,gyshbsh)/robots.txt 20181023022000 {"url": "http://gyshbsh.com/robots.txt", "mime": "text/html", "mime-detected": "text/html", "status": "301", "digest": "3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ", "length": "529", "offset": "638892", "filename": "crawl-data/CC-MAIN-2019-35/segments/1566027315618.73/robotstxt/CC-MAIN-20190820200701-20190820222701-00120.warc.gz"} |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
27 ... |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
28 |
| 57 | 29 Full search: |
| 30 >: find CC*/cdx -type f -name cluster.idx > /tmp/hst/clus | |
| 31 >: cat /tmp/hst/clus | while read c; do printf '%s\t%s\n' $c $(cut -f 1 -d ' ' $c | fgrep -vc ${c:8:4}); done | |
| 32 CC-MAIN-2013-20/cdx/cluster.idx 0 | |
| 33 CC-MAIN-2014-35/cdx/cluster.idx 0 | |
| 34 CC-MAIN-2015-35/cdx/cluster.idx 0 | |
| 35 CC-MAIN-2016-30/cdx/cluster.idx 0 | |
| 36 CC-MAIN-2017-30/cdx/cluster.idx 0 | |
| 37 CC-MAIN-2018-30/cdx/warc/cluster.idx 0 | |
| 38 CC-MAIN-2018-34/cdx/cluster.idx 36 | |
| 39 CC-MAIN-2019-18/cdx/warc/cluster.idx 3 | |
| 40 CC-MAIN-2019-35/cdx/cluster.idx 1 | |
| 41 CC-MAIN-2020-34/cdx/cluster.idx 0 | |
| 42 CC-MAIN-2021-25/cdx/cluster.idx 0 | |
| 43 CC-MAIN-2021-31/cdx/cluster.idx 0 | |
| 44 CC-MAIN-2021-49/cdx/cluster.idx 0 | |
| 45 CC-MAIN-2022-21/cdx/warc/cluster.idx 0 | |
| 46 CC-MAIN-2022-33/cdx/warc/cluster.idx 0 | |
| 47 CC-MAIN-2022-40/cdx/warc/cluster.idx 0 | |
| 48 CC-MAIN-2022-49/cdx/warc/cluster.idx 0 | |
| 49 CC-MAIN-2023-40/cdx/warc/cluster.idx 0 | |
| 50 CC-MAIN-2023-50/cdx/warc/cluster.idx 0 | |
| 51 CC-MAIN-2024-33/cdx/warc/cluster.idx 0 | |
| 52 Emailed this info to Sebastian Nagel 2024-12-17 | |
| 53 | |
|
41
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
54 Tabulate all the date ranges for the WARC files we have |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
55 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
56 >: for d in {2017-30,2019-35,2020-34,2021-25,2023-40,2023-50}; do printf "%s\t" $d; (ls CC-MAIN-$d/*.{?,??}/orig/warc | fgrep .gz | cut -f 3,4 -d - | sort -u |tee /dev/fd/3 | head -1 ) 3> >( tail -1 ) | tr '\n' '\t'; echo; done | cut -f 1,2,4 -d - | sed 's/-20/ 20/;s/.$//' | tr ' ' '\t' > dates.tsv |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
57 >: for d in {2018-30,2018-34}; do printf "%s\t" $d; (ls CC-MAIN-$d/{*.?,*.??} | fgrep warc.gz | cut -f 3,4 -d - | sort -u |tee /dev/fd/3 | { sleep 10 ; head -1 ; } ) 3> >( tail -1 ) | tr '\n' '\t'; echo; done >> dates.tsv |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
58 >: for d in 2019-18; do printf "%s\t" $d; (ls CC-MAIN-$d/{*.?,*.??} | fgrep warc.gz | cut -f 3,4 -d - | sort -u | head -1); done |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
59 2019-18 20190418101243-20190418122248 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
60 >: for d in 2019-18; do printf "%s\t" $d; (ls CC-MAIN-$d/{*.?,*.??} | fgrep warc.gz | cut -f 3,4 -d - | sort -u | tail -1); done |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
61 2019-18 20190426153423-20190426175423 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
62 >: echo 2019-18 20190418101243-20190418122248 20190426153423-20190426175423 >> dates.tsv |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
63 >: pwd |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
64 /beegfs/common_crawl/CC-MAIN-2016-30/cdx/warc |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
65 >: echo {000..299} | tr ' ' '\n' | parallel -j 10 'uz cdx-00{}.gz | cut -f 2 -d " " | sort -u > /tmp/hst/{}' |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
66 >: sort -mu /tmp/hst/??? > /tmp/hst/all |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
67 >: wc -l /tmp/hst/all |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
68 679686 /tmp/hst/all |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
69 >: head -1 /tmp/hst/all |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
70 20160723090435 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
71 >: tail -1 /tmp/hst/all |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
72 20160731110639 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
73 >: cd ../../.. |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
74 >: echo 2016-30 20160723090435 20160731110639 >> dates.tsv |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
75 tweaked and sorted in xemacs: |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
76 2016-30 20160723090435 20160731110639 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
77 2017-30 20170720121902 20170729132938 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
78 2018-30 20180715183800 20180723184955 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
79 2018-34 20180814062251 20180822085454 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
80 2019-18 20190418101243 20190426175423 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
81 2019-35 20190817102624 20190826111356 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
82 2020-34 20200803083123 20200815214756 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
83 2021-25 20210612103920 20210625145905 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
84 2023-40 20230921073711 20231005042006 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
85 2023-50 20231128083443 20231212000408 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
86 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
87 Added to status.xlsx in shortened form, with number of days |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
88 8 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
89 9 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
90 8 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
91 8 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
92 8 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
93 9 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
94 12 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
95 13 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
96 15 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
97 15 |
|
64b7fb44e8dc
extract actual date info for WARC crawls
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
40
diff
changeset
|
98 |
|
42
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
99 Fill a gap by downloading 2022-33 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
100 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
101 >: for s in 0; do ~/bin/getcc_multi.aws CC-MAIN-2022-33 $s 5; done > /tmp/hst/get_22-33_0.log & |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
102 130 minutes... |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
103 >: for s in 1; do ~/bin/getcc_multi.aws CC-MAIN-2022-33 $s 10; done > /tmp/hst/get_22-33_1.log & |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
104 59 minutes |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
105 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
106 Another day to get to a quarter? |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
107 >: for s in {2..23}; do ~/bin/getcc_multi.aws CC-MAIN-2022-33 $s 10; done > /tmp/hst/get_22-33_2-23.log & |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
108 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
109 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
110 And finally 2015-35 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
111 Fetched in just 2 chunks, 0-9 and 10-99, e.g. |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
112 >: for s in {10..99}; do ~/bin/getcc_multi.aws CC-MAIN-2015-35 $s 10; done > /tmp/hst/get_15-35_10-99.log & |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
113 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
114 Much smaller. |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
115 Compare 2023-40, with 900 files per segment: |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
116 >: lss */orig/warc/*-0023?.* | cut -f 5 -d ' ' | stats |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
117 n = 1000 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
118 min = 1.14775e+09 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
119 max = 1.26702e+09 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
120 sum = 1.20192e+12 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
121 mean = 1.20192e+09 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
122 sd = 2.26049e+07 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
123 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
124 with 2015-35, with 353 files per segment |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
125 >: lss */orig/warc/*-0023?-* | cut -f 5 -d ' ' | stats |
| 43 | 126 n = 1000 |
| 127 min = 1.66471e+08 | |
|
42
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
128 max = 9.6322e+08 |
| 43 | 129 sum = 9.19222e+11 |
| 130 mean = 9.19222e+08 | |
| 131 sd = 8.20542e+07 | |
|
42
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
132 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
133 The min files all come from segment 1440644060633.7, whose files are |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
134 _all_ small: |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
135 >: uz *00123-*.gz | wc -l |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
136 12,759,931 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
137 Compare to 1440644060103.8 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
138 >: zcat *00123-*.gz | wc -l |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
139 75,806,738 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
140 Mystery |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
141 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
142 Also faster |
| 43 | 143 Compare 2022-33: |
|
42
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
144 >: fgrep -h BST /tmp/hst/get_22-33_{2-23,24-49,50-99}.log | cut -f 1-7 -d ' ' | while read s; do if read e; then echo $((($(date --date="$e" +%s) - $(date --date="$s" +%s)) / 60)); fi; done | stats n min max mean sd |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
145 98 19 256 75.1 25.2 |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
146 with 2015-35: |
|
0c472ae05f71
nearly finished downloading for now
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
41
diff
changeset
|
147 >: fgrep -h BST /tmp/hst/get_15-35_{0-9,10-99}.log | cut -f 1-7 -d ' ' | while read s; do if read e; then echo $((($(date --date="$e" +%s) - $(date --date="$s" +%s)) / 60)); fi; done | stats n min max mean sd |
| 43 | 148 100 15 40 32.6 2.9 |
| 149 | |
| 150 >: echo {000..299} | tr ' ' '\n' | parallel -j 10 'uz cdx-00{}.gz | cut -f 2 -d " " | sort -u > /tmp/hst/2015_{}' & | |
| 151 >: sort --parallel=10 -mu /tmp/hst/2015_??? > /tmp/hst/2015_all | |
| 152 >: head -1 /tmp/hst/2015_all | |
| 153 20150827191534 | |
| 154 >: tail -1 /tmp/hst/2015_all | |
| 155 20150905180914 | |
| 156 >: wc -l /tmp/hst/2015_all | |
| 157 698128 /tmp/hst/2015_all | |
| 158 | |
| 159 What about wet files -- do they include text from pdfs? What about | |
| 160 truncated pdfs? | |
| 161 | |
| 162 >: time for s in 0; do ~/bin/getcc_wet_multi.aws CC-MAIN-2019-35 $s 10; done > /tmp/hst/get_wet_19-35_0.log & | |
| 163 real 26m3.049s | |
| 164 user 0m1.225s | |
| 165 sys 0m1.310s | |
| 166 | |
| 167 In the segment 0 cdx file (!) we find 3747 probable truncations: | |
| 168 >: zgrep -a '"mime-detected": "application/pdf", ' cdx.gz > /tmp/hst/2019-35_seg0_pdf.idx | |
| 169 >: wc -l /tmp/hst/2019-35_seg0_pdf.idx | |
| 170 42345 /tmp/hst/2019-35_seg0_pdf.idx | |
| 171 >: egrep -a '"length": "10....."' /tmp/hst/2019-35_seg0_pdf.idx > /tmp/hst/2019-35_seg0_long_pdf.idx & | |
| 172 >: wc -l < /tmp/hst/2019-35_seg0_long_pdf.idx | |
| 173 3747 | |
| 174 Of which 70 are in file 0: | |
| 175 >: egrep -a '.-00000\.' /tmp/hst/2019-35_seg0_pdf.idx > /tmp/hst/2019-35_seg0_file0_pdf.idx | |
| 176 >: wc -l /tmp/hst/2019-35_seg0_file0_pdf.idx | |
| 177 70 /tmp/hst/2019-35_seg0_file0_pdf.idx | |
| 178 | |
| 179 In segment 0 file 0 we find 70 application/pdf Content-Type headers: | |
| 180 >: ix.py -h -w -x </tmp/hst/2019-35_seg0_file0_pdf.idx |egrep '^(WARC-Target-URI:|Content-Length:) '|cut -f 2 -d ' ' |tr -d '\r'|while read l1; do read uri; read l2; printf '%s\t%s\t%s\n' $l1 $l2 "$uri"; done > ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | |
| 181 >: wc -l < ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | |
| 182 70 | |
| 183 >: head -3 ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | |
| 184 | |
| 185 | |
| 186 Of which 14 are truncated: | |
| 187 >: fgrep -c 1048576 ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | |
| 188 14 | |
| 189 | |
| 190 E.g. | |
| 191 >: fgrep 1048576 ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | head -3 | |
| 192 1049051 1048576 https://en.heks.ch/sites/default/files/documents/2017-09/HEKS_EPER_Mission_Statement_2016_e.pdf | |
| 193 1049469 1048576 https://bmcmicrobiol.biomedcentral.com/track/pdf/10.1186/s12866-017-0951-4 | |
| 194 1048824 1048576 https://citydocs.fcgov.com/?action=cao-cases&cmd=convert&docid=3332339 | |
| 195 | |
| 196 Are any of the pdfs in the corresponding wet file? | |
| 197 | |
| 198 Yes, 2: | |
| 199 >: cut -f 3 ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | fgrep -af - <(uz 1566027313501.0/orig/wet/*-00000.warc.wet.gz) | |
| 200 WARC-Target-URI: http://bdds.deux-sevres.com/recherche/simple/Editeur/2/Belfond/vignette?format=pdf | |
| 201 WARC-Target-URI: https://museum.wrap.gov.tw/GetFile4.ashx?Serial=201609200919D00 | |
| 202 | |
| 203 Is it in fact corresponding? | |
| 204 >: diff -bw <(uz 1566027313501.0/orig/warc/*-00000.warc.gz | egrep -a '^WARC-Target-URI: ' | uniq | head -1000) <(uz 1566027313501.0/orig/wet/*-00000.warc.wet.gz | egrep -a '^WARC-Target-URI: ' | head -1000)|egrep -c '^<' | |
| 205 19 | |
| 206 | |
| 207 So, yes, mostly. .2% are missing | |
| 208 | |
| 209 Just checking the search: | |
| 210 >: cut -f 3 ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | fgrep -af - <(uz 1566027313501.0/orig/warc/*-00000.warc.gz) | wc -l | |
| 211 210 | |
| 212 Correct | |
| 213 | |
| 214 So, what pdfs make it into the WET: | |
| 215 >: cut -f 3 ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | fgrep -af - <(uz 1566027313501.0/orig/wet/*-00000.warc.wet.gz) > ~/results/CC-MAIN-2019-35/s0_file0_pdf.txt | |
| 216 >: wc -l < ~/results/CC-MAIN-2019-35/s0_file0_pdf.txt | |
| 217 2 | |
| 218 >: cut -f 2 -d ' ' ~/results/CC-MAIN-2019-35/s0_file0_pdf.txt | tr -d '\r' | fgrep -f - ~/results/CC-MAIN-2019-35/seg0_file0_lengths.tsv | |
| 219 11588 10913 http://bdds.deux-sevres.com/recherche/simple/Editeur/2/Belfond/vignette?format=pdf | |
| 220 1048979 1048576 https://museum.wrap.gov.tw/GetFile4.ashx?Serial=201609200919D005 | |
| 221 | |
| 222 Here's the short one: | |
| 223 WARC/1.0 | |
| 224 WARC-Type: response | |
| 225 WARC-Date: 2019-08-17T22:40:17Z | |
| 226 WARC-Record-ID: <urn:uuid:ea98167b-c42a-4233-b57e-994aa627e38a> | |
| 227 Content-Length: 11588 | |
| 228 Content-Type: application/http; msgtype=response | |
| 229 WARC-Warcinfo-ID: <urn:uuid:f689f8d0-24f3-4824-9a38-4f3fee422a4e> | |
| 230 WARC-Concurrent-To: <urn:uuid:2d51c956-0012-4d78-affc-8f57fe9d2e15> | |
| 231 WARC-IP-Address: 92.175.114.24 | |
| 232 WARC-Target-URI: http://bdds.deux-sevres.com/recherche/simple/Editeur/2/Belfond/vignette?format=pdf | |
| 233 WARC-Payload-Digest: sha1:7VVIUDQ4Q6XKNOAURYU4VTMRSZNPHDQA | |
| 234 WARC-Block-Digest: sha1:OSTWXLV772XNHS22T4UBSCSJAAXM2J6T | |
| 235 WARC-Identified-Payload-Type: application/pdf | |
| 236 | |
| 237 HTTP/1.1 200 OK | |
| 238 Cache-Control: must-revalidate, post-check=0, pre-check=0,no-cache | |
| 239 Pragma: public,no-cache | |
| 240 Content-Type: application/pdf",text/html; charset=utf-8 | |
| 241 X-Crawler-Content-Encoding: gzip | |
| 242 Expires: 0 | |
| 243 Server: | |
| 244 X-Powered-By: | |
| 245 Set-Cookie: 166d74d734106ba68b20ea303011f622=301619e3fe31ecb98c8473f0ff5f35a2; path=/ | |
| 246 Content-Disposition: attachment; filename="Mdiathque dpartementale des Deux-Svres - Rsultats de la recherche Belfond.pdf" | |
| 247 Content-Transfer-Encoding: binary | |
| 248 P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM" | |
| 249 X-Content-Encoded-By: | |
| 250 X-Powered-By: | |
| 251 Date: Sat, 17 Aug 2019 22:40:16 GMT | |
| 252 X-Crawler-Content-Length: 5448 | |
| 253 Content-Length: 10913 | |
| 254 | |
| 255 %PDF-1.7 | |
| 256 %<E2><E3><CF><D3> | |
| 257 7 0 obj | |
| 258 << /Type /Page /Parent 1 0 R /LastModified (D:20190818004016+02'00') /Resources 2 | |
| 259 0 R /MediaBox [0.000000 0.000000 595.276000 841.890000] /CropBox [0.000000 0.000 | |
| 260 000 595.276000 841.890000] /BleedBox [0.000000 0.000000 595.276000 841.890000] /T | |
| 261 rimBox [0.000000 0.000000 595.276000 841.890000] /ArtBox [0.000000 0.000000 595.2 | |
| 262 76000 841.890000] /Contents 8 0 R /Rotate 0 /Group << /Type /Group /S /Transparen | |
| 263 cy /CS /DeviceRGB >> /PZ 1 >> | |
| 264 endobj | |
| 265 8 0 obj | |
| 266 | |
| 267 >: uz 1566027313501.0/orig/warc/*-00000.warc.gz|tail -n +1823434 | tail -n +24 | head -c 20000 > ~/results/CC-MAIN-2019-35/mediatheque.pdf | |
| 268 >: ps2ascii mediatheque.pdf | |
| 269 Médiathèque départementale des Deux-Sèvres - Résultats de la recherche Belfond | |
| 270 | |
| 271 Médiathèque départementale des Deux-Sèvres - Résultats de | |
| 272 la recherche Belfond | |
| 273 A charge de revanche | |
| 274 Titre : | |
| 275 Auteur : Grippando, James (1958-....) | |
| 276 ... | |
| 277 etc., three pages, no errors | |
| 278 | |
| 279 >: uz 1566027313501.0/orig/warc/*-00000.warc.gz|fgrep -an https://museum.wrap.gov.tw/GetFile4.ashx | |
| 280 38896837:WARC-Target-URI: https://museum.wrap.gov.tw/GetFile4.ashx?Serial=201609200919D005 | |
| 281 38896858:WARC-Target-URI: https://museum.wrap.gov.tw/GetFile4.ashx?Serial=201609200919D005 | |
| 282 38904590:WARC-Target-URI: https://museum.wrap.gov.tw/GetFile4.ashx?Serial=201609200919D005 | |
| 283 >: uz 1566027313501.0/orig/warc/*-00000.warc.gz|tail -n +38896858 | egrep -an '^%%EOF' | |
| 284 27:%%EOF | |
| 285 1114658:%%EOF | |
| 286 1313299:%%EOF | |
| 287 | |
| 288 Hunh? | |
| 289 | |
| 290 >: uz 1566027313501.0/orig/warc/*-00000.warc.gz|tail -n +38896858 | egrep -an '^(%%EOF|WARC)' | head -30 | |
| 291 1:WARC-Target-URI: https://museum.wrap.gov.tw/GetFile4.ashx?Serial=201609200919D005 | |
| 292 2:WARC-Payload-Digest: sha1:SZ53DQQHENC7DDN7GQ5IS7VMEPAXAMBE | |
| 293 3:WARC-Block-Digest: sha1:QTKJA6A7445Z7264K2YAFBUUM2OYH2T2 | |
| 294 4:WARC-Truncated: length | |
| 295 5:WARC-Identified-Payload-Type: application/pdf | |
| 296 27:%%EOF | |
| 297 7725:WARC/1.0 | |
| 298 7726:WARC-Type: metadata | |
| 299 7727:WARC-Date: 2019-08-17T22:59:14Z | |
| 300 7728:WARC-Record-ID: <urn:uuid:77df2747-e567-45d3-8646-3069ae9a9f25> | |
| 301 7731:WARC-Warcinfo-ID: <urn:uuid:f689f8d0-24f3-4824-9a38-4f3fee422a4e> | |
| 302 7732:WARC-Concurrent-To: <urn:uuid:eceb4adc-d81e-4497-82fe-eea61ce171f4> | |
| 303 7733:WARC-Target-URI: https://museum.wrap.gov.tw/GetFile4.ashx?Serial=201609200919D005 | |
| 304 7739:WARC/1.0 | |
| 305 | |
| 306 OK, so indeed truncated after 7700 lines or so... | |
| 307 >: uz 1566027313501.0/orig/warc/*-00000.warc.gz|tail -n +38896858 | tail -n +21 | head -c 1048576 > ~/results/CC-MAIN-2019-35/museum.pdf | |
| 308 >: ps2ascii ~/results/CC-MAIN-2019-35/museum.pdf | |
| 309 **** Error: An error occurred while reading an XREF table. | |
| 310 **** The file has been damaged. | |
| 311 Look in big_pdf? | |
|
44
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
312 |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
313 ====Modify the original CC indexer to write new indices including lastmod===== |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
314 Looks like WarcRecordWriter.write, in |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
315 src/nutch-cc/src/java/org/commoncrawl/util/WarcRecordWriter, is what |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
316 needs to be editted to include LastModified date |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
317 |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
318 To rebuild nutch-cc, particularly to recompile jar files after editting |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
319 anything |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
320 |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
321 >: cd $HHOME/src/nutch-cc |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
322 >: ant |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
323 |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
324 Fixed deprecation bug in WarcCdxWriter.java |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
325 |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
326 Modified src/java/org/commoncrawl/util/WarcCdxWriter.java |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
327 to include lastmod |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
328 |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
329 Can run just one test, which should allow testing this: |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
330 |
|
7209df5fa5b4
turn attention to nutch-cc and its Cdx code
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
43
diff
changeset
|
331 >: ant test-core -Dtestcase='TestWarcRecordWriter' |
| 45 | 332 |
| 333 Logic is tricky, and there's no easy way in | |
| 334 | |
| 335 Basically, tools/WarcExport.java is launches a hadoop job based on a | |
| 336 hadoop-runnable WarcExport instance. Hadoop will in due course call | |
| 337 ExportReducer.reduce, which will create an instance of WarcCapture | |
| 338 "for each page capture", and call ExportMapper.context.write with that instance (via | |
| 339 some configuration magic with the hadoop job Context). That in turn | |
| 340 uses (more magic) WarcOutputFormat.getRecordWriter, which | |
| 341 (finally!) calls a previously created WarcRecordWriter | |
| 342 instance.write(the capture). | |
| 343 | |
| 344 So to fake a test case, I need to build | |
| 345 1) a WarcRecordWriter instance | |
| 346 2) a WarcCapture instance | |
| 347 and then invoke 1.write(2) | |
| 348 | |
| 349 Got that working, although still can't figure out where in the normal | |
| 350 flow the metadata entry for Response.CONTENT_TYPE gets set. | |
| 351 | |
| 352 Now, add a test that takes a stream of WARC Response extracts and | |
| 353 rewrites their index entries | |
| 354 | |
| 355 >: head -8804 <(uz /beegfs/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00150.gz)|tail -10| ix.py -h -w -x > /tmp/hst/headers.txt | |
| 356 >: cp /tmp/hst/headers.txt src/test/org/commoncrawl/util/ | |
| 357 >: shuf /tmp/hst/headers.txt > src/test/org/commoncrawl/util/headers_mixed.txt | |
| 358 | |
| 359 Won't quite work :-( | |
| 360 How do We reconstruct the Warc filename, offset and length from the | |
| 361 original index? | |
| 362 | |
| 363 Well, we can find a .warc.gz records! | |
| 364 Thanks to https://stackoverflow.com/a/37042747/2595465 | |
| 365 | |
| 366 >: ~/lib/python/unpackz.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz > /tmp/hst/recs.txt | |
| 367 | |
| 368 Nearly working, got 1/3rd of the way through a single WARC and then failed: | |
| 369 | |
| 370 >: n=0 && ~/lib/python/unpackz.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz 2>/tmp/hst/tt.txt|while read o l; do echo $((n+=1)); echo $o $l >> /tmp/hst/r3a; ix.py $l $o CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz| wc -l; done | |
| 371 ... | |
| 372 20 | |
| 373 10215 | |
| 374 CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | |
| 375 Process fail: Compressed file ended before the end-of-stream marker was reached, input: | |
| 376 length=10762, offset=60784640, file=/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | |
| 377 | |
| 378 >: head -10217 /tmp/hst/r3a | tail -4 | |
| 379 60784173 467 | |
| 380 60784640 10762 | |
| 381 60795402 463 | |
| 382 60795865 460 | |
| 383 >: ix.py 467 60784173 CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz|fgrep Target | |
| 384 WARC-Target-URI: http://drycarerestoration.co/corner-furniture-piece/unique-corner-decoration-pieces-or-corner-furniture-pieces-corner-corner-furniture-piece-corner-furniture-pieces-bedroom/ | |
| 385 | |
| 386 >: zcat /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/cdx/warc/cdx.gz | |
| 387 ... | |
| 388 co,drycarerestoration)/corner-furniture-piece/unique-corner-decoration-pieces-or-corner-furniture-pieces-corner-corner-furniture-piece-corner-furniture-pieces-bedroom 20190819020224 {"url": "http://drycarerestoration.co/corner-furniture-piece/unique-corner-decoration-pieces-or-corner-furniture-pieces-corner-corner-furniture-piece-corner-furniture-pieces-bedroom/", "mime": "text/html", "mime-detected": "text/html", "status": "200", "digest": "DTKGJL45XQDXUS7PTXPYR6POMPLG46RZ", "length": "2570", "offset": "60784640", "filename": "crawl-data/CC-MAIN-2019-35/segments/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz", "charset": "UTF-8", "languages": "eng"} | |
| 389 >: ix.py 2570 60784640 CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz|less | |
| 390 >: echo $((10762 - 2570)) | |
| 391 8192 | |
| 392 | |
| 393 Ah, the error I was dreading :-( I _think_ this happens when an | |
| 394 individual record ends exactly on a 8K boundary. | |
| 395 | |
| 396 Yes: | |
| 397 | |
| 398 >: echo $((60784640 % 8192)) | |
| 399 0 | |
| 400 | |
| 46 | 401 Even with buffer 1MB: |
| 402 21 | |
| 403 160245 | |
| 404 CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | |
| 405 Process fail: Compressed file ended before the end-of-stream marker was reached, input: | |
| 406 length=8415, offset=1059033915, file=/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | |
| 407 0 | |
| 408 160246 | |
| 409 | |
| 410 >: tail -60 /tmp/hst/r3b|head -20 | |
| 411 1059013061 423 | |
| 412 1059013484 7218 | |
| 413 1059020702 425 | |
| 414 1059021127 424 | |
| 415 1059021551 11471 | |
| 416 1059033022 426 | |
| 417 1059033448g 467 | |
| 418 1059033915 8415 | |
| 419 | |
| 420 Argh. This is at the _same_ point (before 51 fails before EOF). Ah, | |
| 421 maybe that's the point -- this is the last read before EOF, and it's | |
| 422 not a full buffer! | |
| 423 | |
| 424 >: ix.py 467 1059033448 CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz|less | |
| 425 ... | |
| 426 WARC-Target-URI: https://zowiecarrpsychicmedium.com/tag/oracle/ | |
| 427 | |
| 428 Reran with more instrumentation, took at least all day: | |
| 429 | |
| 430 >: n=0 && ~/lib/python/unpackz.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz 2> /tmp/hst/r3e_err.txt | while read o l; do | |
| 431 echo $((n+=1)); echo $o $l >> /tmp/hst/r3e_val; ix.py $l $o CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | wc -l; | |
| 432 done > /tmp/hst/r3e_log 2>&1 | |
| 433 >: wc -l /tmp/hst/r3e_err.txt | |
| 434 160296 /tmp/hst/r3e_err.txt | |
| 435 >: tail -60 /tmp/hst/r3e_err.txt|cat -n | grep -C2 True\ True | |
| 436 7 b 28738 28738 28312 426 False False | |
| 437 8 b 28312 28312 27845 467 False False | |
| 438 9 b 27845 378162 369747 8415 True True < this is the first hit the last | |
| 439 (partial) block | |
| 440 10 b 369747 369747 369312 435 False True | |
| 441 11 b 369312 369312 368878 434 False True | |
| 442 | |
| 443 >: tail -55 /tmp/hst/r3e_val | head -3 | |
| 444 1059033022 426 | |
| 445 1059033448 467 | |
| 446 1059033915 8415 | |
| 447 >: dd ibs=1 skip=1059033022 count=426 if=/beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz of=/dev/stdout | uz -t | |
| 448 ... | |
| 449 426 bytes copied, 0.00468243 s, 91.0 kB/s | |
| 450 sing<3411>: dd ibs=1 skip=1059033448 count=467 if=/beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz of=/dev/stdout | uz -t | |
| 451 ... | |
| 452 467 bytes copied, 0.00382692 s, 122 kB/s | |
| 453 sing<3412>: dd ibs=1 skip=1059033915 count=8415 if=/beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz of=/dev/stdout | uz -t | |
| 454 igzip: Error (null) does not contain a complete gzip file | |
| 455 ... | |
| 456 8415 bytes (8.4 kB, 8.2 KiB) copied, 0.00968889 s, 869 kB/s | |
| 457 | |
| 458 So, tried one change to use the actually size rather than BUFSIZE at | |
| 459 one point, seems to work now: | |
| 460 | |
| 461 >: time ~/lib/python/unpackz.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz 2> /tmp/hst/r3f_err.txt | tee /tmp/hst/r3f_val | while read l o; do printf '%s\t%s\t%s\n' $l $o 'CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz'; | |
| 462 done 2>&1 | tee /tmp/hst/r3f_log | ix.py -w | egrep -c '^WARC/1\.0' | |
| 463 160296 | |
| 464 real 3m48.393s | |
| 465 user 0m47.997s | |
| 466 sys 0m26.641s | |
| 467 | |
| 468 >: tail /tmp/hst/r3f_val | |
| 469 10851 1059370472 | |
| 470 475 1059381323 | |
| 471 444 1059381798 | |
| 472 22437 1059382242 | |
| 473 447 1059404679 | |
| 474 506 1059405126 | |
| 475 15183 1059405632 | |
| 476 471 1059420815 | |
| 477 457 1059421286 | |
| 478 17754 1059421743 | |
| 479 | |
| 480 >: wc -l /tmp/hst/*_val | |
| 481 171 /tmp/hst/r3d_val | |
| 482 160297 /tmp/hst/r3e_val | |
| 483 160296 /tmp/hst/r3f_val | |
| 484 320764 total | |
| 485 >: uz /tmp/hst/head.warc.gz |egrep -c '^WARC/1\.0.$' | |
| 486 171 | |
| 487 >: tail -n 3 /tmp/hst/*_val | |
| 488 ==> /tmp/hst/r3d_val <== | |
| 489 454 1351795 | |
| 490 414 1352249 | |
| 491 0 1352663 [so the 171 above is bogus, and we're missing one] | |
| 492 | |
| 493 ==> /tmp/hst/r3e_val <== | |
| 494 1059393441 457 | |
| 495 1059393898 17754 | |
| 496 0 [likewise bogus, so see below] | |
| 497 | |
| 498 ==> /tmp/hst/r3f_val <== | |
| 499 471 1059420815 | |
| 500 457 1059421286 | |
| 501 17754 1059421743 [better, but still one missing] | |
| 502 >: uz /tmp/hst/head.warc.gz |egrep '^WARC-Type: ' | tee >(wc -l 1>&2) | tail -4 | |
| 503 WARC-Type: response | |
| 504 WARC-Type: metadata | |
| 505 WARC-Type: request | |
| 506 WARC-Type: response [missing] | |
| 507 171 | |
| 508 >: ls -lt /tmp/hst/*_val | |
| 509 -rw-r--r-- 1 hst dc007 1977 Sep 29 09:27 /tmp/hst/r3d_val | |
| 510 -rw-r--r-- 1 hst dc007 2319237 Sep 28 14:28 /tmp/hst/r3f_val | |
| 511 -rw-r--r-- 1 hst dc007 2319238 Sep 27 19:41 /tmp/hst/r3e_val | |
| 512 >: ls -l ~/lib/python/unpackz.py | |
| 513 -rwxr-xr-x 1 hst dc007 1821 Sep 28 15:13 .../dc007/hst/lib/python/unpackz.py | |
| 514 So e and f are stale, rerun | |
| 515 >: time ~/lib/python/unpackz.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz 2>/tmp/hst/r3f_err.txt| tee /tmp/hst/r3f_val|while read l o; do printf '%s\t%s\t%s\n' $l $o 'CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz' ;done |& tee /tmp/hst/r3f_log |ix.py -w |egrep '^WARC-Type: ' | tail -4 & | |
| 516 >: Reading length, offset, filename tab-delimited triples from stdin... | |
| 517 WARC-Type: response | |
| 518 WARC-Type: metadata | |
| 519 WARC-Type: request | |
| 520 WARC-Type: response | |
| 521 | |
| 522 real 3m49.760s | |
| 523 user 0m47.180s | |
| 524 sys 0m32.218s | |
| 525 So missing the final metadata... | |
| 526 Back to head.warc.gz, with debug info | |
| 527 | |
| 528 >: n=0 && ~/lib/python/unpackz.py /tmp/hst/head.warc.gz 2>/tmp/hst/ttd.txt|while read l o; do echo $((n+=1)); echo $l $o >> /tmp/hst/r3d_val; dd ibs=1 skip=$o count=$l if=/tmp/hst/head.warc.gz of=/dev/stdout 2>/tmp/hst/r3d_ido| uz -t ; done >/tmp/hst/r3d_log 2>&1 | |
| 529 >: tail -2 /tmp/hst/r3d_log | |
| 530 171 | |
| 531 igzip: Error invalid gzip header found for file (null) | |
| 532 >: tail -n 3 /tmp/hst/ttd.txt /tmp/hst/r3d_val | |
| 533 ==> /tmp/hst/ttd.txt <== | |
| 534 b 9697 9697 9243 454 False True | |
| 535 b 9243 9243 8829 414 False True | |
| 536 n 8829 | |
| 537 | |
| 538 ==> /tmp/hst/r3d_val <== | |
| 539 454 1351795 | |
| 540 414 1352249 | |
| 541 0 1352663 | |
| 542 | |
| 543 >: cat -n /tmp/hst/r3f_val | head -172 | tail -4 | |
| 544 169 454 1351795 | |
| 545 170 414 1352249 | |
| 546 171 8829 1352663 | |
| 547 172 446 1361492 | |
| 548 | |
| 549 Fixed, maybe | |
| 550 | |
| 551 >: tail -n 3 /tmp/hst/r3d_log /tmp/hst/r3d_val | |
| 552 ==> /tmp/hst/r3d_log <== | |
| 553 169 | |
| 554 170 | |
| 555 171 | |
| 556 | |
| 557 ==> /tmp/hst/r3d_val <== | |
| 558 454 1351795 | |
| 559 414 1352249 | |
| 560 8829 1352663 | |
| 561 | |
| 562 Yes! | |
| 563 | |
| 564 >: time ~/lib/python/unpackz.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz 2>/tmp/hst/r3f_err| tee /tmp/hst/r3f_val|while read l o; do printf '%s\t%s\t%s\n' $l $o 'CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz' ;done |& tee /tmp/hst/r3f_log |ix.py -w |egrep '^WARC-Type: ' | tail -4 | |
| 565 Reading length, offset, filename tab-delimited triples from stdin... | |
| 566 WARC-Type: metadata | |
| 567 WARC-Type: request | |
| 568 WARC-Type: response | |
| 569 WARC-Type: metadata | |
| 570 | |
| 571 real 3m26.042s | |
| 572 user 0m44.167s | |
| 573 sys 0m24.716s | |
| 574 >: tail -n 3 /tmp/hst/r3f* | |
| 575 ==> /tmp/hst/r3f_err <== | |
| 576 | |
| 577 ==> /tmp/hst/r3f_val <== | |
| 578 457 1059421286 | |
| 579 17754 1059421743 | |
| 580 425 1059439497 | |
| 581 | |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
582 Doubling buffer size doesn't speed up |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
583 >: time ~/lib/python/unpackz.py -b $((2 * 1024 * 1024)) /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz 2>/tmp/hst/r3g_err| tee /tmp/hst/r3g_val|while read l o; do printf '%s\t%s\t%s\n' $l $o 'CC-MAIN-2019-35/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz' ;done |& tee /tmp/hst/r3g_log |ix.py -w |egrep '^WARC-Type: ' | tail -4 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
584 Reading length, offset, filename tab-delimited triples from stdin... |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
585 WARC-Type: metadata |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
586 WARC-Type: request |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
587 WARC-Type: response |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
588 WARC-Type: metadata |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
589 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
590 real 3m34.519s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
591 user 0m52.312s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
592 sys 0m24.875s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
593 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
594 Tried using FileIO.readinto([a fixed buffer]), but didn't immediately |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
595 work. Abandoned because I still don't understand how zlib.decompress |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
596 works at all... |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
597 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
598 Time to convert unpackz to a library which takes a callback |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
599 alternative to an output file -- Done |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
600 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
601 W/o using callback, timing and structure for what we need for |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
602 re-indexing task looks encouraging: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
603 >: time ~/lib/python/cc/unpackz.py -o /dev/stdout /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz |egrep -aA20 '^WARC-Type: response' | cut -f 1 -d ' ' | egrep -a '^WARC-' |sus | tee >(wc -l 1>&2) |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
604 52468 WARC-Block-Digest: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
605 52468 WARC-Concurrent-To: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
606 52468 WARC-Date: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
607 52468 WARC-Identified-Payload-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
608 52468 WARC-IP-Address: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
609 52468 WARC-Payload-Digest: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
610 52468 WARC-Record-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
611 52468 WARC-Target-URI: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
612 52468 WARC-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
613 52468 WARC-Warcinfo-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
614 236 WARC-Truncated: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
615 11 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
616 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
617 real 0m20.308s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
618 user 0m19.720s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
619 sys 0m4.505s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
620 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
621 Whole thing, with no pre-filtering: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
622 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
623 >: time ~/lib/python/cc/unpackz.py -o /dev/stdout /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | cut -f 1 -d ' ' | egrep -a '^(WARC-|Content-|Last-Modified)' |sus | tee >(wc -l 1>&2) |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
624 211794 Content-Length: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
625 211162 Content-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
626 159323 WARC-Target-URI: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
627 159311 WARC-Warcinfo-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
628 159301 WARC-Record-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
629 159299 WARC-Date: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
630 159297 WARC-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
631 105901 WARC-Concurrent-To: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
632 105896 WARC-IP-Address: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
633 52484 WARC-Block-Digest: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
634 52484 WARC-Identified-Payload-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
635 52482 WARC-Payload-Digest: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
636 9239 Last-Modified: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
637 3941 Content-Language: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
638 2262 Content-Security-Policy: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
639 642 Content-language: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
640 326 Content-Security-Policy-Report-Only: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
641 238 WARC-Truncated: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
642 114 Content-Disposition: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
643 352 Content-*: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
644 1 WARC-Filename: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
645 42 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
646 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
647 real 0m30.896s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
648 user 0m37.335s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
649 sys 0m7.542s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
650 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
651 First 51 after WARC-Type: response |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
652 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
653 >: time ~/lib/python/cc/unpackz.py -o /dev/stdout /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz |egrep -aA50 '^WARC-Type: response' | cut -f 1 -d ' ' | egrep -a '^(WARC-|Content-|Last-Modified)' |sus | tee >(wc -l 1>&2) |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
654 106775 Content-Length: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
655 106485 Content-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
656 55215 WARC-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
657 55123 WARC-Date: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
658 54988 WARC-Record-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
659 54551 WARC-Warcinfo-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
660 54246 WARC-Target-URI: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
661 54025 WARC-Concurrent-To: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
662 52806 WARC-IP-Address: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
663 52468 WARC-Block-Digest: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
664 52468 WARC-Identified-Payload-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
665 52468 WARC-Payload-Digest: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
666 9230 Last-Modified: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
667 3938 Content-Language: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
668 2261 Content-Security-Policy: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
669 639 Content-language: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
670 324 Content-Security-Policy-Report-Only: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
671 236 WARC-Truncated: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
672 114 Content-Disposition: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
673 342 Content-*: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
674 41 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
675 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
676 real 0m21.483s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
677 user 0m22.372s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
678 sys 0m5.400s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
679 |
| 58 | 680 So, not worth the risk, let's try python: cdx_extras implements a |
| 681 callback for unpackz that outputs the LM header if it's there | |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
682 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
683 >: time ~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz|wc -l |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
684 9238 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
685 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
686 real 0m25.426s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
687 user 0m23.201s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
688 sys 0m0.711s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
689 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
690 Looks good, but why 9238 instead of 9239??? |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
691 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
692 >: ~/lib/python/cc/unpackz.py -o /dev/stdout /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | egrep -a '^Last-Modified: ' > /tmp/hst/lmo.tsv |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
693 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
694 Argh. Serious bug in unpackz, wasn't handline cross-buffer-boundary |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
695 records correctly. Fixed. Redoing the above... |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
696 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
697 No pre-filter: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
698 >: uz /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz|egrep -c '^WARC/1\.0.$' |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
699 160297 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
700 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
701 >: time ~/lib/python/cc/unpackz.py -o /dev/stdout /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | cut -f 1 -d ' ' | egrep -a '^(WARC-|Content-|Last-Modified)' |sus | tee >(wc -l 1>&2) |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
702 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
703 213719 Content-Length: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
704 213088 Content-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
705 160297 WARC-Date: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
706 160297 WARC-Record-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
707 160297 WARC-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
708 160296 WARC-Target-URI: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
709 160296 WARC-Warcinfo-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
710 106864 WARC-Concurrent-To: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
711 106864 WARC-IP-Address: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
712 53432 WARC-Block-Digest: [consistent with 106297 == (3 * 53432) + 1] |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
713 53432 WARC-Identified-Payload-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
714 53432 WARC-Payload-Digest: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
715 9430 Last-Modified: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
716 4006 Content-Language: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
717 2325 Content-Security-Policy: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
718 653 Content-language: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
719 331 Content-Security-Policy-Report-Only: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
720 298 WARC-Truncated: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
721 128 Content-Disposition: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
722 83 Content-Location: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
723 67 Content-type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
724 51 Content-MD5: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
725 45 Content-Script-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
726 42 Content-Style-Type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
727 31 Content-Transfer-Encoding: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
728 13 Content-disposition: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
729 8 Content-Md5: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
730 5 Content-Description: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
731 5 Content-script-type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
732 5 Content-style-type: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
733 3 Content-transfer-encoding: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
734 2 Content-Encoding-handler: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
735 1 Content-DocumentTitle: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
736 1 Content-Hash: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
737 1 Content-ID: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
738 1 Content-Legth: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
739 1 Content-length: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
740 1 Content-Range: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
741 1 Content-Secure-Policy: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
742 1 Content-security-policy: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
743 1 Content-Type-Options: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
744 1 WARC-Filename: |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
745 42 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
746 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
747 real 0m28.876s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
748 user 0m35.703s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
749 sys 0m6.976s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
750 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
751 >: ~/lib/python/cc/unpackz.py -o /dev/stdout /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | egrep -a '^Last-Modified: ' > /tmp/hst/lmo.tsv |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
752 >: wc -l /tmp/hst/lmo.tsv |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
753 9430 /tmp/hst/lmo.tsv |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
754 >: time ~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz > /tmp/hst/lm.tsv |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
755 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
756 real 0m17.191s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
757 user 0m15.739s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
758 sys 0m0.594s |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
759 >: wc -l /tmp/hst/lm.tsv |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
760 9423 /tmp/hst/lm.tsv |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
761 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
762 >: diff <(sed 's/^Last-Modified: //' /tmp/hst/lmo.tsv | tr -d '\r') <(cut -f 3 /tmp/hst/lm.tsv) |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
763 853d852 |
| 58 | 764 < Mon, 19 Aug 2019 01:46:49 GMT [in XML comment at very end of xHTML] |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
765 4058d4056 |
| 58 | 766 < Tue, 03 Nov 2015 21:31:18 GMT<br /> [in an HTML table] |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
767 4405d4402 |
| 58 | 768 < Mon, 19 Aug 2019 01:54:52 GMT [double lm] |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
769 5237,5238d5233 |
| 58 | 770 < 3 [bogus extension lines to preceding LM] |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
771 < Asia/Amman |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
772 7009d7003 |
| 58 | 773 < Mon, 19 Aug 2019 02:34:20 GMT [in XML comment at very end of xHTML] |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
774 9198d9191 |
| 58 | 775 < Mon, 19 Aug 2019 02:14:49 GMT [in XML comment at very end of xHTML] |
|
47
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
776 |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
777 All good. The only implausable case is |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
778 < Mon, 19 Aug 2019 01:54:52 GMT |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
779 which turns out to be a case of two Last-Modified headers in the same |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
780 the same response record's HTTP headers. RFCs 2616 and 7230 rule it |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
781 out but neither specifies a recovery, so first-wins is as good as |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
782 anything, and indeed 6797 specifies that. |
|
fbdaede4155a
cdx_extras and unpackz.py working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
46
diff
changeset
|
783 |
| 48 | 784 Start looking at how we do the merge of cdx_extras.py with existing index |
| 785 | |
| 58 | 786 ====2024-12-19==== |
| 787 | |
| 788 The above test shows 17.6% of entries have an LM value | |
| 789 For a 3 billion entry dataset, than means 530 million LM entries, call | |
| 790 this n. | |
| 791 | |
| 792 Sizes? For a 10% error rate, we need m bits = -n * ln(.1) / ln(2)^2 | |
| 793 | |
| 794 (- (/ (* n (log .1)) (* (log 2)(log 2))) = 2,535,559,358 =~ 320MB | |
| 795 | |
| 796 That's too much :-) Per segment, that becomes possible? | |
| 797 25,355,594 bits =~ 3.2MB | |
| 798 | |
| 799 But maybe it's _not_ too much. One of the python implementations I | |
| 800 saw uses mmap: | |
| 801 | |
| 802 https://github.com/prashnts/pybloomfiltermmap3 | |
| 803 | |
| 804 Build a Bloom filter with all the URIs whose entries have LM value | |
| 805 _and_ a python hashtable mapping from URI to LM and offset (is that | |
| 806 enough for deduping?) | |
| 807 Rewrite one index file at a time | |
| 808 Probe with each URI, if positive | |
| 809 look up in hashtable and use if found | |
| 810 | |
| 811 >: wc -l ks*.tsv | |
| 812 52369734 ks_0-9.tsv | |
| 813 52489306 ks_10-19.tsv | |
| 814 52381115 ks_20-29.tsv | |
| 815 52438862 ks_30-39.tsv | |
| 816 52512044 ks_40-49.tsv | |
| 817 52476964 ks_50-59.tsv | |
| 818 52317116 ks_60-69.tsv | |
| 819 52200680 ks_70-79.tsv | |
| 820 52382426 ks_80-89.tsv | |
| 821 52295136 ks_90-99.tsv | |
| 822 523863383 total | |
| 68 | 823 Later saved the above in kslines.tsv |
| 58 | 824 |
| 825 >>> from pybloomfilter import BloomFilter | |
| 826 >>> f=BloomFilter(523863383,0.1,'/tmp/hst/uris.bloom') | |
| 827 >>> def bff(f,fn): | |
| 828 ... with open(fn) as uf: | |
| 829 ... while (l:=uf.readline()): | |
| 830 ... f.add(l.split('\t')[2]) | |
| 831 ... | |
| 832 >>> timeit.timeit("bff(f,'/dev/null')",number=1,globals=globals()) | |
| 833 0.00012309104204177856 | |
| 834 >>> timeit.timeit("bff(f,'/work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv')",number=1,globals=globals()) | |
| 835 77.57737312093377 | |
| 836 >>> 'http://71.43.189.10/dermorph' in f | |
| 837 False | |
| 838 >>> 'http://71.43.189.10/dermorph/' in f | |
| 839 True | |
| 840 >>> timeit.timeit("'http://71.43.189.10/dermorph/' in f",number=100000,globals=globals()) | |
| 841 0.02377822808921337 | |
| 842 >>> timeit.timeit("'http://71.43.189.10/dermorph' in f",number=100000,globals=globals()) | |
| 843 0.019318239763379097 | |
| 844 | |
| 845 _That's_ encouraging... | |
| 846 Be sure to f.close() | |
| 847 Use BloomFilter.open for an existing bloom file | |
| 848 Copying a file from /tmp to work/... still gives good (quick) lookup, | |
| 849 but _creating and filling_ a file on work/... takes ... I stopped | |
| 850 waiting after an hour or so. | |
| 59 | 851 |
| 852 How much bigger is .05 false positive? | |
| 853 Less than expected: | |
| 854 >: ls -l /tmp/hst | |
| 855 -rwxr-xr-x 1 hst dc007 408301988 Jan 1 16:52 uris_20.bloom | |
| 856 -rwxr-xr-x 1 hst dc007 313830100 Jan 1 15:04 uris.bloom | |
| 857 And still same (?) fill time: | |
| 858 >>> g=BloomFilter(523863383,0.05,'/tmp/hst/uris_20.bloom') | |
| 859 >>> T=timeit.Timer("bff(g,'/work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv')",globals=globals()) | |
| 860 >>> T.repeat(3,number=1) | |
| 861 [89.64385064691305, 90.9979057777673, 83.9632708914578] | |
| 862 Build a test harness wrt the python dict I'm going to need... | |
| 863 Can't immediately find a way to optimise a dict to have umpty millions | |
| 864 of entries... | |
| 865 >: cat /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv|~/lib/python/cc/lmh/test.py -n 1000000 -r 5 | |
| 866 1000002 | |
| 867 1000002 | |
| 868 1000002 | |
| 869 1000002 | |
| 870 1000002 | |
| 871 [1.229693355038762, 1.3374222852289677, 1.3509841952472925, 1.080365838482976, 1.1893387716263533] | |
| 872 Full as-it-were segment: | |
| 873 >: ~/lib/python/cc/lmh/test.py -n 6000000 -r 5 -f /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv | |
| 874 6000002 | |
| 875 6000002 | |
| 876 6000002 | |
| 877 6000002 | |
| 878 6000002 | |
| 879 [7.250897390767932, 7.237801244482398, 7.239673590287566, 7.32976414449513, 7.23588689416647] | |
| 880 Full 10th of the data: | |
| 881 >: ~/lib/python/cc/lmh/test.py -r 3 -f /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv | |
| 882 52369734 | |
| 883 52369734 | |
| 884 52369734 | |
| 885 [69.63967163302004, 69.09140252694488, 66.49750975705683] | |
| 886 That's tolerable. | |
| 60 | 887 >: ~/lib/python/cc/lmh/test_hash.py -r 3 -f /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv |
| 59 | 888 52369734 |
| 889 52369734 | |
| 890 52369734 | |
| 891 [64.51177835091949, 71.6610240675509, 67.74966451153159] | |
| 892 [0.0034751780331134796, 0.0034532323479652405, 0.0033454522490501404] | |
| 893 Last line is 100000 lookups. | |
| 60 | 894 |
| 895 So, try a test: | |
| 896 >: time ~/lib/python/cc/lmh/test_hash.py -r 1 -p results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.pickle -f /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv | |
| 897 52369734 | |
| 898 [70.98342595621943] | |
| 899 [0.0037928372621536255] | |
| 900 | |
| 901 real 1m51.456s | |
| 902 user 1m32.901s | |
| 903 sys 0m17.937s | |
| 904 >: ls -lh results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.pickle | |
| 905 -rw-r--r-- 1 hst dc007 5.5G Jan 2 12:19 results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.pickle | |
| 906 cdx_out.write(b' ') | |
| 907 cdx_out.write(b' ') | |
| 908 >: time ~/lib/python/cc/lmh/test_lookup1.py | |
| 909 52369734 | |
| 910 1076046 130318 | |
| 911 | |
| 912 real 1m52.668s | |
| 913 user 1m40.751s | |
| 914 sys 0m9.610s | |
| 915 | |
| 916 Not bad. 1.5 minutes per file, plus 10 x 20 secs or so for the | |
| 917 unpickles =~ 453 minutes == 8 hours. | |
| 918 | |
| 919 Try pre-filter with the Bloom filter. | |
|
61
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
920 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
921 Rebuild with byte values: |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
922 >>> from pybloomfilter import BloomFilter |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
923 >>> g=BloomFilter(523863383,0.05,'/tmp/hst/uris_20.bloom') |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
924 >>> def bff(f,fn): |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
925 ... with open(fn,'rb') as uf: |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
926 ... for l in uf: |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
927 ... f.add(l.split(b'\t')[2]) |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
928 ... |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
929 >>> timeit.timeit("bff(g,'/work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv')",number=1,globals=globals()) |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
930 80.17189159989357 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
931 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
932 >: time ~/lib/python/cc/lmh/test_lookup2.py |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
933 52369734 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
934 1076046 entries, 130318 given lastmod, 78 false positives |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
935 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
936 real 1m49.567s |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
937 user 1m38.818s |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
938 sys 0m8.668s |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
939 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
940 Not worth it :-(. |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
941 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
942 Build the rest: |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
943 >: time ~/lib/python/cc/lmh/test_hash.py -r 1 -p results/CC-MAIN-2019-35/warc_lmhx/ks_10-19.pickle -f /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_10-19.tsv |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
944 52489306 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
945 [54.296645537018776] |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
946 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
947 real 1m32.529s |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
948 user 1m16.956s |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
949 sys 0m14.267s |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
950 >: seq 2 9 | parallel -j 8 'time ~/lib/python/cc/lmh/test_hash.py -r 1 -p results/CC-MAIN-2019-35/warc_lmhx/ks_{}0-{}9.pickle -f /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_{}0-{}9.tsv' |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
951 |
|
e6bab0972142
tried pre-filtering with bloom, not much benefit if any
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
60
diff
changeset
|
952 Slightly slower when running in parallel, but done |
|
62
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
953 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
954 Try using cdb file, has to be one seg per file because of 4GB limit of |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
955 cdb C-code |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
956 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
957 >: ~/lib/python/cc/lmh/ks2cdb.py -f <(head -5236974 results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv) -c results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb_in |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
958 5236974 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
959 11.358113696798682 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
960 >: time cdbmake results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb results/CC-MAIN-2019-35/warc_lmhx/ks_0.tmp < results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb_in |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
961 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
962 real 0m4.981s |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
963 user 0m2.865s |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
964 sys 0m1.886s |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
965 >: ls -lh results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb_in |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
966 -rw-r--r-- 1 hst dc007 574M Jan 3 11:34 results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb_in |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
967 >: ls -lh results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
968 -rw-r--r-- 1 hst dc007 642M Jan 3 11:34 results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
969 >: ls -lh results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.{tsv,pickle} |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
970 -rw-r--r-- 1 hst dc007 5.4G Jan 2 13:41 results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.pickle |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
971 -rw-r--r-- 1 hst dc007 8.8G Oct 3 2023 results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
972 >: cdbdump <results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb | head |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
973 +52,10:20190817225913http://85.163.4.0:9999/faces/login.jsp->1566082752 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
974 +58,10:20190817232933https://1.1.1.1/?source=https://www.tuppu.fi->1560198539 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
975 +34,10:20190822010151http://163.30.193.1/->1256804738 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
976 +64,10:20190825182530http://45.70.224.1:20026/index.php?tipo=minhaconta->1565613324 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
977 +32,10:20190818210532http://50.63.77.1/->1366940302 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
978 +41,10:20190825182606http://119.145.170.10:2160/->1497240281 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
979 +43,10:20190825142846http://71.43.189.10/dermorph/->1564555978 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
980 +53,10:20190825143020http://71.43.189.10/dermorph/about.html->1564555936 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
981 +55,10:20190825142857http://71.43.189.10/dermorph/contact.html->1564748606 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
982 +54,10:20190825142842http://71.43.189.10/dermorph/guided.html->1563634912 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
983 sing<4795>: cdbget 20190825142846http://71.43.189.10/dermorph/ <results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
984 1564555978 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
985 >: cdbstats <results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
986 records 5226422 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
987 d0 3920023 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
988 d1 748992 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
989 d2 269375 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
990 d3 124011 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
991 d4 64304 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
992 d5 36623 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
993 d6 21765 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
994 d7 13737 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
995 d8 8712 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
996 d9 5866 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
997 >9 13014 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
998 >: cdbtest <results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
999 found: 5226422 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1000 different record: 0 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1001 bad length: 0 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1002 not found: 0 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1003 untested: 10552 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1004 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1005 All good, _but_ |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1006 >>> r=cdblib.Reader.from_file_path('results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb') |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1007 >>> r.get(b'xyzzy') |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1008 >>> r.get(b'20190825142846http://71.43.189.10/dermorph/') |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1009 b'1564555978' |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1010 >>> from timeit import Timer |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1011 >>> t=Timer("r.get(b'20190825142846http://71.43.189.10/dermorph/')", |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1012 globals=globals()) |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1013 >>> t.timeit(100000) |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1014 0.30162674374878407 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1015 >>> t.timeit(100000) |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1016 0.30807616002857685 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1017 >>> t.timeit(100000) |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1018 0.30170613154768944 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1019 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1020 So, 100 x slower than a dict :-(. Just checking that... |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1021 >>> t = Timer("b'20190825142846http://71.43.189.10/dermorph/' in d",globals={'d':d}) |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1022 >>> r=cdblib.Reader.from_file_path('results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb') |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1023 >>> r.get(b'20190825142846http://71.43.189.10/dermorph/') |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1024 b'1564555978' |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1025 >>> s = Timer("r.get(b'20190825142846http://71.43.189.10/dermorph/')", |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1026 ... globals=globals()) |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1027 >>> (t.repeat(5,100000),s.repeat(5,100000)) |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1028 [0.005662968382239342, 0.005780909210443497, 0.005478940904140472, 0.005713008344173431, 0.005547545850276947] |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1029 [0.30250774696469307, 0.303345350548625, 0.3002819549292326, 0.30161340720951557, 0.30262864381074905]) |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1030 |
|
bc0bdb649c08
tried cdb, slower by 2 OoM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
61
diff
changeset
|
1031 so, forget cdb |
|
63
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1032 Hmm... |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1033 >: time cdbtest <results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1034 found: 5226422 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1035 different record: 0 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1036 bad length: 0 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1037 not found: 0 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1038 untested: 10552 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1039 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1040 real 0m34.151s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1041 user 0m7.513s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1042 sys 0m5.123s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1043 (/ 34.1 5226422) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1044 6.524540115589595e-06 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1045 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1046 Added a bunch of type hints, tried cython |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1047 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1048 >: cython --embed test_lookup1.py |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1049 ...[warnings] |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1050 >: python3 -m autotyping --aggressive test_lookup1.py |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1051 Calculating full-repo metadata... |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1052 ... |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1053 - 0 warnings were generated. |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1054 >: cc -I /usr/include/python3.11 -L /usr/lib/python3.11/config-3.11-x86_64-linux-gnu/ test_lookup1.c -l python3.11 -o test_lookup1 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1055 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1056 >: time ~/lib/python/cc/lmh/test_lookup1.py |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1057 52369734 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1058 1076046 entries, 130318 given lastmod |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1059 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1060 real 1m45.305s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1061 user 1m34.265s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1062 sys 0m8.988s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1063 sing<4969>: time ~/lib/python/cc/lmh/test_lookup1 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1064 52369734 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1065 1076046 entries, 130318 given lastmod |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1066 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1067 real 2m5.626s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1068 user 1m54.493s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1069 sys 0m9.169s |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1070 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1071 Compiled in profiling |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1072 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1073 >: python3 /work/dc007/dc007/hst/lib/python/cc/lmh/test_pl.py |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1074 /work/dc007/dc007/hst/lib/python/cc/lmh/test_lookup1.py |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1075 52369734 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1076 1076046 entries, 130318 given lastmod |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1077 Mon Jan 6 13:56:04 2025 Profile.prof |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1078 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1079 152694273 function calls (152694246 primitive calls) in 169.605 seconds |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1080 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1081 Ordered by: internal time |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1082 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1083 ncalls tottime percall cumtime percall filename:lineno(function) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1084 1 30.001 30.001 30.001 30.001 {built-in method _pickle.load} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1085 11195226 29.825 0.000 68.952 0.000 igzip.py:196(write) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1086 11195226 24.403 0.000 24.403 0.000 {method 'compress' of 'isal_zlib.Compress' objects} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1087 1 23.919 23.919 167.278 167.278 test_lookup1.py:9(mainp) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1088 10543636 14.072 0.000 14.072 0.000 {method 'search' of 're.Pattern' objects} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1089 10543637 11.911 0.000 11.911 0.000 {method 'readline' of '_io.BufferedReader' objects} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1090 21738863 8.646 0.000 12.583 0.000 _compression.py:12(_check_not_closed) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1091 10543637 8.431 0.000 26.779 0.000 gzip.py:397(readline) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1092 21738868 3.937 0.000 3.937 0.000 gzip.py:323(closed) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1093 10543636 3.524 0.000 3.524 0.000 {method 'split' of 'bytes' objects} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1094 11195226 2.951 0.000 2.951 0.000 {built-in method isal.isal_zlib.crc32} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1095 11195236 2.814 0.000 2.814 0.000 {method 'write' of '_io.BufferedWriter' objects} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1096 1 2.327 2.327 169.605 169.605 <string>:1(<module>) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1097 11195324 1.414 0.000 1.414 0.000 {built-in method builtins.isinstance} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1098 11064997/11064989 1.399 0.000 1.399 0.000 {built-in method builtins.len} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1099 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1100 >: python3 /work/dc007/dc007/hst/lib/python/cc/lmh/test_pl.py |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1101 /work/dc007/dc007/hst/lib/python/cc/lmh/test_lookup1.cpython-311-x86_64-linux-gnu.so |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1102 52369734 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1103 1076046 entries, 130318 given lastmod |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1104 Mon Jan 6 14:50:59 2025 Profile.prof |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1105 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1106 131606993 function calls (131606966 primitive calls) in 164.213 seconds |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1107 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1108 Ordered by: internal time |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1109 |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1110 ncalls tottime percall cumtime percall filename:lineno(function) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1111 1 67.658 67.658 164.213 164.213 test_lookup1.py:9(mainp) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1112 11195226 29.944 0.000 69.788 0.000 igzip.py:196(write) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1113 11195226 25.274 0.000 25.274 0.000 {method 'compress' of 'isal_zlib.Compress' objects} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1114 10543637 12.200 0.000 12.200 0.000 {method 'readline' of '_io.BufferedReader' objects} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1115 21738863 8.530 0.000 12.394 0.000 _compression.py:12(_check_not_closed) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1116 10543637 8.205 0.000 26.723 0.000 gzip.py:397(readline) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1117 21738868 3.863 0.000 3.863 0.000 gzip.py:323(closed) |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1118 11195226 3.058 0.000 3.058 0.000 {built-in method isal.isal_zlib.crc32} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1119 11195236 2.784 0.000 2.784 0.000 {method 'write' of '_io.BufferedWriter' objects} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1120 11064996/11064988 1.343 0.000 1.343 0.000 {built-in method builtins.len} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1121 11195324 1.309 0.000 1.309 0.000 {built-in method builtins.isinstance} |
|
663e55844c1d
comparing profiles w and w/o cython
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
62
diff
changeset
|
1122 2 0.040 0.020 0.040 0.020 {built-in method io.open} |
| 64 | 1123 |
| 1124 After a long diversion to get cython access to C extensions working, | |
| 1125 now have a setup that works, in lib/python/cc/lmh: | |
| 1126 | |
| 1127 >: python3 setup.py build_ext -i | |
| 1128 >: python3 -c 'import db' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb xyzzy | |
| 1129 testing... | |
| 1130 0 | |
| 1131 tested | |
| 1132 >: python3 -c 'import db' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ | |
| 1133 testing... | |
| 1134 1 | |
| 1135 tested | |
| 1136 >: python3 -c 'import db' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 | |
| 1137 testing... | |
| 1138 1 | |
| 1139 tested | |
| 1140 1.8913232665508986 | |
| 1141 >: python3 -c 'import db' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb xyzzy 10000000 | |
| 1142 testing... | |
| 1143 0 | |
| 1144 tested | |
| 1145 1.31287781894207 | |
| 1146 sing<3792>: python3 -c 'import db' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb xyzzy 10000000 | |
| 1147 testing... | |
| 1148 0 | |
| 1149 tested | |
| 1150 1.3081446290016174 | |
| 1151 So, maybe that's hopeful | |
| 1152 | |
| 1153 to get the values out: | |
| 1154 https://candide-guevara.github.io/cs_related/2018/02/28/cython-memory-mgt.html | |
| 1155 mmap->memoryview, maybe? | |
| 1156 or | |
| 1157 cdef char[::1] mview = <char[:size:1]>(bp) | |
| 1158 https://stackoverflow.com/questions/50228544/how-do-i-read-a-c-char-array-into-a-python-bytearray-with-cython | |
| 1159 or, if that doesn't work, have to copy: | |
| 1160 https://stackoverflow.com/questions/50228544/how-do-i-read-a-c-char-array-into-a-python-bytearray-with-cython | |
| 1161 | |
| 1162 Not quite there, but, maybe, nearly: | |
| 1163 >: python3 -c 'import db' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 | |
| 1164 testing... | |
| 1165 2488 10 # offset and length? | |
| 1166 tested | |
| 1167 1.9035462848842144 | |
| 1168 >: dd ibs=1 skip=2488 count=10 if=~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb of=/dev/stdout | |
| 1169 1564555978 | |
| 1170 >: cdbget 20190825142846http://71.43.189.10/dermorph/ <~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb | |
| 1171 1564555978 | |
| 65 | 1172 |
| 1173 OK, definitely worth trying: | |
| 1174 >: python3 -c 'import db' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 | |
| 1175 testing... | |
| 1176 2488 10 | |
| 1177 1564555978 | |
| 1178 tested | |
| 1179 1.800725992769003 | |
| 1180 A long struggle to get the module structure better, finally sorted | |
| 1181 cdb.pxd exposes cdb.h from cdb-0.75 (slightly updated) | |
| 1182 db.{pyx, pxd} define an interface cython class to hold a cdb.Cdb instance | |
| 1183 nndb.pyx does a stress test of lookup in a cdb file from 1% of the | |
| 1184 timestamps | |
| 1185 >: python3 -c 'import ndb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 | |
| 1186 testing... /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb b'20190825142846http://71.43.189.10/dermorph/' x 10000000 | |
| 1187 mv 672917132 cdb 140602889433984 | |
| 1188 1 2488 10 | |
| 1189 1564555978 | |
| 1190 tested | |
| 1191 2.055266048759222 | |
|
66
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1192 Oops, that was ndb, and nndb doesn't work! |
| 65 | 1193 |
| 1194 Things to try next: | |
| 67 | 1195 1) Build a bigger .cdb w. as close to 4GB as possible Done |
| 1196 2) Shift to a shared library for cdb-0.75 Done | |
| 65 | 1197 3) Get rid of the single fixed Cdb struct instance and malloc it as |
| 67 | 1198 required Done |
| 1199 3a) Remove debugging output and recompile everything Done | |
| 65 | 1200 4) Build and test the real harness to process .cdx files using .cdb |
|
66
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1201 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1202 Try 50% more, e.g. approx. 1.5 segments |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1203 >: python3 -c "print(1.5 * 5236974)" |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1204 7855461.0 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1205 >: time ~/lib/python/cc/lmh/ks2cdb.py -f <(head -7855461 results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv) -c results/CC-MAIN-2019-35/warc_lmhx/ks_0.5.cdb_in7855461 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1206 real 0m14.585s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1207 user 0m13.909s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1208 sys 0m3.308s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1209 >: time cdbmake results/CC-MAIN-2019-35/warc_lmhx/ks_0.5.cdb results/CC-MAIN-2019-35/warc_lmhx/ks_0.5.tmp < results/CC-MAIN-2019-35/warc_lmhx/ks_0.5.cdb_in |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1210 real 0m6.075s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1211 user 0m3.682s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1212 sys 0m2.337s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1213 >: ls -lh results/CC-MAIN-2019-35/warc_lmhx/ks_0.5.cdb |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1214 -rw-r--r-- 1 hst dc007 991M Jan 27 15:00 results/CC-MAIN-2019-35/warc_lmhx/ks_0.5.cdb |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1215 >: python3 -c 'import ndb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1216 testing... /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb b'20190825142846http://71.43.189.10/dermorph/' x 10000000 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1217 mv 672917132 cdb 140324514468736 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1218 1 2488 10 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1219 1564555978 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1220 tested |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1221 2.016317328438163 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1222 >: python3 -c 'import ndb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.5.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1223 testing... /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0.5.cdb b'20190825142846http://71.43.189.10/dermorph/' x 10000000 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1224 mv 1039033960 cdb 140207982297984 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1225 1 2488 10 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1226 1564555978 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1227 tested |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1228 2.0484518501907587 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1229 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1230 So, that's OK, but still, would need 67 hash files == 265GB |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1231 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1232 Wait, isn't it 4GB max??? |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1233 Build ks_0-9.60.cdb, (as in 60% of 0-9, 31M lines ): |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1234 >: time ~/lib/python/cc/lmh/ks2cdb.py -f <(head -$((4 * 7855461)) ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.tsv) -c ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb_in |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1235 31421844 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1236 real 0m58.676s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1237 user 0m56.312s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1238 sys 0m12.306s |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1239 >: cdbmake ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.tmp < ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb_in |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1240 >: ls -lh ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1241 -rw-r--r-- 1 hst dc007 3.8G Jan 27 15:26 /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1242 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1243 cdbget works: |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1244 >: cdb-0.75/cdbget 20190818122159https://m.europapress.es/navarra/noticia-gobierno-navarra-gamesa-acuerdan-necesidad-reindustrializar-planta-alsasua-20100525100333.html <~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1245 1566130319 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1246 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1247 But cdbtest and cdbstats die with that as input. Looks like it's when int |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1248 overflows to -2MB |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1249 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1250 Careful review of cdb.c/h to change int to uint32 whenever an offset |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1251 in the mmap is stored finally got odb working, in due course should |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1252 try to fix and fork all of cdb |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1253 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1254 Still crashing with 0-9.60 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1255 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1256 Ah, a further int pblm -- editted seek-pos to use uint32, now |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1257 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1258 >: cdb-0.75/cdbtest <~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1259 found: 31403644 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1260 different record: 0 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1261 bad length: 0 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1262 not found: 0 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1263 untested: 18200 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1264 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1265 Finally got a version of nndb (which cimports db as well as cdb) to |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1266 work by moving _all_ uses of _c_cdb into db. Still comparable in |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1267 speed to the mixed version in e.g. odb: |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1268 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1269 >: python3 -c 'import odb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1270 testing... |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1271 mv 672917132 cdb 140317635072864 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1272 2488 10 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1273 1564555978 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1274 tested |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1275 2.1426388323307037 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1276 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1277 >: python3 -c 'import nndb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1278 testing... /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb b'20190825142846http://71.43.189.10/dermorph/' x 10000000 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1279 1 2488 10 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1280 1564555978 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1281 tested |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1282 2.0193762965500355 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1283 2.702429808676243 10000000 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1284 |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1285 Interesting that just adding a counter to the test loop slows it down |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1286 so much: |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1287 'cfind(probe)', |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1288 vs |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1289 '(X:=X+1) if cfind(probe)==1 else None', |
|
0c814f07865a
sequestration of cdb handle complete and working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
65
diff
changeset
|
1290 setup = 'global X' |
| 67 | 1291 |
| 1292 For this test, size of db doesn't matter: | |
| 1293 >: python3 -c 'import nndb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 | |
| 1294 testing... /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0-9.60.cdb b'20190825142846http://71.43.189.10/dermorph/' x 10000000 | |
| 1295 1 2488 10 | |
| 1296 1564555978 | |
| 1297 tested | |
| 1298 1.9914793614298105 | |
| 1299 2.711402891203761 10000000 | |
| 1300 | |
| 1301 >: python3 -c 'import nndb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 | |
| 1302 testing... /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb b'20190825142846http://71.43.189.10/dermorph/' x 10000000 | |
| 1303 1 2488 10 | |
| 1304 1564555978 | |
| 1305 tested | |
| 1306 2.015953341498971 | |
| 1307 2.798953704535961 10000000 | |
| 1308 | |
| 1309 Now using non-static Cdb, it's slower??? Cirrus load is low :-(. | |
| 1310 | |
| 1311 >: python3 -c 'import nndb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 | |
| 1312 testing... /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb b'20190825142846http://71.43.189.10/dermorph/' x 10000000 | |
| 1313 1 2488 10 | |
| 1314 1564555978 | |
| 1315 tested | |
| 1316 2.474294847997953 | |
| 1317 3.2440936170023633 10000000 | |
| 1318 | |
| 68 | 1319 [Forget that, it's back to more or less the same timing as before, |
| 1320 mystery | |
| 1321 >: python3 -c 'import nndb' ~/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb 20190825142846http://71.43.189.10/dermorph/ 10000000 | |
| 1322 testing... /work/dc007/dc007/hst/results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb b'20190825142846http://71.43.189.10/dermorph/' x 10000000 | |
| 1323 1 2488 10 | |
| 1324 1564555978 | |
| 1325 tested | |
| 1326 2.0566618730081245 | |
| 1327 2.6779559339920525 10000000 | |
| 1328 ] | |
| 1329 | |
| 1330 And it works: | |
| 67 | 1331 |
| 1332 >: python3 -c 'import get2' cdb/rts-tmp/sv.cdb cdb/rts-tmp/12.cdb two discard/tcp | |
| 1333 two cdb/rts-tmp/sv.cdb missing | |
| 1334 discard/tcp cdb/rts-tmp/sv.cdb 9 | |
| 1335 two cdb/rts-tmp/12.cdb Goodbye | |
| 1336 discard/tcp cdb/rts-tmp/12.cdb missing | |
| 1337 | |
| 68 | 1338 Spent a morning getting profiling working, but only from/for apps |
| 1339 (cdbget, cdbtest) within the CDB distro. Had to change _exit(0) to | |
| 1340 exit(0) to get the profile data (mcount) actually written out. | |
| 1341 | |
| 1342 And guess what, by far the most time is spent in byte_copy. Noah is | |
| 1343 still correctly predicting things! | |
| 1344 | |
| 1345 Each sample counts as 0.01 seconds. | |
| 1346 % cumulative self self total | |
| 1347 time seconds seconds calls ms/call ms/call name | |
| 1348 41.10 1.54 1.54 119212725 0.00 0.00 byte_copy | |
| 1349 13.21 2.04 0.50 479500623 0.00 0.00 cdb_hashadd | |
| 1350 8.14 2.34 0.31 83427967 0.00 0.00 get | |
| 1351 7.74 2.63 0.29 5226422 0.00 0.00 cdb_findnext | |
| 1352 6.94 2.89 0.26 5226422 0.00 0.00 cdb_hash | |
| 1353 5.60 3.10 0.21 83489614 0.00 0.00 buffer_get | |
| 1354 4.27 3.26 0.16 17427786 0.00 0.00 byte_diff | |
| 1355 3.20 3.38 0.12 main | |
| 1356 2.13 3.46 0.08 5226422 0.00 0.00 seek_cur | |
| 1357 2.00 3.54 0.08 5226422 0.00 0.00 cdb_find | |
| 1358 1.47 3.59 0.06 35723096 0.00 0.00 cdb_read | |
| 1359 1.33 3.64 0.05 10474460 0.00 0.00 getnum | |
| 1360 | |
| 1361 And | |
| 1362 ---------------------------------------------- | |
| 1363 0.00 0.00 15/119212725 buffer_put [19] | |
| 1364 0.46 0.00 35723096/119212725 cdb_read [7] | |
| 1365 1.08 0.00 83489614/119212725 buffer_get [5] | |
| 1366 [4] 41.1 1.54 0.00 119212725 byte_copy [4] | |
| 1367 | |
| 1368 Not clear why this has slowed down 20--25% since Cirrus reboot... | |
| 1369 Obvs. could try increasing buffer size which might well help... | |
| 1370 | |
| 1371 Well, changing to 64K didn't help. | |
| 1372 it's giving the same results... | |
| 1373 | |
| 1374 Anyway, let's do a real experiment: | |
| 1375 | |
| 1376 | |
| 1377 >: seq 1 9 | parallel -j 9 'echo {} $(wc -l < ks_{}0-{}9.tsv)' |sort -k1n,1 >> ks_lines.tsv | |
| 1378 >: rename -n 's/\.(.*)/-\1.tsv/' ks.?? | |
| 1379 >: wc -l < ks.00 | |
| 1380 32468548 | |
| 1381 >: wc -l < ../ks_0-9.60.cdb_in | |
| 1382 31421845 | |
| 1383 >: cdbmake ks-00.cdb ks-00.tmp < ks-00.cdb_in | |
| 1384 >: ls -lh ks-00.cdb_in | |
| 1385 -rw-r--r-- 1 hst dc007 3.5G Feb 10 14:46 ks-00.cdb_in | |
| 1386 | |
| 1387 Try more? | |
| 1388 | |
| 1389 >: split -d -n l/1/15 /tmp/hst/ks.tsv > ks-00_15.tsv | |
| 1390 >: ~/lib/python/cc/lmh/ks2cdb.py -f ks-00_15.tsv -c ks-00_15.cdb_in | |
| 1391 34382190 | |
| 1392 >: cdbmake ks-00_15.cdb ks-00_15.tmp < ks-00_15.cdb_in | |
| 1393 cdbmake: fatal: unable to create ks-00_15.tmp: out of memory | |
| 1394 | |
| 1395 OK, 16 it is. | |
| 1396 >: seq -f "%02g" 1 15 | parallel -j 8 '~/lib/python/cc/lmh/ks2cdb.py -f ks-{}.tsv -c ks-{}.cdb_in && cdbmake ks-{}.cdb ks-{}.tmp < ks -{}.cdb_in' | |
| 1397 33058467 | |
| 1398 /usr/bin/bash: line 1: ks: No such file or directory [oops] | |
| 1399 32925450 | |
| 1400 32194266 | |
| 1401 32611527 | |
| 1402 33059551 | |
| 1403 32235932 | |
| 1404 32466927 | |
| 1405 33341399 | |
| 1406 >: seq -f "%02g" 9 15 | parallel -j 7 '~/lib/python/cc/lmh/ks2cdb.py -f ks-{}.tsv -c ks-{}.cdb_in' | |
| 1407 32282702 | |
| 1408 32852577 | |
| 1409 33016202 | |
| 1410 32294662 | |
| 1411 32584480 | |
| 1412 33069019 | |
| 1413 33401674 | |
| 1414 >: seq -f "%02g" 1 15 | parallel -j 8 'cdbmake ks-{}.cdb ks-{}.tmp < ks-{}.cdb_in' | |
| 1415 >: ls -lh *.cdb | |
| 1416 -rw-r--r-- 1 hst dc007 3.9G Feb 10 14:47 ks-00.cdb | |
| 1417 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:21 ks-01.cdb | |
| 1418 -rw-r--r-- 1 hst dc007 4.0G Feb 10 15:21 ks-02.cdb | |
| 1419 -rw-r--r-- 1 hst dc007 4.0G Feb 10 15:21 ks-03.cdb | |
| 1420 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:21 ks-04.cdb | |
| 1421 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:21 ks-05.cdb | |
| 1422 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:21 ks-06.cdb | |
| 1423 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:21 ks-07.cdb | |
| 1424 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:21 ks-08.cdb | |
| 1425 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:21 ks-09.cdb | |
| 1426 -rw-r--r-- 1 hst dc007 4.0G Feb 10 15:22 ks-10.cdb | |
| 1427 -rw-r--r-- 1 hst dc007 4.0G Feb 10 15:21 ks-11.cdb | |
| 1428 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:22 ks-12.cdb | |
| 1429 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:22 ks-13.cdb | |
| 1430 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:22 ks-14.cdb | |
| 1431 -rw-r--r-- 1 hst dc007 3.9G Feb 10 15:22 ks-15.cdb | |
| 69 | 1432 >: time cdbtest < ks-00.cdb |
| 1433 found: 32449485 | |
| 1434 different record: 0 | |
| 1435 bad length: 0 | |
| 1436 not found: 0 | |
| 1437 untested: 19063 | |
| 68 | 1438 |
| 69 | 1439 real 2m42.674s |
| 1440 user 0m46.146s | |
| 1441 sys 0m29.479s | |
| 1442 >: ls *.cdb | fgrep -v 00 | parallel -j 8 'cdbtest < {} | egrep -v " 0$"' |& tee /tmp/hst/cdbtest.out & | |
| 1443 found: 32588023 | |
| 1444 untested: 23504 | |
| 1445 found: 32180038 | |
| 1446 untested: 14228 | |
| 1447 found: 33045196 | |
| 1448 untested: 13271 | |
| 1449 found: 32216571 | |
| 1450 untested: 19361 | |
| 1451 found: 33329508 | |
| 1452 untested: 11891 | |
| 1453 found: 32449830 | |
| 1454 untested: 17097 | |
| 1455 found: 32913005 | |
| 1456 untested: 12445 | |
| 1457 found: 33046940 | |
| 1458 untested: 12611 | |
| 1459 found: 33056259 | |
| 1460 untested: 12760 | |
| 1461 found: 32277255 | |
| 1462 untested: 17407 | |
| 1463 found: 32270904 | |
| 1464 untested: 11798 | |
| 1465 found: 32838272 | |
| 1466 untested: 14305 | |
| 1467 found: 32560341 | |
| 1468 untested: 24139 | |
| 1469 found: 33389520 | |
| 1470 untested: 12154 | |
| 1471 found: 33002021 | |
| 1472 untested: 14181 | |
| 1473 | |
| 1474 All good. | |
| 1475 Loaded all 16 in to a python image w/o difficulty | |
| 1476 Converted CCdb shim to use cpdef for most methods w/o apparent loss of | |
| 1477 speed. | |
| 1478 | |
| 1479 But, sigh, need to be on segment boundaries so I can tell by the | |
| 1480 filename in an original index entry which cdb to look in. | |
| 1481 >: echo ?/ks_lines.txt ??/ks_lines.txt | xargs cat | tr ' ' '\n' |python3 -c 'M = 33000000 | |
| 1482 import sys | |
| 1483 n = 0 | |
| 1484 for i in range(100): | |
| 1485 j = int(sys.stdin.readline()) | |
| 1486 if n+j > M: | |
| 1487 print(i-1, n, sep = "\t") | |
| 1488 n = j | |
| 1489 else: | |
| 1490 n += j | |
| 1491 ' > ks_divs.tsv | |
| 1492 >: cat ks_divs.tsv | |
| 1493 5 31296954 | |
| 1494 11 31587394 | |
| 1495 17 31428928 | |
| 1496 23 31622785 | |
| 1497 29 31304094 | |
| 1498 35 31467923 | |
| 1499 41 31502409 | |
| 1500 47 31456463 | |
| 1501 53 31462669 | |
| 1502 59 31538406 | |
| 1503 65 31355147 | |
| 1504 71 31281451 | |
| 1505 77 31429609 | |
| 1506 83 31389542 | |
| 1507 89 31444473 | |
| 1508 95 31385531 | |
| 1509 >: wc -l ks_divs.tsv | |
| 1510 16 ks_divs.tsv | |
| 1511 | |
| 1512 So, has to be 17 :-( | |
| 70 | 1513 |
| 1514 >: cut -f 1 ../ks_divs.tsv | { i=0; while read j; do printf "%s\t%s\n" $i $j; i=$((j+1)); done ; printf "%s\t%s\n" $i 99 ; } | parallel --colsep "\t" 'echo cat ../\{{1}..{2}\}/ks.tsv \> ks_{1}-{2}.tsv \&' | |
| 1515 [couldn't make this work as written, hence the echo, followed by copy-paste] | |
| 1516 >: cut -f 1 ../ks_divs.tsv | { i=0; while read j; do printf "ks_%s-%s\n" $i $j; i=$((j+1)); done ; printf "ks_%s-%s\n" $i 99; } | parallel -j 9 '~/lib/python/cc/lmh/ks2cdb.py -f {}.tsv -c {}.cdb_in' & | |
| 1517 >: cut -f 1 ../ks_divs.tsv | { i=0; while read j; do printf "ks_%s-%s\n" $i $j; i=$((j+1)); done ; printf "ks_%s-%s\n" $i 99; } | parallel -j 9 'cdbmake {}.cdb {}.tmp <{}.cdb_in' & | |
| 1518 >: cut -f 1 ../ks_divs.tsv | { i=0; while read j; do printf "ks_%s-%s\n" $i $j; i=$((j+1)); done ; printf "ks_%s-%s\n" $i 99; } | parallel -j 9 'cdb < {}.cdb' > cdbtest.out & | |
|
71
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1519 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1520 Finished deveopment of test_lookup3, now renamed as test_cdb.py. |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1521 Runs, but _very_ slowly, only 117,666 lines output in 30 minutes. |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1522 top shows it's in state I for Idle: |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1523 1238723 hst 20 0 62.3g 89664 43840 I 2.0 0.0 0:01.81 test_cdb |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1524 No better on a compute node. |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1525 Maybe it's thrashing? |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1526 Tried |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1527 >: cythonize -i test_cdb.py |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1528 >: PYTHONPATH=~/lib/python/cc/lmh python3 -c 'import test_cdb |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1529 test_cdb.mainp() |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1530 ' ks_%d-%d.cdb |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1531 No better, AFAICS |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1532 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1533 Try not using python isal/g(un)zipping? That didn't help either (see |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1534 test_cdbp.py, even a version which only tests entries from segment 0) |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1535 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1536 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1537 1546945 hst 20 0 62.3g 50180 41104 I 1.0 0.0 0:00.98 python3 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1538 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1539 But the suppied test code, which processes 31 million keys, is |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1540 averaging |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1541 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1542 Whereas |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1543 >: time cdbtest < ks_0-5.cdb |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1544 found: 31,281,173 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1545 untested: 15781 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1546 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1547 real 2m45.149s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1548 user 0m48.747s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1549 sys 0m31.113s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1550 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1551 31M in 165 seconds (2.75 minutes) == 5.27e-06 (5 microsec???) per key |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1552 compared to nndb result of 2.67 _seconds_ for 10,000,000 (identical) probes |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1553 i.e. 2.67e-7 per probe. |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1554 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1555 Something weird just happened: |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1556 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1557 1777850 hst 20 0 11.2g 334892 325768 R 91.8 0.1 0:43.41 python3 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1558 1777851 hst 20 0 5192 2900 608 S 23.9 0.0 0:10.28 igzip |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1559 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1560 test_cdbp.mainp() |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1561 ' ks_%d-%d.cdb 3 0 > ../cdx-00101 2>/tmp/hst/three |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1562 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1563 real 0m54.498s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1564 user 0m44.699s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1565 sys 0m17.472s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1566 sing<4015>: fgrep -c lastmod ../cdx-00101 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1567 20203 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1568 sing<4016>: date |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1569 Wed Feb 26 05:59:06 PM GMT 2025 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1570 sing<4017>: ls -l date |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1571 ls: cannot access 'date': No such file or directory |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1572 sing<4018>: ls -l ../cdx-00101 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1573 -rw-r--r-- 1 hst dc007 7044044593 Feb 26 17:58 ../cdx-00101 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1574 sing<4019>: ls -l /tmp/hst/three |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1575 -rw-r--r-- 1 hst dc007 0 Feb 26 17:57 /tmp/hst/three |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1576 _Not_ because of adding more: |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1577 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1578 test_cdbp.mainp() |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1579 ' ks_%d-%d.cdb 1 0 > ../cdx-00101 2>/tmp/hst/one |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1580 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1581 real 0m52.195s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1582 user 0m42.983s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1583 sys 0m17.266s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1584 sing<4021>: fgrep -c lastmod ../cdx-00101 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1585 20203 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1586 sing<4022>: ls -l ../cdx-00101 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1587 -rw-r--r-- 1 hst dc007 7044044593 Feb 26 18:04 ../cdx-00101 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1588 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1589 Sometimes fast, sometimes not? |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1590 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1591 test_cdbp.mainp() |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1592 ' ks_%d-%d.cdb 0 1 0 1 > ../cdx-00101 2>/tmp/hst/one |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1593 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1594 real 2m50.546s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1595 user 0m45.288s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1596 sys 0m20.332s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1597 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz | cat|python3 -c 'import test_cdbp |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1598 test_cdbp.mainp() |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1599 ' ks_%d-%d.cdb 0 1 0 1 > ../cdx-00101x 2>/tmp/hst/onex |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1600 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1601 real 0m49.305s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1602 user 0m41.800s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1603 sys 0m22.880s |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1604 |
|
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1605 I thought having the 'cat' in the pipeline was making the difference, |
| 75 | 1606 but no, just as fast w/o. Something very odd. |
| 1607 Next day, losing again: | |
| 1608 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp | |
| 1609 test_cdbp.mainp() | |
| 1610 ' ks_%d-%d.cdb 0 1 0 1 > ../cdx-00101x 2>/tmp/hst/onex | |
| 1611 | |
| 1612 real 2m37.951s | |
| 1613 user 0m45.490s | |
| 1614 sys 0m20.874s | |
| 1615 Try a full 17th anyway: | |
| 1616 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp | |
| 1617 test_cdbp.mainp() | |
| 1618 ' ks_%d-%d.cdb 0 1 0 6 > ../cdx-00101 2>/tmp/hst/six | |
| 1619 | |
| 1620 real 2m56.073s | |
| 1621 user 0m45.699s | |
| 1622 sys 0m19.964s | |
| 1623 Not significantly slower... | |
| 1624 >: fgrep -c lastmod ../cdx-00101 | |
| 1625 111967 | |
| 1626 >: wc -l ../cdx-00101 | |
| 1627 14681149 | |
| 1628 Try 2/17ths | |
| 1629 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp | |
| 1630 test_cdbp.mainp() | |
| 1631 ' ks_%d-%d.cdb 0 2 0 12 > ../cdx-00101 2>/tmp/hst/twelve | |
| 1632 real 4m19.232s | |
| 1633 user 0m50.538s | |
| 1634 sys 0m27.702s | |
| 1635 >: fgrep -c lastmod ../cdx-00101 | |
| 1636 221468 | |
| 1637 At least approx. linear... | |
| 1638 Try a compute node... | |
| 1639 No better | |
| 1640 Try chaining | |
| 1641 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00102.gz | python3 -c 'import test_cdbp | |
| 1642 test_cdbp.mainp() | |
| 1643 ' ks_%d-%d.cdb 0 2 0 12 | python3 -c 'import test_cdbp | |
| 1644 test_cdbp.mainp() | |
| 1645 ' ks_%d-%d.cdb 2 4 12 24 > ../cdx-00102 2> /tmp/hst/twentyfour | |
| 1646 [11 minutes] | |
| 1647 Sigh | |
| 1648 Compute node: | |
| 1649 >: time { uz /beegfs/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00103.gz|python3 -c 'import test_cdbp | |
| 1650 test_cdbp.mainp() | |
| 1651 ' ks_%d-%d.cdb 0 2 0 12 |python3 -c 'import test_cdbp | |
| 1652 test_cdbp.mainp() | |
| 1653 ' ks_%d-%d.cdb 2 4 12 24 |python3 -c 'import test_cdbp | |
| 1654 test_cdbp.mainp() | |
| 1655 ' ks_%d-%d.cdb 4 6 24 36 ; } > ../cdx-00103 2> /tmp/hst/36 & | |
| 1656 Ah, maybe I just hadn't cythonised after last edit... | |
| 1657 OK, yes, working! | |
| 1658 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND | |
| 1659 3675590 hst 20 0 3965368 519288 506468 R 86.9 0.2 0:28.18 python3 | |
| 1660 3675589 hst 20 0 3922944 507340 502332 S 84.3 0.2 0:25.53 python3 | |
| 1661 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp | |
| 1662 test_cdbp.mainp() | |
| 1663 ' ks_%d-%d.cdb 0 1 0 6 | python3 -c 'import test_cdbp | |
| 1664 test_cdbp.mainp() | |
| 1665 ' ks_%d-%d.cdb 1 2 6 12 > ../cdx-00101 | |
| 1666 14681147 entries, 111963 given lastmod | |
| 1667 14681147 entries, 109503 given lastmod | |
| 1668 | |
| 1669 real 0m57.837s | |
| 1670 user 1m23.062s | |
| 1671 sys 0m26.550s | |
| 1672 But with three chained it slowed a bit, top never showed more than 40% | |
| 1673 for the first in the chain: | |
| 1674 >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp | |
| 1675 test_cdbp.mainp() | |
| 1676 ' ks_%d-%d.cdb 0 1 0 6 | python3 -c 'import test_cdbp | |
| 1677 test_cdbp.mainp() | |
| 1678 ' ks_%d-%d.cdb 1 2 6 12 | python3 -c 'import test_cdbp;test_cdbp.mainp() | |
| 1679 ' ks_%d-%d.cdb 2 3 12 18 > ../cdx-00101 > ../cdx-00101 | |
| 1680 14681147 entries, 111963 given lastmod | |
| 1681 14681147 entries, 109503 given lastmod | |
| 1682 14681147 entries, 111299 given lastmod | |
| 1683 | |
| 1684 real 2m39.481s | |
| 1685 user 2m8.402s | |
| 1686 sys 0m33.503s | |
| 1687 Try compute node again... | |
| 1688 Same behaviour -- 3 is bad, 2 is OK | |
| 1689 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND | |
| 1690 2163622 hst 20 0 3961280 291836 281560 S 10.9 0.1 0:15.66 python3 | |
| 1691 2163621 hst 20 0 3922948 289100 278796 I 10.6 0.1 0:15.38 python3 | |
| 1692 2163623 hst 20 0 3945880 306352 289356 S 10.3 0.1 0:15.50 python3 | |
|
71
6935ebce43e0
can't seem to give up on cdb...
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
70
diff
changeset
|
1693 |
|
72
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1694 OK, after many false starts, it's fairly simple: if a cdb file is |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1695 _not_ in the local cache for whereever /work is coming from, mmap |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1696 only accesses bits in 4MB or 8BM or something else relatively small. |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1697 But wc -l ...cdb a) runs in a few seconds and b) gets the whole thing |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1698 in the cache. So, always do that first, and a full 17-step pipeline |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1699 runs nearly as fast as a single step: |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1700 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1701 >: time uz /beegfs/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz | ~/bin/cdb_chain.sh 17 >/tmp/hst/cdb-00101 2>/tmp/hst/ch.out & |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1702 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1703 real 1m5.802s |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1704 user 1m27.873s |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1705 sys 0m34.349s |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1706 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1707 And it matches the earlier version: |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1708 >: fgrep -c lastmod /tmp/hst/cdb-00101 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1709 1864371 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1710 >: ls -l /tmp/hst/cdb-00101 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1711 -rw-r--r-- 1 hst dc007 7090034893 Mar 6 01:24 /tmp/hst/cdb-00101 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1712 >: uz idx/cdx-00101.gz | fgrep -c lastmod |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1713 1864371 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1714 >: uz idx/cdx-00101.gz | wc |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1715 14681147 314748509 7090034893 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1716 >: uz idx/cdx-00101.gz | cmp - /tmp/hst/cdb-00101 |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1717 >: echo $? |
|
7901ce4a39e3
breakthrough wrt performance of chained cdb approach
Henry Thompson <ht@markup.co.uk>
parents:
71
diff
changeset
|
1718 0 |
|
73
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1719 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1720 OK, now to fill up 2023-40 so we can do a real trial. |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1721 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1722 Cythonised lmh.py and warc.py, still running about 14 seconds per file |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1723 Don't actually need these yet, as 2023-40 has a full set of lmh |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1724 outputs, but not many ks.tsv: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1725 >: ls */ks.tsv |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1726 0/ks.tsv 15/ks.tsv 4/ks.tsv 68/ks.tsv |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1727 12/ks.tsv 36/ks.tsv 56/ks.tsv best_two_by_nl1/ks.tsv |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1728 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1729 Note, that to populate cdbs, we don't need most of the logic in |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1730 sort_date.py, because all the sorting and merging isn't needed! |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1731 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1732 Hmm. In fact, just go straight to cdb_in.txt |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1733 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1734 >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 15 0 /tmp/hst 2>/tmp/hst/15/errs |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1735 /tmp/hst/15/lmh.cdb_in |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1736 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1737 real 223m28.517s |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1738 user 168m34.751s |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1739 sys 12m6.020s |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1740 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1741 This compares to 273 minutes for 806 files in the cleanup of segment |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1742 95, see end of old_notes. Best full segment was 96, i.e. 279 for |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1743 900. And those figures are for lmh.py only |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1744 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1745 Are results the same? |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1746 Old: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1747 >: wc -l *.tsv |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1748 8247 ks_errs.tsv |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1749 5324616 ks.tsv |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1750 New: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1751 >: wc -l * |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1752 9147 errs |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1753 5324617 lmh.cdb_in |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1754 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1755 I _think_ that's good -- there's one line per input file added to the |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1756 error output now, and one blank line at the end of the cdb_in file. |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1757 Old: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1758 >: cut -f 2- ks_errs.tsv |sus |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1759 5420 cannot unpack non-iterable NoneType object |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1760 2670 list index out of range |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1761 113 year 641471 is out of range |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1762 20 year 641474 is out of range |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1763 9 hour must be in 0..23 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1764 9 year 642435 is out of range |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1765 3 year 642437 is out of range |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1766 1 year 4262575 is out of range |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1767 1 year 641769 is out of range |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1768 1 year 642434 is out of range |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1769 New: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1770 >: cut -f 2- <(fgrep -v beegfs /tmp/hst/15/errs) | cut -f 1-5 -d ' ' |sus |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1771 8090 Invalid date value or format |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1772 113 year 641471 is out of |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1773 20 year 641474 is out of |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1774 9 hour must be in 0..23 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1775 9 year 642435 is out of |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1776 3 year 642437 is out of |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1777 1 year 4262575 is out of |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1778 1 year 641769 is out of |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1779 1 year 642434 is out of |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1780 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1781 But, failed to fix the case where the URI in the original WARC is not |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1782 %-encoded: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1783 >: diff -bw <(cut -f2- ks.tsv| sed 's/ //;s/\.0$//'|sort -k1,1) <(sed 's/^\+[0-9,]*://;s/->\([0-9]*\)$/ \1/' /tmp/hst/15/lmh.cdb_in_nopercent | sort -k1,1)|head |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1784 4360c4361 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1785 < 20230922034234https://www.hansaton.com/zh-hk/\u6D88\u8CBB\u8005.html 1695354153 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1786 --- |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1787 > 20230922034234https://www.hansaton.com/zh-hk/消費者.html 1695354153 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1788 6574c6575 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1789 < 20230922034248https://blog.arabtherapy.com/\u0639\u0644\u0627\u062C-\u0627\u0644\u062E\u062C\u0644-\u0627\u0644\u0627\u062C\u062A\u0645\u0627\u0639\u064A/ 1695354168 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1790 --- |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1791 > 20230922034248https://blog.arabtherapy.com/علاج-الخجل-الاجتماعي/ 1695354168 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1792 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1793 Could try fixing that at lookup time |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1794 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1795 Fixed in warc2cdb, seems OK for timing: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1796 /tmp/hst/15/lmh.cdb_in |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1797 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1798 real 224m32.815s |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1799 user 169m25.216s |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1800 sys 12m7.867s |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1801 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1802 Lengths are right, difference with ks looks better, try the real |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1803 thing: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1804 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1805 >: cd /beegfs/common_crawl/CC-MAIN-2023-40/cdx/warc |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1806 >: ls | parallel -j 10 "uz '{}' |cut -f 2,4,18 -d ' '|fgrep .15/warc/CC-MAIN | cut -f 1,2 -d ' ' | sort > '/tmp/hst/15/orig_{#}.cdx'" & |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1807 >: sort -m *.cdx | sponge all.cdx |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1808 |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1809 Nope: |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1810 >: diff <(sed 's/->[0-9]*$/",/' test.cdb_in) all.cdx |less |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1811 >: fgrep -n dryelf.com test.cdb_in all.cdx |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1812 test.cdb_in:6710:20230922034248 "http://www.dryelf.com/files/DryCookieInstall.exe", |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1813 test.cdb_in:1600193:20230922043436 "http://www.dryelf.com/files/DryLottoInstall.exe", |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1814 all.cdx:12190045:20230922044250 "http://www.dryelf.com/downloads.asp?Sort=0&Cat=3", |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1815 all.cdx:25401353:20230922053834 "http://www.dryelf.com/downloads.asp?Sort=2&Cat=0", |
|
1283a574260d
working on cross-check of warc2cdb
Henry Thompson <ht@markup.co.uk>
parents:
72
diff
changeset
|
1816 |
|
74
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1817 Stupid -- mime-detected sometimes has a space in it... |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1818 |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1819 >: ls *.gz | parallel -j 10 "uz '{}' |fgrep .15/warc/CC-MAIN | cut -f 2,4 -d ' ' | sort > '/tmp/hst/15/orig_{#}.cdx'" |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1820 >: sort -m *.cdx | sponge all.cdx |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1821 >: wc -l all.cdx |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1822 34742533 |
|
78
45b45810aa12
cleaned up/filled in after merge
Henry Thompson <ht@markup.co.uk>
parents:
76
diff
changeset
|
1823 >: sed -E 's/^\+[0-9,]*:(.{14,14})/\1 "/;s/->[0-9]{10,10}$/",/' /tmp/hst/15/lmh.cdb_in | sort -k1,1 > test.cdb_in |
|
45b45810aa12
cleaned up/filled in after merge
Henry Thompson <ht@markup.co.uk>
parents:
76
diff
changeset
|
1824 |
|
74
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1825 Diff looks plausible, and a few known cases are correct now: |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1826 >: fgrep -n '20230922034234 "https://www.hansaton.com/zh-hk/' test.cdb_in |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1827 4361:20230922034234 "https://www.hansaton.com/zh-hk/%E6%B6%88%E8%B2%BB%E8%80%85.html", |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1828 >: fgrep -n '20230922034234 "https://www.hansaton.com/zh-hk/' all.cdx |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1829 27516:20230922034234 "https://www.hansaton.com/zh-hk/%E6%B6%88%E8%B2%BB%E8%80%85.html", |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1830 >: fgrep dryelf.com test.cdb_in all.cdx |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1831 test.cdb_in:20230922034248 "http://www.dryelf.com/files/DryCookieInstall.exe", |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1832 test.cdb_in:20230922043436 "http://www.dryelf.com/files/DryLottoInstall.exe", |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1833 all.cdx:20230922034248 "http://www.dryelf.com/files/DryCookieInstall.exe", |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1834 all.cdx:20230922043436 "http://www.dryelf.com/files/DryLottoInstall.exe", |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1835 all.cdx:20230922044250 "http://www.dryelf.com/downloads.asp?Sort=0&Cat=3", |
|
ff6ef190f901
cross-check wrt seg 15 of 2023-40
Henry Thompson <ht@markup.co.uk>
parents:
73
diff
changeset
|
1836 all.cdx:20230922053834 "http://www.dryelf.com/downloads.asp?Sort=2&Cat=0", |
| 76 | 1837 |
|
79
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1838 OK, rerun in the right place: |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1839 >: cd ~/results/CC-MAIN-2023-40/warc_lmhx |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1840 >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 15 . 2> 15/w2c_errs &/work/dc007/dc007/hst/lib/python/cc:/work/dc007/dc007/hst/lib/python/cc/lmh |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1841 |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1842 real 210m54.183s |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1843 user 152m34.172s |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1844 sys 12m25.122s |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1845 |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1846 Ran a test of 5 jobs on a single compute node, did 78 files in 18 |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1847 minutes. Try 15, for grins |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1848 >: export PYTHONPATH=/work/dc007/dc007/hst/lib/python/cc:/work/dc007/dc007/hst/lib/python/cc/lmh |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1849 >: cd $W/results/CC-MAIN-2023-40/warc_lmhx |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1850 >: { date ; seq 0 14 | parallel "python3 -c 'import sys,warc2cdb; |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1851 sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 '{}' . 2> '{}'/w2c_errs" ; date ; } & |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1852 Thu Mar 20 12:05:50 AM GMT 2025 |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1853 After an hour, we're only at about 160 per process, and the job only |
|
0099d4269428
trying 15 down to 8 at a time
Henry Thompson <ht@markup.co.uk>
parents:
78
diff
changeset
|
1854 has 3:30 to run, so I'm going to kill 8-14. |
|
80
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1855 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1856 0-7 didn't finish: |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1857 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1858 reran the who |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1859 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1860 >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 0 . 2> 0/w2c_errsle of 0 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1861 ./0/lmh.cdb_in |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1862 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1863 real 224m28.784s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1864 user 169m21.796s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1865 sys 12m25.583s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1866 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1867 Update warc2cdb to accept a rangm patter, use it to try getting the |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1868 rest of segment 1: |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1869 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1870 >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 1 . '{685..899}' 2>> 1/w2c_errs & |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1871 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1872 Oops, complete fail, forgot that the earlier version used glob.glob |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1873 which does _not_ produce a sorted list of files, so need to refetch |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1874 all of 1 as well: |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1875 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1876 >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 1 . 2> 1/w2c_errs |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1877 ./1/lmh.cdb_in |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1878 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1879 real 210m56.444s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1880 user 154m29.608s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1881 sys 11m49.656s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1882 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1883 Here and subsequently, explicity sort the input file list. |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1884 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1885 Compute the missing bits, use to fill the gaps: |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1886 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1887 >: fgrep beegfs 2/w2c_errs |cut -f 8 -d / | tr -d \' | cut -f 5 -d - | cut -f 1 -d .|sort | diff - <(seq -f '%05g' 0 899)|~/lib/python/cc/lmh/hack.py > /tmp/hst/2.pat |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1888 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1889 >: echo python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 2 surplus $(cat /tmp/hst/2.pat) 2> surplus/2/w2c_errs |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1890 time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 2 surplus '{017,024,027,031,{034..035},038,040,043,046,{051..052},059,061,064,084,092,103,105,{108..109},111,114,117,119,{122..123},135,137,139,144,{146..147},150,153,162,186,195,197,206,208,211,{224..225},227,243,251,253,{264..265},269,285,{292..293},296,304,{307..308},{316..317},328,337,341,344,{360..361},373,380,{386..387},392,397,406,{418..419},426,439,445,454,456,459,462,465,467,473,480,{482..483},{486..487},489,{491..492},499,501,506,512,521,523,528,540,{545..546},550,555,562,568,573,{577..578},584,586,588,593,598,{607..608},621,625,{628..629},637,639,{645..647},649,{656..657},659,662,666,669,671,673,{678..679},683,{688..689},{693..694},702,704,706,{709..710},714,730,{740..741},{745..747},{750..751},754,758,763,767,771,{785..787},791,{794..795},800,804,809,{824..825},831,833,{841..842},846,{854..855},862,864,{875..878},880,883,890,{894..896}}' 2> surplus/2/w2c_errs & |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1891 surplus/2/lmh.cdb_in |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1892 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1893 real 47m35.809s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1894 user 35m10.748s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1895 sys 2m54.211s |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1896 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1897 >: fgrep -h beegfs {,surplus/}2/w2c_errs | wc -l |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1898 900 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1899 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1900 Checked that the last 8 of the ones that _were_ logged from the |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1901 original aborted run of segment 2 are complete: |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1902 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1903 >: fgrep beegfs 2/w2c_errs |tail -8 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1904 /beegfs/common_crawl/CC-MAIN-2023-40/...-00030.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1905 /beegfs/common_crawl/CC-MAIN-2023-40/...-00756.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1906 /beegfs/common_crawl/CC-MAIN-2023-40/...-00249.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1907 /beegfs/common_crawl/CC-MAIN-2023-40/...-00301.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1908 /beegfs/common_crawl/CC-MAIN-2023-40/...-00158.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1909 /beegfs/common_crawl/CC-MAIN-2023-40/...-00009.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1910 /beegfs/common_crawl/CC-MAIN-2023-40/...-00294.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1911 /beegfs/common_crawl/CC-MAIN-2023-40/...-00576.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1912 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1913 >: python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 2 surplus '{030,756,249,301,158,009,294,576}' 2> surplus/2x/w2c_errs |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1914 >: fgrep -n beegfs 2/w2c_errs |tail -8 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1915 5765:/beegfs/common_crawl/CC-MAIN-2023-40/1695233506658.2/orig/warc/CC-MAIN-20230924155422-20230924185422-00030.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1916 ... |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1917 5821:/beegfs/common_crawl/CC-MAIN-2023-40/1695233506658.2/orig/warc/CC-MAIN-20230924155422-20230924185422-00576.warc.gz |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1918 >: wc -l 2/w2c_errs |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1919 5839 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1920 >: echo $((5839 - 5765)) |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1921 74 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1922 sing<4424>: tail -75 2/w2c_errs|sort |diff -bw - <(sort surplus/2x/w2c_errs) |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1923 [nothing] |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1924 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1925 Something is wrong? |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1926 >: wc -l 0/lmh.cdb_in |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1927 6414381 0/lmh.cdb_in |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1928 >: seq 0 8 | parallel -j 9 'for i in {00..99}; do uz 0/*00{}$i.warc.gz | egrep -c " .* "; done | btot | tee /dev/stderr' | btot |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1929 713980 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1930 714754 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1931 712334 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1932 712640 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1933 713943 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1934 714063 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1935 712428 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1936 712630 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1937 711997 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1938 6418769 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1939 >: echo $((6418769 - 6414381)) |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1940 4388 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1941 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1942 >: seq 0 8 | parallel -j 9 'for i in {00..99}; do uz 15/*00{}$i.warc.gz | egrep -c " .* "; done | btot | tee /dev/stderr' | btot |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1943 591136 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1944 591189 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1945 591105 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1946 593544 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1947 593214 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1948 594861 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1949 592556 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1950 592249 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1951 593009 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1952 5332863 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1953 >: echo $((5332863 - 5324617)) |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1954 8246 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1955 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1956 Ah, maybe OK: |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1957 wc -l < 0/ks_errs.tsv |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1958 4389 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1959 >: cut -f 2 0/ks_errs.tsv | sus |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1960 3188 cannot unpack non-iterable NoneType object |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1961 1193 list index out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1962 1 year 4263994 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1963 1 year 4278897 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1964 1 year 4355304 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1965 1 year 4361928 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1966 1 year 4370444 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1967 1 year 4430056 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1968 1 year 4433368 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1969 1 year 53310 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1970 >: cut -f 2 <(sed 's/value or format .*/value or format/' w2c_errs)|fgrep -v warc.gz| sus |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1971 4381 Invalid date value or format |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1972 1 year 4263994 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1973 1 year 4278897 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1974 1 year 4355304 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1975 1 year 4361928 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1976 1 year 4370444 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1977 1 year 4430056 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1978 1 year 4433368 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1979 1 year 53310 is out of range |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1980 |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1981 And last line of lmh.cdb_in is a blank EOF indicator |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1982 So |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1983 >: echo $((6418769 - 6414380)) |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1984 4389 |
| 81 | 1985 >: fgrep -v warc.gz w2c_errs |wc -l |
| 1986 4389 | |
|
80
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1987 So lengths of lmh.cdb_in match, given that ks.tsv has string dates and |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1988 lmh.cdb_in has dates successfully converted to numeric timestamps, and |
|
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
1989 non-conversion counts match. |
| 81 | 1990 |
| 1991 Same for 15... | |
| 1992 >: cut -f 2 ks_errs.tsv | sus | |
| 1993 5420 cannot unpack non-iterable NoneType object | |
| 1994 2670 list index out of range | |
| 1995 113 year 641471 is out of range | |
| 1996 20 year 641474 is out of range | |
| 1997 9 hour must be in 0..23 | |
| 1998 9 year 642435 is out of range | |
| 1999 3 year 642437 is out of range | |
| 2000 1 year 4262575 is out of range | |
| 2001 1 year 641769 is out of range | |
| 2002 1 year 642434 is out of range | |
| 2003 >: cut -f 2 <(sed 's/value or format .*/value or format/' w2c_errs)|fgrep -v warc.gz| sus | |
| 2004 8090 Invalid date value or format | |
| 2005 113 year 641471 is out of range | |
| 2006 20 year 641474 is out of range | |
| 2007 9 hour must be in 0..23 | |
| 2008 9 year 642435 is out of range | |
| 2009 3 year 642437 is out of range | |
| 2010 1 year 4262575 is out of range | |
| 2011 1 year 641769 is out of range | |
| 2012 1 year 642434 is out of range | |
| 2013 >: wc -l ks_errs.tsv | |
| 2014 8247 ks_errs.tsv | |
|
80
02fb801ac3c1
2023-40 gap-filling wrt segments 0-2 and cross-validation
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
79
diff
changeset
|
2015 >: fgrep -v warc.gz w2c_errs |wc -l |
| 81 | 2016 8247 |
| 2017 >: wc -l< lmh.cdb_in | |
| 2018 5324617 | |
| 2019 >: seq 0 8 | parallel -j 9 'for i in {00..99}; do uz *00{}$i.warc.gz | fgrep -c " "; done | btot | tee /dev/stderr' | btot | |
| 2020 593009 | |
| 2021 591105 | |
| 2022 593544 | |
| 2023 591136 | |
| 2024 591189 | |
| 2025 594861 | |
| 2026 593214 | |
| 2027 592556 | |
| 2028 592249 | |
| 2029 5332863 | |
| 2030 >: echo $((5332863 - 5324616)) | |
| 2031 8247 | |
| 2032 | |
| 2033 And for 1 | |
| 2034 | |
| 2035 >: seq 0 8 | parallel -j 9 'for i in {00..99}; do uz *00{}$i.warc.gz | fgrep -c " "; done | btot | tee /dev/stderr' | btot | |
| 2036 685262 | |
| 2037 686242 | |
| 2038 684822 | |
| 2039 684268 | |
| 2040 684967 | |
| 2041 684771 | |
| 2042 684520 | |
| 2043 685418 | |
| 2044 686420 | |
| 2045 6166690 | |
| 2046 >: wc -l < lmh.cdb_in | |
| 2047 6160222 | |
| 2048 >: echo $((6166690 - 6160221)) | |
| 2049 6469 | |
| 2050 >: fgrep -v warc.gz w2c_errs |wc -l | |
| 2051 6469 | |
| 2052 | |
| 2053 And for 2? | |
| 2054 | |
| 2055 >: wc -l {,surplus/}2/lmh.cdb_in | |
| 2056 4670389 2/lmh.cdb_in | |
| 2057 52530 surplus/2/lmh.cdb_in | |
| 2058 4722919 total | |
| 2059 sing<4564>: seq 0 8 | parallel -j 9 'for i in {00..99}; do uz 2/*00{}$i.warc.gz | fgrep -c " "; done | btot | tee /dev/stderr' | btot | |
| 2060 659931 | |
| 2061 658839 | |
| 2062 659020 | |
| 2063 660117 | |
| 2064 659896 | |
| 2065 659771 | |
| 2066 659374 | |
| 2067 660426 | |
| 2068 660040 | |
| 2069 5937414 | |
| 2070 | |
| 2071 So surplus/2 is short? | |
| 2072 | |
| 2073 Maybe, but _2_ is short??? | |
| 2074 | |
| 2075 >: fgrep beegfs 2/w2c_errs |cut -f 8 -d / | tr -d \' | cut -f 5 -d - | cut -f 1 -d .|sort | parallel -j 9 'uz 2/*{}.warc.gz | fgrep -c " " | tee /dev/stderr' | btot | |
| 2076 4678728 | |
| 2077 >: echo $((4678728 - 4670388)) | |
| 2078 8340 | |
| 2079 >: fgrep -vh warc.gz 2/w2c_errs |wc -l | |
| 2080 5130 | |
| 2081 | |
| 2082 And so is 4??? | |
| 2083 >: fgrep beegfs 4/w2c_errs |cut -f 8 -d / | tr -d \' | cut -f 5 -d - | cut -f 1 -d .|wc -l | |
| 2084 716 | |
| 2085 >: fgrep beegfs 4/w2c_errs |cut -f 8 -d / | tr -d \' | cut -f 5 -d - | cut -f 1 -d .|sort | parallel -j 9 'uz 4/*{}.warc.gz | fgrep -c " "' | btot | |
| 2086 5003586 | |
| 2087 sing<4577>: fgrep -vh warc.gz 4/w2c_errs |wc -l | |
| 2088 4890 | |
| 2089 sing<4578>: wc -l < 4/lmh.cdb_in | |
| 2090 4975964 | |
| 2091 sing<4579>: echo $((5003586 - 4975963)) | |
| 2092 27623 | |
| 2093 | |
| 2094 Mystery. Just have to redo 2--14. | |
| 2095 Cirrus slurm doesn't allow forcing more than one node, e.g. with | |
| 2096 --nodes=2-7, says you should use --exclusive :-( | |
| 2097 | |
| 2098 So, just force it by hand | |
| 2099 | |
| 2100 >: sbatch --output=slurm_w2c_2-7.out --time=04:00:00 --ntasks=7 -c 3 $HOME/bin/runme.sh $PWD -t 3 -i 'share_by_task.sh -s 2 7 $n $task' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh | |
| 2101 echo $PYTHONPATH | |
| 2102 mkdir -p rerun/$arg | |
| 2103 python3 -c "import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))" 2023-40 $arg ./rerun 2> rerun/$arg/w2c_errs' | |
| 2104 | |
| 2105 >: sbatch --output=slurm_w2c_8-14.out --time=04:00:00 --ntasks=7 -c 3 $HOME/bin/runme.sh $PWD -t 3 -i 'share_by_task.sh -s 8 14 $n $task' 'export PYTHONPATH=$W/hst/lib/python/cc:$W/hst/lib/python/cc/lmh | |
| 2106 echo $PYTHONPATH | |
| 2107 mkdir -p rerun/$arg | |
| 2108 python3 -c "import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))" 2023-40 $arg ./rerun 2> rerun/$arg/w2c_errs' | |
| 2109 | |
| 2110 >: python3 -c "print((900.0/(626/6))*35)" | |
| 2111 301.91693290734827 | |
| 2112 | |
| 2113 Going too slowly, killed the 2nd one after 39 minutes with less than | |
| 2114 110 files processed by the first in 35 minutes, hope it might now just | |
| 2115 squeak in under 240 minutes ... | |
| 2116 | |
| 2117 >: scancel --signal=HUP 7330184 | |
| 2118 >: python3 -c "print((900.0/(925/6))*45.5)" | |
| 2119 265.62162162162167 | |
| 2120 | |
| 2121 >: python3 -c "print((900.0/(1058/6))*52.25)" | |
| 2122 266.68241965973533 | |
| 2123 | |
| 2124 Nope: | |
| 2125 >: fgrep -ch beegfs rerun/{2..7}/w2c_errs | |
| 2126 680 | |
| 2127 804 | |
| 2128 825 | |
| 2129 816 | |
| 2130 825 | |
| 2131 809 | |
| 2132 >: cd rerun | |
| 2133 sing<4601>: for i in {2..7}; do fgrep beegfs $i/w2c_errs |cut -f 8 -d / | tr -d \' | cut -f 5 -d - | cut -f 1 -d .|sort | diff - <(seq -f '%05g' 0 899)|~/lib/python/cc/lmh/hack.py > $i.pat ; done | |
| 2134 sing<4602>: less 2.pat | |
| 2135 sing<4603>: fgrep -c warc.gz 4/w2c_errs | |
| 2136 825 | |
| 2137 sing<4604>: cat 2.pat | |
| 2138 {{680..899}} | |
| 2139 | |
| 2140 less than 10% speed-up after cancelling the 2nd job: | |
| 2141 >: python3 -c "print((900.0/(4759/6))*240)" | |
| 2142 272.32611893254887 | |
| 2143 | |
| 2144 Rerun 4 separately to have one last go at debugging the problem with | |
| 2145 partial runs: | |
| 2146 | |
| 2147 >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 4 ./surplus 2> surplus/4/w2c_errs & | |
| 2148 ./surplus/4/lmh.cdb_in | |
| 2149 | |
| 2150 real 229m55.344s | |
| 2151 user 164m5.765s | |
| 2152 sys 11m24.657s | |
| 2153 | |
| 2154 Is 4/rerun/w2c_errs the same as surplus/4/w2c_errs through file 00824? | |
| 2155 Yes: | |
| 2156 >: fgrep -c beegfs {rerun,surplus}/4/w2c_errs | |
| 2157 rerun/4/w2c_errs:825 | |
| 2158 surplus/4/w2c_errs:900 | |
| 2159 >: diff -bw <(fgrep beegfs rerun/4/w2c_errs) <(fgrep beegfs surplus/4/w2c_errs)|head -2 | |
| 2160 825a826,900 | |
| 2161 > /beegfs/common_crawl/CC-MAIN-2023-40/1695233511021.4/orig/warc/CC-MAIN-20231002200740-20231002230740-00825.warc.gz | |
| 2162 >: diff -bw <(fgrep beegfs rerun/4/w2c_errs) <(fgrep beegfs surplus/4/w2c_errs)|tail -1 | |
| 2163 > /beegfs/common_crawl/CC-MAIN-2023-40/1695233511021.4/orig/warc/CC-MAIN-20231002200740-20231002230740-00899.warc.gz | |
| 58 | 2164 ================ |
| 2165 | |
| 48 | 2166 Try it with the existing _per segment_ index we have for 2019-35 |
| 2167 | |
| 58 | 2168 Assuming we have to key on segment / file and offset, as reconstructing the |
| 48 | 2169 proper index key is such a pain / buggy / is going to change with the year. |
| 2170 | |
| 2171 Stay with segment 49 | |
| 2172 | |
| 2173 >: uz cdx.gz |wc -l | |
| 2174 29,870,307 | |
| 2175 | |
| 2176 >: time uz cdx.gz|egrep -ao ' "length": "[0-9]*", "offset": "[0-9]*"' |wc | |
| 2177 29,870,307 119,481,228 1,241,098,122 | |
| 2178 = 4 * 29,870,307 | |
| 2179 | |
| 2180 So no bogons, not _too_ surprising :-) | |
| 2181 | |
| 2182 Bad news is it's a _big_ file: | |
| 2183 | |
| 2184 >: ls -lh cdx.gz | |
| 2185 -rw-r--r-- 1 hst dc007 2.0G Mar 18 2021 cdx.gz | |
| 2186 | |
| 2187 So not viable to paste offset as a key and then sort on command line, | |
| 2188 or to load it in to python and do the work there... | |
| 2189 | |
| 2190 Do it per warc file and then merge? | |
| 2191 | |
| 2192 >: time uz cdx.gz |fgrep -a warc/CC-MAIN-20190819011034-20190819033034-00558.warc.gz | sort -n -t\" -k28,28 > /tmp/hst/558.warc.cdx | |
| 2193 | |
| 2194 real 0m23.494s | |
| 2195 user 0m14.541s | |
| 2196 sys 0m9.158s | |
| 2197 | |
| 2198 >: wc -l /tmp/hst/558.warc.cdx | |
| 2199 53432 /tmp/hst/558.warc.cdx | |
| 2200 | |
|
49
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2201 >: echo $((600 * 53432)) |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2202 32,059,200 |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2203 |
| 48 | 2204 So, 600 of those, plus approx. same again for extracting, that pbly |
| 2205 _is_ doable in python, not more than 10 hours total, assuming internal | |
| 2206 sort and external merge is not too expensive... | |
|
49
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2207 |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2208 For each segment, suppose we pull out 60 groups of 10 target files |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2209 >: time uz cdx.gz |egrep -a warc/CC-MAIN-2019[^-]*-2019[^-]*-0000..warc.gz > /tmp/hst/0000.warc.cdx |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2210 |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2211 real 0m42.129s |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2212 user 0m35.147s |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2213 sys 0m9.140s |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2214 >: wc -l /tmp/hst/0000.warc.cdx |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2215 533150 |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2216 |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2217 Key it with offset and sort: |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2218 |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2219 >: time egrep -ao ' "length": "[0-9]*", "offset": "[0-9]*"' /tmp/hst/0000.warc.cdx | cut -f 5 -d ' ' | tr -d \" > /tmp/hst/0000_offsets |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2220 |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2221 real 0m5.578s |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2222 user 0m5.593s |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2223 sys 0m0.265s |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2224 |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2225 >: time paste /tmp/hst/0000_offsets /tmp/hst/0000.warc.cdx |sort -nk1,1 | cut -f 2 > /tmp/hst/0000_sorted.warc.cdx |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2226 |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2227 real 0m4.185s |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2228 user 0m2.001s |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2229 sys 0m1.334s |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2230 |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2231 >: time seq 0 9 | parallel -j 10 "~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-*-*-0000'{}'.warc.gz > /tmp/hst/lm_0000'{}'.tsv" |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2232 |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2233 real 0m24.610s |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2234 user 2m54.146s |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2235 sys 0m10.226s |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2236 |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2237 >: head /tmp/hst/lm_00000.tsv |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2238 9398 16432 Mon, 19 Aug 2019 02:44:15 GMT |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2239 20796 26748 Tue, 16 Jul 2019 04:39:09 GMT |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2240 4648 340633 Fri, 07 Dec 2018 09:05:59 GMT |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2241 3465 357109 Sun, 18 Aug 2019 11:48:23 GMT |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2242 7450 914189 Mon, 19 Aug 2019 02:50:08 GMT |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2243 ... |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2244 sing<3956>: fgrep '"length": "9398", "offset": "16432"' /tmp/hst/0000_sorted.warc.cdx |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2245 com,roommeme,0401a)/index.phtml?channel=&op=&p=140&put=show&r2= 20190819024416 {"url": "http://0401a.roommeme.com/index.phtml?PUT=SHOW&R2=&OP=&P=140&CHANNEL=", "mime": "text/html", "mime-detected": "text/html", "status": "200", "digest": "5DNDVX5HQBOOBHISSCOI4UBVMUL63L36", "length": "9398", "offset": "16432", "filename": "crawl-data/CC-MAIN-2019-35/segments/1566027314638.49/warc/CC-MAIN-20190819011034-20190819033034-00000.warc.gz", "charset": "Big5", "languages": "zho"} |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2246 |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2247 bingo |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2248 |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2249 So, the python code is pretty straightfoward: open the 10 individual |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2250 lm-*.tsv outputs into an array, initialise a 10-elt array with the |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2251 first line of each and another with its offset, record the |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2252 fileno(s) of the lowest offset, then iterate |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2253 |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2254 read cdx lines and write unchanged until offset = lowest |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2255 merge line from fileno and output |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2256 remove fileno from list of matches |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2257 read and store a new line for fileno [handle EOF] |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2258 if list of matches is empty, redo setting of lowest |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2259 |
|
deeac8a0a682
tentative plan for merging
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
48
diff
changeset
|
2260 Resort the result by actual key |
| 50 | 2261 |
| 2262 Meanwhile, get a whole test set: | |
| 2263 sbatch --output=slurm_aug_cdx_49_10-599-out --time=01:00:00 --ntasks=10 -c 36 --exclusive $HOME/bin/runme.sh -m 00 59 $PWD -t 18 -b 'export resdir=CC-MAIN-2019-35/aug_cdx/49 | |
| 2264 export DEC=$xarg' "export PYTHONPATH=./lib/python/cc:$PYTHONPATH | |
| 2265 seq 0 9 | parallel -j 10 \"~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-*-*-00\${DEC}'{}'.warc.gz > \$resdir/00\${DEC}'{}'.tsv\"" | |
| 2266 | |
| 2267 Actually finished 360 in the hour. | |
| 2268 | |
| 2269 Leaving | |
| 2270 | |
| 2271 sbatch --output=slurm_aug_cdx_49_360-599-out --time=01:00:00 --ntasks=10 -c 36 --exclusive $HOME/bin/runme.sh -m 36 59 $PWD -t 18 -b 'export resdir=CC-MAIN-2019-35/aug_cdx/49 | |
| 2272 export DEC=$xarg' "export PYTHONPATH=./lib/python/cc:$PYTHONPATH | |
| 2273 seq 0 9 | parallel -j 10 \"~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/1566027314638.49/orig/warc/CC-MAIN-*-*-00\${DEC}'{}'.warc.gz > \$resdir/00\${DEC}'{}'.tsv\"" | |
| 2274 | |
| 2275 But something is wrong, the number of jobs is all wrong: | |
| 2276 | |
| 2277 5>: fgrep -c parallel slurm_aug_cdx_49_0-359-out | |
| 2278 741 | |
| 2279 sing<4046>: ls -lt CC-MAIN-2019-35/aug_cdx/49/|wc -l | |
| 2280 372 | |
| 2281 | |
| 2282 Every file is being produced twice. | |
|
51
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2283 |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2284 Took me a while to figure out my own code :-( |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2285 |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2286 >: sbatch --output=slurm_aug_cdx_49_360-599-out --time=01:00:00 --ntasks=10 -c 36 --exclusive $HOME/bin/runme.sh -m 49 49 $PWD -t 18 -b 'export resdir=CC-MAIN-2019-35/aug_cdx/$xarg |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2287 export SEG=$xarg |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2288 share_by_task.sh -f "%03g\n" -s 360 599 $n $task > /tmp/hst_$task' -i 'cat /tmp/hst_$task' 'export PYTHONPATH=./lib/python/cc:$PYTHONPATH |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2289 ~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/*.$SEG/orig/warc/CC-MAIN-*-*-00${arg}.warc.gz > $resdir/00${arg}.tsv' |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2290 |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2291 Oops, only 560, not 600 |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2292 |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2293 Took 3.5 minutes for 200, so call it 10 for 560, so do 6 more in an |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2294 hour: |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2295 |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2296 >: sbatch --output=slurm_aug_cdx_50-55_out --time=01:00:00 --ntasks=10 -c 36 --exclusive $HOME/bin/runme.sh -m 50 55 $PWD -t 18 -b 'export resdir=CC-MAIN-2019-35/aug_cdx/$xarg |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2297 mkdir -p $resdir |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2298 > export SEG=$xarg |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2299 share_by_task.sh -f "%03g\n" -s 360 599 $n $task > /tmp/hst_$task' -i 'cat /tmp/hst_$task' 'export PYTHONPATH=./lib/python/cc:$PYTHONPATH |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2300 ~/lib/python/cc/cdx_extras.py /beegfs/common_crawl/CC-MAIN-2019-35/*.$SEG/orig/warc/CC-MAIN-*-*-00${arg}.warc.gz > $resdir/00${arg}.tsv' |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2301 |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2302 >: tail slurm_aug_cdx_50-55_out |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2303 ... |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2304 Wed Oct 9 22:25:47 BST 2024 Finished 55 |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2305 >: head -1 slurm_aug_cdx_50-55_out |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2306 Wed Oct 9 21:29:43 BST |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2307 56:04 |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2308 |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2309 >: du -s CC-MAIN-2019-35/aug_cdx |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2310 1,902,916 |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2311 |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2312 Not bad, so order 20MB for the whole thing |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2313 |
|
dc24bb6e524f
done cdx_aux for segments 49--55 of 2019-35
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
50
diff
changeset
|
2314 Next step, compare to my existing cdx with timestamp |
|
52
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2315 |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2316 First check looks about right: |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2317 |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2318 [cd .../warc_lmhx] |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2319 >: seq --format='%03g' 0 299 > /tmp/hst/cdx_nums |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2320 >: parallel -j 20 -a /tmp/hst/cdx_nums 'uz idx/cdx-00{}.gz | egrep -o "\"filename\": \"crawl-data/CC-MAIN-2019-35/segments/[^.]*[.]50.*\"lastmod\":" | sed "s/^.*-00//;s/^\(...\).*/\1/"| sus > /tmp/hst/checkseg_50_{}' |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2321 |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2322 [cd .../aug_cdx/50] |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2323 >: wc -l 00123.tsv |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2324 9333 |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2325 >: egrep -h '123$' /tmp/hst/checkseg_50_??? | acut 1 | btot |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2326 9300 |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2327 >: wc -l 00400.tsv |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2328 9477 00400.tsv |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2329 >: egrep -h '400$' /tmp/hst/checkseg_50_??? | acut 1 | btot |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2330 9439 |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2331 |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2332 Difference is presumable the bogus timestamps aren't in the augmented |
|
8dffb8aa33da
prelim consistency check with published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
51
diff
changeset
|
2333 cdx as shipped. |
|
53
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2334 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2335 Note that the following 'bad' kind of timestamp is fixed before |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2336 sort_date.py does its thing: |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2337 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2338 ... sort_date.sh <(uz $arg/*00???.warc.gz | '"fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/')"' >$arg/ks.tsv |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2339 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2340 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2341 >: egrep -c '[^ ]GMT$' 50/00123.tsv |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2342 22 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2343 >: egrep -c '[^ ]GMT$' 50/00400.tsv |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2344 14 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2345 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2346 >: PYTHONPATH=~/.local/lib/python3.9/site-packages:$PYTHONPATH sort_date.sh <(uz ../warc_lmhx/50/*00123.warc.gz | fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2> /tmp/hst/123_errs | wc -l |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2347 9300 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2348 >: fgrep -c Invalid /tmp/hst/123_errs |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2349 33 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2350 >: PYTHONPATH=~/.local/lib/python3.9/site-packages:$PYTHONPATH sort_date.sh <(uz ../warc_lmhx/50/*00400.warc.gz | fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2> /tmp/hst/400_errs | wc -l |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2351 9439 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2352 >: fgrep -c Invalid /tmp/hst/400_errs |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2353 38 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2354 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2355 All good. |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2356 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2357 But |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2358 >: seq --format='%03g' 0 559 > /tmp/hst/warc_nums |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2359 >: xx () { |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2360 r=$(diff -bw |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2361 <(echo $(( |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2362 $(sort_date.sh <(uz ../warc_lmhx/50/*00$1.warc.gz | |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2363 fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2>/tmp/hst/ec_$1 |wc -l) |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2364 + |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2365 $(fgrep -c Invalid /tmp/hst/ec_$1)))) |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2366 <(wc -l < 50/00$1.tsv)) |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2367 if [ "$r" ] |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2368 then printf "%s:\n%s\n" $2 "$r" |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2369 fi |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2370 } |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2371 >: parallel -j 20 -a /tmp/hst/warc_nums xx '{}' '$(({#} - 1))' | tee /tmp/hst/aug_bugs |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2372 >: fgrep -c 1c1 /tmp/hst/aug_bugs |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2373 77 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2374 sing<4318>: wc -l < /tmp/hst/aug_bugs |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2375 385 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2376 sing<4319>: echo $((77 * 5)) |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2377 385 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2378 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2379 OK, there are a few other error messages from date conversion |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2380 >: xx () { r=$(diff -bw <(echo $(($(sort_date.sh <(uz ../warc_lmhx/50/*00$1.warc.gz | fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2>/tmp/hst/ec_$1 |wc -l) + $(egrep -c 'Invalid|must be in|out of range' /tmp/hst/ec_$1)))) <(wc -l < 50/00$1.tsv)); if [ "$r" ]; then printf "%s:\n%s\n" $2 "$r"; fi; } |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2381 sing<4337>: parallel -j 20 -a /tmp/hst/warc_nums xx '{}' '$(({#} - 1))' | tee /tmp/hst/aug_bugs2 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2382 [nothing] |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2383 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2384 So, I think we can believe we're OK |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2385 But 7 is better than 1: |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2386 >: xx () { r=$(diff -bw <(echo $(($(sort_date.sh <(uz ../warc_lmhx/$3/*00$1.warc.gz | fgrep $'\t'|sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/') 2>/tmp/hst/ec_$1 |wc -l) + $(egrep -c 'Invalid|must be in|out of range' /tmp/hst/ec_$1)))) <(wc -l < $3/00$1.tsv)); if [ "$r" ]; then printf "%s:\n%s\n" $2 "$r"; fi; } |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2387 >: for s in 49 {51..55}; do parallel -j 20 -a /tmp/hst/warc_nums xx '{}' '$(({#} - 1))' $s | tee /tmp/hst/aug_bugs_$s; done |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2388 [nothing] |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2389 |
|
d533894173d0
detailed consistency check with 7 segments from published lmh-augmented cdx
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
52
diff
changeset
|
2390 Next step: ? |