Mercurial > hg > cc > work

comparison lurid3/notes.txt @ 80:02fb801ac3c1

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
2023-40 gap-filling wrt segments 0-2 and cross-validation
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Tue, 25 Mar 2025 19:48:02 +0000
parents 0099d4269428
children b589ae992146
comparison
equal deleted inserted replaced
79:0099d4269428 80:02fb801ac3c1
1850 >: { date ; seq 0 14 | parallel "python3 -c 'import sys,warc2cdb; 1850 >: { date ; seq 0 14 | parallel "python3 -c 'import sys,warc2cdb;
1851 sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 '{}' . 2> '{}'/w2c_errs" ; date ; } & 1851 sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 '{}' . 2> '{}'/w2c_errs" ; date ; } &
1852 Thu Mar 20 12:05:50 AM GMT 2025 1852 Thu Mar 20 12:05:50 AM GMT 2025
1853 After an hour, we're only at about 160 per process, and the job only 1853 After an hour, we're only at about 160 per process, and the job only
1854 has 3:30 to run, so I'm going to kill 8-14. 1854 has 3:30 to run, so I'm going to kill 8-14.
1855
1856 0-7 didn't finish:
1857
1858 reran the who
1859
1860 >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 0 . 2> 0/w2c_errsle of 0
1861 ./0/lmh.cdb_in
1862
1863 real 224m28.784s
1864 user 169m21.796s
1865 sys 12m25.583s
1866
1867 Update warc2cdb to accept a rangm patter, use it to try getting the
1868 rest of segment 1:
1869
1870 >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 1 . '{685..899}' 2>> 1/w2c_errs &
1871
1872 Oops, complete fail, forgot that the earlier version used glob.glob
1873 which does _not_ produce a sorted list of files, so need to refetch
1874 all of 1 as well:
1875
1876 >: time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 1 . 2> 1/w2c_errs
1877 ./1/lmh.cdb_in
1878
1879 real 210m56.444s
1880 user 154m29.608s
1881 sys 11m49.656s
1882
1883 Here and subsequently, explicity sort the input file list.
1884
1885 Compute the missing bits, use to fill the gaps:
1886
1887 >: fgrep beegfs 2/w2c_errs |cut -f 8 -d / | tr -d \' | cut -f 5 -d - | cut -f 1 -d .|sort | diff - <(seq -f '%05g' 0 899)|~/lib/python/cc/lmh/hack.py > /tmp/hst/2.pat
1888
1889 >: echo python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 2 surplus $(cat /tmp/hst/2.pat) 2> surplus/2/w2c_errs
1890 time python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 2 surplus '{017,024,027,031,{034..035},038,040,043,046,{051..052},059,061,064,084,092,103,105,{108..109},111,114,117,119,{122..123},135,137,139,144,{146..147},150,153,162,186,195,197,206,208,211,{224..225},227,243,251,253,{264..265},269,285,{292..293},296,304,{307..308},{316..317},328,337,341,344,{360..361},373,380,{386..387},392,397,406,{418..419},426,439,445,454,456,459,462,465,467,473,480,{482..483},{486..487},489,{491..492},499,501,506,512,521,523,528,540,{545..546},550,555,562,568,573,{577..578},584,586,588,593,598,{607..608},621,625,{628..629},637,639,{645..647},649,{656..657},659,662,666,669,671,673,{678..679},683,{688..689},{693..694},702,704,706,{709..710},714,730,{740..741},{745..747},{750..751},754,758,763,767,771,{785..787},791,{794..795},800,804,809,{824..825},831,833,{841..842},846,{854..855},862,864,{875..878},880,883,890,{894..896}}' 2> surplus/2/w2c_errs &
1891 surplus/2/lmh.cdb_in
1892
1893 real 47m35.809s
1894 user 35m10.748s
1895 sys 2m54.211s
1896
1897 >: fgrep -h beegfs {,surplus/}2/w2c_errs | wc -l
1898 900
1899
1900 Checked that the last 8 of the ones that _were_ logged from the
1901 original aborted run of segment 2 are complete:
1902
1903 >: fgrep beegfs 2/w2c_errs |tail -8
1904 /beegfs/common_crawl/CC-MAIN-2023-40/...-00030.warc.gz
1905 /beegfs/common_crawl/CC-MAIN-2023-40/...-00756.warc.gz
1906 /beegfs/common_crawl/CC-MAIN-2023-40/...-00249.warc.gz
1907 /beegfs/common_crawl/CC-MAIN-2023-40/...-00301.warc.gz
1908 /beegfs/common_crawl/CC-MAIN-2023-40/...-00158.warc.gz
1909 /beegfs/common_crawl/CC-MAIN-2023-40/...-00009.warc.gz
1910 /beegfs/common_crawl/CC-MAIN-2023-40/...-00294.warc.gz
1911 /beegfs/common_crawl/CC-MAIN-2023-40/...-00576.warc.gz
1912
1913 >: python3 -c 'import sys,warc2cdb; sys.exit(warc2cdb.main(*sys.argv[1:]))' 2023-40 2 surplus '{030,756,249,301,158,009,294,576}' 2> surplus/2x/w2c_errs
1914 >: fgrep -n beegfs 2/w2c_errs |tail -8
1915 5765:/beegfs/common_crawl/CC-MAIN-2023-40/1695233506658.2/orig/warc/CC-MAIN-20230924155422-20230924185422-00030.warc.gz
1916 ...
1917 5821:/beegfs/common_crawl/CC-MAIN-2023-40/1695233506658.2/orig/warc/CC-MAIN-20230924155422-20230924185422-00576.warc.gz
1918 >: wc -l 2/w2c_errs
1919 5839
1920 >: echo $((5839 - 5765))
1921 74
1922 sing<4424>: tail -75 2/w2c_errs|sort |diff -bw - <(sort surplus/2x/w2c_errs)
1923 [nothing]
1924
1925 Something is wrong?
1926 >: wc -l 0/lmh.cdb_in
1927 6414381 0/lmh.cdb_in
1928 >: seq 0 8 | parallel -j 9 'for i in {00..99}; do uz 0/*00{}$i.warc.gz | egrep -c " .* "; done | btot | tee /dev/stderr' | btot
1929 713980
1930 714754
1931 712334
1932 712640
1933 713943
1934 714063
1935 712428
1936 712630
1937 711997
1938 6418769
1939 >: echo $((6418769 - 6414381))
1940 4388
1941
1942 >: seq 0 8 | parallel -j 9 'for i in {00..99}; do uz 15/*00{}$i.warc.gz | egrep -c " .* "; done | btot | tee /dev/stderr' | btot
1943 591136
1944 591189
1945 591105
1946 593544
1947 593214
1948 594861
1949 592556
1950 592249
1951 593009
1952 5332863
1953 >: echo $((5332863 - 5324617))
1954 8246
1955
1956 Ah, maybe OK:
1957 wc -l < 0/ks_errs.tsv
1958 4389
1959 >: cut -f 2 0/ks_errs.tsv | sus
1960 3188 cannot unpack non-iterable NoneType object
1961 1193 list index out of range
1962 1 year 4263994 is out of range
1963 1 year 4278897 is out of range
1964 1 year 4355304 is out of range
1965 1 year 4361928 is out of range
1966 1 year 4370444 is out of range
1967 1 year 4430056 is out of range
1968 1 year 4433368 is out of range
1969 1 year 53310 is out of range
1970 >: cut -f 2 <(sed 's/value or format .*/value or format/' w2c_errs)|fgrep -v warc.gz| sus
1971 4381 Invalid date value or format
1972 1 year 4263994 is out of range
1973 1 year 4278897 is out of range
1974 1 year 4355304 is out of range
1975 1 year 4361928 is out of range
1976 1 year 4370444 is out of range
1977 1 year 4430056 is out of range
1978 1 year 4433368 is out of range
1979 1 year 53310 is out of range
1980
1981 And last line of lmh.cdb_in is a blank EOF indicator
1982 So
1983 >: echo $((6418769 - 6414380))
1984 4389
1985 So lengths of lmh.cdb_in match, given that ks.tsv has string dates and
1986 lmh.cdb_in has dates successfully converted to numeric timestamps, and
1987 non-conversion counts match.
1988 >: fgrep -v warc.gz w2c_errs |wc -l
1989 4389
1855 ================ 1990 ================
1856 1991
1857 Try it with the existing _per segment_ index we have for 2019-35 1992 Try it with the existing _per segment_ index we have for 2019-35
1858 1993
1859 Assuming we have to key on segment / file and offset, as reconstructing the 1994 Assuming we have to key on segment / file and offset, as reconstructing the