Mercurial > hg > cc > work

comparison lurid3/notes.txt @ 85:19e2136f595b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
switch to robotstxt crawldiagnostics, back to 2019-35
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Fri, 18 Apr 2025 19:18:52 +0100
parents 5aa8b75aa9f0
children 855665c79545
comparison
equal deleted inserted replaced
84:5aa8b75aa9f0 85:19e2136f595b
2741 >: tail -1 slurm_w2c_0f.out 2741 >: tail -1 slurm_w2c_0f.out
2742 Mon Apr 14 15:03:41 BST 2025 Finished 2742 Mon Apr 14 15:03:41 BST 2025 Finished
2743 2743
2744 47:46 elapsed, mid-day weekday, all on one node... 2744 47:46 elapsed, mid-day weekday, all on one node...
2745 Both cross-checks succeed 2745 Both cross-checks succeed
2746
2747 Where do we stand _now_?
2748 >: ff w2c_errs | xargs -n 1 bash -c 'printf "%s\t%s\t%s\n" "$0" $(fgrep -c beegfs/ $0) "$(ls -l $0 | tr -s \ | cut -f 5,6,7,8 -d \ )"' |fgrep Apr | sort -k5nr,5 2>&1
2749 ./fixu/x0/0/w2c_errs 180 97561 Apr 14 15:02
2750 ./fixu/x0/4/w2c_errs 180 119330 Apr 14 09:41
2751 ./fixu/x1/0/w2c_errs 180 92775 Apr 14 15:03
2752 ./fixu/x1/4/w2c_errs 180 121103 Apr 14 09:39
2753 ./fixu/x2/0/w2c_errs 180 99136 Apr 14 15:02
2754 ./fixu/x2/4/w2c_errs 180 122217 Apr 14 09:40
2755 ./fixu/x3/0/w2c_errs 180 98527 Apr 14 15:03
2756 ./fixu/x3/4/w2c_errs 180 117299 Apr 14 09:41
2757 ./fixu/x4/0/w2c_errs 180 97644 Apr 14 15:02
2758 ./fixu/x4/4/w2c_errs 180 120875 Apr 14 09:41
2759 ./fixu/x0/15/w2c_errs 180 141002 Apr 13 15:52
2760 ./fixu/x1/15/w2c_errs 180 142454 Apr 13 15:46
2761 ./fixu/x2/15/w2c_errs 180 139796 Apr 13 15:46
2762 ./fixu/x3/15/w2c_errs 180 142030 Apr 13 15:45
2763 ./fixu/x4/15/w2c_errs 180 141123 Apr 13 15:45
2764 The above show 0, 4, 15 run in parts, confirmed to match 2024 results
2765 exactly except for extra log lines from warc and warc2cdb in w2c_errs.
2766 ./0/w2c_errs 900 485975 Apr 11 03:24 ***
2767 ./15/w2c_errs 900 706412 Apr 11 03:38 ***
2768 ./1/w2c_errs 900 607072 Apr 11 03:30 ***
2769 ./2/w2c_errs 900 637082 Apr 11 03:35 ***
2770 ./4/w2c_errs 900 601209 Apr 11 03:31 ***
2771 Concatenating errs for 0,4,15 match except for warc max counts after the first
2772 180, which makes sense:
2773 >: diff <(cat fixu/x{1,2,3,4,0}/0/w*) 0/w2c_errs|head
2774 1361c1361
2775 < 114802 records, max record: 1056834, max header: 4513
2776 ---
2777 > 114802 records, max record: 1075973, max header: 10302
2778 1368c1368
2779 < 115126 records, max record: 1058589, max header: 5466
2780 ---
2781 > 115126 records, max record: 1075973, max header: 10302
2782 1377c1377
2783 < 114040 records, max record: 1058589, max header: 7318
2784 >: head -1360 0/w2c_errs |tail -4
2785 b'' Invalid date value or format ""
2786 115030 records, max record: 1075973, max header: 10302
2787 7148 LM headers, 7146 win, 2 lose, 2 non-ASCII URIs
2788 /beegfs/common_crawl/CC-MAIN-2023-40/1695233510671.0/orig/warc/CC-MAIN-20230930082033-20230930112033-00179.warc.gz
2789 However, these runs must have been done before the \u fix
2790 >: diff <(cat fixu/x{1,2,3,4,0}/4/l*) 4/lmh.cdb_in|egrep -c '^[0-9].*a'
2791 0
2792 >: diff <(cat fixu/x{1,2,3,4,0}/4/l*) 4/lmh.cdb_in|egrep -c '^[0-9].*d'
2793 4 [these are the blank lines at the end of the first 4 parts]
2794 >: diff <(cat fixu/x{1,2,3,4,0}/4/l*) 4/lmh.cdb_in|egrep -c '^[0-9].*c'
2795 1435
2796 >: diff <(cat fixu/x{1,2,3,4,0}/4/l*) 4/lmh.cdb_in|egrep -c '^<'
2797 1495
2798 >: diff <(cat fixu/x{1,2,3,4,0}/4/l*) 4/lmh.cdb_in|egrep -c '^>'
2799 1491
2800 >: diff <(cat fixu/x{1,2,3,4,0}/4/l*) 4/lmh.cdb_in|egrep -ic '[\\]u'
2801 1491
2802 >: diff <(cat fixu/x{1,2,3,4,0}/4/l*) 4/lmh.cdb_in|egrep -ic '^<.*[\\]u'
2803 1491
2804 Similarly for 0, 15
2805 ./plus/11/w2c_errs 3 1295 Apr 10 15:36
2806 ./x0/11/w2c_errs 180 119337 Apr 10 16:31
2807 ./x1/11/w2c_errs 180 118047 Apr 10 16:31
2808 ./x2/11/w2c_errs 180 109509 Apr 10 16:35
2809 ./x3/11/w2c_errs 180 118879 Apr 10 16:27
2810 ./x4/11/w2c_errs 180 116116 Apr 10 16:30
2811 That batch was done to test the fix to warc.py for the last buffer
2812 load being missed, but _not_ the \u fix:
2813 >: egrep -ic '[\\]u' x?/11/lmh.cdb_in
2814 x0/11/lmh.cdb_in:0
2815 x1/11/lmh.cdb_in:0
2816 x2/11/lmh.cdb_in:0
2817 x3/11/lmh.cdb_in:0
2818 x4/11/lmh.cdb_in:0
2819 So the runs marked *** above must have been done before that fix and are _not valid_.
2820 And everything else below here is too early to be any use
2821 ./10/w2c_errs 900 473108 Apr 3 17:10
2822 ./11/w2c_errs 188 103194 Apr 3 14:03
2823 ./12/w2c_errs 900 605957 Apr 3 17:09
2824 ./8/w2c_errs 900 468236 Apr 3 17:01
2825 ./9/w2c_errs 900 453757 Apr 3 17:03
2826 ./plus/2/w2c_errs 220 131136 Apr 3 12:06
2827 ./plus/3/w2c_errs 96 52260 Apr 3 11:39
2828 ./plus/5/w2c_errs 84 49769 Apr 3 11:35
2829 ./plus/6/w2c_errs 75 40819 Apr 3 11:33
2830 ./plus/7/w2c_errs 91 54141 Apr 3 11:36
2831 ./rerun/4/a_w2c_errs 75 43572 Apr 2 12:48
2832 Next steps:
2833 Check robots.txt for LM headers: Yes
2834 Fetched robotstxt for 2019-35
2835 Ran warc2cdb on 2019-35 robotstxt, crawldiagnostics (after fixing
2836 bug) and warc:
2837 >: time ~/lib/python/cc/w2c.py 2019-35 0 . crawldiagnostics 2> 0/crawldiagnostics/w2c_errs
2838
2839 ./0/crawldiagnostics/lmh.cdb_in
2840 real 7m13.989s
2841 user 6m24.400s
2842 sys 0m15.013s
2843
2844 >: time ~/lib/python/cc/w2c.py 2019-35 0 . robotstxt 2> 0/robotstxt/w2c_errs
2845 ./0/robotstxt/lmh.cdb_in
2846 real 1m11.277s
2847 user 0m57.372s
2848 sys 0m3.942s
2849
2850 >: time ~/lib/python/cc/w2c.py 2019-35 0 ./x0 warc 000,185 2> x0/0/warc/w2c_errs &
2851 >: less x0/0/warc/w2c_errs
2852 >: time ~/lib/python/cc/w2c.py 2019-35 0 ./x1 warc 186,371 2> x1/0/warc/w2c_errs &
2853 >: time ~/lib/python/cc/w2c.py 2019-35 0 ./x2 warc 372,559 2> x2/0/warc/w2c_errs &
2854 ./x0/0/warc/lmh.cdb_in
2855 real 54m59.239s
2856 user 34m34.656s
2857 sys 2m39.790s
2858
2859 ./x1/0/warc/lmh.cdb_in
2860 real 55m15.233s
2861 user 34m54.959s
2862 sys 2m42.281s
2863
2864 ./x2/0/warc/lmh.cdb_in
2865 real 55m48.226s
2866 user 35m14.087s
2867 sys 2m45.434s
2868
2869 Cross-check warc output:
2870 >: wc -l x?/0/warc/l*
2871 1763032 x0/0/warc/lmh.cdb_in
2872 1763752 x1/0/warc/lmh.cdb_in
2873 1781459 x2/0/warc/lmh.cdb_in
2874 5308243 total
2875 >: wc -l 0/ks.tsv
2876 5308240 0/ks.tsv
2877
2878 >: wc -l 0/ks_errs.tsv
2879 21462 0/ks_errs.tsv
2880 >: wc -l x?/0/warc/w*
2881 7621 x0/0/warc/w2c_errs
2882 7768 x1/0/warc/w2c_errs
2883 7753 x2/0/warc/w2c_errs
2884 23142 total
2885 >: echo $(((23142 - 21462)))
2886 1680
2887 >: echo $((560*3))
2888 1680
2889 >: cut -f 2 0/ks_errs.tsv | sus
2890 14749 cannot unpack non-iterable NoneType object
2891 6604 list index out of range
2892 26 year 641480 is out of range
2893 24 year 641788 is out of range
2894 22 year 641785 is out of range
2895 21 year 641787 is out of range
2896 5 hour must be in 0..23
2897 4 year 641791 is out of range
2898 3 year 641474 is out of range
2899 2 year 641484 is out of range
2900 1 year 640818 is out of range
2901 1 year 641778 is out of range
2902 >: echo $((14749 + 6604))
2903 21353
2904 >: cut -f 2 <(sed 's/value or format .*/value or format/' ./x?/0/warc/w2c_errs)|egrep -v 'warc.gz|LM headers|records,'| sus
2905 21353 Invalid date value or format
2906 26 year 641480 is out of range
2907 24 year 641788 is out of range
2908 22 year 641785 is out of range
2909 21 year 641787 is out of range
2910 5 hour must be in 0..23
2911 4 year 641791 is out of range
2912 3 year 641474 is out of range
2913 2 year 641484 is out of range
2914 1 year 640818 is out of range
2915 1 year 641778 is out of range
2916 All good
2917
2918 Fetch c_d and, if needed r_t for 2023-40
2919 Remove all w2c_errs and ldb.cdb_in from *** downwards
2920 Run a parallel full segment job for 1 2 11 12
2921 Do whatever checks are possible on the results
2922 Fill in r_t? and c_d ldb.cdb_in for 0 1 2 4 11 12 15
2923 Take stock: more 2023-40, or back to 2019-35 to add c_d (and r_t)?
2746 ================ 2924 ================
2747 2925
2748 Try it with the existing _per segment_ index we have for 2019-35 2926 Try it with the existing _per segment_ index we have for 2019-35
2749 2927
2750 Assuming we have to key on segment / file and offset, as reconstructing the 2928 Assuming we have to key on segment / file and offset, as reconstructing the