Mercurial > hg > cc > work

comparison lurid3/notes.txt @ 63:663e55844c1d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
comparing profiles w and w/o cython
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Mon, 06 Jan 2025 17:59:20 +0000
parents bc0bdb649c08
children a70ceb9d1e82
comparison
equal deleted inserted replaced
62:bc0bdb649c08 63:663e55844c1d
1026 >>> (t.repeat(5,100000),s.repeat(5,100000)) 1026 >>> (t.repeat(5,100000),s.repeat(5,100000))
1027 [0.005662968382239342, 0.005780909210443497, 0.005478940904140472, 0.005713008344173431, 0.005547545850276947] 1027 [0.005662968382239342, 0.005780909210443497, 0.005478940904140472, 0.005713008344173431, 0.005547545850276947]
1028 [0.30250774696469307, 0.303345350548625, 0.3002819549292326, 0.30161340720951557, 0.30262864381074905]) 1028 [0.30250774696469307, 0.303345350548625, 0.3002819549292326, 0.30161340720951557, 0.30262864381074905])
1029 1029
1030 so, forget cdb 1030 so, forget cdb
1031 Hmm...
1032 >: time cdbtest <results/CC-MAIN-2019-35/warc_lmhx/ks_0.cdb
1033 found: 5226422
1034 different record: 0
1035 bad length: 0
1036 not found: 0
1037 untested: 10552
1038
1039 real 0m34.151s
1040 user 0m7.513s
1041 sys 0m5.123s
1042 (/ 34.1 5226422)
1043 6.524540115589595e-06
1044
1045 Added a bunch of type hints, tried cython
1046
1047 >: cython --embed test_lookup1.py
1048 ...[warnings]
1049 >: python3 -m autotyping --aggressive test_lookup1.py
1050 Calculating full-repo metadata...
1051 ...
1052 - 0 warnings were generated.
1053 >: cc -I /usr/include/python3.11 -L /usr/lib/python3.11/config-3.11-x86_64-linux-gnu/ test_lookup1.c -l python3.11 -o test_lookup1
1054
1055 >: time ~/lib/python/cc/lmh/test_lookup1.py
1056 52369734
1057 1076046 entries, 130318 given lastmod
1058
1059 real 1m45.305s
1060 user 1m34.265s
1061 sys 0m8.988s
1062 sing<4969>: time ~/lib/python/cc/lmh/test_lookup1
1063 52369734
1064 1076046 entries, 130318 given lastmod
1065
1066 real 2m5.626s
1067 user 1m54.493s
1068 sys 0m9.169s
1069
1070 Compiled in profiling
1071
1072 >: python3 /work/dc007/dc007/hst/lib/python/cc/lmh/test_pl.py
1073 /work/dc007/dc007/hst/lib/python/cc/lmh/test_lookup1.py
1074 52369734
1075 1076046 entries, 130318 given lastmod
1076 Mon Jan 6 13:56:04 2025 Profile.prof
1077
1078 152694273 function calls (152694246 primitive calls) in 169.605 seconds
1079
1080 Ordered by: internal time
1081
1082 ncalls tottime percall cumtime percall filename:lineno(function)
1083 1 30.001 30.001 30.001 30.001 {built-in method _pickle.load}
1084 11195226 29.825 0.000 68.952 0.000 igzip.py:196(write)
1085 11195226 24.403 0.000 24.403 0.000 {method 'compress' of 'isal_zlib.Compress' objects}
1086 1 23.919 23.919 167.278 167.278 test_lookup1.py:9(mainp)
1087 10543636 14.072 0.000 14.072 0.000 {method 'search' of 're.Pattern' objects}
1088 10543637 11.911 0.000 11.911 0.000 {method 'readline' of '_io.BufferedReader' objects}
1089 21738863 8.646 0.000 12.583 0.000 _compression.py:12(_check_not_closed)
1090 10543637 8.431 0.000 26.779 0.000 gzip.py:397(readline)
1091 21738868 3.937 0.000 3.937 0.000 gzip.py:323(closed)
1092 10543636 3.524 0.000 3.524 0.000 {method 'split' of 'bytes' objects}
1093 11195226 2.951 0.000 2.951 0.000 {built-in method isal.isal_zlib.crc32}
1094 11195236 2.814 0.000 2.814 0.000 {method 'write' of '_io.BufferedWriter' objects}
1095 1 2.327 2.327 169.605 169.605 <string>:1(<module>)
1096 11195324 1.414 0.000 1.414 0.000 {built-in method builtins.isinstance}
1097 11064997/11064989 1.399 0.000 1.399 0.000 {built-in method builtins.len}
1098
1099 >: python3 /work/dc007/dc007/hst/lib/python/cc/lmh/test_pl.py
1100 /work/dc007/dc007/hst/lib/python/cc/lmh/test_lookup1.cpython-311-x86_64-linux-gnu.so
1101 52369734
1102 1076046 entries, 130318 given lastmod
1103 Mon Jan 6 14:50:59 2025 Profile.prof
1104
1105 131606993 function calls (131606966 primitive calls) in 164.213 seconds
1106
1107 Ordered by: internal time
1108
1109 ncalls tottime percall cumtime percall filename:lineno(function)
1110 1 67.658 67.658 164.213 164.213 test_lookup1.py:9(mainp)
1111 11195226 29.944 0.000 69.788 0.000 igzip.py:196(write)
1112 11195226 25.274 0.000 25.274 0.000 {method 'compress' of 'isal_zlib.Compress' objects}
1113 10543637 12.200 0.000 12.200 0.000 {method 'readline' of '_io.BufferedReader' objects}
1114 21738863 8.530 0.000 12.394 0.000 _compression.py:12(_check_not_closed)
1115 10543637 8.205 0.000 26.723 0.000 gzip.py:397(readline)
1116 21738868 3.863 0.000 3.863 0.000 gzip.py:323(closed)
1117 11195226 3.058 0.000 3.058 0.000 {built-in method isal.isal_zlib.crc32}
1118 11195236 2.784 0.000 2.784 0.000 {method 'write' of '_io.BufferedWriter' objects}
1119 11064996/11064988 1.343 0.000 1.343 0.000 {built-in method builtins.len}
1120 11195324 1.309 0.000 1.309 0.000 {built-in method builtins.isinstance}
1121 2 0.040 0.020 0.040 0.020 {built-in method io.open}
1031 ================ 1122 ================
1032 1123
1033 1124
1034 Try it with the existing _per segment_ index we have for 2019-35 1125 Try it with the existing _per segment_ index we have for 2019-35
1035 1126