# HG changeset patch
# User Henry S. Thompson <ht@inf.ed.ac.uk>
# Date 1742401646 0
# Node ID 737b74f85e38e659a980286f092044691a5b15e8
# Parent  6935ebce43e08b027f9c79ffe2f1057499234d97
performance problems

diff -r 6935ebce43e0 -r 737b74f85e38 lurid3/notes.txt
--- a/lurid3/notes.txt	Wed Feb 26 19:53:07 2025 +0000
+++ b/lurid3/notes.txt	Wed Mar 19 16:27:26 2025 +0000
@@ -1603,8 +1603,94 @@
 sys     0m22.880s
 
 I thought having the 'cat' in the pipeline was making the difference,
-but no, just as fast w/o.  Something very odd
+but no, just as fast w/o.  Something very odd.
+Next day, losing again:
+  >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp
+  test_cdbp.mainp()
+  '  ks_%d-%d.cdb 0 1 0 1 >  ../cdx-00101x 2>/tmp/hst/onex
+
+  real    2m37.951s
+  user    0m45.490s
+  sys     0m20.874s
+Try a full 17th anyway:
+  >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp
+  test_cdbp.mainp()
+  '  ks_%d-%d.cdb 0 1 0 6 >  ../cdx-00101 2>/tmp/hst/six
+
+  real    2m56.073s
+  user    0m45.699s
+  sys     0m19.964s
+Not significantly slower...
+  >: fgrep -c lastmod ../cdx-00101
+  111967
+  >: wc -l ../cdx-00101
+  14681149
+Try 2/17ths
+  >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp
+  test_cdbp.mainp()
+  '  ks_%d-%d.cdb 0 2 0 12 >  ../cdx-00101 2>/tmp/hst/twelve
 
+  real    4m19.232s
+  user    0m50.538s
+  sys     0m27.702s
+  >: fgrep -c lastmod ../cdx-00101
+     221468
+At least approx. linear...
+Try a compute node...
+No better
+Try chaining
+  >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00102.gz | python3 -c 'import test_cdbp
+  test_cdbp.mainp()
+  ' ks_%d-%d.cdb 0 2 0 12 | python3 -c 'import test_cdbp
+  test_cdbp.mainp()
+  ' ks_%d-%d.cdb 2 4 12 24 > ../cdx-00102 2> /tmp/hst/twentyfour
+  [11 minutes]
+Sigh
+Compute node:
+  >: time { uz /beegfs/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00103.gz|python3 -c 'import test_cdbp
+test_cdbp.mainp()
+'  ks_%d-%d.cdb 0 2 0 12 |python3 -c 'import test_cdbp
+test_cdbp.mainp()
+'  ks_%d-%d.cdb 2 4 12 24 |python3 -c 'import test_cdbp
+test_cdbp.mainp()
+'  ks_%d-%d.cdb 4 6 24 36 ; } >  ../cdx-00103 2> /tmp/hst/36 &
+Ah, maybe I just hadn't cythonised after last edit...
+OK, yes, working!
+    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND  
+3675590 hst       20   0 3965368 519288 506468 R  86.9   0.2   0:28.18 python3  
+3675589 hst       20   0 3922944 507340 502332 S  84.3   0.2   0:25.53 python3  
+  >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp
+  test_cdbp.mainp()
+  '  ks_%d-%d.cdb 0 1 0 6 | python3 -c 'import test_cdbp
+  test_cdbp.mainp()
+  '  ks_%d-%d.cdb 1 2 6 12 >  ../cdx-00101
+  14681147 entries, 111963 given lastmod
+  14681147 entries, 109503 given lastmod
+
+  real    0m57.837s
+  user    1m23.062s
+  sys     0m26.550s
+But with three chained it slowed a bit, top never showed more than 40%
+for the first in the chain:
+  >: time uz /mnt/beegfs/pod12/common_crawl/CC-MAIN-2019-35/cdx/warc/cdx-00101.gz|python3 -c 'import test_cdbp
+  test_cdbp.mainp()
+  '  ks_%d-%d.cdb 0 1 0 6 | python3 -c 'import test_cdbp
+  test_cdbp.mainp()
+  '  ks_%d-%d.cdb 1 2 6 12 | python3 -c 'import test_cdbp;test_cdbp.mainp()
+  '  ks_%d-%d.cdb 2 3 12 18 >  ../cdx-00101 >  ../cdx-00101
+  14681147 entries, 111963 given lastmod
+  14681147 entries, 109503 given lastmod
+  14681147 entries, 111299 given lastmod
+
+  real    2m39.481s
+  user    2m8.402s
+  sys     0m33.503s
+Try compute node again...
+Same behaviour -- 3 is bad, 2 is OK
+    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
+2163622 hst       20   0 3961280 291836 281560 S  10.9   0.1   0:15.66 python3
+2163621 hst       20   0 3922948 289100 278796 I  10.6   0.1   0:15.38 python3
+2163623 hst       20   0 3945880 306352 289356 S  10.3   0.1   0:15.50 python3
 ================
 
 Try it with the existing _per segment_ index we have for 2019-35