Mercurial > hg > cc > work
changeset 89:367dca1afea4
begin actual cdb creation experiment on expanded data (including r_t, c_d) for 2019-35
| author | Henry S. Thompson <ht@inf.ed.ac.uk> |
|---|---|
| date | Fri, 02 May 2025 17:09:22 +0100 |
| parents | d622b1b5e60d |
| children | 82d406eca22b |
| files | lurid3/notes.txt |
| diffstat | 1 files changed, 78 insertions(+), 0 deletions(-) [+] |
line wrap: on
line diff
--- a/lurid3/notes.txt Fri May 02 13:10:21 2025 +0100 +++ b/lurid3/notes.txt Fri May 02 17:09:22 2025 +0100 @@ -3694,3 +3694,81 @@ 1 +100,10:20190822110255http://legacy.abcquant.com/index.php?option=com_content&view=article&id=158&Itemid=208-> ... +Promising... + +Next question: how many .cdb files can we get away with for 2019-35, +given the 4GB limit? + +I settled on 16 .cdb files for 2023-40, but compared to that 2019-35 is + 62% wrt number of /warc/...warc.gz files + 73% wrt number of lines in 10 random files in segment 36 + 97% wrt number of valid LM headers in segment 36 (!) + 16% valid LM headers in 10 2023-40 files vs. 18% for 10 2019-35 files + +Let's start with 17 just for grins... + >: mkdir ncdb + >: cd ncdb + >: mkdir 0 + >: cd 0 + >: for i in {0..4}; do for t in warc robotstxt crawldiagnostics; do head -n $(($(wc -l < ../../x$i/0/$t/lmh.cdb_in) - 1)) ../../x$i/0/$t/lmh.cdb_in >> lmh.cdb_in; done ; done ; echo >> lmh.cdb_in + >: tail -2 lmh.cdb_in + +57,10:20190817230302https://zoom.nl/image/edit?image_id=2705705->1566082403 + + >: wc -l lmh.cdb_in + 5909614 lmh.cdb_in + >: egrep -c '^$' lmh.cdb_in + 1 + >: wc -l ../../x?/0/*/lmh.cdb_in | tail -1 + 5909628 total + >: echo ../../x?/0/*/lmh.cdb_in | wc -w + 15 + >: echo $((5909628 - 14)) + 5909614 + >: wc -l ../../0/ks.tsv + 5308240 ../../0/ks.tsv + +So, that's a more-than-10% increase +And + >: echo $(($(wc -l< ../../cdb/ks_0-5.tsv) / 6)) + 5216159 +well over that average, pbly won't now be able to fit 6 +into 4GB, but let's try... + + >: cd .. + >: for s in {1..5}; do (cd $s; for i in {0..4}; do for t in warc robotstxt crawldiagnostics; do head -n $(($(wc -l < ../../x$i/$s/$t/lmh.cdb_in) - 1)) ../../x$i/$s/$t/lmh.cdb_in >> lmh.cdb_in; done ; done ; echo >> lmh.cdb_in); done + >: wc -l */l* + 5909614 0/lmh.cdb_in + 5828355 1/lmh.cdb_in + 5720232 2/lmh.cdb_in + 5727498 3/lmh.cdb_in + 5834213 4/lmh.cdb_in + 5856536 5/lmh.cdb_in + 34876448 total + +More than the old 34,000,000 cutoff... + +Ah, we don't actually want the empty last lines in the individual +cdb_in files, since we don't know how many we will eventually +combine... + >: ls -l */*.cdb_in + -rw-r--r-- 1 hst dc007 658140014 May 2 16:35 0/lmh.cdb_in + -rw-r--r-- 1 hst dc007 648024030 May 2 16:54 1/lmh.cdb_in + -rw-r--r-- 1 hst dc007 636163088 May 2 16:54 2/lmh.cdb_in + -rw-r--r-- 1 hst dc007 637397790 May 2 16:54 3/lmh.cdb_in + -rw-r--r-- 1 hst dc007 652157651 May 2 16:54 4/lmh.cdb_in + -rw-r--r-- 1 hst dc007 652391161 May 2 16:54 5/lmh.cdb_in + >: for f in */*.cdb_in; do truncate $f -s -1; done + >: ls -l */*.cdb_in + -rw-r--r-- 1 hst dc007 658140013 May 2 17:05 0/lmh.cdb_in + -rw-r--r-- 1 hst dc007 648024029 May 2 17:05 1/lmh.cdb_in + -rw-r--r-- 1 hst dc007 636163087 May 2 17:05 2/lmh.cdb_in + -rw-r--r-- 1 hst dc007 637397789 May 2 17:05 3/lmh.cdb_in + -rw-r--r-- 1 hst dc007 652157650 May 2 17:05 4/lmh.cdb_in + -rw-r--r-- 1 hst dc007 652391160 May 2 17:05 5/lmh.cdb_in + >: { cat */*.cdb_in ; echo ; } > lmh_0-5.cdb_in + >: ls -l lmh_0-5.cdb_in + -rw-r--r-- 1 hst dc007 3884273729 May 2 17:06 lmh_0-5.cdb_in + >: cdbmake lmh_0-5.cdb lmh.tmp < lmh_0-5.cdb_in + cdbmake: fatal: unable to create lmh.tmp: out of memory + +Sigh. So, 5 at a time it is.