| Tue, 28 Apr 2026 16:16:45 +0100 |
Henry S. Thompson |
cdbs built for 2025-33
|
| Thu, 23 Apr 2026 16:07:46 +0100 |
Henry S. Thompson |
in production mode for 2025-33...
|
| Wed, 22 Apr 2026 09:32:26 +0100 |
Henry S. Thompson |
disk space hacking
|
| Mon, 20 Apr 2026 18:25:30 +0100 |
Henry S. Thompson |
got --array working, maybe
|
| Fri, 10 Apr 2026 16:31:51 +0100 |
Henry S. Thompson |
2023-40 index with lastmod now complete
|
| Thu, 09 Apr 2026 15:11:08 +0100 |
Henry S. Thompson |
bug in shrink_key.py fixed, 2024-30 index w. lastmod complete
|
| Mon, 06 Apr 2026 23:10:22 +0100 |
Henry S. Thompson |
figure out the right number and sizes for the full
|
| Mon, 06 Apr 2026 16:22:50 +0100 |
Henry S. Thompson |
add in the 2-bit file type info,
|
| Sat, 04 Apr 2026 21:22:57 +0100 |
Henry S. Thompson |
found a new duplicate problem, diagnosed it
|
| Fri, 03 Apr 2026 22:44:57 +0100 |
Henry S. Thompson |
done processing crawldiagnostics and robotstxt for 2023-40
|
| Thu, 02 Apr 2026 21:36:55 +0100 |
Henry S. Thompson |
complete satisfactory uniqueness check on 2023-40/warc new cdb_in format
|
| Sat, 28 Mar 2026 21:39:32 +0000 |
Henry S. Thompson |
2023-40 w2c.py done for all 10 deciles, needs dup testing
|
| Fri, 27 Mar 2026 12:36:06 +0000 |
Henry S. Thompson |
abandoned --array again
|
| Mon, 16 Mar 2026 13:50:31 +0000 |
Henry S. Thompson |
noting block offsets while scanning warc files now working in gzip_plus/warc/test_warc
|
| Thu, 12 Mar 2026 18:10:13 +0000 |
Henry S. Thompson |
progress wrt block boundary logging
|
| Wed, 11 Mar 2026 18:49:57 +0000 |
Henry S. Thompson |
towards using file+offset as cdb key
|
| Fri, 06 Mar 2026 14:42:05 +0000 |
Henry S. Thompson |
...
|
| Thu, 05 Mar 2026 18:23:31 +0000 |
Henry S. Thompson |
glitch in crawldiagnostics...
|
| Wed, 04 Mar 2026 21:10:37 +0000 |
Henry S. Thompson |
cd and rt warc2cdb.py done for 2023-40
|
| Mon, 02 Mar 2026 22:49:21 +0000 |
Henry S. Thompson |
got all 2019-35 statuses-by-year
|
| Mon, 02 Mar 2026 10:55:33 +0000 |
Henry S. Thompson |
complete stats for status code from 2019-35
|
| Fri, 27 Feb 2026 20:44:17 +0000 |
Henry S. Thompson |
brief excursion to look at status codes by year
|
| Thu, 05 Feb 2026 14:53:35 +0000 |
Henry S. Thompson |
new grouped date-shrinking working better
|
| Wed, 28 Jan 2026 18:35:47 +0000 |
Henry S. Thompson |
minor glitch w. warc2cdb.py,
|
| Wed, 28 Jan 2026 12:26:10 +0000 |
Henry S. Thompson |
w2c.py now looks good, running all of 2023-40
|
| Tue, 27 Jan 2026 19:15:03 +0000 |
Henry S. Thompson |
reran w2c.py and cdb_one on x0/24
|
| Tue, 27 Jan 2026 17:32:08 +0000 |
Henry S. Thompson |
found the 'bug', fixed in warc2cdb.py
|
| Mon, 26 Jan 2026 12:51:56 +0000 |
Henry S. Thompson |
got it!
|
| Sun, 25 Jan 2026 21:45:14 +0000 |
Henry S. Thompson |
Probably \unnnn is the problem?
|
| Sun, 25 Jan 2026 21:31:08 +0000 |
Henry S. Thompson |
Maybe \unnnn is the problem?
|
| Sat, 24 Jan 2026 12:02:23 +0000 |
Henry S. Thompson |
still searching for missing lm hits in shrunk.cdb
|
| Thu, 22 Jan 2026 18:30:01 +0000 |
Henry S. Thompson |
simple key check fails to reproduce the error
|
| Wed, 21 Jan 2026 19:12:15 +0000 |
Henry S. Thompson |
still no agreement
|
| Wed, 21 Jan 2026 15:47:48 +0000 |
Henry S. Thompson |
still redoing
|
| Tue, 20 Jan 2026 19:50:40 +0000 |
Henry S. Thompson |
use shrink_key.shrink_date, now encorporating LM shrinkage, everywhere, including warc2cdb.py
|
| Mon, 19 Jan 2026 18:31:56 +0000 |
Henry S. Thompson |
but first...
|
| Mon, 19 Jan 2026 17:26:16 +0000 |
Henry S. Thompson |
talk myself through the problem
|
| Mon, 19 Jan 2026 12:06:54 +0000 |
Henry S. Thompson |
move date shrink to shrink.py
|
| Fri, 16 Jan 2026 13:34:15 +0000 |
Henry S. Thompson |
...
|
| Wed, 14 Jan 2026 14:06:30 +0000 |
Henry S. Thompson |
...
|
| Wed, 14 Jan 2026 10:55:11 +0000 |
Henry Thompson |
...
|
| Fri, 09 Jan 2026 18:01:19 +0000 |
Henry S. Thompson |
...
|
| Thu, 18 Dec 2025 14:45:59 +0000 |
Henry S. Thompson |
ByteIO ?
|
| Wed, 17 Dec 2025 15:23:33 +0000 |
Henry S. Thompson |
shrinking...
|
| Fri, 12 Dec 2025 21:50:10 +0000 |
Henry S. Thompson |
filled in
|
| Wed, 10 Dec 2025 19:15:14 +0000 |
Henry S. Thompson |
explore fixing lack of shrinkage in 2023-40
|
| Sat, 06 Dec 2025 18:28:21 +0000 |
Henry S. Thompson |
parquet example, _inter alia_
|
| Wed, 03 Dec 2025 19:16:13 +0000 |
Henry S. Thompson |
got min-max for 2024-33, updated/tidied status.xlsx to match dates.tsv
|
| Mon, 01 Dec 2025 17:15:01 +0000 |
Henry S. Thompson |
2025-33 index
|
| Wed, 05 Nov 2025 15:23:14 +0000 |
Henry S. Thompson |
minor
|
| Wed, 05 Nov 2025 15:21:19 +0000 |
Henry S. Thompson |
as submitted
|
| Wed, 05 Nov 2025 15:13:44 +0000 |
Henry S. Thompson |
as submitted
|
| Wed, 05 Nov 2025 12:03:27 +0000 |
Henry S. Thompson |
shrink to 2 pages
|
| Wed, 05 Nov 2025 11:47:42 +0000 |
Henry S. Thompson |
nearly ready
|
| Tue, 04 Nov 2025 11:26:21 +0000 |
Henry S. Thompson |
as submitted
|
| Tue, 04 Nov 2025 11:22:15 +0000 |
Henry S. Thompson |
moved
|
| Tue, 04 Nov 2025 11:16:15 +0000 |
Henry S. Thompson |
moved
|
| Wed, 21 May 2025 22:04:18 +0100 |
Henry S. Thompson |
repair broken cases, cdb_in for 2023-40/warc now complete, but not trimmed
|
| Mon, 19 May 2025 22:16:32 +0100 |
Henry S. Thompson |
2019-35 reindexing with c_d and r_t nearly done, just needs cluster.idx
|
| Tue, 13 May 2025 14:45:25 +0100 |
Henry Thompson |
cdb_chain working
|
