log

age author description
Wed, 21 Jul 2021 20:05:42 +0000 Henry S. Thompson add cl arg --fpath replacing FPAT, which is now default value
Wed, 21 Jul 2021 20:04:11 +0000 Henry S. Thompson more paths
Wed, 14 Jul 2021 16:50:30 +0000 Henry S. Thompson add usage/help info
Wed, 14 Jul 2021 16:49:54 +0000 Henry S. Thompson add usage/help info
Wed, 14 Jul 2021 16:49:35 +0000 Henry S. Thompson parameterise the temp file and move it to /dev/shm
Wed, 14 Jul 2021 15:30:29 +0000 Henry S. Thompson sic
Fri, 09 Jul 2021 14:20:51 +0000 Henry S. Thompson use printf safely
Fri, 09 Jul 2021 13:46:10 +0000 Henry S. Thompson handle multiple L-M lines :-(
Fri, 09 Jul 2021 13:45:43 +0000 Henry S. Thompson improve error handling
Fri, 09 Jul 2021 13:45:04 +0000 Henry S. Thompson more focussed, better SLURM_... vars
Tue, 29 Jun 2021 08:00:40 +0000 Henry S. Thompson bits and pieces
Tue, 29 Jun 2021 07:53:47 +0000 Henry S. Thompson better btot
Mon, 28 Jun 2021 21:50:30 +0000 Henry S. Thompson extract Last Modified via cdx
Mon, 28 Jun 2021 17:16:34 +0000 Henry S. Thompson fix path to qpdf
Mon, 28 Jun 2021 17:16:15 +0000 Henry S. Thompson silently skip robotstxt
Mon, 28 Jun 2021 17:15:19 +0000 Henry S. Thompson workaround histcontrol
Mon, 28 Jun 2021 15:40:10 +0000 Henry S. Thompson support field edit
Mon, 28 Jun 2021 14:01:41 +0000 Henry S. Thompson for use in processing CC index files
Wed, 16 Jun 2021 16:12:46 +0000 Henry S. Thompson implement --cmd
Wed, 16 Jun 2021 16:12:16 +0000 Henry S. Thompson qpdf needs LD_LIB_PATH
Tue, 15 Jun 2021 18:04:34 +0000 Henry S. Thompson refactor final processing loop,
Tue, 15 Jun 2021 16:58:31 +0000 Henry S. Thompson frame size
Tue, 15 Jun 2021 16:58:03 +0000 Henry S. Thompson include sh-script
Mon, 26 Apr 2021 17:18:29 +0000 Henry S. Thompson all parts working, idempotency achieved
Mon, 26 Apr 2021 17:17:58 +0000 Henry S. Thompson debugging
Mon, 26 Apr 2021 17:17:38 +0000 Henry S. Thompson (none)
Mon, 26 Apr 2021 15:28:23 +0000 Henry S. Thompson warc and headers parts working
Thu, 22 Apr 2021 21:31:03 +0000 Henry S. Thompson back to IGzipFile
Thu, 22 Apr 2021 21:10:02 +0000 Henry S. Thompson approved Popen version using .communicate
Thu, 22 Apr 2021 19:06:55 +0000 Henry S. Thompson using Popen to run igzip (also not great)
Tue, 20 Apr 2021 19:11:57 +0000 Henry S. Thompson added support for copying to/using /dev/shm or /tmp
Tue, 20 Apr 2021 12:26:09 +0000 Henry S. Thompson working with -x and rich directory structure
Tue, 20 Apr 2021 11:12:35 +0000 Henry S. Thompson convert to rich directory structure per 2019-35
Mon, 19 Apr 2021 18:09:51 +0000 Henry S. Thompson -x barely working
Mon, 19 Apr 2021 18:09:25 +0000 Henry S. Thompson never should have added
Mon, 19 Apr 2021 13:08:16 +0000 Henry S. Thompson better dd error handling
Mon, 19 Apr 2021 13:07:58 +0000 Henry S. Thompson (none)
Sun, 18 Apr 2021 17:03:45 +0000 Henry S. Thompson bare minimum working
Fri, 16 Apr 2021 18:28:00 +0000 Henry S. Thompson triple args checked, filename opened
Fri, 16 Apr 2021 13:15:23 +0000 Henry S. Thompson help format hacking done
Fri, 16 Apr 2021 12:55:05 +0000 Henry S. Thompson basic help format hacking works
Fri, 16 Apr 2021 09:01:16 +0000 Henry S. Thompson (none)
Fri, 16 Apr 2021 09:00:17 +0000 Henry S. Thompson (none)
Thu, 15 Apr 2021 19:22:27 +0000 Henry S. Thompson just strugging with argparse
Thu, 15 Apr 2021 10:59:25 +0000 Henry S. Thompson support a command to receive each result,
Wed, 14 Apr 2021 20:15:32 +0000 Henry S. Thompson accepts index lines, less line-at-a-time
Wed, 14 Apr 2021 10:08:41 +0000 Henry S. Thompson working with one input
Wed, 14 Apr 2021 08:57:43 +0000 Henry S. Thompson -w and -h working
Tue, 13 Apr 2021 17:52:31 +0000 Henry S. Thompson working on flags
Tue, 13 Apr 2021 17:02:09 +0000 Henry S. Thompson new
Tue, 16 Mar 2021 16:20:02 +0000 Henry S. Thompson working with locking and copying
Mon, 15 Mar 2021 14:26:42 +0000 Henry S. Thompson working for -t 2 -c 2
Mon, 15 Mar 2021 14:20:00 +0000 Henry S. Thompson minor
Sun, 14 Mar 2021 21:28:02 +0000 Henry S. Thompson prepare for real parallel distribution
Sun, 14 Mar 2021 21:25:01 +0000 Henry S. Thompson environment improvements
Wed, 03 Mar 2021 19:33:56 +0000 Henry S. Thompson trying to move to slurm
Sat, 09 May 2020 16:16:28 +0100 Henry S. Thompson improved F handling/logging
Fri, 08 May 2020 19:52:36 +0100 Henry S. Thompson keep separate antecedants separate, buggy?
Thu, 07 May 2020 18:47:24 +0100 Henry S. Thompson track redirects, need to us full crawldiagnostics.warc.gz for "location:" and "Uri:"
Thu, 07 May 2020 11:33:24 +0100 Henry S. Thompson refactor, change summary print (problem?)