cc/cirrus_home: revision graph

graph

less more | rev 153: (0) -120 tip

add -c switch to btot
Mon, 01 Nov 2021 21:23:13 +0000, by Henry S. Thompson
use sqlite3 just to tabulate
Thu, 28 Oct 2021 12:11:08 +0000, by Henry S. Thompson
fixed
Tue, 26 Oct 2021 14:07:34 +0000, by Henry S. Thompson
working, with compound driver files
Tue, 26 Oct 2021 14:05:35 +0000, by Henry S. Thompson
better comments
Mon, 25 Oct 2021 15:07:03 +0000, by Henry S. Thompson
do the work for cdx2sql
Mon, 25 Oct 2021 15:05:46 +0000, by Henry S. Thompson
change test to use Master
Mon, 25 Oct 2021 15:05:25 +0000, by Henry S. Thompson
works for 0--9
Fri, 22 Oct 2021 12:36:15 +0000, by Henry S. Thompson
replace too-complex invocation of cdx2tsv
Thu, 21 Oct 2021 19:18:47 +0000, by Henry S. Thompson
basic, works
Wed, 20 Oct 2021 17:14:18 +0000, by Henry S. Thompson
too clever by half, keys won't work in parallel for e.g. media types
Wed, 20 Oct 2021 15:47:55 +0000, by Henry S. Thompson
working, w. pickle
Tue, 19 Oct 2021 12:57:50 +0000, by Henry S. Thompson
mail-lib
Tue, 19 Oct 2021 12:56:14 +0000, by Henry S. Thompson
move to ec164.guest
Tue, 19 Oct 2021 12:55:30 +0000, by Henry S. Thompson
fixed bug(s) wrt large payload files
Fri, 23 Jul 2021 22:19:15 +0000, by Henry S. Thompson
just barely working
Fri, 23 Jul 2021 16:23:46 +0000, by Henry S. Thompson
add cl arg --fpath replacing FPAT, which is now default value
Wed, 21 Jul 2021 20:05:42 +0000, by Henry S. Thompson
more paths
Wed, 21 Jul 2021 20:04:11 +0000, by Henry S. Thompson
add usage/help info
Wed, 14 Jul 2021 16:50:30 +0000, by Henry S. Thompson
add usage/help info
Wed, 14 Jul 2021 16:49:54 +0000, by Henry S. Thompson
parameterise the temp file and move it to /dev/shm
Wed, 14 Jul 2021 16:49:35 +0000, by Henry S. Thompson
sic
Wed, 14 Jul 2021 15:30:29 +0000, by Henry S. Thompson
use printf safely
Fri, 09 Jul 2021 14:20:51 +0000, by Henry S. Thompson
handle multiple L-M lines :-(
Fri, 09 Jul 2021 13:46:10 +0000, by Henry S. Thompson
improve error handling
Fri, 09 Jul 2021 13:45:43 +0000, by Henry S. Thompson
more focussed, better SLURM_... vars
Fri, 09 Jul 2021 13:45:04 +0000, by Henry S. Thompson
bits and pieces
Tue, 29 Jun 2021 08:00:40 +0000, by Henry S. Thompson
better btot
Tue, 29 Jun 2021 07:53:47 +0000, by Henry S. Thompson
extract Last Modified via cdx
Mon, 28 Jun 2021 21:50:30 +0000, by Henry S. Thompson
fix path to qpdf
Mon, 28 Jun 2021 17:16:34 +0000, by Henry S. Thompson
silently skip robotstxt
Mon, 28 Jun 2021 17:16:15 +0000, by Henry S. Thompson
workaround histcontrol
Mon, 28 Jun 2021 17:15:19 +0000, by Henry S. Thompson
support field edit
Mon, 28 Jun 2021 15:40:10 +0000, by Henry S. Thompson
for use in processing CC index files
Mon, 28 Jun 2021 14:01:41 +0000, by Henry S. Thompson
implement --cmd
Wed, 16 Jun 2021 16:12:46 +0000, by Henry S. Thompson
qpdf needs LD_LIB_PATH
Wed, 16 Jun 2021 16:12:16 +0000, by Henry S. Thompson
refactor final processing loop,
Tue, 15 Jun 2021 18:04:34 +0000, by Henry S. Thompson
frame size
Tue, 15 Jun 2021 16:58:31 +0000, by Henry S. Thompson
include sh-script
Tue, 15 Jun 2021 16:58:03 +0000, by Henry S. Thompson
all parts working, idempotency achieved
Mon, 26 Apr 2021 17:18:29 +0000, by Henry S. Thompson
debugging
Mon, 26 Apr 2021 17:17:58 +0000, by Henry S. Thompson
(none)
Mon, 26 Apr 2021 17:17:38 +0000, by Henry S. Thompson
warc and headers parts working
Mon, 26 Apr 2021 15:28:23 +0000, by Henry S. Thompson
back to IGzipFile
Thu, 22 Apr 2021 21:31:03 +0000, by Henry S. Thompson
approved Popen version using .communicate
Thu, 22 Apr 2021 21:10:02 +0000, by Henry S. Thompson
using Popen to run igzip (also not great)
Thu, 22 Apr 2021 19:06:55 +0000, by Henry S. Thompson
added support for copying to/using /dev/shm or /tmp
Tue, 20 Apr 2021 19:11:57 +0000, by Henry S. Thompson
working with -x and rich directory structure
Tue, 20 Apr 2021 12:26:09 +0000, by Henry S. Thompson
convert to rich directory structure per 2019-35
Tue, 20 Apr 2021 11:12:35 +0000, by Henry S. Thompson
-x barely working
Mon, 19 Apr 2021 18:09:51 +0000, by Henry S. Thompson
never should have added
Mon, 19 Apr 2021 18:09:25 +0000, by Henry S. Thompson
better dd error handling
Mon, 19 Apr 2021 13:08:16 +0000, by Henry S. Thompson
(none)
Mon, 19 Apr 2021 13:07:58 +0000, by Henry S. Thompson
bare minimum working
Sun, 18 Apr 2021 17:03:45 +0000, by Henry S. Thompson
triple args checked, filename opened
Fri, 16 Apr 2021 18:28:00 +0000, by Henry S. Thompson
help format hacking done
Fri, 16 Apr 2021 13:15:23 +0000, by Henry S. Thompson
basic help format hacking works
Fri, 16 Apr 2021 12:55:05 +0000, by Henry S. Thompson
(none)
Fri, 16 Apr 2021 09:01:16 +0000, by Henry S. Thompson
(none)
Fri, 16 Apr 2021 09:00:17 +0000, by Henry S. Thompson
just strugging with argparse
Thu, 15 Apr 2021 19:22:27 +0000, by Henry S. Thompson
support a command to receive each result,
Thu, 15 Apr 2021 10:59:25 +0000, by Henry S. Thompson
accepts index lines, less line-at-a-time
Wed, 14 Apr 2021 20:15:32 +0000, by Henry S. Thompson
working with one input
Wed, 14 Apr 2021 10:08:41 +0000, by Henry S. Thompson
-w and -h working
Wed, 14 Apr 2021 08:57:43 +0000, by Henry S. Thompson
working on flags
Tue, 13 Apr 2021 17:52:31 +0000, by Henry S. Thompson
new
Tue, 13 Apr 2021 17:02:09 +0000, by Henry S. Thompson
working with locking and copying
Tue, 16 Mar 2021 16:20:02 +0000, by Henry S. Thompson
working for -t 2 -c 2
Mon, 15 Mar 2021 14:26:42 +0000, by Henry S. Thompson
minor
Mon, 15 Mar 2021 14:20:00 +0000, by Henry S. Thompson
prepare for real parallel distribution
Sun, 14 Mar 2021 21:28:02 +0000, by Henry S. Thompson
environment improvements
Sun, 14 Mar 2021 21:25:01 +0000, by Henry S. Thompson
trying to move to slurm
Wed, 03 Mar 2021 19:33:56 +0000, by Henry S. Thompson
improved F handling/logging
Sat, 09 May 2020 16:16:28 +0100, by Henry S. Thompson
keep separate antecedants separate, buggy?
Fri, 08 May 2020 19:52:36 +0100, by Henry S. Thompson
track redirects, need to us full crawldiagnostics.warc.gz for "location:" and "Uri:"
Thu, 07 May 2020 18:47:24 +0100, by Henry S. Thompson
refactor, change summary print (problem?)
Thu, 07 May 2020 11:33:24 +0100, by Henry S. Thompson
bare framework working
Wed, 06 May 2020 18:28:52 +0100, by Henry S. Thompson
starting on tool to assemble as complete as we have info wrt a seed URI
Wed, 06 May 2020 14:25:44 +0100, by Henry S. Thompson
use local .m2/repository for Hadoop 3.4.0
Wed, 06 May 2020 14:24:42 +0100, by Henry S. Thompson
works for big files with Hadoop 3.4.0
Wed, 06 May 2020 14:23:33 +0100, by Henry S. Thompson
x
Wed, 06 May 2020 14:22:48 +0100, by Henry S. Thompson
log trucations
Tue, 28 Apr 2020 19:02:34 +0100, by Henry S. Thompson
impose some limits
Tue, 28 Apr 2020 19:02:14 +0100, by Henry S. Thompson
x
Tue, 28 Apr 2020 19:01:41 +0100, by Henry S. Thompson
x
Fri, 24 Apr 2020 20:12:44 +0100, by Henry S. Thompson
mostly from Sebastian
Fri, 24 Apr 2020 20:12:29 +0100, by Henry S. Thompson
misc
Fri, 24 Apr 2020 20:03:29 +0100, by Henry S. Thompson
misc
Fri, 24 Apr 2020 20:01:35 +0100, by Henry S. Thompson
fix from Sebastian
Fri, 24 Apr 2020 20:01:25 +0100, by Henry S. Thompson
misc
Fri, 24 Apr 2020 19:57:16 +0100, by Henry S. Thompson
misc
Fri, 24 Apr 2020 19:55:11 +0100, by Henry S. Thompson
several efficiency (hofentlich) tweaks
Fri, 24 Apr 2020 15:20:33 +0100, by Henry S. Thompson
x
Thu, 23 Apr 2020 17:26:55 +0100, by Henry S. Thompson
switch for use on login server, invoke by hand with 0/1 as only cmd line arg
Thu, 23 Apr 2020 17:25:25 +0100, by Henry S. Thompson
java stuff
Wed, 22 Apr 2020 18:42:40 +0100, by Henry S. Thompson
try nutch fetch for big pdfs
Wed, 22 Apr 2020 18:42:23 +0100, by Henry S. Thompson
final most general versin
Wed, 15 Apr 2020 18:44:18 +0100, by Henry S. Thompson
too big for /dev/shm, split in half
Tue, 14 Apr 2020 17:52:34 +0100, by Henry S. Thompson
one-off to convert big extracts.tar into lots of smaller ones
Tue, 14 Apr 2020 16:10:22 +0100, by Henry S. Thompson
as used successfully for 3rd run
Mon, 13 Apr 2020 17:29:31 +0100, by Henry S. Thompson
ready to try another pass with robust diff checking
Mon, 13 Apr 2020 15:24:32 +0100, by Henry S. Thompson
working towards more robust diff checking
Mon, 13 Apr 2020 14:12:12 +0100, by Henry S. Thompson
a few tweaks after 2nd parallel run
Sat, 11 Apr 2020 13:41:46 +0100, by Henry S. Thompson
another few log fixes
Fri, 10 Apr 2020 18:45:30 +0100, by Henry S. Thompson
as running, modulo 1 log output wrong
Fri, 10 Apr 2020 18:42:08 +0100, by Henry S. Thompson
log more, work around more glitches
Fri, 10 Apr 2020 18:22:48 +0100, by Henry S. Thompson
x
Fri, 10 Apr 2020 18:22:24 +0100, by Henry S. Thompson
start try to work around failures
Wed, 08 Apr 2020 14:11:04 +0100, by Henry S. Thompson
parallelised version of reExtract.sh
Wed, 08 Apr 2020 11:27:33 +0100, by Henry S. Thompson
complete change of array var construction, used it for log file names too, tar update enabled, so maybe complete but w/o any parallel
Tue, 07 Apr 2020 18:00:29 +0100, by Henry S. Thompson
added computation of required additions to tar file, but not actually added
Sat, 04 Apr 2020 15:31:58 +0100, by Henry S. Thompson
refactored, not tested
Fri, 03 Apr 2020 19:04:06 +0100, by Henry S. Thompson
done through re-extraction, fixing tars still to come
Fri, 03 Apr 2020 17:35:17 +0100, by Henry S. Thompson
sketching more
Thu, 02 Apr 2020 19:21:21 +0100, by Henry S. Thompson
towards re-running extraction in part
Thu, 02 Apr 2020 19:14:23 +0100, by Henry S. Thompson
up the time limit
Thu, 02 Apr 2020 19:13:40 +0100, by Henry S. Thompson
clean up after ourselves
Thu, 02 Apr 2020 19:13:14 +0100, by Henry S. Thompson
fixed scope pblm in tar step
Thu, 26 Mar 2020 15:29:12 +0000, by Henry S. Thompson
sync up filenames and log names,
Thu, 26 Mar 2020 12:24:30 +0000, by Henry S. Thompson
pass through extract args
Thu, 26 Mar 2020 12:23:33 +0000, by Henry S. Thompson

less more | rev 153: (0) -120 tip

Mercurial > hg > cc > cirrus_home

graph