view lurid4/notes.txt @ 155:ee8e50b3af26

complete
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Wed, 14 Oct 2026 11:11:05 +0100
parents
children
line wrap: on
line source

Using this for 2026 MSc work for the time being, see
../lurid3/notes.txt for my own ongoing research

See class/msc/starting_work_with_CC.txt
              managingCirrusEnvironment.txt

for intro exercises

It appears that under sing
   lib/python/plinks
still works, meaning that /usr/local/src/
  pdfx/.git/config:       url = https://github.com/htInEdin/pdfx.git
  pdfminer.six/.git/config:       url = https://github.com/htInEdin/pdfminer.six.git
versions still work together

But

  htInEdin/pdfminer.six Public
  forked from pdfminer/pdfminer.six [private branch preferLoggingToWarning]
  ...
  This branch is 143 commits behind pdfminer/pdfminer.six:master
  [master updated a few months ago]

And

  htInEdin/pdfx Public
  forked from metachris/pdfx  [private branch tidy_up_link_extraction]
  This branch is up to date with metachris/pdfx:master [not updated since 2021]

Next step -- try updating and rebuilding pdfminer.six and see what happens
Works in pdfminer.six.upstream, try a merge...

Here's part of an answer for a first exercise in parallelism:

  >: time ls cdx*.gz |parallel -j 4 'printf "%d\\t%s\\n" {#} $(uz {} |wc -l |tee -a /tmp/hst/lci ) ' |cut -f 2 |btot &
[2] 3482184
  >: 3,618,211,612

  real    5m41.947s
  user    16m27.802s
  sys     8m22.556s

Download more wat files for Phil:
  sing>: cd $CCC
  >: ./getccwat.aws CC-MAIN-2023-40 0 4
Finished:
  >: ls */orig/wat | fgrep -c .warc.wat.gz
  90,000