Mercurial > hg > cc > work

annotate lurid4/notes.txt @ 155:ee8e50b3af26

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
complete
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Wed, 14 Oct 2026 11:11:05 +0100
parents
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
155
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
1 Using this for 2026 MSc work for the time being, see
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
2 ../lurid3/notes.txt for my own ongoing research
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
3
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
4 See class/msc/starting_work_with_CC.txt
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
5 managingCirrusEnvironment.txt
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
6
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
7 for intro exercises
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
8
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
9 It appears that under sing
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
10 lib/python/plinks
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
11 still works, meaning that /usr/local/src/
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
12 pdfx/.git/config: url = https://github.com/htInEdin/pdfx.git
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
13 pdfminer.six/.git/config: url = https://github.com/htInEdin/pdfminer.six.git
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
14 versions still work together
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
15
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
16 But
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
17
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
18 htInEdin/pdfminer.six Public
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
19 forked from pdfminer/pdfminer.six [private branch preferLoggingToWarning]
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
20 ...
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
21 This branch is 143 commits behind pdfminer/pdfminer.six:master
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
22 [master updated a few months ago]
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
23
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
24 And
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
25
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
26 htInEdin/pdfx Public
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
27 forked from metachris/pdfx [private branch tidy_up_link_extraction]
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
28 This branch is up to date with metachris/pdfx:master [not updated since 2021]
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
29
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
30 Next step -- try updating and rebuilding pdfminer.six and see what happens
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
31 Works in pdfminer.six.upstream, try a merge...
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
32
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
33 Here's part of an answer for a first exercise in parallelism:
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
34
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
35 >: time ls cdx*.gz |parallel -j 4 'printf "%d\\t%s\\n" {#} $(uz {} |wc -l |tee -a /tmp/hst/lci ) ' |cut -f 2 |btot &
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
36 [2] 3482184
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
37 >: 3,618,211,612
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
38
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
39 real 5m41.947s
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
40 user 16m27.802s
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
41 sys 8m22.556s
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
42
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
43 Download more wat files for Phil:
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
44 sing>: cd $CCC
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
45 >: ./getccwat.aws CC-MAIN-2023-40 0 4
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
46 Finished:
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
47 >: ls */orig/wat | fgrep -c .warc.wat.gz
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
48 90,000
ee8e50b3af26 complete
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
49