Mercurial > hg > cc > cirrus_work

annotate bin/lmh_warc.py @ 91:460f0599e8cd

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
mostly working, but need to reorder in case of cfid and friends
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Tue, 05 Sep 2023 17:32:46 +0100
parents 120d90b47d74
children 86df63d251cf
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
64
b14187ccfb46 revert to just showing first LM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 55
diff changeset
1 #!/usr/bin/env python3
b14187ccfb46 revert to just showing first LM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 55
diff changeset
2
b14187ccfb46 revert to just showing first LM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 55
diff changeset
3 import re,warc,sys
42
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
4 TUPAT=re.compile(b'^WARC-Target-URI: (.*?)\r',re.MULTILINE)
79
120d90b47d74 include timestamp
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 64
diff changeset
5 DPAT=re.compile(b'^WARC-Date: (.*?)\r',re.MULTILINE)
42
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
6 LMPAT=re.compile(b'^Last-Modified: (.*?)\r',re.MULTILINE)
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
7
79
120d90b47d74 include timestamp
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 64
diff changeset
8 DTAB=bytearray(range(256))
120d90b47d74 include timestamp
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 64
diff changeset
9 DDEL=b'TZ-:'
120d90b47d74 include timestamp
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 64
diff changeset
10
55
11a886a84a49 finds multiples
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 42
diff changeset
11 OUT=open(sys.stdout.fileno(),'wb')
11a886a84a49 finds multiples
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 42
diff changeset
12
42
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
13 def showmeLMH(wtype,buf,part):
79
120d90b47d74 include timestamp
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 64
diff changeset
14 global URI, DATE
42
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
15 if part==1:
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
16 if (m:=TUPAT.search(buf)):
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
17 URI=m[1]
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
18 else:
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
19 raise ValueError(b"No target URI in %s ??"%buf)
79
120d90b47d74 include timestamp
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 64
diff changeset
20 if (md:=DPAT.search(buf)):
120d90b47d74 include timestamp
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 64
diff changeset
21 DATE=md[1]
120d90b47d74 include timestamp
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 64
diff changeset
22 else:
120d90b47d74 include timestamp
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 64
diff changeset
23 raise ValueError(b"No date in %s ??"%buf)
42
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
24 else:
64
b14187ccfb46 revert to just showing first LM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 55
diff changeset
25 mm=LMPAT.search(buf)
42
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
26 OUT.write(URI)
55
11a886a84a49 finds multiples
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 42
diff changeset
27 if mm:
64
b14187ccfb46 revert to just showing first LM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 55
diff changeset
28 OUT.write(b'\t')
79
120d90b47d74 include timestamp
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 64
diff changeset
29 OUT.write(DATE.translate(DTAB,DDEL))
120d90b47d74 include timestamp
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 64
diff changeset
30 OUT.write(b'\t')
64
b14187ccfb46 revert to just showing first LM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 55
diff changeset
31 OUT.write(mm[1])
42
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
32 OUT.write(b'\n')
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
33
64
b14187ccfb46 revert to just showing first LM
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 55
diff changeset
34 warc.warc(sys.argv[1],showmeLMH,[b'response'],parts=3)
42
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
35