Mercurial > hg > cc > cirrus_work

annotate bin/sort_date.py @ 91:460f0599e8cd

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
mostly working, but need to reorder in case of cfid and friends
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Tue, 05 Sep 2023 17:32:46 +0100
parents 49faf679d7df
children 135a8c56dcc2
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
73
e8c667bf8965 compute timestamps, key and sort lmh lines
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
1 #!/usr/bin/python3
e8c667bf8965 compute timestamps, key and sort lmh lines
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
2 # Assumes you have used grep -v $'\t' on input for speed
e8c667bf8965 compute timestamps, key and sort lmh lines
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
3 # Recommended to also sed '/GMT$/s/\([^ ]\)GMT$/\1 GMT/'
e8c667bf8965 compute timestamps, key and sort lmh lines
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
4 # to fix a common 'bad' timestamp (~ .2% of inputs)
e8c667bf8965 compute timestamps, key and sort lmh lines
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
5 import email.utils
e8c667bf8965 compute timestamps, key and sort lmh lines
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
6 import sys
85
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
7 from urllib.parse import urlsplit, quote, unquote
83
d92bd8527718 use surt instead of trying to create index term by hand
Henry Thompson <ht@markup.co.uk>
parents: 82
diff changeset
8 from surt import surt
85
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
9
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
10 import re, codecs
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
11
88
49faf679d7df final keystroke fixes, recurse and decimal www stripping
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 86
diff changeset
12 WPAT = re.compile('(,www\\d*)+\\)')
49faf679d7df final keystroke fixes, recurse and decimal www stripping
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 86
diff changeset
13
77
bf09a1d80d7b make CC's own sorting explicit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 76
diff changeset
14 # Thanks to https://stackoverflow.com/a/8776871
bf09a1d80d7b make CC's own sorting explicit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 76
diff changeset
15 import locale
bf09a1d80d7b make CC's own sorting explicit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 76
diff changeset
16 from functools import cmp_to_key
76
eeef811f734d handle corner cases with final . and initial www..+
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 75
diff changeset
17
85
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
18 def percent_encode(ude):
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
19 #print(ude.object,ude.object[ude.start:ude.end])
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
20 return (''.join('%%%X'%c for c in ude.object[ude.start:ude.end]),
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
21 ude.end)
75
177f7df2bf46 handle %-encoded utf-8 as idna
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 73
diff changeset
22
85
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
23 codecs.register_error('percent',percent_encode)
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
24
86
3a2ae6057242 handle double .www, more keep-me chars
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 85
diff changeset
25 # From RFC-3986:
3a2ae6057242 handle double .www, more keep-me chars
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 85
diff changeset
26 # gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@"
3a2ae6057242 handle double .www, more keep-me chars
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 85
diff changeset
27 # sub-delims = "!" / "$" / "&" / "'" / "(" / ")"
3a2ae6057242 handle double .www, more keep-me chars
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 85
diff changeset
28 # / "*" / "+" / "," / ";" / "="
3a2ae6057242 handle double .www, more keep-me chars
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 85
diff changeset
29 # But # _is_ escaped in Java surt results
3a2ae6057242 handle double .www, more keep-me chars
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 85
diff changeset
30 # and additionally " \ : < = > ? \ ^ _ ` { | } are not
3a2ae6057242 handle double .www, more keep-me chars
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 85
diff changeset
31
88
49faf679d7df final keystroke fixes, recurse and decimal www stripping
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 86
diff changeset
32 # Note also that although quote already does _not_ quote - . / _ ~
49faf679d7df final keystroke fixes, recurse and decimal www stripping
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 86
diff changeset
33 # they are included below as that's what we find in surt.surt 0.
49faf679d7df final keystroke fixes, recurse and decimal www stripping
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 86
diff changeset
34
49faf679d7df final keystroke fixes, recurse and decimal www stripping
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 86
diff changeset
35 # Also, Java surt strips _all_ leading 'www\d*.',
86
3a2ae6057242 handle double .www, more keep-me chars
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 85
diff changeset
36 # where python3 surt only strips the first one.
3a2ae6057242 handle double .www, more keep-me chars
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 85
diff changeset
37
85
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
38 def cdx_key(uristring):
86
3a2ae6057242 handle double .www, more keep-me chars
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 85
diff changeset
39 _surt = quote(unquote(surt(uristring),
3a2ae6057242 handle double .www, more keep-me chars
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 85
diff changeset
40 errors='percent'),
88
49faf679d7df final keystroke fixes, recurse and decimal www stripping
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 86
diff changeset
41 safe='''!"$&'()*+,-./:;<=>?@[\]^_`{|}~''' # '
49faf679d7df final keystroke fixes, recurse and decimal www stripping
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 86
diff changeset
42 ).lower()
49faf679d7df final keystroke fixes, recurse and decimal www stripping
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 86
diff changeset
43 return WPAT.sub(')',_surt)
73
e8c667bf8965 compute timestamps, key and sort lmh lines
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
44
e8c667bf8965 compute timestamps, key and sort lmh lines
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
45 def keyed(l):
85
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
46 uri, cc_stamp, dateTime = l.split('\t',2)
77
bf09a1d80d7b make CC's own sorting explicit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 76
diff changeset
47 #print('ul',uri,file=sys.stderr)
73
e8c667bf8965 compute timestamps, key and sort lmh lines
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
48 try:
76
eeef811f734d handle corner cases with final . and initial www..+
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 75
diff changeset
49 try:
eeef811f734d handle corner cases with final . and initial www..+
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 75
diff changeset
50 epoch = email.utils.parsedate_to_datetime(dateTime).timestamp()
eeef811f734d handle corner cases with final . and initial www..+
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 75
diff changeset
51 except OverflowError:
eeef811f734d handle corner cases with final . and initial www..+
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 75
diff changeset
52 epoch = 32535215999.0
85
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
53 return ((cdx_key(uri), cc_stamp), epoch)
75
177f7df2bf46 handle %-encoded utf-8 as idna
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 73
diff changeset
54 except (TypeError,IndexError,ValueError) as e:
73
e8c667bf8965 compute timestamps, key and sort lmh lines
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
55 print(dateTime.rstrip(),e,sep='\t',file=sys.stderr)
e8c667bf8965 compute timestamps, key and sort lmh lines
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
56 return
e8c667bf8965 compute timestamps, key and sort lmh lines
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
57
85
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
58 with open(sys.argv[1],"r") as ff:
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
59 # crucial that the following is done _after_ the file is opened
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
60 # with the default (utf-8) locale!
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
61 locale.setlocale(locale.LC_ALL, "C")
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
62 ctk=cmp_to_key(locale.strcoll)
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
63 for tl in sorted((kk for l in ff if (kk:=keyed(l)) is not None),
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
64 key=lambda x:x[0]):
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 84
diff changeset
65 print(tl[0][0],tl[0][1],tl[1],sep='\t')
77
bf09a1d80d7b make CC's own sorting explicit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 76
diff changeset
66