Mercurial > hg > cc > cirrus_shared

annotate bin/cdx2tsv.py @ 7:10dfd4feeaa0

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
tidy
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Thu, 11 Jun 2026 21:15:35 +0100
parents ca6349278b82
children c7c72311b731
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
1 #!/usr/bin/env python3
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
2 '''Extract named fields, with optional post-processing, in order,
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
3 from a Common Crawl index row
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
4
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
5 Field specs on command line are either an atom or
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
6 a tuple of atom and expression with
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
7 free variable f
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
8 For example
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
9 > cdx2tsv.py mime '(url,f.split(":",maxsplit=1)[0])' < xyzzy.cdx
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
10 will output media type and URI scheme
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
11
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
12 Note that 'key' and 'timestamp' are also available as global variables,
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
13 so e.g. '(key,key.split(",")[0])' will yield the TLD.
6
ca6349278b82 better usage notes
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 0
diff changeset
14 For a callback instead of printing, include -c cb-mod for a module to import
0
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
15 and within which to call init() once and the cbf function for each index line
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
16 '''
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
17 import json,sys
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
18
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
19 index = cbf = None
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
20 while len(sys.argv)>2 and sys.argv[1][0]=='-':
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
21 if sys.argv[1] == '-i':
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
22 sys.argv.pop(1)
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
23 index=str(int(sys.argv.pop(1)))
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
24 elif sys.argv[1] == '-c':
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
25 sys.argv.pop(1)
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
26 cbmodname = sys.argv.pop(1)
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
27 from importlib import import_module
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
28 cbm = import_module(cbmodname)
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
29 cbf=cbm.cbf
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
30 cbm.init()
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
31 else:
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
32 break
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
33
6
ca6349278b82 better usage notes
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 0
diff changeset
34 if len(sys.argv)==1 or sys.argv[1][0]=='-':
0
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
35 print("""Reads index lines from stdin and extracts values from json dict part
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
36
6
ca6349278b82 better usage notes
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 0
diff changeset
37 Usage: cdx2tsv.py [-i n] [-c callback-module] fieldspecs...
0
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
38
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
39 fieldspec is either a name or a quoted python tuple of a name and
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
40 an expression with free variable f which will be evaluated with f
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
41 having the field value.
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
42
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
43 For example
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
44 cdx2tsv.py mime '(url,f.split(":",maxsplit=1)[0])' < xyzzy.cdx
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
45 will output media type and URI scheme
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
46
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
47 -i n will include n, typically the index number, as 1st column
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
48
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
49 -c cbfile will apply cbfile:cbf to each list of values instead of printing
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
50 [this is probably a better approach than using tuple args with
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
51 lots of tricky lambdas]
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
52 """,
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
53 file=sys.stderr)
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
54 exit(1)
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
55
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
56 fields=sys.argv[1:]
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
57
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
58 stash={} # This can be used to save an expensive field computation for re-use:
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
59 # E.g. '(filename,(lambda g:(stash.__setitem__("T1",g) or g)[3].split(".")[1])(f.split("/",maxsplit=5)))' '(filename,(stash["T1"])[4][0])
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
60
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
61 fields=[((lambda x,y:(x,eval("lambda f:%s"%y,globals())))(*(f[1:-1].split(',',maxsplit=1))) if f[0]=='(' else f) for f in fields]
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
62 for l in sys.stdin:
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
63 (surt,stamp,jj)=l.rstrip().split(' ',maxsplit=2)
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
64 key = surt.split(')')
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
65 ja=json.loads(jj)
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
66 args = ((ja.get(f,'NA') if isinstance(f,str) else
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
67 (f[1](ja[f[0]] if f[0] in ja else 'NA'))) for f in (ja.keys()
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
68 if fields==["*"] else fields))
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
69 if cbf:
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
70 cbf(*args)
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
71 else:
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
72 if index:
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
73 print(index,end='\t')
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
74 print('\t'.join(args))
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
75
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
76 if cbf:
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
77 cbm.finish()
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
78