Mercurial > hg > cc > cirrus_shared

annotate bin/cdx2tsv.py @ 9:f6f1deba62af default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
sic
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Thu, 11 Jun 2026 21:16:59 +0100
parents c7c72311b731
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
1 #!/usr/bin/env python3
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
2 '''Extract named fields, with optional post-processing, in order,
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
3 from a Common Crawl index row
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
4
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
5 Field specs on command line are either an atom or
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
6 a tuple of atom and expression with
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
7 free variable f
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
8 For example
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
9 > cdx2tsv.py mime '(url,f.split(":",maxsplit=1)[0])' < xyzzy.cdx
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
10 will output media type and URI scheme
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
11
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
12 Note that 'key' and 'timestamp' are also available as global variables,
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
13 so e.g. '(key,key.split(",")[0])' will yield the TLD.
8
c7c72311b731 fix big comment,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 6
diff changeset
14 For a callback instead of printing, include -c cb-mod.cb-file for
c7c72311b731 fix big comment,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 6
diff changeset
15 a file from a module to import
0
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
16 and within which to call init() once and the cbf function for each index line
8
c7c72311b731 fix big comment,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 6
diff changeset
17 and finally finish()
0
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
18 '''
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
19 import json,sys
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
20
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
21 index = cbf = None
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
22 while len(sys.argv)>2 and sys.argv[1][0]=='-':
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
23 if sys.argv[1] == '-i':
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
24 sys.argv.pop(1)
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
25 index=str(int(sys.argv.pop(1)))
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
26 elif sys.argv[1] == '-c':
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
27 sys.argv.pop(1)
8
c7c72311b731 fix big comment,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 6
diff changeset
28 (cbmodname,cbfilename) = sys.argv.pop(1).split('.')
c7c72311b731 fix big comment,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 6
diff changeset
29 import importlib
c7c72311b731 fix big comment,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 6
diff changeset
30 cbm = importlib.import_module(cbfilename,cbmodname)
0
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
31 cbf=cbm.cbf
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
32 cbm.init()
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
33 else:
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
34 break
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
35
6
ca6349278b82 better usage notes
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 0
diff changeset
36 if len(sys.argv)==1 or sys.argv[1][0]=='-':
0
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
37 print("""Reads index lines from stdin and extracts values from json dict part
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
38
6
ca6349278b82 better usage notes
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 0
diff changeset
39 Usage: cdx2tsv.py [-i n] [-c callback-module] fieldspecs...
0
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
40
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
41 fieldspec is either a name or a quoted python tuple of a name and
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
42 an expression with free variable f which will be evaluated with f
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
43 having the field value.
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
44
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
45 For example
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
46 cdx2tsv.py mime '(url,f.split(":",maxsplit=1)[0])' < xyzzy.cdx
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
47 will output media type and URI scheme
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
48
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
49 -i n will include n, typically the index number, as 1st column
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
50
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
51 -c cbfile will apply cbfile:cbf to each list of values instead of printing
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
52 [this is probably a better approach than using tuple args with
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
53 lots of tricky lambdas]
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
54 """,
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
55 file=sys.stderr)
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
56 exit(1)
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
57
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
58 fields=sys.argv[1:]
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
59
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
60 stash={} # This can be used to save an expensive field computation for re-use:
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
61 # E.g. '(filename,(lambda g:(stash.__setitem__("T1",g) or g)[3].split(".")[1])(f.split("/",maxsplit=5)))' '(filename,(stash["T1"])[4][0])
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
62
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
63 fields=[((lambda x,y:(x,eval("lambda f:%s"%y,globals())))(*(f[1:-1].split(',',maxsplit=1))) if f[0]=='(' else f) for f in fields]
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
64 for l in sys.stdin:
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
65 (surt,stamp,jj)=l.rstrip().split(' ',maxsplit=2)
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
66 key = surt.split(')')
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
67 ja=json.loads(jj)
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
68 args = ((ja.get(f,'NA') if isinstance(f,str) else
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
69 (f[1](ja[f[0]] if f[0] in ja else 'NA'))) for f in (ja.keys()
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
70 if fields==["*"] else fields))
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
71 if cbf:
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
72 cbf(*args)
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
73 else:
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
74 if index:
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
75 print(index,end='\t')
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
76 print('\t'.join(args))
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
77
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
78 if cbf:
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
79 cbm.finish()
a6c7f23cca94 adding callback functionality
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
80