Mercurial > hg > cc > cirrus_home

annotate bin/cdx2tsv.py @ 121:863ea87be6bb

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
support field edit
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Mon, 28 Jun 2021 15:40:10 +0000
parents d0b544e53dda
children a76cc0df2754
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
120
d0b544e53dda for use in processing CC index files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
1 #!/usr/bin/env python3
121
863ea87be6bb support field edit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 120
diff changeset
2 '''Extract named fields, with optional post-processing, in order,
863ea87be6bb support field edit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 120
diff changeset
3 from a Common Crawl index row
863ea87be6bb support field edit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 120
diff changeset
4
863ea87be6bb support field edit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 120
diff changeset
5 Field specs on command line are either an atom or
863ea87be6bb support field edit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 120
diff changeset
6 a tuple of atom and expression with
863ea87be6bb support field edit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 120
diff changeset
7 free variable f
863ea87be6bb support field edit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 120
diff changeset
8 For example
863ea87be6bb support field edit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 120
diff changeset
9 > cdx2tsv.py mime '(url,f.split(":",maxsplit=1)[0])' < xyzzy.cdx
863ea87be6bb support field edit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 120
diff changeset
10 will output media type and URI scheme'''
120
d0b544e53dda for use in processing CC index files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
11 import json,sys
d0b544e53dda for use in processing CC index files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
12
d0b544e53dda for use in processing CC index files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
13 fields=sys.argv[1:]
d0b544e53dda for use in processing CC index files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
14
121
863ea87be6bb support field edit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 120
diff changeset
15 fields=[((lambda x,y:(x,eval("lambda f:%s"%y)))(*(f[1:-1].split(',',maxsplit=1))) if f[0]=='(' else f) for f in fields]
120
d0b544e53dda for use in processing CC index files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
16 for l in sys.stdin:
d0b544e53dda for use in processing CC index files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
17 (key,stamp,jj)=l.rstrip().split(' ',maxsplit=2)
d0b544e53dda for use in processing CC index files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
18 ja=json.loads(jj)
121
863ea87be6bb support field edit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 120
diff changeset
19 print('\t'.join((ja.get(f,'NA') if isinstance(f,str) else
863ea87be6bb support field edit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 120
diff changeset
20 ((f[1](ja[f[0]]) if f[0] in ja else 'NA'))) for f in (ja.keys() if fields==["*"] else fields)))
863ea87be6bb support field edit
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 120
diff changeset
21