Mercurial > hg > cc > cirrus_home

annotate bin/cdx2sql.py @ 145:a6d2b299ccdd

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
replace too-complex invocation of cdx2tsv
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Thu, 21 Oct 2021 19:18:47 +0000
parents
children c8e41c543c0b
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
145
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
1 #!/usr/bin/env python3
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
2 '''Replicate part of Jingrui MSc, tabulate a single index
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
3 file info of segment, language, http[s], mime for feeding to sqlite3
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
4
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
5 Borrows from cdx2tsv
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
6
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
7 Usage: cdx2sql.py infiledir i | \
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
8 sqlite3 seg$i.db '.read ../cdx.sql' '.mode tabs' \
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
9 '.import /dev/stdin props' '.quit' 2> seg$i.log ; done &'''
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
10
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
11 import sys, json, io
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
12 from isal import igzip
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
13
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
14 dir=sys.argv[1]
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
15 index=int(sys.argv[2])
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
16
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
17 with igzip.open("%s/cdx-%05.0f.gz"%(dir,index),'rt') as f:
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
18 for l in f:
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
19 (key,stamp,jj)=l.rstrip().split(' ',maxsplit=2)
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
20 ja=json.loads(jj.replace('\\','\\\\')) # some values observed
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
21 # to include \t :-(
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
22 fnf=ja['filename'].split('/',maxsplit=5)
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
23 # Segment number
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
24 seg=int(fnf[3].split('.')[1])
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
25 # Record type (w for warc, r for robots.txt, c for crawl diagnostics)
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
26 wr=fnf[4][0]
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
27 # URI scheme
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
28 sch=int((ja['url'].split(':',maxsplit=1)[0])=='https')
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
29 # Content-type
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
30 m=ja['mime']
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
31 m=m.split('/',maxsplit=1)
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
32 m=(m[0],m[1] if len(m)>1 else '')
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
33 # Sniffed content-type
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
34 md=ja.get('mime-detected','/').split('/',maxsplit=1)
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
35 md=(md[0],md[1] if len(md)>1 else '')
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
36 # Language(s)
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
37 langs=ja.get('languages',None)
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
38 if langs is None:
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
39 langs=('',)
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
40 ll=0
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
41 else:
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
42 langs=langs.split(',')
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
43 ll=len(langs)
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
44 print(seg,wr,sch,'\t'.join(m),
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
45 '\t'.join(md),ll,sep='\t',end='\t')
a6d2b299ccdd replace too-complex invocation of cdx2tsv
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
46 print('\t'.join(langs),'\t'*(3-len(langs)),sep='')