Mercurial > hg > cc > cirrus_home

annotate bin/ix.py @ 94:d60073ec798a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
just strugging with argparse
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Thu, 15 Apr 2021 19:22:27 +0000
parents
children 2b880f2ce894
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
94
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
1 #!/usr/bin/env python
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
2 '''Extract request records from Common Crawl WARC-format files
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
3 given length, offset and file triples.
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
4 Input one triple on command line, or
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
5 triples from stdin as tab-delimited lines
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
6 or complete cdx index lines.
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
7
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
8 Note that if no output flag(s) is/are given, the whole WARC record will be output, more efficiently than would be the case if all three flags were given.'''
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
9
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
10 import argparse
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
11
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
12 parser = argparse.ArgumentParser(
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
13 description='''Extract records from warc files given length, offset and file triples.
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
14 Input one triple on command line, or
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
15 triples from stdin as tab-delimited lines
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
16 or complete cdx index lines.''',
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
17 epilog='''[ -x | length option file]
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
18
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
19 Note that if no output flag(s) is/are given,
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
20 the whole WARC record will be output, more efficiently than
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
21 would be the case if all three flags were given.''',
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
22 add_help=False,
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
23 conflict_handler='resolve',
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
24 formatter_class=argparse.RawDescriptionHelpFormatter
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
25 )
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
26
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
27 parser.add_argument('--help',help='Show help',action='store_true')
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
28 parser.add_argument('-d','--debug',help='Debug output',action='store_true')
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
29 parser.add_argument('-w','--warc',help='output WARC headers',
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
30 action='store_true')
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
31 parser.add_argument('-h','--headers',help='output HTTP headers',
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
32 action='store_true')
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
33 parser.add_argument('-b','--body',help='output HTTP body',
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
34 action='store_true')
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
35 parser.add_argument('-e','--exec',help='pipes each result thru EXEC')
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
36 sg=parser.add_mutually_exclusive_group()
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
37 sg.add_argument('-x','--index',
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
38 help='take lines of triples from a cdx index file as input',
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
39 action='store_true')
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
40 tg=sg.add_argument_group('triple','explicit triple')
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
41 tg.add_argument('length',type=int,
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
42 help='length in bytes of gzipped record')
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
43 tg.add_argument('offset',type=int,
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
44 help='start position in bytes of gzipped record')
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
45 tg.add_argument('filename',
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
46 help='name of gzipped Common Crawl WARC-format file')
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
47
d60073ec798a just strugging with argparse
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
48 parser.print_help()