cc/cirrus_home: bin/ix.py annotate

annotate bin/ix.py @ 107:007f35b9df9c

added support for copying to/using /dev/shm or /tmp

author	Henry S. Thompson <ht@inf.ed.ac.uk>
date	Tue, 20 Apr 2021 19:11:57 +0000
parents	815b33c3254a
children	9e5b117dc461

rev	line source
105 baf56ff538f8 convert to rich directory structure per 2019-35 Henry S. Thompson <ht@inf.ed.ac.uk> parents: 104 diff changeset	1 #!/usr/bin/env python3
94 d60073ec798a just strugging with argparse Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	2 '''Extract request records from Common Crawl WARC-format files
105 baf56ff538f8 convert to rich directory structure per 2019-35 Henry S. Thompson <ht@inf.ed.ac.uk> parents: 104 diff changeset	3 given length, offset and filename triples.
94 d60073ec798a just strugging with argparse Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	4 Input one triple on command line, or
d60073ec798a just strugging with argparse Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	5 triples from stdin as tab-delimited lines
d60073ec798a just strugging with argparse Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	6 or complete cdx index lines.
105 baf56ff538f8 convert to rich directory structure per 2019-35 Henry S. Thompson <ht@inf.ed.ac.uk> parents: 104 diff changeset	7 In all cases by 'filename' is meant crawlid/segmentid/filename
94 d60073ec798a just strugging with argparse Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	8
98 1a4c5fdc2923 help format hacking done Henry S. Thompson <ht@inf.ed.ac.uk> parents: 97 diff changeset	9 Note that if no output flag(s) is/are given, the whole WARC record will be output, more efficiently than would be the case if -whb is given.'''
94 d60073ec798a just strugging with argparse Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	10
107 007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	11 import sys, argparse, regex, os, shutil
97 2b880f2ce894 basic help format hacking works Henry S. Thompson <ht@inf.ed.ac.uk> parents: 94 diff changeset	12
98 1a4c5fdc2923 help format hacking done Henry S. Thompson <ht@inf.ed.ac.uk> parents: 97 diff changeset	13 HACK_USAGE=regex.compile('\[-x\]\n\s*\[length\] \[offset\] \[filename\]')
105 baf56ff538f8 convert to rich directory structure per 2019-35 Henry S. Thompson <ht@inf.ed.ac.uk> parents: 104 diff changeset	14 BINOUT=sys.stdout.buffer
107 007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	15 FPAT="/%s/%s/orig/warc/%s"
94 d60073ec798a just strugging with argparse Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	16
97 2b880f2ce894 basic help format hacking works Henry S. Thompson <ht@inf.ed.ac.uk> parents: 94 diff changeset	17 class HackFormat(argparse.RawDescriptionHelpFormatter):
2b880f2ce894 basic help format hacking works Henry S. Thompson <ht@inf.ed.ac.uk> parents: 94 diff changeset	18 def format_help(self):
2b880f2ce894 basic help format hacking works Henry S. Thompson <ht@inf.ed.ac.uk> parents: 94 diff changeset	19 global FOO
2b880f2ce894 basic help format hacking works Henry S. Thompson <ht@inf.ed.ac.uk> parents: 94 diff changeset	20 FOO=argparse.RawDescriptionHelpFormatter.format_help(self)
98 1a4c5fdc2923 help format hacking done Henry S. Thompson <ht@inf.ed.ac.uk> parents: 97 diff changeset	21 return HACK_USAGE.sub('\n [ ( -x \| length offset filename ) ]',
1a4c5fdc2923 help format hacking done Henry S. Thompson <ht@inf.ed.ac.uk> parents: 97 diff changeset	22 FOO)
100 f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	23
107 007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	24 def process(options,buf,root,filename,offset,length,whole):
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	25 rfn=root+filename
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	26 if root!="/beegfs/common_crawl":
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	27 if not os.path.exists(rfn):
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	28 if not os.path.exists(os.path.dirname(rfn)):
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	29 os.makedirs(os.path.dirname(rfn))
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	30 with open('/beegfs/common_crawl'+filename,'rb',0) as infile, \
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	31 open(rfn,'wb',0) as outfile:
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	32 shutil.copyfileobj(infile,outfile,2048*1024)
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	33 # while True:
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	34 # l=infile.readinto(buf)
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	35 # if l is None:
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	36 # break
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	37 # print(l,file=sys.stderr)
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	38 # outfile.write(memoryview(buf)[:l])
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	39 infile.close()
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	40 outfile.close()
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	41 file=open(rfn,'rb',0)
100 f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	42 if whole:
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	43 file.seek(offset)
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	44 bv=memoryview(buf)[:length]
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	45 nb=file.readinto(bv)
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	46 if nb!=length:
104 61122560ae0c -x barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 100 diff changeset	47 print("losing",file.name,length,nb,file=sys.stderr)
105 baf56ff538f8 convert to rich directory structure per 2019-35 Henry S. Thompson <ht@inf.ed.ac.uk> parents: 104 diff changeset	48 BINOUT.write(bv)
104 61122560ae0c -x barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 100 diff changeset	49 file.close()
100 f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	50
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	51 def main():
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	52 parser = argparse.ArgumentParser(
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	53 description='''Extract records from warc files given length, offset and file triples.
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	54 Input one triple on command line, or
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	55 triples from stdin as tab-delimited lines
105 baf56ff538f8 convert to rich directory structure per 2019-35 Henry S. Thompson <ht@inf.ed.ac.uk> parents: 104 diff changeset	56 or complete cdx index lines.
baf56ff538f8 convert to rich directory structure per 2019-35 Henry S. Thompson <ht@inf.ed.ac.uk> parents: 104 diff changeset	57 In all cases by 'filename' is meant crawlid/segmentid/filename''',
100 f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	58 epilog='''Note that if no output flag(s) is/are given,
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	59 the whole WARC record will be output, more efficiently than
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	60 would be the case if all three flags were given.''',
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	61 add_help=False,
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	62 conflict_handler='resolve',
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	63 formatter_class=HackFormat
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	64 )
97 2b880f2ce894 basic help format hacking works Henry S. Thompson <ht@inf.ed.ac.uk> parents: 94 diff changeset	65
100 f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	66 parser.add_argument('--help',help='Show help',action='help')
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	67 parser.add_argument('-d','--debug',help='Debug output',action='store_true')
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	68 parser.add_argument('-w','--warc',help='output WARC headers',
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	69 action='store_true')
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	70 parser.add_argument('-h','--headers',help='output HTTP headers',
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	71 action='store_true')
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	72 parser.add_argument('-b','--body',help='output HTTP body',
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	73 action='store_true')
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	74 parser.add_argument('-c','--cmd',help='pipes each result thru CMD')
107 007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	75 parser.add_argument('-r','--root',nargs='?',
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	76 help='File path root, create a copy there if necessary',
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	77 default='/beegfs/common_crawl'),
100 f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	78 sg=parser.add_mutually_exclusive_group()
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	79 sg.add_argument('-x','--index',
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	80 help='take lines of triples from a cdx index file as input',
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	81 action='store_true')
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	82 sg.add_argument('length',type=int,
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	83 help='length in bytes of gzipped record',
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	84 nargs='?')
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	85 parser.add_argument('offset',type=int,
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	86 help='start position in bytes of gzipped record',
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	87 nargs='?')
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	88 parser.add_argument('filename',
105 baf56ff538f8 convert to rich directory structure per 2019-35 Henry S. Thompson <ht@inf.ed.ac.uk> parents: 104 diff changeset	89 help='pathname of gzipped Common Crawl WARC-format file',
baf56ff538f8 convert to rich directory structure per 2019-35 Henry S. Thompson <ht@inf.ed.ac.uk> parents: 104 diff changeset	90 nargs='?')
100 f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	91 # Hack the order of optional and positional in the help output
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	92 parser._action_groups.sort(key=lambda g:g.title)
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	93 #parser.print_help()
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	94 pa=parser.parse_args(sys.argv[1:])
106 815b33c3254a working with -x and rich directory structure Henry S. Thompson <ht@inf.ed.ac.uk> parents: 105 diff changeset	95 #print(pa,file=sys.stderr)
100 f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	96 if pa.length is not None:
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	97 # We have to enforce our own check..
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	98 if pa.offset is None or pa.filename is None:
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	99 parser.error("length, offset and filename must all be supplied together")
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	100
105 baf56ff538f8 convert to rich directory structure per 2019-35 Henry S. Thompson <ht@inf.ed.ac.uk> parents: 104 diff changeset	101 buf=bytearray(2024*1024)
100 f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	102
104 61122560ae0c -x barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 100 diff changeset	103 whole=not (pa.warc or pa.headers or pa.body)
100 f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	104 if pa.length is not None:
107 007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	105 process(pa,buf,pa.root,FPAT%list(pa.filename.split('/')),
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	106 pa.offset,pa.length,whole)
100 f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	107 exit(0)
104 61122560ae0c -x barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 100 diff changeset	108 if pa.index:
61122560ae0c -x barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 100 diff changeset	109 CDX=regex.compile('length": "([0-9])", "offset": "([0-9])", "filename": "crawl-data/([^/])/segments/([^/])/warc/(.*\.gz)"')
61122560ae0c -x barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 100 diff changeset	110 for l in sys.stdin:
61122560ae0c -x barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 100 diff changeset	111 m=CDX.search(l)
61122560ae0c -x barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 100 diff changeset	112 if m is None:
61122560ae0c -x barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 100 diff changeset	113 print("index line problem: \"%s\""%l.lstrip(),file=sys.stderr)
61122560ae0c -x barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 100 diff changeset	114 exit(2)
105 baf56ff538f8 convert to rich directory structure per 2019-35 Henry S. Thompson <ht@inf.ed.ac.uk> parents: 104 diff changeset	115 f=FPAT%(m[3:6])
107 007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	116 process(pa,buf,pa.root,f,
007f35b9df9c added support for copying to/using /dev/shm or /tmp Henry S. Thompson <ht@inf.ed.ac.uk> parents: 106 diff changeset	117 int(m[2]),int(m[1]),whole)
104 61122560ae0c -x barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 100 diff changeset	118 exit(0)
100 f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	119 if __name__ == "__main__":
f7623dbd8eb5 bare minimum working Henry S. Thompson <ht@inf.ed.ac.uk> parents: 99 diff changeset	120 main()

Mercurial > hg > cc > cirrus_home

annotate bin/ix.py @ 107:007f35b9df9c