cc/cirrus_home: bin/warc.py annotate

annotate bin/warc.py @ 176:97137f5bbe0f

working, about to move to work tree

author	Henry S. Thompson <ht@inf.ed.ac.uk>
date	Wed, 05 Jul 2023 14:50:00 +0100
parents	d123ef7fdb82
children

rev	line source
138 9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	1 #!/usr/bin/env python3
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	2 '''Stream a warc format file, invoking a callback on each record.
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	3 Callback can be limited by WARC-Type, record part'''
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	4 import sys,os,io
138 9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	5
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	6 if (debug:=(sys.argv[1]=='-d')):
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	7 sys.argv.pop(1)
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	8
176 97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	9 def warc(callback,types=['response'],whole=False,parts=7):
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	10 types=[(t if isinstance(t,bytes) else bytes(t,'utf8')) for t in types]
138 9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	11 nb=0
139 e96d444b0f84 fixed bug(s) wrt large payload files Henry S. Thompson <ht@inf.ed.ac.uk> parents: 138 diff changeset	12 stream=open(sys.argv[1],'rb',0)
138 9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	13 bufsize=12810241024
9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	14 buf=bytearray(12810241024)
9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	15 l=b'\r\n'
176 97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	16 while not stream.closed:
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	17 bp=0
138 9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	18 while l==b'\r\n':
9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	19 l=stream.readline()
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	20 nb+=(ln:=len(l))
176 97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	21 if ln==0:
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	22 break
138 9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	23 if l!=b'WARC/1.0\r\n':
9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	24 raise ValueError("Not a WARC file? At %s: %s[%s]"%(nb-len(l),
9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	25 l.decode('latin-1'),len(l)))
9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	26 wtype=None
9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	27 length=None
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	28 state=1
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	29 tr=None # Was this record truncated?
138 9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	30 while l!=b'\r\n':
176 97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	31 # WARC header
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	32 if parts & 1:
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	33 buf[bp:(bp:=bp+ln)]=l
138 9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	34 l=stream.readline()
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	35 nb+=(ln:=len(l))
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	36 if l.startswith(b"Content-Length: "):
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	37 length=wl=int(l[16:].rstrip())
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	38 elif l.startswith(b"WARC-Truncated: "):
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	39 tr=l[16:].rstrip()
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	40 tr="EMPTY" if tr=="" else tr
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	41 elif l.startswith(b'WARC-Type: '):
138 9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	42 wtype = l[11:-2]
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	43 start_2=bp
176 97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	44 if (wtype in types):
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	45 if whole:
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	46 buf[bp:(bp:=bp+ln)]=l
176 97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	47 elif (parts & 1):
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	48 callback(wtype,buf[:start_2],1)
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	49 if parts==1:
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	50 start_2=0
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	51 else:
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	52 start_2=bp
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	53 else:
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	54 start_2=0
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	55 bv=memoryview(buf)[start_2:start_2+length]
138 9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	56 ii=0
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	57 while True and not stream.closed:
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	58 if (i:=stream.readinto(bv))==0:
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	59 break
138 9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	60 ii+=i
9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	61 if ii>=length:
9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	62 break
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	63 bv=memoryview(buf)[start_2+ii:start_2+length]
138 9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	64 if ii!=length:
9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	65 raise ValueError("Chunk read losing, from %s got %s expected %s"%(nb,ii,length))
9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	66 nb+=length
9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	67 if wtype in types:
176 97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	68 if whole:
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	69 callback(wtype,buf[0:start_2+length],7)
138 9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	70 continue
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	71 # Only output parts (1 = WARC header, 2 = HTTP header, 4 = body) that are wanted
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	72 bl=None # for HTTP Content-Length for the length of the body?
176 97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	73 L_start=start_2
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	74 state=2
176 97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	75 bv=memoryview(buf)[start_2:start_2+length]
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	76 with io.BytesIO(bv) as rec_text:
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	77 for L in rec_text:
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	78 if state==2:
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	79 # HTTP header
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	80 wl -= len(L)
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	81 if not (L==b"" or L.startswith(b"\r")):
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	82 # Non-empty, it's (a continuation of) a header
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	83 if bl is None and L.startswith(b"Content-Length: "):
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	84 bl=int(L[16:].rstrip())
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	85 else:
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	86 # Blank line, HTTP header is finished
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	87 if parts & 2:
176 97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	88 callback(wtype,buf[start_2:start_2+L_start],2)
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	89 state=4
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	90 # The above is just for sanity, because we do _not_
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	91 # continue with the outer loop,
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	92 # since we can now block-output the entire rest of the
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	93 # input buffer.
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	94 if bl is not None:
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	95 if bl!=wl:
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	96 print("length mismatch: %s %s %s here: %s given: %s trunc: %s"%\
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	97 (length,offset,filename,wl,bl,tr),file=sys.stderr)
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	98 # HTTP body
176 97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	99 balance=start_2+rec_text.tell()
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	100 #print(balance,bl,wl,ll,ll-balance,file=sys.stderr)
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	101 # Output whatever is left
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	102 if parts & 4:
176 97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	103 callback(wtype,buf[balance:balance+wl],4)
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	104 state=1
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	105
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	106 L_start=rec_text.tell()
138 9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	107 OUT=open(sys.stdout.fileno(),'wb')
9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	108
139 e96d444b0f84 fixed bug(s) wrt large payload files Henry S. Thompson <ht@inf.ed.ac.uk> parents: 138 diff changeset	109 import re
176 97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	110 TUPAT=re.compile(b'^WARC-Target-URI: (.*?)\r',re.MULTILINE)
139 e96d444b0f84 fixed bug(s) wrt large payload files Henry S. Thompson <ht@inf.ed.ac.uk> parents: 138 diff changeset	111 LMPAT=re.compile(b'^Last-Modified: (.*?)\r',re.MULTILINE)
e96d444b0f84 fixed bug(s) wrt large payload files Henry S. Thompson <ht@inf.ed.ac.uk> parents: 138 diff changeset	112
176 97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	113 def showmeLMH(wtype,buf,part):
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	114 global URI
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	115 if part==1:
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	116 if (m:=TUPAT.search(buf)):
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	117 URI=m[1]
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	118 else:
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	119 raise ValueError(b"No target URI in %s ??"%buf)
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	120 else:
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	121 m=LMPAT.search(buf)
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	122 OUT.write(URI)
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	123 if m:
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	124 OUT.write(b'\t')
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	125 OUT.write(m[1])
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	126 OUT.write(b'\n')
139 e96d444b0f84 fixed bug(s) wrt large payload files Henry S. Thompson <ht@inf.ed.ac.uk> parents: 138 diff changeset	127
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	128 def showme(wtype,buf,part):
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	129 if debug:
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	130 breakpoint()
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	131 OUT.write(b"%d\n%b"%(part,buf))
138 9ea12f7b304b just barely working Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	132
176 97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	133 warc(showmeLMH,[b'response'],parts=3)
175 d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	134
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	135 #warc(showme,[b'response','warcinfo','request','metadata'],int(sys.argv[2]))
d123ef7fdb82 working on implementing types and parts: Henry S. Thompson <ht@inf.ed.ac.uk> parents: 139 diff changeset	136
176 97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	137 #warc(showme,[b'response'],parts=int(sys.argv[2]))
97137f5bbe0f working, about to move to work tree Henry S. Thompson <ht@inf.ed.ac.uk> parents: 175 diff changeset	138 #warc(showme,[b'response'],whole=True)

Mercurial > hg > cc > cirrus_home

annotate bin/warc.py @ 176:97137f5bbe0f