Mercurial > hg > cc > cirrus_work

annotate bin/warc.py @ 109:52c6a9b0fc8c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
loosen must-match criterion in the both-messy case
author Henry Thompson <ht@markup.co.uk>
date Tue, 19 Sep 2023 19:29:41 +0100
parents b8d4a5ede7a3
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
1 #!/usr/bin/env python3
46
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
2 '''Stream a warc format file, unzipping if necessary, invoking a
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
3 callback on each record. Callback can be limited by WARC-Type, record
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
4 part'''
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
5
42
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 39
diff changeset
6 import sys,io
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 39
diff changeset
7 from isal import igzip
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
8
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
9 RESP = b'response'
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
10 REQ = b'request'
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
11 META = b'metadata'
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
12 INFO = b'warcinfo'
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
13
42
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 39
diff changeset
14 def warc(filename,callback,types=['response'],whole=False,parts=7,debug=False):
61
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
15 '''parts is a bit-mask:
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
16 1 for warc header;
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
17 2 for req/resp HTTP header, warcinfo/metadata features;
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
18 4 for req/resp body'''
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
19 # should do some sanity checking wrt parts and types
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
20 types=[(t if isinstance(t,bytes) else bytes(t,'utf8')) for t in types]
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
21 nb=0
46
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
22 if filename.endswith(".gz"):
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
23 stream=igzip.IGzipFile(filename=filename)
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
24 else:
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
25 stream=open(filename,'rb',0)
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
26 bufSize=2*1024*1024
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
27 hdrMax=16*1024
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
28 buf=bytearray(bufSize)
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
29 bufView=memoryview(buf)
56
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
30 fpos=bl=stream.readinto(buf)
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
31 bp=0
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
32 done=False
50
55943918794e a bit better
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 49
diff changeset
33 while True:
67
b8d4a5ede7a3 fix eof bug, expand error messages
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 66
diff changeset
34 while buf.startswith(b'\r\n',bp,bl): # will Fail if buffer (nearly) empty
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
35 bp+=2
56
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
36 start_1=bp
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
37 if not buf.startswith(b'WARC/1.0\r\n',bp):
56
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
38 if done and bl-bp==0:
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
39 # really done
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
40 return
67
b8d4a5ede7a3 fix eof bug, expand error messages
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 66
diff changeset
41 raise ValueError("Not a WARC file? In %s at %s of %s (%s): %s[%s]"%(filename,
b8d4a5ede7a3 fix eof bug, expand error messages
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 66
diff changeset
42 bp,bl,fpos,
b8d4a5ede7a3 fix eof bug, expand error messages
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 66
diff changeset
43 (buf[bp:min(bl,bp+20)] if bp<bl else buf[bl-20:bl]).decode('latin-1'),
b8d4a5ede7a3 fix eof bug, expand error messages
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 66
diff changeset
44 bl-bp))
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
45 bp+=10
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
46 wtype=None
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
47 length=None
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
48 state=1
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
49 tr=None # Was this record truncated?
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
50 while not buf.startswith(b'\r\n',bp):
56
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
51 # there should always be enough in the buffer to complete this loop,
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
52 # because of the buffer update logic below
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
53 eol=buf.index(b'\r\n',bp)+2
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
54 if buf.startswith(b"Content-Length: ",bp):
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
55 length=wl=int(bufView[bp+16:eol-2])
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
56 elif buf.startswith(b"WARC-Truncated: ",bp):
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
57 if bp+16==eol-2:
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
58 tr=b"EMPTY"
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
59 else:
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
60 tr=bytes(bufView[bp+16:eol-2])
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
61 elif buf.startswith(b'WARC-Type: ',bp):
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
62 if buf.startswith(b's',bp+13):
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
63 wtype = RESP
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
64 elif buf.startswith(b'q',bp+13):
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
65 wtype = REQ
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
66 elif buf.startswith(b'm',bp+11):
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
67 wtype = META
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
68 elif buf.startswith(b'w',bp+11):
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
69 wtype = INFO
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
70 else:
67
b8d4a5ede7a3 fix eof bug, expand error messages
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 66
diff changeset
71 raise ValueError("Unknown WARC-Type: %s in %s at %s"%(
b8d4a5ede7a3 fix eof bug, expand error messages
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 66
diff changeset
72 bytes(bufView[bp+11:eol-2]),filename,
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
73 fpos-(bl-bp)))
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
74 bp=eol
51
c0b4359dd26a working better, gets confused by 3-part response
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 50
diff changeset
75 bp=eol+2
57
61b0a1582af8 works with all types, part=1
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 56
diff changeset
76 if done:
61b0a1582af8 works with all types, part=1
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 56
diff changeset
77 if (bp+length)>bl:
67
b8d4a5ede7a3 fix eof bug, expand error messages
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 66
diff changeset
78 raise ValueError("Done but need more! %s + %s > %s in %s"%(bp,
b8d4a5ede7a3 fix eof bug, expand error messages
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 66
diff changeset
79 length,bl,filename))
57
61b0a1582af8 works with all types, part=1
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 56
diff changeset
80 elif (bp+(length+hdrMax))>bl:
56
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
81 # Need more data
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
82 if wtype in types:
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
83 # we need to keep from start_1 to bl
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
84 keepFrom=start_1
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
85 keepLen=bl-keepFrom
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
86 buf[0:keepLen]=bufView[keepFrom:bl]
60
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
87 eol=eol-start_1
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
88 start_1=0
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
89 bp=eol+2
56
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
90 else:
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
91 # we can skip the rest of this part
60
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
92 if (bp+length)<=bl:
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
93 # we have at least some bytes from the next part
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
94 keepLen=bl-(bp+length)
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
95 buf[0:keepLen]=bufView[bl-keepLen:bl]
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
96 else:
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
97 # we don't have all of the bytes from the current part
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
98 # so can skip the rest of it
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
99 keepLen=0
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
100 fpos=stream.seek(fpos+(bp+length-bl))
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
101 bp=0
57
61b0a1582af8 works with all types, part=1
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 56
diff changeset
102 spaceToFill=bufSize-keepLen
61b0a1582af8 works with all types, part=1
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 56
diff changeset
103 with memoryview(buf)[keepLen:bufSize] as xBuf:
56
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
104 nb=stream.readinto(xBuf)
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
105 fpos+=nb
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
106 bl=keepLen+nb
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
107 if nb<spaceToFill:
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
108 done=True
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
109 if wtype not in types:
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
110 continue
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
111 if (wtype in types):
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
112 # Output whole or part 1 as required
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
113 if whole:
61
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
114 bp+=length
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
115 OUT=callback(wtype,bufView[start_1:bp],7)
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
116 continue
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
117 elif (parts & 1):
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
118 OUT=callback(wtype,bufView[start_1:eol],1)
50
55943918794e a bit better
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 49
diff changeset
119 if parts!=1:
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
120 while buf.startswith(b'\r\n',bp):
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
121 bp+=2
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
122 start_2=bp
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
123 eob=bp+length
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
124 while buf.startswith(b'\r\n',eob-2):
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
125 eob-=2
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
126 # Only output parts (2 = HTTP header, 4 = body) that are wanted
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
127 if parts & 2:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
128 if wtype is META or wtype is INFO:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
129 # rest of the part
66
75f1d3bc60d9 part 2 is now working for all types
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 62
diff changeset
130 OUT=callback(wtype,bufView[start_2:eob],2)
75f1d3bc60d9 part 2 is now working for all types
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 62
diff changeset
131 else:
75f1d3bc60d9 part 2 is now working for all types
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 62
diff changeset
132 # request and response have http headers
75f1d3bc60d9 part 2 is now working for all types
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 62
diff changeset
133 eo2=buf.index(b'\r\n\r\n',start_2)
75f1d3bc60d9 part 2 is now working for all types
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 62
diff changeset
134 OUT=callback(wtype,bufView[start_2:eo2+2],2)
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
135 if parts & 4:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
136 for L in rec_text:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
137 if state==2:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
138 # HTTP header
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
139 wl -= len(L)
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
140 if not (L==b"" or L.startswith(b"\r")):
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
141 # Non-empty, it's (a continuation of) a header
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
142 if bl is None and L.startswith(b"Content-Length: "):
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
143 bl=int(L[16:].rstrip())
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
144 else:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
145 # Blank line, HTTP header is finished
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
146 if parts & 2:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
147 callback(wtype,bufView[start_2:start_2+L_start],2)
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
148 state=4
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
149 # The above is just for sanity, because we do _not_
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
150 # continue with the outer loop,
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
151 # since we can now block-output the entire rest of the
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
152 # input buffer.
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
153 if bl is not None:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
154 if bl!=wl:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
155 print("length mismatch: %s %s %s here: %s given: %s trunc: %s"%\
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
156 (length,offset,filename,wl,bl,tr),file=sys.stderr)
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
157 # HTTP body
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
158 balance=start_2+rec_text.tell()
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
159 #print(balance,bl,wl,ll,ll-balance,file=sys.stderr)
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
160 # Output whatever is left
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
161 if parts & 4:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
162 callback(wtype,bufView[balance:balance+wl],4)
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
163 state=1
61
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
164
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
165 L_start=rec_text.tell()
58
299e3d0f2310 but skip at eobp is not working (with test 2)
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 57
diff changeset
166 bp+=length
61
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
167 #print('end of loop',wtype,start_1,bp,eol,length,bl,file=sys.stderr)