Mercurial > hg > cc > cirrus_work

annotate bin/warc.py @ 62:11cbaee8bbc8

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Test 2 works with parts=1,2,3. Tests 3 and 4 work; Test 1 works with parts=1, gives correct output for warcinfo and metadata with parts=1,2,3.
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Fri, 14 Jul 2023 17:38:54 +0100
parents f182d09ad1cd
children 75f1d3bc60d9
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
1 #!/usr/bin/env python3
46
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
2 '''Stream a warc format file, unzipping if necessary, invoking a
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
3 callback on each record. Callback can be limited by WARC-Type, record
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
4 part'''
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
5
42
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 39
diff changeset
6 import sys,io
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 39
diff changeset
7 from isal import igzip
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
8
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
9 RESP = b'response'
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
10 REQ = b'request'
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
11 META = b'metadata'
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
12 INFO = b'warcinfo'
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
13
42
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 39
diff changeset
14 def warc(filename,callback,types=['response'],whole=False,parts=7,debug=False):
61
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
15 '''parts is a bit-mask:
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
16 1 for warc header;
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
17 2 for req/resp HTTP header, warcinfo/metadata features;
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
18 4 for req/resp body'''
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
19 # should do some sanity checking wrt parts and types
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
20 types=[(t if isinstance(t,bytes) else bytes(t,'utf8')) for t in types]
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
21 nb=0
46
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
22 if filename.endswith(".gz"):
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
23 stream=igzip.IGzipFile(filename=filename)
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
24 else:
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
25 stream=open(filename,'rb',0)
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
26 bufSize=2*1024*1024
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
27 hdrMax=16*1024
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
28 buf=bytearray(bufSize)
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
29 bufView=memoryview(buf)
56
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
30 fpos=bl=stream.readinto(buf)
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
31 bp=0
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
32 done=False
50
55943918794e a bit better
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 49
diff changeset
33 while True:
56
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
34 while buf.startswith(b'\r\n',bp): # will Fail if buffer (nearly) empty
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
35 bp+=2
56
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
36 start_1=bp
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
37 if not buf.startswith(b'WARC/1.0\r\n',bp):
56
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
38 if done and bl-bp==0:
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
39 # really done
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
40 return
51
c0b4359dd26a working better, gets confused by 3-part response
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 50
diff changeset
41 raise ValueError("Not a WARC file? At %s: %s[%s]"%(bp,
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
42 buf[bp:min(bl,bp+20)].decode('latin-1'), bl-bp))
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
43 bp+=10
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
44 wtype=None
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
45 length=None
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
46 state=1
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
47 tr=None # Was this record truncated?
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
48 while not buf.startswith(b'\r\n',bp):
56
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
49 # there should always be enough in the buffer to complete this loop,
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
50 # because of the buffer update logic below
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
51 eol=buf.index(b'\r\n',bp)+2
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
52 if buf.startswith(b"Content-Length: ",bp):
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
53 length=wl=int(bufView[bp+16:eol-2])
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
54 elif buf.startswith(b"WARC-Truncated: ",bp):
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
55 if bp+16==eol-2:
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
56 tr=b"EMPTY"
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
57 else:
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
58 tr=bytes(bufView[bp+16:eol-2])
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
59 elif buf.startswith(b'WARC-Type: ',bp):
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
60 if buf.startswith(b's',bp+13):
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
61 wtype = RESP
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
62 elif buf.startswith(b'q',bp+13):
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
63 wtype = REQ
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
64 elif buf.startswith(b'm',bp+11):
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
65 wtype = META
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
66 elif buf.startswith(b'w',bp+11):
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
67 wtype = INFO
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
68 else:
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
69 raise ValueError("Unknown WARC-Type: %s at %s"%(
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
70 bytes(bufView[bp+11:eol-2]),
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
71 fpos-(bl-bp)))
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
72 bp=eol
51
c0b4359dd26a working better, gets confused by 3-part response
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 50
diff changeset
73 bp=eol+2
57
61b0a1582af8 works with all types, part=1
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 56
diff changeset
74 if done:
61b0a1582af8 works with all types, part=1
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 56
diff changeset
75 if (bp+length)>bl:
61b0a1582af8 works with all types, part=1
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 56
diff changeset
76 raise ValueError("Done but need more! %s + %s > %s"%(bp,
61b0a1582af8 works with all types, part=1
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 56
diff changeset
77 length,bl))
61b0a1582af8 works with all types, part=1
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 56
diff changeset
78 elif (bp+(length+hdrMax))>bl:
56
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
79 # Need more data
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
80 if wtype in types:
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
81 # we need to keep from start_1 to bl
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
82 keepFrom=start_1
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
83 keepLen=bl-keepFrom
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
84 buf[0:keepLen]=bufView[keepFrom:bl]
60
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
85 eol=eol-start_1
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
86 start_1=0
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
87 bp=eol+2
56
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
88 else:
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
89 # we can skip the rest of this part
60
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
90 if (bp+length)<=bl:
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
91 # we have at least some bytes from the next part
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
92 keepLen=bl-(bp+length)
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
93 buf[0:keepLen]=bufView[bl-keepLen:bl]
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
94 else:
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
95 # we don't have all of the bytes from the current part
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
96 # so can skip the rest of it
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
97 keepLen=0
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
98 fpos=stream.seek(fpos+(bp+length-bl))
7b68c3ebc35a tests 1 & 2 now working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 59
diff changeset
99 bp=0
57
61b0a1582af8 works with all types, part=1
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 56
diff changeset
100 spaceToFill=bufSize-keepLen
61b0a1582af8 works with all types, part=1
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 56
diff changeset
101 with memoryview(buf)[keepLen:bufSize] as xBuf:
56
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
102 nb=stream.readinto(xBuf)
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
103 fpos+=nb
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
104 bl=keepLen+nb
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
105 if nb<spaceToFill:
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
106 done=True
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
107 if wtype not in types:
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
108 continue
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
109 if (wtype in types):
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
110 # Output whole or part 1 as required
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
111 if whole:
61
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
112 bp+=length
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
113 OUT=callback(wtype,bufView[start_1:bp],7)
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
114 continue
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
115 elif (parts & 1):
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
116 OUT=callback(wtype,bufView[start_1:eol],1)
50
55943918794e a bit better
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 49
diff changeset
117 if parts!=1:
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
118 while buf.startswith(b'\r\n',bp):
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
119 bp+=2
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
120 start_2=bp
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
121 eob=bp+length
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
122 while buf.startswith(b'\r\n',eob-2):
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
123 eob-=2
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
124 bv=bufView[start_2:eob]
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
125 # Only output parts (2 = HTTP header, 4 = body) that are wanted
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
126 if parts & 2:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
127 if wtype is META or wtype is INFO:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
128 # rest of the part
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
129 OUT=callback(wtype,bv,2)
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
130 if parts & 4:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
131 for L in rec_text:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
132 if state==2:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
133 # HTTP header
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
134 wl -= len(L)
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
135 if not (L==b"" or L.startswith(b"\r")):
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
136 # Non-empty, it's (a continuation of) a header
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
137 if bl is None and L.startswith(b"Content-Length: "):
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
138 bl=int(L[16:].rstrip())
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
139 else:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
140 # Blank line, HTTP header is finished
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
141 if parts & 2:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
142 callback(wtype,bufView[start_2:start_2+L_start],2)
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
143 state=4
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
144 # The above is just for sanity, because we do _not_
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
145 # continue with the outer loop,
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
146 # since we can now block-output the entire rest of the
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
147 # input buffer.
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
148 if bl is not None:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
149 if bl!=wl:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
150 print("length mismatch: %s %s %s here: %s given: %s trunc: %s"%\
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
151 (length,offset,filename,wl,bl,tr),file=sys.stderr)
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
152 # HTTP body
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
153 balance=start_2+rec_text.tell()
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
154 #print(balance,bl,wl,ll,ll-balance,file=sys.stderr)
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
155 # Output whatever is left
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
156 if parts & 4:
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
157 callback(wtype,bufView[balance:balance+wl],4)
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
158 state=1
61
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
159
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
160 L_start=rec_text.tell()
58
299e3d0f2310 but skip at eobp is not working (with test 2)
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 57
diff changeset
161 bp+=length
61
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
162 #print('end of loop',wtype,start_1,bp,eol,length,bl,file=sys.stderr)
50
55943918794e a bit better
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 49
diff changeset
163 #while not buf.startswith(b'\r\n',bp):
55943918794e a bit better
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 49
diff changeset
164 continue