Mercurial > hg > cc > cirrus_work

annotate lib/python/cc/warc.py @ 344:279793350225 trim

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
decompOneBlock working with gzip
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Wed, 11 Mar 2026 13:24:48 +0000
parents 9ecf193718c1
children 8c9e7578ed30
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
1 #!/usr/bin/env python3
46
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
2 '''Stream a warc format file, unzipping if necessary, invoking a
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
3 callback on each record. Callback can be limited by WARC-Type, record
44d3a4f4ea51 support on-board unzipping, reduce buffer size to 2MB
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 43
diff changeset
4 part'''
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
5
164
4315a36b1672 refactor to provide for buffer overflow fix
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 120
diff changeset
6 import sys, io
42
689a0e311cd2 make warc.py a library, separate out testing
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 39
diff changeset
7 from isal import igzip
289
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
8 import cython, typing, gzip
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
9
287
fe78af4ea7c5 in the midst of trying to rethink the refill logic
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 283
diff changeset
10 # see warc.pxd for function signatures
283
6739e08d19ff type decls, cythonize works
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 164
diff changeset
11
289
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
12 INFO: int = 0
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
13 RESP: int = 1
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
14 REQ: int = 2
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
15 META: int = 3
293
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
16 REVISIT: int = 4
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
17
289
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
18 BUFSIZE: int = 16 * 1024 * 1024
308
d52602a64e09 double buffer size to deal with massive header cases
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 293
diff changeset
19 BUFMIN: int = 3 * 1024 * 1024 # 1.5MiB, will need to be increased
288
d3fc7b5c73d0 park that, try fixed large buffer and large-enough min to ensure we always have a whole record in view
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 287
diff changeset
20 # to 5.5MiB for CC-MAIN-2025-13 (Mar) and thereafter
287
fe78af4ea7c5 in the midst of trying to rethink the refill logic
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 283
diff changeset
21
288
d3fc7b5c73d0 park that, try fixed large buffer and large-enough min to ensure we always have a whole record in view
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 287
diff changeset
22 HDRMAX: int = 0 # will grow
290
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
23 RECORDMAX: int = 0 # will grow
164
4315a36b1672 refactor to provide for buffer overflow fix
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 120
diff changeset
24
287
fe78af4ea7c5 in the midst of trying to rethink the refill logic
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 283
diff changeset
25 def warc(filename: str,
fe78af4ea7c5 in the midst of trying to rethink the refill logic
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 283
diff changeset
26 callback: typing.Callable[[bytes, typing.ByteString, int], typing.BinaryIO],
289
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
27 types: typing.List[int] = [RESP], whole: bool = False, parts: int = 7,
287
fe78af4ea7c5 in the midst of trying to rethink the refill logic
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 283
diff changeset
28 debug: bool = False):
61
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
29 '''parts is a bit-mask:
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
30 1 for warc header;
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
31 2 for req/resp HTTP header, warcinfo/metadata features;
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
32 4 for req/resp body'''
288
d3fc7b5c73d0 park that, try fixed large buffer and large-enough min to ensure we always have a whole record in view
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 287
diff changeset
33 # Not currently trying to depend on this, but I believe that
289
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
34 # warcinfo: record-headers+1bl+crawl-headers+2bl
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
35 # request: record-headers+1bl+HTTP-headers+3bl
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
36 # response: record-headers+1bl+HTTP-headers+[1bl or 2bl]+HTTP-body+1bl
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
37 # metadata: record-headers+1bl+metadata-headers+3bl
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
38 global BUFSIZE, HDRMAX, BUFMIN, RECORDMAX
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
39 _out: typing.BinaryIO
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
40 # should do some sanity checking wrt parts and types
289
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
41 stream: typing.BinaryIO
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
42 with gzip.open(filename, 'r') as fh:
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
43 try:
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
44 fh.read(1)
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
45 except gzip.BadGzipFile:
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
46 stream = open(filename,'rb',0)
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
47 else:
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
48 stream = igzip.IGzipFile(filename=filename)
287
fe78af4ea7c5 in the midst of trying to rethink the refill logic
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 283
diff changeset
49 buf: char[::1] = bytearray(BUFSIZE)
289
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
50 bufView: char[::1] = memoryview(buf)
290
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
51 fpos: int = 0
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
52 bp: int = 0
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
53 bl: int = stream.readinto(buf)
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
54 n: int = 0
289
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
55 done: bool = bl < BUFSIZE
290
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
56 while buf.startswith(b'\r\n',bp):
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
57 bp+=2
293
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
58 while not (done and bp >= bl):
290
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
59 start_1: int = bp
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
60 if not buf.startswith(b'WARC/1.0\r\n',bp):
292
a3d55cc7da18 fix another long-tail bug
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 291
diff changeset
61 breakpoint()
67
b8d4a5ede7a3 fix eof bug, expand error messages
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 66
diff changeset
62 raise ValueError("Not a WARC file? In %s at %s of %s (%s): %s[%s]"%(filename,
b8d4a5ede7a3 fix eof bug, expand error messages
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 66
diff changeset
63 bp,bl,fpos,
b8d4a5ede7a3 fix eof bug, expand error messages
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 66
diff changeset
64 (buf[bp:min(bl,bp+20)] if bp<bl else buf[bl-20:bl]).decode('latin-1'),
b8d4a5ede7a3 fix eof bug, expand error messages
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 66
diff changeset
65 bl-bp))
290
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
66 bp += 10
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
67 n += 1
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
68 wtype: int = -1
287
fe78af4ea7c5 in the midst of trying to rethink the refill logic
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 283
diff changeset
69 length: int = 0
288
d3fc7b5c73d0 park that, try fixed large buffer and large-enough min to ensure we always have a whole record in view
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 287
diff changeset
70 tr: bytes = b'' # Was this record truncated?
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
71 while not buf.startswith(b'\r\n',bp):
56
f8c8f79b2532 rework completely to refill as much as possible only when necessary,
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 54
diff changeset
72 # there should always be enough in the buffer to complete this loop,
289
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
73 # because of the buffer update logic at the end
287
fe78af4ea7c5 in the midst of trying to rethink the refill logic
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 283
diff changeset
74 eol = buf.index(b'\r\n', bp)
289
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
75 if buf.startswith(b"Content-Length: ",bp):
290
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
76 length=wl=int(bufView[bp+16:eol])
287
fe78af4ea7c5 in the midst of trying to rethink the refill logic
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 283
diff changeset
77 if buf.startswith(b"WARC-Truncated: ",bp):
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
78 if bp+16==eol-2:
287
fe78af4ea7c5 in the midst of trying to rethink the refill logic
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 283
diff changeset
79 tr = b"EMPTY"
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
80 else:
287
fe78af4ea7c5 in the midst of trying to rethink the refill logic
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 283
diff changeset
81 tr = bytes(bufView[bp+16:eol-2])
48
d0d2fd9830d6 starting on conversion to direct-querying of buffer
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 46
diff changeset
82 elif buf.startswith(b'WARC-Type: ',bp):
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
83 if buf.startswith(b's',bp+13):
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
84 wtype = RESP
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
85 elif buf.startswith(b'q',bp+13):
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
86 wtype = REQ
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
87 elif buf.startswith(b'm',bp+11):
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
88 wtype = META
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
89 elif buf.startswith(b'w',bp+11):
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
90 wtype = INFO
293
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
91 elif buf.startswith(b'v',bp+13):
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
92 wtype = REVISIT
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
93 else:
67
b8d4a5ede7a3 fix eof bug, expand error messages
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 66
diff changeset
94 raise ValueError("Unknown WARC-Type: %s in %s at %s"%(
b8d4a5ede7a3 fix eof bug, expand error messages
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 66
diff changeset
95 bytes(bufView[bp+11:eol-2]),filename,
59
5d40d7511374 avoid slicing buf by using memoryview to save copying
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 58
diff changeset
96 fpos-(bl-bp)))
290
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
97 bp=eol+2
289
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
98 # record header done
288
d3fc7b5c73d0 park that, try fixed large buffer and large-enough min to ensure we always have a whole record in view
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 287
diff changeset
99 if (hl:=(bp - start_1)) > HDRMAX:
d3fc7b5c73d0 park that, try fixed large buffer and large-enough min to ensure we always have a whole record in view
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 287
diff changeset
100 HDRMAX = hl
289
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
101 #if done:
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
102 # if (bp+length)>bl:
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
103 # raise ValueError("Done but need more! %s + %s > %s in %s"%(bp,
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
104 # length,bl,filename))
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
105 if (wtype in types):
62
11cbaee8bbc8 Test 2 works with parts=1,2,3.
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 61
diff changeset
106 # Output whole or part 1 as required
39
8661062a50b1 moved from home bin
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
107 if whole:
293
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
108 _out=callback(wtype,bufView[start_1:bp+length],7)
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
109 else:
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
110 if (parts & 1):
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
111 bp = eol+2
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
112 _out=callback(wtype,bufView[start_1:bp],1)
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
113 if parts!=1:
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
114 while buf.startswith(b'\r\n',bp):
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
115 bp+=2
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
116 start_2=bp
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
117 eob=bp+length
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
118 while buf.startswith(b'\r\n',eob-2):
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
119 eob-=2
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
120 # Only output parts (2 = HTTP header, 4 = body) that are wanted
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
121 if parts & 2:
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
122 if wtype == RESP or wtype == REQ :
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
123 # request and response have http headers
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
124 eo2=buf.index(b'\r\n\r\n',start_2)
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
125 _out=callback(wtype,bufView[start_2:eo2+2],2)
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
126 else:
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
127 # rest of the part
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
128 _out=callback(wtype,bufView[start_2:eob],2)
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
129 if parts & 4:
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
130 raise ValueError("Not implemented: body part (4): %s"%parts)
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
131 #bp += length
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
132 #if buf[bp] != 13:
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
133 # # Why does this sometimes happen, e.g. when doing
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
134 # python3 ~/lib/python/cc/test_warc.py 4 /beegfs/common_crawl/CC-MAIN-2019-35/1566027313501.0/orig/crawldiagnostics/CC-MAIN-20190817222907-20190818004907-00000.warc.gz
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
135 # at a point where bp+length is 11018, looking at >\n\r\n
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
136 # bp += 1 [doesn't work]
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
137 bp = buf.index(b'\r\n',bp+length)
12d13a1d387f extend, then fix, to get it working for crawldiagnostics warc files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 292
diff changeset
138 # check if refill needed
289
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
139 rl: int
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
140 if (rl := (bp - start_1)) > RECORDMAX:
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
141 RECORDMAX = rl
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
142 keepLen: int
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
143 if (not done) and (keepLen := bl - bp) < BUFMIN:
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
144 # we need to shift and read more
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
145 buf[0:keepLen]=bufView[bp:bl]
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
146 with memoryview(buf)[keepLen:BUFSIZE] as xBuf:
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
147 nb=stream.readinto(xBuf)
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
148 bl = keepLen+nb
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
149 done = bl < BUFSIZE
f17aef7ba4a7 try simpler refill
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 288
diff changeset
150 bp = 0
290
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
151 while buf.startswith(b'\r\n',bp):
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
152 bp+=2
61
f182d09ad1cd whole working
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 60
diff changeset
153 #print('end of loop',wtype,start_1,bp,eol,length,bl,file=sys.stderr)
290
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
154 print('%d records, max record: %d, max header: %d'%(n, RECORDMAX, HDRMAX),
52c9d1875608 simple refill working?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 289
diff changeset
155 file=sys.stderr)
343
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
156
344
279793350225 decompOneBlock working with gzip
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 343
diff changeset
157 import zlib, gzip, struct
279793350225 decompOneBlock working with gzip
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 343
diff changeset
158
343
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
159 def decompOneBlock(fp: io.BytesIO):
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
160 """Decompress one block of a gzip compressed stream in one shot.
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
161 Return the decompressed string and the stream repositioned
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
162 at the start of the next block.
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
163 """
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
164 data = fp.getbuffer()
344
279793350225 decompOneBlock working with gzip
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 343
diff changeset
165 if gzip._read_gzip_header(fp) is None:
343
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
166 return b""
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
167 # Use a zlib raw deflate compressor
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
168 do = zlib.decompressobj(wbits=-zlib.MAX_WBITS)
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
169 # Read all the data except the header
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
170 decompressed = do.decompress(data[fp.tell():])
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
171 if not do.eof or len(do.unused_data) < 8:
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
172 raise EOFError("Compressed file ended before the end-of-stream "
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
173 "marker was reached")
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
174 crc, length = struct.unpack("<II", do.unused_data[:8])
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
175 if crc != zlib.crc32(decompressed):
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
176 raise BadGzipFile("CRC check failed")
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
177 if length != (len(decompressed) & 0xffffffff):
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
178 raise BadGzipFile("Incorrect length of data produced")
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
179 eob = 8
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
180 unused = len(do.unused_data) - 8
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
181 while unused > 0 and do.unused_data[eob] == 0:
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
182 eob += 1
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
183 unused -= 1
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
184 if unused > 0:
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
185 fp.seek(-unused, 2)
9ecf193718c1 process exactly one block at a time, maybe?
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 308
diff changeset
186 return decompressed