Mercurial > hg > cc > cirrus_work
comparison bin/count_warc.py @ 44:083229195d12
just count part length
| author | Henry S. Thompson <ht@inf.ed.ac.uk> |
|---|---|
| date | Wed, 05 Jul 2023 17:51:44 +0100 |
| parents | |
| children |
comparison
equal
deleted
inserted
replaced
| 43:69be1131bcc5 | 44:083229195d12 |
|---|---|
| 1 #!/usr/bin/env python3 | |
| 2 import warc,sys | |
| 3 | |
| 4 OUT=open(sys.stdout.fileno(),'wb') | |
| 5 | |
| 6 if (debug:=(sys.argv[1]=='-d')): | |
| 7 sys.argv.pop(1) | |
| 8 | |
| 9 def countme(wtype,buf,part): | |
| 10 if debug: | |
| 11 breakpoint() | |
| 12 OUT.write(b"%d\n"%len(buf)) | |
| 13 | |
| 14 #warc(showme,[b'response','warcinfo','request','metadata'],int(sys.argv[2])) | |
| 15 #warc(showme,[b'response'],whole=True) | |
| 16 | |
| 17 warc.warc(sys.argv[1],countme,[b'response'],parts=int(sys.argv[2]),debug=debug) |