annotate bin/percent_encode.py @ 115:0b1e6e134aca

robotstxt and crawldiagnostics get free ride, get rid of DFQ and xq, big simplification and refactor as a result, fix bug in date stream eof handling
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Tue, 26 Sep 2023 17:42:57 +0100
parents 1daa8e444cfe
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
85
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
1 '''Handle unquoting of non-UTF-8 bytes by %-encoding them'''
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
2 import codecs
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
3
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
4 def percent_encode(ude):
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
5 #print(ude.object,ude.object[ude.start:ude.end])
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
6 return (''.join('%%%X'%c for c in ude.object[ude.start:ude.end]),
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
7 ude.end)
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
8
1daa8e444cfe work-around for weird handling of %-encoding in Java impl. of SURT
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
9 codecs.register_error('percent',percent_encode)