|
771
|
1 /* Text encoding conversion functions; coding-system object.
|
|
|
2 #### rename me to coding-system.c or coding.c
|
|
428
|
3 Copyright (C) 1991, 1995 Free Software Foundation, Inc.
|
|
|
4 Copyright (C) 1995 Sun Microsystems, Inc.
|
|
1318
|
5 Copyright (C) 2000, 2001, 2002, 2003 Ben Wing.
|
|
428
|
6
|
|
|
7 This file is part of XEmacs.
|
|
|
8
|
|
|
9 XEmacs is free software; you can redistribute it and/or modify it
|
|
|
10 under the terms of the GNU General Public License as published by the
|
|
|
11 Free Software Foundation; either version 2, or (at your option) any
|
|
|
12 later version.
|
|
|
13
|
|
|
14 XEmacs is distributed in the hope that it will be useful, but WITHOUT
|
|
|
15 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
|
|
|
16 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
|
|
|
17 for more details.
|
|
|
18
|
|
|
19 You should have received a copy of the GNU General Public License
|
|
|
20 along with XEmacs; see the file COPYING. If not, write to
|
|
|
21 the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
|
|
|
22 Boston, MA 02111-1307, USA. */
|
|
|
23
|
|
771
|
24 /* Synched up with: Not in FSF. */
|
|
|
25
|
|
|
26 /* Authorship:
|
|
|
27
|
|
|
28 Current primary author: Ben Wing <ben@xemacs.org>
|
|
|
29
|
|
|
30 Rewritten by Ben Wing <ben@xemacs.org>, based originally on coding.c
|
|
|
31 from Mule 2.? but probably does not share one line of code with that
|
|
|
32 original source. Rewriting work started around Dec. 1994. or Jan. 1995.
|
|
|
33 Proceeded in earnest till Nov. 1995.
|
|
|
34
|
|
|
35 Around Feb. 17, 1998, Andy Piper renamed what was then mule-coding.c to
|
|
|
36 file-coding.c, with the intention of using it to do end-of-line conversion
|
|
|
37 on non-MULE machines (specifically, on Windows machines). He separated
|
|
|
38 out the MULE stuff from non-MULE using ifdef's, and searched throughout
|
|
|
39 the rest of the source tree looking for coding-system-related code that
|
|
|
40 was ifdef MULE but should be ifdef HAVE_CODING_SYSTEMS.
|
|
|
41
|
|
|
42 Sept. 4 - 8, 1998, Tomohiko Morioka added the UCS_4 and UTF_8 coding system
|
|
|
43 types, providing a primitive means of decoding and encoding externally-
|
|
|
44 formatted Unicode/UCS_4 and Unicode/UTF_8 data.
|
|
|
45
|
|
|
46 January 25, 2000, Martin Buchholz redid and fleshed out the coding
|
|
|
47 system alias handling that was first added in prototype form by
|
|
|
48 Hrjove Niksic, April 15, 1999.
|
|
|
49
|
|
|
50 April to May 2000, Ben Wing: More major reorganization. Adding features
|
|
|
51 needed for MS Windows (multibyte, unicode, unicode-to-multibyte), the
|
|
|
52 "chain" coding system for chaining two together, and doing a lot of
|
|
|
53 reorganization in preparation for properly abstracting out the different
|
|
|
54 coding system types.
|
|
|
55
|
|
|
56 June 2001, Ben Wing: Added Unicode support. Eliminated previous
|
|
|
57 junky Unicode translation support.
|
|
|
58
|
|
|
59 August 2001, Ben Wing: Moved Unicode support to unicode.c. Finished
|
|
|
60 abstracting everything except detection, which is hard to abstract (see
|
|
|
61 just below).
|
|
|
62
|
|
|
63 September 2001, Ben Wing: Moved Mule code to mule-coding.c, Windows code
|
|
|
64 to intl-win32.c. Lots more rewriting; very little code is untouched
|
|
|
65 from before April 2000. Abstracted the detection code, added multiple
|
|
|
66 levels of likelihood to increase the reliability of the algorithm.
|
|
|
67
|
|
|
68 October 2001, Ben Wing: HAVE_CODING_SYSTEMS is always now defined.
|
|
|
69 Removed the conditionals.
|
|
|
70 */
|
|
|
71
|
|
2297
|
72 /* sjt sez:
|
|
|
73
|
|
|
74 There should be no elementary coding systems in the Lisp API, only chains.
|
|
|
75 Chains should be declared, not computed, as a sequence of coding formats.
|
|
|
76 (Probably the internal representation can be a vector for efficiency but
|
|
|
77 programmers would probably rather work with lists.) A stream has a token
|
|
|
78 type. Most streams are octet streams. Text is a stream of characters (in
|
|
|
79 _internal_ format; a file on disk is not text!) An octet-stream has no
|
|
|
80 implicit semantics, so its format must always be specified. The only type
|
|
|
81 currently having semantics is characters. This means that the chain [euc-jp
|
|
|
82 -> internal -> shift_jis) may be specified (euc-jp, shift_jis), and if no
|
|
|
83 euc-jp -> shift_jis converter is available, then the chain is automatically
|
|
|
84 constructed. (N.B. I f we have fixed width buffers in the future, then we
|
|
|
85 could have ASCII -> 8-bit char -> 16-bit char -> ISO-2022-JP (with escape
|
|
|
86 sequences).
|
|
|
87
|
|
|
88 EOL handling is a char <-> char coding. It should not be part of another
|
|
|
89 coding system except as a convenience for users. For text coding,
|
|
|
90 automatically insert EOL handlers between char <-> octet boundaries.
|
|
|
91 */
|
|
|
92
|
|
771
|
93 /* Comments about future work
|
|
|
94
|
|
|
95 ------------------------------------------------------------------
|
|
|
96 ABOUT DETECTION
|
|
|
97 ------------------------------------------------------------------
|
|
|
98
|
|
|
99 however, in general the detection code has major problems and needs lots
|
|
|
100 of work:
|
|
|
101
|
|
|
102 -- instead of merely "yes" or "no" for particular categories, we need a
|
|
|
103 more flexible system, with various levels of likelihood. Currently
|
|
|
104 I've created a system with six levels, as follows:
|
|
|
105
|
|
|
106 [see file-coding.h]
|
|
|
107
|
|
|
108 Let's consider what this might mean for an ASCII text detector. (In
|
|
|
109 order to have accurate detection, especially given the iteration I
|
|
|
110 proposed below, we need active detectors for *all* types of data we
|
|
|
111 might reasonably encounter, such as ASCII text files, binary files,
|
|
|
112 and possibly other sorts of ASCII files, and not assume that simply
|
|
|
113 "falling back to no detection" will work at all well.)
|
|
|
114
|
|
|
115 An ASCII text detector DOES NOT report ASCII text as level 0, since
|
|
|
116 that's what the detector is looking for. Such a detector ideally
|
|
|
117 wants all bytes in the range 0x20 - 0x7E (no high bytes!), except for
|
|
|
118 whitespace control chars and perhaps a few others; LF, CR, or CRLF
|
|
|
119 sequences at regular intervals (where "regular" might mean an average
|
|
|
120 < 100 chars and 99% < 300 for code and other stuff of the "text file
|
|
|
121 w/line breaks" variety, but for the "text file w/o line breaks"
|
|
|
122 variety, excluding blank lines, averages could easily be 600 or more
|
|
|
123 with 2000-3000 char "lines" not so uncommon); similar statistical
|
|
|
124 variance between odds and evens (not Unicode); frequent occurrences of
|
|
|
125 the space character; letters more common than non-letters; etc. Also
|
|
|
126 checking for too little variability between frequencies of characters
|
|
|
127 and for exclusion of particular characters based on character ranges
|
|
|
128 can catch ASCII encodings like base-64, UUEncode, UTF-7, etc.
|
|
|
129 Granted, this doesn't even apply to everything called "ASCII", and we
|
|
|
130 could potentially distinguish off ASCII for code, ASCII for text,
|
|
|
131 etc. as separate categories. However, it does give us a lot to work
|
|
|
132 off of, in deciding what likelihood to choose -- and it shows there's
|
|
|
133 in fact a lot of detectable patterns to look for even in something
|
|
|
134 seemingly so generic as ASCII. The detector would report most text
|
|
|
135 files in level 1 or level 2. EUC encodings, Shift-JIS, etc. probably
|
|
|
136 go to level -1 because they also pass the EOL test and all other tests
|
|
|
137 for the ASCII part of the text, but have lots of high bytes, which in
|
|
|
138 essence turn them into binary. Aberrant text files like something in
|
|
|
139 BASE64 encoding might get placed in level 0, because they pass most
|
|
|
140 tests but fail dramatically the frequency test; but they should not be
|
|
|
141 reported as any lower, because that would cause explicit prompting,
|
|
|
142 and the user should be able any valid text file without prompting.
|
|
|
143 The escape sequences and the base-64-type checks might send 7-bit
|
|
|
144 iso2022 to 0, but probably not -1, for similar reasons.
|
|
|
145
|
|
|
146 -- The assumed algorithm for the above detection levels is to in essence
|
|
|
147 sort categories first by detection level and then by priority.
|
|
|
148 Perhaps, however, we would want smarter algorithms, or at least
|
|
|
149 something user-controllable -- in particular, when (other than no
|
|
|
150 category at level 0 or greater) do we prompt the user to pick a
|
|
|
151 category?
|
|
|
152
|
|
|
153 -- Improvements in how the detection algorithm works: we want to handle
|
|
|
154 lots of different ways something could be encoded, including multiple
|
|
|
155 stacked encodings. trying to specify a series of detection levels
|
|
|
156 (check for base64 first, then check for gzip, then check for an i18n
|
|
|
157 decoding, then for crlf) won't generally work. for example, what
|
|
|
158 about the same encoding appearing more than once? for example, take
|
|
|
159 euc-jp, base64'd, then gzip'd, then base64'd again: this could well
|
|
|
160 happen, and you could specify the encodings specifically as
|
|
|
161 base64|gzip|base64|euc-jp, but we'd like to autodetect it without
|
|
|
162 worrying about exactly what order these things appear in. we should
|
|
|
163 allow for iterating over detection/decoding cycles until we reach
|
|
|
164 some maximum (we got stuck in a loop, due to incorrect category
|
|
|
165 tables or detection algorithms), have no reported detection levels
|
|
|
166 over -1, or we end up with no change after a decoding pass (i.e. the
|
|
|
167 coding system associated with a chosen category was `no-conversion'
|
|
|
168 or something equivalent). it might make sense to divide things into
|
|
|
169 two phases (internal and external), where the internal phase has a
|
|
|
170 separate category list and would probably mostly end up handling EOL
|
|
|
171 detection; but the i think about it, the more i disagree. with
|
|
|
172 properly written detectors, and properly organized tables (in
|
|
|
173 general, those decodings that are more "distinctive" and thus
|
|
|
174 detectable with greater certainty go lower on the list), we shouldn't
|
|
|
175 need two phases. for example, let's say the example above was also
|
|
|
176 in CRLF format. The EOL detector (which really detects *plain text*
|
|
|
177 with a particular EOL type) would return at most level 0 for all
|
|
|
178 results until the text file is reached, whereas the base64, gzip or
|
|
|
179 euc-jp decoders will return higher. Once the text file is reached,
|
|
|
180 the EOL detector will return 0 or higher for the CRLF encoding, and
|
|
2297
|
181 all other detectors will return 0 or lower; thus, we will successfully
|
|
771
|
182 proceed through CRLF decoding, or at worst prompt the user. (The only
|
|
|
183 external-vs-internal distinction that might make sense here is to
|
|
|
184 favor coding systems of the correct source type over those that
|
|
|
185 require conversion between external and internal; if done right, this
|
|
|
186 could allow the CRLF detector to return level 1 for all CRLF-encoded
|
|
|
187 text files, even those that look like Base-64 or similar encoding, so
|
|
|
188 that CRLF encoding will always get decoded without prompting, but not
|
|
|
189 interfere with other decoders. On the other hand, this
|
|
|
190 external-vs-internal distinction may not matter at all -- with
|
|
|
191 automatic internal-external conversion, CRLF decoding can occur
|
|
|
192 before or after decoding of euc-jp, base64, iso2022, or similar,
|
|
|
193 without any difference in the final results.)
|
|
|
194
|
|
2297
|
195 #### What are we trying to say? In base64, the CRLF decoding before
|
|
|
196 base64 decoding is irrelevant, they will be thrown out as whitespace
|
|
|
197 is not significant in base64.
|
|
|
198
|
|
|
199 [sjt considers all of this to be rather bogus. Ideas like "greater
|
|
|
200 certainty" and "distinctive" can and should be quantified. The issue
|
|
|
201 of proper table organization should be a question of optimization.]
|
|
|
202
|
|
|
203 [sjt wonders if it might not be a good idea to use Unicode's newline
|
|
|
204 character as the internal representation so that (for non-Unicode
|
|
|
205 coding systems) we can catch EOL bugs on Unix too.]
|
|
|
206
|
|
771
|
207 -- There need to be two priority lists and two
|
|
|
208 category->coding-system lists. Once is general, the other
|
|
|
209 category->langenv-specific. The user sets the former, the langenv
|
|
|
210 category->the latter. The langenv-specific entries take precedence
|
|
|
211 category->over the others. This works similarly to the
|
|
|
212 category->category->Unicode charset priority list.
|
|
|
213
|
|
|
214 -- The simple list of coding categories per detectors is not enough.
|
|
|
215 Instead of coding categories, we need parameters. For example,
|
|
|
216 Unicode might have separate detectors for UTF-8, UTF-7, UTF-16,
|
|
|
217 and perhaps UCS-4; or UTF-16/UCS-4 would be one detection type.
|
|
|
218 UTF-16 would have parameters such as "little-endian" and "needs BOM",
|
|
|
219 and possibly another one like "collapse/expand/leave alone composite
|
|
|
220 sequences" once we add this support. Usually these parameters
|
|
|
221 correspond directly to a coding system parameter. Different
|
|
|
222 likelihood values can be specified for each parameter as well as for
|
|
|
223 the detection type as a whole. The user can specify particular
|
|
|
224 coding systems for a particular combination of detection type and
|
|
|
225 parameters, or can give "default parameters" associated with a
|
|
|
226 detection type. In the latter case, we create a new coding system as
|
|
|
227 necessary that corresponds to the detected type and parameters.
|
|
|
228
|
|
|
229 -- a better means of presentation. rather than just coming up
|
|
|
230 with the new file decoded according to the detected coding
|
|
|
231 system, allow the user to browse through the file and
|
|
|
232 conveniently reject it if it looks wrong; then detection
|
|
|
233 starts again, but with that possibility removed. in cases where
|
|
|
234 certainty is low and thus more than one possibility is presented,
|
|
|
235 the user can browse each one and select one or reject them all.
|
|
|
236
|
|
|
237 -- fail-safe: even after the user has made a choice, if they
|
|
|
238 later on realize they have the wrong coding system, they can
|
|
|
239 go back, and we've squirreled away the original data so they
|
|
|
240 can start the process over. this may be tricky.
|
|
|
241
|
|
|
242 -- using a larger buffer for detection. we use just a small
|
|
|
243 piece, which can give quite random results. we may need to
|
|
|
244 buffer up all the data we look through because we can't
|
|
|
245 necessarily rewind. the idea is we proceed until we get a
|
|
|
246 result that's at least at a certain level of certainty
|
|
|
247 (e.g. "probable") or we reached a maximum limit of how much
|
|
|
248 we want to buffer.
|
|
|
249
|
|
|
250 -- dealing with interactive systems. we might need to go ahead
|
|
|
251 and present the data before we've finished detection, and
|
|
|
252 then re-decode it, perhaps multiple times, as we get better
|
|
|
253 detection results.
|
|
|
254
|
|
|
255 -- Clearly some of these are more important than others. at the
|
|
|
256 very least, the "better means of presentation" should be
|
|
2297
|
257 implemented as soon as possible, along with a very simple means
|
|
771
|
258 of fail-safe whenever the data is readibly available, e.g. it's
|
|
|
259 coming from a file, which is the most common scenario.
|
|
|
260
|
|
2297
|
261 --ben [at least that's what sjt thinks]
|
|
|
262
|
|
|
263 *****
|
|
|
264
|
|
|
265 While this is clearly something of an improvement over earlier designs,
|
|
|
266 it doesn't deal with the most important issue: to do better than categories
|
|
|
267 (which in the medium term is mostly going to mean "which flavor of Unicode
|
|
|
268 is this?"), we need to look at statistical behavior rather than ruling out
|
|
|
269 categories via presence of specific sequences. This means the stream
|
|
|
270 processor should
|
|
|
271
|
|
|
272 (1) keep octet distributions (octet, 2-, 3-, 4- octet sequences)
|
|
|
273 (2) in some kind of compressed form
|
|
|
274 (3) look for "skip features" (eg, characteristic behavior of leading
|
|
|
275 bytes for UTF-7, UTF-8, UTF-16, Mule code)
|
|
|
276 (4) pick up certain "simple" regexps
|
|
|
277 (5) provide "triggers" to determine when statistical detectors should be
|
|
|
278 invoked, such as octet count
|
|
|
279 (6) and "magic" like Unicode signatures or file(1) magic.
|
|
|
280
|
|
|
281 --sjt
|
|
|
282
|
|
771
|
283
|
|
|
284 ------------------------------------------------------------------
|
|
|
285 ABOUT FORMATS
|
|
|
286 ------------------------------------------------------------------
|
|
|
287
|
|
|
288 when calling make-coding-system, the name can be a cons of (format1 .
|
|
|
289 format2), specifying that it decodes format1->format2 and encodes the other
|
|
|
290 way. if only one name is given, that is assumed to be format1, and the
|
|
|
291 other is either `external' or `internal' depending on the end type.
|
|
|
292 normally the user when decoding gives the decoding order in formats, but
|
|
|
293 can leave off the last one, `internal', which is assumed. a multichain
|
|
|
294 might look like gzip|multibyte|unicode, using the coding systems named
|
|
|
295 `gzip', `(unicode . multibyte)' and `unicode'. the way this actually works
|
|
|
296 is by searching for gzip->multibyte; if not found, look for gzip->external
|
|
|
297 or gzip->internal. (In general we automatically do conversion between
|
|
|
298 internal and external as necessary: thus gzip|crlf does the expected, and
|
|
|
299 maps to gzip->external, external->internal, crlf->internal, which when
|
|
|
300 fully specified would be gzip|external:external|internal:crlf|internal --
|
|
|
301 see below.) To forcibly fit together two converters that have explicitly
|
|
|
302 specified and incompatible names (say you have unicode->multibyte and
|
|
|
303 iso8859-1->ebcdic and you know that the multibyte and iso8859-1 in this
|
|
|
304 case are compatible), you can force-cast using :, like this:
|
|
|
305 ebcdic|iso8859-1:multibyte|unicode. (again, if you force-cast between
|
|
|
306 internal and external formats, the conversion happens automatically.)
|
|
|
307
|
|
|
308 --------------------------------------------------------------------------
|
|
|
309 ABOUT PDUMP, UNICODE, AND RUNNING XEMACS FROM A DIRECTORY WITH WEIRD CHARS
|
|
|
310 --------------------------------------------------------------------------
|
|
|
311
|
|
|
312 -- there's the problem that XEmacs can't be run in a directory with
|
|
|
313 non-ASCII/Latin-1 chars in it, since it will be doing Unicode
|
|
|
314 processing before we've had a chance to load the tables. In fact,
|
|
|
315 even finding the tables in such a situation is problematic using
|
|
|
316 the normal commands. my idea is to eventually load the stuff
|
|
|
317 extremely extremely early, at the same time as the pdump data gets
|
|
|
318 loaded. in fact, the unicode table data (stored in an efficient
|
|
|
319 binary format) can even be stuck into the pdump file (which would
|
|
|
320 mean as a resource to the executable, for windows). we'd need to
|
|
|
321 extend pdump a bit: to allow for attaching extra data to the pdump
|
|
|
322 file. (something like pdump_attach_extra_data (addr, length)
|
|
|
323 returns a number of some sort, an index into the file, which you
|
|
|
324 can then retrieve with pdump_load_extra_data(), which returns an
|
|
|
325 addr (mmap()ed or loaded), and later you pdump_unload_extra_data()
|
|
|
326 when finished. we'd probably also need
|
|
|
327 pdump_attach_extra_data_append(), which appends data to the data
|
|
|
328 just written out with pdump_attach_extra_data(). this way,
|
|
|
329 multiple tables in memory can be written out into one contiguous
|
|
|
330 table. (we'd use the tar-like trick of allowing new blocks to be
|
|
|
331 written without going back to change the old blocks -- we just rely
|
|
|
332 on the end of file/end of memory.) this same mechanism could be
|
|
|
333 extracted out of pdump and used to handle the non-pdump situation
|
|
|
334 (or alternatively, we could just dump either the memory image of
|
|
|
335 the tables themselves or the compressed binary version). in the
|
|
|
336 case of extra unicode tables not known about at compile time that
|
|
|
337 get loaded before dumping, we either just dump them into the image
|
|
|
338 (pdump and all) or extract them into the compressed binary format,
|
|
|
339 free the original tables, and treat them like all other tables.
|
|
|
340
|
|
|
341 --------------------------------------------------------------------------
|
|
|
342 HANDLING WRITING A FILE SAFELY, WITHOUT DATA LOSS
|
|
|
343 --------------------------------------------------------------------------
|
|
|
344
|
|
|
345 -- When writing a file, we need error detection; otherwise somebody
|
|
|
346 will create a Unicode file without realizing the coding system
|
|
|
347 of the buffer is Raw, and then lose all the non-ASCII/Latin-1
|
|
|
348 text when it's written out. We need two levels
|
|
|
349
|
|
|
350 1. first, a "safe-charset" level that checks before any actual
|
|
|
351 encoding to see if all characters in the document can safely
|
|
|
352 be represented using the given coding system. FSF has a
|
|
|
353 "safe-charset" property of coding systems, but it's stupid
|
|
|
354 because this information can be automatically derived from
|
|
|
355 the coding system, at least the vast majority of the time.
|
|
|
356 What we need is some sort of
|
|
|
357 alternative-coding-system-precedence-list, langenv-specific,
|
|
|
358 where everything on it can be checked for safe charsets and
|
|
|
359 then the user given a list of possibilities. When the user
|
|
|
360 does "save with specified encoding", they should see the same
|
|
|
361 precedence list. Again like with other precedence lists,
|
|
|
362 there's also a global one, and presumably all coding systems
|
|
|
363 not on other list get appended to the end (and perhaps not
|
|
|
364 checked at all when doing safe-checking?). safe-checking
|
|
|
365 should work something like this: compile a list of all
|
|
|
366 charsets used in the buffer, along with a count of chars
|
|
2297
|
367 used. that way, "slightly unsafe" coding systems can perhaps
|
|
|
368 be presented at the end, which will lose only a few characters
|
|
771
|
369 and are perhaps what the users were looking for.
|
|
|
370
|
|
2297
|
371 [sjt sez this whole step is a crock. If a universal coding system
|
|
|
372 is unacceptable, the user had better know what he/she is doing,
|
|
|
373 and explicitly specify a lossy encoding.
|
|
|
374 In principle, we can simply check for characters being writable as
|
|
|
375 we go along. Eg, via an "unrepresentable character handler." We
|
|
|
376 still have the buffer contents. If we can't successfully save,
|
|
|
377 then ask the user what to do. (Do we ever simply destroy previous
|
|
|
378 file version before completing a write?)]
|
|
|
379
|
|
771
|
380 2. when actually writing out, we need error checking in case an
|
|
|
381 individual char in a charset can't be written even though the
|
|
|
382 charsets are safe. again, the user gets the choice of other
|
|
|
383 reasonable coding systems.
|
|
|
384
|
|
2297
|
385 [sjt -- something is very confused, here; safe charsets should be
|
|
|
386 defined as those charsets all of whose characters can be encoded.]
|
|
|
387
|
|
771
|
388 3. same thing (error checking, list of alternatives, etc.) needs
|
|
|
389 to happen when reading! all of this will be a lot of work!
|
|
|
390
|
|
|
391
|
|
|
392 --ben
|
|
2297
|
393
|
|
|
394 I don't much like Ben's scheme. First, this isn't an issue of I/O,
|
|
|
395 it's a coding issue. It can happen in many places, not just on stream
|
|
|
396 I/O. Error checking should take place on all translations. Second,
|
|
|
397 the two-pass algorithm should be avoided if possible. In some cases
|
|
|
398 (eg, output to a tty) we won't be able to go back and change the
|
|
|
399 previously output data. Third, the whole idea of having a buffer full
|
|
|
400 of arbitrary characters which we're going to somehow shoehorn into a
|
|
|
401 file based on some twit user's less than informed idea of a coding system
|
|
|
402 is kind of laughable from the start. If we're going to say that a buffer
|
|
|
403 has a coding system, shouldn't we enforce restrictions on what you can
|
|
|
404 put into it? Fourth, what's the point of having safe charsets if some
|
|
|
405 of the characters in them are unsafe? Fifth, what makes you think we're
|
|
|
406 going to have a list of charsets? It seems to me that there might be
|
|
|
407 reasons to have user-defined charsets (eg, "German" vs "French" subsets
|
|
|
408 of ISO 8859/15). Sixth, the idea of having language environment determine
|
|
|
409 precedence doesn't seem very useful to me. Users who are working with a
|
|
|
410 language that corresponds to the language environment are not going to
|
|
|
411 run into safe charsets problems. It's users who are outside of their
|
|
|
412 usual language environment who run into trouble. Also, the reason for
|
|
|
413 specifying anything other than a universal coding system is normally
|
|
|
414 restrictions imposed by other users or applications. Seventh, the
|
|
|
415 statistical feedback isn't terribly useful. Users rarely "want" a
|
|
|
416 coding system, they want their file saved in a useful way. We could
|
|
|
417 add a FORCE argument to conversions for those who really want a specific
|
|
|
418 coding system. But mostly, a user might want to edit out a few unsafe
|
|
|
419 characters. So (up to some maximum) we should keep a list of unsafe
|
|
|
420 text positions, and provide a convenient function for traversing them.
|
|
|
421
|
|
|
422 --sjt
|
|
771
|
423 */
|
|
428
|
424
|
|
|
425 #include <config.h>
|
|
|
426 #include "lisp.h"
|
|
|
427
|
|
|
428 #include "buffer.h"
|
|
|
429 #include "elhash.h"
|
|
|
430 #include "insdel.h"
|
|
|
431 #include "lstream.h"
|
|
440
|
432 #include "opaque.h"
|
|
771
|
433 #include "file-coding.h"
|
|
|
434
|
|
|
435 #ifdef HAVE_ZLIB
|
|
|
436 #include "zlib.h"
|
|
428
|
437 #endif
|
|
|
438
|
|
|
439 Lisp_Object Vkeyboard_coding_system;
|
|
|
440 Lisp_Object Vterminal_coding_system;
|
|
|
441 Lisp_Object Vcoding_system_for_read;
|
|
|
442 Lisp_Object Vcoding_system_for_write;
|
|
|
443 Lisp_Object Vfile_name_coding_system;
|
|
|
444
|
|
771
|
445 #ifdef DEBUG_XEMACS
|
|
|
446 Lisp_Object Vdebug_coding_detection;
|
|
440
|
447 #endif
|
|
771
|
448
|
|
|
449 typedef struct coding_system_type_entry
|
|
|
450 {
|
|
|
451 struct coding_system_methods *meths;
|
|
|
452 } coding_system_type_entry;
|
|
|
453
|
|
|
454 typedef struct
|
|
|
455 {
|
|
|
456 Dynarr_declare (coding_system_type_entry);
|
|
|
457 } coding_system_type_entry_dynarr;
|
|
|
458
|
|
|
459 static coding_system_type_entry_dynarr *the_coding_system_type_entry_dynarr;
|
|
|
460
|
|
1204
|
461 static const struct memory_description cste_description_1[] = {
|
|
771
|
462 { XD_STRUCT_PTR, offsetof (coding_system_type_entry, meths), 1, &coding_system_methods_description },
|
|
|
463 { XD_END }
|
|
|
464 };
|
|
|
465
|
|
1204
|
466 static const struct sized_memory_description cste_description = {
|
|
771
|
467 sizeof (coding_system_type_entry),
|
|
|
468 cste_description_1
|
|
|
469 };
|
|
|
470
|
|
1204
|
471 static const struct memory_description csted_description_1[] = {
|
|
771
|
472 XD_DYNARR_DESC (coding_system_type_entry_dynarr, &cste_description),
|
|
428
|
473 { XD_END }
|
|
|
474 };
|
|
|
475
|
|
1204
|
476 static const struct sized_memory_description csted_description = {
|
|
771
|
477 sizeof (coding_system_type_entry_dynarr),
|
|
|
478 csted_description_1
|
|
|
479 };
|
|
|
480
|
|
|
481 static Lisp_Object Vcoding_system_type_list;
|
|
|
482
|
|
|
483 /* Coding system currently associated with each coding category. */
|
|
|
484 Lisp_Object coding_category_system[MAX_DETECTOR_CATEGORIES];
|
|
|
485
|
|
|
486 /* Table of all coding categories in decreasing order of priority.
|
|
|
487 This describes a permutation of the possible coding categories. */
|
|
|
488 int coding_category_by_priority[MAX_DETECTOR_CATEGORIES];
|
|
|
489
|
|
|
490 /* Value used with to give a unique name to nameless coding systems */
|
|
|
491 int coding_system_tick;
|
|
|
492
|
|
|
493 int coding_detector_count;
|
|
|
494 int coding_detector_category_count;
|
|
|
495
|
|
|
496 detector_dynarr *all_coding_detectors;
|
|
|
497
|
|
1204
|
498 static const struct memory_description struct_detector_category_description_1[]
|
|
771
|
499 =
|
|
|
500 {
|
|
|
501 { XD_LISP_OBJECT, offsetof (struct detector_category, sym) },
|
|
|
502 { XD_END }
|
|
|
503 };
|
|
|
504
|
|
1204
|
505 static const struct sized_memory_description struct_detector_category_description =
|
|
771
|
506 {
|
|
|
507 sizeof (struct detector_category),
|
|
|
508 struct_detector_category_description_1
|
|
428
|
509 };
|
|
|
510
|
|
1204
|
511 static const struct memory_description detector_category_dynarr_description_1[] =
|
|
771
|
512 {
|
|
|
513 XD_DYNARR_DESC (detector_category_dynarr,
|
|
|
514 &struct_detector_category_description),
|
|
|
515 { XD_END }
|
|
|
516 };
|
|
|
517
|
|
1204
|
518 static const struct sized_memory_description detector_category_dynarr_description = {
|
|
771
|
519 sizeof (detector_category_dynarr),
|
|
|
520 detector_category_dynarr_description_1
|
|
|
521 };
|
|
|
522
|
|
1204
|
523 static const struct memory_description struct_detector_description_1[]
|
|
771
|
524 =
|
|
|
525 {
|
|
|
526 { XD_STRUCT_PTR, offsetof (struct detector, cats), 1,
|
|
|
527 &detector_category_dynarr_description },
|
|
|
528 { XD_END }
|
|
|
529 };
|
|
|
530
|
|
1204
|
531 static const struct sized_memory_description struct_detector_description =
|
|
771
|
532 {
|
|
|
533 sizeof (struct detector),
|
|
|
534 struct_detector_description_1
|
|
|
535 };
|
|
|
536
|
|
1204
|
537 static const struct memory_description detector_dynarr_description_1[] =
|
|
771
|
538 {
|
|
|
539 XD_DYNARR_DESC (detector_dynarr, &struct_detector_description),
|
|
|
540 { XD_END }
|
|
|
541 };
|
|
|
542
|
|
1204
|
543 static const struct sized_memory_description detector_dynarr_description = {
|
|
771
|
544 sizeof (detector_dynarr),
|
|
|
545 detector_dynarr_description_1
|
|
|
546 };
|
|
428
|
547
|
|
|
548 Lisp_Object Qcoding_systemp;
|
|
|
549
|
|
771
|
550 Lisp_Object Qraw_text;
|
|
428
|
551
|
|
|
552 Lisp_Object Qmnemonic, Qeol_type;
|
|
|
553 Lisp_Object Qcr, Qcrlf, Qlf;
|
|
|
554 Lisp_Object Qeol_cr, Qeol_crlf, Qeol_lf;
|
|
|
555 Lisp_Object Qpost_read_conversion;
|
|
|
556 Lisp_Object Qpre_write_conversion;
|
|
|
557
|
|
771
|
558 Lisp_Object Qtranslation_table_for_decode;
|
|
|
559 Lisp_Object Qtranslation_table_for_encode;
|
|
|
560 Lisp_Object Qsafe_chars;
|
|
|
561 Lisp_Object Qsafe_charsets;
|
|
|
562 Lisp_Object Qmime_charset;
|
|
|
563 Lisp_Object Qvalid_codes;
|
|
|
564
|
|
|
565 Lisp_Object Qno_conversion;
|
|
|
566 Lisp_Object Qconvert_eol;
|
|
440
|
567 Lisp_Object Qescape_quoted;
|
|
771
|
568 Lisp_Object Qencode, Qdecode;
|
|
|
569
|
|
|
570 Lisp_Object Qconvert_eol_lf, Qconvert_eol_cr, Qconvert_eol_crlf;
|
|
|
571 Lisp_Object Qconvert_eol_autodetect;
|
|
|
572
|
|
|
573 Lisp_Object Qnear_certainty, Qquite_probable, Qsomewhat_likely;
|
|
1494
|
574 Lisp_Object Qslightly_likely;
|
|
771
|
575 Lisp_Object Qas_likely_as_unlikely, Qsomewhat_unlikely, Qquite_improbable;
|
|
|
576 Lisp_Object Qnearly_impossible;
|
|
|
577
|
|
|
578 Lisp_Object Qdo_eol, Qdo_coding;
|
|
|
579
|
|
|
580 Lisp_Object Qcanonicalize_after_coding;
|
|
|
581
|
|
1347
|
582 Lisp_Object QScoding_system_cookie;
|
|
|
583
|
|
771
|
584 /* This is used to convert autodetected coding systems into existing
|
|
|
585 systems. For example, the chain undecided->convert-eol-autodetect may
|
|
|
586 have its separate parts detected as mswindows-multibyte and
|
|
|
587 convert-eol-crlf, and the result needs to be mapped to
|
|
|
588 mswindows-multibyte-dos. */
|
|
|
589 /* #### It's not clear we need this whole chain-canonicalize mechanism
|
|
|
590 any more. */
|
|
|
591 static Lisp_Object Vchain_canonicalize_hash_table;
|
|
|
592
|
|
|
593 #ifdef HAVE_ZLIB
|
|
|
594 Lisp_Object Qgzip;
|
|
428
|
595 #endif
|
|
771
|
596
|
|
2297
|
597 /* Maps symbols (coding system names) to either coding system objects or
|
|
|
598 (for aliases) other names. */
|
|
771
|
599 static Lisp_Object Vcoding_system_hash_table;
|
|
428
|
600
|
|
|
601 int enable_multibyte_characters;
|
|
|
602
|
|
|
603 EXFUN (Fcopy_coding_system, 2);
|
|
|
604
|
|
|
605
|
|
|
606 /************************************************************************/
|
|
771
|
607 /* Coding system object methods */
|
|
428
|
608 /************************************************************************/
|
|
|
609
|
|
|
610 static Lisp_Object
|
|
|
611 mark_coding_system (Lisp_Object obj)
|
|
|
612 {
|
|
|
613 Lisp_Coding_System *codesys = XCODING_SYSTEM (obj);
|
|
|
614
|
|
1204
|
615 #define MARKED_SLOT(x) mark_object (codesys->x);
|
|
|
616 #include "coding-system-slots.h"
|
|
771
|
617
|
|
|
618 MAYBE_CODESYSMETH (codesys, mark, (obj));
|
|
428
|
619
|
|
1204
|
620 return Qnil;
|
|
428
|
621 }
|
|
|
622
|
|
|
623 static void
|
|
771
|
624 print_coding_system_properties (Lisp_Object obj, Lisp_Object printcharfun)
|
|
|
625 {
|
|
|
626 Lisp_Coding_System *c = XCODING_SYSTEM (obj);
|
|
|
627 print_internal (c->methods->type, printcharfun, 1);
|
|
|
628 MAYBE_CODESYSMETH (c, print, (obj, printcharfun, 1));
|
|
|
629 if (CODING_SYSTEM_EOL_TYPE (c) != EOL_AUTODETECT)
|
|
|
630 write_fmt_string_lisp (printcharfun, " eol-type=%s",
|
|
|
631 1, Fcoding_system_property (obj, Qeol_type));
|
|
|
632 }
|
|
|
633
|
|
|
634 static void
|
|
428
|
635 print_coding_system (Lisp_Object obj, Lisp_Object printcharfun,
|
|
2286
|
636 int UNUSED (escapeflag))
|
|
428
|
637 {
|
|
|
638 Lisp_Coding_System *c = XCODING_SYSTEM (obj);
|
|
|
639 if (print_readably)
|
|
771
|
640 printing_unreadable_object
|
|
|
641 ("printing unreadable object #<coding-system 0x%x>", c->header.uid);
|
|
|
642
|
|
|
643 write_fmt_string_lisp (printcharfun, "#<coding-system %s ", 1, c->name);
|
|
|
644 print_coding_system_properties (obj, printcharfun);
|
|
826
|
645 write_c_string (printcharfun, ">");
|
|
428
|
646 }
|
|
|
647
|
|
771
|
648 /* Print an abbreviated version of a coding system (but still containing
|
|
|
649 all the information), for use within a coding system print method. */
|
|
|
650
|
|
|
651 static void
|
|
|
652 print_coding_system_in_print_method (Lisp_Object cs, Lisp_Object printcharfun,
|
|
2286
|
653 int UNUSED (escapeflag))
|
|
771
|
654 {
|
|
800
|
655 write_fmt_string_lisp (printcharfun, "%s[", 1, XCODING_SYSTEM_NAME (cs));
|
|
771
|
656 print_coding_system_properties (cs, printcharfun);
|
|
826
|
657 write_c_string (printcharfun, "]");
|
|
771
|
658 }
|
|
|
659
|
|
428
|
660 static void
|
|
|
661 finalize_coding_system (void *header, int for_disksave)
|
|
|
662 {
|
|
771
|
663 Lisp_Object cs = wrap_coding_system ((Lisp_Coding_System *) header);
|
|
428
|
664 /* Since coding systems never go away, this function is not
|
|
|
665 necessary. But it would be necessary if we changed things
|
|
|
666 so that coding systems could go away. */
|
|
|
667 if (!for_disksave) /* see comment in lstream.c */
|
|
771
|
668 MAYBE_XCODESYSMETH (cs, finalize, (cs));
|
|
|
669 }
|
|
|
670
|
|
|
671 static Bytecount
|
|
|
672 sizeof_coding_system (const void *header)
|
|
|
673 {
|
|
|
674 const Lisp_Coding_System *p = (const Lisp_Coding_System *) header;
|
|
|
675 return offsetof (Lisp_Coding_System, data) + p->methods->extra_data_size;
|
|
428
|
676 }
|
|
|
677
|
|
1204
|
678 static const struct memory_description coding_system_methods_description_1[]
|
|
771
|
679 = {
|
|
|
680 { XD_LISP_OBJECT,
|
|
|
681 offsetof (struct coding_system_methods, type) },
|
|
|
682 { XD_LISP_OBJECT,
|
|
|
683 offsetof (struct coding_system_methods, predicate_symbol) },
|
|
|
684 { XD_END }
|
|
|
685 };
|
|
|
686
|
|
1204
|
687 const struct sized_memory_description coding_system_methods_description = {
|
|
771
|
688 sizeof (struct coding_system_methods),
|
|
|
689 coding_system_methods_description_1
|
|
|
690 };
|
|
|
691
|
|
1204
|
692 static const struct sized_memory_description coding_system_extra_description_map[] =
|
|
|
693 {
|
|
|
694 { offsetof (Lisp_Coding_System, methods) },
|
|
|
695 { offsetof (struct coding_system_methods, extra_description) },
|
|
|
696 { -1 },
|
|
771
|
697 };
|
|
|
698
|
|
1204
|
699 static const struct memory_description coding_system_description[] =
|
|
428
|
700 {
|
|
771
|
701 { XD_STRUCT_PTR, offsetof (Lisp_Coding_System, methods), 1,
|
|
|
702 &coding_system_methods_description },
|
|
1204
|
703 #define MARKED_SLOT(x) { XD_LISP_OBJECT, offsetof (Lisp_Coding_System, x) },
|
|
|
704 #define MARKED_SLOT_ARRAY(slot, size) \
|
|
|
705 { XD_LISP_OBJECT_ARRAY, offsetof (Lisp_Coding_System, slot), size },
|
|
|
706 #include "coding-system-slots.h"
|
|
|
707 { XD_STRUCT_ARRAY, offsetof (Lisp_Coding_System, data), 1,
|
|
|
708 coding_system_extra_description_map },
|
|
|
709 { XD_END }
|
|
771
|
710 };
|
|
|
711
|
|
1204
|
712 static const struct memory_description coding_system_empty_extra_description_1[] =
|
|
|
713 {
|
|
|
714 { XD_END }
|
|
|
715 };
|
|
|
716
|
|
|
717 const struct sized_memory_description coding_system_empty_extra_description = {
|
|
|
718 0, coding_system_empty_extra_description_1
|
|
|
719 };
|
|
|
720
|
|
934
|
721 DEFINE_LRECORD_SEQUENCE_IMPLEMENTATION ("coding-system", coding_system,
|
|
|
722 1, /*dumpable-flag*/
|
|
|
723 mark_coding_system,
|
|
|
724 print_coding_system,
|
|
|
725 finalize_coding_system,
|
|
|
726 0, 0, coding_system_description,
|
|
|
727 sizeof_coding_system,
|
|
|
728 Lisp_Coding_System);
|
|
771
|
729
|
|
|
730 /************************************************************************/
|
|
|
731 /* Creating coding systems */
|
|
|
732 /************************************************************************/
|
|
|
733
|
|
|
734 static struct coding_system_methods *
|
|
|
735 decode_coding_system_type (Lisp_Object type, Error_Behavior errb)
|
|
428
|
736 {
|
|
771
|
737 int i;
|
|
|
738
|
|
|
739 for (i = 0; i < Dynarr_length (the_coding_system_type_entry_dynarr); i++)
|
|
428
|
740 {
|
|
771
|
741 if (EQ (type,
|
|
|
742 Dynarr_at (the_coding_system_type_entry_dynarr, i).meths->type))
|
|
|
743 return Dynarr_at (the_coding_system_type_entry_dynarr, i).meths;
|
|
428
|
744 }
|
|
771
|
745
|
|
|
746 maybe_invalid_constant ("Invalid coding system type", type,
|
|
|
747 Qcoding_system, errb);
|
|
|
748
|
|
|
749 return 0;
|
|
428
|
750 }
|
|
|
751
|
|
771
|
752 static int
|
|
|
753 valid_coding_system_type_p (Lisp_Object type)
|
|
428
|
754 {
|
|
771
|
755 return decode_coding_system_type (type, ERROR_ME_NOT) != 0;
|
|
|
756 }
|
|
|
757
|
|
|
758 DEFUN ("valid-coding-system-type-p", Fvalid_coding_system_type_p, 1, 1, 0, /*
|
|
|
759 Given a CODING-SYSTEM-TYPE, return non-nil if it is valid.
|
|
|
760 Valid types depend on how XEmacs was compiled but may include
|
|
|
761 'undecided, 'chain, 'integer, 'ccl, 'iso2022, 'big5, 'shift-jis,
|
|
|
762 'utf-16, 'ucs-4, 'utf-8, etc.
|
|
|
763 */
|
|
|
764 (coding_system_type))
|
|
|
765 {
|
|
|
766 return valid_coding_system_type_p (coding_system_type) ? Qt : Qnil;
|
|
|
767 }
|
|
|
768
|
|
|
769 DEFUN ("coding-system-type-list", Fcoding_system_type_list, 0, 0, 0, /*
|
|
|
770 Return a list of valid coding system types.
|
|
|
771 */
|
|
|
772 ())
|
|
|
773 {
|
|
|
774 return Fcopy_sequence (Vcoding_system_type_list);
|
|
|
775 }
|
|
|
776
|
|
|
777 void
|
|
|
778 add_entry_to_coding_system_type_list (struct coding_system_methods *meths)
|
|
|
779 {
|
|
|
780 struct coding_system_type_entry entry;
|
|
|
781
|
|
|
782 entry.meths = meths;
|
|
|
783 Dynarr_add (the_coding_system_type_entry_dynarr, entry);
|
|
|
784 Vcoding_system_type_list = Fcons (meths->type, Vcoding_system_type_list);
|
|
428
|
785 }
|
|
|
786
|
|
|
787 DEFUN ("coding-system-p", Fcoding_system_p, 1, 1, 0, /*
|
|
|
788 Return t if OBJECT is a coding system.
|
|
|
789 A coding system is an object that defines how text containing multiple
|
|
|
790 character sets is encoded into a stream of (typically 8-bit) bytes.
|
|
|
791 The coding system is used to decode the stream into a series of
|
|
|
792 characters (which may be from multiple charsets) when the text is read
|
|
|
793 from a file or process, and is used to encode the text back into the
|
|
|
794 same format when it is written out to a file or process.
|
|
|
795
|
|
|
796 For example, many ISO2022-compliant coding systems (such as Compound
|
|
|
797 Text, which is used for inter-client data under the X Window System)
|
|
|
798 use escape sequences to switch between different charsets -- Japanese
|
|
|
799 Kanji, for example, is invoked with "ESC $ ( B"; ASCII is invoked
|
|
|
800 with "ESC ( B"; and Cyrillic is invoked with "ESC - L". See
|
|
|
801 `make-coding-system' for more information.
|
|
|
802
|
|
|
803 Coding systems are normally identified using a symbol, and the
|
|
|
804 symbol is accepted in place of the actual coding system object whenever
|
|
|
805 a coding system is called for. (This is similar to how faces work.)
|
|
|
806 */
|
|
|
807 (object))
|
|
|
808 {
|
|
|
809 return CODING_SYSTEMP (object) ? Qt : Qnil;
|
|
|
810 }
|
|
|
811
|
|
|
812 DEFUN ("find-coding-system", Ffind_coding_system, 1, 1, 0, /*
|
|
|
813 Retrieve the coding system of the given name.
|
|
|
814
|
|
|
815 If CODING-SYSTEM-OR-NAME is a coding-system object, it is simply
|
|
|
816 returned. Otherwise, CODING-SYSTEM-OR-NAME should be a symbol.
|
|
|
817 If there is no such coding system, nil is returned. Otherwise the
|
|
|
818 associated coding system object is returned.
|
|
|
819 */
|
|
|
820 (coding_system_or_name))
|
|
|
821 {
|
|
|
822 if (NILP (coding_system_or_name))
|
|
|
823 coding_system_or_name = Qbinary;
|
|
440
|
824 else if (CODING_SYSTEMP (coding_system_or_name))
|
|
|
825 return coding_system_or_name;
|
|
428
|
826 else
|
|
|
827 CHECK_SYMBOL (coding_system_or_name);
|
|
|
828
|
|
440
|
829 while (1)
|
|
|
830 {
|
|
|
831 coding_system_or_name =
|
|
|
832 Fgethash (coding_system_or_name, Vcoding_system_hash_table, Qnil);
|
|
|
833
|
|
771
|
834 if (CODING_SYSTEMP (coding_system_or_name)
|
|
|
835 || NILP (coding_system_or_name))
|
|
440
|
836 return coding_system_or_name;
|
|
|
837 }
|
|
428
|
838 }
|
|
|
839
|
|
|
840 DEFUN ("get-coding-system", Fget_coding_system, 1, 1, 0, /*
|
|
|
841 Retrieve the coding system of the given name.
|
|
|
842 Same as `find-coding-system' except that if there is no such
|
|
|
843 coding system, an error is signaled instead of returning nil.
|
|
|
844 */
|
|
|
845 (name))
|
|
|
846 {
|
|
|
847 Lisp_Object coding_system = Ffind_coding_system (name);
|
|
|
848
|
|
|
849 if (NILP (coding_system))
|
|
563
|
850 invalid_argument ("No such coding system", name);
|
|
428
|
851 return coding_system;
|
|
|
852 }
|
|
|
853
|
|
771
|
854 int
|
|
|
855 coding_system_is_binary (Lisp_Object coding_system)
|
|
|
856 {
|
|
|
857 Lisp_Coding_System *cs = XCODING_SYSTEM (coding_system);
|
|
|
858 return
|
|
|
859 (EQ (CODING_SYSTEM_TYPE (cs), Qno_conversion) &&
|
|
|
860 CODING_SYSTEM_EOL_TYPE (cs) == EOL_LF &&
|
|
|
861 EQ (CODING_SYSTEM_POST_READ_CONVERSION (cs), Qnil) &&
|
|
|
862 EQ (CODING_SYSTEM_PRE_WRITE_CONVERSION (cs), Qnil));
|
|
|
863 }
|
|
|
864
|
|
|
865 static Lisp_Object
|
|
|
866 coding_system_real_canonical (Lisp_Object cs)
|
|
|
867 {
|
|
|
868 if (!NILP (XCODING_SYSTEM_CANONICAL (cs)))
|
|
|
869 return XCODING_SYSTEM_CANONICAL (cs);
|
|
|
870 return cs;
|
|
|
871 }
|
|
|
872
|
|
|
873 /* Return true if coding system is of the "standard" type that decodes
|
|
|
874 bytes into characters (suitable for decoding a text file). */
|
|
|
875 int
|
|
|
876 coding_system_is_for_text_file (Lisp_Object coding_system)
|
|
|
877 {
|
|
|
878 return (XCODESYSMETH_OR_GIVEN
|
|
|
879 (coding_system, conversion_end_type,
|
|
|
880 (coding_system_real_canonical (coding_system)),
|
|
|
881 DECODES_BYTE_TO_CHARACTER) ==
|
|
|
882 DECODES_BYTE_TO_CHARACTER);
|
|
|
883 }
|
|
|
884
|
|
|
885 static int
|
|
|
886 decoding_source_sink_type_is_char (Lisp_Object cs, enum source_or_sink sex)
|
|
|
887 {
|
|
|
888 enum source_sink_type type =
|
|
|
889 XCODESYSMETH_OR_GIVEN (cs, conversion_end_type,
|
|
|
890 (coding_system_real_canonical (cs)),
|
|
|
891 DECODES_BYTE_TO_CHARACTER);
|
|
|
892 if (sex == CODING_SOURCE)
|
|
|
893 return (type == DECODES_CHARACTER_TO_CHARACTER ||
|
|
|
894 type == DECODES_CHARACTER_TO_BYTE);
|
|
|
895 else
|
|
|
896 return (type == DECODES_CHARACTER_TO_CHARACTER ||
|
|
|
897 type == DECODES_BYTE_TO_CHARACTER);
|
|
|
898 }
|
|
|
899
|
|
|
900 static int
|
|
|
901 encoding_source_sink_type_is_char (Lisp_Object cs, enum source_or_sink sex)
|
|
|
902 {
|
|
|
903 return decoding_source_sink_type_is_char (cs,
|
|
|
904 /* Sex change */
|
|
|
905 sex == CODING_SOURCE ?
|
|
|
906 CODING_SINK : CODING_SOURCE);
|
|
|
907 }
|
|
|
908
|
|
|
909 /* Like Ffind_coding_system() but check that the coding system is of the
|
|
|
910 "standard" type that decodes bytes into characters (suitable for
|
|
|
911 decoding a text file), and if not, returns an appropriate wrapper that
|
|
|
912 does. Also, if EOL_WRAP is non-zero, check whether this coding system
|
|
|
913 wants EOL auto-detection, and if so, wrap with a convert-eol coding
|
|
|
914 system to do this. */
|
|
|
915
|
|
|
916 Lisp_Object
|
|
|
917 find_coding_system_for_text_file (Lisp_Object name, int eol_wrap)
|
|
|
918 {
|
|
|
919 Lisp_Object coding_system = Ffind_coding_system (name);
|
|
|
920 Lisp_Object wrapper = coding_system;
|
|
|
921
|
|
|
922 if (NILP (coding_system))
|
|
|
923 return Qnil;
|
|
|
924 if (!coding_system_is_for_text_file (coding_system))
|
|
|
925 {
|
|
|
926 wrapper = XCODING_SYSTEM_TEXT_FILE_WRAPPER (coding_system);
|
|
|
927 if (NILP (wrapper))
|
|
|
928 {
|
|
|
929 Lisp_Object chain;
|
|
|
930 if (!decoding_source_sink_type_is_char (coding_system, CODING_SINK))
|
|
|
931 chain = list2 (coding_system, Qbinary);
|
|
|
932 else
|
|
|
933 chain = list1 (coding_system);
|
|
|
934 if (decoding_source_sink_type_is_char (coding_system, CODING_SOURCE))
|
|
|
935 chain = Fcons (Qbinary, chain);
|
|
|
936 wrapper =
|
|
|
937 make_internal_coding_system
|
|
|
938 (coding_system,
|
|
|
939 "internal-text-file-wrapper",
|
|
|
940 Qchain,
|
|
|
941 Qunbound, list4 (Qchain, chain,
|
|
|
942 Qcanonicalize_after_coding, coding_system));
|
|
|
943 XCODING_SYSTEM_TEXT_FILE_WRAPPER (coding_system) = wrapper;
|
|
|
944 }
|
|
|
945 }
|
|
|
946
|
|
|
947 if (!eol_wrap || XCODING_SYSTEM_EOL_TYPE (coding_system) != EOL_AUTODETECT)
|
|
|
948 return wrapper;
|
|
|
949
|
|
|
950 coding_system = wrapper;
|
|
|
951 wrapper = XCODING_SYSTEM_AUTO_EOL_WRAPPER (coding_system);
|
|
|
952 if (!NILP (wrapper))
|
|
|
953 return wrapper;
|
|
|
954 wrapper =
|
|
|
955 make_internal_coding_system
|
|
|
956 (coding_system,
|
|
|
957 "internal-auto-eol-wrapper",
|
|
|
958 Qundecided, Qunbound,
|
|
|
959 list4 (Qcoding_system, coding_system,
|
|
|
960 Qdo_eol, Qt));
|
|
|
961 XCODING_SYSTEM_AUTO_EOL_WRAPPER (coding_system) = wrapper;
|
|
|
962 return wrapper;
|
|
|
963 }
|
|
|
964
|
|
|
965 /* Like Fget_coding_system() but verify that the coding system is of the
|
|
|
966 "standard" type that decodes bytes into characters (suitable for
|
|
|
967 decoding a text file), and if not, returns an appropriate wrapper that
|
|
|
968 does. Also, if EOL_WRAP is non-zero, check whether this coding system
|
|
|
969 wants EOL auto-detection, and if so, wrap with a convert-eol coding
|
|
|
970 system to do this. */
|
|
|
971
|
|
|
972 Lisp_Object
|
|
|
973 get_coding_system_for_text_file (Lisp_Object name, int eol_wrap)
|
|
|
974 {
|
|
|
975 Lisp_Object coding_system = find_coding_system_for_text_file (name,
|
|
|
976 eol_wrap);
|
|
|
977 if (NILP (coding_system))
|
|
|
978 invalid_argument ("No such coding system", name);
|
|
|
979 return coding_system;
|
|
|
980 }
|
|
|
981
|
|
|
982 /* We store the coding systems in hash tables with the names as the
|
|
|
983 key and the actual coding system object as the value. Occasionally
|
|
|
984 we need to use them in a list format. These routines provide us
|
|
|
985 with that. */
|
|
428
|
986 struct coding_system_list_closure
|
|
|
987 {
|
|
|
988 Lisp_Object *coding_system_list;
|
|
771
|
989 int normal;
|
|
|
990 int internal;
|
|
428
|
991 };
|
|
|
992
|
|
|
993 static int
|
|
2286
|
994 add_coding_system_to_list_mapper (Lisp_Object key, Lisp_Object UNUSED (value),
|
|
428
|
995 void *coding_system_list_closure)
|
|
|
996 {
|
|
|
997 /* This function can GC */
|
|
|
998 struct coding_system_list_closure *cscl =
|
|
|
999 (struct coding_system_list_closure *) coding_system_list_closure;
|
|
|
1000 Lisp_Object *coding_system_list = cscl->coding_system_list;
|
|
|
1001
|
|
771
|
1002 /* We can't just use VALUE because KEY might be an alias, and we need
|
|
|
1003 the real coding system object. */
|
|
|
1004 if (XCODING_SYSTEM (Ffind_coding_system (key))->internal_p ?
|
|
|
1005 cscl->internal : cscl->normal)
|
|
|
1006 *coding_system_list = Fcons (key, *coding_system_list);
|
|
428
|
1007 return 0;
|
|
|
1008 }
|
|
|
1009
|
|
2297
|
1010 /* #### should we specify a conventional for "all coding systems"? */
|
|
771
|
1011 DEFUN ("coding-system-list", Fcoding_system_list, 0, 1, 0, /*
|
|
428
|
1012 Return a list of the names of all defined coding systems.
|
|
771
|
1013 If INTERNAL is nil, only the normal (non-internal) coding systems are
|
|
|
1014 included. (Internal coding systems are created for various internal
|
|
|
1015 purposes, such as implementing EOL types of CRLF and CR; generally, you do
|
|
|
1016 not want to see these.) If it is t, only the internal coding systems are
|
|
|
1017 included. If it is any other non-nil value both normal and internal are
|
|
|
1018 included.
|
|
428
|
1019 */
|
|
771
|
1020 (internal))
|
|
428
|
1021 {
|
|
|
1022 Lisp_Object coding_system_list = Qnil;
|
|
|
1023 struct gcpro gcpro1;
|
|
|
1024 struct coding_system_list_closure coding_system_list_closure;
|
|
|
1025
|
|
|
1026 GCPRO1 (coding_system_list);
|
|
|
1027 coding_system_list_closure.coding_system_list = &coding_system_list;
|
|
771
|
1028 coding_system_list_closure.normal = !EQ (internal, Qt);
|
|
|
1029 coding_system_list_closure.internal = !NILP (internal);
|
|
428
|
1030 elisp_maphash (add_coding_system_to_list_mapper, Vcoding_system_hash_table,
|
|
|
1031 &coding_system_list_closure);
|
|
|
1032 UNGCPRO;
|
|
|
1033
|
|
|
1034 return coding_system_list;
|
|
|
1035 }
|
|
|
1036
|
|
|
1037 DEFUN ("coding-system-name", Fcoding_system_name, 1, 1, 0, /*
|
|
|
1038 Return the name of the given coding system.
|
|
|
1039 */
|
|
|
1040 (coding_system))
|
|
|
1041 {
|
|
|
1042 coding_system = Fget_coding_system (coding_system);
|
|
|
1043 return XCODING_SYSTEM_NAME (coding_system);
|
|
|
1044 }
|
|
|
1045
|
|
|
1046 static Lisp_Coding_System *
|
|
771
|
1047 allocate_coding_system (struct coding_system_methods *codesys_meths,
|
|
|
1048 Bytecount data_size,
|
|
|
1049 Lisp_Object name)
|
|
428
|
1050 {
|
|
771
|
1051 Bytecount total_size = offsetof (Lisp_Coding_System, data) + data_size;
|
|
428
|
1052 Lisp_Coding_System *codesys =
|
|
1204
|
1053 (Lisp_Coding_System *) basic_alloc_lcrecord (total_size,
|
|
|
1054 &lrecord_coding_system);
|
|
|
1055
|
|
771
|
1056 codesys->methods = codesys_meths;
|
|
1204
|
1057 #define MARKED_SLOT(x) codesys->x = Qnil;
|
|
|
1058 #include "coding-system-slots.h"
|
|
|
1059
|
|
771
|
1060 CODING_SYSTEM_EOL_TYPE (codesys) = EOL_LF;
|
|
|
1061 CODING_SYSTEM_NAME (codesys) = name;
|
|
|
1062
|
|
|
1063 MAYBE_CODESYSMETH (codesys, init, (wrap_coding_system (codesys)));
|
|
428
|
1064
|
|
|
1065 return codesys;
|
|
|
1066 }
|
|
|
1067
|
|
771
|
1068 static enum eol_type
|
|
|
1069 symbol_to_eol_type (Lisp_Object symbol)
|
|
|
1070 {
|
|
|
1071 CHECK_SYMBOL (symbol);
|
|
|
1072 if (NILP (symbol)) return EOL_AUTODETECT;
|
|
|
1073 if (EQ (symbol, Qlf)) return EOL_LF;
|
|
|
1074 if (EQ (symbol, Qcrlf)) return EOL_CRLF;
|
|
|
1075 if (EQ (symbol, Qcr)) return EOL_CR;
|
|
|
1076
|
|
|
1077 invalid_constant ("Unrecognized eol type", symbol);
|
|
1204
|
1078 RETURN_NOT_REACHED (EOL_AUTODETECT);
|
|
771
|
1079 }
|
|
|
1080
|
|
|
1081 static Lisp_Object
|
|
|
1082 eol_type_to_symbol (enum eol_type type)
|
|
|
1083 {
|
|
|
1084 switch (type)
|
|
|
1085 {
|
|
|
1086 default: abort ();
|
|
|
1087 case EOL_LF: return Qlf;
|
|
|
1088 case EOL_CRLF: return Qcrlf;
|
|
|
1089 case EOL_CR: return Qcr;
|
|
|
1090 case EOL_AUTODETECT: return Qnil;
|
|
|
1091 }
|
|
|
1092 }
|
|
|
1093
|
|
|
1094 struct subsidiary_type
|
|
|
1095 {
|
|
|
1096 Char_ASCII *extension;
|
|
|
1097 Char_ASCII *mnemonic_ext;
|
|
|
1098 enum eol_type eol;
|
|
|
1099 };
|
|
|
1100
|
|
|
1101 static struct subsidiary_type coding_subsidiary_list[] =
|
|
|
1102 { { "-unix", "", EOL_LF },
|
|
|
1103 { "-dos", ":T", EOL_CRLF },
|
|
|
1104 { "-mac", ":t", EOL_CR } };
|
|
|
1105
|
|
|
1106 /* kludge */
|
|
428
|
1107 static void
|
|
771
|
1108 setup_eol_coding_systems (Lisp_Object codesys)
|
|
428
|
1109 {
|
|
793
|
1110 int len = XSTRING_LENGTH (XSYMBOL (XCODING_SYSTEM_NAME (codesys))->name);
|
|
867
|
1111 Ibyte *codesys_name = (Ibyte *) ALLOCA (len + 7);
|
|
771
|
1112 int mlen = -1;
|
|
867
|
1113 Ibyte *codesys_mnemonic = 0;
|
|
771
|
1114 Lisp_Object codesys_name_sym, sub_codesys;
|
|
|
1115 int i;
|
|
|
1116
|
|
|
1117 memcpy (codesys_name,
|
|
793
|
1118 XSTRING_DATA (XSYMBOL (XCODING_SYSTEM_NAME (codesys))->name), len);
|
|
771
|
1119
|
|
|
1120 if (STRINGP (XCODING_SYSTEM_MNEMONIC (codesys)))
|
|
428
|
1121 {
|
|
771
|
1122 mlen = XSTRING_LENGTH (XCODING_SYSTEM_MNEMONIC (codesys));
|
|
867
|
1123 codesys_mnemonic = (Ibyte *) ALLOCA (mlen + 7);
|
|
771
|
1124 memcpy (codesys_mnemonic,
|
|
|
1125 XSTRING_DATA (XCODING_SYSTEM_MNEMONIC (codesys)), mlen);
|
|
|
1126 }
|
|
|
1127
|
|
|
1128 /* Create three "subsidiary" coding systems, decoding data encoded using
|
|
|
1129 each of the three EOL types. We do this for each subsidiary by
|
|
|
1130 copying the original coding system, setting the EOL type
|
|
|
1131 appropriately, and setting the CANONICAL member of the new coding
|
|
|
1132 system to be a chain consisting of the original coding system followed
|
|
|
1133 by a convert-eol coding system to do the EOL decoding. For EOL type
|
|
|
1134 LF, however, we don't need any decoding, so we skip creating a
|
|
|
1135 CANONICAL.
|
|
|
1136
|
|
|
1137 If the original coding system is not a text-type coding system
|
|
|
1138 (decodes byte->char), we need to coerce it to one by the appropriate
|
|
|
1139 wrapping in CANONICAL. */
|
|
|
1140
|
|
|
1141 for (i = 0; i < countof (coding_subsidiary_list); i++)
|
|
|
1142 {
|
|
|
1143 Char_ASCII *extension = coding_subsidiary_list[i].extension;
|
|
|
1144 Char_ASCII *mnemonic_ext = coding_subsidiary_list[i].mnemonic_ext;
|
|
|
1145 enum eol_type eol = coding_subsidiary_list[i].eol;
|
|
|
1146
|
|
|
1147 qxestrcpy_c (codesys_name + len, extension);
|
|
|
1148 codesys_name_sym = intern_int (codesys_name);
|
|
|
1149 if (mlen != -1)
|
|
|
1150 qxestrcpy_c (codesys_mnemonic + mlen, mnemonic_ext);
|
|
|
1151
|
|
|
1152 sub_codesys = Fcopy_coding_system (codesys, codesys_name_sym);
|
|
|
1153 if (mlen != -1)
|
|
|
1154 XCODING_SYSTEM_MNEMONIC (sub_codesys) =
|
|
|
1155 build_intstring (codesys_mnemonic);
|
|
|
1156
|
|
|
1157 if (eol != EOL_LF)
|
|
|
1158 {
|
|
|
1159 Lisp_Object chain = list2 (get_coding_system_for_text_file
|
|
|
1160 (codesys, 0),
|
|
|
1161 eol == EOL_CR ? Qconvert_eol_cr :
|
|
|
1162 Qconvert_eol_crlf);
|
|
|
1163 Lisp_Object canon =
|
|
|
1164 make_internal_coding_system
|
|
|
1165 (sub_codesys, "internal-subsidiary-eol-wrapper",
|
|
|
1166 Qchain, Qunbound,
|
|
|
1167 mlen != -1 ?
|
|
|
1168 list6 (Qmnemonic, build_intstring (codesys_mnemonic),
|
|
|
1169 Qchain, chain,
|
|
|
1170 Qcanonicalize_after_coding, sub_codesys) :
|
|
|
1171 list4 (Qchain, chain,
|
|
|
1172 Qcanonicalize_after_coding, sub_codesys));
|
|
|
1173 XCODING_SYSTEM_CANONICAL (sub_codesys) = canon;
|
|
|
1174 }
|
|
|
1175 XCODING_SYSTEM_EOL_TYPE (sub_codesys) = eol;
|
|
|
1176 XCODING_SYSTEM_SUBSIDIARY_PARENT (sub_codesys) = codesys;
|
|
|
1177 XCODING_SYSTEM (codesys)->eol[eol] = sub_codesys;
|
|
428
|
1178 }
|
|
|
1179 }
|
|
|
1180
|
|
771
|
1181 /* Basic function to create new coding systems. For `make-coding-system',
|
|
|
1182 NAME-OR-EXISTING is the NAME argument, PREFIX is null, and TYPE,
|
|
|
1183 DESCRIPTION, and PROPS are the same. All created coding systems are put
|
|
|
1184 in a hash table indexed by NAME.
|
|
|
1185
|
|
|
1186 If PREFIX is a string, NAME-OR-EXISTING should specify an existing
|
|
|
1187 coding system (or nil), and an internal coding system will be created.
|
|
|
1188 The name of the coding system will be constructed by combining PREFIX
|
|
|
1189 with the name of the existing coding system (if given), and a number
|
|
|
1190 will be appended to insure uniqueness. In such a case, if Qunbound is
|
|
|
1191 given for DESCRIPTION, the description gets created based on the
|
|
|
1192 generated name. Also, if no mnemonic is given in the properties list, a
|
|
|
1193 mnemonic is created based on the generated name.
|
|
|
1194
|
|
|
1195 For internal coding systems, the coding system is marked as internal
|
|
|
1196 (see `coding-system-list'), and no subsidiaries will be created or
|
|
|
1197 eol-wrapping will happen. Otherwise:
|
|
|
1198
|
|
|
1199 -- if the eol-type property is `lf' or t, the coding system is merely
|
|
|
1200 created and returned. (For t, the coding system will be wrapped with
|
|
|
1201 an EOL autodetector when it's used to read a file.)
|
|
|
1202
|
|
|
1203 -- if eol-type is `crlf' or `cr', after the coding system object is
|
|
|
1204 created, it will be wrapped in a chain with the appropriate
|
|
|
1205 convert-eol coding system (either `convert-eol-crlf' or
|
|
|
1206 `convert-eol-cr'), so that CRLF->LF or CR->LF conversion is done at
|
|
|
1207 decoding time, and the opposite at encoding time. The resulting
|
|
|
1208 chain becomes the CANONICAL field of the coding system object.
|
|
|
1209
|
|
|
1210 -- if eol-type is nil or omitted, "subsidiaries" are generated: Three
|
|
|
1211 coding systems where the original coding system (before wrapping with
|
|
|
1212 convert-eol-autodetect) is either unwrapped or wrapped with
|
|
|
1213 convert-eol-crlf or convert-eol-cr, respectively, so that coding systems
|
|
|
1214 to handle LF, CRLF, and CR end-of-line indicators are created. (This
|
|
|
1215 crazy crap is based on existing behavior in other Mule versions,
|
|
|
1216 including FSF Emacs.)
|
|
|
1217 */
|
|
428
|
1218
|
|
|
1219 static Lisp_Object
|
|
771
|
1220 make_coding_system_1 (Lisp_Object name_or_existing, Char_ASCII *prefix,
|
|
|
1221 Lisp_Object type, Lisp_Object description,
|
|
|
1222 Lisp_Object props)
|
|
428
|
1223 {
|
|
771
|
1224 Lisp_Coding_System *cs;
|
|
|
1225 int need_to_setup_eol_systems = 1;
|
|
|
1226 enum eol_type eol_wrapper = EOL_AUTODETECT;
|
|
|
1227 struct coding_system_methods *meths;
|
|
|
1228 Lisp_Object csobj;
|
|
|
1229 Lisp_Object defmnem = Qnil;
|
|
|
1230
|
|
|
1231 if (NILP (type))
|
|
|
1232 type = Qundecided;
|
|
|
1233 meths = decode_coding_system_type (type, ERROR_ME);
|
|
|
1234
|
|
|
1235 if (prefix)
|
|
428
|
1236 {
|
|
867
|
1237 Ibyte *newname =
|
|
771
|
1238 emacs_sprintf_malloc (NULL, "%s-%s-%d",
|
|
|
1239 prefix,
|
|
867
|
1240 NILP (name_or_existing) ? (Ibyte *) "nil" :
|
|
771
|
1241 XSTRING_DATA (Fsymbol_name (XCODING_SYSTEM_NAME
|
|
|
1242 (name_or_existing))),
|
|
|
1243 ++coding_system_tick);
|
|
|
1244 name_or_existing = intern_int (newname);
|
|
1726
|
1245 xfree (newname, Ibyte *);
|
|
771
|
1246
|
|
|
1247 if (UNBOUNDP (description))
|
|
|
1248 {
|
|
|
1249 newname =
|
|
|
1250 emacs_sprintf_malloc
|
|
|
1251 (NULL, "For Internal Use (%s)",
|
|
|
1252 XSTRING_DATA (Fsymbol_name (name_or_existing)));
|
|
|
1253 description = build_intstring (newname);
|
|
1726
|
1254 xfree (newname, Ibyte *);
|
|
771
|
1255 }
|
|
|
1256
|
|
|
1257 newname = emacs_sprintf_malloc (NULL, "Int%d", coding_system_tick);
|
|
|
1258 defmnem = build_intstring (newname);
|
|
1726
|
1259 xfree (newname, Ibyte *);
|
|
428
|
1260 }
|
|
771
|
1261 else
|
|
|
1262 CHECK_SYMBOL (name_or_existing);
|
|
|
1263
|
|
|
1264 if (!NILP (Ffind_coding_system (name_or_existing)))
|
|
|
1265 invalid_operation ("Cannot redefine existing coding system",
|
|
|
1266 name_or_existing);
|
|
|
1267
|
|
|
1268 cs = allocate_coding_system (meths, meths->extra_data_size,
|
|
|
1269 name_or_existing);
|
|
793
|
1270 csobj = wrap_coding_system (cs);
|
|
771
|
1271
|
|
|
1272 cs->internal_p = !!prefix;
|
|
|
1273
|
|
|
1274 if (NILP (description))
|
|
|
1275 description = build_string ("");
|
|
|
1276 else
|
|
|
1277 CHECK_STRING (description);
|
|
|
1278 CODING_SYSTEM_DESCRIPTION (cs) = description;
|
|
|
1279
|
|
|
1280 if (!NILP (defmnem))
|
|
|
1281 CODING_SYSTEM_MNEMONIC (cs) = defmnem;
|
|
|
1282
|
|
|
1283 {
|
|
|
1284 EXTERNAL_PROPERTY_LIST_LOOP_3 (key, value, props)
|
|
|
1285 {
|
|
|
1286 int recognized = 1;
|
|
|
1287
|
|
|
1288 if (EQ (key, Qmnemonic))
|
|
|
1289 {
|
|
|
1290 if (!NILP (value))
|
|
|
1291 CHECK_STRING (value);
|
|
|
1292 CODING_SYSTEM_MNEMONIC (cs) = value;
|
|
|
1293 }
|
|
|
1294
|
|
|
1295 else if (EQ (key, Qdocumentation))
|
|
|
1296 {
|
|
|
1297 if (!NILP (value))
|
|
|
1298 CHECK_STRING (value);
|
|
|
1299 CODING_SYSTEM_DOCUMENTATION (cs) = value;
|
|
|
1300 }
|
|
|
1301
|
|
|
1302 else if (EQ (key, Qeol_type))
|
|
|
1303 {
|
|
|
1304 need_to_setup_eol_systems = NILP (value);
|
|
|
1305 if (EQ (value, Qt))
|
|
|
1306 value = Qnil;
|
|
|
1307 eol_wrapper = symbol_to_eol_type (value);
|
|
|
1308 }
|
|
|
1309
|
|
|
1310 else if (EQ (key, Qpost_read_conversion))
|
|
|
1311 CODING_SYSTEM_POST_READ_CONVERSION (cs) = value;
|
|
|
1312 else if (EQ (key, Qpre_write_conversion))
|
|
|
1313 CODING_SYSTEM_PRE_WRITE_CONVERSION (cs) = value;
|
|
|
1314 /* FSF compatibility */
|
|
|
1315 else if (EQ (key, Qtranslation_table_for_decode))
|
|
|
1316 ;
|
|
|
1317 else if (EQ (key, Qtranslation_table_for_encode))
|
|
|
1318 ;
|
|
|
1319 else if (EQ (key, Qsafe_chars))
|
|
|
1320 ;
|
|
|
1321 else if (EQ (key, Qsafe_charsets))
|
|
|
1322 ;
|
|
|
1323 else if (EQ (key, Qmime_charset))
|
|
|
1324 ;
|
|
|
1325 else if (EQ (key, Qvalid_codes))
|
|
|
1326 ;
|
|
|
1327 else
|
|
|
1328 recognized = CODESYSMETH_OR_GIVEN (cs, putprop,
|
|
|
1329 (csobj, key, value), 0);
|
|
|
1330
|
|
|
1331 if (!recognized)
|
|
|
1332 invalid_constant ("Unrecognized property", key);
|
|
|
1333 }
|
|
|
1334 }
|
|
|
1335
|
|
|
1336 {
|
|
|
1337 XCODING_SYSTEM_CANONICAL (csobj) =
|
|
|
1338 CODESYSMETH_OR_GIVEN (cs, canonicalize, (csobj), Qnil);
|
|
|
1339 XCODING_SYSTEM_EOL_TYPE (csobj) = EOL_AUTODETECT; /* for copy-coding-system
|
|
|
1340 below */
|
|
|
1341
|
|
|
1342 if (need_to_setup_eol_systems && !cs->internal_p)
|
|
|
1343 setup_eol_coding_systems (csobj);
|
|
|
1344 else if (eol_wrapper == EOL_CR || eol_wrapper == EOL_CRLF)
|
|
|
1345 {
|
|
|
1346 /* If a specific eol-type (other than LF) was specified, we handle
|
|
|
1347 this by converting the coding system into a chain that wraps the
|
|
|
1348 coding system along with a convert-eol system after it, in
|
|
|
1349 exactly that same switcheroo fashion that the normal
|
|
|
1350 canonicalize method works -- BUT we will run into a problem if
|
|
|
1351 we do it the obvious way, because when `chain' creates its
|
|
|
1352 substreams, the substream containing the coding system we're
|
|
|
1353 creating will have canonicalization expansion done on it,
|
|
|
1354 leading to infinite recursion. So we have to generate a new,
|
|
|
1355 internal coding system with the previous value of CANONICAL. */
|
|
867
|
1356 Ibyte *newname =
|
|
771
|
1357 emacs_sprintf_malloc
|
|
|
1358 (NULL, "internal-eol-copy-%s-%d",
|
|
|
1359 XSTRING_DATA (Fsymbol_name (name_or_existing)),
|
|
|
1360 ++coding_system_tick);
|
|
|
1361 Lisp_Object newnamesym = intern_int (newname);
|
|
|
1362 Lisp_Object copied = Fcopy_coding_system (csobj, newnamesym);
|
|
1726
|
1363 xfree (newname, Ibyte *);
|
|
771
|
1364
|
|
|
1365 XCODING_SYSTEM_CANONICAL (csobj) =
|
|
|
1366 make_internal_coding_system
|
|
|
1367 (csobj,
|
|
|
1368 "internal-eol-wrapper",
|
|
|
1369 Qchain, Qunbound,
|
|
|
1370 list4 (Qchain,
|
|
|
1371 list2 (copied,
|
|
|
1372 eol_wrapper == EOL_CR ?
|
|
|
1373 Qconvert_eol_cr :
|
|
|
1374 Qconvert_eol_crlf),
|
|
|
1375 Qcanonicalize_after_coding,
|
|
|
1376 csobj));
|
|
|
1377 }
|
|
|
1378 XCODING_SYSTEM_EOL_TYPE (csobj) = eol_wrapper;
|
|
|
1379 }
|
|
|
1380
|
|
|
1381 Fputhash (name_or_existing, csobj, Vcoding_system_hash_table);
|
|
|
1382
|
|
|
1383 return csobj;
|
|
428
|
1384 }
|
|
|
1385
|
|
771
|
1386 Lisp_Object
|
|
|
1387 make_internal_coding_system (Lisp_Object existing, Char_ASCII *prefix,
|
|
|
1388 Lisp_Object type, Lisp_Object description,
|
|
|
1389 Lisp_Object props)
|
|
|
1390 {
|
|
|
1391 return make_coding_system_1 (existing, prefix, type, description, props);
|
|
|
1392 }
|
|
428
|
1393
|
|
|
1394 DEFUN ("make-coding-system", Fmake_coding_system, 2, 4, 0, /*
|
|
|
1395 Register symbol NAME as a coding system.
|
|
|
1396
|
|
|
1397 TYPE describes the conversion method used and should be one of
|
|
|
1398
|
|
|
1399 nil or 'undecided
|
|
|
1400 Automatic conversion. XEmacs attempts to detect the coding system
|
|
|
1401 used in the file.
|
|
771
|
1402 'chain
|
|
|
1403 Chain two or more coding systems together to make a combination coding
|
|
|
1404 system.
|
|
428
|
1405 'no-conversion
|
|
|
1406 No conversion. Use this for binary files and such. On output,
|
|
|
1407 graphic characters that are not in ASCII or Latin-1 will be
|
|
|
1408 replaced by a ?. (For a no-conversion-encoded buffer, these
|
|
|
1409 characters will only be present if you explicitly insert them.)
|
|
771
|
1410 'convert-eol
|
|
|
1411 Convert CRLF sequences or CR to LF.
|
|
428
|
1412 'shift-jis
|
|
|
1413 Shift-JIS (a Japanese encoding commonly used in PC operating systems).
|
|
771
|
1414 'unicode
|
|
|
1415 Any Unicode encoding (UCS-4, UTF-8, UTF-16, etc.).
|
|
|
1416 'mswindows-unicode-to-multibyte
|
|
|
1417 (MS Windows only) Converts from Windows Unicode to Windows Multibyte
|
|
|
1418 (any code page encoding) upon encoding, and the other way upon decoding.
|
|
|
1419 'mswindows-multibyte
|
|
|
1420 Converts to or from Windows Multibyte (any code page encoding).
|
|
|
1421 This is resolved into a chain of `mswindows-unicode' and
|
|
|
1422 `mswindows-unicode-to-multibyte'.
|
|
428
|
1423 'iso2022
|
|
|
1424 Any ISO2022-compliant encoding. Among other things, this includes
|
|
|
1425 JIS (the Japanese encoding commonly used for e-mail), EUC (the
|
|
|
1426 standard Unix encoding for Japanese and other languages), and
|
|
|
1427 Compound Text (the encoding used in X11). You can specify more
|
|
442
|
1428 specific information about the conversion with the PROPS argument.
|
|
428
|
1429 'big5
|
|
|
1430 Big5 (the encoding commonly used for Taiwanese).
|
|
|
1431 'ccl
|
|
|
1432 The conversion is performed using a user-written pseudo-code
|
|
|
1433 program. CCL (Code Conversion Language) is the name of this
|
|
|
1434 pseudo-code.
|
|
771
|
1435 'gzip
|
|
|
1436 GZIP compression format.
|
|
428
|
1437 'internal
|
|
|
1438 Write out or read in the raw contents of the memory representing
|
|
|
1439 the buffer's text. This is primarily useful for debugging
|
|
|
1440 purposes, and is only enabled when XEmacs has been compiled with
|
|
|
1441 DEBUG_XEMACS defined (via the --debug configure option).
|
|
|
1442 WARNING: Reading in a file using 'internal conversion can result
|
|
|
1443 in an internal inconsistency in the memory representing a
|
|
|
1444 buffer's text, which will produce unpredictable results and may
|
|
|
1445 cause XEmacs to crash. Under normal circumstances you should
|
|
|
1446 never use 'internal conversion.
|
|
|
1447
|
|
771
|
1448 DESCRIPTION is a short English phrase describing the coding system,
|
|
|
1449 suitable for use as a menu item. (See also the `documentation' property
|
|
|
1450 below.)
|
|
428
|
1451
|
|
|
1452 PROPS is a property list, describing the specific nature of the
|
|
|
1453 character set. Recognized properties are:
|
|
|
1454
|
|
|
1455 'mnemonic
|
|
|
1456 String to be displayed in the modeline when this coding system is
|
|
|
1457 active.
|
|
|
1458
|
|
771
|
1459 'documentation
|
|
|
1460 Detailed documentation on the coding system.
|
|
|
1461
|
|
428
|
1462 'eol-type
|
|
|
1463 End-of-line conversion to be used. It should be one of
|
|
|
1464
|
|
|
1465 nil
|
|
|
1466 Automatically detect the end-of-line type (LF, CRLF,
|
|
|
1467 or CR). Also generate subsidiary coding systems named
|
|
|
1468 `NAME-unix', `NAME-dos', and `NAME-mac', that are
|
|
|
1469 identical to this coding system but have an EOL-TYPE
|
|
|
1470 value of 'lf, 'crlf, and 'cr, respectively.
|
|
|
1471 'lf
|
|
|
1472 The end of a line is marked externally using ASCII LF.
|
|
|
1473 Since this is also the way that XEmacs represents an
|
|
|
1474 end-of-line internally, specifying this option results
|
|
|
1475 in no end-of-line conversion. This is the standard
|
|
|
1476 format for Unix text files.
|
|
|
1477 'crlf
|
|
|
1478 The end of a line is marked externally using ASCII
|
|
|
1479 CRLF. This is the standard format for MS-DOS text
|
|
|
1480 files.
|
|
|
1481 'cr
|
|
|
1482 The end of a line is marked externally using ASCII CR.
|
|
|
1483 This is the standard format for Macintosh text files.
|
|
|
1484 t
|
|
|
1485 Automatically detect the end-of-line type but do not
|
|
|
1486 generate subsidiary coding systems. (This value is
|
|
|
1487 converted to nil when stored internally, and
|
|
|
1488 `coding-system-property' will return nil.)
|
|
|
1489
|
|
|
1490 'post-read-conversion
|
|
771
|
1491 The value is a function to call after some text is inserted and
|
|
|
1492 decoded by the coding system itself and before any functions in
|
|
|
1493 `after-change-functions' are called. (#### Not actually true in
|
|
|
1494 XEmacs. `after-change-functions' will be called twice if
|
|
|
1495 `post-read-conversion' changes something.) The argument of this
|
|
|
1496 function is the same as for a function in
|
|
|
1497 `after-insert-file-functions', i.e. LENGTH of the text inserted,
|
|
|
1498 with point at the head of the text to be decoded.
|
|
428
|
1499
|
|
|
1500 'pre-write-conversion
|
|
771
|
1501 The value is a function to call after all functions in
|
|
|
1502 `write-region-annotate-functions' and `buffer-file-format' are
|
|
|
1503 called, and before the text is encoded by the coding system itself.
|
|
|
1504 The arguments to this function are the same as those of a function
|
|
|
1505 in `write-region-annotate-functions', i.e. FROM and TO, specifying
|
|
|
1506 a region of text.
|
|
|
1507
|
|
|
1508
|
|
|
1509
|
|
|
1510 The following properties are allowed for FSF compatibility but currently
|
|
|
1511 ignored:
|
|
|
1512
|
|
|
1513 'translation-table-for-decode
|
|
|
1514 The value is a translation table to be applied on decoding. See
|
|
|
1515 the function `make-translation-table' for the format of translation
|
|
|
1516 table. This is not applicable to CCL-based coding systems.
|
|
|
1517
|
|
|
1518 'translation-table-for-encode
|
|
|
1519 The value is a translation table to be applied on encoding. This is
|
|
|
1520 not applicable to CCL-based coding systems.
|
|
|
1521
|
|
|
1522 'safe-chars
|
|
|
1523 The value is a char table. If a character has non-nil value in it,
|
|
|
1524 the character is safely supported by the coding system. This
|
|
|
1525 overrides the specification of safe-charsets.
|
|
|
1526
|
|
|
1527 'safe-charsets
|
|
|
1528 The value is a list of charsets safely supported by the coding
|
|
|
1529 system. The value t means that all charsets Emacs handles are
|
|
|
1530 supported. Even if some charset is not in this list, it doesn't
|
|
|
1531 mean that the charset can't be encoded in the coding system;
|
|
|
1532 it just means that some other receiver of text encoded
|
|
|
1533 in the coding system won't be able to handle that charset.
|
|
|
1534
|
|
|
1535 'mime-charset
|
|
|
1536 The value is a symbol of which name is `MIME-charset' parameter of
|
|
|
1537 the coding system.
|
|
|
1538
|
|
|
1539 'valid-codes (meaningful only for a coding system based on CCL)
|
|
|
1540 The value is a list to indicate valid byte ranges of the encoded
|
|
|
1541 file. Each element of the list is an integer or a cons of integer.
|
|
|
1542 In the former case, the integer value is a valid byte code. In the
|
|
|
1543 latter case, the integers specifies the range of valid byte codes.
|
|
|
1544
|
|
|
1545
|
|
|
1546
|
|
|
1547 The following additional property is recognized if TYPE is 'convert-eol:
|
|
|
1548
|
|
|
1549 'subtype
|
|
793
|
1550 One of `lf', `crlf', `cr' or nil (for autodetection). When decoding,
|
|
|
1551 the corresponding sequence will be converted to LF. When encoding,
|
|
|
1552 the opposite happens. This coding system converts characters to
|
|
771
|
1553 characters.
|
|
|
1554
|
|
428
|
1555
|
|
|
1556
|
|
|
1557 The following additional properties are recognized if TYPE is 'iso2022:
|
|
|
1558
|
|
|
1559 'charset-g0
|
|
|
1560 'charset-g1
|
|
|
1561 'charset-g2
|
|
|
1562 'charset-g3
|
|
|
1563 The character set initially designated to the G0 - G3 registers.
|
|
|
1564 The value should be one of
|
|
|
1565
|
|
|
1566 -- A charset object (designate that character set)
|
|
|
1567 -- nil (do not ever use this register)
|
|
|
1568 -- t (no character set is initially designated to
|
|
|
1569 the register, but may be later on; this automatically
|
|
|
1570 sets the corresponding `force-g*-on-output' property)
|
|
|
1571
|
|
|
1572 'force-g0-on-output
|
|
|
1573 'force-g1-on-output
|
|
|
1574 'force-g2-on-output
|
|
|
1575 'force-g2-on-output
|
|
|
1576 If non-nil, send an explicit designation sequence on output before
|
|
|
1577 using the specified register.
|
|
|
1578
|
|
|
1579 'short
|
|
|
1580 If non-nil, use the short forms "ESC $ @", "ESC $ A", and
|
|
|
1581 "ESC $ B" on output in place of the full designation sequences
|
|
|
1582 "ESC $ ( @", "ESC $ ( A", and "ESC $ ( B".
|
|
|
1583
|
|
|
1584 'no-ascii-eol
|
|
|
1585 If non-nil, don't designate ASCII to G0 at each end of line on output.
|
|
|
1586 Setting this to non-nil also suppresses other state-resetting that
|
|
|
1587 normally happens at the end of a line.
|
|
|
1588
|
|
|
1589 'no-ascii-cntl
|
|
|
1590 If non-nil, don't designate ASCII to G0 before control chars on output.
|
|
|
1591
|
|
|
1592 'seven
|
|
|
1593 If non-nil, use 7-bit environment on output. Otherwise, use 8-bit
|
|
|
1594 environment.
|
|
|
1595
|
|
|
1596 'lock-shift
|
|
|
1597 If non-nil, use locking-shift (SO/SI) instead of single-shift
|
|
|
1598 or designation by escape sequence.
|
|
|
1599
|
|
|
1600 'no-iso6429
|
|
|
1601 If non-nil, don't use ISO6429's direction specification.
|
|
|
1602
|
|
|
1603 'escape-quoted
|
|
|
1604 If non-nil, literal control characters that are the same as
|
|
|
1605 the beginning of a recognized ISO2022 or ISO6429 escape sequence
|
|
|
1606 (in particular, ESC (0x1B), SO (0x0E), SI (0x0F), SS2 (0x8E),
|
|
|
1607 SS3 (0x8F), and CSI (0x9B)) are "quoted" with an escape character
|
|
|
1608 so that they can be properly distinguished from an escape sequence.
|
|
|
1609 (Note that doing this results in a non-portable encoding.) This
|
|
|
1610 encoding flag is used for byte-compiled files. Note that ESC
|
|
|
1611 is a good choice for a quoting character because there are no
|
|
|
1612 escape sequences whose second byte is a character from the Control-0
|
|
|
1613 or Control-1 character sets; this is explicitly disallowed by the
|
|
|
1614 ISO2022 standard.
|
|
|
1615
|
|
|
1616 'input-charset-conversion
|
|
|
1617 A list of conversion specifications, specifying conversion of
|
|
|
1618 characters in one charset to another when decoding is performed.
|
|
|
1619 Each specification is a list of two elements: the source charset,
|
|
|
1620 and the destination charset.
|
|
|
1621
|
|
|
1622 'output-charset-conversion
|
|
|
1623 A list of conversion specifications, specifying conversion of
|
|
|
1624 characters in one charset to another when encoding is performed.
|
|
|
1625 The form of each specification is the same as for
|
|
|
1626 'input-charset-conversion.
|
|
|
1627
|
|
|
1628
|
|
771
|
1629
|
|
428
|
1630 The following additional properties are recognized (and required)
|
|
|
1631 if TYPE is 'ccl:
|
|
|
1632
|
|
|
1633 'decode
|
|
|
1634 CCL program used for decoding (converting to internal format).
|
|
|
1635
|
|
|
1636 'encode
|
|
|
1637 CCL program used for encoding (converting to external format).
|
|
771
|
1638
|
|
|
1639
|
|
|
1640 The following additional properties are recognized if TYPE is 'chain:
|
|
|
1641
|
|
|
1642 'chain
|
|
|
1643 List of coding systems to be chained together, in decoding order.
|
|
|
1644
|
|
|
1645 'canonicalize-after-coding
|
|
|
1646 Coding system to be returned by the detector routines in place of
|
|
|
1647 this coding system.
|
|
|
1648
|
|
|
1649
|
|
|
1650
|
|
|
1651 The following additional properties are recognized if TYPE is 'unicode:
|
|
|
1652
|
|
|
1653 'type
|
|
|
1654 One of `utf-16', `utf-8', `ucs-4', or `utf-7' (the latter is not
|
|
|
1655 yet implemented). `utf-16' is the basic two-byte encoding;
|
|
|
1656 `ucs-4' is the four-byte encoding; `utf-8' is an ASCII-compatible
|
|
|
1657 variable-width 8-bit encoding; `utf-7' is a 7-bit encoding using
|
|
|
1658 only characters that will safely pass through all mail gateways.
|
|
2297
|
1659 [[ This should be \"transformation format\". There should also be
|
|
|
1660 `ucs-2' (or `bmp' -- no surrogates) and `utf-32' (range checked). ]]
|
|
771
|
1661
|
|
|
1662 'little-endian
|
|
|
1663 If non-nil, `utf-16' and `ucs-4' will write out the groups of two
|
|
|
1664 or four bytes little-endian instead of big-endian. This is required,
|
|
|
1665 for example, under Windows.
|
|
|
1666
|
|
|
1667 'need-bom
|
|
|
1668 If non-nil, a byte order mark (BOM, or Unicode FFFE) should be
|
|
|
1669 written out at the beginning of the data. This serves both to
|
|
|
1670 identify the endianness of the following data and to mark the
|
|
|
1671 data as Unicode (at least, this is how Windows uses it).
|
|
2297
|
1672 [[ The correct term is \"signature\", since this technique may also
|
|
|
1673 be used with UTF-8. That is the term used in the standard. ]]
|
|
771
|
1674
|
|
|
1675
|
|
|
1676 The following additional properties are recognized if TYPE is
|
|
|
1677 'mswindows-multibyte:
|
|
|
1678
|
|
|
1679 'code-page
|
|
|
1680 Either a number (specifying a particular code page) or one of the
|
|
|
1681 symbols `ansi', `oem', `mac', or `ebcdic', specifying the ANSI,
|
|
|
1682 OEM, Macintosh, or EBCDIC code page associated with a particular
|
|
|
1683 locale (given by the `locale' property). NOTE: EBCDIC code pages
|
|
|
1684 only exist in Windows 2000 and later.
|
|
|
1685
|
|
|
1686 'locale
|
|
|
1687 If `code-page' is a symbol, this specifies the locale whose code
|
|
|
1688 page of the corresponding type should be used. This should be
|
|
|
1689 one of the following: A cons of two strings, (LANGUAGE
|
|
|
1690 . SUBLANGUAGE) (see `mswindows-set-current-locale'); a string (a
|
|
|
1691 language; SUBLANG_DEFAULT, i.e. the default sublanguage, is
|
|
|
1692 used); or one of the symbols `current', `user-default', or
|
|
|
1693 `system-default', corresponding to the values of
|
|
|
1694 `mswindows-current-locale', `mswindows-user-default-locale', or
|
|
|
1695 `mswindows-system-default-locale', respectively.
|
|
|
1696
|
|
|
1697
|
|
|
1698
|
|
|
1699 The following additional properties are recognized if TYPE is 'undecided:
|
|
2297
|
1700 [[ Doesn't GNU use \"detect-*\" for the following two? ]]
|
|
771
|
1701
|
|
|
1702 'do-eol
|
|
|
1703 Do EOL detection.
|
|
|
1704
|
|
|
1705 'do-coding
|
|
|
1706 Do encoding detection.
|
|
|
1707
|
|
|
1708 'coding-system
|
|
|
1709 If encoding detection is not done, use the specified coding system
|
|
|
1710 to do decoding. This is used internally when implementing coding
|
|
|
1711 systems with an EOL type that specifies autodetection (the default),
|
|
|
1712 so that the detector routines return the proper subsidiary.
|
|
|
1713
|
|
|
1714
|
|
|
1715
|
|
|
1716 The following additional property is recognized if TYPE is 'gzip:
|
|
|
1717
|
|
|
1718 'level
|
|
|
1719 Compression level: 0 through 9, or `default' (currently 6).
|
|
|
1720
|
|
428
|
1721 */
|
|
771
|
1722 (name, type, description, props))
|
|
428
|
1723 {
|
|
771
|
1724 return make_coding_system_1 (name, 0, type, description, props);
|
|
428
|
1725 }
|
|
|
1726
|
|
|
1727 DEFUN ("copy-coding-system", Fcopy_coding_system, 2, 2, 0, /*
|
|
|
1728 Copy OLD-CODING-SYSTEM to NEW-NAME.
|
|
|
1729 If NEW-NAME does not name an existing coding system, a new one will
|
|
|
1730 be created.
|
|
771
|
1731 If you are using this function to create an alias, think again:
|
|
|
1732 Use `define-coding-system-alias' instead.
|
|
428
|
1733 */
|
|
|
1734 (old_coding_system, new_name))
|
|
|
1735 {
|
|
|
1736 Lisp_Object new_coding_system;
|
|
|
1737 old_coding_system = Fget_coding_system (old_coding_system);
|
|
771
|
1738 new_coding_system =
|
|
|
1739 UNBOUNDP (new_name) ? Qnil : Ffind_coding_system (new_name);
|
|
428
|
1740 if (NILP (new_coding_system))
|
|
|
1741 {
|
|
793
|
1742 new_coding_system =
|
|
|
1743 wrap_coding_system
|
|
|
1744 (allocate_coding_system
|
|
|
1745 (XCODING_SYSTEM (old_coding_system)->methods,
|
|
|
1746 XCODING_SYSTEM (old_coding_system)->methods->extra_data_size,
|
|
|
1747 new_name));
|
|
771
|
1748 if (!UNBOUNDP (new_name))
|
|
|
1749 Fputhash (new_name, new_coding_system, Vcoding_system_hash_table);
|
|
428
|
1750 }
|
|
771
|
1751 else if (XCODING_SYSTEM (old_coding_system)->methods !=
|
|
|
1752 XCODING_SYSTEM (new_coding_system)->methods)
|
|
|
1753 invalid_operation_2 ("Coding systems not same type",
|
|
|
1754 old_coding_system, new_coding_system);
|
|
428
|
1755
|
|
|
1756 {
|
|
|
1757 Lisp_Coding_System *to = XCODING_SYSTEM (new_coding_system);
|
|
|
1758 Lisp_Coding_System *from = XCODING_SYSTEM (old_coding_system);
|
|
771
|
1759 copy_sized_lcrecord (to, from, sizeof_coding_system (from));
|
|
428
|
1760 to->name = new_name;
|
|
|
1761 }
|
|
|
1762 return new_coding_system;
|
|
|
1763 }
|
|
|
1764
|
|
771
|
1765 DEFUN ("coding-system-canonical-name-p", Fcoding_system_canonical_name_p,
|
|
|
1766 1, 1, 0, /*
|
|
440
|
1767 Return t if OBJECT names a coding system, and is not a coding system alias.
|
|
428
|
1768 */
|
|
440
|
1769 (object))
|
|
|
1770 {
|
|
|
1771 return CODING_SYSTEMP (Fgethash (object, Vcoding_system_hash_table, Qnil))
|
|
|
1772 ? Qt : Qnil;
|
|
|
1773 }
|
|
|
1774
|
|
2297
|
1775 /* #### Shouldn't this really be a find/get pair? */
|
|
|
1776
|
|
440
|
1777 DEFUN ("coding-system-alias-p", Fcoding_system_alias_p, 1, 1, 0, /*
|
|
|
1778 Return t if OBJECT is a coding system alias.
|
|
|
1779 All coding system aliases are created by `define-coding-system-alias'.
|
|
|
1780 */
|
|
|
1781 (object))
|
|
428
|
1782 {
|
|
440
|
1783 return SYMBOLP (Fgethash (object, Vcoding_system_hash_table, Qzero))
|
|
|
1784 ? Qt : Qnil;
|
|
|
1785 }
|
|
|
1786
|
|
|
1787 DEFUN ("coding-system-aliasee", Fcoding_system_aliasee, 1, 1, 0, /*
|
|
|
1788 Return the coding-system symbol for which symbol ALIAS is an alias.
|
|
|
1789 */
|
|
|
1790 (alias))
|
|
|
1791 {
|
|
|
1792 Lisp_Object aliasee = Fgethash (alias, Vcoding_system_hash_table, Qnil);
|
|
|
1793 if (SYMBOLP (aliasee))
|
|
|
1794 return aliasee;
|
|
|
1795 else
|
|
563
|
1796 invalid_argument ("Symbol is not a coding system alias", alias);
|
|
1204
|
1797 RETURN_NOT_REACHED (Qnil);
|
|
440
|
1798 }
|
|
|
1799
|
|
|
1800 /* A maphash function, for removing dangling coding system aliases. */
|
|
|
1801 static int
|
|
2286
|
1802 dangling_coding_system_alias_p (Lisp_Object UNUSED (alias),
|
|
440
|
1803 Lisp_Object aliasee,
|
|
|
1804 void *dangling_aliases)
|
|
|
1805 {
|
|
|
1806 if (SYMBOLP (aliasee)
|
|
|
1807 && NILP (Fgethash (aliasee, Vcoding_system_hash_table, Qnil)))
|
|
428
|
1808 {
|
|
440
|
1809 (*(int *) dangling_aliases)++;
|
|
|
1810 return 1;
|
|
428
|
1811 }
|
|
440
|
1812 else
|
|
|
1813 return 0;
|
|
|
1814 }
|
|
|
1815
|
|
|
1816 DEFUN ("define-coding-system-alias", Fdefine_coding_system_alias, 2, 2, 0, /*
|
|
|
1817 Define symbol ALIAS as an alias for coding system ALIASEE.
|
|
|
1818
|
|
|
1819 You can use this function to redefine an alias that has already been defined,
|
|
|
1820 but you cannot redefine a name which is the canonical name for a coding system.
|
|
|
1821 \(a canonical name of a coding system is what is returned when you call
|
|
|
1822 `coding-system-name' on a coding system).
|
|
|
1823
|
|
|
1824 ALIASEE itself can be an alias, which allows you to define nested aliases.
|
|
|
1825
|
|
|
1826 You are forbidden, however, from creating alias loops or `dangling' aliases.
|
|
|
1827 These will be detected, and an error will be signaled if you attempt to do so.
|
|
|
1828
|
|
|
1829 If ALIASEE is nil, then ALIAS will simply be undefined.
|
|
|
1830
|
|
|
1831 See also `coding-system-alias-p', `coding-system-aliasee',
|
|
|
1832 and `coding-system-canonical-name-p'.
|
|
|
1833 */
|
|
|
1834 (alias, aliasee))
|
|
|
1835 {
|
|
2286
|
1836 Lisp_Object probe;
|
|
440
|
1837
|
|
|
1838 CHECK_SYMBOL (alias);
|
|
|
1839
|
|
|
1840 if (!NILP (Fcoding_system_canonical_name_p (alias)))
|
|
563
|
1841 invalid_change
|
|
440
|
1842 ("Symbol is the canonical name of a coding system and cannot be redefined",
|
|
|
1843 alias);
|
|
|
1844
|
|
|
1845 if (NILP (aliasee))
|
|
|
1846 {
|
|
771
|
1847 Lisp_Object subsidiary_unix = add_suffix_to_symbol (alias, "-unix");
|
|
|
1848 Lisp_Object subsidiary_dos = add_suffix_to_symbol (alias, "-dos");
|
|
|
1849 Lisp_Object subsidiary_mac = add_suffix_to_symbol (alias, "-mac");
|
|
440
|
1850
|
|
|
1851 Fremhash (alias, Vcoding_system_hash_table);
|
|
|
1852
|
|
|
1853 /* Undefine subsidiary aliases,
|
|
|
1854 presumably created by a previous call to this function */
|
|
|
1855 if (! NILP (Fcoding_system_alias_p (subsidiary_unix)) &&
|
|
|
1856 ! NILP (Fcoding_system_alias_p (subsidiary_dos)) &&
|
|
|
1857 ! NILP (Fcoding_system_alias_p (subsidiary_mac)))
|
|
|
1858 {
|
|
|
1859 Fdefine_coding_system_alias (subsidiary_unix, Qnil);
|
|
|
1860 Fdefine_coding_system_alias (subsidiary_dos, Qnil);
|
|
|
1861 Fdefine_coding_system_alias (subsidiary_mac, Qnil);
|
|
|
1862 }
|
|
|
1863
|
|
|
1864 /* Undefine dangling coding system aliases. */
|
|
|
1865 {
|
|
|
1866 int dangling_aliases;
|
|
|
1867
|
|
|
1868 do {
|
|
|
1869 dangling_aliases = 0;
|
|
|
1870 elisp_map_remhash (dangling_coding_system_alias_p,
|
|
|
1871 Vcoding_system_hash_table,
|
|
|
1872 &dangling_aliases);
|
|
|
1873 } while (dangling_aliases > 0);
|
|
|
1874 }
|
|
|
1875
|
|
|
1876 return Qnil;
|
|
|
1877 }
|
|
|
1878
|
|
|
1879 if (CODING_SYSTEMP (aliasee))
|
|
|
1880 aliasee = XCODING_SYSTEM_NAME (aliasee);
|
|
|
1881
|
|
|
1882 /* Checks that aliasee names a coding-system */
|
|
2286
|
1883 (void) Fget_coding_system (aliasee);
|
|
440
|
1884
|
|
|
1885 /* Check for coding system alias loops */
|
|
|
1886 if (EQ (alias, aliasee))
|
|
563
|
1887 alias_loop: invalid_operation_2
|
|
440
|
1888 ("Attempt to create a coding system alias loop", alias, aliasee);
|
|
|
1889
|
|
|
1890 for (probe = aliasee;
|
|
|
1891 SYMBOLP (probe);
|
|
|
1892 probe = Fgethash (probe, Vcoding_system_hash_table, Qzero))
|
|
|
1893 {
|
|
|
1894 if (EQ (probe, alias))
|
|
|
1895 goto alias_loop;
|
|
|
1896 }
|
|
|
1897
|
|
|
1898 Fputhash (alias, aliasee, Vcoding_system_hash_table);
|
|
|
1899
|
|
|
1900 /* Set up aliases for subsidiaries.
|
|
2297
|
1901 #### There must be a better way to handle subsidiary coding systems.
|
|
|
1902 Inquiring Minds Want To Know: shouldn't they always be chains? */
|
|
440
|
1903 {
|
|
|
1904 static const char *suffixes[] = { "-unix", "-dos", "-mac" };
|
|
|
1905 int i;
|
|
|
1906 for (i = 0; i < countof (suffixes); i++)
|
|
|
1907 {
|
|
|
1908 Lisp_Object alias_subsidiary =
|
|
771
|
1909 add_suffix_to_symbol (alias, suffixes[i]);
|
|
440
|
1910 Lisp_Object aliasee_subsidiary =
|
|
771
|
1911 add_suffix_to_symbol (aliasee, suffixes[i]);
|
|
440
|
1912
|
|
|
1913 if (! NILP (Ffind_coding_system (aliasee_subsidiary)))
|
|
|
1914 Fdefine_coding_system_alias (alias_subsidiary, aliasee_subsidiary);
|
|
|
1915 }
|
|
|
1916 }
|
|
428
|
1917 /* FSF return value is a vector of [ALIAS-unix ALIAS-dos ALIAS-mac],
|
|
|
1918 but it doesn't look intentional, so I'd rather return something
|
|
|
1919 meaningful or nothing at all. */
|
|
|
1920 return Qnil;
|
|
|
1921 }
|
|
|
1922
|
|
|
1923 static Lisp_Object
|
|
771
|
1924 subsidiary_coding_system (Lisp_Object coding_system, enum eol_type type)
|
|
428
|
1925 {
|
|
|
1926 Lisp_Coding_System *cs = XCODING_SYSTEM (coding_system);
|
|
|
1927 Lisp_Object new_coding_system;
|
|
|
1928
|
|
|
1929 switch (type)
|
|
|
1930 {
|
|
|
1931 case EOL_AUTODETECT: return coding_system;
|
|
|
1932 case EOL_LF: new_coding_system = CODING_SYSTEM_EOL_LF (cs); break;
|
|
|
1933 case EOL_CR: new_coding_system = CODING_SYSTEM_EOL_CR (cs); break;
|
|
|
1934 case EOL_CRLF: new_coding_system = CODING_SYSTEM_EOL_CRLF (cs); break;
|
|
442
|
1935 default: abort (); return Qnil;
|
|
428
|
1936 }
|
|
|
1937
|
|
|
1938 return NILP (new_coding_system) ? coding_system : new_coding_system;
|
|
|
1939 }
|
|
|
1940
|
|
|
1941 DEFUN ("subsidiary-coding-system", Fsubsidiary_coding_system, 2, 2, 0, /*
|
|
|
1942 Return the subsidiary coding system of CODING-SYSTEM with eol type EOL-TYPE.
|
|
771
|
1943 The logically opposite operation is `coding-system-base'.
|
|
428
|
1944 */
|
|
|
1945 (coding_system, eol_type))
|
|
|
1946 {
|
|
771
|
1947 coding_system = get_coding_system_for_text_file (coding_system, 0);
|
|
428
|
1948
|
|
|
1949 return subsidiary_coding_system (coding_system,
|
|
|
1950 symbol_to_eol_type (eol_type));
|
|
|
1951 }
|
|
|
1952
|
|
771
|
1953 DEFUN ("coding-system-base", Fcoding_system_base,
|
|
|
1954 1, 1, 0, /*
|
|
|
1955 Return the base coding system of CODING-SYSTEM.
|
|
|
1956 If CODING-SYSTEM is a subsidiary, this returns its parent; otherwise, it
|
|
|
1957 returns CODING-SYSTEM.
|
|
|
1958 The logically opposite operation is `subsidiary-coding-system'.
|
|
|
1959 */
|
|
|
1960 (coding_system))
|
|
|
1961 {
|
|
|
1962 Lisp_Object base;
|
|
|
1963
|
|
|
1964 coding_system = Fget_coding_system (coding_system);
|
|
|
1965 if (EQ (XCODING_SYSTEM_NAME (coding_system), Qbinary))
|
|
|
1966 return Fget_coding_system (Qraw_text); /* hack! */
|
|
|
1967 base = XCODING_SYSTEM_SUBSIDIARY_PARENT (coding_system);
|
|
|
1968 if (!NILP (base))
|
|
|
1969 return base;
|
|
|
1970 return coding_system;
|
|
|
1971 }
|
|
|
1972
|
|
|
1973 DEFUN ("coding-system-used-for-io", Fcoding_system_used_for_io,
|
|
|
1974 1, 1, 0, /*
|
|
|
1975 Return the coding system actually used for I/O.
|
|
|
1976 In some cases (e.g. when a particular EOL type is specified) this won't be
|
|
2297
|
1977 the coding system itself. This can be useful when trying to determine
|
|
|
1978 precisely how data was decoded.
|
|
771
|
1979 */
|
|
|
1980 (coding_system))
|
|
|
1981 {
|
|
|
1982 Lisp_Object canon;
|
|
|
1983
|
|
|
1984 coding_system = Fget_coding_system (coding_system);
|
|
|
1985 canon = XCODING_SYSTEM_CANONICAL (coding_system);
|
|
|
1986 if (!NILP (canon))
|
|
|
1987 return canon;
|
|
|
1988 return coding_system;
|
|
|
1989 }
|
|
|
1990
|
|
428
|
1991
|
|
|
1992 /************************************************************************/
|
|
|
1993 /* Coding system accessors */
|
|
|
1994 /************************************************************************/
|
|
|
1995
|
|
771
|
1996 DEFUN ("coding-system-description", Fcoding_system_description, 1, 1, 0, /*
|
|
|
1997 Return the description for CODING-SYSTEM.
|
|
|
1998 The `description' of a coding system is a short English phrase giving the
|
|
|
1999 name rendered according to English punctuation rules, plus possibly some
|
|
|
2000 explanatory text (typically in the form of a parenthetical phrase). The
|
|
|
2001 description is intended to be short enough that it can appear as a menu item,
|
|
|
2002 and clear enough to be recognizable even to someone who is assumed to have
|
|
|
2003 some basic familiarity with different encodings but may not know all the
|
|
|
2004 technical names; thus, for `cn-gb-2312' is described as "Chinese EUC" and
|
|
|
2005 `hz-gb-2312' is described as "Hz/ZW (Chinese)", where the actual name of
|
|
|
2006 the encoding is given, followed by a note that this is a Chinese encoding,
|
|
|
2007 because the great majority of people encountering this would have no idea
|
|
|
2008 what it is, and giving the language indicates whether the encoding should
|
|
|
2009 just be ignored or (conceivably) investigated more thoroughly.
|
|
428
|
2010 */
|
|
|
2011 (coding_system))
|
|
|
2012 {
|
|
|
2013 coding_system = Fget_coding_system (coding_system);
|
|
771
|
2014 return XCODING_SYSTEM_DESCRIPTION (coding_system);
|
|
428
|
2015 }
|
|
|
2016
|
|
|
2017 DEFUN ("coding-system-type", Fcoding_system_type, 1, 1, 0, /*
|
|
|
2018 Return the type of CODING-SYSTEM.
|
|
|
2019 */
|
|
|
2020 (coding_system))
|
|
|
2021 {
|
|
771
|
2022 coding_system = Fget_coding_system (coding_system);
|
|
|
2023 return XCODING_SYSTEM_TYPE (coding_system);
|
|
428
|
2024 }
|
|
|
2025
|
|
|
2026 DEFUN ("coding-system-property", Fcoding_system_property, 2, 2, 0, /*
|
|
|
2027 Return the PROP property of CODING-SYSTEM.
|
|
|
2028 */
|
|
|
2029 (coding_system, prop))
|
|
|
2030 {
|
|
|
2031 coding_system = Fget_coding_system (coding_system);
|
|
|
2032 CHECK_SYMBOL (prop);
|
|
|
2033
|
|
|
2034 if (EQ (prop, Qname))
|
|
|
2035 return XCODING_SYSTEM_NAME (coding_system);
|
|
|
2036 else if (EQ (prop, Qtype))
|
|
|
2037 return Fcoding_system_type (coding_system);
|
|
771
|
2038 else if (EQ (prop, Qdescription))
|
|
|
2039 return XCODING_SYSTEM_DESCRIPTION (coding_system);
|
|
428
|
2040 else if (EQ (prop, Qmnemonic))
|
|
|
2041 return XCODING_SYSTEM_MNEMONIC (coding_system);
|
|
771
|
2042 else if (EQ (prop, Qdocumentation))
|
|
|
2043 return XCODING_SYSTEM_DOCUMENTATION (coding_system);
|
|
428
|
2044 else if (EQ (prop, Qeol_type))
|
|
771
|
2045 return eol_type_to_symbol (XCODING_SYSTEM_EOL_TYPE
|
|
|
2046 (coding_system));
|
|
428
|
2047 else if (EQ (prop, Qeol_lf))
|
|
|
2048 return XCODING_SYSTEM_EOL_LF (coding_system);
|
|
|
2049 else if (EQ (prop, Qeol_crlf))
|
|
|
2050 return XCODING_SYSTEM_EOL_CRLF (coding_system);
|
|
|
2051 else if (EQ (prop, Qeol_cr))
|
|
|
2052 return XCODING_SYSTEM_EOL_CR (coding_system);
|
|
|
2053 else if (EQ (prop, Qpost_read_conversion))
|
|
|
2054 return XCODING_SYSTEM_POST_READ_CONVERSION (coding_system);
|
|
|
2055 else if (EQ (prop, Qpre_write_conversion))
|
|
|
2056 return XCODING_SYSTEM_PRE_WRITE_CONVERSION (coding_system);
|
|
771
|
2057 else
|
|
|
2058 {
|
|
|
2059 Lisp_Object value = CODESYSMETH_OR_GIVEN (XCODING_SYSTEM (coding_system),
|
|
|
2060 getprop,
|
|
|
2061 (coding_system, prop),
|
|
|
2062 Qunbound);
|
|
|
2063 if (UNBOUNDP (value))
|
|
|
2064 invalid_constant ("Unrecognized property", prop);
|
|
|
2065 return value;
|
|
|
2066 }
|
|
|
2067 }
|
|
|
2068
|
|
|
2069
|
|
|
2070 /************************************************************************/
|
|
|
2071 /* Coding stream functions */
|
|
|
2072 /************************************************************************/
|
|
|
2073
|
|
|
2074 /* A coding stream is a stream used for encoding or decoding text. The
|
|
|
2075 coding-stream object keeps track of the actual coding system, the stream
|
|
|
2076 that is at the other end, and data that needs to be persistent across
|
|
|
2077 the lifetime of the stream. */
|
|
|
2078
|
|
1204
|
2079 extern const struct sized_memory_description chain_coding_stream_description;
|
|
|
2080 extern const struct sized_memory_description undecided_coding_stream_description;
|
|
|
2081
|
|
|
2082 static const struct memory_description coding_stream_data_description_1 []= {
|
|
|
2083 { XD_STRUCT_PTR, chain_coding_system, 1, &chain_coding_stream_description},
|
|
|
2084 { XD_STRUCT_PTR, undecided_coding_system, 1, &undecided_coding_stream_description},
|
|
|
2085 { XD_END }
|
|
|
2086 };
|
|
|
2087
|
|
|
2088 static const struct sized_memory_description coding_stream_data_description = {
|
|
|
2089 sizeof (void *), coding_stream_data_description_1
|
|
|
2090 };
|
|
|
2091
|
|
|
2092 static const struct memory_description coding_lstream_description[] = {
|
|
|
2093 { XD_INT, offsetof (struct coding_stream, type) },
|
|
|
2094 { XD_LISP_OBJECT, offsetof (struct coding_stream, orig_codesys) },
|
|
|
2095 { XD_LISP_OBJECT, offsetof (struct coding_stream, codesys) },
|
|
|
2096 { XD_LISP_OBJECT, offsetof (struct coding_stream, other_end) },
|
|
|
2097 { XD_UNION, offsetof (struct coding_stream, data),
|
|
|
2098 XD_INDIRECT (0, 0), &coding_stream_data_description },
|
|
|
2099 { XD_END }
|
|
|
2100 };
|
|
|
2101
|
|
|
2102 DEFINE_LSTREAM_IMPLEMENTATION_WITH_DATA ("coding", coding);
|
|
771
|
2103
|
|
|
2104 /* Encoding and decoding are parallel operations, so we create just one
|
|
|
2105 stream for both. "Decoding" may involve the extra step of autodetection
|
|
|
2106 of the data format, but that's only because of the conventional
|
|
|
2107 definition of decoding as converting from external- to
|
|
|
2108 internal-formatted data.
|
|
|
2109
|
|
2297
|
2110 [[ REWRITE ME! ]]
|
|
|
2111
|
|
771
|
2112 #### We really need to abstract out the concept of "data formats" and
|
|
|
2113 define "converters" that convert from and to specified formats,
|
|
|
2114 eliminating the idea of decoding and encoding. When specifying a
|
|
|
2115 conversion process, we need to give the data formats themselves, not the
|
|
|
2116 conversion processes -- e.g. a coding system called "Unicode->multibyte"
|
|
|
2117 converts in both directions, and we could auto-detect the format of data
|
|
|
2118 at either end. */
|
|
|
2119
|
|
|
2120 static Bytecount
|
|
|
2121 coding_reader (Lstream *stream, unsigned char *data, Bytecount size)
|
|
|
2122 {
|
|
|
2123 unsigned char *orig_data = data;
|
|
|
2124 Bytecount read_size;
|
|
|
2125 int error_occurred = 0;
|
|
|
2126 struct coding_stream *str = CODING_STREAM_DATA (stream);
|
|
|
2127
|
|
|
2128 /* We need to interface to coding_{de,en}code_1(), which expects to take
|
|
|
2129 some amount of data and store the result into a Dynarr. We have
|
|
|
2130 coding_{de,en}code_1() store into c->runoff, and take data from there
|
|
|
2131 as necessary. */
|
|
|
2132
|
|
|
2133 /* We loop until we have enough data, reading chunks from the other
|
|
|
2134 end and converting it. */
|
|
|
2135 while (1)
|
|
|
2136 {
|
|
|
2137 /* Take data from convert_to if we can. Make sure to take at
|
|
|
2138 most SIZE bytes, and delete the data from convert_to. */
|
|
|
2139 if (Dynarr_length (str->convert_to) > 0)
|
|
|
2140 {
|
|
|
2141 Bytecount chunk =
|
|
|
2142 min (size, (Bytecount) Dynarr_length (str->convert_to));
|
|
|
2143 memcpy (data, Dynarr_atp (str->convert_to, 0), chunk);
|
|
|
2144 Dynarr_delete_many (str->convert_to, 0, chunk);
|
|
|
2145 data += chunk;
|
|
|
2146 size -= chunk;
|
|
|
2147 }
|
|
|
2148
|
|
|
2149 if (size == 0)
|
|
|
2150 break; /* No more room for data */
|
|
|
2151
|
|
|
2152 if (str->eof)
|
|
|
2153 break;
|
|
|
2154
|
|
|
2155 {
|
|
|
2156 /* Exhausted convert_to, so get some more. Read into convert_from,
|
|
|
2157 after existing "rejected" data from the last conversion. */
|
|
|
2158 Bytecount rejected = Dynarr_length (str->convert_from);
|
|
|
2159 /* #### 1024 is arbitrary; we really need to separate 0 from EOF,
|
|
|
2160 and when we get 0, keep taking more data until we don't get 0 --
|
|
|
2161 we don't know how much data the conversion routine might need
|
|
2297
|
2162 before it can generate any data of its own (eg, bzip2). */
|
|
814
|
2163 Bytecount readmore =
|
|
|
2164 str->one_byte_at_a_time ? (Bytecount) 1 :
|
|
|
2165 max (size, (Bytecount) 1024);
|
|
771
|
2166
|
|
|
2167 Dynarr_add_many (str->convert_from, 0, readmore);
|
|
|
2168 read_size = Lstream_read (str->other_end,
|
|
|
2169 Dynarr_atp (str->convert_from, rejected),
|
|
|
2170 readmore);
|
|
|
2171 /* Trim size down to how much we actually got */
|
|
|
2172 Dynarr_set_size (str->convert_from, rejected + max (0, read_size));
|
|
|
2173 }
|
|
|
2174
|
|
|
2175 if (read_size < 0) /* LSTREAM_ERROR */
|
|
|
2176 {
|
|
|
2177 error_occurred = 1;
|
|
|
2178 break;
|
|
|
2179 }
|
|
|
2180 if (read_size == 0) /* LSTREAM_EOF */
|
|
|
2181 /* There might be some more end data produced in the translation,
|
|
|
2182 so we set a flag and call the conversion method once more to
|
|
|
2183 output any final stuff it may be holding, any "go back to a sane
|
|
|
2184 state" escape sequences, etc. The conversion method is free to
|
|
|
2185 look at this flag, and we use it above to stop looping. */
|
|
|
2186 str->eof = 1;
|
|
|
2187 {
|
|
|
2188 Bytecount processed;
|
|
|
2189 Bytecount to_process = Dynarr_length (str->convert_from);
|
|
|
2190
|
|
|
2191 /* Convert the data, and save any rejected data in convert_from */
|
|
|
2192 processed =
|
|
|
2193 XCODESYSMETH (str->codesys, convert,
|
|
|
2194 (str, Dynarr_atp (str->convert_from, 0),
|
|
|
2195 str->convert_to, to_process));
|
|
|
2196 if (processed < 0)
|
|
|
2197 {
|
|
|
2198 error_occurred = 1;
|
|
|
2199 break;
|
|
|
2200 }
|
|
|
2201 assert (processed <= to_process);
|
|
|
2202 if (processed < to_process)
|
|
|
2203 memmove (Dynarr_atp (str->convert_from, 0),
|
|
|
2204 Dynarr_atp (str->convert_from, processed),
|
|
|
2205 to_process - processed);
|
|
|
2206 Dynarr_set_size (str->convert_from, to_process - processed);
|
|
|
2207 }
|
|
|
2208 }
|
|
|
2209
|
|
|
2210 if (data - orig_data == 0)
|
|
|
2211 return error_occurred ? -1 : 0;
|
|
|
2212 else
|
|
|
2213 return data - orig_data;
|
|
|
2214 }
|
|
|
2215
|
|
|
2216 static Bytecount
|
|
|
2217 coding_writer (Lstream *stream, const unsigned char *data, Bytecount size)
|
|
|
2218 {
|
|
|
2219 struct coding_stream *str = CODING_STREAM_DATA (stream);
|
|
|
2220
|
|
|
2221 /* Convert all our data into convert_to, and then attempt to write
|
|
|
2222 it all out to the other end. */
|
|
|
2223 Dynarr_reset (str->convert_to);
|
|
|
2224 size = XCODESYSMETH (str->codesys, convert,
|
|
|
2225 (str, data, str->convert_to, size));
|
|
|
2226 if (Lstream_write (str->other_end, Dynarr_atp (str->convert_to, 0),
|
|
|
2227 Dynarr_length (str->convert_to)) < 0)
|
|
|
2228 return -1;
|
|
|
2229 else
|
|
|
2230 /* The return value indicates how much of the incoming data was
|
|
|
2231 processed, not how many bytes were written. */
|
|
|
2232 return size;
|
|
|
2233 }
|
|
|
2234
|
|
|
2235 static int
|
|
|
2236 encode_decode_source_sink_type_is_char (Lisp_Object cs,
|
|
|
2237 enum source_or_sink sex,
|
|
|
2238 enum encode_decode direction)
|
|
|
2239 {
|
|
|
2240 return (direction == CODING_DECODE ?
|
|
|
2241 decoding_source_sink_type_is_char (cs, sex) :
|
|
|
2242 encoding_source_sink_type_is_char (cs, sex));
|
|
|
2243 }
|
|
|
2244
|
|
|
2245 /* Ensure that the convert methods only get full characters sent to them to
|
|
|
2246 convert if the source of that conversion is characters; and that no such
|
|
|
2247 full-character checking happens when the source is bytes. Keep in mind
|
|
|
2248 that (1) the conversion_end_type return values take the perspective of
|
|
|
2249 encoding; (2) the source for decoding is the same as the sink for
|
|
|
2250 encoding; (3) when writing, the data is given to us, and we set our own
|
|
|
2251 stream to be character mode or not; (4) when reading, the data comes
|
|
|
2252 from the other_end stream, and we set that one to be character mode or
|
|
|
2253 not. This is consistent with the comment above the prototype for
|
|
|
2254 Lstream_set_character_mode(), which lays out rules for who is allowed to
|
|
|
2255 modify the character type mode on a stream.
|
|
|
2256
|
|
814
|
2257 If we're a read stream, we're always setting character mode on the
|
|
|
2258 source, but we also set it on ourselves consistent with the flag that
|
|
|
2259 can disable this (see again the comment above
|
|
|
2260 Lstream_set_character_mode()).
|
|
|
2261 */
|
|
771
|
2262
|
|
|
2263 static void
|
|
|
2264 set_coding_character_mode (Lstream *stream)
|
|
|
2265 {
|
|
|
2266 struct coding_stream *str = CODING_STREAM_DATA (stream);
|
|
|
2267 Lstream *stream_to_set =
|
|
|
2268 stream->flags & LSTREAM_FL_WRITE ? stream : str->other_end;
|
|
|
2269 if (encode_decode_source_sink_type_is_char
|
|
|
2270 (str->codesys, CODING_SOURCE, str->direction))
|
|
|
2271 Lstream_set_character_mode (stream_to_set);
|
|
|
2272 else
|
|
|
2273 Lstream_unset_character_mode (stream_to_set);
|
|
814
|
2274 if (str->set_char_mode_on_us_when_reading &&
|
|
|
2275 (stream->flags & LSTREAM_FL_READ))
|
|
|
2276 {
|
|
|
2277 if (encode_decode_source_sink_type_is_char
|
|
|
2278 (str->codesys, CODING_SINK, str->direction))
|
|
|
2279 Lstream_set_character_mode (stream);
|
|
|
2280 else
|
|
|
2281 Lstream_unset_character_mode (stream);
|
|
|
2282 }
|
|
771
|
2283 }
|
|
|
2284
|
|
|
2285 static Lisp_Object
|
|
|
2286 coding_marker (Lisp_Object stream)
|
|
|
2287 {
|
|
|
2288 struct coding_stream *str = CODING_STREAM_DATA (XLSTREAM (stream));
|
|
|
2289
|
|
|
2290 mark_object (str->orig_codesys);
|
|
|
2291 mark_object (str->codesys);
|
|
|
2292 MAYBE_XCODESYSMETH (str->codesys, mark_coding_stream, (str));
|
|
|
2293 return wrap_lstream (str->other_end);
|
|
|
2294 }
|
|
|
2295
|
|
|
2296 static int
|
|
|
2297 coding_rewinder (Lstream *stream)
|
|
|
2298 {
|
|
|
2299 struct coding_stream *str = CODING_STREAM_DATA (stream);
|
|
|
2300 MAYBE_XCODESYSMETH (str->codesys, rewind_coding_stream, (str));
|
|
|
2301
|
|
|
2302 str->ch = 0;
|
|
|
2303 Dynarr_reset (str->convert_to);
|
|
|
2304 Dynarr_reset (str->convert_from);
|
|
|
2305 return Lstream_rewind (str->other_end);
|
|
|
2306 }
|
|
|
2307
|
|
|
2308 static int
|
|
|
2309 coding_seekable_p (Lstream *stream)
|
|
|
2310 {
|
|
|
2311 struct coding_stream *str = CODING_STREAM_DATA (stream);
|
|
|
2312 return Lstream_seekable_p (str->other_end);
|
|
|
2313 }
|
|
|
2314
|
|
|
2315 static int
|
|
|
2316 coding_flusher (Lstream *stream)
|
|
|
2317 {
|
|
|
2318 struct coding_stream *str = CODING_STREAM_DATA (stream);
|
|
|
2319 return Lstream_flush (str->other_end);
|
|
|
2320 }
|
|
|
2321
|
|
|
2322 static int
|
|
|
2323 coding_closer (Lstream *stream)
|
|
|
2324 {
|
|
|
2325 struct coding_stream *str = CODING_STREAM_DATA (stream);
|
|
|
2326 if (stream->flags & LSTREAM_FL_WRITE)
|
|
|
2327 {
|
|
|
2328 str->eof = 1;
|
|
|
2329 coding_writer (stream, 0, 0);
|
|
|
2330 str->eof = 0;
|
|
|
2331 }
|
|
|
2332 /* It's safe to free the runoff dynarrs now because they are used only
|
|
|
2333 during conversion. We need to keep the type-specific data around,
|
|
|
2334 though, because of canonicalize_after_coding. */
|
|
|
2335 if (str->convert_to)
|
|
|
2336 {
|
|
|
2337 Dynarr_free (str->convert_to);
|
|
|
2338 str->convert_to = 0;
|
|
|
2339 }
|
|
|
2340 if (str->convert_from)
|
|
428
|
2341 {
|
|
771
|
2342 Dynarr_free (str->convert_from);
|
|
|
2343 str->convert_from = 0;
|
|
|
2344 }
|
|
|
2345
|
|
800
|
2346 if (str->no_close_other)
|
|
|
2347 return Lstream_flush (str->other_end);
|
|
|
2348 else
|
|
|
2349 return Lstream_close (str->other_end);
|
|
771
|
2350 }
|
|
|
2351
|
|
|
2352 static void
|
|
|
2353 coding_finalizer (Lstream *stream)
|
|
|
2354 {
|
|
|
2355 struct coding_stream *str = CODING_STREAM_DATA (stream);
|
|
|
2356
|
|
|
2357 assert (!str->finalized);
|
|
|
2358 MAYBE_XCODESYSMETH (str->codesys, finalize_coding_stream, (str));
|
|
|
2359 if (str->data)
|
|
|
2360 {
|
|
1726
|
2361 xfree (str->data, void *);
|
|
771
|
2362 str->data = 0;
|
|
|
2363 }
|
|
|
2364 str->finalized = 1;
|
|
|
2365 }
|
|
|
2366
|
|
|
2367 static Lisp_Object
|
|
|
2368 coding_stream_canonicalize_after_coding (Lstream *stream)
|
|
|
2369 {
|
|
|
2370 struct coding_stream *str = CODING_STREAM_DATA (stream);
|
|
|
2371
|
|
|
2372 return XCODESYSMETH_OR_GIVEN (str->codesys, canonicalize_after_coding,
|
|
|
2373 (str), str->codesys);
|
|
|
2374 }
|
|
|
2375
|
|
|
2376 Lisp_Object
|
|
|
2377 coding_stream_detected_coding_system (Lstream *stream)
|
|
|
2378 {
|
|
|
2379 Lisp_Object codesys =
|
|
|
2380 coding_stream_canonicalize_after_coding (stream);
|
|
|
2381 if (NILP (codesys))
|
|
|
2382 return Fget_coding_system (Qidentity);
|
|
|
2383 return codesys;
|
|
|
2384 }
|
|
|
2385
|
|
|
2386 Lisp_Object
|
|
|
2387 coding_stream_coding_system (Lstream *stream)
|
|
|
2388 {
|
|
|
2389 return CODING_STREAM_DATA (stream)->codesys;
|
|
|
2390 }
|
|
|
2391
|
|
|
2392 /* Change the coding system associated with a stream. */
|
|
|
2393
|
|
|
2394 void
|
|
|
2395 set_coding_stream_coding_system (Lstream *lstr, Lisp_Object codesys)
|
|
|
2396 {
|
|
|
2397 struct coding_stream *str = CODING_STREAM_DATA (lstr);
|
|
|
2398 if (EQ (str->orig_codesys, codesys))
|
|
|
2399 return;
|
|
|
2400 /* We do the equivalent of closing the stream, destroying it, and
|
|
|
2401 reinitializing it. This includes flushing out the data and signalling
|
|
|
2402 EOF, if we're a writing stream; we also replace the type-specific data
|
|
|
2403 with the data appropriate for the new coding system. */
|
|
|
2404 if (!NILP (str->codesys))
|
|
|
2405 {
|
|
|
2406 if (lstr->flags & LSTREAM_FL_WRITE)
|
|
|
2407 {
|
|
|
2408 Lstream_flush (lstr);
|
|
|
2409 str->eof = 1;
|
|
|
2410 coding_writer (lstr, 0, 0);
|
|
|
2411 str->eof = 0;
|
|
|
2412 }
|
|
|
2413 MAYBE_XCODESYSMETH (str->codesys, finalize_coding_stream, (str));
|
|
|
2414 }
|
|
|
2415 str->orig_codesys = codesys;
|
|
|
2416 str->codesys = coding_system_real_canonical (codesys);
|
|
|
2417
|
|
|
2418 if (str->data)
|
|
|
2419 {
|
|
1726
|
2420 xfree (str->data, void *);
|
|
771
|
2421 str->data = 0;
|
|
428
|
2422 }
|
|
771
|
2423 if (XCODING_SYSTEM_METHODS (str->codesys)->coding_data_size)
|
|
1204
|
2424 {
|
|
|
2425 str->data =
|
|
|
2426 xmalloc_and_zero (XCODING_SYSTEM_METHODS (str->codesys)->
|
|
|
2427 coding_data_size);
|
|
|
2428 str->type = XCODING_SYSTEM_METHODS (str->codesys)->enumtype;
|
|
|
2429 }
|
|
771
|
2430 MAYBE_XCODESYSMETH (str->codesys, init_coding_stream, (str));
|
|
|
2431 /* The new coding system may have different ideas regarding whether its
|
|
|
2432 ends are characters or bytes. */
|
|
|
2433 set_coding_character_mode (lstr);
|
|
|
2434 }
|
|
|
2435
|
|
|
2436 /* WARNING WARNING WARNING WARNING!!!!! If you open up a coding
|
|
|
2437 stream for writing, no automatic code detection will be performed.
|
|
|
2438 The reason for this is that automatic code detection requires a
|
|
|
2439 seekable input. Things will also fail if you open a coding
|
|
|
2440 stream for reading using a non-fully-specified coding system and
|
|
|
2441 a non-seekable input stream. */
|
|
|
2442
|
|
|
2443 static Lisp_Object
|
|
|
2444 make_coding_stream_1 (Lstream *stream, Lisp_Object codesys,
|
|
800
|
2445 const char *mode, enum encode_decode direction,
|
|
802
|
2446 int flags)
|
|
771
|
2447 {
|
|
|
2448 Lstream *lstr = Lstream_new (lstream_coding, mode);
|
|
|
2449 struct coding_stream *str = CODING_STREAM_DATA (lstr);
|
|
|
2450
|
|
|
2451 codesys = Fget_coding_system (codesys);
|
|
|
2452 xzero (*str);
|
|
|
2453 str->codesys = Qnil;
|
|
|
2454 str->orig_codesys = Qnil;
|
|
|
2455 str->us = lstr;
|
|
|
2456 str->other_end = stream;
|
|
|
2457 str->convert_to = Dynarr_new (unsigned_char);
|
|
|
2458 str->convert_from = Dynarr_new (unsigned_char);
|
|
|
2459 str->direction = direction;
|
|
814
|
2460 if (flags & LSTREAM_FL_NO_CLOSE_OTHER)
|
|
802
|
2461 str->no_close_other = 1;
|
|
814
|
2462 if (flags & LSTREAM_FL_READ_ONE_BYTE_AT_A_TIME)
|
|
802
|
2463 str->one_byte_at_a_time = 1;
|
|
814
|
2464 if (!(flags & LSTREAM_FL_NO_INIT_CHAR_MODE_WHEN_READING))
|
|
|
2465 str->set_char_mode_on_us_when_reading = 1;
|
|
802
|
2466
|
|
771
|
2467 set_coding_stream_coding_system (lstr, codesys);
|
|
793
|
2468 return wrap_lstream (lstr);
|
|
771
|
2469 }
|
|
|
2470
|
|
814
|
2471 /* FLAGS:
|
|
|
2472
|
|
|
2473 LSTREAM_FL_NO_CLOSE_OTHER
|
|
|
2474 Don't close STREAM (the stream at the other end) when this stream is
|
|
|
2475 closed.
|
|
|
2476
|
|
|
2477 LSTREAM_FL_READ_ONE_BYTE_AT_A_TIME
|
|
|
2478 When reading from STREAM, read and process one byte at a time rather
|
|
|
2479 than in large chunks. This is for reading from TTY's, so we don't
|
|
|
2480 block. #### We should instead create a non-blocking filedesc stream
|
|
|
2481 that emulates the behavior as necessary using select(), when the
|
|
|
2482 fcntls don't work. (As seems to be the case on Cygwin.)
|
|
|
2483
|
|
|
2484 LSTREAM_FL_NO_INIT_CHAR_MODE_WHEN_READING
|
|
|
2485 When reading from STREAM, read and process one byte at a time rather
|
|
|
2486 than in large chunks. This is for reading from TTY's, so we don't
|
|
|
2487 block. #### We should instead create a non-blocking filedesc stream
|
|
|
2488 that emulates the behavior as necessary using select(), when the
|
|
|
2489 fcntls don't work. (As seems to be the case on Cygwin.)
|
|
|
2490 */
|
|
771
|
2491 Lisp_Object
|
|
|
2492 make_coding_input_stream (Lstream *stream, Lisp_Object codesys,
|
|
802
|
2493 enum encode_decode direction, int flags)
|
|
771
|
2494 {
|
|
800
|
2495 return make_coding_stream_1 (stream, codesys, "r", direction,
|
|
802
|
2496 flags);
|
|
771
|
2497 }
|
|
|
2498
|
|
814
|
2499 /* FLAGS:
|
|
|
2500
|
|
|
2501 LSTREAM_FL_NO_CLOSE_OTHER
|
|
|
2502 Don't close STREAM (the stream at the other end) when this stream is
|
|
|
2503 closed.
|
|
|
2504 */
|
|
771
|
2505 Lisp_Object
|
|
|
2506 make_coding_output_stream (Lstream *stream, Lisp_Object codesys,
|
|
802
|
2507 enum encode_decode direction, int flags)
|
|
771
|
2508 {
|
|
800
|
2509 return make_coding_stream_1 (stream, codesys, "w", direction,
|
|
802
|
2510 flags);
|
|
771
|
2511 }
|
|
|
2512
|
|
|
2513 static Lisp_Object
|
|
|
2514 encode_decode_coding_region (Lisp_Object start, Lisp_Object end,
|
|
|
2515 Lisp_Object coding_system, Lisp_Object buffer,
|
|
|
2516 enum encode_decode direction)
|
|
|
2517 {
|
|
|
2518 Charbpos b, e;
|
|
|
2519 struct buffer *buf = decode_buffer (buffer, 0);
|
|
|
2520 Lisp_Object instream = Qnil, to_outstream = Qnil, outstream = Qnil;
|
|
|
2521 Lisp_Object from_outstream = Qnil, auto_outstream = Qnil;
|
|
|
2522 Lisp_Object lb_outstream = Qnil;
|
|
|
2523 Lisp_Object next;
|
|
|
2524 Lstream *istr, *ostr;
|
|
|
2525 struct gcpro gcpro1, gcpro2, gcpro3, gcpro4, gcpro5;
|
|
|
2526 struct gcpro ngcpro1;
|
|
|
2527 int source_char, sink_char;
|
|
|
2528
|
|
|
2529 get_buffer_range_char (buf, start, end, &b, &e, 0);
|
|
|
2530 barf_if_buffer_read_only (buf, b, e);
|
|
|
2531
|
|
|
2532 GCPRO5 (instream, to_outstream, outstream, from_outstream, lb_outstream);
|
|
|
2533 NGCPRO1 (auto_outstream);
|
|
|
2534
|
|
|
2535 coding_system = Fget_coding_system (coding_system);
|
|
|
2536 source_char = encode_decode_source_sink_type_is_char (coding_system,
|
|
|
2537 CODING_SOURCE,
|
|
|
2538 direction);
|
|
|
2539 sink_char = encode_decode_source_sink_type_is_char (coding_system,
|
|
|
2540 CODING_SINK,
|
|
|
2541 direction);
|
|
|
2542
|
|
|
2543 /* Order is IN <---> [TO] -> OUT -> [FROM] -> [AUTODETECT-EOL] -> LB */
|
|
|
2544 instream = make_lisp_buffer_input_stream (buf, b, e, 0);
|
|
|
2545 next = lb_outstream = make_lisp_buffer_output_stream (buf, b, 0);
|
|
|
2546
|
|
|
2547 if (direction == CODING_DECODE &&
|
|
|
2548 XCODING_SYSTEM_EOL_TYPE (coding_system) == EOL_AUTODETECT)
|
|
|
2549 next = auto_outstream =
|
|
|
2550 make_coding_output_stream
|
|
800
|
2551 (XLSTREAM (next), Fget_coding_system (Qconvert_eol_autodetect),
|
|
|
2552 CODING_DECODE, 0);
|
|
771
|
2553
|
|
|
2554 if (!sink_char)
|
|
|
2555 next = from_outstream =
|
|
800
|
2556 make_coding_output_stream (XLSTREAM (next), Qbinary, CODING_DECODE, 0);
|
|
771
|
2557 outstream = make_coding_output_stream (XLSTREAM (next), coding_system,
|
|
800
|
2558 direction, 0);
|
|
771
|
2559 if (!source_char)
|
|
428
|
2560 {
|
|
771
|
2561 to_outstream =
|
|
|
2562 make_coding_output_stream (XLSTREAM (outstream),
|
|
800
|
2563 Qbinary, CODING_ENCODE, 0);
|
|
771
|
2564 ostr = XLSTREAM (to_outstream);
|
|
|
2565 }
|
|
|
2566 else
|
|
|
2567 ostr = XLSTREAM (outstream);
|
|
|
2568 istr = XLSTREAM (instream);
|
|
|
2569
|
|
|
2570 /* The chain of streams looks like this:
|
|
|
2571
|
|
2297
|
2572 [BUFFER] <----- (( read from/send to loop ))
|
|
771
|
2573 ------> [CHAR->BYTE i.e. ENCODE AS BINARY if source is
|
|
|
2574 in bytes]
|
|
|
2575 ------> [ENCODE/DECODE AS SPECIFIED]
|
|
|
2576 ------> [BYTE->CHAR i.e. DECODE AS BINARY
|
|
|
2577 if sink is in bytes]
|
|
|
2578 ------> [AUTODETECT EOL if
|
|
|
2579 we're decoding and
|
|
|
2580 coding system calls
|
|
|
2581 for this]
|
|
|
2582 ------> [BUFFER]
|
|
|
2583 */
|
|
2297
|
2584 /* Of course, this is just horrible. BYTE<->CHAR should only be available
|
|
|
2585 to I/O routines. It should not be visible to Mule proper.
|
|
|
2586
|
|
|
2587 A comment on the implementation. Hrvoje and Kyle worry about the
|
|
|
2588 inefficiency of repeated copying among buffers that chained coding
|
|
|
2589 systems entail. But this may not be as time inefficient as it appears
|
|
|
2590 in the Mule ("house rules") context. The issue is how do you do chain
|
|
|
2591 coding systems without copying? In theory you could have
|
|
|
2592
|
|
|
2593 IChar external_to_raw (ExtChar *cp, State *s);
|
|
|
2594 IChar decode_utf16 (IChar c, State *s);
|
|
|
2595 IChar decode_crlf (ExtChar *cp, State *s);
|
|
|
2596
|
|
|
2597 typedef Ichar (*Converter[]) (Ichar, State*);
|
|
|
2598
|
|
|
2599 Converter utf16[2] = { &decode_utf16, &decode_crlf };
|
|
|
2600
|
|
|
2601 void convert (ExtChar *inbuf, IChar *outbuf, Converter cvtr)
|
|
|
2602 {
|
|
|
2603 int i;
|
|
|
2604 ExtChar c;
|
|
|
2605 State s;
|
|
|
2606
|
|
|
2607 while (c = external_to_raw (*inbuf++, &s))
|
|
|
2608 {
|
|
|
2609 for (i = 0; i < sizeof(cvtr)/sizeof(Converter); ++i)
|
|
|
2610 if (s.ready)
|
|
|
2611 c = (*cvtr[i]) (c, &s);
|
|
|
2612 }
|
|
|
2613 if (s.ready)
|
|
|
2614 *outbuf++ = c;
|
|
|
2615 }
|
|
|
2616
|
|
|
2617 But this is a lot of function calls; what Ben is doing is basically
|
|
|
2618 reducing this to one call per buffer-full. The only way to avoid this
|
|
|
2619 is to hardcode all the "interesting" coding systems, maybe using
|
|
|
2620 inline or macros to give structure. But this is still a huge amount
|
|
|
2621 of work, and code.
|
|
|
2622
|
|
|
2623 One advantage to the call-per-char approach is that we might be able
|
|
|
2624 to do something about the marker/extent destruction that coding
|
|
|
2625 normally entails.
|
|
|
2626 */
|
|
771
|
2627 while (1)
|
|
|
2628 {
|
|
|
2629 char tempbuf[1024]; /* some random amount */
|
|
|
2630 Charbpos newpos, even_newer_pos;
|
|
|
2631 Charbpos oldpos = lisp_buffer_stream_startpos (istr);
|
|
|
2632 Bytecount size_in_bytes =
|
|
|
2633 Lstream_read (istr, tempbuf, sizeof (tempbuf));
|
|
|
2634
|
|
|
2635 if (!size_in_bytes)
|
|
|
2636 break;
|
|
|
2637 newpos = lisp_buffer_stream_startpos (istr);
|
|
|
2638 Lstream_write (ostr, tempbuf, size_in_bytes);
|
|
|
2639 even_newer_pos = lisp_buffer_stream_startpos (istr);
|
|
|
2640 buffer_delete_range (buf, even_newer_pos - (newpos - oldpos),
|
|
|
2641 even_newer_pos, 0);
|
|
428
|
2642 }
|
|
771
|
2643
|
|
|
2644 {
|
|
|
2645 Charcount retlen =
|
|
|
2646 lisp_buffer_stream_startpos (XLSTREAM (instream)) - b;
|
|
|
2647 Lstream_close (istr);
|
|
|
2648 Lstream_close (ostr);
|
|
|
2649 NUNGCPRO;
|
|
|
2650 UNGCPRO;
|
|
|
2651 Lstream_delete (istr);
|
|
|
2652 if (!NILP (from_outstream))
|
|
|
2653 Lstream_delete (XLSTREAM (from_outstream));
|
|
|
2654 Lstream_delete (XLSTREAM (outstream));
|
|
|
2655 if (!NILP (to_outstream))
|
|
|
2656 Lstream_delete (XLSTREAM (to_outstream));
|
|
|
2657 if (!NILP (auto_outstream))
|
|
|
2658 Lstream_delete (XLSTREAM (auto_outstream));
|
|
|
2659 Lstream_delete (XLSTREAM (lb_outstream));
|
|
|
2660 return make_int (retlen);
|
|
|
2661 }
|
|
|
2662 }
|
|
|
2663
|
|
|
2664 DEFUN ("decode-coding-region", Fdecode_coding_region, 3, 4, 0, /*
|
|
|
2665 Decode the text between START and END which is encoded in CODING-SYSTEM.
|
|
|
2666 This is useful if you've read in encoded text from a file without decoding
|
|
|
2667 it (e.g. you read in a JIS-formatted file but used the `binary' or
|
|
|
2668 `no-conversion' coding system, so that it shows up as "^[$B!<!+^[(B").
|
|
|
2669 Return length of decoded text.
|
|
|
2670 BUFFER defaults to the current buffer if unspecified.
|
|
|
2671 */
|
|
|
2672 (start, end, coding_system, buffer))
|
|
|
2673 {
|
|
|
2674 return encode_decode_coding_region (start, end, coding_system, buffer,
|
|
|
2675 CODING_DECODE);
|
|
|
2676 }
|
|
|
2677
|
|
|
2678 DEFUN ("encode-coding-region", Fencode_coding_region, 3, 4, 0, /*
|
|
|
2679 Encode the text between START and END using CODING-SYSTEM.
|
|
|
2680 This will, for example, convert Japanese characters into stuff such as
|
|
|
2681 "^[$B!<!+^[(B" if you use the JIS encoding. Return length of encoded
|
|
|
2682 text. BUFFER defaults to the current buffer if unspecified.
|
|
|
2683 */
|
|
|
2684 (start, end, coding_system, buffer))
|
|
|
2685 {
|
|
|
2686 return encode_decode_coding_region (start, end, coding_system, buffer,
|
|
|
2687 CODING_ENCODE);
|
|
428
|
2688 }
|
|
|
2689
|
|
|
2690
|
|
|
2691 /************************************************************************/
|
|
771
|
2692 /* Chain methods */
|
|
428
|
2693 /************************************************************************/
|
|
|
2694
|
|
771
|
2695 /* #### Need a way to create "opposite-direction" coding systems. */
|
|
|
2696
|
|
|
2697 /* Chain two or more coding systems together to make a combination coding
|
|
|
2698 system. */
|
|
|
2699
|
|
|
2700 struct chain_coding_system
|
|
|
2701 {
|
|
|
2702 /* List of coding systems, in decode order */
|
|
|
2703 Lisp_Object *chain;
|
|
|
2704 /* Number of coding systems in list */
|
|
|
2705 int count;
|
|
|
2706 /* Coding system to return as a result of canonicalize-after-coding */
|
|
|
2707 Lisp_Object canonicalize_after_coding;
|
|
|
2708 };
|
|
|
2709
|
|
|
2710 struct chain_coding_stream
|
|
|
2711 {
|
|
|
2712 int initted;
|
|
|
2713 /* Lstreams for chain coding system */
|
|
|
2714 Lisp_Object *lstreams;
|
|
|
2715 int lstream_count;
|
|
|
2716 };
|
|
|
2717
|
|
1204
|
2718 static const struct memory_description chain_coding_system_description[] = {
|
|
|
2719 { XD_INT, offsetof (struct chain_coding_system, count) },
|
|
|
2720 { XD_STRUCT_PTR, offsetof (struct chain_coding_system, chain),
|
|
|
2721 XD_INDIRECT (0, 0), &lisp_object_description },
|
|
|
2722 { XD_LISP_OBJECT, offsetof (struct chain_coding_system,
|
|
|
2723 canonicalize_after_coding) },
|
|
771
|
2724 { XD_END }
|
|
|
2725 };
|
|
|
2726
|
|
1204
|
2727 static const struct memory_description chain_coding_stream_description_1 [] = {
|
|
|
2728 { XD_INT, offsetof (struct chain_coding_stream, lstream_count) },
|
|
|
2729 { XD_STRUCT_PTR, offsetof (struct chain_coding_stream, lstreams),
|
|
|
2730 XD_INDIRECT (0, 0), &lisp_object_description },
|
|
771
|
2731 { XD_END }
|
|
|
2732 };
|
|
|
2733
|
|
1204
|
2734 const struct sized_memory_description chain_coding_stream_description = {
|
|
|
2735 sizeof (struct chain_coding_stream), chain_coding_stream_description_1
|
|
|
2736 };
|
|
|
2737
|
|
|
2738 DEFINE_CODING_SYSTEM_TYPE_WITH_DATA (chain);
|
|
|
2739
|
|
771
|
2740 static Lisp_Object
|
|
|
2741 chain_canonicalize (Lisp_Object codesys)
|
|
|
2742 {
|
|
|
2743 /* We make use of the fact that this method is called at init time, after
|
|
|
2744 properties have been parsed. init_method is called too early. */
|
|
|
2745 /* #### It's not clear we need this whole chain-canonicalize mechanism
|
|
|
2746 any more. */
|
|
|
2747 Lisp_Object chain = Flist (XCODING_SYSTEM_CHAIN_COUNT (codesys),
|
|
|
2748 XCODING_SYSTEM_CHAIN_CHAIN (codesys));
|
|
|
2749 chain = Fcons (XCODING_SYSTEM_PRE_WRITE_CONVERSION (codesys),
|
|
|
2750 Fcons (XCODING_SYSTEM_POST_READ_CONVERSION (codesys),
|
|
|
2751 chain));
|
|
|
2752 Fputhash (chain, codesys, Vchain_canonicalize_hash_table);
|
|
|
2753 return codesys;
|
|
|
2754 }
|
|
|
2755
|
|
|
2756 static Lisp_Object
|
|
|
2757 chain_canonicalize_after_coding (struct coding_stream *str)
|
|
|
2758 {
|
|
|
2759 Lisp_Object cac =
|
|
|
2760 XCODING_SYSTEM_CHAIN_CANONICALIZE_AFTER_CODING (str->codesys);
|
|
|
2761 if (!NILP (cac))
|
|
|
2762 return cac;
|
|
|
2763 return str->codesys;
|
|
|
2764 #if 0
|
|
|
2765 struct chain_coding_stream *data = CODING_STREAM_TYPE_DATA (str, chain);
|
|
|
2766 Lisp_Object us = str->codesys, codesys;
|
|
|
2767 int i;
|
|
|
2768 Lisp_Object chain;
|
|
|
2769 Lisp_Object tail;
|
|
|
2770 int changed = 0;
|
|
|
2771
|
|
|
2772 /* #### It's not clear we need this whole chain-canonicalize mechanism
|
|
|
2773 any more. */
|
|
|
2774 if (str->direction == CODING_ENCODE || !data->initted)
|
|
|
2775 return us;
|
|
|
2776
|
|
|
2777 chain = Flist (XCODING_SYSTEM_CHAIN_COUNT (us),
|
|
|
2778 XCODING_SYSTEM_CHAIN_CHAIN (us));
|
|
|
2779
|
|
|
2780 tail = chain;
|
|
|
2781 for (i = 0; i < XCODING_SYSTEM_CHAIN_COUNT (us); i++)
|
|
|
2782 {
|
|
|
2783 codesys = (coding_stream_canonicalize_after_coding
|
|
|
2784 (XLSTREAM (data->lstreams[i])));
|
|
|
2785 if (!EQ (codesys, XCAR (tail)))
|
|
|
2786 changed = 1;
|
|
|
2787 XCAR (tail) = codesys;
|
|
|
2788 tail = XCDR (tail);
|
|
|
2789 }
|
|
|
2790
|
|
|
2791 if (!changed)
|
|
|
2792 return us;
|
|
|
2793
|
|
|
2794 chain = delq_no_quit (Qnil, chain);
|
|
|
2795
|
|
|
2796 if (NILP (XCODING_SYSTEM_PRE_WRITE_CONVERSION (us)) &&
|
|
|
2797 NILP (XCODING_SYSTEM_POST_READ_CONVERSION (us)))
|
|
|
2798 {
|
|
|
2799 if (NILP (chain))
|
|
|
2800 return Qnil;
|
|
|
2801 if (NILP (XCDR (chain)))
|
|
|
2802 return XCAR (chain);
|
|
|
2803 }
|
|
|
2804
|
|
|
2805 codesys = Fgethash (Fcons (XCODING_SYSTEM_PRE_WRITE_CONVERSION (us),
|
|
|
2806 Fcons (XCODING_SYSTEM_POST_READ_CONVERSION (us),
|
|
|
2807 chain)), Vchain_canonicalize_hash_table,
|
|
|
2808 Qnil);
|
|
|
2809 if (!NILP (codesys))
|
|
|
2810 return codesys;
|
|
|
2811 return make_internal_coding_system
|
|
|
2812 (us, "internal-chain-canonicalizer-wrapper",
|
|
|
2813 Qchain, Qunbound, list2 (Qchain, chain));
|
|
|
2814 #endif /* 0 */
|
|
|
2815 }
|
|
|
2816
|
|
|
2817 static void
|
|
|
2818 chain_init (Lisp_Object codesys)
|
|
|
2819 {
|
|
|
2820 XCODING_SYSTEM_CHAIN_CANONICALIZE_AFTER_CODING (codesys) = Qnil;
|
|
|
2821 }
|
|
|
2822
|
|
|
2823 static void
|
|
|
2824 chain_mark (Lisp_Object codesys)
|
|
|
2825 {
|
|
|
2826 int i;
|
|
|
2827
|
|
|
2828 for (i = 0; i < XCODING_SYSTEM_CHAIN_COUNT (codesys); i++)
|
|
|
2829 mark_object (XCODING_SYSTEM_CHAIN_CHAIN (codesys)[i]);
|
|
|
2830 mark_object (XCODING_SYSTEM_CHAIN_CANONICALIZE_AFTER_CODING (codesys));
|
|
|
2831 }
|
|
|
2832
|
|
|
2833 static void
|
|
|
2834 chain_mark_coding_stream_1 (struct chain_coding_stream *data)
|
|
|
2835 {
|
|
|
2836 int i;
|
|
|
2837
|
|
|
2838 for (i = 0; i < data->lstream_count; i++)
|
|
|
2839 mark_object (data->lstreams[i]);
|
|
|
2840 }
|
|
|
2841
|
|
|
2842 static void
|
|
|
2843 chain_mark_coding_stream (struct coding_stream *str)
|
|
|
2844 {
|
|
|
2845 chain_mark_coding_stream_1 (CODING_STREAM_TYPE_DATA (str, chain));
|
|
|
2846 }
|
|
|
2847
|
|
|
2848 static void
|
|
|
2849 chain_print (Lisp_Object cs, Lisp_Object printcharfun, int escapeflag)
|
|
|
2850 {
|
|
|
2851 int i;
|
|
|
2852
|
|
826
|
2853 write_c_string (printcharfun, "(");
|
|
771
|
2854 for (i = 0; i < XCODING_SYSTEM_CHAIN_COUNT (cs); i++)
|
|
|
2855 {
|
|
826
|
2856 write_c_string (printcharfun, i == 0 ? "" : "->");
|
|
771
|
2857 print_coding_system_in_print_method (XCODING_SYSTEM_CHAIN_CHAIN (cs)[i],
|
|
|
2858 printcharfun, escapeflag);
|
|
|
2859 }
|
|
|
2860 {
|
|
|
2861 Lisp_Object cac = XCODING_SYSTEM_CHAIN_CANONICALIZE_AFTER_CODING (cs);
|
|
|
2862 if (!NILP (cac))
|
|
|
2863 {
|
|
|
2864 if (i > 0)
|
|
826
|
2865 write_c_string (printcharfun, " ");
|
|
|
2866 write_c_string (printcharfun, "canonicalize-after-coding=");
|
|
771
|
2867 print_coding_system_in_print_method (cac, printcharfun, escapeflag);
|
|
|
2868 }
|
|
|
2869 }
|
|
|
2870
|
|
826
|
2871 write_c_string (printcharfun, ")");
|
|
771
|
2872 }
|
|
|
2873
|
|
|
2874 static void
|
|
|
2875 chain_rewind_coding_stream_1 (struct chain_coding_stream *data)
|
|
|
2876 {
|
|
|
2877 /* Each will rewind the next; there is always at least one stream (the
|
|
|
2878 dynarr stream at the end) if we're initted */
|
|
|
2879 if (data->initted)
|
|
|
2880 Lstream_rewind (XLSTREAM (data->lstreams[0]));
|
|
|
2881 }
|
|
|
2882
|
|
|
2883 static void
|
|
|
2884 chain_rewind_coding_stream (struct coding_stream *str)
|
|
|
2885 {
|
|
|
2886 chain_rewind_coding_stream_1 (CODING_STREAM_TYPE_DATA (str, chain));
|
|
|
2887 }
|
|
|
2888
|
|
|
2889 static void
|
|
|
2890 chain_init_coding_streams_1 (struct chain_coding_stream *data,
|
|
|
2891 unsigned_char_dynarr *dst,
|
|
|
2892 int ncodesys, Lisp_Object *codesys,
|
|
|
2893 enum encode_decode direction)
|
|
|
2894 {
|
|
|
2895 int i;
|
|
|
2896 Lisp_Object lstream_out;
|
|
|
2897
|
|
|
2898 data->lstream_count = ncodesys + 1;
|
|
|
2899 data->lstreams = xnew_array (Lisp_Object, data->lstream_count);
|
|
|
2900
|
|
|
2901 lstream_out = make_dynarr_output_stream (dst);
|
|
|
2902 Lstream_set_buffering (XLSTREAM (lstream_out), LSTREAM_UNBUFFERED, 0);
|
|
|
2903 data->lstreams[data->lstream_count - 1] = lstream_out;
|
|
|
2904
|
|
|
2905 for (i = ncodesys - 1; i >= 0; i--)
|
|
|
2906 {
|
|
|
2907 data->lstreams[i] =
|
|
|
2908 make_coding_output_stream
|
|
|
2909 (XLSTREAM (lstream_out),
|
|
|
2910 codesys[direction == CODING_ENCODE ? ncodesys - (i + 1) : i],
|
|
800
|
2911 direction, 0);
|
|
771
|
2912 lstream_out = data->lstreams[i];
|
|
|
2913 Lstream_set_buffering (XLSTREAM (lstream_out), LSTREAM_UNBUFFERED,
|
|
|
2914 0);
|
|
|
2915 }
|
|
|
2916 data->initted = 1;
|
|
|
2917 }
|
|
|
2918
|
|
|
2919 static Bytecount
|
|
|
2920 chain_convert (struct coding_stream *str, const UExtbyte *src,
|
|
|
2921 unsigned_char_dynarr *dst, Bytecount n)
|
|
|
2922 {
|
|
|
2923 struct chain_coding_stream *data = CODING_STREAM_TYPE_DATA (str, chain);
|
|
|
2924
|
|
|
2925 if (str->eof)
|
|
|
2926 {
|
|
|
2927 /* Each will close the next; there is always at least one stream (the
|
|
|
2928 dynarr stream at the end) if we're initted. We need to close now
|
|
|
2929 because more data may be generated. */
|
|
|
2930 if (data->initted)
|
|
|
2931 Lstream_close (XLSTREAM (data->lstreams[0]));
|
|
|
2932 return n;
|
|
|
2933 }
|
|
|
2934
|
|
|
2935 if (!data->initted)
|
|
|
2936 chain_init_coding_streams_1
|
|
|
2937 (data, dst, XCODING_SYSTEM_CHAIN_COUNT (str->codesys),
|
|
|
2938 XCODING_SYSTEM_CHAIN_CHAIN (str->codesys), str->direction);
|
|
|
2939
|
|
|
2940 if (Lstream_write (XLSTREAM (data->lstreams[0]), src, n) < 0)
|
|
|
2941 return -1;
|
|
|
2942 return n;
|
|
|
2943 }
|
|
|
2944
|
|
|
2945 static void
|
|
|
2946 chain_finalize_coding_stream_1 (struct chain_coding_stream *data)
|
|
|
2947 {
|
|
|
2948 if (data->lstreams)
|
|
|
2949 {
|
|
2297
|
2950 /* During GC, these objects are unmarked, and are about to be freed.
|
|
|
2951 We do NOT want them on the free list, and that will cause lots of
|
|
|
2952 nastiness including crashes. Just let them be freed normally. */
|
|
771
|
2953 if (!gc_in_progress)
|
|
|
2954 {
|
|
|
2955 int i;
|
|
2297
|
2956 /* Order of deletion is important here! Delete from the head of
|
|
|
2957 the chain and work your way towards the tail. In general,
|
|
|
2958 when you delete an object, there should be *NO* pointers to it
|
|
|
2959 anywhere. Deleting back-to-front would be a problem because
|
|
|
2960 there are pointers going forward. If there were pointers in
|
|
|
2961 both directions, you'd have to disconnect the pointers to a
|
|
|
2962 particular object before deleting it. */
|
|
771
|
2963 for (i = 0; i < data->lstream_count; i++)
|
|
|
2964 Lstream_delete (XLSTREAM ((data->lstreams)[i]));
|
|
|
2965 }
|
|
1726
|
2966 xfree (data->lstreams, Lisp_Object *);
|
|
771
|
2967 }
|
|
|
2968 }
|
|
|
2969
|
|
|
2970 static void
|
|
|
2971 chain_finalize_coding_stream (struct coding_stream *str)
|
|
|
2972 {
|
|
|
2973 chain_finalize_coding_stream_1 (CODING_STREAM_TYPE_DATA (str, chain));
|
|
|
2974 }
|
|
|
2975
|
|
|
2976 static void
|
|
|
2977 chain_finalize (Lisp_Object c)
|
|
|
2978 {
|
|
|
2979 if (XCODING_SYSTEM_CHAIN_CHAIN (c))
|
|
1726
|
2980 xfree (XCODING_SYSTEM_CHAIN_CHAIN (c), Lisp_Object *);
|
|
771
|
2981 }
|
|
|
2982
|
|
428
|
2983 static int
|
|
771
|
2984 chain_putprop (Lisp_Object codesys, Lisp_Object key, Lisp_Object value)
|
|
|
2985 {
|
|
|
2986 if (EQ (key, Qchain))
|
|
|
2987 {
|
|
|
2988 Lisp_Object tail;
|
|
|
2989 Lisp_Object *cslist;
|
|
|
2990 int count = 0;
|
|
|
2991 int i;
|
|
|
2992
|
|
|
2993 EXTERNAL_LIST_LOOP (tail, value)
|
|
|
2994 {
|
|
|
2995 Fget_coding_system (XCAR (tail));
|
|
|
2996 count++;
|
|
|
2997 }
|
|
|
2998
|
|
|
2999 cslist = xnew_array (Lisp_Object, count);
|
|
|
3000 XCODING_SYSTEM_CHAIN_CHAIN (codesys) = cslist;
|
|
|
3001
|
|
|
3002 count = 0;
|
|
|
3003 EXTERNAL_LIST_LOOP (tail, value)
|
|
|
3004 {
|
|
|
3005 cslist[count] = Fget_coding_system (XCAR (tail));
|
|
|
3006 count++;
|
|
|
3007 }
|
|
|
3008
|
|
|
3009 XCODING_SYSTEM_CHAIN_COUNT (codesys) = count;
|
|
|
3010
|
|
|
3011 for (i = 0; i < count - 1; i++)
|
|
|
3012 {
|
|
|
3013 if (decoding_source_sink_type_is_char (cslist[i], CODING_SINK) !=
|
|
|
3014 decoding_source_sink_type_is_char (cslist[i + 1], CODING_SOURCE))
|
|
|
3015 invalid_argument_2 ("Sink of first must match source of second",
|
|
|
3016 cslist[i], cslist[i + 1]);
|
|
|
3017 }
|
|
|
3018 }
|
|
|
3019 else if (EQ (key, Qcanonicalize_after_coding))
|
|
|
3020 XCODING_SYSTEM_CHAIN_CANONICALIZE_AFTER_CODING (codesys) =
|
|
|
3021 Fget_coding_system (value);
|
|
|
3022 else
|
|
|
3023 return 0;
|
|
|
3024 return 1;
|
|
|
3025 }
|
|
|
3026
|
|
|
3027 static Lisp_Object
|
|
|
3028 chain_getprop (Lisp_Object coding_system, Lisp_Object prop)
|
|
|
3029 {
|
|
|
3030 if (EQ (prop, Qchain))
|
|
|
3031 {
|
|
|
3032 Lisp_Object result = Qnil;
|
|
|
3033 int i;
|
|
|
3034
|
|
|
3035 for (i = 0; i < XCODING_SYSTEM_CHAIN_COUNT (coding_system); i++)
|
|
|
3036 result = Fcons (XCODING_SYSTEM_CHAIN_CHAIN (coding_system)[i],
|
|
|
3037 result);
|
|
|
3038
|
|
|
3039 return Fnreverse (result);
|
|
|
3040 }
|
|
|
3041 else if (EQ (prop, Qcanonicalize_after_coding))
|
|
|
3042 return XCODING_SYSTEM_CHAIN_CANONICALIZE_AFTER_CODING (coding_system);
|
|
|
3043 else
|
|
|
3044 return Qunbound;
|
|
|
3045 }
|
|
|
3046
|
|
|
3047 static enum source_sink_type
|
|
|
3048 chain_conversion_end_type (Lisp_Object codesys)
|
|
|
3049 {
|
|
|
3050 Lisp_Object *cslist = XCODING_SYSTEM_CHAIN_CHAIN (codesys);
|
|
|
3051 int n = XCODING_SYSTEM_CHAIN_COUNT (codesys);
|
|
|
3052 int charp_source, charp_sink;
|
|
|
3053
|
|
|
3054 if (n == 0)
|
|
|
3055 return DECODES_BYTE_TO_BYTE; /* arbitrary */
|
|
|
3056 charp_source = decoding_source_sink_type_is_char (cslist[0], CODING_SOURCE);
|
|
|
3057 charp_sink = decoding_source_sink_type_is_char (cslist[n - 1], CODING_SINK);
|
|
|
3058
|
|
|
3059 switch (charp_source * 2 + charp_sink)
|
|
|
3060 {
|
|
|
3061 case 0: return DECODES_BYTE_TO_BYTE;
|
|
|
3062 case 1: return DECODES_BYTE_TO_CHARACTER;
|
|
|
3063 case 2: return DECODES_CHARACTER_TO_BYTE;
|
|
|
3064 case 3: return DECODES_CHARACTER_TO_CHARACTER;
|
|
|
3065 }
|
|
|
3066
|
|
|
3067 abort ();
|
|
|
3068 return DECODES_BYTE_TO_BYTE;
|
|
|
3069 }
|
|
|
3070
|
|
|
3071
|
|
|
3072 /************************************************************************/
|
|
|
3073 /* No-conversion methods */
|
|
|
3074 /************************************************************************/
|
|
|
3075
|
|
|
3076 /* "No conversion"; used for binary files. We use quotes because there
|
|
|
3077 really is some conversion being applied (it does byte<->char
|
|
|
3078 conversion), but it appears to the user as if the text is read in
|
|
2297
|
3079 without conversion.
|
|
|
3080
|
|
|
3081 #### Shouldn't we _call_ it that, then? And while we're at it,
|
|
|
3082 separate it into "to_internal" and "to_external"? */
|
|
771
|
3083 DEFINE_CODING_SYSTEM_TYPE (no_conversion);
|
|
|
3084
|
|
|
3085 /* This is used when reading in "binary" files -- i.e. files that may
|
|
|
3086 contain all 256 possible byte values and that are not to be
|
|
|
3087 interpreted as being in any particular encoding. */
|
|
|
3088 static Bytecount
|
|
|
3089 no_conversion_convert (struct coding_stream *str,
|
|
|
3090 const UExtbyte *src,
|
|
|
3091 unsigned_char_dynarr *dst, Bytecount n)
|
|
|
3092 {
|
|
|
3093 UExtbyte c;
|
|
|
3094 unsigned int ch = str->ch;
|
|
|
3095 Bytecount orign = n;
|
|
|
3096
|
|
|
3097 if (str->direction == CODING_DECODE)
|
|
|
3098 {
|
|
|
3099 while (n--)
|
|
|
3100 {
|
|
|
3101 c = *src++;
|
|
|
3102
|
|
|
3103 DECODE_ADD_BINARY_CHAR (c, dst);
|
|
|
3104 }
|
|
|
3105
|
|
|
3106 if (str->eof)
|
|
|
3107 DECODE_OUTPUT_PARTIAL_CHAR (ch, dst);
|
|
|
3108 }
|
|
|
3109 else
|
|
|
3110 {
|
|
|
3111
|
|
|
3112 while (n--)
|
|
|
3113 {
|
|
|
3114 c = *src++;
|
|
826
|
3115 if (byte_ascii_p (c))
|
|
771
|
3116 {
|
|
|
3117 assert (ch == 0);
|
|
|
3118 Dynarr_add (dst, c);
|
|
|
3119 }
|
|
|
3120 #ifdef MULE
|
|
867
|
3121 else if (ibyte_leading_byte_p (c))
|
|
771
|
3122 {
|
|
|
3123 assert (ch == 0);
|
|
|
3124 if (c == LEADING_BYTE_LATIN_ISO8859_1 ||
|
|
|
3125 c == LEADING_BYTE_CONTROL_1)
|
|
|
3126 ch = c;
|
|
|
3127 else
|
|
2297
|
3128 /* #### This is just plain unacceptable. */
|
|
771
|
3129 Dynarr_add (dst, '~'); /* untranslatable character */
|
|
|
3130 }
|
|
|
3131 else
|
|
|
3132 {
|
|
|
3133 if (ch == LEADING_BYTE_LATIN_ISO8859_1)
|
|
|
3134 Dynarr_add (dst, c);
|
|
|
3135 else if (ch == LEADING_BYTE_CONTROL_1)
|
|
|
3136 {
|
|
|
3137 assert (c < 0xC0);
|
|
|
3138 Dynarr_add (dst, c - 0x20);
|
|
|
3139 }
|
|
|
3140 /* else it should be the second or third byte of an
|
|
|
3141 untranslatable character, so ignore it */
|
|
|
3142 ch = 0;
|
|
|
3143 }
|
|
|
3144 #endif /* MULE */
|
|
|
3145
|
|
|
3146 }
|
|
|
3147 }
|
|
|
3148
|
|
|
3149 str->ch = ch;
|
|
|
3150 return orign;
|
|
|
3151 }
|
|
|
3152
|
|
|
3153 DEFINE_DETECTOR (no_conversion);
|
|
|
3154 DEFINE_DETECTOR_CATEGORY (no_conversion, no_conversion);
|
|
|
3155
|
|
|
3156 struct no_conversion_detector
|
|
|
3157 {
|
|
|
3158 int dummy;
|
|
|
3159 };
|
|
|
3160
|
|
|
3161 static void
|
|
2286
|
3162 no_conversion_detect (struct detection_state *st, const UExtbyte *UNUSED (src),
|
|
|
3163 Bytecount UNUSED (n))
|
|
771
|
3164 {
|
|
|
3165 /* Hack until we get better handling of this stuff! */
|
|
|
3166 DET_RESULT (st, no_conversion) = DET_SLIGHTLY_LIKELY;
|
|
|
3167 }
|
|
|
3168
|
|
|
3169
|
|
|
3170 /************************************************************************/
|
|
|
3171 /* Convert-eol methods */
|
|
|
3172 /************************************************************************/
|
|
|
3173
|
|
|
3174 /* This is used to handle end-of-line (EOL) differences. It is
|
|
|
3175 character-to-character, and works (when encoding) *BEFORE* sending
|
|
|
3176 data to the main encoding routine -- thus, that routine must handle
|
|
|
3177 different EOL types itself if it does line-oriented type processing.
|
|
|
3178 This is unavoidable because we don't know whether the output of the
|
|
|
3179 main encoding routine is ASCII compatible (Unicode is definitely not,
|
|
2297
|
3180 for example). [[ sjt sez this is bogus. There should be _no_ EOL
|
|
|
3181 processing (or processing of any kind) after conversion to external. ]]
|
|
771
|
3182
|
|
793
|
3183 There is one parameter: `subtype', either `cr', `lf', `crlf', or nil.
|
|
771
|
3184 */
|
|
|
3185
|
|
|
3186 struct convert_eol_coding_system
|
|
|
3187 {
|
|
|
3188 enum eol_type subtype;
|
|
2132
|
3189 int dummy; /* On some architectures (eg ia64) the portable dumper can
|
|
|
3190 produce unaligned access errors without this field. Probably
|
|
|
3191 because the combined structure of this structure and
|
|
|
3192 Lisp_Coding_System is not properly aligned. */
|
|
771
|
3193 };
|
|
|
3194
|
|
|
3195 #define CODING_SYSTEM_CONVERT_EOL_SUBTYPE(codesys) \
|
|
|
3196 (CODING_SYSTEM_TYPE_DATA (codesys, convert_eol)->subtype)
|
|
|
3197 #define XCODING_SYSTEM_CONVERT_EOL_SUBTYPE(codesys) \
|
|
|
3198 (XCODING_SYSTEM_TYPE_DATA (codesys, convert_eol)->subtype)
|
|
|
3199
|
|
|
3200 struct convert_eol_coding_stream
|
|
|
3201 {
|
|
|
3202 enum eol_type actual;
|
|
|
3203 };
|
|
|
3204
|
|
1204
|
3205 static const struct memory_description
|
|
771
|
3206 convert_eol_coding_system_description[] = {
|
|
|
3207 { XD_END }
|
|
|
3208 };
|
|
|
3209
|
|
1204
|
3210 DEFINE_CODING_SYSTEM_TYPE_WITH_DATA (convert_eol);
|
|
|
3211
|
|
771
|
3212 static void
|
|
2286
|
3213 convert_eol_print (Lisp_Object cs, Lisp_Object printcharfun,
|
|
|
3214 int UNUSED (escapeflag))
|
|
771
|
3215 {
|
|
|
3216 struct convert_eol_coding_system *data =
|
|
|
3217 XCODING_SYSTEM_TYPE_DATA (cs, convert_eol);
|
|
|
3218
|
|
|
3219 write_fmt_string (printcharfun, "(%s)",
|
|
|
3220 data->subtype == EOL_LF ? "lf" :
|
|
|
3221 data->subtype == EOL_CRLF ? "crlf" :
|
|
|
3222 data->subtype == EOL_CR ? "cr" :
|
|
793
|
3223 data->subtype == EOL_AUTODETECT ? "nil" :
|
|
771
|
3224 (abort(), ""));
|
|
|
3225 }
|
|
|
3226
|
|
|
3227 static enum source_sink_type
|
|
2286
|
3228 convert_eol_conversion_end_type (Lisp_Object UNUSED (codesys))
|
|
771
|
3229 {
|
|
|
3230 return DECODES_CHARACTER_TO_CHARACTER;
|
|
|
3231 }
|
|
|
3232
|
|
|
3233 static int
|
|
|
3234 convert_eol_putprop (Lisp_Object codesys,
|
|
|
3235 Lisp_Object key,
|
|
|
3236 Lisp_Object value)
|
|
|
3237 {
|
|
|
3238 struct convert_eol_coding_system *data =
|
|
|
3239 XCODING_SYSTEM_TYPE_DATA (codesys, convert_eol);
|
|
|
3240
|
|
|
3241 if (EQ (key, Qsubtype))
|
|
|
3242 {
|
|
|
3243 if (EQ (value, Qlf) /* || EQ (value, Qunix) */)
|
|
|
3244 data->subtype = EOL_LF;
|
|
|
3245 else if (EQ (value, Qcrlf) /* || EQ (value, Qdos) */)
|
|
|
3246 data->subtype = EOL_CRLF;
|
|
|
3247 else if (EQ (value, Qcr) /* || EQ (value, Qmac) */)
|
|
|
3248 data->subtype = EOL_CR;
|
|
793
|
3249 else if (EQ (value, Qnil))
|
|
771
|
3250 data->subtype = EOL_AUTODETECT;
|
|
|
3251 else invalid_constant ("Unrecognized eol type", value);
|
|
|
3252 }
|
|
|
3253 else
|
|
|
3254 return 0;
|
|
|
3255 return 1;
|
|
|
3256 }
|
|
|
3257
|
|
|
3258 static Lisp_Object
|
|
|
3259 convert_eol_getprop (Lisp_Object coding_system, Lisp_Object prop)
|
|
|
3260 {
|
|
|
3261 struct convert_eol_coding_system *data =
|
|
|
3262 XCODING_SYSTEM_TYPE_DATA (coding_system, convert_eol);
|
|
|
3263
|
|
|
3264 if (EQ (prop, Qsubtype))
|
|
|
3265 {
|
|
|
3266 switch (data->subtype)
|
|
|
3267 {
|
|
|
3268 case EOL_LF: return Qlf;
|
|
|
3269 case EOL_CRLF: return Qcrlf;
|
|
|
3270 case EOL_CR: return Qcr;
|
|
793
|
3271 case EOL_AUTODETECT: return Qnil;
|
|
771
|
3272 default: abort ();
|
|
|
3273 }
|
|
|
3274 }
|
|
|
3275
|
|
|
3276 return Qunbound;
|
|
|
3277 }
|
|
|
3278
|
|
|
3279 static void
|
|
|
3280 convert_eol_init_coding_stream (struct coding_stream *str)
|
|
|
3281 {
|
|
|
3282 struct convert_eol_coding_stream *data =
|
|
|
3283 CODING_STREAM_TYPE_DATA (str, convert_eol);
|
|
|
3284 data->actual = XCODING_SYSTEM_CONVERT_EOL_SUBTYPE (str->codesys);
|
|
|
3285 }
|
|
|
3286
|
|
|
3287 static Bytecount
|
|
867
|
3288 convert_eol_convert (struct coding_stream *str, const Ibyte *src,
|
|
771
|
3289 unsigned_char_dynarr *dst, Bytecount n)
|
|
|
3290 {
|
|
|
3291 if (str->direction == CODING_DECODE)
|
|
|
3292 {
|
|
|
3293 struct convert_eol_coding_stream *data =
|
|
|
3294 CODING_STREAM_TYPE_DATA (str, convert_eol);
|
|
|
3295
|
|
|
3296 if (data->actual == EOL_AUTODETECT)
|
|
|
3297 {
|
|
|
3298 Bytecount n2 = n;
|
|
867
|
3299 const Ibyte *src2 = src;
|
|
771
|
3300
|
|
|
3301 for (; n2; n2--)
|
|
|
3302 {
|
|
867
|
3303 Ibyte c = *src2++;
|
|
771
|
3304 if (c == '\n')
|
|
|
3305 {
|
|
|
3306 data->actual = EOL_LF;
|
|
|
3307 break;
|
|
|
3308 }
|
|
|
3309 else if (c == '\r')
|
|
|
3310 {
|
|
|
3311 if (n2 == 1)
|
|
|
3312 {
|
|
|
3313 /* If we're seeing a '\r' at the end of the data, then
|
|
|
3314 reject the '\r' right now so it doesn't become an
|
|
|
3315 issue in the code below -- unless we're at the end of
|
|
|
3316 the stream, in which case we can't do that (because
|
|
|
3317 then the '\r' will never get written out), and in any
|
|
|
3318 case we should be recognizing it at EOL_CR format. */
|
|
|
3319 if (str->eof)
|
|
|
3320 data->actual = EOL_CR;
|
|
|
3321 else
|
|
|
3322 n--;
|
|
|
3323 break;
|
|
|
3324 }
|
|
|
3325 else if (*src2 == '\n')
|
|
|
3326 data->actual = EOL_CRLF;
|
|
|
3327 else
|
|
|
3328 data->actual = EOL_CR;
|
|
|
3329 break;
|
|
|
3330 }
|
|
|
3331 }
|
|
|
3332 }
|
|
|
3333
|
|
|
3334 /* str->eof is set, the caller reached EOF on the other end and has
|
|
|
3335 no new data to give us. The only data we get is the data we
|
|
|
3336 rejected from last time. */
|
|
|
3337 if (data->actual == EOL_LF || data->actual == EOL_AUTODETECT ||
|
|
|
3338 (str->eof))
|
|
|
3339 Dynarr_add_many (dst, src, n);
|
|
|
3340 else
|
|
|
3341 {
|
|
867
|
3342 const Ibyte *end = src + n;
|
|
771
|
3343 while (1)
|
|
|
3344 {
|
|
|
3345 /* Find the next section with no \r and add it. */
|
|
867
|
3346 const Ibyte *runstart = src;
|
|
|
3347 src = (Ibyte *) memchr (src, '\r', end - src);
|
|
771
|
3348 if (!src)
|
|
|
3349 src = end;
|
|
|
3350 Dynarr_add_many (dst, runstart, src - runstart);
|
|
|
3351 /* Stop if at end ... */
|
|
|
3352 if (src == end)
|
|
|
3353 break;
|
|
|
3354 /* ... else, translate as necessary. */
|
|
|
3355 src++;
|
|
|
3356 if (data->actual == EOL_CR)
|
|
|
3357 Dynarr_add (dst, '\n');
|
|
|
3358 /* We need to be careful here with CRLF. If we see a CR at the
|
|
|
3359 end of the data, we don't know if it's part of a CRLF, so we
|
|
|
3360 reject it. Otherwise: If it's part of a CRLF, eat it and
|
|
|
3361 loop; the following LF gets added next time around. If it's
|
|
|
3362 not part of a CRLF, add the CR and loop. The following
|
|
|
3363 character will be processed in the next loop iteration. This
|
|
|
3364 correctly handles a sequence like CR+CR+LF. */
|
|
|
3365 else if (src == end)
|
|
|
3366 return n - 1; /* reject the CR at the end; we'll get it again
|
|
|
3367 next time the convert method is called */
|
|
|
3368 else if (*src != '\n')
|
|
|
3369 Dynarr_add (dst, '\r');
|
|
|
3370 }
|
|
|
3371 }
|
|
|
3372
|
|
|
3373 return n;
|
|
|
3374 }
|
|
|
3375 else
|
|
|
3376 {
|
|
|
3377 enum eol_type subtype =
|
|
|
3378 XCODING_SYSTEM_CONVERT_EOL_SUBTYPE (str->codesys);
|
|
867
|
3379 const Ibyte *end = src + n;
|
|
771
|
3380
|
|
|
3381 /* We try to be relatively efficient here. */
|
|
|
3382 if (subtype == EOL_LF)
|
|
|
3383 Dynarr_add_many (dst, src, n);
|
|
|
3384 else
|
|
|
3385 {
|
|
|
3386 while (1)
|
|
|
3387 {
|
|
|
3388 /* Find the next section with no \n and add it. */
|
|
867
|
3389 const Ibyte *runstart = src;
|
|
|
3390 src = (Ibyte *) memchr (src, '\n', end - src);
|
|
771
|
3391 if (!src)
|
|
|
3392 src = end;
|
|
|
3393 Dynarr_add_many (dst, runstart, src - runstart);
|
|
|
3394 /* Stop if at end ... */
|
|
|
3395 if (src == end)
|
|
|
3396 break;
|
|
|
3397 /* ... else, skip over \n and add its translation. */
|
|
|
3398 src++;
|
|
|
3399 Dynarr_add (dst, '\r');
|
|
|
3400 if (subtype == EOL_CRLF)
|
|
|
3401 Dynarr_add (dst, '\n');
|
|
|
3402 }
|
|
|
3403 }
|
|
|
3404
|
|
|
3405 return n;
|
|
|
3406 }
|
|
|
3407 }
|
|
|
3408
|
|
|
3409 static Lisp_Object
|
|
|
3410 convert_eol_canonicalize_after_coding (struct coding_stream *str)
|
|
|
3411 {
|
|
|
3412 struct convert_eol_coding_stream *data =
|
|
|
3413 CODING_STREAM_TYPE_DATA (str, convert_eol);
|
|
|
3414
|
|
|
3415 if (str->direction == CODING_ENCODE)
|
|
|
3416 return str->codesys;
|
|
|
3417
|
|
|
3418 switch (data->actual)
|
|
|
3419 {
|
|
|
3420 case EOL_LF: return Fget_coding_system (Qconvert_eol_lf);
|
|
|
3421 case EOL_CRLF: return Fget_coding_system (Qconvert_eol_crlf);
|
|
|
3422 case EOL_CR: return Fget_coding_system (Qconvert_eol_cr);
|
|
|
3423 case EOL_AUTODETECT: return str->codesys;
|
|
|
3424 default: abort (); return Qnil;
|
|
|
3425 }
|
|
|
3426 }
|
|
|
3427
|
|
|
3428
|
|
|
3429 /************************************************************************/
|
|
|
3430 /* Undecided methods */
|
|
|
3431 /************************************************************************/
|
|
|
3432
|
|
|
3433 /* Do autodetection. We can autodetect the EOL type only, the coding
|
|
|
3434 system only, or both. We only do autodetection when decoding; when
|
|
|
3435 encoding, we just pass the data through.
|
|
|
3436
|
|
|
3437 When doing just EOL detection, a coding system can be specified; if so,
|
|
|
3438 we will decode this data through the coding system before doing EOL
|
|
|
3439 detection. The reason for specifying this is so that
|
|
|
3440 canonicalize-after-coding works: We will canonicalize the specified
|
|
|
3441 coding system into the appropriate EOL type. When doing both coding and
|
|
|
3442 EOL detection, we do similar canonicalization, and also catch situations
|
|
|
3443 where the EOL type is overspecified, i.e. the detected coding system
|
|
|
3444 specifies an EOL type, and either switch to the equivalent
|
|
|
3445 non-EOL-processing coding system (if possible), or terminate EOL
|
|
|
3446 detection and use the specified EOL type. This prevents data from being
|
|
|
3447 EOL-processed twice.
|
|
|
3448 */
|
|
|
3449
|
|
|
3450 struct undecided_coding_system
|
|
|
3451 {
|
|
|
3452 int do_eol, do_coding;
|
|
|
3453 Lisp_Object cs;
|
|
|
3454 };
|
|
|
3455
|
|
|
3456 struct undecided_coding_stream
|
|
|
3457 {
|
|
|
3458 Lisp_Object actual;
|
|
|
3459 /* Either 2 or 3 lstreams here; see undecided_convert */
|
|
|
3460 struct chain_coding_stream c;
|
|
|
3461
|
|
|
3462 struct detection_state *st;
|
|
|
3463 };
|
|
|
3464
|
|
1204
|
3465 static const struct memory_description undecided_coding_system_description[] = {
|
|
|
3466 { XD_LISP_OBJECT, offsetof (struct undecided_coding_system, cs) },
|
|
771
|
3467 { XD_END }
|
|
|
3468 };
|
|
|
3469
|
|
1204
|
3470 static const struct memory_description undecided_coding_stream_description_1 [] = {
|
|
|
3471 { XD_LISP_OBJECT, offsetof (struct undecided_coding_stream, actual) },
|
|
|
3472 { XD_STRUCT_ARRAY, offsetof (struct undecided_coding_stream, c),
|
|
|
3473 1, &chain_coding_stream_description },
|
|
|
3474 { XD_END }
|
|
|
3475 };
|
|
|
3476
|
|
|
3477 const struct sized_memory_description undecided_coding_stream_description = {
|
|
|
3478 sizeof (struct undecided_coding_stream), undecided_coding_stream_description_1
|
|
|
3479 };
|
|
|
3480
|
|
|
3481 DEFINE_CODING_SYSTEM_TYPE_WITH_DATA (undecided);
|
|
|
3482
|
|
771
|
3483 static void
|
|
|
3484 undecided_init (Lisp_Object codesys)
|
|
|
3485 {
|
|
|
3486 struct undecided_coding_system *data =
|
|
|
3487 XCODING_SYSTEM_TYPE_DATA (codesys, undecided);
|
|
|
3488
|
|
|
3489 data->cs = Qnil;
|
|
|
3490 }
|
|
|
3491
|
|
|
3492 static void
|
|
|
3493 undecided_mark (Lisp_Object codesys)
|
|
|
3494 {
|
|
|
3495 struct undecided_coding_system *data =
|
|
|
3496 XCODING_SYSTEM_TYPE_DATA (codesys, undecided);
|
|
|
3497
|
|
|
3498 mark_object (data->cs);
|
|
|
3499 }
|
|
|
3500
|
|
|
3501 static void
|
|
|
3502 undecided_print (Lisp_Object cs, Lisp_Object printcharfun, int escapeflag)
|
|
|
3503 {
|
|
|
3504 struct undecided_coding_system *data =
|
|
|
3505 XCODING_SYSTEM_TYPE_DATA (cs, undecided);
|
|
|
3506 int need_space = 0;
|
|
|
3507
|
|
826
|
3508 write_c_string (printcharfun, "(");
|
|
771
|
3509 if (data->do_eol)
|
|
|
3510 {
|
|
826
|
3511 write_c_string (printcharfun, "do-eol");
|
|
771
|
3512 need_space = 1;
|
|
|
3513 }
|
|
|
3514 if (data->do_coding)
|
|
|
3515 {
|
|
|
3516 if (need_space)
|
|
826
|
3517 write_c_string (printcharfun, " ");
|
|
|
3518 write_c_string (printcharfun, "do-coding");
|
|
771
|
3519 need_space = 1;
|
|
|
3520 }
|
|
|
3521 if (!NILP (data->cs))
|
|
|
3522 {
|
|
|
3523 if (need_space)
|
|
826
|
3524 write_c_string (printcharfun, " ");
|
|
|
3525 write_c_string (printcharfun, "coding-system=");
|
|
771
|
3526 print_coding_system_in_print_method (data->cs, printcharfun, escapeflag);
|
|
|
3527 }
|
|
826
|
3528 write_c_string (printcharfun, ")");
|
|
771
|
3529 }
|
|
|
3530
|
|
|
3531 static void
|
|
|
3532 undecided_mark_coding_stream (struct coding_stream *str)
|
|
|
3533 {
|
|
1204
|
3534 mark_object (CODING_STREAM_TYPE_DATA (str, undecided)->actual);
|
|
771
|
3535 chain_mark_coding_stream_1 (&CODING_STREAM_TYPE_DATA (str, undecided)->c);
|
|
|
3536 }
|
|
|
3537
|
|
|
3538 static int
|
|
|
3539 undecided_putprop (Lisp_Object codesys, Lisp_Object key, Lisp_Object value)
|
|
|
3540 {
|
|
|
3541 struct undecided_coding_system *data =
|
|
|
3542 XCODING_SYSTEM_TYPE_DATA (codesys, undecided);
|
|
|
3543
|
|
|
3544 if (EQ (key, Qdo_eol))
|
|
|
3545 data->do_eol = 1;
|
|
|
3546 else if (EQ (key, Qdo_coding))
|
|
|
3547 data->do_coding = 1;
|
|
|
3548 else if (EQ (key, Qcoding_system))
|
|
|
3549 data->cs = get_coding_system_for_text_file (value, 0);
|
|
|
3550 else
|
|
|
3551 return 0;
|
|
|
3552 return 1;
|
|
|
3553 }
|
|
|
3554
|
|
|
3555 static Lisp_Object
|
|
|
3556 undecided_getprop (Lisp_Object codesys, Lisp_Object prop)
|
|
|
3557 {
|
|
|
3558 struct undecided_coding_system *data =
|
|
|
3559 XCODING_SYSTEM_TYPE_DATA (codesys, undecided);
|
|
|
3560
|
|
|
3561 if (EQ (prop, Qdo_eol))
|
|
|
3562 return data->do_eol ? Qt : Qnil;
|
|
|
3563 if (EQ (prop, Qdo_coding))
|
|
|
3564 return data->do_coding ? Qt : Qnil;
|
|
|
3565 if (EQ (prop, Qcoding_system))
|
|
|
3566 return data->cs;
|
|
|
3567 return Qunbound;
|
|
|
3568 }
|
|
|
3569
|
|
|
3570 static struct detection_state *
|
|
|
3571 allocate_detection_state (void)
|
|
|
3572 {
|
|
|
3573 int i;
|
|
|
3574 Bytecount size = MAX_ALIGN_SIZE (sizeof (struct detection_state));
|
|
|
3575 struct detection_state *block;
|
|
|
3576
|
|
|
3577 for (i = 0; i < coding_detector_count; i++)
|
|
|
3578 size += MAX_ALIGN_SIZE (Dynarr_at (all_coding_detectors, i).data_size);
|
|
|
3579
|
|
|
3580 block = (struct detection_state *) xmalloc_and_zero (size);
|
|
|
3581
|
|
|
3582 size = MAX_ALIGN_SIZE (sizeof (struct detection_state));
|
|
|
3583 for (i = 0; i < coding_detector_count; i++)
|
|
|
3584 {
|
|
|
3585 block->data_offset[i] = size;
|
|
|
3586 size += MAX_ALIGN_SIZE (Dynarr_at (all_coding_detectors, i).data_size);
|
|
|
3587 }
|
|
|
3588
|
|
|
3589 return block;
|
|
|
3590 }
|
|
|
3591
|
|
|
3592 static void
|
|
|
3593 free_detection_state (struct detection_state *st)
|
|
|
3594 {
|
|
|
3595 int i;
|
|
|
3596
|
|
|
3597 for (i = 0; i < coding_detector_count; i++)
|
|
|
3598 {
|
|
|
3599 if (Dynarr_at (all_coding_detectors, i).finalize_detection_state_method)
|
|
|
3600 Dynarr_at (all_coding_detectors, i).finalize_detection_state_method
|
|
|
3601 (st);
|
|
|
3602 }
|
|
|
3603
|
|
1726
|
3604 xfree (st, struct detection_state *);
|
|
771
|
3605 }
|
|
|
3606
|
|
|
3607 static int
|
|
|
3608 coding_category_symbol_to_id (Lisp_Object symbol)
|
|
428
|
3609 {
|
|
|
3610 int i;
|
|
|
3611
|
|
|
3612 CHECK_SYMBOL (symbol);
|
|
771
|
3613 for (i = 0; i < coding_detector_count; i++)
|
|
|
3614 {
|
|
|
3615 detector_category_dynarr *cats =
|
|
|
3616 Dynarr_at (all_coding_detectors, i).cats;
|
|
|
3617 int j;
|
|
|
3618
|
|
|
3619 for (j = 0; j < Dynarr_length (cats); j++)
|
|
|
3620 if (EQ (Dynarr_at (cats, j).sym, symbol))
|
|
|
3621 return Dynarr_at (cats, j).id;
|
|
|
3622 }
|
|
|
3623
|
|
563
|
3624 invalid_constant ("Unrecognized coding category", symbol);
|
|
1204
|
3625 RETURN_NOT_REACHED (0);
|
|
428
|
3626 }
|
|
|
3627
|
|
771
|
3628 static Lisp_Object
|
|
|
3629 coding_category_id_to_symbol (int id)
|
|
428
|
3630 {
|
|
|
3631 int i;
|
|
771
|
3632
|
|
|
3633 for (i = 0; i < coding_detector_count; i++)
|
|
|
3634 {
|
|
|
3635 detector_category_dynarr *cats =
|
|
|
3636 Dynarr_at (all_coding_detectors, i).cats;
|
|
|
3637 int j;
|
|
|
3638
|
|
|
3639 for (j = 0; j < Dynarr_length (cats); j++)
|
|
|
3640 if (id == Dynarr_at (cats, j).id)
|
|
|
3641 return Dynarr_at (cats, j).sym;
|
|
|
3642 }
|
|
|
3643
|
|
|
3644 abort ();
|
|
|
3645 return Qnil; /* (usually) not reached */
|
|
428
|
3646 }
|
|
|
3647
|
|
771
|
3648 static Lisp_Object
|
|
|
3649 detection_result_number_to_symbol (enum detection_result result)
|
|
428
|
3650 {
|
|
1494
|
3651 /* let compiler warn if not all enumerators are handled */
|
|
|
3652 switch (result) {
|
|
|
3653 #define FROB(sym, num) case num: return (sym)
|
|
771
|
3654 FROB (Qnear_certainty, DET_NEAR_CERTAINTY);
|
|
|
3655 FROB (Qquite_probable, DET_QUITE_PROBABLE);
|
|
|
3656 FROB (Qsomewhat_likely, DET_SOMEWHAT_LIKELY);
|
|
1494
|
3657 FROB (Qslightly_likely, DET_SLIGHTLY_LIKELY);
|
|
771
|
3658 FROB (Qas_likely_as_unlikely, DET_AS_LIKELY_AS_UNLIKELY);
|
|
|
3659 FROB (Qsomewhat_unlikely, DET_SOMEWHAT_UNLIKELY);
|
|
|
3660 FROB (Qquite_improbable, DET_QUITE_IMPROBABLE);
|
|
|
3661 FROB (Qnearly_impossible, DET_NEARLY_IMPOSSIBLE);
|
|
|
3662 #undef FROB
|
|
1494
|
3663 }
|
|
771
|
3664
|
|
|
3665 abort ();
|
|
|
3666 return Qnil; /* (usually) not reached */
|
|
|
3667 }
|
|
|
3668
|
|
778
|
3669 #if 0 /* not used */
|
|
771
|
3670 static enum detection_result
|
|
|
3671 detection_result_symbol_to_number (Lisp_Object symbol)
|
|
|
3672 {
|
|
1494
|
3673 /* using switch here would be bad style, and doesn't help */
|
|
771
|
3674 #define FROB(sym, num) if (EQ (symbol, sym)) return (num)
|
|
|
3675 FROB (Qnear_certainty, DET_NEAR_CERTAINTY);
|
|
|
3676 FROB (Qquite_probable, DET_QUITE_PROBABLE);
|
|
|
3677 FROB (Qsomewhat_likely, DET_SOMEWHAT_LIKELY);
|
|
1494
|
3678 FROB (Qslightly_likely, DET_SLIGHTLY_LIKELY);
|
|
771
|
3679 FROB (Qas_likely_as_unlikely, DET_AS_LIKELY_AS_UNLIKELY);
|
|
|
3680 FROB (Qsomewhat_unlikely, DET_SOMEWHAT_UNLIKELY);
|
|
|
3681 FROB (Qquite_improbable, DET_QUITE_IMPROBABLE);
|
|
|
3682 FROB (Qnearly_impossible, DET_NEARLY_IMPOSSIBLE);
|
|
|
3683 #undef FROB
|
|
|
3684
|
|
|
3685 invalid_constant ("Unrecognized detection result", symbol);
|
|
|
3686 return ((enum detection_result) 0); /* not reached */
|
|
|
3687 }
|
|
778
|
3688 #endif /* 0 */
|
|
771
|
3689
|
|
|
3690 /* Set all detection results for a given detector to a specified value. */
|
|
|
3691 void
|
|
|
3692 set_detection_results (struct detection_state *st, int detector, int given)
|
|
|
3693 {
|
|
|
3694 detector_category_dynarr *cats =
|
|
|
3695 Dynarr_at (all_coding_detectors, detector).cats;
|
|
|
3696 int i;
|
|
|
3697
|
|
|
3698 for (i = 0; i < Dynarr_length (cats); i++)
|
|
|
3699 st->categories[Dynarr_at (cats, i).id] = given;
|
|
|
3700 }
|
|
428
|
3701
|
|
|
3702 static int
|
|
|
3703 acceptable_control_char_p (int c)
|
|
|
3704 {
|
|
|
3705 switch (c)
|
|
|
3706 {
|
|
|
3707 /* Allow and ignore control characters that you might
|
|
|
3708 reasonably see in a text file */
|
|
|
3709 case '\r':
|
|
|
3710 case '\n':
|
|
|
3711 case '\t':
|
|
|
3712 case 7: /* bell */
|
|
|
3713 case 8: /* backspace */
|
|
|
3714 case 11: /* vertical tab */
|
|
|
3715 case 12: /* form feed */
|
|
|
3716 case 26: /* MS-DOS C-z junk */
|
|
|
3717 case 31: /* '^_' -- for info */
|
|
|
3718 return 1;
|
|
|
3719 default:
|
|
|
3720 return 0;
|
|
|
3721 }
|
|
|
3722 }
|
|
|
3723
|
|
771
|
3724 #ifdef DEBUG_XEMACS
|
|
|
3725
|
|
|
3726 static UExtbyte
|
|
|
3727 hex_digit_to_char (int digit)
|
|
428
|
3728 {
|
|
771
|
3729 if (digit < 10)
|
|
|
3730 return digit + '0';
|
|
|
3731 else
|
|
|
3732 return digit - 10 + 'A';
|
|
428
|
3733 }
|
|
|
3734
|
|
771
|
3735 static void
|
|
|
3736 output_bytes_in_ascii_and_hex (const UExtbyte *src, Bytecount n)
|
|
428
|
3737 {
|
|
771
|
3738 UExtbyte *ascii = alloca_array (UExtbyte, n + 1);
|
|
|
3739 UExtbyte *hex = alloca_array (UExtbyte, 3 * n + 1);
|
|
|
3740 int i;
|
|
|
3741
|
|
|
3742 for (i = 0; i < n; i++)
|
|
428
|
3743 {
|
|
771
|
3744 UExtbyte c = src[i];
|
|
|
3745 if (c < 0x20)
|
|
|
3746 ascii[i] = '.';
|
|
428
|
3747 else
|
|
771
|
3748 ascii[i] = c;
|
|
|
3749 hex[3 * i] = hex_digit_to_char (c >> 4);
|
|
|
3750 hex[3 * i + 1] = hex_digit_to_char (c & 0xF);
|
|
|
3751 hex[3 * i + 2] = ' ';
|
|
428
|
3752 }
|
|
771
|
3753 ascii[i] = '\0';
|
|
|
3754 hex[3 * i - 1] = '\0';
|
|
|
3755 stderr_out ("%s %s", ascii, hex);
|
|
428
|
3756 }
|
|
|
3757
|
|
771
|
3758 #endif /* DEBUG_XEMACS */
|
|
|
3759
|
|
|
3760 /* Attempt to determine the encoding of the given text. Before calling
|
|
|
3761 this function for the first time, you must zero out the detection state.
|
|
428
|
3762
|
|
|
3763 Returns:
|
|
|
3764
|
|
771
|
3765 0 == keep going
|
|
|
3766 1 == stop
|
|
428
|
3767 */
|
|
|
3768
|
|
|
3769 static int
|
|
771
|
3770 detect_coding_type (struct detection_state *st, const UExtbyte *src,
|
|
|
3771 Bytecount n)
|
|
428
|
3772 {
|
|
771
|
3773 Bytecount n2 = n;
|
|
|
3774 const UExtbyte *src2 = src;
|
|
|
3775 int i;
|
|
|
3776
|
|
|
3777 #ifdef DEBUG_XEMACS
|
|
|
3778 if (!NILP (Vdebug_coding_detection))
|
|
|
3779 {
|
|
|
3780 int bytes = min (16, n);
|
|
|
3781 stderr_out ("detect_coding_type: processing %ld bytes\n", n);
|
|
|
3782 stderr_out ("First %d: ", bytes);
|
|
|
3783 output_bytes_in_ascii_and_hex (src, bytes);
|
|
|
3784 stderr_out ("\nLast %d: ", bytes);
|
|
|
3785 output_bytes_in_ascii_and_hex (src + n - bytes, bytes);
|
|
|
3786 stderr_out ("\n");
|
|
|
3787 }
|
|
|
3788 #endif /* DEBUG_XEMACS */
|
|
428
|
3789 if (!st->seen_non_ascii)
|
|
|
3790 {
|
|
771
|
3791 for (; n2; n2--, src2++)
|
|
428
|
3792 {
|
|
771
|
3793 UExtbyte c = *src2;
|
|
428
|
3794 if ((c < 0x20 && !acceptable_control_char_p (c)) || c >= 0x80)
|
|
|
3795 {
|
|
|
3796 st->seen_non_ascii = 1;
|
|
|
3797 break;
|
|
|
3798 }
|
|
|
3799 }
|
|
|
3800 }
|
|
|
3801
|
|
771
|
3802 for (i = 0; i < coding_detector_count; i++)
|
|
|
3803 Dynarr_at (all_coding_detectors, i).detect_method (st, src, n);
|
|
|
3804
|
|
|
3805 st->bytes_seen += n;
|
|
|
3806
|
|
|
3807 #ifdef DEBUG_XEMACS
|
|
|
3808 if (!NILP (Vdebug_coding_detection))
|
|
|
3809 {
|
|
|
3810 stderr_out ("seen_non_ascii: %d\n", st->seen_non_ascii);
|
|
1494
|
3811 if (coding_detector_category_count <= 0)
|
|
|
3812 stderr_out ("found %d detector categories\n",
|
|
|
3813 coding_detector_category_count);
|
|
771
|
3814 for (i = 0; i < coding_detector_category_count; i++)
|
|
|
3815 stderr_out_lisp
|
|
|
3816 ("%s: %s\n",
|
|
|
3817 2,
|
|
|
3818 coding_category_id_to_symbol (i),
|
|
|
3819 detection_result_number_to_symbol ((enum detection_result)
|
|
|
3820 st->categories[i]));
|
|
|
3821 }
|
|
|
3822 #endif /* DEBUG_XEMACS */
|
|
|
3823
|
|
|
3824 {
|
|
|
3825 int not_unlikely = 0;
|
|
|
3826 int retval;
|
|
|
3827
|
|
|
3828 for (i = 0; i < coding_detector_category_count; i++)
|
|
|
3829 if (st->categories[i] >= 0)
|
|
|
3830 not_unlikely++;
|
|
|
3831
|
|
|
3832 retval = (not_unlikely <= 1
|
|
|
3833 #if 0 /* this is bogus */
|
|
|
3834 || st->bytes_seen >= MAX_BYTES_PROCESSED_FOR_DETECTION
|
|
428
|
3835 #endif
|
|
771
|
3836 );
|
|
|
3837
|
|
|
3838 #ifdef DEBUG_XEMACS
|
|
|
3839 if (!NILP (Vdebug_coding_detection))
|
|
|
3840 stderr_out ("detect_coding_type: returning %d (%s)\n",
|
|
|
3841 retval, retval ? "stop" : "keep going");
|
|
|
3842 #endif /* DEBUG_XEMACS */
|
|
|
3843
|
|
|
3844 return retval;
|
|
428
|
3845 }
|
|
|
3846 }
|
|
|
3847
|
|
|
3848 static Lisp_Object
|
|
771
|
3849 detected_coding_system (struct detection_state *st)
|
|
428
|
3850 {
|
|
771
|
3851 int i;
|
|
|
3852 int even = 1;
|
|
|
3853
|
|
|
3854 if (st->seen_non_ascii)
|
|
|
3855 {
|
|
|
3856 for (i = 0; i < coding_detector_category_count; i++)
|
|
|
3857 if (st->categories[i] != DET_AS_LIKELY_AS_UNLIKELY)
|
|
|
3858 {
|
|
|
3859 even = 0;
|
|
|
3860 break;
|
|
|
3861 }
|
|
|
3862 }
|
|
|
3863
|
|
|
3864 /* #### Here we are ignoring the results of detection when it's all
|
|
|
3865 ASCII. This is obviously a bad thing. But we need to fix up the
|
|
|
3866 existing detection methods somewhat before we can switch. */
|
|
|
3867 if (even)
|
|
428
|
3868 {
|
|
|
3869 /* If the file was entirely or basically ASCII, use the
|
|
|
3870 default value of `buffer-file-coding-system'. */
|
|
|
3871 Lisp_Object retval =
|
|
|
3872 XBUFFER (Vbuffer_defaults)->buffer_file_coding_system;
|
|
|
3873 if (!NILP (retval))
|
|
|
3874 {
|
|
771
|
3875 retval = find_coding_system_for_text_file (retval, 0);
|
|
428
|
3876 if (NILP (retval))
|
|
|
3877 {
|
|
|
3878 warn_when_safe
|
|
|
3879 (Qbad_variable, Qwarning,
|
|
|
3880 "Invalid `default-buffer-file-coding-system', set to nil");
|
|
|
3881 XBUFFER (Vbuffer_defaults)->buffer_file_coding_system = Qnil;
|
|
|
3882 }
|
|
|
3883 }
|
|
|
3884 if (NILP (retval))
|
|
|
3885 retval = Fget_coding_system (Qraw_text);
|
|
|
3886 return retval;
|
|
|
3887 }
|
|
|
3888 else
|
|
|
3889 {
|
|
771
|
3890 int likelihood;
|
|
|
3891 Lisp_Object retval = Qnil;
|
|
|
3892
|
|
|
3893 /* Look through the coding categories first by likelihood and then by
|
|
|
3894 priority and find the first one that is allowed. */
|
|
|
3895
|
|
|
3896 for (likelihood = DET_HIGHEST; likelihood >= DET_LOWEST; likelihood--)
|
|
428
|
3897 {
|
|
771
|
3898 for (i = 0; i < coding_detector_category_count; i++)
|
|
|
3899 {
|
|
|
3900 int cat = coding_category_by_priority[i];
|
|
|
3901 if (st->categories[cat] == likelihood &&
|
|
|
3902 !NILP (coding_category_system[cat]))
|
|
|
3903 {
|
|
|
3904 retval = (get_coding_system_for_text_file
|
|
|
3905 (coding_category_system[cat], 0));
|
|
|
3906 if (likelihood < DET_AS_LIKELY_AS_UNLIKELY)
|
|
|
3907 warn_when_safe_lispobj
|
|
|
3908 (intern ("detection"),
|
|
793
|
3909 Qwarning,
|
|
771
|
3910 emacs_sprintf_string_lisp
|
|
|
3911 (
|
|
|
3912 "Detected coding %s is unlikely to be correct (likelihood == `%s')",
|
|
|
3913 Qnil, 2, XCODING_SYSTEM_NAME (retval),
|
|
|
3914 detection_result_number_to_symbol
|
|
|
3915 ((enum detection_result) likelihood)));
|
|
|
3916 return retval;
|
|
|
3917 }
|
|
|
3918 }
|
|
428
|
3919 }
|
|
771
|
3920
|
|
|
3921 return Fget_coding_system (Qraw_text);
|
|
428
|
3922 }
|
|
|
3923 }
|
|
|
3924
|
|
1347
|
3925 /* Look for a coding system in the string (skipping over leading
|
|
|
3926 blanks). If found, return it, otherwise nil. */
|
|
|
3927
|
|
|
3928 static Lisp_Object
|
|
|
3929 snarf_coding_system (const Ibyte *p, Bytecount len)
|
|
|
3930 {
|
|
|
3931 Bytecount n;
|
|
|
3932 Ibyte *name;
|
|
|
3933
|
|
|
3934 while (*p == ' ' || *p == '\t') p++, len--;
|
|
|
3935 len = min (len, 1000);
|
|
|
3936 name = alloca_ibytes (len + 1);
|
|
|
3937 memcpy (name, p, len);
|
|
|
3938 name[len] = '\0';
|
|
|
3939
|
|
|
3940 /* Get coding system name */
|
|
|
3941 /* Characters valid in a MIME charset name (rfc 1521),
|
|
|
3942 and in a Lisp symbol name. */
|
|
|
3943 n = qxestrspn (name,
|
|
|
3944 "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
|
|
|
3945 "abcdefghijklmnopqrstuvwxyz"
|
|
|
3946 "0123456789"
|
|
|
3947 "!$%&*+-.^_{|}~");
|
|
|
3948 if (n > 0)
|
|
|
3949 {
|
|
|
3950 name[n] = '\0';
|
|
|
3951 return find_coding_system_for_text_file (intern_int (name), 0);
|
|
|
3952 }
|
|
|
3953
|
|
|
3954 return Qnil;
|
|
|
3955 }
|
|
|
3956
|
|
428
|
3957 /* Given a seekable read stream and potential coding system and EOL type
|
|
|
3958 as specified, do any autodetection that is called for. If the
|
|
|
3959 coding system and/or EOL type are not `autodetect', they will be left
|
|
|
3960 alone; but this function will never return an autodetect coding system
|
|
|
3961 or EOL type.
|
|
|
3962
|
|
|
3963 This function does not automatically fetch subsidiary coding systems;
|
|
|
3964 that should be unnecessary with the explicit eol-type argument. */
|
|
|
3965
|
|
|
3966 #define LENGTH(string_constant) (sizeof (string_constant) - 1)
|
|
|
3967
|
|
771
|
3968 static Lisp_Object
|
|
|
3969 unwind_free_detection_state (Lisp_Object opaque)
|
|
|
3970 {
|
|
|
3971 struct detection_state *st =
|
|
|
3972 (struct detection_state *) get_opaque_ptr (opaque);
|
|
|
3973 free_detection_state (st);
|
|
|
3974 free_opaque_ptr (opaque);
|
|
|
3975 return Qnil;
|
|
|
3976 }
|
|
|
3977
|
|
1347
|
3978 /* #### This duplicates code in `find-coding-system-magic-cookie-in-file'
|
|
|
3979 in files.el. Look into combining them. */
|
|
|
3980
|
|
771
|
3981 static Lisp_Object
|
|
|
3982 look_for_coding_system_magic_cookie (const UExtbyte *data, Bytecount len)
|
|
428
|
3983 {
|
|
771
|
3984 const UExtbyte *p;
|
|
|
3985 const UExtbyte *scan_end;
|
|
|
3986
|
|
|
3987 /* Look for initial "-*-"; mode line prefix */
|
|
|
3988 for (p = data,
|
|
|
3989 scan_end = data + len - LENGTH ("-*-coding:?-*-");
|
|
|
3990 p <= scan_end
|
|
|
3991 && *p != '\n'
|
|
|
3992 && *p != '\r';
|
|
|
3993 p++)
|
|
|
3994 if (*p == '-' && *(p+1) == '*' && *(p+2) == '-')
|
|
|
3995 {
|
|
|
3996 const UExtbyte *local_vars_beg = p + 3;
|
|
|
3997 /* Look for final "-*-"; mode line suffix */
|
|
|
3998 for (p = local_vars_beg,
|
|
|
3999 scan_end = data + len - LENGTH ("-*-");
|
|
|
4000 p <= scan_end
|
|
428
|
4001 && *p != '\n'
|
|
|
4002 && *p != '\r';
|
|
771
|
4003 p++)
|
|
|
4004 if (*p == '-' && *(p+1) == '*' && *(p+2) == '-')
|
|
|
4005 {
|
|
|
4006 const UExtbyte *suffix = p;
|
|
|
4007 /* Look for "coding:" */
|
|
|
4008 for (p = local_vars_beg,
|
|
|
4009 scan_end = suffix - LENGTH ("coding:?");
|
|
|
4010 p <= scan_end;
|
|
|
4011 p++)
|
|
|
4012 if (memcmp ("coding:", p, LENGTH ("coding:")) == 0
|
|
|
4013 && (p == local_vars_beg
|
|
|
4014 || (*(p-1) == ' ' ||
|
|
|
4015 *(p-1) == '\t' ||
|
|
|
4016 *(p-1) == ';')))
|
|
|
4017 {
|
|
|
4018 p += LENGTH ("coding:");
|
|
1347
|
4019 return snarf_coding_system (p, suffix - p);
|
|
771
|
4020 break;
|
|
|
4021 }
|
|
|
4022 break;
|
|
|
4023 }
|
|
|
4024 break;
|
|
|
4025 }
|
|
|
4026
|
|
1348
|
4027 #if 0
|
|
|
4028 /* #### Totally wrong as is, rewrite */
|
|
1347
|
4029 /* Look for initial ;;;###coding system */
|
|
|
4030
|
|
|
4031 {
|
|
|
4032 Bytecount ind = fast_string_match (QScoding_system_cookie,
|
|
|
4033 data, Qnil, 0, len, 0, ERROR_ME_NOT,
|
|
|
4034 1);
|
|
|
4035 if (ind >= 0)
|
|
|
4036 return
|
|
|
4037 snarf_coding_system (data + ind + LENGTH (";;;###coding system: "),
|
|
|
4038 len - ind - LENGTH (";;;###coding system: "));
|
|
|
4039 }
|
|
1348
|
4040 #endif /* 0 */
|
|
1347
|
4041
|
|
|
4042 return Qnil;
|
|
771
|
4043 }
|
|
|
4044
|
|
|
4045 static Lisp_Object
|
|
|
4046 determine_real_coding_system (Lstream *stream)
|
|
|
4047 {
|
|
|
4048 struct detection_state *st = allocate_detection_state ();
|
|
|
4049 int depth = record_unwind_protect (unwind_free_detection_state,
|
|
|
4050 make_opaque_ptr (st));
|
|
|
4051 UExtbyte buf[4096];
|
|
|
4052 Bytecount nread = Lstream_read (stream, buf, sizeof (buf));
|
|
|
4053 Lisp_Object coding_system = look_for_coding_system_magic_cookie (buf, nread);
|
|
|
4054
|
|
|
4055 if (NILP (coding_system))
|
|
|
4056 {
|
|
|
4057 while (1)
|
|
|
4058 {
|
|
|
4059 if (detect_coding_type (st, buf, nread))
|
|
428
|
4060 break;
|
|
771
|
4061 nread = Lstream_read (stream, buf, sizeof (buf));
|
|
|
4062 if (nread == 0)
|
|
|
4063 break;
|
|
428
|
4064 }
|
|
771
|
4065
|
|
|
4066 coding_system = detected_coding_system (st);
|
|
428
|
4067 }
|
|
|
4068
|
|
|
4069 Lstream_rewind (stream);
|
|
771
|
4070
|
|
|
4071 unbind_to (depth);
|
|
|
4072 return coding_system;
|
|
|
4073 }
|
|
|
4074
|
|
|
4075 static void
|
|
|
4076 undecided_init_coding_stream (struct coding_stream *str)
|
|
|
4077 {
|
|
|
4078 struct undecided_coding_stream *data =
|
|
|
4079 CODING_STREAM_TYPE_DATA (str, undecided);
|
|
|
4080 struct undecided_coding_system *csdata =
|
|
|
4081 XCODING_SYSTEM_TYPE_DATA (str->codesys, undecided);
|
|
|
4082
|
|
|
4083 data->actual = Qnil;
|
|
|
4084
|
|
|
4085 if (str->direction == CODING_DECODE)
|
|
|
4086 {
|
|
|
4087 Lstream *lst = str->other_end;
|
|
|
4088
|
|
|
4089 if ((lst->flags & LSTREAM_FL_READ) &&
|
|
|
4090 Lstream_seekable_p (lst) &&
|
|
|
4091 csdata->do_coding)
|
|
|
4092 /* We can determine the coding system now. */
|
|
|
4093 data->actual = determine_real_coding_system (lst);
|
|
|
4094 }
|
|
1494
|
4095
|
|
|
4096 #ifdef DEBUG_XEMACS
|
|
|
4097 if (!NILP (Vdebug_coding_detection))
|
|
|
4098 stderr_out_lisp ("detected coding system: %s\n", 1, data->actual);
|
|
|
4099 #endif /* DEBUG_XEMACS */
|
|
771
|
4100 }
|
|
|
4101
|
|
|
4102 static void
|
|
|
4103 undecided_rewind_coding_stream (struct coding_stream *str)
|
|
|
4104 {
|
|
|
4105 chain_rewind_coding_stream_1 (&CODING_STREAM_TYPE_DATA (str, undecided)->c);
|
|
|
4106 }
|
|
|
4107
|
|
|
4108 static void
|
|
|
4109 undecided_finalize_coding_stream (struct coding_stream *str)
|
|
|
4110 {
|
|
|
4111 struct undecided_coding_stream *data =
|
|
|
4112 CODING_STREAM_TYPE_DATA (str, undecided);
|
|
|
4113
|
|
|
4114 chain_finalize_coding_stream_1
|
|
|
4115 (&CODING_STREAM_TYPE_DATA (str, undecided)->c);
|
|
|
4116 if (data->st)
|
|
|
4117 free_detection_state (data->st);
|
|
|
4118 }
|
|
|
4119
|
|
|
4120 static Lisp_Object
|
|
|
4121 undecided_canonicalize (Lisp_Object codesys)
|
|
|
4122 {
|
|
|
4123 struct undecided_coding_system *csdata =
|
|
|
4124 XCODING_SYSTEM_TYPE_DATA (codesys, undecided);
|
|
|
4125 if (!csdata->do_eol && !csdata->do_coding)
|
|
|
4126 return NILP (csdata->cs) ? Fget_coding_system (Qbinary) : csdata->cs;
|
|
|
4127 if (csdata->do_eol && !csdata->do_coding && NILP (csdata->cs))
|
|
|
4128 return Fget_coding_system (Qconvert_eol_autodetect);
|
|
|
4129 return codesys;
|
|
|
4130 }
|
|
|
4131
|
|
|
4132 static Bytecount
|
|
|
4133 undecided_convert (struct coding_stream *str, const UExtbyte *src,
|
|
|
4134 unsigned_char_dynarr *dst, Bytecount n)
|
|
|
4135 {
|
|
|
4136 int first_time = 0;
|
|
|
4137
|
|
|
4138 if (str->direction == CODING_DECODE)
|
|
|
4139 {
|
|
|
4140 /* At this point, we have only the following possibilities:
|
|
|
4141
|
|
|
4142 do_eol && do_coding
|
|
|
4143 do_coding only
|
|
|
4144 do_eol only and a coding system was specified
|
|
|
4145
|
|
|
4146 Other possibilities are removed during undecided_canonicalize.
|
|
|
4147
|
|
|
4148 Therefore, our substreams are either
|
|
|
4149
|
|
|
4150 lstream_coding -> lstream_dynarr, or
|
|
|
4151 lstream_coding -> lstream_eol -> lstream_dynarr.
|
|
|
4152 */
|
|
|
4153 struct undecided_coding_system *csdata =
|
|
|
4154 XCODING_SYSTEM_TYPE_DATA (str->codesys, undecided);
|
|
|
4155 struct undecided_coding_stream *data =
|
|
|
4156 CODING_STREAM_TYPE_DATA (str, undecided);
|
|
|
4157
|
|
|
4158 if (str->eof)
|
|
|
4159 {
|
|
|
4160 /* Each will close the next. We need to close now because more
|
|
|
4161 data may be generated. */
|
|
|
4162 if (data->c.initted)
|
|
|
4163 Lstream_close (XLSTREAM (data->c.lstreams[0]));
|
|
|
4164 return n;
|
|
|
4165 }
|
|
|
4166
|
|
|
4167 if (!data->c.initted)
|
|
|
4168 {
|
|
|
4169 data->c.lstream_count = csdata->do_eol ? 3 : 2;
|
|
|
4170 data->c.lstreams = xnew_array (Lisp_Object, data->c.lstream_count);
|
|
|
4171
|
|
|
4172 data->c.lstreams[data->c.lstream_count - 1] =
|
|
|
4173 make_dynarr_output_stream (dst);
|
|
|
4174 Lstream_set_buffering
|
|
|
4175 (XLSTREAM (data->c.lstreams[data->c.lstream_count - 1]),
|
|
|
4176 LSTREAM_UNBUFFERED, 0);
|
|
|
4177 if (csdata->do_eol)
|
|
|
4178 {
|
|
|
4179 data->c.lstreams[1] =
|
|
|
4180 make_coding_output_stream
|
|
|
4181 (XLSTREAM (data->c.lstreams[data->c.lstream_count - 1]),
|
|
|
4182 Fget_coding_system (Qconvert_eol_autodetect),
|
|
800
|
4183 CODING_DECODE, 0);
|
|
771
|
4184 Lstream_set_buffering
|
|
|
4185 (XLSTREAM (data->c.lstreams[1]),
|
|
|
4186 LSTREAM_UNBUFFERED, 0);
|
|
|
4187 }
|
|
|
4188
|
|
|
4189 data->c.lstreams[0] =
|
|
|
4190 make_coding_output_stream
|
|
|
4191 (XLSTREAM (data->c.lstreams[1]),
|
|
|
4192 /* Substitute binary if we need to detect the encoding */
|
|
|
4193 csdata->do_coding ? Qbinary : csdata->cs,
|
|
800
|
4194 CODING_DECODE, 0);
|
|
771
|
4195 Lstream_set_buffering (XLSTREAM (data->c.lstreams[0]),
|
|
|
4196 LSTREAM_UNBUFFERED, 0);
|
|
|
4197
|
|
|
4198 first_time = 1;
|
|
|
4199 data->c.initted = 1;
|
|
|
4200 }
|
|
|
4201
|
|
|
4202 /* If necessary, do encoding-detection now. We do this when we're a
|
|
|
4203 writing stream or a non-seekable reading stream, meaning that we
|
|
|
4204 can't just process the whole input, rewind, and start over. */
|
|
|
4205
|
|
|
4206 if (csdata->do_coding)
|
|
|
4207 {
|
|
|
4208 int actual_was_nil = NILP (data->actual);
|
|
|
4209 if (NILP (data->actual))
|
|
|
4210 {
|
|
|
4211 if (!data->st)
|
|
|
4212 data->st = allocate_detection_state ();
|
|
|
4213 if (first_time)
|
|
|
4214 /* #### This is cheesy. What we really ought to do is buffer
|
|
|
4215 up a certain minimum amount of data to get a better result.
|
|
|
4216 */
|
|
|
4217 data->actual = look_for_coding_system_magic_cookie (src, n);
|
|
|
4218 if (NILP (data->actual))
|
|
|
4219 {
|
|
|
4220 /* #### This is cheesy. What we really ought to do is buffer
|
|
|
4221 up a certain minimum amount of data so as to get a less
|
|
|
4222 random result when doing subprocess detection. */
|
|
|
4223 detect_coding_type (data->st, src, n);
|
|
|
4224 data->actual = detected_coding_system (data->st);
|
|
|
4225 }
|
|
|
4226 }
|
|
|
4227 /* We need to set the detected coding system if we actually have
|
|
|
4228 such a coding system but didn't before. That is the case
|
|
|
4229 either when we just detected it in the previous code or when
|
|
|
4230 it was detected during undecided_init_coding_stream(). We
|
|
|
4231 can check for that using first_time. */
|
|
|
4232 if (!NILP (data->actual) && (actual_was_nil || first_time))
|
|
|
4233 {
|
|
|
4234 /* If the detected coding system doesn't allow for EOL
|
|
|
4235 autodetection, try to get the equivalent that does;
|
|
|
4236 otherwise, disable EOL detection (overriding whatever
|
|
|
4237 may already have been detected). */
|
|
|
4238 if (XCODING_SYSTEM_EOL_TYPE (data->actual) != EOL_AUTODETECT)
|
|
|
4239 {
|
|
|
4240 if (!NILP (XCODING_SYSTEM_SUBSIDIARY_PARENT (data->actual)))
|
|
|
4241 data->actual =
|
|
|
4242 XCODING_SYSTEM_SUBSIDIARY_PARENT (data->actual);
|
|
|
4243 else if (data->c.lstream_count == 3)
|
|
|
4244 set_coding_stream_coding_system
|
|
|
4245 (XLSTREAM (data->c.lstreams[1]),
|
|
|
4246 Fget_coding_system (Qidentity));
|
|
|
4247 }
|
|
|
4248 set_coding_stream_coding_system
|
|
|
4249 (XLSTREAM (data->c.lstreams[0]), data->actual);
|
|
|
4250 }
|
|
|
4251 }
|
|
|
4252
|
|
|
4253 if (Lstream_write (XLSTREAM (data->c.lstreams[0]), src, n) < 0)
|
|
|
4254 return -1;
|
|
|
4255 return n;
|
|
|
4256 }
|
|
|
4257 else
|
|
|
4258 return no_conversion_convert (str, src, dst, n);
|
|
|
4259 }
|
|
|
4260
|
|
|
4261 static Lisp_Object
|
|
|
4262 undecided_canonicalize_after_coding (struct coding_stream *str)
|
|
|
4263 {
|
|
|
4264 struct undecided_coding_stream *data =
|
|
|
4265 CODING_STREAM_TYPE_DATA (str, undecided);
|
|
|
4266 Lisp_Object ret, eolret;
|
|
|
4267
|
|
|
4268 if (str->direction == CODING_ENCODE)
|
|
|
4269 return str->codesys;
|
|
|
4270
|
|
|
4271 if (!data->c.initted)
|
|
|
4272 return Fget_coding_system (Qundecided);
|
|
|
4273
|
|
|
4274 ret = coding_stream_canonicalize_after_coding
|
|
|
4275 (XLSTREAM (data->c.lstreams[0]));
|
|
|
4276 if (NILP (ret))
|
|
|
4277 ret = Fget_coding_system (Qundecided);
|
|
|
4278 if (XCODING_SYSTEM_EOL_TYPE (ret) != EOL_AUTODETECT)
|
|
|
4279 return ret;
|
|
|
4280 eolret = coding_stream_canonicalize_after_coding
|
|
|
4281 (XLSTREAM (data->c.lstreams[1]));
|
|
|
4282 if (!EQ (XCODING_SYSTEM_TYPE (eolret), Qconvert_eol))
|
|
|
4283 return ret;
|
|
|
4284 return
|
|
|
4285 Fsubsidiary_coding_system (ret, Fcoding_system_property (eolret,
|
|
|
4286 Qsubtype));
|
|
|
4287 }
|
|
|
4288
|
|
|
4289
|
|
|
4290 /************************************************************************/
|
|
|
4291 /* Lisp interface: Coding category functions and detection */
|
|
|
4292 /************************************************************************/
|
|
|
4293
|
|
|
4294 DEFUN ("coding-category-list", Fcoding_category_list, 0, 0, 0, /*
|
|
|
4295 Return a list of all recognized coding categories.
|
|
|
4296 */
|
|
|
4297 ())
|
|
|
4298 {
|
|
|
4299 int i;
|
|
|
4300 Lisp_Object list = Qnil;
|
|
|
4301
|
|
|
4302 for (i = 0; i < coding_detector_count; i++)
|
|
|
4303 {
|
|
|
4304 detector_category_dynarr *cats =
|
|
|
4305 Dynarr_at (all_coding_detectors, i).cats;
|
|
|
4306 int j;
|
|
|
4307
|
|
|
4308 for (j = 0; j < Dynarr_length (cats); j++)
|
|
|
4309 list = Fcons (Dynarr_at (cats, j).sym, list);
|
|
|
4310 }
|
|
|
4311
|
|
|
4312 return Fnreverse (list);
|
|
|
4313 }
|
|
|
4314
|
|
|
4315 DEFUN ("set-coding-priority-list", Fset_coding_priority_list, 1, 1, 0, /*
|
|
|
4316 Change the priority order of the coding categories.
|
|
|
4317 LIST should be list of coding categories, in descending order of
|
|
|
4318 priority. Unspecified coding categories will be lower in priority
|
|
|
4319 than all specified ones, in the same relative order they were in
|
|
|
4320 previously.
|
|
|
4321 */
|
|
|
4322 (list))
|
|
|
4323 {
|
|
|
4324 int *category_to_priority =
|
|
|
4325 alloca_array (int, coding_detector_category_count);
|
|
|
4326 int i, j;
|
|
|
4327 Lisp_Object rest;
|
|
|
4328
|
|
|
4329 /* First generate a list that maps coding categories to priorities. */
|
|
|
4330
|
|
|
4331 for (i = 0; i < coding_detector_category_count; i++)
|
|
|
4332 category_to_priority[i] = -1;
|
|
|
4333
|
|
|
4334 /* Highest priority comes from the specified list. */
|
|
|
4335 i = 0;
|
|
|
4336 EXTERNAL_LIST_LOOP (rest, list)
|
|
|
4337 {
|
|
|
4338 int cat = coding_category_symbol_to_id (XCAR (rest));
|
|
|
4339
|
|
|
4340 if (category_to_priority[cat] >= 0)
|
|
|
4341 sferror ("Duplicate coding category in list", XCAR (rest));
|
|
|
4342 category_to_priority[cat] = i++;
|
|
|
4343 }
|
|
|
4344
|
|
|
4345 /* Now go through the existing categories by priority to retrieve
|
|
|
4346 the categories not yet specified and preserve their priority
|
|
|
4347 order. */
|
|
|
4348 for (j = 0; j < coding_detector_category_count; j++)
|
|
|
4349 {
|
|
|
4350 int cat = coding_category_by_priority[j];
|
|
|
4351 if (category_to_priority[cat] < 0)
|
|
|
4352 category_to_priority[cat] = i++;
|
|
|
4353 }
|
|
|
4354
|
|
|
4355 /* Now we need to construct the inverse of the mapping we just
|
|
|
4356 constructed. */
|
|
|
4357
|
|
|
4358 for (i = 0; i < coding_detector_category_count; i++)
|
|
|
4359 coding_category_by_priority[category_to_priority[i]] = i;
|
|
|
4360
|
|
|
4361 /* Phew! That was confusing. */
|
|
|
4362 return Qnil;
|
|
|
4363 }
|
|
|
4364
|
|
|
4365 DEFUN ("coding-priority-list", Fcoding_priority_list, 0, 0, 0, /*
|
|
|
4366 Return a list of coding categories in descending order of priority.
|
|
|
4367 */
|
|
|
4368 ())
|
|
|
4369 {
|
|
|
4370 int i;
|
|
|
4371 Lisp_Object list = Qnil;
|
|
|
4372
|
|
|
4373 for (i = 0; i < coding_detector_category_count; i++)
|
|
|
4374 list =
|
|
|
4375 Fcons (coding_category_id_to_symbol (coding_category_by_priority[i]),
|
|
|
4376 list);
|
|
|
4377 return Fnreverse (list);
|
|
|
4378 }
|
|
|
4379
|
|
|
4380 DEFUN ("set-coding-category-system", Fset_coding_category_system, 2, 2, 0, /*
|
|
|
4381 Change the coding system associated with a coding category.
|
|
|
4382 */
|
|
|
4383 (coding_category, coding_system))
|
|
|
4384 {
|
|
|
4385 coding_category_system[coding_category_symbol_to_id (coding_category)] =
|
|
|
4386 Fget_coding_system (coding_system);
|
|
|
4387 return Qnil;
|
|
|
4388 }
|
|
|
4389
|
|
|
4390 DEFUN ("coding-category-system", Fcoding_category_system, 1, 1, 0, /*
|
|
|
4391 Return the coding system associated with a coding category.
|
|
|
4392 */
|
|
|
4393 (coding_category))
|
|
|
4394 {
|
|
|
4395 Lisp_Object sys =
|
|
|
4396 coding_category_system[coding_category_symbol_to_id (coding_category)];
|
|
|
4397
|
|
|
4398 if (!NILP (sys))
|
|
|
4399 return XCODING_SYSTEM_NAME (sys);
|
|
|
4400 return Qnil;
|
|
|
4401 }
|
|
|
4402
|
|
800
|
4403 /* Detect the encoding of STREAM. Assumes stream is at the begnning and will
|
|
|
4404 read through to the end of STREAM, leaving it there but open. */
|
|
|
4405
|
|
771
|
4406 Lisp_Object
|
|
|
4407 detect_coding_stream (Lisp_Object stream)
|
|
|
4408 {
|
|
|
4409 Lisp_Object val = Qnil;
|
|
|
4410 struct gcpro gcpro1, gcpro2, gcpro3;
|
|
|
4411 UExtbyte random_buffer[65536];
|
|
|
4412 Lisp_Object binary_instream =
|
|
|
4413 make_coding_input_stream
|
|
|
4414 (XLSTREAM (stream), Qbinary,
|
|
814
|
4415 CODING_ENCODE, LSTREAM_FL_NO_CLOSE_OTHER);
|
|
771
|
4416 Lisp_Object decstream =
|
|
|
4417 make_coding_input_stream
|
|
|
4418 (XLSTREAM (binary_instream),
|
|
800
|
4419 Qundecided, CODING_DECODE, 0);
|
|
771
|
4420 Lstream *decstr = XLSTREAM (decstream);
|
|
|
4421
|
|
|
4422 GCPRO3 (decstream, stream, binary_instream);
|
|
|
4423 /* Read and discard all data; detection happens as a side effect of this,
|
|
|
4424 and we examine what was detected afterwards. */
|
|
|
4425 while (Lstream_read (decstr, random_buffer, sizeof (random_buffer)) > 0)
|
|
|
4426 ;
|
|
|
4427
|
|
|
4428 val = coding_stream_detected_coding_system (decstr);
|
|
|
4429 Lstream_close (decstr);
|
|
|
4430 Lstream_delete (decstr);
|
|
|
4431 Lstream_delete (XLSTREAM (binary_instream));
|
|
|
4432 UNGCPRO;
|
|
|
4433 return val;
|
|
428
|
4434 }
|
|
|
4435
|
|
|
4436 DEFUN ("detect-coding-region", Fdetect_coding_region, 2, 3, 0, /*
|
|
|
4437 Detect coding system of the text in the region between START and END.
|
|
444
|
4438 Return a list of possible coding systems ordered by priority.
|
|
|
4439 If only ASCII characters are found, return 'undecided or one of
|
|
428
|
4440 its subsidiary coding systems according to a detected end-of-line
|
|
|
4441 type. Optional arg BUFFER defaults to the current buffer.
|
|
|
4442 */
|
|
|
4443 (start, end, buffer))
|
|
|
4444 {
|
|
|
4445 Lisp_Object val = Qnil;
|
|
|
4446 struct buffer *buf = decode_buffer (buffer, 0);
|
|
665
|
4447 Charbpos b, e;
|
|
771
|
4448 Lisp_Object lb_instream;
|
|
428
|
4449
|
|
|
4450 get_buffer_range_char (buf, start, end, &b, &e, 0);
|
|
|
4451 lb_instream = make_lisp_buffer_input_stream (buf, b, e, 0);
|
|
771
|
4452
|
|
|
4453 val = detect_coding_stream (lb_instream);
|
|
|
4454 Lstream_delete (XLSTREAM (lb_instream));
|
|
428
|
4455 return val;
|
|
|
4456 }
|
|
|
4457
|
|
|
4458
|
|
771
|
4459
|
|
|
4460 #ifdef DEBUG_XEMACS
|
|
|
4461
|
|
428
|
4462 /************************************************************************/
|
|
771
|
4463 /* Internal methods */
|
|
|
4464 /************************************************************************/
|
|
|
4465
|
|
|
4466 /* Raw (internally-formatted) data. */
|
|
|
4467 DEFINE_CODING_SYSTEM_TYPE (internal);
|
|
428
|
4468
|
|
665
|
4469 static Bytecount
|
|
2286
|
4470 internal_convert (struct coding_stream *UNUSED (str), const UExtbyte *src,
|
|
771
|
4471 unsigned_char_dynarr *dst, Bytecount n)
|
|
|
4472 {
|
|
|
4473 Bytecount orign = n;
|
|
|
4474 Dynarr_add_many (dst, src, n);
|
|
|
4475 return orign;
|
|
|
4476 }
|
|
|
4477
|
|
|
4478 #endif /* DEBUG_XEMACS */
|
|
|
4479
|
|
|
4480
|
|
|
4481
|
|
|
4482 #ifdef HAVE_ZLIB
|
|
|
4483
|
|
|
4484 /************************************************************************/
|
|
|
4485 /* Gzip methods */
|
|
|
4486 /************************************************************************/
|
|
|
4487
|
|
|
4488 struct gzip_coding_system
|
|
428
|
4489 {
|
|
771
|
4490 int level; /* 0 through 9, or -1 for default */
|
|
|
4491 };
|
|
|
4492
|
|
|
4493 #define CODING_SYSTEM_GZIP_LEVEL(codesys) \
|
|
|
4494 (CODING_SYSTEM_TYPE_DATA (codesys, gzip)->level)
|
|
|
4495 #define XCODING_SYSTEM_GZIP_LEVEL(codesys) \
|
|
|
4496 (XCODING_SYSTEM_TYPE_DATA (codesys, gzip)->level)
|
|
|
4497
|
|
|
4498 struct gzip_coding_stream
|
|
428
|
4499 {
|
|
771
|
4500 z_stream stream;
|
|
|
4501 int stream_initted;
|
|
|
4502 int reached_eof; /* #### this should be handled by the caller, once we
|
|
|
4503 return LSTREAM_EOF */
|
|
|
4504 };
|
|
|
4505
|
|
1204
|
4506 static const struct memory_description
|
|
771
|
4507 gzip_coding_system_description[] = {
|
|
|
4508 { XD_END }
|
|
|
4509 };
|
|
|
4510
|
|
1204
|
4511 DEFINE_CODING_SYSTEM_TYPE_WITH_DATA (gzip);
|
|
|
4512
|
|
771
|
4513 enum source_sink_type
|
|
|
4514 gzip_conversion_end_type (Lisp_Object codesys)
|
|
|
4515 {
|
|
|
4516 return DECODES_BYTE_TO_BYTE;
|
|
428
|
4517 }
|
|
|
4518
|
|
|
4519 static void
|
|
771
|
4520 gzip_init (Lisp_Object codesys)
|
|
|
4521 {
|
|
|
4522 struct gzip_coding_system *data = XCODING_SYSTEM_TYPE_DATA (codesys, gzip);
|
|
|
4523 data->level = -1;
|
|
|
4524 }
|
|
|
4525
|
|
|
4526 static void
|
|
|
4527 gzip_print (Lisp_Object cs, Lisp_Object printcharfun, int escapeflag)
|
|
428
|
4528 {
|
|
771
|
4529 struct gzip_coding_system *data = XCODING_SYSTEM_TYPE_DATA (cs, gzip);
|
|
|
4530
|
|
826
|
4531 write_c_string (printcharfun, "(");
|
|
771
|
4532 if (data->level == -1)
|
|
826
|
4533 write_c_string (printcharfun, "default");
|
|
771
|
4534 else
|
|
|
4535 print_internal (make_int (data->level), printcharfun, 0);
|
|
826
|
4536 write_c_string (printcharfun, ")");
|
|
428
|
4537 }
|
|
|
4538
|
|
|
4539 static int
|
|
771
|
4540 gzip_putprop (Lisp_Object codesys, Lisp_Object key, Lisp_Object value)
|
|
428
|
4541 {
|
|
771
|
4542 struct gzip_coding_system *data = XCODING_SYSTEM_TYPE_DATA (codesys, gzip);
|
|
|
4543
|
|
|
4544 if (EQ (key, Qlevel))
|
|
428
|
4545 {
|
|
771
|
4546 if (EQ (value, Qdefault))
|
|
|
4547 data->level = -1;
|
|
|
4548 else
|
|
428
|
4549 {
|
|
771
|
4550 CHECK_INT (value);
|
|
|
4551 check_int_range (XINT (value), 0, 9);
|
|
|
4552 data->level = XINT (value);
|
|
428
|
4553 }
|
|
|
4554 }
|
|
|
4555 else
|
|
771
|
4556 return 0;
|
|
|
4557 return 1;
|
|
428
|
4558 }
|
|
|
4559
|
|
|
4560 static Lisp_Object
|
|
771
|
4561 gzip_getprop (Lisp_Object coding_system, Lisp_Object prop)
|
|
428
|
4562 {
|
|
771
|
4563 struct gzip_coding_system *data =
|
|
|
4564 XCODING_SYSTEM_TYPE_DATA (coding_system, gzip);
|
|
|
4565
|
|
|
4566 if (EQ (prop, Qlevel))
|
|
428
|
4567 {
|
|
771
|
4568 if (data->level == -1)
|
|
|
4569 return Qdefault;
|
|
|
4570 return make_int (data->level);
|
|
428
|
4571 }
|
|
771
|
4572
|
|
|
4573 return Qunbound;
|
|
428
|
4574 }
|
|
|
4575
|
|
|
4576 static void
|
|
771
|
4577 gzip_init_coding_stream (struct coding_stream *str)
|
|
428
|
4578 {
|
|
771
|
4579 struct gzip_coding_stream *data = CODING_STREAM_TYPE_DATA (str, gzip);
|
|
|
4580 if (data->stream_initted)
|
|
428
|
4581 {
|
|
771
|
4582 if (str->direction == CODING_DECODE)
|
|
|
4583 inflateEnd (&data->stream);
|
|
|
4584 else
|
|
|
4585 deflateEnd (&data->stream);
|
|
|
4586 data->stream_initted = 0;
|
|
428
|
4587 }
|
|
771
|
4588 data->reached_eof = 0;
|
|
428
|
4589 }
|
|
|
4590
|
|
|
4591 static void
|
|
771
|
4592 gzip_rewind_coding_stream (struct coding_stream *str)
|
|
428
|
4593 {
|
|
771
|
4594 gzip_init_coding_stream (str);
|
|
428
|
4595 }
|
|
|
4596
|
|
771
|
4597 static Bytecount
|
|
|
4598 gzip_convert (struct coding_stream *str,
|
|
|
4599 const UExtbyte *src,
|
|
|
4600 unsigned_char_dynarr *dst, Bytecount n)
|
|
428
|
4601 {
|
|
771
|
4602 struct gzip_coding_stream *data = CODING_STREAM_TYPE_DATA (str, gzip);
|
|
|
4603 int zerr;
|
|
|
4604 if (str->direction == CODING_DECODE)
|
|
428
|
4605 {
|
|
771
|
4606 if (data->reached_eof)
|
|
|
4607 return n; /* eat the data */
|
|
|
4608
|
|
|
4609 if (!data->stream_initted)
|
|
428
|
4610 {
|
|
771
|
4611 xzero (data->stream);
|
|
|
4612 if (inflateInit (&data->stream) != Z_OK)
|
|
|
4613 return LSTREAM_ERROR;
|
|
|
4614 data->stream_initted = 1;
|
|
428
|
4615 }
|
|
771
|
4616
|
|
|
4617 data->stream.next_in = (Bytef *) src;
|
|
|
4618 data->stream.avail_in = n;
|
|
|
4619
|
|
|
4620 /* Normally we stop when we've fed all data to the decompressor; but
|
|
|
4621 if we're at the end of the input, and the decompressor hasn't
|
|
|
4622 reported EOF, we need to keep going, as there might be more output
|
|
|
4623 to generate. Z_OK from the decompressor means input was processed
|
|
|
4624 or output was generated; if neither, we break out of the loop.
|
|
|
4625 Other return values are:
|
|
|
4626
|
|
|
4627 Z_STREAM_END EOF from decompressor
|
|
|
4628 Z_DATA_ERROR Corrupted data
|
|
|
4629 Z_BUF_ERROR No progress possible (this should happen if
|
|
|
4630 we try to feed it an incomplete file)
|
|
|
4631 Z_MEM_ERROR Out of memory
|
|
|
4632 Z_STREAM_ERROR (should never happen)
|
|
|
4633 Z_NEED_DICT (#### when will this happen?)
|
|
|
4634 */
|
|
|
4635 while (data->stream.avail_in > 0 || str->eof)
|
|
|
4636 {
|
|
|
4637 /* Reserve an output buffer of the same size as the input buffer;
|
|
|
4638 if that's not enough, we keep reserving the same size. */
|
|
|
4639 Bytecount reserved = n;
|
|
|
4640 Dynarr_add_many (dst, 0, reserved);
|
|
|
4641 /* Careful here! Don't retrieve the pointer until after
|
|
|
4642 reserving the space, or it might be bogus */
|
|
|
4643 data->stream.next_out =
|
|
|
4644 Dynarr_atp (dst, Dynarr_length (dst) - reserved);
|
|
|
4645 data->stream.avail_out = reserved;
|
|
|
4646 zerr = inflate (&data->stream, Z_NO_FLUSH);
|
|
|
4647 /* Lop off the unused portion */
|
|
|
4648 Dynarr_set_size (dst, Dynarr_length (dst) - data->stream.avail_out);
|
|
|
4649 if (zerr != Z_OK)
|
|
|
4650 break;
|
|
|
4651 }
|
|
|
4652
|
|
|
4653 if (zerr == Z_STREAM_END)
|
|
|
4654 data->reached_eof = 1;
|
|
|
4655
|
|
|
4656 if ((Bytecount) data->stream.avail_in < n)
|
|
|
4657 return n - data->stream.avail_in;
|
|
|
4658
|
|
|
4659 if (zerr == Z_OK || zerr == Z_STREAM_END)
|
|
|
4660 return 0;
|
|
|
4661
|
|
|
4662 return LSTREAM_ERROR;
|
|
428
|
4663 }
|
|
|
4664 else
|
|
|
4665 {
|
|
771
|
4666 if (!data->stream_initted)
|
|
|
4667 {
|
|
|
4668 int level = XCODING_SYSTEM_GZIP_LEVEL (str->codesys);
|
|
|
4669 xzero (data->stream);
|
|
|
4670 if (deflateInit (&data->stream,
|
|
|
4671 level == -1 ? Z_DEFAULT_COMPRESSION : level) !=
|
|
|
4672 Z_OK)
|
|
|
4673 return LSTREAM_ERROR;
|
|
|
4674 data->stream_initted = 1;
|
|
428
|
4675 }
|
|
771
|
4676
|
|
|
4677 data->stream.next_in = (Bytef *) src;
|
|
|
4678 data->stream.avail_in = n;
|
|
|
4679
|
|
|
4680 /* Normally we stop when we've fed all data to the compressor; but if
|
|
|
4681 we're at the end of the input, and the compressor hasn't reported
|
|
|
4682 EOF, we need to keep going, as there might be more output to
|
|
|
4683 generate. (To signal EOF on our end, we set the FLUSH parameter
|
|
|
4684 to Z_FINISH; when all data is output, Z_STREAM_END will be
|
|
|
4685 returned.) Z_OK from the compressor means input was processed or
|
|
|
4686 output was generated; if neither, we break out of the loop. Other
|
|
|
4687 return values are:
|
|
|
4688
|
|
|
4689 Z_STREAM_END EOF from compressor
|
|
|
4690 Z_BUF_ERROR No progress possible (should never happen)
|
|
|
4691 Z_STREAM_ERROR (should never happen)
|
|
|
4692 */
|
|
|
4693 while (data->stream.avail_in > 0 || str->eof)
|
|
|
4694 {
|
|
|
4695 /* Reserve an output buffer of the same size as the input buffer;
|
|
|
4696 if that's not enough, we keep reserving the same size. */
|
|
|
4697 Bytecount reserved = n;
|
|
|
4698 Dynarr_add_many (dst, 0, reserved);
|
|
|
4699 /* Careful here! Don't retrieve the pointer until after
|
|
|
4700 reserving the space, or it might be bogus */
|
|
|
4701 data->stream.next_out =
|
|
|
4702 Dynarr_atp (dst, Dynarr_length (dst) - reserved);
|
|
|
4703 data->stream.avail_out = reserved;
|
|
|
4704 zerr =
|
|
|
4705 deflate (&data->stream,
|
|
|
4706 str->eof ? Z_FINISH : Z_NO_FLUSH);
|
|
|
4707 /* Lop off the unused portion */
|
|
|
4708 Dynarr_set_size (dst, Dynarr_length (dst) - data->stream.avail_out);
|
|
|
4709 if (zerr != Z_OK)
|
|
|
4710 break;
|
|
|
4711 }
|
|
|
4712
|
|
|
4713 if ((Bytecount) data->stream.avail_in < n)
|
|
|
4714 return n - data->stream.avail_in;
|
|
|
4715
|
|
|
4716 if (zerr == Z_OK || zerr == Z_STREAM_END)
|
|
|
4717 return 0;
|
|
|
4718
|
|
|
4719 return LSTREAM_ERROR;
|
|
428
|
4720 }
|
|
|
4721 }
|
|
|
4722
|
|
771
|
4723 #endif /* HAVE_ZLIB */
|
|
428
|
4724
|
|
|
4725
|
|
|
4726 /************************************************************************/
|
|
|
4727 /* Initialization */
|
|
|
4728 /************************************************************************/
|
|
|
4729
|
|
|
4730 void
|
|
|
4731 syms_of_file_coding (void)
|
|
|
4732 {
|
|
442
|
4733 INIT_LRECORD_IMPLEMENTATION (coding_system);
|
|
|
4734
|
|
771
|
4735 DEFSUBR (Fvalid_coding_system_type_p);
|
|
|
4736 DEFSUBR (Fcoding_system_type_list);
|
|
428
|
4737 DEFSUBR (Fcoding_system_p);
|
|
|
4738 DEFSUBR (Ffind_coding_system);
|
|
|
4739 DEFSUBR (Fget_coding_system);
|
|
|
4740 DEFSUBR (Fcoding_system_list);
|
|
|
4741 DEFSUBR (Fcoding_system_name);
|
|
|
4742 DEFSUBR (Fmake_coding_system);
|
|
|
4743 DEFSUBR (Fcopy_coding_system);
|
|
440
|
4744 DEFSUBR (Fcoding_system_canonical_name_p);
|
|
|
4745 DEFSUBR (Fcoding_system_alias_p);
|
|
|
4746 DEFSUBR (Fcoding_system_aliasee);
|
|
428
|
4747 DEFSUBR (Fdefine_coding_system_alias);
|
|
|
4748 DEFSUBR (Fsubsidiary_coding_system);
|
|
771
|
4749 DEFSUBR (Fcoding_system_base);
|
|
|
4750 DEFSUBR (Fcoding_system_used_for_io);
|
|
428
|
4751
|
|
|
4752 DEFSUBR (Fcoding_system_type);
|
|
771
|
4753 DEFSUBR (Fcoding_system_description);
|
|
428
|
4754 DEFSUBR (Fcoding_system_property);
|
|
|
4755
|
|
|
4756 DEFSUBR (Fcoding_category_list);
|
|
|
4757 DEFSUBR (Fset_coding_priority_list);
|
|
|
4758 DEFSUBR (Fcoding_priority_list);
|
|
|
4759 DEFSUBR (Fset_coding_category_system);
|
|
|
4760 DEFSUBR (Fcoding_category_system);
|
|
|
4761
|
|
|
4762 DEFSUBR (Fdetect_coding_region);
|
|
|
4763 DEFSUBR (Fdecode_coding_region);
|
|
|
4764 DEFSUBR (Fencode_coding_region);
|
|
563
|
4765 DEFSYMBOL_MULTIWORD_PREDICATE (Qcoding_systemp);
|
|
|
4766 DEFSYMBOL (Qno_conversion);
|
|
771
|
4767 DEFSYMBOL (Qconvert_eol);
|
|
|
4768 DEFSYMBOL (Qconvert_eol_autodetect);
|
|
|
4769 DEFSYMBOL (Qconvert_eol_lf);
|
|
|
4770 DEFSYMBOL (Qconvert_eol_cr);
|
|
|
4771 DEFSYMBOL (Qconvert_eol_crlf);
|
|
563
|
4772 DEFSYMBOL (Qraw_text);
|
|
771
|
4773
|
|
563
|
4774 DEFSYMBOL (Qmnemonic);
|
|
|
4775 DEFSYMBOL (Qeol_type);
|
|
|
4776 DEFSYMBOL (Qpost_read_conversion);
|
|
|
4777 DEFSYMBOL (Qpre_write_conversion);
|
|
|
4778
|
|
771
|
4779 DEFSYMBOL (Qtranslation_table_for_decode);
|
|
|
4780 DEFSYMBOL (Qtranslation_table_for_encode);
|
|
|
4781 DEFSYMBOL (Qsafe_chars);
|
|
|
4782 DEFSYMBOL (Qsafe_charsets);
|
|
|
4783 DEFSYMBOL (Qmime_charset);
|
|
|
4784 DEFSYMBOL (Qvalid_codes);
|
|
|
4785
|
|
563
|
4786 DEFSYMBOL (Qcr);
|
|
|
4787 DEFSYMBOL (Qlf);
|
|
|
4788 DEFSYMBOL (Qcrlf);
|
|
|
4789 DEFSYMBOL (Qeol_cr);
|
|
|
4790 DEFSYMBOL (Qeol_lf);
|
|
|
4791 DEFSYMBOL (Qeol_crlf);
|
|
|
4792 DEFSYMBOL (Qencode);
|
|
|
4793 DEFSYMBOL (Qdecode);
|
|
428
|
4794
|
|
771
|
4795 DEFSYMBOL (Qnear_certainty);
|
|
|
4796 DEFSYMBOL (Qquite_probable);
|
|
|
4797 DEFSYMBOL (Qsomewhat_likely);
|
|
1494
|
4798 DEFSYMBOL (Qslightly_likely);
|
|
771
|
4799 DEFSYMBOL (Qas_likely_as_unlikely);
|
|
|
4800 DEFSYMBOL (Qsomewhat_unlikely);
|
|
|
4801 DEFSYMBOL (Qquite_improbable);
|
|
|
4802 DEFSYMBOL (Qnearly_impossible);
|
|
|
4803
|
|
|
4804 DEFSYMBOL (Qdo_eol);
|
|
|
4805 DEFSYMBOL (Qdo_coding);
|
|
|
4806
|
|
|
4807 DEFSYMBOL (Qcanonicalize_after_coding);
|
|
|
4808
|
|
|
4809 DEFSYMBOL (Qescape_quoted);
|
|
|
4810
|
|
|
4811 #ifdef HAVE_ZLIB
|
|
|
4812 DEFSYMBOL (Qgzip);
|
|
|
4813 #endif
|
|
|
4814
|
|
428
|
4815 }
|
|
|
4816
|
|
|
4817 void
|
|
|
4818 lstream_type_create_file_coding (void)
|
|
|
4819 {
|
|
771
|
4820 LSTREAM_HAS_METHOD (coding, reader);
|
|
|
4821 LSTREAM_HAS_METHOD (coding, writer);
|
|
|
4822 LSTREAM_HAS_METHOD (coding, rewinder);
|
|
|
4823 LSTREAM_HAS_METHOD (coding, seekable_p);
|
|
|
4824 LSTREAM_HAS_METHOD (coding, marker);
|
|
|
4825 LSTREAM_HAS_METHOD (coding, flusher);
|
|
|
4826 LSTREAM_HAS_METHOD (coding, closer);
|
|
|
4827 LSTREAM_HAS_METHOD (coding, finalizer);
|
|
|
4828 }
|
|
|
4829
|
|
|
4830 void
|
|
|
4831 coding_system_type_create (void)
|
|
|
4832 {
|
|
|
4833 int i;
|
|
|
4834
|
|
|
4835 staticpro (&Vcoding_system_hash_table);
|
|
|
4836 Vcoding_system_hash_table =
|
|
|
4837 make_lisp_hash_table (50, HASH_TABLE_NON_WEAK, HASH_TABLE_EQ);
|
|
|
4838
|
|
|
4839 the_coding_system_type_entry_dynarr = Dynarr_new (coding_system_type_entry);
|
|
|
4840 dump_add_root_struct_ptr (&the_coding_system_type_entry_dynarr,
|
|
|
4841 &csted_description);
|
|
|
4842
|
|
|
4843 Vcoding_system_type_list = Qnil;
|
|
|
4844 staticpro (&Vcoding_system_type_list);
|
|
|
4845
|
|
|
4846 /* Initialize to something reasonable ... */
|
|
|
4847 for (i = 0; i < MAX_DETECTOR_CATEGORIES; i++)
|
|
|
4848 {
|
|
|
4849 coding_category_system[i] = Qnil;
|
|
1204
|
4850 dump_add_root_lisp_object (&coding_category_system[i]);
|
|
771
|
4851 coding_category_by_priority[i] = i;
|
|
|
4852 }
|
|
|
4853
|
|
|
4854 dump_add_opaque (coding_category_by_priority,
|
|
|
4855 sizeof (coding_category_by_priority));
|
|
|
4856
|
|
|
4857 all_coding_detectors = Dynarr_new2 (detector_dynarr, struct detector);
|
|
|
4858 dump_add_root_struct_ptr (&all_coding_detectors,
|
|
|
4859 &detector_dynarr_description);
|
|
|
4860
|
|
|
4861 dump_add_opaque_int (&coding_system_tick);
|
|
|
4862 dump_add_opaque_int (&coding_detector_count);
|
|
|
4863 dump_add_opaque_int (&coding_detector_category_count);
|
|
|
4864
|
|
|
4865 INITIALIZE_CODING_SYSTEM_TYPE (no_conversion,
|
|
|
4866 "no-conversion-coding-system-p");
|
|
|
4867 CODING_SYSTEM_HAS_METHOD (no_conversion, convert);
|
|
|
4868
|
|
|
4869 INITIALIZE_DETECTOR (no_conversion);
|
|
|
4870 DETECTOR_HAS_METHOD (no_conversion, detect);
|
|
|
4871 INITIALIZE_DETECTOR_CATEGORY (no_conversion, no_conversion);
|
|
|
4872
|
|
|
4873 INITIALIZE_CODING_SYSTEM_TYPE_WITH_DATA (convert_eol,
|
|
|
4874 "convert-eol-coding-system-p");
|
|
|
4875 CODING_SYSTEM_HAS_METHOD (convert_eol, print);
|
|
|
4876 CODING_SYSTEM_HAS_METHOD (convert_eol, convert);
|
|
|
4877 CODING_SYSTEM_HAS_METHOD (convert_eol, getprop);
|
|
|
4878 CODING_SYSTEM_HAS_METHOD (convert_eol, putprop);
|
|
|
4879 CODING_SYSTEM_HAS_METHOD (convert_eol, conversion_end_type);
|
|
|
4880 CODING_SYSTEM_HAS_METHOD (convert_eol, canonicalize_after_coding);
|
|
|
4881 CODING_SYSTEM_HAS_METHOD (convert_eol, init_coding_stream);
|
|
|
4882
|
|
|
4883 INITIALIZE_CODING_SYSTEM_TYPE_WITH_DATA (undecided,
|
|
|
4884 "undecided-coding-system-p");
|
|
|
4885 CODING_SYSTEM_HAS_METHOD (undecided, init);
|
|
|
4886 CODING_SYSTEM_HAS_METHOD (undecided, mark);
|
|
|
4887 CODING_SYSTEM_HAS_METHOD (undecided, print);
|
|
|
4888 CODING_SYSTEM_HAS_METHOD (undecided, convert);
|
|
|
4889 CODING_SYSTEM_HAS_METHOD (undecided, putprop);
|
|
|
4890 CODING_SYSTEM_HAS_METHOD (undecided, getprop);
|
|
|
4891 CODING_SYSTEM_HAS_METHOD (undecided, init_coding_stream);
|
|
|
4892 CODING_SYSTEM_HAS_METHOD (undecided, rewind_coding_stream);
|
|
|
4893 CODING_SYSTEM_HAS_METHOD (undecided, finalize_coding_stream);
|
|
|
4894 CODING_SYSTEM_HAS_METHOD (undecided, mark_coding_stream);
|
|
|
4895 CODING_SYSTEM_HAS_METHOD (undecided, canonicalize);
|
|
|
4896 CODING_SYSTEM_HAS_METHOD (undecided, canonicalize_after_coding);
|
|
|
4897
|
|
|
4898 INITIALIZE_CODING_SYSTEM_TYPE_WITH_DATA (chain, "chain-coding-system-p");
|
|
|
4899
|
|
|
4900 CODING_SYSTEM_HAS_METHOD (chain, print);
|
|
|
4901 CODING_SYSTEM_HAS_METHOD (chain, canonicalize);
|
|
|
4902 CODING_SYSTEM_HAS_METHOD (chain, init);
|
|
|
4903 CODING_SYSTEM_HAS_METHOD (chain, mark);
|
|
|
4904 CODING_SYSTEM_HAS_METHOD (chain, mark_coding_stream);
|
|
|
4905 CODING_SYSTEM_HAS_METHOD (chain, convert);
|
|
|
4906 CODING_SYSTEM_HAS_METHOD (chain, rewind_coding_stream);
|
|
|
4907 CODING_SYSTEM_HAS_METHOD (chain, finalize_coding_stream);
|
|
|
4908 CODING_SYSTEM_HAS_METHOD (chain, finalize);
|
|
|
4909 CODING_SYSTEM_HAS_METHOD (chain, putprop);
|
|
|
4910 CODING_SYSTEM_HAS_METHOD (chain, getprop);
|
|
|
4911 CODING_SYSTEM_HAS_METHOD (chain, conversion_end_type);
|
|
|
4912 CODING_SYSTEM_HAS_METHOD (chain, canonicalize_after_coding);
|
|
|
4913
|
|
|
4914 #ifdef DEBUG_XEMACS
|
|
|
4915 INITIALIZE_CODING_SYSTEM_TYPE (internal, "internal-coding-system-p");
|
|
|
4916 CODING_SYSTEM_HAS_METHOD (internal, convert);
|
|
|
4917 #endif
|
|
|
4918
|
|
|
4919 #ifdef HAVE_ZLIB
|
|
|
4920 INITIALIZE_CODING_SYSTEM_TYPE_WITH_DATA (gzip, "gzip-coding-system-p");
|
|
|
4921 CODING_SYSTEM_HAS_METHOD (gzip, conversion_end_type);
|
|
|
4922 CODING_SYSTEM_HAS_METHOD (gzip, convert);
|
|
|
4923 CODING_SYSTEM_HAS_METHOD (gzip, init);
|
|
|
4924 CODING_SYSTEM_HAS_METHOD (gzip, print);
|
|
|
4925 CODING_SYSTEM_HAS_METHOD (gzip, init_coding_stream);
|
|
|
4926 CODING_SYSTEM_HAS_METHOD (gzip, rewind_coding_stream);
|
|
|
4927 CODING_SYSTEM_HAS_METHOD (gzip, putprop);
|
|
|
4928 CODING_SYSTEM_HAS_METHOD (gzip, getprop);
|
|
|
4929 #endif
|
|
|
4930 }
|
|
|
4931
|
|
|
4932 void
|
|
|
4933 reinit_coding_system_type_create (void)
|
|
|
4934 {
|
|
|
4935 REINITIALIZE_CODING_SYSTEM_TYPE (no_conversion);
|
|
|
4936 REINITIALIZE_CODING_SYSTEM_TYPE (convert_eol);
|
|
|
4937 REINITIALIZE_CODING_SYSTEM_TYPE (undecided);
|
|
|
4938 REINITIALIZE_CODING_SYSTEM_TYPE (chain);
|
|
|
4939 #if 0
|
|
|
4940 REINITIALIZE_CODING_SYSTEM_TYPE (text_file_wrapper);
|
|
|
4941 #endif /* 0 */
|
|
|
4942 #ifdef DEBUG_XEMACS
|
|
|
4943 REINITIALIZE_CODING_SYSTEM_TYPE (internal);
|
|
|
4944 #endif
|
|
|
4945 #ifdef HAVE_ZLIB
|
|
|
4946 REINITIALIZE_CODING_SYSTEM_TYPE (gzip);
|
|
|
4947 #endif
|
|
|
4948 }
|
|
|
4949
|
|
|
4950 void
|
|
|
4951 reinit_vars_of_file_coding (void)
|
|
|
4952 {
|
|
428
|
4953 }
|
|
|
4954
|
|
|
4955 void
|
|
|
4956 vars_of_file_coding (void)
|
|
|
4957 {
|
|
771
|
4958 reinit_vars_of_file_coding ();
|
|
|
4959
|
|
|
4960 /* We always have file-coding support */
|
|
428
|
4961 Fprovide (intern ("file-coding"));
|
|
|
4962
|
|
1347
|
4963 QScoding_system_cookie = build_string (";;;###coding system: ");
|
|
|
4964 staticpro (&QScoding_system_cookie);
|
|
|
4965
|
|
1242
|
4966 #ifdef HAVE_DEFAULT_EOL_DETECTION
|
|
2297
|
4967 /* #### Find a more appropriate place for this comment.
|
|
|
4968 WARNING: The existing categories are intimately tied to the function
|
|
1242
|
4969 `coding-system-category' in coding.el. If you change a category, or
|
|
|
4970 change the layout of any coding system associated with a category, you
|
|
|
4971 need to check that function and make sure it's written properly. */
|
|
|
4972
|
|
|
4973 Fprovide (intern ("unix-default-eol-detection"));
|
|
|
4974 #endif
|
|
|
4975
|
|
428
|
4976 DEFVAR_LISP ("keyboard-coding-system", &Vkeyboard_coding_system /*
|
|
|
4977 Coding system used for TTY keyboard input.
|
|
|
4978 Not used under a windowing system.
|
|
|
4979 */ );
|
|
|
4980 Vkeyboard_coding_system = Qnil;
|
|
|
4981
|
|
|
4982 DEFVAR_LISP ("terminal-coding-system", &Vterminal_coding_system /*
|
|
|
4983 Coding system used for TTY display output.
|
|
|
4984 Not used under a windowing system.
|
|
|
4985 */ );
|
|
|
4986 Vterminal_coding_system = Qnil;
|
|
|
4987
|
|
|
4988 DEFVAR_LISP ("coding-system-for-read", &Vcoding_system_for_read /*
|
|
440
|
4989 Overriding coding system used when reading from a file or process.
|
|
|
4990 You should bind this variable with `let', but do not set it globally.
|
|
|
4991 If this is non-nil, it specifies the coding system that will be used
|
|
|
4992 to decode input on read operations, such as from a file or process.
|
|
|
4993 It overrides `buffer-file-coding-system-for-read',
|
|
428
|
4994 `insert-file-contents-pre-hook', etc. Use those variables instead of
|
|
440
|
4995 this one for permanent changes to the environment. */ );
|
|
428
|
4996 Vcoding_system_for_read = Qnil;
|
|
|
4997
|
|
|
4998 DEFVAR_LISP ("coding-system-for-write",
|
|
|
4999 &Vcoding_system_for_write /*
|
|
440
|
5000 Overriding coding system used when writing to a file or process.
|
|
|
5001 You should bind this variable with `let', but do not set it globally.
|
|
|
5002 If this is non-nil, it specifies the coding system that will be used
|
|
|
5003 to encode output for write operations, such as to a file or process.
|
|
|
5004 It overrides `buffer-file-coding-system', `write-region-pre-hook', etc.
|
|
|
5005 Use those variables instead of this one for permanent changes to the
|
|
|
5006 environment. */ );
|
|
428
|
5007 Vcoding_system_for_write = Qnil;
|
|
|
5008
|
|
|
5009 DEFVAR_LISP ("file-name-coding-system", &Vfile_name_coding_system /*
|
|
|
5010 Coding system used to convert pathnames when accessing files.
|
|
|
5011 */ );
|
|
|
5012 Vfile_name_coding_system = Qnil;
|
|
|
5013
|
|
|
5014 DEFVAR_BOOL ("enable-multibyte-characters", &enable_multibyte_characters /*
|
|
771
|
5015 Setting this has no effect. It is purely for FSF compatibility.
|
|
428
|
5016 */ );
|
|
|
5017 enable_multibyte_characters = 1;
|
|
771
|
5018
|
|
|
5019 Vchain_canonicalize_hash_table =
|
|
|
5020 make_lisp_hash_table (50, HASH_TABLE_NON_WEAK, HASH_TABLE_EQUAL);
|
|
|
5021 staticpro (&Vchain_canonicalize_hash_table);
|
|
|
5022
|
|
|
5023 #ifdef DEBUG_XEMACS
|
|
|
5024 DEFVAR_LISP ("debug-coding-detection", &Vdebug_coding_detection /*
|
|
|
5025 If non-nil, display debug information about detection operations in progress.
|
|
|
5026 Information is displayed on stderr.
|
|
|
5027 */ );
|
|
|
5028 Vdebug_coding_detection = Qnil;
|
|
|
5029 #endif
|
|
428
|
5030 }
|
|
|
5031
|
|
2297
|
5032 /* #### reformat this for consistent appearance? */
|
|
|
5033
|
|
428
|
5034 void
|
|
|
5035 complex_vars_of_file_coding (void)
|
|
|
5036 {
|
|
771
|
5037 Fmake_coding_system
|
|
|
5038 (Qconvert_eol_cr, Qconvert_eol,
|
|
|
5039 build_msg_string ("Convert CR to LF"),
|
|
|
5040 nconc2 (list6 (Qdocumentation,
|
|
|
5041 build_msg_string (
|
|
|
5042 "Converts CR (used to mark the end of a line on Macintosh systems) to LF\n"
|
|
|
5043 "(used internally and under Unix to mark the end of a line)."),
|
|
|
5044 Qmnemonic, build_string ("CR->LF"),
|
|
|
5045 Qsubtype, Qcr),
|
|
|
5046 /* VERY IMPORTANT! Tell make-coding-system not to generate
|
|
|
5047 subsidiaries -- it needs the coding systems we're creating
|
|
|
5048 to do so! */
|
|
|
5049 list2 (Qeol_type, Qlf)));
|
|
|
5050
|
|
|
5051 Fmake_coding_system
|
|
|
5052 (Qconvert_eol_lf, Qconvert_eol,
|
|
|
5053 build_msg_string ("Convert LF to LF (do nothing)"),
|
|
|
5054 nconc2 (list6 (Qdocumentation,
|
|
|
5055 build_msg_string (
|
|
|
5056 "Do nothing."),
|
|
|
5057 Qmnemonic, build_string ("LF->LF"),
|
|
|
5058 Qsubtype, Qlf),
|
|
|
5059 /* VERY IMPORTANT! Tell make-coding-system not to generate
|
|
|
5060 subsidiaries -- it needs the coding systems we're creating
|
|
|
5061 to do so! */
|
|
|
5062 list2 (Qeol_type, Qlf)));
|
|
|
5063
|
|
|
5064 Fmake_coding_system
|
|
|
5065 (Qconvert_eol_crlf, Qconvert_eol,
|
|
|
5066 build_msg_string ("Convert CRLF to LF"),
|
|
|
5067 nconc2 (list6 (Qdocumentation,
|
|
|
5068 build_msg_string (
|
|
|
5069 "Converts CR+LF (used to mark the end of a line on Macintosh systems) to LF\n"
|
|
|
5070 "(used internally and under Unix to mark the end of a line)."),
|
|
|
5071 Qmnemonic, build_string ("CRLF->LF"),
|
|
|
5072 Qsubtype, Qcrlf),
|
|
|
5073 /* VERY IMPORTANT! Tell make-coding-system not to generate
|
|
|
5074 subsidiaries -- it needs the coding systems we're creating
|
|
|
5075 to do so! */
|
|
|
5076 list2 (Qeol_type, Qlf)));
|
|
|
5077
|
|
|
5078 Fmake_coding_system
|
|
|
5079 (Qconvert_eol_autodetect, Qconvert_eol,
|
|
|
5080 build_msg_string ("Autodetect EOL type"),
|
|
|
5081 nconc2 (list6 (Qdocumentation,
|
|
|
5082 build_msg_string (
|
|
|
5083 "Autodetect the end-of-line type."),
|
|
|
5084 Qmnemonic, build_string ("Auto-EOL"),
|
|
793
|
5085 Qsubtype, Qnil),
|
|
771
|
5086 /* VERY IMPORTANT! Tell make-coding-system not to generate
|
|
|
5087 subsidiaries -- it needs the coding systems we're creating
|
|
|
5088 to do so! */
|
|
|
5089 list2 (Qeol_type, Qlf)));
|
|
|
5090
|
|
|
5091 Fmake_coding_system
|
|
|
5092 (Qundecided, Qundecided,
|
|
|
5093 build_msg_string ("Undecided (auto-detect)"),
|
|
|
5094 nconc2 (list4 (Qdocumentation,
|
|
|
5095 build_msg_string
|
|
|
5096 ("Automatically detects the correct encoding."),
|
|
|
5097 Qmnemonic, build_string ("Auto")),
|
|
|
5098 list6 (Qdo_eol, Qt, Qdo_coding, Qt,
|
|
|
5099 /* We do EOL detection ourselves so we don't need to be
|
|
|
5100 wrapped in an EOL detector. (It doesn't actually hurt,
|
|
|
5101 though, I don't think.) */
|
|
|
5102 Qeol_type, Qlf)));
|
|
|
5103
|
|
|
5104 Fmake_coding_system
|
|
|
5105 (intern ("undecided-dos"), Qundecided,
|
|
|
5106 build_msg_string ("Undecided (auto-detect) (CRLF)"),
|
|
|
5107 nconc2 (list4 (Qdocumentation,
|
|
|
5108 build_msg_string
|
|
|
5109 ("Automatically detects the correct encoding; EOL type of CRLF forced."),
|
|
|
5110 Qmnemonic, build_string ("Auto")),
|
|
|
5111 list4 (Qdo_coding, Qt,
|
|
|
5112 Qeol_type, Qcrlf)));
|
|
|
5113
|
|
|
5114 Fmake_coding_system
|
|
|
5115 (intern ("undecided-unix"), Qundecided,
|
|
|
5116 build_msg_string ("Undecided (auto-detect) (LF)"),
|
|
|
5117 nconc2 (list4 (Qdocumentation,
|
|
|
5118 build_msg_string
|
|
|
5119 ("Automatically detects the correct encoding; EOL type of LF forced."),
|
|
|
5120 Qmnemonic, build_string ("Auto")),
|
|
|
5121 list4 (Qdo_coding, Qt,
|
|
|
5122 Qeol_type, Qlf)));
|
|
|
5123
|
|
|
5124 Fmake_coding_system
|
|
|
5125 (intern ("undecided-mac"), Qundecided,
|
|
|
5126 build_msg_string ("Undecided (auto-detect) (CR)"),
|
|
|
5127 nconc2 (list4 (Qdocumentation,
|
|
|
5128 build_msg_string
|
|
|
5129 ("Automatically detects the correct encoding; EOL type of CR forced."),
|
|
|
5130 Qmnemonic, build_string ("Auto")),
|
|
|
5131 list4 (Qdo_coding, Qt,
|
|
|
5132 Qeol_type, Qcr)));
|
|
|
5133
|
|
428
|
5134 /* Need to create this here or we're really screwed. */
|
|
|
5135 Fmake_coding_system
|
|
|
5136 (Qraw_text, Qno_conversion,
|
|
771
|
5137 build_msg_string ("Raw Text"),
|
|
|
5138 list4 (Qdocumentation,
|
|
|
5139 build_msg_string ("Raw text converts only line-break codes, and acts otherwise like `binary'."),
|
|
|
5140 Qmnemonic, build_string ("Raw")));
|
|
428
|
5141
|
|
|
5142 Fmake_coding_system
|
|
|
5143 (Qbinary, Qno_conversion,
|
|
771
|
5144 build_msg_string ("Binary"),
|
|
|
5145 list6 (Qdocumentation,
|
|
|
5146 build_msg_string (
|
|
|
5147 "This coding system is as close as it comes to doing no conversion.\n"
|
|
|
5148 "On input, each byte is converted directly into the character\n"
|
|
|
5149 "with the corresponding code -- i.e. from the `ascii', `control-1',\n"
|
|
|
5150 "or `latin-1' character sets. On output, these characters are\n"
|
|
|
5151 "converted back to the corresponding bytes, and other characters\n"
|
|
|
5152 "are converted to the default character, i.e. `~'."),
|
|
|
5153 Qeol_type, Qlf,
|
|
428
|
5154 Qmnemonic, build_string ("Binary")));
|
|
|
5155
|
|
771
|
5156 /* Formerly aliased to raw-text! Completely bogus and not even the same
|
|
|
5157 as FSF Emacs. */
|
|
|
5158 Fdefine_coding_system_alias (Qno_conversion, Qbinary);
|
|
|
5159 Fdefine_coding_system_alias (intern ("no-conversion-unix"),
|
|
|
5160 intern ("raw-text-unix"));
|
|
|
5161 Fdefine_coding_system_alias (intern ("no-conversion-dos"),
|
|
|
5162 intern ("raw-text-dos"));
|
|
|
5163 Fdefine_coding_system_alias (intern ("no-conversion-mac"),
|
|
|
5164 intern ("raw-text-mac"));
|
|
|
5165
|
|
1318
|
5166 /* These three below will get their defaults set correctly
|
|
|
5167 in code-init.el. We init them now so we can handle stuff at dump
|
|
771
|
5168 time before we get to code-init.el. */
|
|
1318
|
5169 Fdefine_coding_system_alias (Qnative, Qbinary);
|
|
440
|
5170 Fdefine_coding_system_alias (Qterminal, Qbinary);
|
|
|
5171 Fdefine_coding_system_alias (Qkeyboard, Qbinary);
|
|
|
5172
|
|
1318
|
5173 Fdefine_coding_system_alias (Qfile_name, Qnative);
|
|
771
|
5174 Fdefine_coding_system_alias (Qidentity, Qconvert_eol_lf);
|
|
|
5175
|
|
428
|
5176 /* Need this for bootstrapping */
|
|
771
|
5177 coding_category_system[detector_category_no_conversion] =
|
|
428
|
5178 Fget_coding_system (Qraw_text);
|
|
|
5179 }
|
