Mercurial > hg > xemacs-beta
annotate src/file-coding.h @ 5428:73eef12660cd
Converted gnuserv docs to GPLv3 or later.
| author | Mats Lidell <matsl@xemacs.org> |
|---|---|
| date | Sat, 06 Nov 2010 22:57:08 +0100 |
| parents | 308d34e9f07d |
| children | 65d65b52d608 |
| rev | line source |
|---|---|
| 771 | 1 /* Header for encoding conversion functions; coding-system object. |
| 2 #### rename me to coding-system.h | |
| 428 | 3 Copyright (C) 1991, 1995 Free Software Foundation, Inc. |
| 4 Copyright (C) 1995 Sun Microsystems, Inc. | |
| 793 | 5 Copyright (C) 2000, 2001, 2002 Ben Wing. |
| 428 | 6 |
| 7 This file is part of XEmacs. | |
| 8 | |
|
5402
308d34e9f07d
Changed bulk of GPLv2 or later files identified by script
Mats Lidell <matsl@xemacs.org>
parents:
5127
diff
changeset
|
9 XEmacs is free software: you can redistribute it and/or modify it |
| 428 | 10 under the terms of the GNU General Public License as published by the |
|
5402
308d34e9f07d
Changed bulk of GPLv2 or later files identified by script
Mats Lidell <matsl@xemacs.org>
parents:
5127
diff
changeset
|
11 Free Software Foundation, either version 3 of the License, or (at your |
|
308d34e9f07d
Changed bulk of GPLv2 or later files identified by script
Mats Lidell <matsl@xemacs.org>
parents:
5127
diff
changeset
|
12 option) any later version. |
| 428 | 13 |
| 14 XEmacs is distributed in the hope that it will be useful, but WITHOUT | |
| 15 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or | |
| 16 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License | |
| 17 for more details. | |
| 18 | |
| 19 You should have received a copy of the GNU General Public License | |
|
5402
308d34e9f07d
Changed bulk of GPLv2 or later files identified by script
Mats Lidell <matsl@xemacs.org>
parents:
5127
diff
changeset
|
20 along with XEmacs. If not, see <http://www.gnu.org/licenses/>. */ |
| 428 | 21 |
| 22 /* Synched up with: Mule 2.3. Not in FSF. */ | |
| 23 | |
| 771 | 24 /* Authorship: |
| 25 | |
| 26 Current primary author: Ben Wing <ben@xemacs.org> | |
| 27 | |
| 28 Written by Ben Wing <ben@xemacs.org> for XEmacs, 1995, loosely based | |
| 29 on code written 91.10.09 by K.Handa <handa@etl.go.jp>. | |
| 30 Rewritten again 2000-2001 by Ben Wing to support properly | |
| 31 abstracted coding systems. | |
| 32 September 2001: Finished last part of abstraction, the detection | |
| 33 mechanism. | |
| 34 */ | |
| 428 | 35 |
| 440 | 36 #ifndef INCLUDED_file_coding_h_ |
| 37 #define INCLUDED_file_coding_h_ | |
| 428 | 38 |
| 771 | 39 /* Capsule description of the different structures, what their purpose is, |
| 40 how they fit together, and where various bits of data are stored. | |
| 41 | |
| 2297 | 42 A "coding system" is an algorithm for converting stream data in one format |
| 43 into stream data in another format. Currently most of the coding systems | |
| 44 we have created concern internationalized text, and convert between the | |
| 45 XEmacs internal format for multilingual text, and various external | |
| 771 | 46 representations of such text. However, any such conversion is possible, |
| 47 for example, compressing or uncompressing text using the gzip algorithm. | |
| 48 All coding systems provide both encode and decode routines, so that the | |
| 2297 | 49 conversion can go both ways. Unfortunately encoding and decoding may not |
| 50 be exact inverses, even for a specific instance of a coding system. Care | |
| 51 must be taken when this is not the case. | |
| 771 | 52 |
| 53 The way we handle this is by dividing the various potential coding | |
| 54 systems into types, analogous to classes in C++. Each coding system | |
| 55 type encompasses a series of related coding systems that it can | |
| 56 implement, and it has properties which control how exactly the encoding | |
| 57 works. A particular set of values for each of the properties makes up a | |
| 58 "coding system", and specifies one particular encoding. A `struct | |
| 59 Lisp_Coding_System' object encapsulates those settings -- its type, the | |
| 60 values chosen for all properties of that type, a name for the coding | |
| 61 system, some documentation. | |
| 62 | |
| 63 In addition, there are of course methods associated with a coding system | |
| 64 type, implementing the encoding, decoding, etc. These are stored in a | |
| 65 `struct coding_system_methods' object, one per coding-system type, which | |
| 66 contains mostly function pointers. This is retrievable from the | |
| 67 coding-system object (i.e. the struct Lisp_Coding_System), which has a | |
| 68 pointer to it. | |
| 69 | |
| 70 In order to actually use a coding system to do an encoding or decoding | |
| 71 operation, you need to use a coding Lstream. | |
| 72 | |
| 73 Now let's look more at attached data. All coding systems have certain | |
| 74 common data fields -- name, type, documentation, etc. -- as well as a | |
| 75 bunch more that are defined by the coding system type. To handle this | |
| 76 cleanly, each coding system type defines a structure that holds just the | |
| 77 fields of data particular to it, and calls it e.g. `struct | |
| 78 iso2022_coding_system' for coding system type `iso2022'. When the | |
| 79 memory block holding the coding system object is created, it is sized | |
| 80 such that it can hold both the struct Lisp_Coding_System and the struct | |
| 81 iso2022_coding_system (or whatever) directly following it. (This is a | |
| 82 common trick; another possibility is to have a void * pointer in the | |
| 83 struct Lisp_Coding_System, which points to another memory block holding | |
| 84 the struct iso2022_coding_system.) A macro is provided | |
| 85 (CODING_SYSTEM_TYPE_DATA) to retrieve a pointer of the right type to the | |
| 86 type-specific data contained within the overall `struct | |
| 87 Lisp_Coding_System' block. | |
| 88 | |
| 89 Lstreams, similarly, are objects of type `struct lstream' holding data | |
| 90 about the stream operation (how much data has been read or written, any | |
| 91 buffered data, any error conditions, etc.), and like coding systems have | |
| 92 different types. They have a structure called `Lstream_implementation', | |
| 93 one per lstream type, exactly analogous to `struct | |
| 94 coding_system_methods'. In addition, they have type-specific data | |
| 95 (specifying, e.g., the file number, FILE *, memory location, other | |
| 96 lstream, etc. to read the data from or write it to, and for conversion | |
| 97 processes, the current state of the process -- are we decoding ASCII or | |
| 98 Kanji characters? are we in the middle of a processing an escape | |
| 99 sequence? etc.). This type-specific data is stored in a structure | |
| 100 named `struct coding_stream'. Just like for coding systems, the | |
| 101 type-independent data in the `struct lstream' and the type-dependent | |
| 102 data in the `struct coding_stream' are stored together in the same | |
| 103 memory block. | |
| 428 | 104 |
| 771 | 105 Now things get a bit tricky. The `struct coding_stream' is |
| 106 type-specific from the point of view of an lstream, but not from the | |
| 107 point of view of a coding system. It contains only general data about | |
| 108 the conversion process, e.g. the name of the coding system used for | |
| 109 conversion, the lstream that we take data from or write it to (depending | |
| 110 on whether this was created as a read stream or a write stream), a | |
| 111 buffer to hold extra data we retrieved but can't send on yet, some | |
| 112 flags, etc. It also needs some data specific to the particular coding | |
| 113 system and thus to the particular operation going on. This data is held | |
| 114 in a structure named (e.g.) `struct iso2022_coding_stream', and it's | |
| 115 held in a separate memory block and pointed to by the generic `struct | |
| 116 coding_stream'. It's not glommed into a single memory block both | |
| 117 because that would require making changes to the generic lstream code | |
| 118 and more importantly because the coding system used in a particular | |
| 119 coding lstream can be changed at any point during the lifetime of the | |
| 120 lstream, and possibly multiple times. (For example, it can be set using | |
| 121 the Lisp primitives `set-process-input-coding-system' and | |
| 122 `set-console-tty-input-coding-system', as well as getting set when a | |
| 123 conversion operation was started with coding system `undecided' and the | |
| 2297 | 124 correct coding system was then detected.) #### This suggests implementing |
| 125 compound text extended segments by saving the state of the ctext stream, | |
| 126 and installing an appropriate for the duration of the segment. | |
| 428 | 127 |
| 771 | 128 IMPORTANT NOTE: There are at least two ancillary data structures |
| 129 associated with a coding system type. (There may also be detection data; | |
| 130 see elsewhere.) It's important, when writing a coding system type, to | |
| 131 keep straight which type of data goes where. In particular, `struct | |
| 132 foo_coding_system' is attached to the coding system object itself. This | |
| 133 is a permanent object and there's only one per coding system. It's | |
| 134 created once, usually at init time, and never destroyed. So, `struct | |
| 135 foo_coding_system' should in general not contain dynamic data! (Just | |
| 136 data describing the properties of the coding system.) In particular, | |
| 137 *NO* data about any conversion in progress. There may be many | |
| 138 conversions going on simultaneously using a particular coding system, | |
| 139 and by storing conversion data in the coding system, these conversions | |
| 140 will overwrite each other's data. | |
| 141 | |
| 142 Instead, use the lstream object, whose purpose is to encapsulate a | |
| 143 particular conversion and all associated data. From the lstream object, | |
| 144 you can get the struct coding_stream using something like | |
| 145 | |
| 146 struct coding_stream *str = LSTREAM_TYPE_DATA (lstr, coding); | |
| 147 | |
| 148 But usually this structure is already passed to you as one of the | |
| 149 parameters of the method being invoked. | |
| 150 | |
| 151 From the struct coding_stream, you can retrieve the | |
| 152 coding-system-type-specific data using something like | |
| 153 | |
| 154 struct foo_coding_stream *data = CODING_STREAM_TYPE_DATA (str, foo); | |
| 155 | |
| 156 Then, use this structure to hold all data relevant to the particular | |
| 157 conversion being done. | |
| 158 | |
| 159 Initialize this structure whenever init_coding_stream_method is called | |
| 160 (this may happen more than once), and finalize it (free resources, etc.) | |
| 161 when finalize_coding_stream_method is called. | |
| 162 */ | |
| 163 | |
| 164 struct coding_stream; | |
| 165 struct detection_state; | |
| 166 | |
| 1204 | 167 extern const struct sized_memory_description coding_system_methods_description; |
| 771 | 168 |
| 169 struct coding_system_methods; | |
| 170 | |
| 171 enum source_sink_type | |
| 428 | 172 { |
| 771 | 173 DECODES_CHARACTER_TO_BYTE, |
| 174 DECODES_BYTE_TO_BYTE, | |
| 175 DECODES_BYTE_TO_CHARACTER, | |
| 176 DECODES_CHARACTER_TO_CHARACTER | |
| 428 | 177 }; |
| 178 | |
| 179 enum eol_type | |
| 180 { | |
| 181 EOL_LF, | |
| 182 EOL_CRLF, | |
| 771 | 183 EOL_CR, |
| 1429 | 184 EOL_AUTODETECT |
| 428 | 185 }; |
| 186 | |
| 187 struct Lisp_Coding_System | |
| 188 { | |
|
5127
a9c41067dd88
more cleanups, terminology clarification, lots of doc work
Ben Wing <ben@xemacs.org>
parents:
5124
diff
changeset
|
189 NORMAL_LISP_OBJECT_HEADER header; |
| 771 | 190 struct coding_system_methods *methods; |
| 428 | 191 |
| 1204 | 192 #define CODING_SYSTEM_SLOT_DECLARATION |
| 193 #define MARKED_SLOT(x) Lisp_Object x; | |
| 194 #include "coding-system-slots.h" | |
| 771 | 195 |
| 1204 | 196 /* Eol type requested by user. See comment about EOL junk in |
| 197 coding-system-slots.h. */ | |
| 771 | 198 enum eol_type eol_type; |
| 428 | 199 |
| 2132 | 200 /* If true, this is an internal coding system, which will not show up in |
| 201 coding-system-list unless a special parameter is given to it. */ | |
| 202 int internal_p; | |
| 203 | |
| 771 | 204 /* type-specific extra data attached to a coding_system */ |
| 205 char data[1]; | |
| 428 | 206 }; |
| 207 typedef struct Lisp_Coding_System Lisp_Coding_System; | |
| 208 | |
|
5118
e0db3c197671
merge up to latest default branch, doesn't compile yet
Ben Wing <ben@xemacs.org>
parents:
4690
diff
changeset
|
209 DECLARE_LISP_OBJECT (coding_system, Lisp_Coding_System); |
| 440 | 210 #define XCODING_SYSTEM(x) XRECORD (x, coding_system, Lisp_Coding_System) |
| 617 | 211 #define wrap_coding_system(p) wrap_record (p, coding_system) |
| 428 | 212 #define CODING_SYSTEMP(x) RECORDP (x, coding_system) |
| 213 #define CHECK_CODING_SYSTEM(x) CHECK_RECORD (x, coding_system) | |
| 214 #define CONCHECK_CODING_SYSTEM(x) CONCHECK_RECORD (x, coding_system) | |
| 215 | |
| 1204 | 216 enum coding_system_variant |
| 217 { | |
| 218 no_conversion_coding_system, | |
| 219 convert_eol_coding_system, | |
| 220 undecided_coding_system, | |
| 221 chain_coding_system, | |
| 222 text_file_wrapper_coding_system, | |
| 223 internal_coding_system, | |
| 224 gzip_coding_system, | |
| 225 mswindows_multibyte_to_unicode_coding_system, | |
| 226 mswindows_multibyte_coding_system, | |
| 227 iso2022_coding_system, | |
| 228 ccl_coding_system, | |
| 229 shift_jis_coding_system, | |
| 230 big5_coding_system, | |
|
4690
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
231 unicode_coding_system, |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
232 fixed_width_coding_system |
| 1204 | 233 }; |
| 234 | |
| 771 | 235 struct coding_system_methods |
| 236 { | |
| 237 Lisp_Object type; | |
| 238 Lisp_Object predicate_symbol; | |
| 239 | |
| 1204 | 240 /* Type expressed as an enum, needed for KKCC marking of the |
| 241 type-specific lstream data; copied into the struct coding_stream. */ | |
| 242 | |
| 243 enum coding_system_variant enumtype; | |
| 244 | |
| 771 | 245 /* Implementation specific methods: */ |
| 246 | |
| 247 /* Init method: Initialize coding-system data. Optional. */ | |
| 248 void (*init_method) (Lisp_Object coding_system); | |
| 249 | |
| 250 /* Mark method: Mark any Lisp objects in the type-specific data | |
| 251 attached to the coding-system object. Optional. */ | |
| 252 void (*mark_method) (Lisp_Object coding_system); | |
| 253 | |
| 254 /* Print method: Print the type-specific properties of this coding | |
| 255 system, as part of `print'-ing the object. If this method is defined | |
| 256 and prints anything, it should print a space as the first thing it | |
| 257 does. Optional. */ | |
| 258 void (*print_method) (Lisp_Object cs, Lisp_Object printcharfun, | |
| 259 int escapeflag); | |
| 260 | |
| 261 /* Canonicalize method: Convert this coding system to another one; called | |
| 262 once, at creation time, after all properties have been parsed. The | |
| 263 returned value should be a coding system created with | |
| 264 make_internal_coding_system() (passing the existing coding system as the | |
| 265 first argument), and will become the coding system returned by | |
| 266 `make-coding-system'. Optional. | |
| 267 | |
| 268 NOTE: There are *three* different uses of "canonical" or "canonicalize" | |
| 269 w.r.t. coding systems, and it's important to keep them straight. | |
| 270 | |
| 271 1. The canonicalize method. Used to specify a different coding | |
| 272 system, used when doing conversions, in place of the actual coding | |
| 273 system itself. Stored in the CANONICAL field of a coding system. | |
| 274 | |
| 275 2. The canonicalize-after-coding method. Used to return the encoding | |
| 276 that was "actually" used to decode some text, such that this | |
| 277 particular encoding can be used to encode the text again with the | |
| 278 expectation that the result will be the same as the original encoding. | |
| 279 Particularly important with auto-detecting coding systems. | |
| 280 | |
| 281 3. From the perspective of aliases, a "canonical" coding system is one | |
| 282 that's not an alias to some other coding system, and "canonicalization" | |
| 283 is the process of traversing the alias pointers to find the canonical | |
| 284 coding system that's equivalent to the alias. | |
| 285 */ | |
| 286 Lisp_Object (*canonicalize_method) (Lisp_Object coding_system); | |
| 287 | |
| 288 /* Canonicalize after coding method: Convert this coding system to | |
| 289 another one, after coding (usually decoding) has finished. This is | |
| 290 meant to be used by auto-detecting coding systems, which should return | |
| 291 the actually detected coding system. Optional. */ | |
| 292 Lisp_Object (*canonicalize_after_coding_method) | |
| 293 (struct coding_stream *str); | |
| 294 | |
| 295 /* Convert method: Decode or encode the data in SRC of size N, writing | |
| 296 the results into the Dynarr DST. If the conversion_end_type method | |
| 297 indicates that the source is characters (as opposed to bytes), you are | |
| 298 guaranteed to get only whole characters in the data in SRC/N. STR, a | |
| 299 struct coding_stream, stores all necessary state and other info about | |
| 300 the conversion. Coding-specific state (struct TYPE_coding_stream) can | |
| 301 be retrieved from STR using CODING_STREAM_TYPE_DATA(). Return value | |
| 302 indicates the number of bytes of the *INPUT* that were converted (not | |
| 303 the number of bytes written to the Dynarr!). This can be less than | |
| 304 the total amount of input passed in; if so, the remainder is | |
| 305 considered "rejected" and will appear again at the beginning of the | |
| 306 data passed in the next time the convert method is called. When EOF | |
| 307 is returned on the other end and there's no more data, the convert | |
| 308 method will be called one last time, STR->eof set and the passed-in | |
| 309 data will consist only of any rejected data from the previous | |
| 310 call. (At this point, file handles and similar resources can be | |
| 311 closed, but do NOT arbitrarily free data structures in the | |
| 312 type-specific data, because there are operations that can be done on | |
| 313 closed streams to query the results of the processing -- specifically, | |
| 314 for coding streams, there's the canonicalize_after_coding() method.) | |
| 315 Required. */ | |
| 316 Bytecount (*convert_method) (struct coding_stream *str, | |
| 317 const unsigned char *src, | |
| 318 unsigned_char_dynarr *dst, Bytecount n); | |
| 319 | |
|
4690
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
320 /* Query method: Check whether the buffer text between point and END |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
321 can be encoded by this coding system. Returns |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
322 either nil (meaning the text can be encoded by the coding system) or a |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
323 range table object describing the stretches that the coding system |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
324 cannot encode. |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
325 |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
326 Possible values for flags are below, search for |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
327 QUERY_METHOD_IGNORE_INVALID_SEQUENCES. |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
328 |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
329 Coding systems are expected to be able to behave sensibly with all |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
330 possible octets on decoding, which is why this method is only available |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
331 for encoding. */ |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
332 Lisp_Object (*query_method) (Lisp_Object coding_system, struct buffer *buf, |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
333 Charbpos end, int flags); |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
334 |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
335 /* Same as the previous method, but this works in the context of |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
336 lstreams. (Where the data do need to be copied, unfortunately.) The |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
337 intention is to implement the query method for the mswindows-multibyte |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
338 coding systems in terms of a query_lstream method. */ |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
339 Lisp_Object (*query_lstream_method) (struct coding_stream *str, |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
340 const Ibyte *start, Bytecount n); |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
341 |
| 771 | 342 /* Coding mark method: Mark any Lisp objects in the type-specific data |
| 343 attached to `struct coding_stream'. Optional. */ | |
| 344 void (*mark_coding_stream_method) (struct coding_stream *str); | |
| 345 | |
| 346 /* Init coding stream method: Initialize the type-specific data attached | |
| 347 to the coding stream (i.e. in struct TYPE_coding_stream), when the | |
| 348 coding stream is opened. The type-specific data will be zeroed out. | |
| 349 Optional. */ | |
| 350 void (*init_coding_stream_method) (struct coding_stream *str); | |
| 351 | |
| 352 /* Rewind coding stream method: Reset any necessary type-specific data as | |
| 353 a result of the stream being rewound. Optional. */ | |
| 354 void (*rewind_coding_stream_method) (struct coding_stream *str); | |
| 355 | |
| 356 /* Finalize coding stream method: Clean up the type-specific data | |
| 357 attached to the coding stream (i.e. in struct TYPE_coding_stream). | |
| 358 Happens when the Lstream is deleted using Lstream_delete() or is | |
| 359 garbage-collected. Most streams are deleted after they've been used, | |
| 360 so it's less likely (but still possible) that allocated data will | |
| 361 stick around until GC time. (File handles can also be closed when EOF | |
| 362 is signalled; but some data must stick around after this point, for | |
| 363 the benefit of canonicalize_after_coding. See the convert method.) | |
|
5124
623d57b7fbe8
separate regular and disksave finalization, print method fixes.
Ben Wing <ben@xemacs.org>
parents:
5120
diff
changeset
|
364 Called only once. Optional. */ |
| 771 | 365 void (*finalize_coding_stream_method) (struct coding_stream *str); |
| 366 | |
| 367 /* Finalize method: Clean up type-specific data (e.g. free allocated | |
| 368 data) attached to the coding system (i.e. in struct | |
| 369 TYPE_coding_system), when the coding system is about to be garbage | |
|
5124
623d57b7fbe8
separate regular and disksave finalization, print method fixes.
Ben Wing <ben@xemacs.org>
parents:
5120
diff
changeset
|
370 collected. (Currently not called.) Called only once. Optional. */ |
| 771 | 371 void (*finalize_method) (Lisp_Object codesys); |
| 372 | |
| 373 /* Conversion end type method: Does this coding system encode bytes -> | |
| 374 characters, characters -> characters, bytes -> bytes, or | |
| 375 characters -> bytes?. Default is characters -> bytes. Optional. */ | |
| 376 enum source_sink_type (*conversion_end_type_method) (Lisp_Object codesys); | |
| 377 | |
| 378 /* Putprop method: Set the value of a type-specific property. If | |
| 379 the property name is unrecognized, return 0. If the value is disallowed | |
| 380 or erroneous, signal an error. Currently called only at creation time. | |
| 381 Optional. */ | |
| 382 int (*putprop_method) (Lisp_Object codesys, | |
| 383 Lisp_Object key, | |
| 384 Lisp_Object value); | |
| 385 | |
| 386 /* Getprop method: Return the value of a type-specific property. If | |
| 387 the property name is unrecognized, return Qunbound. Optional. | |
| 388 */ | |
| 389 Lisp_Object (*getprop_method) (Lisp_Object coding_system, | |
| 390 Lisp_Object prop); | |
| 391 | |
| 392 /* These next three are set as part of the call to | |
| 393 INITIALIZE_CODING_SYSTEM_TYPE_WITH_DATA. */ | |
| 394 | |
| 395 /* Description of the extra data (struct foo_coding_system) attached to a | |
| 1204 | 396 coding system, for pdump purposes. */ |
| 397 const struct sized_memory_description *extra_description; | |
| 771 | 398 /* size of struct foo_coding_system -- extra data associated with |
| 399 the coding system */ | |
| 400 int extra_data_size; | |
| 401 /* size of struct foo_coding_stream -- extra data associated with the | |
| 402 struct coding_stream, needed for each active coding process | |
| 403 using this coding system. note that we can have more than one | |
| 404 process active at once (simply by creating more than one coding | |
| 405 lstream using this coding system), so we can't store this data in | |
| 406 the coding system object. */ | |
| 407 int coding_data_size; | |
| 408 }; | |
| 409 | |
|
4690
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
410 /* Values for flags, as passed to query_method. */ |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
411 |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
412 #define QUERY_METHOD_IGNORE_INVALID_SEQUENCES 0x0001 |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
413 #define QUERY_METHOD_ERRORP 0x0002 |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
414 #define QUERY_METHOD_HIGHLIGHT 0x0004 |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
415 |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
416 enum query_coding_failure_reasons |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
417 { |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
418 query_coding_succeeded = 0, |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
419 query_coding_unencodable = 1, |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
420 query_coding_invalid_sequence = 2 |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
421 }; |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
422 |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
423 extern Lisp_Object Qquery_coding_warning_face; |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
424 |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
425 Lisp_Object default_query_method (Lisp_Object, struct buffer *, Charbpos, |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
426 int); |
|
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
427 |
| 771 | 428 /***** Calling a coding-system method *****/ |
| 429 | |
| 430 #define RAW_CODESYSMETH(cs, m) ((cs)->methods->m##_method) | |
| 431 #define HAS_CODESYSMETH_P(cs, m) (!!RAW_CODESYSMETH (cs, m)) | |
| 432 #define CODESYSMETH(cs, m, args) (((cs)->methods->m##_method) args) | |
| 433 | |
| 434 /* Call a void-returning coding-system method, if it exists. */ | |
| 435 #define MAYBE_CODESYSMETH(cs, m, args) do { \ | |
| 436 Lisp_Coding_System *maybe_codesysmeth_cs = (cs); \ | |
| 437 if (HAS_CODESYSMETH_P (maybe_codesysmeth_cs, m)) \ | |
| 438 CODESYSMETH (maybe_codesysmeth_cs, m, args); \ | |
| 439 } while (0) | |
| 440 | |
| 441 /* Call a coding-system method, if it exists, or return GIVEN. | |
| 442 NOTE: Multiply-evaluates CS. */ | |
| 443 #define CODESYSMETH_OR_GIVEN(cs, m, args, given) \ | |
| 444 (HAS_CODESYSMETH_P (cs, m) ? \ | |
| 445 CODESYSMETH (cs, m, args) : (given)) | |
| 446 | |
| 447 #define XCODESYSMETH(cs, m, args) \ | |
| 448 CODESYSMETH (XCODING_SYSTEM (cs), m, args) | |
| 449 #define MAYBE_XCODESYSMETH(cs, m, args) \ | |
| 450 MAYBE_CODESYSMETH (XCODING_SYSTEM (cs), m, args) | |
| 451 #define XCODESYSMETH_OR_GIVEN(cs, m, args, given) \ | |
| 452 CODESYSMETH_OR_GIVEN (XCODING_SYSTEM (cs), m, args, given) | |
| 453 | |
| 454 /***** Defining new coding-system types *****/ | |
| 455 | |
| 1204 | 456 extern const struct sized_memory_description coding_system_empty_extra_description; |
| 771 | 457 |
| 800 | 458 #ifdef ERROR_CHECK_TYPES |
| 771 | 459 #define DECLARE_CODING_SYSTEM_TYPE(type) \ |
| 460 \ | |
| 461 extern struct coding_system_methods * type##_coding_system_methods; \ | |
| 826 | 462 DECLARE_INLINE_HEADER ( \ |
| 463 struct type##_coding_system * \ | |
| 771 | 464 error_check_##type##_coding_system_data (Lisp_Coding_System *cs) \ |
| 826 | 465 ) \ |
| 771 | 466 { \ |
| 467 assert (CODING_SYSTEM_TYPE_P (cs, type)); \ | |
| 468 /* Catch accidental use of INITIALIZE_CODING_SYSTEM_TYPE in place \ | |
| 469 of INITIALIZE_CODING_SYSTEM_TYPE_WITH_DATA. */ \ | |
| 470 assert (cs->methods->extra_data_size > 0); \ | |
| 471 return (struct type##_coding_system *) cs->data; \ | |
| 472 } \ | |
| 473 \ | |
| 826 | 474 DECLARE_INLINE_HEADER ( \ |
| 475 struct type##_coding_stream * \ | |
| 771 | 476 error_check_##type##_coding_stream_data (struct coding_stream *s) \ |
| 826 | 477 ) \ |
| 771 | 478 { \ |
| 479 assert (XCODING_SYSTEM_TYPE_P (s->codesys, type)); \ | |
| 480 return (struct type##_coding_stream *) s->data; \ | |
| 481 } \ | |
| 482 \ | |
| 826 | 483 DECLARE_INLINE_HEADER ( \ |
| 484 Lisp_Coding_System * \ | |
| 771 | 485 error_check_##type##_coding_system_type (Lisp_Object obj) \ |
| 826 | 486 ) \ |
| 771 | 487 { \ |
| 488 Lisp_Coding_System *cs = XCODING_SYSTEM (obj); \ | |
| 489 assert (CODING_SYSTEM_TYPE_P (cs, type)); \ | |
| 490 return cs; \ | |
| 491 } \ | |
| 492 \ | |
| 493 DECLARE_NOTHING | |
| 494 #else | |
| 495 #define DECLARE_CODING_SYSTEM_TYPE(type) \ | |
| 496 extern struct coding_system_methods * type##_coding_system_methods | |
| 800 | 497 #endif /* ERROR_CHECK_TYPES */ |
| 771 | 498 |
| 499 #define DEFINE_CODING_SYSTEM_TYPE(type) \ | |
| 500 struct coding_system_methods * type##_coding_system_methods | |
| 501 | |
| 1204 | 502 #define DEFINE_CODING_SYSTEM_TYPE_WITH_DATA(type) \ |
| 503 struct coding_system_methods * type##_coding_system_methods; \ | |
| 504 static const struct sized_memory_description \ | |
| 505 type##_coding_system_description_0 = { \ | |
| 506 sizeof (struct type##_coding_system), \ | |
| 507 type##_coding_system_description \ | |
| 508 } | |
| 509 | |
| 771 | 510 #define INITIALIZE_CODING_SYSTEM_TYPE(ty, pred_sym) do { \ |
| 511 ty##_coding_system_methods = \ | |
| 512 xnew_and_zero (struct coding_system_methods); \ | |
| 513 ty##_coding_system_methods->type = Q##ty; \ | |
| 514 ty##_coding_system_methods->extra_description = \ | |
| 1204 | 515 &coding_system_empty_extra_description; \ |
| 516 ty##_coding_system_methods->enumtype = ty##_coding_system; \ | |
|
4690
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
517 ty##_coding_system_methods->query_method = default_query_method; \ |
| 771 | 518 defsymbol_nodump (&ty##_coding_system_methods->predicate_symbol, \ |
| 519 pred_sym); \ | |
| 520 add_entry_to_coding_system_type_list (ty##_coding_system_methods); \ | |
| 2367 | 521 dump_add_root_block_ptr (&ty##_coding_system_methods, \ |
| 771 | 522 &coding_system_methods_description); \ |
| 523 } while (0) | |
| 524 | |
| 525 #define REINITIALIZE_CODING_SYSTEM_TYPE(type) do { \ | |
| 526 staticpro_nodump (&type##_coding_system_methods->predicate_symbol); \ | |
| 527 } while (0) | |
| 528 | |
| 529 /* This assumes the existence of two structures: | |
| 530 | |
| 531 struct foo_coding_system (attached to the coding system) | |
| 532 struct foo_coding_stream (per coding process, attached to the | |
| 533 struct coding_stream) | |
| 1204 | 534 const struct memory_description foo_coding_system_description[] |
| 535 (data description of struct foo_coding_system) | |
| 771 | 536 |
| 1204 | 537 For an example of how to do the description, see |
| 771 | 538 chain_coding_system_description. |
| 539 */ | |
| 540 #define INITIALIZE_CODING_SYSTEM_TYPE_WITH_DATA(type, pred_sym) \ | |
| 541 do { \ | |
| 542 INITIALIZE_CODING_SYSTEM_TYPE (type, pred_sym); \ | |
| 543 type##_coding_system_methods->extra_data_size = \ | |
| 544 sizeof (struct type##_coding_system); \ | |
| 545 type##_coding_system_methods->extra_description = \ | |
| 1204 | 546 &type##_coding_system_description_0; \ |
| 771 | 547 type##_coding_system_methods->coding_data_size = \ |
| 548 sizeof (struct type##_coding_stream); \ | |
| 549 } while (0) | |
| 550 | |
| 551 /* Declare that coding-system-type TYPE has method METH; used in | |
| 552 initialization routines */ | |
| 553 #define CODING_SYSTEM_HAS_METHOD(type, meth) \ | |
| 554 (type##_coding_system_methods->meth##_method = type##_##meth) | |
| 555 | |
| 556 /***** Macros for accessing coding-system types *****/ | |
| 557 | |
| 558 #define CODING_SYSTEM_TYPE_P(cs, type) \ | |
| 559 ((cs)->methods == type##_coding_system_methods) | |
| 560 #define XCODING_SYSTEM_TYPE_P(cs, type) \ | |
| 561 CODING_SYSTEM_TYPE_P (XCODING_SYSTEM (cs), type) | |
| 562 | |
| 800 | 563 #ifdef ERROR_CHECK_TYPES |
| 771 | 564 # define CODING_SYSTEM_TYPE_DATA(cs, type) \ |
| 565 error_check_##type##_coding_system_data (cs) | |
| 566 #else | |
| 567 # define CODING_SYSTEM_TYPE_DATA(cs, type) \ | |
| 568 ((struct type##_coding_system *) \ | |
| 569 (cs)->data) | |
| 570 #endif | |
| 571 | |
| 572 #define XCODING_SYSTEM_TYPE_DATA(cs, type) \ | |
| 573 CODING_SYSTEM_TYPE_DATA (XCODING_SYSTEM_OF_TYPE (cs, type), type) | |
| 574 | |
| 800 | 575 #ifdef ERROR_CHECK_TYPES |
| 771 | 576 # define XCODING_SYSTEM_OF_TYPE(x, type) \ |
| 577 error_check_##type##_coding_system_type (x) | |
| 578 # define XSETCODING_SYSTEM_OF_TYPE(x, p, type) do \ | |
| 579 { \ | |
| 793 | 580 x = wrap_coding_system (p); \ |
| 581 assert (CODING_SYSTEM_TYPEP (XCODING_SYSTEM (x), type)); \ | |
| 771 | 582 } while (0) |
| 583 #else | |
| 584 # define XCODING_SYSTEM_OF_TYPE(x, type) XCODING_SYSTEM (x) | |
| 793 | 585 # define XSETCODING_SYSTEM_OF_TYPE(x, p, type) do \ |
| 586 { \ | |
| 587 x = wrap_coding_system (p); \ | |
| 588 } while (0) | |
| 771 | 589 #endif /* ERROR_CHECK_TYPE_CHECK */ |
| 590 | |
| 591 #define CODING_SYSTEM_TYPEP(x, type) \ | |
| 592 (CODING_SYSTEMP (x) && CODING_SYSTEM_TYPE_P (XCODING_SYSTEM (x), type)) | |
| 593 #define CHECK_CODING_SYSTEM_OF_TYPE(x, type) do { \ | |
| 594 CHECK_CODING_SYSTEM (x); \ | |
| 595 if (!CODING_SYSTEM_TYPE_P (XCODING_SYSTEM (x), type)) \ | |
| 596 dead_wrong_type_argument \ | |
| 597 (type##_coding_system_methods->predicate_symbol, x); \ | |
| 598 } while (0) | |
| 599 #define CONCHECK_CODING_SYSTEM_OF_TYPE(x, type) do { \ | |
| 600 CONCHECK_CODING_SYSTEM (x); \ | |
| 601 if (!(CODING_SYSTEM_TYPEP (x, type))) \ | |
| 602 x = wrong_type_argument \ | |
| 603 (type##_coding_system_methods->predicate_symbol, x); \ | |
| 604 } while (0) | |
| 605 | |
| 606 #define CODING_SYSTEM_METHODS(codesys) ((codesys)->methods) | |
| 428 | 607 #define CODING_SYSTEM_NAME(codesys) ((codesys)->name) |
| 771 | 608 #define CODING_SYSTEM_DESCRIPTION(codesys) ((codesys)->description) |
| 609 #define CODING_SYSTEM_TYPE(codesys) ((codesys)->methods->type) | |
| 428 | 610 #define CODING_SYSTEM_MNEMONIC(codesys) ((codesys)->mnemonic) |
| 771 | 611 #define CODING_SYSTEM_DOCUMENTATION(codesys) ((codesys)->documentation) |
| 428 | 612 #define CODING_SYSTEM_POST_READ_CONVERSION(codesys) \ |
| 613 ((codesys)->post_read_conversion) | |
| 614 #define CODING_SYSTEM_PRE_WRITE_CONVERSION(codesys) \ | |
| 615 ((codesys)->pre_write_conversion) | |
| 616 #define CODING_SYSTEM_EOL_TYPE(codesys) ((codesys)->eol_type) | |
| 771 | 617 #define CODING_SYSTEM_EOL_LF(codesys) ((codesys)->eol[EOL_LF]) |
| 618 #define CODING_SYSTEM_EOL_CRLF(codesys) ((codesys)->eol[EOL_CRLF]) | |
| 619 #define CODING_SYSTEM_EOL_CR(codesys) ((codesys)->eol[EOL_CR]) | |
| 620 #define CODING_SYSTEM_TEXT_FILE_WRAPPER(codesys) ((codesys)->text_file_wrapper) | |
| 621 #define CODING_SYSTEM_AUTO_EOL_WRAPPER(codesys) ((codesys)->auto_eol_wrapper) | |
| 622 #define CODING_SYSTEM_SUBSIDIARY_PARENT(codesys) ((codesys)->subsidiary_parent) | |
| 623 #define CODING_SYSTEM_CANONICAL(codesys) ((codesys)->canonical) | |
|
4568
1d74a1d115ee
Add #'query-coding-region tests; do the work necessary to get them running.
Aidan Kehoe <kehoea@parhasard.net>
parents:
3017
diff
changeset
|
624 #define CODING_SYSTEM_SAFE_CHARSETS(codesys) ((codesys)->safe_charsets) |
|
1d74a1d115ee
Add #'query-coding-region tests; do the work necessary to get them running.
Aidan Kehoe <kehoea@parhasard.net>
parents:
3017
diff
changeset
|
625 #define CODING_SYSTEM_SAFE_CHARS(codesys) ((codesys)->safe_chars) |
| 428 | 626 |
| 771 | 627 #define CODING_SYSTEM_CHAIN_CHAIN(codesys) \ |
| 628 (CODING_SYSTEM_TYPE_DATA (codesys, chain)->chain) | |
| 629 #define CODING_SYSTEM_CHAIN_COUNT(codesys) \ | |
| 630 (CODING_SYSTEM_TYPE_DATA (codesys, chain)->count) | |
| 631 #define CODING_SYSTEM_CHAIN_CANONICALIZE_AFTER_CODING(codesys) \ | |
| 632 (CODING_SYSTEM_TYPE_DATA (codesys, chain)->canonicalize_after_coding) | |
| 428 | 633 |
| 771 | 634 #define XCODING_SYSTEM_METHODS(codesys) \ |
| 635 CODING_SYSTEM_METHODS (XCODING_SYSTEM (codesys)) | |
| 428 | 636 #define XCODING_SYSTEM_NAME(codesys) \ |
| 637 CODING_SYSTEM_NAME (XCODING_SYSTEM (codesys)) | |
| 771 | 638 #define XCODING_SYSTEM_DESCRIPTION(codesys) \ |
| 639 CODING_SYSTEM_DESCRIPTION (XCODING_SYSTEM (codesys)) | |
| 428 | 640 #define XCODING_SYSTEM_TYPE(codesys) \ |
| 641 CODING_SYSTEM_TYPE (XCODING_SYSTEM (codesys)) | |
| 642 #define XCODING_SYSTEM_MNEMONIC(codesys) \ | |
| 643 CODING_SYSTEM_MNEMONIC (XCODING_SYSTEM (codesys)) | |
| 771 | 644 #define XCODING_SYSTEM_DOCUMENTATION(codesys) \ |
| 645 CODING_SYSTEM_DOCUMENTATION (XCODING_SYSTEM (codesys)) | |
| 428 | 646 #define XCODING_SYSTEM_POST_READ_CONVERSION(codesys) \ |
| 647 CODING_SYSTEM_POST_READ_CONVERSION (XCODING_SYSTEM (codesys)) | |
| 648 #define XCODING_SYSTEM_PRE_WRITE_CONVERSION(codesys) \ | |
| 649 CODING_SYSTEM_PRE_WRITE_CONVERSION (XCODING_SYSTEM (codesys)) | |
| 650 #define XCODING_SYSTEM_EOL_TYPE(codesys) \ | |
| 651 CODING_SYSTEM_EOL_TYPE (XCODING_SYSTEM (codesys)) | |
| 652 #define XCODING_SYSTEM_EOL_LF(codesys) \ | |
| 653 CODING_SYSTEM_EOL_LF (XCODING_SYSTEM (codesys)) | |
| 654 #define XCODING_SYSTEM_EOL_CRLF(codesys) \ | |
| 655 CODING_SYSTEM_EOL_CRLF (XCODING_SYSTEM (codesys)) | |
| 656 #define XCODING_SYSTEM_EOL_CR(codesys) \ | |
| 657 CODING_SYSTEM_EOL_CR (XCODING_SYSTEM (codesys)) | |
| 771 | 658 #define XCODING_SYSTEM_TEXT_FILE_WRAPPER(codesys) \ |
| 659 CODING_SYSTEM_TEXT_FILE_WRAPPER (XCODING_SYSTEM (codesys)) | |
| 660 #define XCODING_SYSTEM_AUTO_EOL_WRAPPER(codesys) \ | |
| 661 CODING_SYSTEM_AUTO_EOL_WRAPPER (XCODING_SYSTEM (codesys)) | |
| 662 #define XCODING_SYSTEM_SUBSIDIARY_PARENT(codesys) \ | |
| 663 CODING_SYSTEM_SUBSIDIARY_PARENT (XCODING_SYSTEM (codesys)) | |
| 664 #define XCODING_SYSTEM_CANONICAL(codesys) \ | |
| 665 CODING_SYSTEM_CANONICAL (XCODING_SYSTEM (codesys)) | |
|
4568
1d74a1d115ee
Add #'query-coding-region tests; do the work necessary to get them running.
Aidan Kehoe <kehoea@parhasard.net>
parents:
3017
diff
changeset
|
666 #define XCODING_SYSTEM_SAFE_CHARSETS(codesys) \ |
|
1d74a1d115ee
Add #'query-coding-region tests; do the work necessary to get them running.
Aidan Kehoe <kehoea@parhasard.net>
parents:
3017
diff
changeset
|
667 CODING_SYSTEM_SAFE_CHARSETS (XCODING_SYSTEM (codesys)) |
|
1d74a1d115ee
Add #'query-coding-region tests; do the work necessary to get them running.
Aidan Kehoe <kehoea@parhasard.net>
parents:
3017
diff
changeset
|
668 #define XCODING_SYSTEM_SAFE_CHARS(codesys) \ |
|
1d74a1d115ee
Add #'query-coding-region tests; do the work necessary to get them running.
Aidan Kehoe <kehoea@parhasard.net>
parents:
3017
diff
changeset
|
669 CODING_SYSTEM_SAFE_CHARS (XCODING_SYSTEM (codesys)) |
| 428 | 670 |
| 771 | 671 #define XCODING_SYSTEM_CHAIN_CHAIN(codesys) \ |
| 672 CODING_SYSTEM_CHAIN_CHAIN (XCODING_SYSTEM (codesys)) | |
| 673 #define XCODING_SYSTEM_CHAIN_COUNT(codesys) \ | |
| 674 CODING_SYSTEM_CHAIN_COUNT (XCODING_SYSTEM (codesys)) | |
| 675 #define XCODING_SYSTEM_CHAIN_CANONICALIZE_AFTER_CODING(codesys) \ | |
| 676 CODING_SYSTEM_CHAIN_CANONICALIZE_AFTER_CODING (XCODING_SYSTEM (codesys)) | |
| 428 | 677 |
| 771 | 678 /**************************************************/ |
| 679 /* Detection */ | |
| 680 /**************************************************/ | |
| 428 | 681 |
| 771 | 682 #define MAX_DETECTOR_CATEGORIES 256 |
| 683 #define MAX_DETECTORS 64 | |
| 428 | 684 |
| 771 | 685 #define MAX_BYTES_PROCESSED_FOR_DETECTION 65536 |
| 428 | 686 |
| 771 | 687 struct detection_state |
| 428 | 688 { |
| 771 | 689 int seen_non_ascii; |
| 690 Bytecount bytes_seen; | |
| 428 | 691 |
| 771 | 692 char categories[MAX_DETECTOR_CATEGORIES]; |
| 693 Bytecount data_offset[MAX_DETECTORS]; | |
| 694 /* ... more data follows; data_offset[detector_##TYPE] points to | |
| 695 the data for that type */ | |
| 428 | 696 }; |
| 697 | |
| 771 | 698 #define DETECTION_STATE_DATA(st, type) \ |
| 699 ((struct type##_detector *) \ | |
| 700 ((char *) (st) + (st)->data_offset[detector_##type])) | |
| 428 | 701 |
| 448 | 702 /* Distinguishable categories of encodings. |
| 703 | |
| 704 This list determines the initial priority of the categories. | |
| 705 | |
| 706 For better or worse, currently Mule files are encoded in 7-bit ISO 2022. | |
| 707 For this reason, under Mule ISO_7 gets highest priority. | |
| 708 | |
| 709 Putting NO_CONVERSION second prevents "binary corruption" in the | |
| 710 default case in all but the (presumably) extremely rare case of a | |
| 711 binary file which contains redundant escape sequences but no 8-bit | |
| 712 characters. | |
| 713 | |
| 714 The remaining priorities are based on perceived "internationalization | |
| 715 political correctness." An exception is UCS-4 at the bottom, since | |
| 716 basically everything is compatible with UCS-4, but it is likely to | |
| 717 be very rare as an external encoding. */ | |
| 718 | |
| 771 | 719 /* Macros to define code of control characters for ISO2022's functions. */ |
| 720 /* Used by the detection routines of other coding system types as well. */ | |
| 721 /* code */ /* function */ | |
| 722 #define ISO_CODE_LF 0x0A /* line-feed */ | |
| 723 #define ISO_CODE_CR 0x0D /* carriage-return */ | |
| 724 #define ISO_CODE_SO 0x0E /* shift-out */ | |
| 725 #define ISO_CODE_SI 0x0F /* shift-in */ | |
| 726 #define ISO_CODE_ESC 0x1B /* escape */ | |
| 727 #define ISO_CODE_DEL 0x7F /* delete */ | |
| 728 #define ISO_CODE_SS2 0x8E /* single-shift-2 */ | |
| 729 #define ISO_CODE_SS3 0x8F /* single-shift-3 */ | |
| 730 #define ISO_CODE_CSI 0x9B /* control-sequence-introduce */ | |
| 731 | |
| 732 enum detection_result | |
| 733 { | |
| 734 /* Basically means a magic cookie was seen indicating this type, or | |
| 735 something similar. */ | |
| 736 DET_NEAR_CERTAINTY = 4, | |
| 737 DET_HIGHEST = 4, | |
| 738 /* Characteristics seen that are unlikely to be other coding system types | |
| 739 -- e.g. ISO-2022 escape sequences, or perhaps a consistent pattern of | |
| 740 alternating zero bytes in UTF-16, along with Unicode LF or CRLF | |
| 741 sequences at regular intervals. (Zero bytes are unlikely or impossible | |
| 742 in most text encodings.) */ | |
| 743 DET_QUITE_PROBABLE = 3, | |
| 744 /* Strong or medium statistical likelihood. At least some | |
| 745 characteristics seen that match what's normally found in this encoding | |
| 746 -- e.g. in Shift-JIS, a number of two-byte Japanese character | |
| 747 sequences in the right range, and nothing out of range; or in Unicode, | |
| 748 much higher statistical variance in the odd bytes than in the even | |
| 749 bytes, or vice-versa (perhaps the presence of regular EOL sequences | |
| 750 would bump this too to DET_QUITE_PROBABLE). This is quite often a | |
| 751 statistical test. */ | |
| 752 DET_SOMEWHAT_LIKELY = 2, | |
| 753 /* Weak statistical likelihood. Pretty much any features at all that | |
| 754 characterize this encoding, and nothing that rules against it. */ | |
| 755 DET_SLIGHTLY_LIKELY = 1, | |
| 756 /* Default state. Perhaps it indicates pure ASCII or something similarly | |
| 757 vague seen in Shift-JIS, or, exactly as the level says, it might mean | |
| 758 in a statistical-based detector that the pros and cons are balanced | |
| 759 out. This is also the lowest level that will be accepted by the | |
| 760 auto-detector without asking the user: If all available detectors | |
| 761 report lower levels for all categories with attached coding systems, | |
| 762 the user will be shown the results and explicitly prompted for action. | |
| 763 The user will also be prompted if this is the highest available level | |
| 764 and more than one detector reports the level. (See below about the | |
| 765 consequent necessity of an "ASCII" detector, which will return level 1 | |
| 766 or higher for most plain text files.) */ | |
| 767 DET_AS_LIKELY_AS_UNLIKELY = 0, | |
| 768 /* Some characteristics seen that are unusual for this encoding -- | |
| 769 e.g. unusual control characters in a plain-text encoding, lots of | |
| 770 8-bit characters, or little statistical variance in the odd and even | |
| 771 bytes in UTF-16. */ | |
| 772 DET_SOMEWHAT_UNLIKELY = -1, | |
| 773 /* This indicates that there is very little chance the data is in the | |
| 774 right format; this is probably the lowest level you can get when | |
| 775 presenting random binary data to a text file, because there are no | |
| 776 "specific sequences" you can see that would totally rule out | |
| 777 recognition. */ | |
| 778 DET_QUITE_IMPROBABLE = -2, | |
| 779 /* An erroneous sequence was seen. */ | |
| 780 DET_NEARLY_IMPOSSIBLE = -3, | |
| 1429 | 781 DET_LOWEST = -3 |
| 771 | 782 }; |
| 783 | |
| 784 extern int coding_detector_count; | |
| 785 extern int coding_detector_category_count; | |
| 786 | |
| 787 struct detector_category | |
| 428 | 788 { |
| 771 | 789 int id; |
| 790 Lisp_Object sym; | |
| 791 }; | |
| 792 | |
| 793 typedef struct | |
| 794 { | |
| 795 Dynarr_declare (struct detector_category); | |
| 796 } detector_category_dynarr; | |
| 797 | |
| 798 struct detector | |
| 799 { | |
| 800 int id; | |
| 801 detector_category_dynarr *cats; | |
| 802 Bytecount data_size; | |
| 803 /* Detect method: Required. */ | |
| 804 void (*detect_method) (struct detection_state *st, | |
| 805 const unsigned char *src, Bytecount n); | |
| 806 /* Finalize detection state method: Clean up any allocated data in the | |
|
5124
623d57b7fbe8
separate regular and disksave finalization, print method fixes.
Ben Wing <ben@xemacs.org>
parents:
5120
diff
changeset
|
807 detection state. Called only once. Optional. */ |
| 771 | 808 void (*finalize_detection_state_method) (struct detection_state *st); |
| 428 | 809 }; |
| 810 | |
| 771 | 811 /* Lvalue for a particular detection result -- detection state ST, |
| 812 category CAT */ | |
| 813 #define DET_RESULT(st, cat) ((st)->categories[detector_category_##cat]) | |
| 814 /* In state ST, set all detection results associated with detector DET to | |
| 815 RESULT. */ | |
| 816 #define SET_DET_RESULTS(st, det, result) \ | |
| 817 set_detection_results (st, detector_##det, result) | |
| 818 | |
| 819 typedef struct | |
| 820 { | |
| 821 Dynarr_declare (struct detector); | |
| 822 } detector_dynarr; | |
| 823 | |
| 824 extern detector_dynarr *all_coding_detectors; | |
| 825 | |
| 826 #define DEFINE_DETECTOR_CATEGORY(detector, cat) \ | |
| 827 int detector_category_##cat | |
| 828 #define DECLARE_DETECTOR_CATEGORY(detector, cat) \ | |
| 829 extern int detector_category_##cat | |
| 830 #define INITIALIZE_DETECTOR_CATEGORY(detector, cat) \ | |
| 831 do { \ | |
| 832 struct detector_category dog; \ | |
| 833 xzero (dog); \ | |
| 834 detector_category_##cat = coding_detector_category_count++; \ | |
| 835 dump_add_opaque_int (&detector_category_##cat); \ | |
| 836 dog.id = detector_category_##cat; \ | |
| 837 dog.sym = Q##cat; \ | |
| 838 Dynarr_add (Dynarr_at (all_coding_detectors, detector_##detector).cats, \ | |
| 839 dog); \ | |
| 840 } while (0) | |
| 841 | |
| 842 #define DEFINE_DETECTOR(Detector) \ | |
| 843 int detector_##Detector | |
| 844 #define DECLARE_DETECTOR(Detector) \ | |
| 845 extern int detector_##Detector | |
| 846 #define INITIALIZE_DETECTOR(Detector) \ | |
| 847 do { \ | |
| 848 struct detector det; \ | |
| 849 xzero (det); \ | |
| 850 detector_##Detector = coding_detector_count++; \ | |
| 851 dump_add_opaque_int (&detector_##Detector); \ | |
| 852 det.id = detector_##Detector; \ | |
| 853 det.cats = Dynarr_new2 (detector_category_dynarr, \ | |
| 854 struct detector_category); \ | |
| 855 det.data_size = sizeof (struct Detector##_detector); \ | |
| 856 Dynarr_add (all_coding_detectors, det); \ | |
| 857 } while (0) | |
| 858 #define DETECTOR_HAS_METHOD(Detector, Meth) \ | |
| 859 Dynarr_at (all_coding_detectors, detector_##Detector).Meth##_method = \ | |
| 802 | 860 Detector##_##Meth |
| 771 | 861 |
| 862 | |
| 863 /**************************************************/ | |
| 864 /* Decoding/Encoding */ | |
| 865 /**************************************************/ | |
| 866 | |
| 867 /* Is the source (SOURCEP == 1) or sink (SOURCEP == 0) when encoding specified | |
| 868 in characters? */ | |
| 869 | |
| 870 enum source_or_sink | |
| 871 { | |
| 872 CODING_SOURCE, | |
| 873 CODING_SINK | |
| 874 }; | |
| 875 | |
| 876 enum encode_decode | |
| 877 { | |
| 878 CODING_ENCODE, | |
| 879 CODING_DECODE | |
| 880 }; | |
| 881 | |
| 882 /* Data structure attached to an lstream of type `coding', | |
| 883 containing values specific to the coding process. Additional | |
| 884 data is stored in the DATA field below; the exact form of that data | |
| 885 is controlled by the type of the coding system that governs the | |
| 886 conversion (field CODESYS). CODESYS may be set at any time | |
| 887 throughout the lifetime of the lstream and possibly more than once. | |
| 888 See long comment above for more info. */ | |
| 889 | |
| 890 struct coding_stream | |
| 891 { | |
| 1204 | 892 /* Enumerated constant listing which type of console this is (TTY, X, |
| 893 MS-Windows, etc.). This duplicates the method structure in | |
| 894 XCODING_SYSTEM (str->codesys)->methods->type, which formerly was the | |
| 895 only way to determine the coding system type. We need this constant | |
| 896 now for KKCC, so that it can be used in an XD_UNION clause to | |
| 897 determine the Lisp objects in the type-specific data. */ | |
| 898 enum coding_system_variant type; | |
| 899 | |
| 771 | 900 /* Coding system that governs the conversion. */ |
| 901 Lisp_Object codesys; | |
| 902 /* Original coding system, pre-canonicalization. */ | |
| 903 Lisp_Object orig_codesys; | |
| 904 | |
| 905 /* Back pointer to current stream. */ | |
| 906 Lstream *us; | |
| 907 | |
| 908 /* Stream that we read the unprocessed data from or write the processed | |
| 909 data to. */ | |
| 910 Lstream *other_end; | |
| 911 | |
| 912 /* In order to handle both reading to and writing from a coding stream, | |
| 913 we phrase the conversion methods like write methods -- we can | |
| 914 implement reading in terms of a write method but not vice-versa, | |
| 915 because the write method is forced to take only what it's given but | |
| 916 the read method can read more data from the other end if necessary. | |
| 917 On the other hand, the write method is free to generate all the data | |
| 2297 | 918 it wants (and just write it to the other end), but the read method |
| 771 | 919 can return only as much as was asked for, so we need to implement our |
| 920 own buffering. */ | |
| 921 | |
| 922 /* If we are reading, then we can return only a fixed amount of data, but | |
| 923 the converter is free to return as much as it wants, so we direct it | |
| 924 to store the data here and lop off chunks as we need them. If we are | |
| 925 writing, we use this because the converter takes a Dynarr but we are | |
| 926 supposed to write into a fixed buffer. (NOTE: This introduces an extra | |
| 927 memory copy.) */ | |
| 928 unsigned_char_dynarr *convert_to; | |
| 929 | |
| 930 /* The conversion method might reject some of the data -- this typically | |
| 931 includes partial characters, partial escape sequences, etc. When | |
| 932 writing, we just pass the rejection up to the Lstream module, and it | |
| 933 will buffer the data. When reading, however, we need to do the | |
| 934 buffering ourselves, and we put it here, combined with newly read | |
| 935 data. */ | |
| 936 unsigned_char_dynarr *convert_from; | |
| 937 | |
| 938 /* If set, this is the last chunk of data being processed. When this is | |
| 939 finished, output any necessary terminating control characters, escape | |
| 940 sequences, etc. */ | |
| 941 unsigned int eof:1; | |
| 942 | |
| 943 /* CH holds a partially built-up character. This is really part of the | |
| 944 state-dependent data and should be moved there. */ | |
| 945 unsigned int ch; | |
| 946 | |
| 947 /* Coding-system-specific data holding extra state about the | |
| 948 conversion. Logically a struct TYPE_coding_stream; a pointer | |
| 800 | 949 to such a struct, with (when ERROR_CHECK_TYPES is defined) |
| 771 | 950 error-checking that this is really a structure of that type |
| 951 (checking the corresponding coding system type) can be retrieved using | |
| 952 CODING_STREAM_TYPE_DATA(). Allocated at the same time that | |
| 953 CODESYS is set (which may occur at any time, even multiple times, | |
| 954 during the lifetime of the stream). The size comes from | |
| 955 methods->coding_data_size. */ | |
| 956 void *data; | |
| 957 | |
| 958 enum encode_decode direction; | |
| 959 | |
| 800 | 960 /* If set, don't close the stream at the other end when being closed. */ |
| 961 unsigned int no_close_other:1; | |
| 802 | 962 /* If set, read only one byte at a time from other end to avoid any |
| 963 possible blocking. */ | |
| 964 unsigned int one_byte_at_a_time:1; | |
| 814 | 965 /* If set, and we're a read stream, we init char mode on ourselves as |
| 966 necessary to prevent the caller from getting partial characters. (the | |
| 967 default) */ | |
| 968 unsigned int set_char_mode_on_us_when_reading:1; | |
| 800 | 969 |
| 771 | 970 /* #### Temporary test */ |
| 971 unsigned int finalized:1; | |
| 972 }; | |
| 973 | |
| 974 #define CODING_STREAM_DATA(stream) LSTREAM_TYPE_DATA (stream, coding) | |
| 975 | |
| 800 | 976 #ifdef ERROR_CHECK_TYPES |
| 771 | 977 # define CODING_STREAM_TYPE_DATA(s, type) \ |
| 978 error_check_##type##_coding_stream_data (s) | |
| 979 #else | |
| 980 # define CODING_STREAM_TYPE_DATA(s, type) \ | |
| 981 ((struct type##_coding_stream *) (s)->data) | |
| 982 #endif | |
| 983 | |
| 984 /* C should be a binary character in the range 0 - 255; convert | |
| 985 to internal format and add to Dynarr DST. */ | |
| 986 | |
| 428 | 987 #ifdef MULE |
| 771 | 988 |
| 989 #define DECODE_ADD_BINARY_CHAR(c, dst) \ | |
| 990 do { \ | |
| 826 | 991 if (byte_ascii_p (c)) \ |
| 771 | 992 Dynarr_add (dst, c); \ |
| 826 | 993 else if (byte_c1_p (c)) \ |
| 771 | 994 { \ |
| 995 Dynarr_add (dst, LEADING_BYTE_CONTROL_1); \ | |
| 996 Dynarr_add (dst, c + 0x20); \ | |
| 997 } \ | |
| 998 else \ | |
| 999 { \ | |
| 1000 Dynarr_add (dst, LEADING_BYTE_LATIN_ISO8859_1); \ | |
| 1001 Dynarr_add (dst, c); \ | |
| 1002 } \ | |
| 1003 } while (0) | |
| 1004 | |
| 1005 #else /* not MULE */ | |
| 1006 | |
| 1007 #define DECODE_ADD_BINARY_CHAR(c, dst) \ | |
| 1008 do { \ | |
| 1009 Dynarr_add (dst, c); \ | |
| 1010 } while (0) | |
| 1011 | |
| 1012 #endif /* MULE */ | |
| 1013 | |
| 1014 #define DECODE_OUTPUT_PARTIAL_CHAR(ch, dst) \ | |
| 1015 do { \ | |
| 1016 if (ch) \ | |
| 1017 { \ | |
| 1018 DECODE_ADD_BINARY_CHAR (ch, dst); \ | |
| 1019 ch = 0; \ | |
| 1020 } \ | |
| 1021 } while (0) | |
| 428 | 1022 |
| 1023 #ifdef MULE | |
| 1024 /* Convert shift-JIS code (sj1, sj2) into internal string | |
| 1025 representation (c1, c2). (The leading byte is assumed.) */ | |
| 1026 | |
| 771 | 1027 #define DECODE_SHIFT_JIS(sj1, sj2, c1, c2) \ |
| 428 | 1028 do { \ |
| 1029 int I1 = sj1, I2 = sj2; \ | |
| 1030 if (I2 >= 0x9f) \ | |
| 1031 c1 = (I1 << 1) - ((I1 >= 0xe0) ? 0xe0 : 0x60), \ | |
| 1032 c2 = I2 + 2; \ | |
| 1033 else \ | |
| 1034 c1 = (I1 << 1) - ((I1 >= 0xe0) ? 0xe1 : 0x61), \ | |
| 1035 c2 = I2 + ((I2 >= 0x7f) ? 0x60 : 0x61); \ | |
| 1036 } while (0) | |
| 1037 | |
| 1038 /* Convert the internal string representation of a Shift-JIS character | |
| 1039 (c1, c2) into Shift-JIS code (sj1, sj2). The leading byte is | |
| 1040 assumed. */ | |
| 1041 | |
| 771 | 1042 #define ENCODE_SHIFT_JIS(c1, c2, sj1, sj2) \ |
| 428 | 1043 do { \ |
| 1044 int I1 = c1, I2 = c2; \ | |
| 1045 if (I1 & 1) \ | |
| 1046 sj1 = (I1 >> 1) + ((I1 < 0xdf) ? 0x31 : 0x71), \ | |
| 1047 sj2 = I2 - ((I2 >= 0xe0) ? 0x60 : 0x61); \ | |
| 1048 else \ | |
| 1049 sj1 = (I1 >> 1) + ((I1 < 0xdf) ? 0x30 : 0x70), \ | |
| 1050 sj2 = I2 - 2; \ | |
| 1051 } while (0) | |
| 1052 #endif /* MULE */ | |
| 1053 | |
| 771 | 1054 DECLARE_CODING_SYSTEM_TYPE (no_conversion); |
| 1055 DECLARE_CODING_SYSTEM_TYPE (convert_eol); | |
| 1056 #if 0 | |
| 1057 DECLARE_CODING_SYSTEM_TYPE (text_file_wrapper); | |
| 1058 #endif /* 0 */ | |
| 1059 DECLARE_CODING_SYSTEM_TYPE (undecided); | |
| 1060 DECLARE_CODING_SYSTEM_TYPE (chain); | |
| 1061 | |
| 1062 #ifdef DEBUG_XEMACS | |
| 1063 DECLARE_CODING_SYSTEM_TYPE (internal); | |
| 1064 #endif | |
| 1065 | |
| 1066 #ifdef MULE | |
| 1067 DECLARE_CODING_SYSTEM_TYPE (iso2022); | |
| 1068 DECLARE_CODING_SYSTEM_TYPE (ccl); | |
|
4690
257b468bf2ca
Move the #'query-coding-region implementation to C.
Aidan Kehoe <kehoea@parhasard.net>
parents:
4569
diff
changeset
|
1069 DECLARE_CODING_SYSTEM_TYPE (fixed_width); |
| 771 | 1070 DECLARE_CODING_SYSTEM_TYPE (shift_jis); |
| 1071 DECLARE_CODING_SYSTEM_TYPE (big5); | |
| 1072 #endif | |
| 1073 | |
| 1074 #ifdef HAVE_ZLIB | |
| 1075 DECLARE_CODING_SYSTEM_TYPE (gzip); | |
| 1076 #endif | |
| 428 | 1077 |
| 771 | 1078 DECLARE_CODING_SYSTEM_TYPE (unicode); |
| 428 | 1079 |
| 1315 | 1080 #ifdef WIN32_ANY |
| 771 | 1081 DECLARE_CODING_SYSTEM_TYPE (mswindows_multibyte_to_unicode); |
| 1082 DECLARE_CODING_SYSTEM_TYPE (mswindows_multibyte); | |
| 428 | 1083 #endif |
| 771 | 1084 |
| 1085 Lisp_Object coding_stream_detected_coding_system (Lstream *stream); | |
| 1086 Lisp_Object coding_stream_coding_system (Lstream *stream); | |
| 1087 void set_coding_stream_coding_system (Lstream *stream, | |
| 1088 Lisp_Object codesys); | |
| 1089 Lisp_Object detect_coding_stream (Lisp_Object stream); | |
| 867 | 1090 Ichar decode_big5_char (int o1, int o2); |
| 771 | 1091 void add_entry_to_coding_system_type_list (struct coding_system_methods *m); |
| 1092 Lisp_Object make_internal_coding_system (Lisp_Object existing, | |
|
4528
726060ee587c
First draft of g++ 4.3 warning removal patch. Builds. *Needs ChangeLogs.*
Stephen J. Turnbull <stephen@xemacs.org>
parents:
4522
diff
changeset
|
1093 const Ascbyte *prefix, |
| 771 | 1094 Lisp_Object type, |
| 1095 Lisp_Object description, | |
| 1096 Lisp_Object props); | |
| 802 | 1097 |
| 814 | 1098 #define LSTREAM_FL_NO_CLOSE_OTHER (1 << 16) |
| 1099 #define LSTREAM_FL_READ_ONE_BYTE_AT_A_TIME (1 << 17) | |
| 1100 #define LSTREAM_FL_NO_INIT_CHAR_MODE_WHEN_READING (1 << 18) | |
| 1101 | |
| 771 | 1102 Lisp_Object make_coding_input_stream (Lstream *stream, Lisp_Object codesys, |
| 800 | 1103 enum encode_decode direction, |
| 802 | 1104 int flags); |
| 771 | 1105 Lisp_Object make_coding_output_stream (Lstream *stream, Lisp_Object codesys, |
| 800 | 1106 enum encode_decode direction, |
| 802 | 1107 int flags); |
| 771 | 1108 void set_detection_results (struct detection_state *st, int detector, |
| 1109 int given); | |
| 428 | 1110 |
| 440 | 1111 #endif /* INCLUDED_file_coding_h_ */ |
| 1112 |