|
428
|
1 @c -*-texinfo-*-
|
|
|
2 @c This is part of the XEmacs Lisp Reference Manual.
|
|
775
|
3 @c Copyright (C) 1996 Ben Wing, 2001-2002 Free Software Foundation.
|
|
428
|
4 @c See the file lispref.texi for copying conditions.
|
|
|
5 @setfilename ../../info/internationalization.info
|
|
|
6 @node MULE, Tips, Internationalization, top
|
|
|
7 @chapter MULE
|
|
|
8
|
|
442
|
9 @dfn{MULE} is the name originally given to the version of GNU Emacs
|
|
428
|
10 extended for multi-lingual (and in particular Asian-language) support.
|
|
442
|
11 ``MULE'' is short for ``MUlti-Lingual Emacs''. It is an extension and
|
|
|
12 complete rewrite of Nemacs (``Nihon Emacs'' where ``Nihon'' is the
|
|
|
13 Japanese word for ``Japan''), which only provided support for Japanese.
|
|
|
14 XEmacs refers to its multi-lingual support as @dfn{MULE support} since
|
|
|
15 it is based on @dfn{MULE}.
|
|
428
|
16
|
|
|
17 @menu
|
|
|
18 * Internationalization Terminology::
|
|
|
19 Definition of various internationalization terms.
|
|
|
20 * Charsets:: Sets of related characters.
|
|
|
21 * MULE Characters:: Working with characters in XEmacs/MULE.
|
|
|
22 * Composite Characters:: Making new characters by overstriking other ones.
|
|
|
23 * Coding Systems:: Ways of representing a string of chars using integers.
|
|
|
24 * CCL:: A special language for writing fast converters.
|
|
|
25 * Category Tables:: Subdividing charsets into groups.
|
|
775
|
26 * Unicode Support:: The universal coded character set.
|
|
1183
|
27 * Charset Unification:: Handling overlapping character sets.
|
|
|
28 * Charsets and Coding Systems:: Tables and reference information.
|
|
428
|
29 @end menu
|
|
|
30
|
|
442
|
31 @node Internationalization Terminology, Charsets, , MULE
|
|
428
|
32 @section Internationalization Terminology
|
|
|
33
|
|
442
|
34 In internationalization terminology, a string of text is divided up
|
|
428
|
35 into @dfn{characters}, which are the printable units that make up the
|
|
|
36 text. A single character is (for example) a capital @samp{A}, the
|
|
442
|
37 number @samp{2}, a Katakana character, a Hangul character, a Kanji
|
|
|
38 ideograph (an @dfn{ideograph} is a ``picture'' character, such as is
|
|
|
39 used in Japanese Kanji, Chinese Hanzi, and Korean Hanja; typically there
|
|
|
40 are thousands of such ideographs in each language), etc. The basic
|
|
|
41 property of a character is that it is the smallest unit of text with
|
|
1261
|
42 semantic significance in text processing---i.e., characters are abstract
|
|
|
43 units defined by their meaning, not by their exact appearance.
|
|
442
|
44
|
|
|
45 Human beings normally process text visually, so to a first approximation
|
|
|
46 a character may be identified with its shape. Note that the same
|
|
|
47 character may be drawn by two different people (or in two different
|
|
|
48 fonts) in slightly different ways, although the "basic shape" will be the
|
|
|
49 same. But consider the works of Scott Kim; human beings can recognize
|
|
|
50 hugely variant shapes as the "same" character. Sometimes, especially
|
|
|
51 where characters are extremely complicated to write, completely
|
|
|
52 different shapes may be defined as the "same" character in national
|
|
|
53 standards. The Taiwanese variant of Hanzi is generally the most
|
|
444
|
54 complicated; over the centuries, the Japanese, Koreans, and the People's
|
|
442
|
55 Republic of China have adopted simplifications of the shape, but the
|
|
|
56 line of descent from the original shape is recorded, and the meanings
|
|
|
57 and pronunciation of different forms of the same character are
|
|
|
58 considered to be identical within each language. (Of course, it may
|
|
|
59 take a specialist to recognize the related form; the point is that the
|
|
|
60 relations are standardized, despite the differing shapes.)
|
|
428
|
61
|
|
|
62 In some cases, the differences will be significant enough that it is
|
|
|
63 actually possible to identify two or more distinct shapes that both
|
|
|
64 represent the same character. For example, the lowercase letters
|
|
440
|
65 @samp{a} and @samp{g} each have two distinct possible shapes---the
|
|
428
|
66 @samp{a} can optionally have a curved tail projecting off the top, and
|
|
|
67 the @samp{g} can be formed either of two loops, or of one loop and a
|
|
|
68 tail hanging off the bottom. Such distinct possible shapes of a
|
|
|
69 character are called @dfn{glyphs}. The important characteristic of two
|
|
|
70 glyphs making up the same character is that the choice between one or
|
|
|
71 the other is purely stylistic and has no linguistic effect on a word
|
|
|
72 (this is the reason why a capital @samp{A} and lowercase @samp{a}
|
|
440
|
73 are different characters rather than different glyphs---e.g.
|
|
428
|
74 @samp{Aspen} is a city while @samp{aspen} is a kind of tree).
|
|
|
75
|
|
|
76 Note that @dfn{character} and @dfn{glyph} are used differently
|
|
|
77 here than elsewhere in XEmacs.
|
|
|
78
|
|
442
|
79 A @dfn{character set} is essentially a set of related characters. ASCII,
|
|
428
|
80 for example, is a set of 94 characters (or 128, if you count
|
|
|
81 non-printing characters). Other character sets are ISO8859-1 (ASCII
|
|
|
82 plus various accented characters and other international symbols),
|
|
442
|
83 JIS X 0201 (ASCII, more or less, plus half-width Katakana), JIS X 0208
|
|
|
84 (Japanese Kanji), JIS X 0212 (a second set of less-used Japanese Kanji),
|
|
428
|
85 GB2312 (Mainland Chinese Hanzi), etc.
|
|
|
86
|
|
442
|
87 The definition of a character set will implicitly or explicitly give
|
|
|
88 it an @dfn{ordering}, a way of assigning a number to each character in
|
|
|
89 the set. For many character sets, there is a natural ordering, for
|
|
|
90 example the ``ABC'' ordering of the Roman letters. But it is not clear
|
|
|
91 whether digits should come before or after the letters, and in fact
|
|
|
92 different European languages treat the ordering of accented characters
|
|
|
93 differently. It is useful to use the natural order where available, of
|
|
|
94 course. The number assigned to any particular character is called the
|
|
|
95 character's @dfn{code point}. (Within a given character set, each
|
|
|
96 character has a unique code point. Thus the word "set" is ill-chosen;
|
|
|
97 different orderings of the same characters are different character sets.
|
|
|
98 Identifying characters is simple enough for alphabetic character sets,
|
|
|
99 but the difference in ordering can cause great headaches when the same
|
|
|
100 thousands of characters are used by different cultures as in the Hanzi.)
|
|
428
|
101
|
|
1261
|
102 It's important to understand that a character is defined not by any
|
|
|
103 number attached to it, but by its meaning. For example, ASCII and
|
|
|
104 EBCDIC are two charsets containing exactly the same characters
|
|
|
105 (lowercase and uppercase letters, numbers 0 through 9, particular
|
|
|
106 punctuation marks) but with different numberings. The @samp{comma}
|
|
|
107 character in ASCII and EBCDIC, for instance, is the same character
|
|
|
108 despite having a different numbering. Conversely, when comparing ASCII
|
|
|
109 and JIS-Roman, which look the same except that the latter has a yen sign
|
|
|
110 substituted for the backslash, we would say that the backslash and yen
|
|
|
111 sign are @emph{not} the same characters, despite having the same number
|
|
|
112 (95) and despite the fact that all other characters are present in both
|
|
|
113 charsets, with the same numbering. ASCII and JIS-Roman, then, do
|
|
|
114 @emph{not} have exactly the same characters in them (ASCII has a
|
|
|
115 backslash character but no yen-sign character, and vice-versa for
|
|
|
116 JIS-Roman), unlike ASCII and EBCDIC, even though the numberings in ASCII
|
|
|
117 and JIS-Roman are closer.
|
|
|
118
|
|
|
119 Sometimes, a code point is not a single number, but instead a group of
|
|
|
120 numbers, called @dfn{position codes}. In such cases, the number of
|
|
|
121 position codes required to index a particular character in a character
|
|
|
122 set is called the @dfn{dimension} of the character set. Character sets
|
|
|
123 indexed by more than one position code typically use byte-sized position
|
|
|
124 codes. Small character sets, e.g. ASCII, invariably use a single
|
|
|
125 position code, but for larger character sets, the choice of whether to
|
|
|
126 use multiple position codes or a single large (16-bit or 32-bit) number
|
|
|
127 is arbitrary. Unicode typically uses a single large number, but
|
|
|
128 language-specific or "national" character sets often use multiple
|
|
|
129 (usually two) position codes. For example, JIS X 0208, i.e. Japanese
|
|
|
130 Kanji, has thousands of characters, and is of dimension two -- every
|
|
|
131 character is indexed by two position codes, each in the range 1 through
|
|
|
132 94. (This number ``94'' is not a coincidence; it is the same as the
|
|
|
133 number of printable characters in ASCII, and was chosen so that JIS
|
|
|
134 characters could be directly encoded using two printable ASCII
|
|
|
135 characters.) Note that the choice of the range here is somewhat
|
|
|
136 arbitrary -- it could just as easily be 0 through 93, 2 through 95, etc.
|
|
|
137 In fact, the range for JIS position codes (and for other character sets
|
|
|
138 modeled after it) is often given as range 33 through 126, so as to
|
|
|
139 directly match ASCII printing characters.
|
|
428
|
140
|
|
|
141 An @dfn{encoding} is a way of numerically representing characters from
|
|
|
142 one or more character sets into a stream of like-sized numerical values
|
|
1261
|
143 called @dfn{words} -- typically 8-bit bytes, but sometimes 16-bit or
|
|
2818
|
144 32-bit quantities. In a context where dealing with Japanese motivates
|
|
|
145 much of XEmacs' design in this area, it's important to clearly
|
|
|
146 distinguish between charsets and encodings. For a simple charset like
|
|
|
147 ASCII, there is only one encoding normally used -- each character is
|
|
|
148 represented by a single byte, with the same value as its code point.
|
|
|
149 For more complicated charsets, however, or when a single encoding needs
|
|
|
150 to represent more than charset, things are not so obvious. Unicode
|
|
|
151 version 2, for example, is a large charset with thousands of characters,
|
|
|
152 each indexed by a 16-bit number, often represented in hex, e.g. 0x05D0
|
|
|
153 for the Hebrew letter "aleph". One obvious encoding (actually two
|
|
|
154 encodings, depending on which of the two possible byte orderings is
|
|
|
155 chosen) simply uses two bytes per character. This encoding is
|
|
|
156 convenient for internal processing of Unicode text; however, it's
|
|
|
157 incompatible with ASCII, and thus external text (files, e-mail, etc.)
|
|
|
158 that is encoded this way is completely uninterpretable by programs
|
|
|
159 lacking Unicode support. For this reason, a different, ASCII-compatible
|
|
|
160 encoding, e.g. UTF-8, is usually used for external text. UTF-8
|
|
|
161 represents Unicode characters with one to three bytes (often extended to
|
|
|
162 six bytes to handle characters with up to 31-bit indices). Unicode
|
|
|
163 characters 00 to 7F (identical with ASCII) are directly represented with
|
|
|
164 one byte, and other characters with two or more bytes, each in the range
|
|
|
165 80 to FF. Applications that don't understand Unicode will still be able
|
|
|
166 to process ASCII characters represented in UTF-8-encoded text, and will
|
|
|
167 typically ignore (and hopefully preserve) the high-bit characters.
|
|
|
168
|
|
|
169 Similarly, Shift-JIS and EUC-JP are different encodings normally used to
|
|
|
170 encode the same character set(s), these character sets being subsets of
|
|
|
171 Unicode. However, the obvious approach of unifying XEmacs' internal
|
|
|
172 encoding across character sets, as was part of the motivation behind
|
|
|
173 Unicode, wasn't taken. This means that characters in these character
|
|
|
174 sets that are identical to characters in other character sets---for
|
|
|
175 example, the Greek alphabet is in the large Japanese character sets and
|
|
|
176 at least one European character set--are unfortunately disjoint.
|
|
1261
|
177
|
|
|
178 Naive use of code points is also not possible if more than one
|
|
|
179 character set is to be used in the encoding. For example, printed
|
|
442
|
180 Japanese text typically requires characters from multiple character sets
|
|
|
181 -- ASCII, JIS X 0208, and JIS X 0212, to be specific. Each of these is
|
|
1261
|
182 indexed using one or more position codes in the range 1 through 94 (or
|
|
|
183 33 through 126), so the position codes could not be used directly or
|
|
|
184 there would be no way to tell which character was meant. Different
|
|
|
185 Japanese encodings handle this differently -- JIS uses special escape
|
|
|
186 characters to denote different character sets; EUC sets the high bit of
|
|
|
187 the position codes for JIS X 0208 and JIS X 0212, and puts a special
|
|
|
188 extra byte before each JIS X 0212 character; etc.
|
|
|
189
|
|
|
190 The encodings described above are all 7-bit or 8-bit encodings. The
|
|
|
191 fixed-width Unicode encoding previous described, however, is sometimes
|
|
|
192 considered to be a 16-bit encoding, in which case the issue of byte
|
|
|
193 ordering does not come up. (Imagine, for example, that the text is
|
|
|
194 represented as an array of shorts.) Similarly, Unicode version 3 (which
|
|
|
195 has characters with indices above 0xFFFF), and other very large
|
|
|
196 character sets, may be represented internally as 32-bit encodings,
|
|
|
197 i.e. arrays of ints. However, it does not make too much sense to talk
|
|
|
198 about 16-bit or 32-bit encodings for external data, since nowadays 8-bit
|
|
|
199 data is a universal standard -- the closest you can get is fixed-width
|
|
|
200 encodings using two or four bytes to encode 16-bit or 32-bit values. (A
|
|
|
201 "7-bit" encoding is used when it cannot be guaranteed that the high bit
|
|
|
202 of 8-bit data will be correctly preserved. Some e-mail gateways, for
|
|
|
203 example, strip the high bit of text passing through them. These same
|
|
|
204 gateways often handle non-printable characters incorrectly, and so 7-bit
|
|
|
205 encodings usually avoid using bytes with such values.)
|
|
442
|
206
|
|
|
207 A general method of handling text using multiple character sets
|
|
|
208 (whether for multilingual text, or simply text in an extremely
|
|
|
209 complicated single language like Japanese) is defined in the
|
|
|
210 international standard ISO 2022. ISO 2022 will be discussed in more
|
|
|
211 detail later (@pxref{ISO 2022}), but for now suffice it to say that text
|
|
|
212 needs control functions (at least spacing), and if escape sequences are
|
|
|
213 to be used, an escape sequence introducer. It was decided to make all
|
|
|
214 text streams compatible with ASCII in the sense that the codes 0--31
|
|
|
215 (and 128-159) would always be control codes, never graphic characters,
|
|
|
216 and where defined by the character set the @samp{SPC} character would be
|
|
|
217 assigned code 32, and @samp{DEL} would be assigned 127. Thus there are
|
|
|
218 94 code points remaining if 7 bits are used. This is the reason that
|
|
|
219 most character sets are defined using position codes in the range 1
|
|
|
220 through 94. Then ISO 2022 compatible encodings are produced by shifting
|
|
|
221 the position codes 1 to 94 into character codes 33 to 126, or (if 8 bit
|
|
|
222 codes are available) into character codes 161 to 254.
|
|
428
|
223
|
|
|
224 Encodings are classified as either @dfn{modal} or @dfn{non-modal}. In
|
|
442
|
225 a @dfn{modal encoding}, there are multiple states that the encoding can
|
|
|
226 be in, and the interpretation of the values in the stream depends on the
|
|
428
|
227 current global state of the encoding. Special values in the encoding,
|
|
|
228 called @dfn{escape sequences}, are used to change the global state.
|
|
|
229 JIS, for example, is a modal encoding. The bytes @samp{ESC $ B}
|
|
|
230 indicate that, from then on, bytes are to be interpreted as position
|
|
442
|
231 codes for JIS X 0208, rather than as ASCII. This effect is cancelled
|
|
428
|
232 using the bytes @samp{ESC ( B}, which mean ``switch from whatever the
|
|
442
|
233 current state is to ASCII''. To switch to JIS X 0212, the escape
|
|
|
234 sequence @samp{ESC $ ( D}. (Note that here, as is common, the escape
|
|
|
235 sequences do in fact begin with @samp{ESC}. This is not necessarily the
|
|
|
236 case, however. Some encodings use control characters called "locking
|
|
|
237 shifts" (effect persists until cancelled) to switch character sets.)
|
|
428
|
238
|
|
442
|
239 A @dfn{non-modal encoding} has no global state that extends past the
|
|
428
|
240 character currently being interpreted. EUC, for example, is a
|
|
442
|
241 non-modal encoding. Characters in JIS X 0208 are encoded by setting
|
|
|
242 the high bit of the position codes, and characters in JIS X 0212 are
|
|
428
|
243 encoded by doing the same but also prefixing the character with the
|
|
|
244 byte 0x8F.
|
|
|
245
|
|
|
246 The advantage of a modal encoding is that it is generally more
|
|
442
|
247 space-efficient, and is easily extendible because there are essentially
|
|
428
|
248 an arbitrary number of escape sequences that can be created. The
|
|
|
249 disadvantage, however, is that it is much more difficult to work with
|
|
|
250 if it is not being processed in a sequential manner. In the non-modal
|
|
|
251 EUC encoding, for example, the byte 0x41 always refers to the letter
|
|
|
252 @samp{A}; whereas in JIS, it could either be the letter @samp{A}, or
|
|
442
|
253 one of the two position codes in a JIS X 0208 character, or one of the
|
|
|
254 two position codes in a JIS X 0212 character. Determining exactly which
|
|
428
|
255 one is meant could be difficult and time-consuming if the previous
|
|
442
|
256 bytes in the string have not already been processed, or impossible if
|
|
|
257 they are drawn from an external stream that cannot be rewound.
|
|
428
|
258
|
|
|
259 Non-modal encodings are further divided into @dfn{fixed-width} and
|
|
|
260 @dfn{variable-width} formats. A fixed-width encoding always uses
|
|
|
261 the same number of words per character, whereas a variable-width
|
|
|
262 encoding does not. EUC is a good example of a variable-width
|
|
|
263 encoding: one to three bytes are used per character, depending on
|
|
|
264 the character set. 16-bit and 32-bit encodings are nearly always
|
|
|
265 fixed-width, and this is in fact one of the main reasons for using
|
|
|
266 an encoding with a larger word size. The advantages of fixed-width
|
|
|
267 encodings should be obvious. The advantages of variable-width
|
|
|
268 encodings are that they are generally more space-efficient and allow
|
|
442
|
269 for compatibility with existing 8-bit encodings such as ASCII. (For
|
|
|
270 example, in Unicode ASCII characters are simply promoted to a 16-bit
|
|
|
271 representation. That means that every ASCII character contains a
|
|
|
272 @samp{NUL} byte; evidently all of the standard string manipulation
|
|
|
273 functions will lose badly in a fixed-width Unicode environment.)
|
|
428
|
274
|
|
442
|
275 The bytes in an 8-bit encoding are often referred to as @dfn{octets}
|
|
|
276 rather than simply as bytes. This terminology dates back to the days
|
|
|
277 before 8-bit bytes were universal, when some computers had 9-bit bytes,
|
|
|
278 others had 10-bit bytes, etc.
|
|
428
|
279
|
|
442
|
280 @node Charsets, MULE Characters, Internationalization Terminology, MULE
|
|
428
|
281 @section Charsets
|
|
|
282
|
|
|
283 A @dfn{charset} in MULE is an object that encapsulates a
|
|
|
284 particular character set as well as an ordering of those characters.
|
|
|
285 Charsets are permanent objects and are named using symbols, like
|
|
|
286 faces.
|
|
|
287
|
|
|
288 @defun charsetp object
|
|
|
289 This function returns non-@code{nil} if @var{object} is a charset.
|
|
|
290 @end defun
|
|
|
291
|
|
|
292 @menu
|
|
|
293 * Charset Properties:: Properties of a charset.
|
|
|
294 * Basic Charset Functions:: Functions for working with charsets.
|
|
|
295 * Charset Property Functions:: Functions for accessing charset properties.
|
|
|
296 * Predefined Charsets:: Predefined charset objects.
|
|
|
297 @end menu
|
|
|
298
|
|
442
|
299 @node Charset Properties, Basic Charset Functions, , Charsets
|
|
428
|
300 @subsection Charset Properties
|
|
|
301
|
|
|
302 Charsets have the following properties:
|
|
|
303
|
|
|
304 @table @code
|
|
|
305 @item name
|
|
|
306 A symbol naming the charset. Every charset must have a different name;
|
|
|
307 this allows a charset to be referred to using its name rather than
|
|
|
308 the actual charset object.
|
|
|
309 @item doc-string
|
|
|
310 A documentation string describing the charset.
|
|
|
311 @item registry
|
|
|
312 A regular expression matching the font registry field for this character
|
|
|
313 set. For example, both the @code{ascii} and @code{latin-iso8859-1}
|
|
|
314 charsets use the registry @code{"ISO8859-1"}. This field is used to
|
|
|
315 choose an appropriate font when the user gives a general font
|
|
|
316 specification such as @samp{-*-courier-medium-r-*-140-*}, i.e. a
|
|
|
317 14-point upright medium-weight Courier font.
|
|
|
318 @item dimension
|
|
|
319 Number of position codes used to index a character in the character set.
|
|
|
320 XEmacs/MULE can only handle character sets of dimension 1 or 2.
|
|
|
321 This property defaults to 1.
|
|
|
322 @item chars
|
|
|
323 Number of characters in each dimension. In XEmacs/MULE, the only
|
|
|
324 allowed values are 94 or 96. (There are a couple of pre-defined
|
|
|
325 character sets, such as ASCII, that do not follow this, but you cannot
|
|
|
326 define new ones like this.) Defaults to 94. Note that if the dimension
|
|
|
327 is 2, the character set thus described is 94x94 or 96x96.
|
|
|
328 @item columns
|
|
|
329 Number of columns used to display a character in this charset.
|
|
|
330 Only used in TTY mode. (Under X, the actual width of a character
|
|
|
331 can be derived from the font used to display the characters.)
|
|
|
332 If unspecified, defaults to the dimension. (This is almost
|
|
|
333 always the correct value, because character sets with dimension 2
|
|
|
334 are usually ideograph character sets, which need two columns to
|
|
|
335 display the intricate ideographs.)
|
|
|
336 @item direction
|
|
|
337 A symbol, either @code{l2r} (left-to-right) or @code{r2l}
|
|
|
338 (right-to-left). Defaults to @code{l2r}. This specifies the
|
|
|
339 direction that the text should be displayed in, and will be
|
|
|
340 left-to-right for most charsets but right-to-left for Hebrew
|
|
|
341 and Arabic. (Right-to-left display is not currently implemented.)
|
|
|
342 @item final
|
|
|
343 Final byte of the standard ISO 2022 escape sequence designating this
|
|
|
344 charset. Must be supplied. Each combination of (@var{dimension},
|
|
|
345 @var{chars}) defines a separate namespace for final bytes, and each
|
|
|
346 charset within a particular namespace must have a different final byte.
|
|
|
347 Note that ISO 2022 restricts the final byte to the range 0x30 - 0x7E if
|
|
|
348 dimension == 1, and 0x30 - 0x5F if dimension == 2. Note also that final
|
|
|
349 bytes in the range 0x30 - 0x3F are reserved for user-defined (not
|
|
|
350 official) character sets. For more information on ISO 2022, see @ref{Coding
|
|
|
351 Systems}.
|
|
|
352 @item graphic
|
|
|
353 0 (use left half of font on output) or 1 (use right half of font on
|
|
|
354 output). Defaults to 0. This specifies how to convert the position
|
|
|
355 codes that index a character in a character set into an index into the
|
|
|
356 font used to display the character set. With @code{graphic} set to 0,
|
|
|
357 position codes 33 through 126 map to font indices 33 through 126; with
|
|
|
358 it set to 1, position codes 33 through 126 map to font indices 161
|
|
|
359 through 254 (i.e. the same number but with the high bit set). For
|
|
|
360 example, for a font whose registry is ISO8859-1, the left half of the
|
|
|
361 font (octets 0x20 - 0x7F) is the @code{ascii} charset, while the right
|
|
|
362 half (octets 0xA0 - 0xFF) is the @code{latin-iso8859-1} charset.
|
|
|
363 @item ccl-program
|
|
|
364 A compiled CCL program used to convert a character in this charset into
|
|
|
365 an index into the font. This is in addition to the @code{graphic}
|
|
|
366 property. If a CCL program is defined, the position codes of a
|
|
|
367 character will first be processed according to @code{graphic} and
|
|
|
368 then passed through the CCL program, with the resulting values used
|
|
|
369 to index the font.
|
|
|
370
|
|
442
|
371 This is used, for example, in the Big5 character set (used in Taiwan).
|
|
428
|
372 This character set is not ISO-2022-compliant, and its size (94x157) does
|
|
|
373 not fit within the maximum 96x96 size of ISO-2022-compliant character
|
|
|
374 sets. As a result, XEmacs/MULE splits it (in a rather complex fashion,
|
|
|
375 so as to group the most commonly used characters together) into two
|
|
|
376 charset objects (@code{big5-1} and @code{big5-2}), each of size 94x94,
|
|
|
377 and each charset object uses a CCL program to convert the modified
|
|
|
378 position codes back into standard Big5 indices to retrieve a character
|
|
|
379 from a Big5 font.
|
|
|
380 @end table
|
|
|
381
|
|
442
|
382 Most of the above properties can only be set when the charset is
|
|
|
383 initialized, and cannot be changed later.
|
|
|
384 @xref{Charset Property Functions}.
|
|
428
|
385
|
|
442
|
386 @node Basic Charset Functions, Charset Property Functions, Charset Properties, Charsets
|
|
428
|
387 @subsection Basic Charset Functions
|
|
|
388
|
|
|
389 @defun find-charset charset-or-name
|
|
|
390 This function retrieves the charset of the given name. If
|
|
|
391 @var{charset-or-name} is a charset object, it is simply returned.
|
|
|
392 Otherwise, @var{charset-or-name} should be a symbol. If there is no
|
|
|
393 such charset, @code{nil} is returned. Otherwise the associated charset
|
|
|
394 object is returned.
|
|
|
395 @end defun
|
|
|
396
|
|
|
397 @defun get-charset name
|
|
|
398 This function retrieves the charset of the given name. Same as
|
|
|
399 @code{find-charset} except an error is signalled if there is no such
|
|
|
400 charset instead of returning @code{nil}.
|
|
|
401 @end defun
|
|
|
402
|
|
|
403 @defun charset-list
|
|
|
404 This function returns a list of the names of all defined charsets.
|
|
|
405 @end defun
|
|
|
406
|
|
|
407 @defun make-charset name doc-string props
|
|
|
408 This function defines a new character set. This function is for use
|
|
442
|
409 with MULE support. @var{name} is a symbol, the name by which the
|
|
428
|
410 character set is normally referred. @var{doc-string} is a string
|
|
|
411 describing the character set. @var{props} is a property list,
|
|
|
412 describing the specific nature of the character set. The recognized
|
|
|
413 properties are @code{registry}, @code{dimension}, @code{columns},
|
|
|
414 @code{chars}, @code{final}, @code{graphic}, @code{direction}, and
|
|
|
415 @code{ccl-program}, as previously described.
|
|
|
416 @end defun
|
|
|
417
|
|
|
418 @defun make-reverse-direction-charset charset new-name
|
|
|
419 This function makes a charset equivalent to @var{charset} but which goes
|
|
|
420 in the opposite direction. @var{new-name} is the name of the new
|
|
|
421 charset. The new charset is returned.
|
|
|
422 @end defun
|
|
|
423
|
|
|
424 @defun charset-from-attributes dimension chars final &optional direction
|
|
|
425 This function returns a charset with the given @var{dimension},
|
|
|
426 @var{chars}, @var{final}, and @var{direction}. If @var{direction} is
|
|
|
427 omitted, both directions will be checked (left-to-right will be returned
|
|
|
428 if character sets exist for both directions).
|
|
|
429 @end defun
|
|
|
430
|
|
|
431 @defun charset-reverse-direction-charset charset
|
|
|
432 This function returns the charset (if any) with the same dimension,
|
|
|
433 number of characters, and final byte as @var{charset}, but which is
|
|
|
434 displayed in the opposite direction.
|
|
|
435 @end defun
|
|
|
436
|
|
442
|
437 @node Charset Property Functions, Predefined Charsets, Basic Charset Functions, Charsets
|
|
428
|
438 @subsection Charset Property Functions
|
|
|
439
|
|
442
|
440 All of these functions accept either a charset name or charset object.
|
|
428
|
441
|
|
|
442 @defun charset-property charset prop
|
|
|
443 This function returns property @var{prop} of @var{charset}.
|
|
|
444 @xref{Charset Properties}.
|
|
|
445 @end defun
|
|
|
446
|
|
442
|
447 Convenience functions are also provided for retrieving individual
|
|
428
|
448 properties of a charset.
|
|
|
449
|
|
|
450 @defun charset-name charset
|
|
|
451 This function returns the name of @var{charset}. This will be a symbol.
|
|
|
452 @end defun
|
|
|
453
|
|
444
|
454 @defun charset-description charset
|
|
|
455 This function returns the documentation string of @var{charset}.
|
|
428
|
456 @end defun
|
|
|
457
|
|
|
458 @defun charset-registry charset
|
|
|
459 This function returns the registry of @var{charset}.
|
|
|
460 @end defun
|
|
|
461
|
|
|
462 @defun charset-dimension charset
|
|
|
463 This function returns the dimension of @var{charset}.
|
|
|
464 @end defun
|
|
|
465
|
|
|
466 @defun charset-chars charset
|
|
|
467 This function returns the number of characters per dimension of
|
|
|
468 @var{charset}.
|
|
|
469 @end defun
|
|
|
470
|
|
444
|
471 @defun charset-width charset
|
|
428
|
472 This function returns the number of display columns per character (in
|
|
|
473 TTY mode) of @var{charset}.
|
|
|
474 @end defun
|
|
|
475
|
|
|
476 @defun charset-direction charset
|
|
440
|
477 This function returns the display direction of @var{charset}---either
|
|
428
|
478 @code{l2r} or @code{r2l}.
|
|
|
479 @end defun
|
|
|
480
|
|
444
|
481 @defun charset-iso-final-char charset
|
|
428
|
482 This function returns the final byte of the ISO 2022 escape sequence
|
|
|
483 designating @var{charset}.
|
|
|
484 @end defun
|
|
|
485
|
|
444
|
486 @defun charset-iso-graphic-plane charset
|
|
428
|
487 This function returns either 0 or 1, depending on whether the position
|
|
|
488 codes of characters in @var{charset} map to the left or right half
|
|
|
489 of their font, respectively.
|
|
|
490 @end defun
|
|
|
491
|
|
|
492 @defun charset-ccl-program charset
|
|
|
493 This function returns the CCL program, if any, for converting
|
|
|
494 position codes of characters in @var{charset} into font indices.
|
|
|
495 @end defun
|
|
|
496
|
|
1734
|
497 The two properties of a charset that can currently be set after the
|
|
|
498 charset has been created are the CCL program and the font registry.
|
|
428
|
499
|
|
|
500 @defun set-charset-ccl-program charset ccl-program
|
|
|
501 This function sets the @code{ccl-program} property of @var{charset} to
|
|
|
502 @var{ccl-program}.
|
|
|
503 @end defun
|
|
|
504
|
|
1734
|
505 @defun set-charset-registry charset registry
|
|
|
506 This function sets the @code{registry} property of @var{charset} to
|
|
|
507 @var{registry}.
|
|
|
508 @end defun
|
|
|
509
|
|
442
|
510 @node Predefined Charsets, , Charset Property Functions, Charsets
|
|
428
|
511 @subsection Predefined Charsets
|
|
|
512
|
|
442
|
513 The following charsets are predefined in the C code.
|
|
428
|
514
|
|
|
515 @example
|
|
|
516 Name Type Fi Gr Dir Registry
|
|
|
517 --------------------------------------------------------------
|
|
|
518 ascii 94 B 0 l2r ISO8859-1
|
|
|
519 control-1 94 0 l2r ---
|
|
|
520 latin-iso8859-1 94 A 1 l2r ISO8859-1
|
|
|
521 latin-iso8859-2 96 B 1 l2r ISO8859-2
|
|
|
522 latin-iso8859-3 96 C 1 l2r ISO8859-3
|
|
|
523 latin-iso8859-4 96 D 1 l2r ISO8859-4
|
|
|
524 cyrillic-iso8859-5 96 L 1 l2r ISO8859-5
|
|
|
525 arabic-iso8859-6 96 G 1 r2l ISO8859-6
|
|
|
526 greek-iso8859-7 96 F 1 l2r ISO8859-7
|
|
|
527 hebrew-iso8859-8 96 H 1 r2l ISO8859-8
|
|
|
528 latin-iso8859-9 96 M 1 l2r ISO8859-9
|
|
|
529 thai-tis620 96 T 1 l2r TIS620
|
|
|
530 katakana-jisx0201 94 I 1 l2r JISX0201.1976
|
|
|
531 latin-jisx0201 94 J 0 l2r JISX0201.1976
|
|
|
532 japanese-jisx0208-1978 94x94 @@ 0 l2r JISX0208.1978
|
|
|
533 japanese-jisx0208 94x94 B 0 l2r JISX0208.19(83|90)
|
|
|
534 japanese-jisx0212 94x94 D 0 l2r JISX0212
|
|
|
535 chinese-gb2312 94x94 A 0 l2r GB2312
|
|
|
536 chinese-cns11643-1 94x94 G 0 l2r CNS11643.1
|
|
|
537 chinese-cns11643-2 94x94 H 0 l2r CNS11643.2
|
|
|
538 chinese-big5-1 94x94 0 0 l2r Big5
|
|
|
539 chinese-big5-2 94x94 1 0 l2r Big5
|
|
|
540 korean-ksc5601 94x94 C 0 l2r KSC5601
|
|
|
541 composite 96x96 0 l2r ---
|
|
|
542 @end example
|
|
|
543
|
|
442
|
544 The following charsets are predefined in the Lisp code.
|
|
428
|
545
|
|
|
546 @example
|
|
|
547 Name Type Fi Gr Dir Registry
|
|
|
548 --------------------------------------------------------------
|
|
|
549 arabic-digit 94 2 0 l2r MuleArabic-0
|
|
|
550 arabic-1-column 94 3 0 r2l MuleArabic-1
|
|
|
551 arabic-2-column 94 4 0 r2l MuleArabic-2
|
|
|
552 sisheng 94 0 0 l2r sisheng_cwnn\|OMRON_UDC_ZH
|
|
|
553 chinese-cns11643-3 94x94 I 0 l2r CNS11643.1
|
|
|
554 chinese-cns11643-4 94x94 J 0 l2r CNS11643.1
|
|
|
555 chinese-cns11643-5 94x94 K 0 l2r CNS11643.1
|
|
|
556 chinese-cns11643-6 94x94 L 0 l2r CNS11643.1
|
|
|
557 chinese-cns11643-7 94x94 M 0 l2r CNS11643.1
|
|
|
558 ethiopic 94x94 2 0 l2r Ethio
|
|
|
559 ascii-r2l 94 B 0 r2l ISO8859-1
|
|
|
560 ipa 96 0 1 l2r MuleIPA
|
|
1734
|
561 vietnamese-viscii-lower 96 1 1 l2r VISCII1.1
|
|
|
562 vietnamese-viscii-upper 96 2 1 l2r VISCII1.1
|
|
428
|
563 @end example
|
|
|
564
|
|
|
565 For all of the above charsets, the dimension and number of columns are
|
|
|
566 the same.
|
|
|
567
|
|
442
|
568 Note that ASCII, Control-1, and Composite are handled specially.
|
|
428
|
569 This is why some of the fields are blank; and some of the filled-in
|
|
|
570 fields (e.g. the type) are not really accurate.
|
|
|
571
|
|
442
|
572 @node MULE Characters, Composite Characters, Charsets, MULE
|
|
428
|
573 @section MULE Characters
|
|
|
574
|
|
|
575 @defun make-char charset arg1 &optional arg2
|
|
|
576 This function makes a multi-byte character from @var{charset} and octets
|
|
|
577 @var{arg1} and @var{arg2}.
|
|
|
578 @end defun
|
|
|
579
|
|
444
|
580 @defun char-charset character
|
|
|
581 This function returns the character set of char @var{character}.
|
|
428
|
582 @end defun
|
|
|
583
|
|
444
|
584 @defun char-octet character &optional n
|
|
428
|
585 This function returns the octet (i.e. position code) numbered @var{n}
|
|
444
|
586 (should be 0 or 1) of char @var{character}. @var{n} defaults to 0 if omitted.
|
|
428
|
587 @end defun
|
|
|
588
|
|
|
589 @defun find-charset-region start end &optional buffer
|
|
|
590 This function returns a list of the charsets in the region between
|
|
|
591 @var{start} and @var{end}. @var{buffer} defaults to the current buffer
|
|
|
592 if omitted.
|
|
|
593 @end defun
|
|
|
594
|
|
|
595 @defun find-charset-string string
|
|
|
596 This function returns a list of the charsets in @var{string}.
|
|
|
597 @end defun
|
|
|
598
|
|
442
|
599 @node Composite Characters, Coding Systems, MULE Characters, MULE
|
|
428
|
600 @section Composite Characters
|
|
|
601
|
|
442
|
602 Composite characters are not yet completely implemented.
|
|
428
|
603
|
|
|
604 @defun make-composite-char string
|
|
|
605 This function converts a string into a single composite character. The
|
|
|
606 character is the result of overstriking all the characters in the
|
|
|
607 string.
|
|
|
608 @end defun
|
|
|
609
|
|
444
|
610 @defun composite-char-string character
|
|
428
|
611 This function returns a string of the characters comprising a composite
|
|
|
612 character.
|
|
|
613 @end defun
|
|
|
614
|
|
|
615 @defun compose-region start end &optional buffer
|
|
|
616 This function composes the characters in the region from @var{start} to
|
|
|
617 @var{end} in @var{buffer} into one composite character. The composite
|
|
|
618 character replaces the composed characters. @var{buffer} defaults to
|
|
|
619 the current buffer if omitted.
|
|
|
620 @end defun
|
|
|
621
|
|
|
622 @defun decompose-region start end &optional buffer
|
|
|
623 This function decomposes any composite characters in the region from
|
|
|
624 @var{start} to @var{end} in @var{buffer}. This converts each composite
|
|
|
625 character into one or more characters, the individual characters out of
|
|
|
626 which the composite character was formed. Non-composite characters are
|
|
|
627 left as-is. @var{buffer} defaults to the current buffer if omitted.
|
|
|
628 @end defun
|
|
|
629
|
|
442
|
630 @node Coding Systems, CCL, Composite Characters, MULE
|
|
|
631 @section Coding Systems
|
|
|
632
|
|
|
633 A coding system is an object that defines how text containing multiple
|
|
|
634 character sets is encoded into a stream of (typically 8-bit) bytes. The
|
|
|
635 coding system is used to decode the stream into a series of characters
|
|
|
636 (which may be from multiple charsets) when the text is read from a file
|
|
|
637 or process, and is used to encode the text back into the same format
|
|
|
638 when it is written out to a file or process.
|
|
|
639
|
|
|
640 For example, many ISO-2022-compliant coding systems (such as Compound
|
|
|
641 Text, which is used for inter-client data under the X Window System) use
|
|
|
642 escape sequences to switch between different charsets -- Japanese Kanji,
|
|
|
643 for example, is invoked with @samp{ESC $ ( B}; ASCII is invoked with
|
|
|
644 @samp{ESC ( B}; and Cyrillic is invoked with @samp{ESC - L}. See
|
|
|
645 @code{make-coding-system} for more information.
|
|
|
646
|
|
|
647 Coding systems are normally identified using a symbol, and the symbol is
|
|
|
648 accepted in place of the actual coding system object whenever a coding
|
|
|
649 system is called for. (This is similar to how faces and charsets work.)
|
|
|
650
|
|
|
651 @defun coding-system-p object
|
|
|
652 This function returns non-@code{nil} if @var{object} is a coding system.
|
|
|
653 @end defun
|
|
428
|
654
|
|
442
|
655 @menu
|
|
|
656 * Coding System Types:: Classifying coding systems.
|
|
|
657 * ISO 2022:: An international standard for
|
|
|
658 charsets and encodings.
|
|
|
659 * EOL Conversion:: Dealing with different ways of denoting
|
|
|
660 the end of a line.
|
|
|
661 * Coding System Properties:: Properties of a coding system.
|
|
|
662 * Basic Coding System Functions:: Working with coding systems.
|
|
|
663 * Coding System Property Functions:: Retrieving a coding system's properties.
|
|
|
664 * Encoding and Decoding Text:: Encoding and decoding text.
|
|
|
665 * Detection of Textual Encoding:: Determining how text is encoded.
|
|
|
666 * Big5 and Shift-JIS Functions:: Special functions for these non-standard
|
|
|
667 encodings.
|
|
|
668 * Predefined Coding Systems:: Coding systems implemented by MULE.
|
|
|
669 @end menu
|
|
428
|
670
|
|
442
|
671 @node Coding System Types, ISO 2022, , Coding Systems
|
|
|
672 @subsection Coding System Types
|
|
|
673
|
|
|
674 The coding system type determines the basic algorithm XEmacs will use to
|
|
|
675 decode or encode a data stream. Character encodings will be converted
|
|
|
676 to the MULE encoding, escape sequences processed, and newline sequences
|
|
|
677 converted to XEmacs's internal representation. There are three basic
|
|
|
678 classes of coding system type: no-conversion, ISO-2022, and special.
|
|
|
679
|
|
|
680 No conversion allows you to look at the file's internal representation.
|
|
|
681 Since XEmacs is basically a text editor, "no conversion" does convert
|
|
|
682 newline conventions by default. (Use the 'binary coding-system if this
|
|
|
683 is not desired.)
|
|
428
|
684
|
|
442
|
685 ISO 2022 (@pxref{ISO 2022}) is the basic international standard regulating
|
|
|
686 use of "coded character sets for the exchange of data", ie, text
|
|
|
687 streams. ISO 2022 contains functions that make it possible to encode
|
|
|
688 text streams to comply with restrictions of the Internet mail system and
|
|
|
689 de facto restrictions of most file systems (eg, use of the separator
|
|
|
690 character in file names). Coding systems which are not ISO 2022
|
|
|
691 conformant can be difficult to handle. Perhaps more important, they are
|
|
|
692 not adaptable to multilingual information interchange, with the obvious
|
|
|
693 exception of ISO 10646 (Unicode). (Unicode is partially supported by
|
|
|
694 XEmacs with the addition of the Lisp package ucs-conv.)
|
|
|
695
|
|
|
696 The special class of coding systems includes automatic detection, CCL (a
|
|
|
697 "little language" embedded as an interpreter, useful for translating
|
|
|
698 between variants of a single character set), non-ISO-2022-conformant
|
|
|
699 encodings like Unicode, Shift JIS, and Big5, and MULE internal coding.
|
|
|
700 (NB: this list is based on XEmacs 21.2. Terminology may vary slightly
|
|
|
701 for other versions of XEmacs and for GNU Emacs 20.)
|
|
|
702
|
|
|
703 @table @code
|
|
|
704 @item no-conversion
|
|
|
705 No conversion, for binary files, and a few special cases of non-ISO-2022
|
|
|
706 coding systems where conversion is done by hook functions (usually
|
|
|
707 implemented in CCL). On output, graphic characters that are not in
|
|
|
708 ASCII or Latin-1 will be replaced by a @samp{?}. (For a
|
|
|
709 no-conversion-encoded buffer, these characters will only be present if
|
|
|
710 you explicitly insert them.)
|
|
|
711 @item iso2022
|
|
|
712 Any ISO-2022-compliant encoding. Among others, this includes JIS (the
|
|
|
713 Japanese encoding commonly used for e-mail), national variants of EUC
|
|
|
714 (the standard Unix encoding for Japanese and other languages), and
|
|
|
715 Compound Text (an encoding used in X11). You can specify more specific
|
|
|
716 information about the conversion with the @var{flags} argument.
|
|
|
717 @item ucs-4
|
|
|
718 ISO 10646 UCS-4 encoding. A 31-bit fixed-width superset of Unicode.
|
|
|
719 @item utf-8
|
|
|
720 ISO 10646 UTF-8 encoding. A ``file system safe'' transformation format
|
|
|
721 that can be used with both UCS-4 and Unicode.
|
|
|
722 @item undecided
|
|
|
723 Automatic conversion. XEmacs attempts to detect the coding system used
|
|
|
724 in the file.
|
|
|
725 @item shift-jis
|
|
|
726 Shift-JIS (a Japanese encoding commonly used in PC operating systems).
|
|
|
727 @item big5
|
|
|
728 Big5 (the encoding commonly used for Taiwanese).
|
|
|
729 @item ccl
|
|
|
730 The conversion is performed using a user-written pseudo-code program.
|
|
|
731 CCL (Code Conversion Language) is the name of this pseudo-code. For
|
|
|
732 example, CCL is used to map KOI8-R characters (an encoding for Russian
|
|
|
733 Cyrillic) to ISO8859-5 (the form used internally by MULE).
|
|
|
734 @item internal
|
|
|
735 Write out or read in the raw contents of the memory representing the
|
|
|
736 buffer's text. This is primarily useful for debugging purposes, and is
|
|
|
737 only enabled when XEmacs has been compiled with @code{DEBUG_XEMACS} set
|
|
|
738 (the @samp{--debug} configure option). @strong{Warning}: Reading in a
|
|
|
739 file using @code{internal} conversion can result in an internal
|
|
|
740 inconsistency in the memory representing a buffer's text, which will
|
|
|
741 produce unpredictable results and may cause XEmacs to crash. Under
|
|
|
742 normal circumstances you should never use @code{internal} conversion.
|
|
428
|
743 @end table
|
|
|
744
|
|
442
|
745 @node ISO 2022, EOL Conversion, Coding System Types, Coding Systems
|
|
|
746 @section ISO 2022
|
|
|
747
|
|
|
748 This section briefly describes the ISO 2022 encoding standard. A more
|
|
|
749 thorough treatment is available in the original document of ISO
|
|
|
750 2022 as well as various national standards (such as JIS X 0202).
|
|
428
|
751
|
|
442
|
752 Character sets (@dfn{charsets}) are classified into the following four
|
|
|
753 categories, according to the number of characters in the charset:
|
|
|
754 94-charset, 96-charset, 94x94-charset, and 96x96-charset. This means
|
|
|
755 that although an ISO 2022 coding system may have variable width
|
|
|
756 characters, each charset used is fixed-width (in contrast to the MULE
|
|
|
757 character set and UTF-8, for example).
|
|
|
758
|
|
|
759 ISO 2022 provides for switching between character sets via escape
|
|
|
760 sequences. This switching is somewhat complicated, because ISO 2022
|
|
|
761 provides for both legacy applications like Internet mail that accept
|
|
444
|
762 only 7 significant bits in some contexts (RFC 822 headers, for example),
|
|
442
|
763 and more modern "8-bit clean" applications. It also provides for
|
|
|
764 compact and transparent representation of languages like Japanese which
|
|
|
765 mix ASCII and a national script (even outside of computer programs).
|
|
428
|
766
|
|
442
|
767 First, ISO 2022 codified prevailing practice by dividing the code space
|
|
|
768 into "control" and "graphic" regions. The code points 0x00-0x1F and
|
|
|
769 0x80-0x9F are reserved for "control characters", while "graphic
|
|
|
770 characters" must be assigned to code points in the regions 0x20-0x7F and
|
|
|
771 0xA0-0xFF. The positions 0x20 and 0x7F are special, and under some
|
|
|
772 circumstances must be assigned the graphic character "ASCII SPACE" and
|
|
|
773 the control character "ASCII DEL" respectively.
|
|
428
|
774
|
|
442
|
775 The various regions are given the name C0 (0x00-0x1F), GL (0x20-0x7F),
|
|
|
776 C1 (0x80-0x9F), and GR (0xA0-0xFF). GL and GR stand for "graphic left"
|
|
|
777 and "graphic right", respectively, because of the standard method of
|
|
|
778 displaying graphic character sets in tables with the high byte indexing
|
|
444
|
779 columns and the low byte indexing rows. I don't find it very intuitive,
|
|
442
|
780 but these are called "registers".
|
|
|
781
|
|
|
782 An ISO 2022-conformant encoding for a graphic character set must use a
|
|
|
783 fixed number of bytes per character, and the values must fit into a
|
|
|
784 single register; that is, each byte must range over either 0x20-0x7F, or
|
|
|
785 0xA0-0xFF. It is not allowed to extend the range of the repertoire of a
|
|
|
786 character set by using both ranges at the same. This is why a standard
|
|
|
787 character set such as ISO 8859-1 is actually considered by ISO 2022 to
|
|
|
788 be an aggregation of two character sets, ASCII and LATIN-1, and why it
|
|
|
789 is technically incorrect to refer to ISO 8859-1 as "Latin 1". Also, a
|
|
|
790 single character's bytes must all be drawn from the same register; this
|
|
|
791 is why Shift JIS (for Japanese) and Big 5 (for Chinese) are not ISO
|
|
|
792 2022-compatible encodings.
|
|
428
|
793
|
|
442
|
794 The reason for this restriction becomes clear when you attempt to define
|
|
|
795 an efficient, robust encoding for a language like Japanese. Like ISO
|
|
|
796 8859, Japanese encodings are aggregations of several character sets. In
|
|
|
797 practice, the vast majority of characters are drawn from the "JIS Roman"
|
|
|
798 character set (a derivative of ASCII; it won't hurt to think of it as
|
|
|
799 ASCII) and the JIS X 0208 standard "basic Japanese" character set
|
|
|
800 including not only ideographic characters ("kanji") but syllabic
|
|
|
801 Japanese characters ("kana"), a wide variety of symbols, and many
|
|
|
802 alphabetic characters (Roman, Greek, and Cyrillic) as well. Although
|
|
|
803 JIS X 0208 includes the whole Roman alphabet, as a 2-byte code it is not
|
|
|
804 suited to programming; thus the inclusion of ASCII in the standard
|
|
|
805 Japanese encodings.
|
|
428
|
806
|
|
442
|
807 For normal Japanese text such as in newspapers, a broad repertoire of
|
|
|
808 approximately 3000 characters is used. Evidently this won't fit into
|
|
|
809 one byte; two must be used. But much of the text processed by Japanese
|
|
|
810 computers is computer source code, nearly all of which is ASCII. A not
|
|
|
811 insignificant portion of ordinary text is English (as such or as
|
|
|
812 borrowed Japanese vocabulary) or other languages which can represented
|
|
|
813 at least approximately in ASCII, as well. It seems reasonable then to
|
|
|
814 represent ASCII in one byte, and JIS X 0208 in two. And this is exactly
|
|
|
815 what the Extended Unix Code for Japanese (EUC-JP) does. ASCII is
|
|
|
816 invoked to the GL register, and JIS X 0208 is invoked to the GR
|
|
|
817 register. Thus, each byte can be tested for its character set by
|
|
|
818 looking at the high bit; if set, it is Japanese, if clear, it is ASCII.
|
|
|
819 Furthermore, since control characters like newline can never be part of
|
|
|
820 a graphic character, even in the case of corruption in transmission the
|
|
|
821 stream will be resynchronized at every line break, on the order of 60-80
|
|
|
822 bytes. This coding system requires no escape sequences or special
|
|
|
823 control codes to represent 99.9% of all Japanese text.
|
|
428
|
824
|
|
442
|
825 Note carefully the distinction between the character sets (ASCII and JIS
|
|
|
826 X 0208), the encoding (EUC-JP), and the coding system (ISO 2022). The
|
|
|
827 JIS X 0208 character set is used in three different encodings for
|
|
|
828 Japanese, but in ISO-2022-JP it is invoked into GL (so the high bit is
|
|
|
829 always clear), in EUC-JP it is invoked into GR (setting the high bit in
|
|
|
830 the process), and in Shift JIS the high bit may be set or reset, and the
|
|
|
831 significant bits are shifted within the 16-bit character so that the two
|
|
|
832 main character sets can coexist with a third (the "halfwidth katakana"
|
|
|
833 of JIS X 0201). As the name implies, the ISO-2022-JP encoding is also a
|
|
|
834 version of the ISO-2022 coding system.
|
|
428
|
835
|
|
442
|
836 In order to systematically treat subsidiary character sets (like the
|
|
|
837 "halfwidth katakana" already mentioned, and the "supplementary kanji" of
|
|
|
838 JIS X 0212), four further registers are defined: G0, G1, G2, and G3.
|
|
|
839 Unlike GL and GR, they are not logically distinguished by internal
|
|
|
840 format. Instead, the process of "invocation" mentioned earlier is
|
|
|
841 broken into two steps: first, a character set is @dfn{designated} to one
|
|
|
842 of the registers G0-G3 by use of an @dfn{escape sequence} of the form:
|
|
428
|
843
|
|
|
844 @example
|
|
440
|
845 ESC [@var{I}] @var{I} @var{F}
|
|
428
|
846 @end example
|
|
|
847
|
|
442
|
848 where @var{I} is an intermediate character or characters in the range
|
|
|
849 0x20 - 0x3F, and @var{F}, from the range 0x30-0x7Fm is the final
|
|
|
850 character identifying this charset. (Final characters in the range
|
|
|
851 0x30-0x3F are reserved for private use and will never have a publicly
|
|
|
852 registered meaning.)
|
|
|
853
|
|
|
854 Then that register is @dfn{invoked} to either GL or GR, either
|
|
|
855 automatically (designations to G0 normally involve invocation to GL as
|
|
|
856 well), or by use of shifting (affecting only the following character in
|
|
|
857 the data stream) or locking (effective until the next designation or
|
|
|
858 locking) control sequences. An encoding conformant to ISO 2022 is
|
|
|
859 typically defined by designating the initial contents of the G0-G3
|
|
901
|
860 registers, specifying a 7 or 8 bit environment, and specifying whether
|
|
442
|
861 further designations will be recognized.
|
|
|
862
|
|
|
863 Some examples of character sets and the registered final characters
|
|
|
864 @var{F} used to designate them:
|
|
428
|
865
|
|
442
|
866 @need 1000
|
|
|
867 @table @asis
|
|
|
868 @item 94-charset
|
|
|
869 ASCII (B), left (J) and right (I) half of JIS X 0201, ...
|
|
|
870 @item 96-charset
|
|
|
871 Latin-1 (A), Latin-2 (B), Latin-3 (C), ...
|
|
|
872 @item 94x94-charset
|
|
|
873 GB2312 (A), JIS X 0208 (B), KSC5601 (C), ...
|
|
|
874 @item 96x96-charset
|
|
|
875 none for the moment
|
|
|
876 @end table
|
|
|
877
|
|
|
878 The meanings of the various characters in these sequences, where not
|
|
|
879 specified by the ISO 2022 standard (such as the ESC character), are
|
|
|
880 assigned by @dfn{ECMA}, the European Computer Manufacturers Association.
|
|
|
881
|
|
|
882 The meaning of intermediate characters are:
|
|
428
|
883
|
|
|
884 @example
|
|
|
885 @group
|
|
440
|
886 $ [0x24]: indicate charset of dimension 2 (94x94 or 96x96).
|
|
|
887 ( [0x28]: designate to G0 a 94-charset whose final byte is @var{F}.
|
|
|
888 ) [0x29]: designate to G1 a 94-charset whose final byte is @var{F}.
|
|
|
889 * [0x2A]: designate to G2 a 94-charset whose final byte is @var{F}.
|
|
|
890 + [0x2B]: designate to G3 a 94-charset whose final byte is @var{F}.
|
|
442
|
891 , [0x2C]: designate to G0 a 96-charset whose final byte is @var{F}.
|
|
440
|
892 - [0x2D]: designate to G1 a 96-charset whose final byte is @var{F}.
|
|
|
893 . [0x2E]: designate to G2 a 96-charset whose final byte is @var{F}.
|
|
|
894 / [0x2F]: designate to G3 a 96-charset whose final byte is @var{F}.
|
|
428
|
895 @end group
|
|
|
896 @end example
|
|
|
897
|
|
442
|
898 The comma may be used in files read and written only by MULE, as a MULE
|
|
|
899 extension, but this is illegal in ISO 2022. (The reason is that in ISO
|
|
|
900 2022 G0 must be a 94-member character set, with 0x20 assigned the value
|
|
|
901 SPACE, and 0x7F assigned the value DEL.)
|
|
428
|
902
|
|
442
|
903 Here are examples of designations:
|
|
428
|
904
|
|
|
905 @example
|
|
|
906 @group
|
|
440
|
907 ESC ( B : designate to G0 ASCII
|
|
|
908 ESC - A : designate to G1 Latin-1
|
|
|
909 ESC $ ( A or ESC $ A : designate to G0 GB2312
|
|
|
910 ESC $ ( B or ESC $ B : designate to G0 JISX0208
|
|
|
911 ESC $ ) C : designate to G1 KSC5601
|
|
428
|
912 @end group
|
|
|
913 @end example
|
|
|
914
|
|
442
|
915 (The short forms used to designate GB2312 and JIS X 0208 are for
|
|
|
916 backwards compatibility; the long forms are preferred.)
|
|
|
917
|
|
|
918 To use a charset designated to G2 or G3, and to use a charset designated
|
|
428
|
919 to G1 in a 7-bit environment, you must explicitly invoke G1, G2, or G3
|
|
|
920 into GL. There are two types of invocation, Locking Shift (forever) and
|
|
|
921 Single Shift (one character only).
|
|
|
922
|
|
442
|
923 Locking Shift is done as follows:
|
|
428
|
924
|
|
|
925 @example
|
|
440
|
926 LS0 or SI (0x0F): invoke G0 into GL
|
|
|
927 LS1 or SO (0x0E): invoke G1 into GL
|
|
|
928 LS2: invoke G2 into GL
|
|
|
929 LS3: invoke G3 into GL
|
|
|
930 LS1R: invoke G1 into GR
|
|
|
931 LS2R: invoke G2 into GR
|
|
|
932 LS3R: invoke G3 into GR
|
|
428
|
933 @end example
|
|
|
934
|
|
442
|
935 Single Shift is done as follows:
|
|
428
|
936
|
|
|
937 @example
|
|
|
938 @group
|
|
440
|
939 SS2 or ESC N: invoke G2 into GL
|
|
|
940 SS3 or ESC O: invoke G3 into GL
|
|
428
|
941 @end group
|
|
|
942 @end example
|
|
|
943
|
|
442
|
944 The shift functions (such as LS1R and SS3) are represented by control
|
|
|
945 characters (from C1) in 8 bit environments and by escape sequences in 7
|
|
|
946 bit environments.
|
|
|
947
|
|
428
|
948 (#### Ben says: I think the above is slightly incorrect. It appears that
|
|
|
949 SS2 invokes G2 into GR and SS3 invokes G3 into GR, whereas ESC N and
|
|
444
|
950 ESC O behave as indicated. The above definitions will not parse
|
|
428
|
951 EUC-encoded text correctly, and it looks like the code in mule-coding.c
|
|
|
952 has similar problems.)
|
|
|
953
|
|
442
|
954 Evidently there are a lot of ISO-2022-compliant ways of encoding
|
|
|
955 multilingual text. Now, in the world, there exist many coding systems
|
|
|
956 such as X11's Compound Text, Japanese JUNET code, and so-called EUC
|
|
|
957 (Extended UNIX Code); all of these are variants of ISO 2022.
|
|
428
|
958
|
|
442
|
959 In MULE, we characterize a version of ISO 2022 by the following
|
|
|
960 attributes:
|
|
428
|
961
|
|
|
962 @enumerate
|
|
|
963 @item
|
|
442
|
964 The character sets initially designated to G0 thru G3.
|
|
428
|
965 @item
|
|
442
|
966 Whether short form designations are allowed for Japanese and Chinese.
|
|
428
|
967 @item
|
|
442
|
968 Whether ASCII should be designated to G0 before control characters.
|
|
428
|
969 @item
|
|
442
|
970 Whether ASCII should be designated to G0 at the end of line.
|
|
428
|
971 @item
|
|
|
972 7-bit environment or 8-bit environment.
|
|
|
973 @item
|
|
442
|
974 Whether Locking Shifts are used or not.
|
|
428
|
975 @item
|
|
442
|
976 Whether to use ASCII or the variant JIS X 0201-1976-Roman.
|
|
428
|
977 @item
|
|
442
|
978 Whether to use JIS X 0208-1983 or the older version JIS X 0208-1976.
|
|
428
|
979 @end enumerate
|
|
|
980
|
|
|
981 (The last two are only for Japanese.)
|
|
|
982
|
|
442
|
983 By specifying these attributes, you can create any variant
|
|
428
|
984 of ISO 2022.
|
|
|
985
|
|
442
|
986 Here are several examples:
|
|
428
|
987
|
|
|
988 @example
|
|
|
989 @group
|
|
442
|
990 ISO-2022-JP -- Coding system used in Japanese email (RFC 1463 #### check).
|
|
440
|
991 1. G0 <- ASCII, G1..3 <- never used
|
|
|
992 2. Yes.
|
|
|
993 3. Yes.
|
|
|
994 4. Yes.
|
|
|
995 5. 7-bit environment
|
|
|
996 6. No.
|
|
|
997 7. Use ASCII
|
|
442
|
998 8. Use JIS X 0208-1983
|
|
428
|
999 @end group
|
|
|
1000
|
|
|
1001 @group
|
|
442
|
1002 ctext -- X11 Compound Text
|
|
|
1003 1. G0 <- ASCII, G1 <- Latin-1, G2,3 <- never used.
|
|
440
|
1004 2. No.
|
|
|
1005 3. No.
|
|
|
1006 4. Yes.
|
|
442
|
1007 5. 8-bit environment.
|
|
440
|
1008 6. No.
|
|
442
|
1009 7. Use ASCII.
|
|
|
1010 8. Use JIS X 0208-1983.
|
|
428
|
1011 @end group
|
|
|
1012
|
|
|
1013 @group
|
|
442
|
1014 euc-china -- Chinese EUC. Often called the "GB encoding", but that is
|
|
|
1015 technically incorrect.
|
|
|
1016 1. G0 <- ASCII, G1 <- GB 2312, G2,3 <- never used.
|
|
440
|
1017 2. No.
|
|
|
1018 3. Yes.
|
|
|
1019 4. Yes.
|
|
442
|
1020 5. 8-bit environment.
|
|
440
|
1021 6. No.
|
|
442
|
1022 7. Use ASCII.
|
|
|
1023 8. Use JIS X 0208-1983.
|
|
428
|
1024 @end group
|
|
|
1025
|
|
|
1026 @group
|
|
442
|
1027 ISO-2022-KR -- Coding system used in Korean email.
|
|
|
1028 1. G0 <- ASCII, G1 <- KSC 5601, G2,3 <- never used.
|
|
440
|
1029 2. No.
|
|
|
1030 3. Yes.
|
|
|
1031 4. Yes.
|
|
442
|
1032 5. 7-bit environment.
|
|
440
|
1033 6. Yes.
|
|
442
|
1034 7. Use ASCII.
|
|
|
1035 8. Use JIS X 0208-1983.
|
|
428
|
1036 @end group
|
|
|
1037 @end example
|
|
|
1038
|
|
442
|
1039 MULE creates all of these coding systems by default.
|
|
428
|
1040
|
|
442
|
1041 @node EOL Conversion, Coding System Properties, ISO 2022, Coding Systems
|
|
428
|
1042 @subsection EOL Conversion
|
|
|
1043
|
|
|
1044 @table @code
|
|
|
1045 @item nil
|
|
|
1046 Automatically detect the end-of-line type (LF, CRLF, or CR). Also
|
|
|
1047 generate subsidiary coding systems named @code{@var{name}-unix},
|
|
|
1048 @code{@var{name}-dos}, and @code{@var{name}-mac}, that are identical to
|
|
|
1049 this coding system but have an EOL-TYPE value of @code{lf}, @code{crlf},
|
|
|
1050 and @code{cr}, respectively.
|
|
|
1051 @item lf
|
|
|
1052 The end of a line is marked externally using ASCII LF. Since this is
|
|
|
1053 also the way that XEmacs represents an end-of-line internally,
|
|
|
1054 specifying this option results in no end-of-line conversion. This is
|
|
|
1055 the standard format for Unix text files.
|
|
|
1056 @item crlf
|
|
|
1057 The end of a line is marked externally using ASCII CRLF. This is the
|
|
|
1058 standard format for MS-DOS text files.
|
|
|
1059 @item cr
|
|
|
1060 The end of a line is marked externally using ASCII CR. This is the
|
|
|
1061 standard format for Macintosh text files.
|
|
|
1062 @item t
|
|
|
1063 Automatically detect the end-of-line type but do not generate subsidiary
|
|
|
1064 coding systems. (This value is converted to @code{nil} when stored
|
|
|
1065 internally, and @code{coding-system-property} will return @code{nil}.)
|
|
|
1066 @end table
|
|
|
1067
|
|
442
|
1068 @node Coding System Properties, Basic Coding System Functions, EOL Conversion, Coding Systems
|
|
428
|
1069 @subsection Coding System Properties
|
|
|
1070
|
|
|
1071 @table @code
|
|
|
1072 @item mnemonic
|
|
|
1073 String to be displayed in the modeline when this coding system is
|
|
|
1074 active.
|
|
|
1075
|
|
|
1076 @item eol-type
|
|
|
1077 End-of-line conversion to be used. It should be one of the types
|
|
|
1078 listed in @ref{EOL Conversion}.
|
|
|
1079
|
|
442
|
1080 @item eol-lf
|
|
444
|
1081 The coding system which is the same as this one, except that it uses the
|
|
442
|
1082 Unix line-breaking convention.
|
|
|
1083
|
|
|
1084 @item eol-crlf
|
|
444
|
1085 The coding system which is the same as this one, except that it uses the
|
|
442
|
1086 DOS line-breaking convention.
|
|
|
1087
|
|
|
1088 @item eol-cr
|
|
444
|
1089 The coding system which is the same as this one, except that it uses the
|
|
442
|
1090 Macintosh line-breaking convention.
|
|
|
1091
|
|
428
|
1092 @item post-read-conversion
|
|
|
1093 Function called after a file has been read in, to perform the decoding.
|
|
444
|
1094 Called with two arguments, @var{start} and @var{end}, denoting a region of
|
|
428
|
1095 the current buffer to be decoded.
|
|
|
1096
|
|
|
1097 @item pre-write-conversion
|
|
|
1098 Function called before a file is written out, to perform the encoding.
|
|
444
|
1099 Called with two arguments, @var{start} and @var{end}, denoting a region of
|
|
428
|
1100 the current buffer to be encoded.
|
|
|
1101 @end table
|
|
|
1102
|
|
442
|
1103 The following additional properties are recognized if @var{type} is
|
|
428
|
1104 @code{iso2022}:
|
|
|
1105
|
|
|
1106 @table @code
|
|
|
1107 @item charset-g0
|
|
|
1108 @itemx charset-g1
|
|
|
1109 @itemx charset-g2
|
|
|
1110 @itemx charset-g3
|
|
|
1111 The character set initially designated to the G0 - G3 registers.
|
|
|
1112 The value should be one of
|
|
|
1113
|
|
|
1114 @itemize @bullet
|
|
|
1115 @item
|
|
|
1116 A charset object (designate that character set)
|
|
|
1117 @item
|
|
|
1118 @code{nil} (do not ever use this register)
|
|
|
1119 @item
|
|
|
1120 @code{t} (no character set is initially designated to the register, but
|
|
|
1121 may be later on; this automatically sets the corresponding
|
|
|
1122 @code{force-g*-on-output} property)
|
|
|
1123 @end itemize
|
|
|
1124
|
|
|
1125 @item force-g0-on-output
|
|
|
1126 @itemx force-g1-on-output
|
|
|
1127 @itemx force-g2-on-output
|
|
|
1128 @itemx force-g3-on-output
|
|
|
1129 If non-@code{nil}, send an explicit designation sequence on output
|
|
|
1130 before using the specified register.
|
|
|
1131
|
|
|
1132 @item short
|
|
|
1133 If non-@code{nil}, use the short forms @samp{ESC $ @@}, @samp{ESC $ A},
|
|
|
1134 and @samp{ESC $ B} on output in place of the full designation sequences
|
|
|
1135 @samp{ESC $ ( @@}, @samp{ESC $ ( A}, and @samp{ESC $ ( B}.
|
|
|
1136
|
|
|
1137 @item no-ascii-eol
|
|
|
1138 If non-@code{nil}, don't designate ASCII to G0 at each end of line on
|
|
|
1139 output. Setting this to non-@code{nil} also suppresses other
|
|
|
1140 state-resetting that normally happens at the end of a line.
|
|
|
1141
|
|
|
1142 @item no-ascii-cntl
|
|
|
1143 If non-@code{nil}, don't designate ASCII to G0 before control chars on
|
|
|
1144 output.
|
|
|
1145
|
|
|
1146 @item seven
|
|
|
1147 If non-@code{nil}, use 7-bit environment on output. Otherwise, use 8-bit
|
|
|
1148 environment.
|
|
|
1149
|
|
|
1150 @item lock-shift
|
|
|
1151 If non-@code{nil}, use locking-shift (SO/SI) instead of single-shift or
|
|
|
1152 designation by escape sequence.
|
|
|
1153
|
|
|
1154 @item no-iso6429
|
|
|
1155 If non-@code{nil}, don't use ISO6429's direction specification.
|
|
|
1156
|
|
|
1157 @item escape-quoted
|
|
444
|
1158 If non-@code{nil}, literal control characters that are the same as the
|
|
428
|
1159 beginning of a recognized ISO 2022 or ISO 6429 escape sequence (in
|
|
|
1160 particular, ESC (0x1B), SO (0x0E), SI (0x0F), SS2 (0x8E), SS3 (0x8F),
|
|
|
1161 and CSI (0x9B)) are ``quoted'' with an escape character so that they can
|
|
|
1162 be properly distinguished from an escape sequence. (Note that doing
|
|
|
1163 this results in a non-portable encoding.) This encoding flag is used for
|
|
|
1164 byte-compiled files. Note that ESC is a good choice for a quoting
|
|
|
1165 character because there are no escape sequences whose second byte is a
|
|
|
1166 character from the Control-0 or Control-1 character sets; this is
|
|
|
1167 explicitly disallowed by the ISO 2022 standard.
|
|
|
1168
|
|
|
1169 @item input-charset-conversion
|
|
|
1170 A list of conversion specifications, specifying conversion of characters
|
|
|
1171 in one charset to another when decoding is performed. Each
|
|
|
1172 specification is a list of two elements: the source charset, and the
|
|
|
1173 destination charset.
|
|
|
1174
|
|
|
1175 @item output-charset-conversion
|
|
|
1176 A list of conversion specifications, specifying conversion of characters
|
|
|
1177 in one charset to another when encoding is performed. The form of each
|
|
|
1178 specification is the same as for @code{input-charset-conversion}.
|
|
|
1179 @end table
|
|
|
1180
|
|
442
|
1181 The following additional properties are recognized (and required) if
|
|
428
|
1182 @var{type} is @code{ccl}:
|
|
|
1183
|
|
|
1184 @table @code
|
|
|
1185 @item decode
|
|
|
1186 CCL program used for decoding (converting to internal format).
|
|
|
1187
|
|
|
1188 @item encode
|
|
|
1189 CCL program used for encoding (converting to external format).
|
|
|
1190 @end table
|
|
|
1191
|
|
442
|
1192 The following properties are used internally: @var{eol-cr},
|
|
|
1193 @var{eol-crlf}, @var{eol-lf}, and @var{base}.
|
|
|
1194
|
|
|
1195 @node Basic Coding System Functions, Coding System Property Functions, Coding System Properties, Coding Systems
|
|
428
|
1196 @subsection Basic Coding System Functions
|
|
|
1197
|
|
|
1198 @defun find-coding-system coding-system-or-name
|
|
|
1199 This function retrieves the coding system of the given name.
|
|
|
1200
|
|
442
|
1201 If @var{coding-system-or-name} is a coding-system object, it is simply
|
|
428
|
1202 returned. Otherwise, @var{coding-system-or-name} should be a symbol.
|
|
|
1203 If there is no such coding system, @code{nil} is returned. Otherwise
|
|
|
1204 the associated coding system object is returned.
|
|
|
1205 @end defun
|
|
|
1206
|
|
|
1207 @defun get-coding-system name
|
|
|
1208 This function retrieves the coding system of the given name. Same as
|
|
|
1209 @code{find-coding-system} except an error is signalled if there is no
|
|
|
1210 such coding system instead of returning @code{nil}.
|
|
|
1211 @end defun
|
|
|
1212
|
|
|
1213 @defun coding-system-list
|
|
|
1214 This function returns a list of the names of all defined coding systems.
|
|
|
1215 @end defun
|
|
|
1216
|
|
|
1217 @defun coding-system-name coding-system
|
|
|
1218 This function returns the name of the given coding system.
|
|
|
1219 @end defun
|
|
|
1220
|
|
442
|
1221 @defun coding-system-base coding-system
|
|
|
1222 Returns the base coding system (undecided EOL convention)
|
|
|
1223 coding system.
|
|
|
1224 @end defun
|
|
|
1225
|
|
428
|
1226 @defun make-coding-system name type &optional doc-string props
|
|
|
1227 This function registers symbol @var{name} as a coding system.
|
|
|
1228
|
|
|
1229 @var{type} describes the conversion method used and should be one of
|
|
|
1230 the types listed in @ref{Coding System Types}.
|
|
|
1231
|
|
|
1232 @var{doc-string} is a string describing the coding system.
|
|
|
1233
|
|
|
1234 @var{props} is a property list, describing the specific nature of the
|
|
|
1235 character set. Recognized properties are as in @ref{Coding System
|
|
|
1236 Properties}.
|
|
|
1237 @end defun
|
|
|
1238
|
|
|
1239 @defun copy-coding-system old-coding-system new-name
|
|
|
1240 This function copies @var{old-coding-system} to @var{new-name}. If
|
|
|
1241 @var{new-name} does not name an existing coding system, a new one will
|
|
|
1242 be created.
|
|
|
1243 @end defun
|
|
|
1244
|
|
|
1245 @defun subsidiary-coding-system coding-system eol-type
|
|
|
1246 This function returns the subsidiary coding system of
|
|
|
1247 @var{coding-system} with eol type @var{eol-type}.
|
|
|
1248 @end defun
|
|
|
1249
|
|
442
|
1250 @node Coding System Property Functions, Encoding and Decoding Text, Basic Coding System Functions, Coding Systems
|
|
428
|
1251 @subsection Coding System Property Functions
|
|
|
1252
|
|
|
1253 @defun coding-system-doc-string coding-system
|
|
|
1254 This function returns the doc string for @var{coding-system}.
|
|
|
1255 @end defun
|
|
|
1256
|
|
|
1257 @defun coding-system-type coding-system
|
|
|
1258 This function returns the type of @var{coding-system}.
|
|
|
1259 @end defun
|
|
|
1260
|
|
|
1261 @defun coding-system-property coding-system prop
|
|
|
1262 This function returns the @var{prop} property of @var{coding-system}.
|
|
|
1263 @end defun
|
|
|
1264
|
|
442
|
1265 @node Encoding and Decoding Text, Detection of Textual Encoding, Coding System Property Functions, Coding Systems
|
|
428
|
1266 @subsection Encoding and Decoding Text
|
|
|
1267
|
|
|
1268 @defun decode-coding-region start end coding-system &optional buffer
|
|
|
1269 This function decodes the text between @var{start} and @var{end} which
|
|
|
1270 is encoded in @var{coding-system}. This is useful if you've read in
|
|
|
1271 encoded text from a file without decoding it (e.g. you read in a
|
|
|
1272 JIS-formatted file but used the @code{binary} or @code{no-conversion} coding
|
|
|
1273 system, so that it shows up as @samp{^[$B!<!+^[(B}). The length of the
|
|
|
1274 encoded text is returned. @var{buffer} defaults to the current buffer
|
|
|
1275 if unspecified.
|
|
|
1276 @end defun
|
|
|
1277
|
|
|
1278 @defun encode-coding-region start end coding-system &optional buffer
|
|
|
1279 This function encodes the text between @var{start} and @var{end} using
|
|
|
1280 @var{coding-system}. This will, for example, convert Japanese
|
|
|
1281 characters into stuff such as @samp{^[$B!<!+^[(B} if you use the JIS
|
|
|
1282 encoding. The length of the encoded text is returned. @var{buffer}
|
|
|
1283 defaults to the current buffer if unspecified.
|
|
|
1284 @end defun
|
|
|
1285
|
|
442
|
1286 @node Detection of Textual Encoding, Big5 and Shift-JIS Functions, Encoding and Decoding Text, Coding Systems
|
|
428
|
1287 @subsection Detection of Textual Encoding
|
|
|
1288
|
|
|
1289 @defun coding-category-list
|
|
|
1290 This function returns a list of all recognized coding categories.
|
|
|
1291 @end defun
|
|
|
1292
|
|
|
1293 @defun set-coding-priority-list list
|
|
|
1294 This function changes the priority order of the coding categories.
|
|
|
1295 @var{list} should be a list of coding categories, in descending order of
|
|
|
1296 priority. Unspecified coding categories will be lower in priority than
|
|
|
1297 all specified ones, in the same relative order they were in previously.
|
|
|
1298 @end defun
|
|
|
1299
|
|
|
1300 @defun coding-priority-list
|
|
|
1301 This function returns a list of coding categories in descending order of
|
|
|
1302 priority.
|
|
|
1303 @end defun
|
|
|
1304
|
|
|
1305 @defun set-coding-category-system coding-category coding-system
|
|
|
1306 This function changes the coding system associated with a coding category.
|
|
|
1307 @end defun
|
|
|
1308
|
|
|
1309 @defun coding-category-system coding-category
|
|
|
1310 This function returns the coding system associated with a coding category.
|
|
|
1311 @end defun
|
|
|
1312
|
|
|
1313 @defun detect-coding-region start end &optional buffer
|
|
|
1314 This function detects coding system of the text in the region between
|
|
|
1315 @var{start} and @var{end}. Returned value is a list of possible coding
|
|
|
1316 systems ordered by priority. If only ASCII characters are found, it
|
|
|
1317 returns @code{autodetect} or one of its subsidiary coding systems
|
|
|
1318 according to a detected end-of-line type. Optional arg @var{buffer}
|
|
|
1319 defaults to the current buffer.
|
|
|
1320 @end defun
|
|
|
1321
|
|
442
|
1322 @node Big5 and Shift-JIS Functions, Predefined Coding Systems, Detection of Textual Encoding, Coding Systems
|
|
428
|
1323 @subsection Big5 and Shift-JIS Functions
|
|
|
1324
|
|
442
|
1325 These are special functions for working with the non-standard
|
|
428
|
1326 Shift-JIS and Big5 encodings.
|
|
|
1327
|
|
|
1328 @defun decode-shift-jis-char code
|
|
442
|
1329 This function decodes a JIS X 0208 character of Shift-JIS coding-system.
|
|
428
|
1330 @var{code} is the character code in Shift-JIS as a cons of type bytes.
|
|
|
1331 The corresponding character is returned.
|
|
|
1332 @end defun
|
|
|
1333
|
|
444
|
1334 @defun encode-shift-jis-char character
|
|
|
1335 This function encodes a JIS X 0208 character @var{character} to
|
|
|
1336 SHIFT-JIS coding-system. The corresponding character code in SHIFT-JIS
|
|
|
1337 is returned as a cons of two bytes.
|
|
428
|
1338 @end defun
|
|
|
1339
|
|
|
1340 @defun decode-big5-char code
|
|
|
1341 This function decodes a Big5 character @var{code} of BIG5 coding-system.
|
|
|
1342 @var{code} is the character code in BIG5. The corresponding character
|
|
|
1343 is returned.
|
|
|
1344 @end defun
|
|
|
1345
|
|
444
|
1346 @defun encode-big5-char character
|
|
|
1347 This function encodes the Big5 character @var{character} to BIG5
|
|
428
|
1348 coding-system. The corresponding character code in Big5 is returned.
|
|
|
1349 @end defun
|
|
|
1350
|
|
442
|
1351 @node Predefined Coding Systems, , Big5 and Shift-JIS Functions, Coding Systems
|
|
|
1352 @subsection Coding Systems Implemented
|
|
|
1353
|
|
|
1354 MULE initializes most of the commonly used coding systems at XEmacs's
|
|
|
1355 startup. A few others are initialized only when the relevant language
|
|
|
1356 environment is selected and support libraries are loaded. (NB: The
|
|
444
|
1357 following list is based on XEmacs 21.2.19, the development branch at the
|
|
442
|
1358 time of writing. The list may be somewhat different for other
|
|
|
1359 versions. Recent versions of GNU Emacs 20 implement a few more rare
|
|
|
1360 coding systems; work is being done to port these to XEmacs.)
|
|
|
1361
|
|
444
|
1362 Unfortunately, there is not a consistent naming convention for character
|
|
|
1363 sets, and for practical purposes coding systems often take their name
|
|
442
|
1364 from their principal character sets (ASCII, KOI8-R, Shift JIS). Others
|
|
444
|
1365 take their names from the coding system (ISO-2022-JP, EUC-KR), and a few
|
|
|
1366 from their non-text usages (internal, binary). To provide for this, and
|
|
442
|
1367 for the fact that many coding systems have several common names, an
|
|
|
1368 aliasing system is provided. Finally, some effort has been made to use
|
|
|
1369 names that are registered as MIME charsets (this is why the name
|
|
|
1370 'shift_jis contains that un-Lisp-y underscore).
|
|
|
1371
|
|
|
1372 There is a systematic naming convention regarding end-of-line (EOL)
|
|
|
1373 conventions for different systems. A coding system whose name ends in
|
|
|
1374 "-unix" forces the assumptions that lines are broken by newlines (0x0A).
|
|
|
1375 A coding system whose name ends in "-mac" forces the assumptions that
|
|
|
1376 lines are broken by ASCII CRs (0x0D). A coding system whose name ends
|
|
|
1377 in "-dos" forces the assumptions that lines are broken by CRLF sequences
|
|
|
1378 (0x0D 0x0A). These subsidiary coding systems are automatically derived
|
|
|
1379 from a base coding system. Use of the base coding system implies
|
|
|
1380 autodetection of the text file convention. (The fact that the -unix,
|
|
|
1381 -mac, and -dos are derived from a base system results in them showing up
|
|
|
1382 as "aliases" in `list-coding-systems'.) These subsidiaries have a
|
|
|
1383 consistent modeline indicator as well. "-dos" coding systems have ":T"
|
|
|
1384 appended to their modeline indicator, while "-mac" coding systems have
|
|
|
1385 ":t" appended (eg, "ISO8:t" for iso-2022-8-mac).
|
|
|
1386
|
|
|
1387 In the following table, each coding system is given with its mode line
|
|
|
1388 indicator in parentheses. Non-textual coding systems are listed first,
|
|
|
1389 followed by textual coding systems and their aliases. (The coding system
|
|
|
1390 subsidiary modeline indicators ":T" and ":t" will be omitted from the
|
|
|
1391 table of coding systems.)
|
|
|
1392
|
|
|
1393 ### SJT 1999-08-23 Maybe should order these by language? Definitely
|
|
|
1394 need language usage for the ISO-8859 family.
|
|
|
1395
|
|
|
1396 Note that although true coding system aliases have been implemented for
|
|
444
|
1397 XEmacs 21.2, the coding system initialization has not yet been converted
|
|
442
|
1398 as of 21.2.19. So coding systems described as aliases have the same
|
|
|
1399 properties as the aliased coding system, but will not be equal as Lisp
|
|
|
1400 objects.
|
|
|
1401
|
|
|
1402 @table @code
|
|
|
1403
|
|
|
1404 @item automatic-conversion
|
|
|
1405 @itemx undecided
|
|
|
1406 @itemx undecided-dos
|
|
|
1407 @itemx undecided-mac
|
|
|
1408 @itemx undecided-unix
|
|
|
1409
|
|
|
1410 Modeline indicator: @code{Auto}. A type @code{undecided} coding system.
|
|
|
1411 Attempts to determine an appropriate coding system from file contents or
|
|
|
1412 the environment.
|
|
|
1413
|
|
|
1414 @item raw-text
|
|
|
1415 @itemx no-conversion
|
|
|
1416 @itemx raw-text-dos
|
|
|
1417 @itemx raw-text-mac
|
|
|
1418 @itemx raw-text-unix
|
|
|
1419 @itemx no-conversion-dos
|
|
|
1420 @itemx no-conversion-mac
|
|
|
1421 @itemx no-conversion-unix
|
|
|
1422
|
|
|
1423 Modeline indicator: @code{Raw}. A type @code{no-conversion} coding system,
|
|
|
1424 which converts only line-break-codes. An implementation quirk means
|
|
|
1425 that this coding system is also used for ISO8859-1.
|
|
|
1426
|
|
|
1427 @item binary
|
|
|
1428 Modeline indicator: @code{Binary}. A type @code{no-conversion} coding
|
|
|
1429 system which does no character coding or EOL conversions. An alias for
|
|
|
1430 @code{raw-text-unix}.
|
|
|
1431
|
|
|
1432 @item alternativnyj
|
|
|
1433 @itemx alternativnyj-dos
|
|
|
1434 @itemx alternativnyj-mac
|
|
|
1435 @itemx alternativnyj-unix
|
|
|
1436
|
|
|
1437 Modeline indicator: @code{Cy.Alt}. A type @code{ccl} coding system used for
|
|
|
1438 Alternativnyj, an encoding of the Cyrillic alphabet.
|
|
|
1439
|
|
|
1440 @item big5
|
|
|
1441 @itemx big5-dos
|
|
|
1442 @itemx big5-mac
|
|
|
1443 @itemx big5-unix
|
|
|
1444
|
|
|
1445 Modeline indicator: @code{Zh/Big5}. A type @code{big5} coding system used for
|
|
|
1446 BIG5, the most common encoding of traditional Chinese as used in Taiwan.
|
|
|
1447
|
|
|
1448 @item cn-gb-2312
|
|
|
1449 @itemx cn-gb-2312-dos
|
|
|
1450 @itemx cn-gb-2312-mac
|
|
|
1451 @itemx cn-gb-2312-unix
|
|
|
1452
|
|
|
1453 Modeline indicator: @code{Zh-GB/EUC}. A type @code{iso2022} coding system used
|
|
|
1454 for simplified Chinese (as used in the People's Republic of China), with
|
|
|
1455 the @code{ascii} (G0), @code{chinese-gb2312} (G1), and @code{sisheng}
|
|
|
1456 (G2) character sets initially designated. Chinese EUC (Extended Unix
|
|
|
1457 Code).
|
|
|
1458
|
|
|
1459 @item ctext-hebrew
|
|
|
1460 @itemx ctext-hebrew-dos
|
|
|
1461 @itemx ctext-hebrew-mac
|
|
|
1462 @itemx ctext-hebrew-unix
|
|
|
1463
|
|
|
1464 Modeline indicator: @code{CText/Hbrw}. A type @code{iso2022} coding system
|
|
|
1465 with the @code{ascii} (G0) and @code{hebrew-iso8859-8} (G1) character
|
|
|
1466 sets initially designated for Hebrew.
|
|
|
1467
|
|
|
1468 @item ctext
|
|
|
1469 @itemx ctext-dos
|
|
|
1470 @itemx ctext-mac
|
|
|
1471 @itemx ctext-unix
|
|
|
1472
|
|
|
1473 Modeline indicator: @code{CText}. A type @code{iso2022} 8-bit coding system
|
|
|
1474 with the @code{ascii} (G0) and @code{latin-iso8859-1} (G1) character
|
|
|
1475 sets initially designated. X11 Compound Text Encoding. Often
|
|
|
1476 mistakenly recognized instead of EUC encodings; usual cause is
|
|
|
1477 inappropriate setting of @code{coding-priority-list}.
|
|
|
1478
|
|
|
1479 @item escape-quoted
|
|
|
1480
|
|
|
1481 Modeline indicator: @code{ESC/Quot}. A type @code{iso2022} 8-bit coding
|
|
|
1482 system with the @code{ascii} (G0) and @code{latin-iso8859-1} (G1)
|
|
|
1483 character sets initially designated and escape quoting. Unix EOL
|
|
|
1484 conversion (ie, no conversion). It is used for .ELC files.
|
|
|
1485
|
|
|
1486 @item euc-jp
|
|
|
1487 @itemx euc-jp-dos
|
|
|
1488 @itemx euc-jp-mac
|
|
|
1489 @itemx euc-jp-unix
|
|
|
1490
|
|
|
1491 Modeline indicator: @code{Ja/EUC}. A type @code{iso2022} 8-bit coding system
|
|
|
1492 with @code{ascii} (G0), @code{japanese-jisx0208} (G1),
|
|
|
1493 @code{katakana-jisx0201} (G2), and @code{japanese-jisx0212} (G3)
|
|
|
1494 initially designated. Japanese EUC (Extended Unix Code).
|
|
|
1495
|
|
|
1496 @item euc-kr
|
|
|
1497 @itemx euc-kr-dos
|
|
|
1498 @itemx euc-kr-mac
|
|
|
1499 @itemx euc-kr-unix
|
|
|
1500
|
|
|
1501 Modeline indicator: @code{ko/EUC}. A type @code{iso2022} 8-bit coding system
|
|
|
1502 with @code{ascii} (G0) and @code{korean-ksc5601} (G1) initially
|
|
|
1503 designated. Korean EUC (Extended Unix Code).
|
|
|
1504
|
|
|
1505 @item hz-gb-2312
|
|
|
1506 Modeline indicator: @code{Zh-GB/Hz}. A type @code{no-conversion} coding
|
|
|
1507 system with Unix EOL convention (ie, no conversion) using
|
|
|
1508 post-read-decode and pre-write-encode functions to translate the Hz/ZW
|
|
|
1509 coding system used for Chinese.
|
|
|
1510
|
|
|
1511 @item iso-2022-7bit
|
|
|
1512 @itemx iso-2022-7bit-unix
|
|
|
1513 @itemx iso-2022-7bit-dos
|
|
|
1514 @itemx iso-2022-7bit-mac
|
|
|
1515 @itemx iso-2022-7
|
|
|
1516
|
|
|
1517 Modeline indicator: @code{ISO7}. A type @code{iso2022} 7-bit coding system
|
|
|
1518 with @code{ascii} (G0) initially designated. Other character sets must
|
|
|
1519 be explicitly designated to be used.
|
|
|
1520
|
|
|
1521 @item iso-2022-7bit-ss2
|
|
|
1522 @itemx iso-2022-7bit-ss2-dos
|
|
|
1523 @itemx iso-2022-7bit-ss2-mac
|
|
|
1524 @itemx iso-2022-7bit-ss2-unix
|
|
|
1525
|
|
|
1526 Modeline indicator: @code{ISO7/SS}. A type @code{iso2022} 7-bit coding system
|
|
|
1527 with @code{ascii} (G0) initially designated. Other character sets must
|
|
|
1528 be explicitly designated to be used. SS2 is used to invoke a
|
|
|
1529 96-charset, one character at a time.
|
|
|
1530
|
|
|
1531 @item iso-2022-8
|
|
|
1532 @itemx iso-2022-8-dos
|
|
|
1533 @itemx iso-2022-8-mac
|
|
|
1534 @itemx iso-2022-8-unix
|
|
|
1535
|
|
|
1536 Modeline indicator: @code{ISO8}. A type @code{iso2022} 8-bit coding system
|
|
|
1537 with @code{ascii} (G0) and @code{latin-iso8859-1} (G1) initially
|
|
|
1538 designated. Other character sets must be explicitly designated to be
|
|
|
1539 used. No single-shift or locking-shift.
|
|
|
1540
|
|
|
1541 @item iso-2022-8bit-ss2
|
|
|
1542 @itemx iso-2022-8bit-ss2-dos
|
|
|
1543 @itemx iso-2022-8bit-ss2-mac
|
|
|
1544 @itemx iso-2022-8bit-ss2-unix
|
|
|
1545
|
|
|
1546 Modeline indicator: @code{ISO8/SS}. A type @code{iso2022} 8-bit coding system
|
|
|
1547 with @code{ascii} (G0) and @code{latin-iso8859-1} (G1) initially
|
|
|
1548 designated. Other character sets must be explicitly designated to be
|
|
|
1549 used. SS2 is used to invoke a 96-charset, one character at a time.
|
|
|
1550
|
|
|
1551 @item iso-2022-int-1
|
|
|
1552 @itemx iso-2022-int-1-dos
|
|
|
1553 @itemx iso-2022-int-1-mac
|
|
|
1554 @itemx iso-2022-int-1-unix
|
|
|
1555
|
|
|
1556 Modeline indicator: @code{INT-1}. A type @code{iso2022} 7-bit coding system
|
|
|
1557 with @code{ascii} (G0) and @code{korean-ksc5601} (G1) initially
|
|
|
1558 designated. ISO-2022-INT-1.
|
|
|
1559
|
|
|
1560 @item iso-2022-jp-1978-irv
|
|
|
1561 @itemx iso-2022-jp-1978-irv-dos
|
|
|
1562 @itemx iso-2022-jp-1978-irv-mac
|
|
|
1563 @itemx iso-2022-jp-1978-irv-unix
|
|
|
1564
|
|
|
1565 Modeline indicator: @code{Ja-78/7bit}. A type @code{iso2022} 7-bit coding
|
|
|
1566 system. For compatibility with old Japanese terminals; if you need to
|
|
|
1567 know, look at the source.
|
|
|
1568
|
|
|
1569 @item iso-2022-jp
|
|
|
1570 @itemx iso-2022-jp-2 (ISO7/SS)
|
|
|
1571 @itemx iso-2022-jp-dos
|
|
|
1572 @itemx iso-2022-jp-mac
|
|
|
1573 @itemx iso-2022-jp-unix
|
|
|
1574 @itemx iso-2022-jp-2-dos
|
|
|
1575 @itemx iso-2022-jp-2-mac
|
|
|
1576 @itemx iso-2022-jp-2-unix
|
|
|
1577
|
|
|
1578 Modeline indicator: @code{MULE/7bit}. A type @code{iso2022} 7-bit coding
|
|
|
1579 system with @code{ascii} (G0) initially designated, and complex
|
|
|
1580 specifications to insure backward compatibility with old Japanese
|
|
|
1581 systems. Used for communication with mail and news in Japan. The "-2"
|
|
|
1582 versions also use SS2 to invoke a 96-charset one character at a time.
|
|
|
1583
|
|
|
1584 @item iso-2022-kr
|
|
|
1585 Modeline indicator: @code{Ko/7bit} A type @code{iso2022} 7-bit coding
|
|
|
1586 system with @code{ascii} (G0) and @code{korean-ksc5601} (G1) initially
|
|
|
1587 designated. Used for e-mail in Korea.
|
|
|
1588
|
|
|
1589 @item iso-2022-lock
|
|
|
1590 @itemx iso-2022-lock-dos
|
|
|
1591 @itemx iso-2022-lock-mac
|
|
|
1592 @itemx iso-2022-lock-unix
|
|
|
1593
|
|
|
1594 Modeline indicator: @code{ISO7/Lock}. A type @code{iso2022} 7-bit coding
|
|
|
1595 system with @code{ascii} (G0) initially designated, using Locking-Shift
|
|
|
1596 to invoke a 96-charset.
|
|
|
1597
|
|
|
1598 @item iso-8859-1
|
|
|
1599 @itemx iso-8859-1-dos
|
|
|
1600 @itemx iso-8859-1-mac
|
|
|
1601 @itemx iso-8859-1-unix
|
|
|
1602
|
|
|
1603 Due to implementation, this is not a type @code{iso2022} coding system,
|
|
|
1604 but rather an alias for the @code{raw-text} coding system.
|
|
|
1605
|
|
|
1606 @item iso-8859-2
|
|
|
1607 @itemx iso-8859-2-dos
|
|
|
1608 @itemx iso-8859-2-mac
|
|
|
1609 @itemx iso-8859-2-unix
|
|
|
1610
|
|
|
1611 Modeline indicator: @code{MIME/Ltn-2}. A type @code{iso2022} coding
|
|
|
1612 system with @code{ascii} (G0) and @code{latin-iso8859-2} (G1) initially
|
|
|
1613 invoked.
|
|
|
1614
|
|
|
1615 @item iso-8859-3
|
|
|
1616 @itemx iso-8859-3-dos
|
|
|
1617 @itemx iso-8859-3-mac
|
|
|
1618 @itemx iso-8859-3-unix
|
|
|
1619
|
|
|
1620 Modeline indicator: @code{MIME/Ltn-3}. A type @code{iso2022} coding system
|
|
|
1621 with @code{ascii} (G0) and @code{latin-iso8859-3} (G1) initially
|
|
|
1622 invoked.
|
|
|
1623
|
|
|
1624 @item iso-8859-4
|
|
|
1625 @itemx iso-8859-4-dos
|
|
|
1626 @itemx iso-8859-4-mac
|
|
|
1627 @itemx iso-8859-4-unix
|
|
|
1628
|
|
|
1629 Modeline indicator: @code{MIME/Ltn-4}. A type @code{iso2022} coding system
|
|
|
1630 with @code{ascii} (G0) and @code{latin-iso8859-4} (G1) initially
|
|
|
1631 invoked.
|
|
|
1632
|
|
|
1633 @item iso-8859-5
|
|
|
1634 @itemx iso-8859-5-dos
|
|
|
1635 @itemx iso-8859-5-mac
|
|
|
1636 @itemx iso-8859-5-unix
|
|
|
1637
|
|
|
1638 Modeline indicator: @code{ISO8/Cyr}. A type @code{iso2022} coding system with
|
|
|
1639 @code{ascii} (G0) and @code{cyrillic-iso8859-5} (G1) initially invoked.
|
|
|
1640
|
|
|
1641 @item iso-8859-7
|
|
|
1642 @itemx iso-8859-7-dos
|
|
|
1643 @itemx iso-8859-7-mac
|
|
|
1644 @itemx iso-8859-7-unix
|
|
|
1645
|
|
|
1646 Modeline indicator: @code{Grk}. A type @code{iso2022} coding system with
|
|
|
1647 @code{ascii} (G0) and @code{greek-iso8859-7} (G1) initially invoked.
|
|
|
1648
|
|
|
1649 @item iso-8859-8
|
|
|
1650 @itemx iso-8859-8-dos
|
|
|
1651 @itemx iso-8859-8-mac
|
|
|
1652 @itemx iso-8859-8-unix
|
|
|
1653
|
|
|
1654 Modeline indicator: @code{MIME/Hbrw}. A type @code{iso2022} coding system with
|
|
|
1655 @code{ascii} (G0) and @code{hebrew-iso8859-8} (G1) initially invoked.
|
|
|
1656
|
|
|
1657 @item iso-8859-9
|
|
|
1658 @itemx iso-8859-9-dos
|
|
|
1659 @itemx iso-8859-9-mac
|
|
|
1660 @itemx iso-8859-9-unix
|
|
|
1661
|
|
|
1662 Modeline indicator: @code{MIME/Ltn-5}. A type @code{iso2022} coding system
|
|
|
1663 with @code{ascii} (G0) and @code{latin-iso8859-9} (G1) initially
|
|
|
1664 invoked.
|
|
|
1665
|
|
|
1666 @item koi8-r
|
|
|
1667 @itemx koi8-r-dos
|
|
|
1668 @itemx koi8-r-mac
|
|
|
1669 @itemx koi8-r-unix
|
|
|
1670
|
|
|
1671 Modeline indicator: @code{KOI8}. A type @code{ccl} coding-system used for
|
|
|
1672 KOI8-R, an encoding of the Cyrillic alphabet.
|
|
|
1673
|
|
|
1674 @item shift_jis
|
|
|
1675 @itemx shift_jis-dos
|
|
|
1676 @itemx shift_jis-mac
|
|
|
1677 @itemx shift_jis-unix
|
|
|
1678
|
|
|
1679 Modeline indicator: @code{Ja/SJIS}. A type @code{shift-jis} coding-system
|
|
|
1680 implementing the Shift-JIS encoding for Japanese. The underscore is to
|
|
|
1681 conform to the MIME charset implementing this encoding.
|
|
|
1682
|
|
|
1683 @item tis-620
|
|
|
1684 @itemx tis-620-dos
|
|
|
1685 @itemx tis-620-mac
|
|
|
1686 @itemx tis-620-unix
|
|
|
1687
|
|
|
1688 Modeline indicator: @code{TIS620}. A type @code{ccl} encoding for Thai. The
|
|
|
1689 external encoding is defined by TIS620, the internal encoding is
|
|
|
1690 peculiar to MULE, and called @code{thai-xtis}.
|
|
|
1691
|
|
|
1692 @item viqr
|
|
|
1693
|
|
|
1694 Modeline indicator: @code{VIQR}. A type @code{no-conversion} coding
|
|
|
1695 system with Unix EOL convention (ie, no conversion) using
|
|
|
1696 post-read-decode and pre-write-encode functions to translate the VIQR
|
|
|
1697 coding system for Vietnamese.
|
|
|
1698
|
|
|
1699 @item viscii
|
|
|
1700 @itemx viscii-dos
|
|
|
1701 @itemx viscii-mac
|
|
|
1702 @itemx viscii-unix
|
|
|
1703
|
|
|
1704 Modeline indicator: @code{VISCII}. A type @code{ccl} coding-system used
|
|
|
1705 for VISCII 1.1 for Vietnamese. Differs slightly from VSCII; VISCII is
|
|
|
1706 given priority by XEmacs.
|
|
|
1707
|
|
|
1708 @item vscii
|
|
|
1709 @itemx vscii-dos
|
|
|
1710 @itemx vscii-mac
|
|
|
1711 @itemx vscii-unix
|
|
|
1712
|
|
|
1713 Modeline indicator: @code{VSCII}. A type @code{ccl} coding-system used
|
|
|
1714 for VSCII 1.1 for Vietnamese. Differs slightly from VISCII, which is
|
|
|
1715 given priority by XEmacs. Use
|
|
|
1716 @code{(prefer-coding-system 'vietnamese-vscii)} to give priority to VSCII.
|
|
|
1717
|
|
|
1718 @end table
|
|
|
1719
|
|
428
|
1720 @node CCL, Category Tables, Coding Systems, MULE
|
|
|
1721 @section CCL
|
|
|
1722
|
|
442
|
1723 CCL (Code Conversion Language) is a simple structured programming
|
|
428
|
1724 language designed for character coding conversions. A CCL program is
|
|
|
1725 compiled to CCL code (represented by a vector of integers) and executed
|
|
|
1726 by the CCL interpreter embedded in Emacs. The CCL interpreter
|
|
|
1727 implements a virtual machine with 8 registers called @code{r0}, ...,
|
|
|
1728 @code{r7}, a number of control structures, and some I/O operators. Take
|
|
|
1729 care when using registers @code{r0} (used in implicit @dfn{set}
|
|
|
1730 statements) and especially @code{r7} (used internally by several
|
|
444
|
1731 statements and operations, especially for multiple return values and I/O
|
|
428
|
1732 operations).
|
|
|
1733
|
|
442
|
1734 CCL is used for code conversion during process I/O and file I/O for
|
|
428
|
1735 non-ISO2022 coding systems. (It is the only way for a user to specify a
|
|
|
1736 code conversion function.) It is also used for calculating the code
|
|
|
1737 point of an X11 font from a character code. However, since CCL is
|
|
|
1738 designed as a powerful programming language, it can be used for more
|
|
|
1739 generic calculation where efficiency is demanded. A combination of
|
|
|
1740 three or more arithmetic operations can be calculated faster by CCL than
|
|
|
1741 by Emacs Lisp.
|
|
|
1742
|
|
442
|
1743 @strong{Warning:} The code in @file{src/mule-ccl.c} and
|
|
428
|
1744 @file{$packages/lisp/mule-base/mule-ccl.el} is the definitive
|
|
|
1745 description of CCL's semantics. The previous version of this section
|
|
|
1746 contained several typos and obsolete names left from earlier versions of
|
|
|
1747 MULE, and many may remain. (I am not an experienced CCL programmer; the
|
|
|
1748 few who know CCL well find writing English painful.)
|
|
|
1749
|
|
442
|
1750 A CCL program transforms an input data stream into an output data
|
|
428
|
1751 stream. The input stream, held in a buffer of constant bytes, is left
|
|
|
1752 unchanged. The buffer may be filled by an external input operation,
|
|
|
1753 taken from an Emacs buffer, or taken from a Lisp string. The output
|
|
|
1754 buffer is a dynamic array of bytes, which can be written by an external
|
|
|
1755 output operation, inserted into an Emacs buffer, or returned as a Lisp
|
|
|
1756 string.
|
|
|
1757
|
|
442
|
1758 A CCL program is a (Lisp) list containing two or three members. The
|
|
428
|
1759 first member is the @dfn{buffer magnification}, which indicates the
|
|
|
1760 required minimum size of the output buffer as a multiple of the input
|
|
|
1761 buffer. It is followed by the @dfn{main block} which executes while
|
|
|
1762 there is input remaining, and an optional @dfn{EOF block} which is
|
|
|
1763 executed when the input is exhausted. Both the main block and the EOF
|
|
|
1764 block are CCL blocks.
|
|
|
1765
|
|
442
|
1766 A @dfn{CCL block} is either a CCL statement or list of CCL statements.
|
|
444
|
1767 A @dfn{CCL statement} is either a @dfn{set statement} (either an integer
|
|
428
|
1768 or an @dfn{assignment}, which is a list of a register to receive the
|
|
444
|
1769 assignment, an assignment operator, and an expression) or a @dfn{control
|
|
428
|
1770 statement} (a list starting with a keyword, whose allowable syntax
|
|
|
1771 depends on the keyword).
|
|
|
1772
|
|
|
1773 @menu
|
|
|
1774 * CCL Syntax:: CCL program syntax in BNF notation.
|
|
|
1775 * CCL Statements:: Semantics of CCL statements.
|
|
|
1776 * CCL Expressions:: Operators and expressions in CCL.
|
|
|
1777 * Calling CCL:: Running CCL programs.
|
|
2640
|
1778 * CCL Example:: A trivial program to transform the Web's URL encoding.
|
|
428
|
1779 @end menu
|
|
|
1780
|
|
442
|
1781 @node CCL Syntax, CCL Statements, , CCL
|
|
428
|
1782 @comment Node, Next, Previous, Up
|
|
|
1783 @subsection CCL Syntax
|
|
|
1784
|
|
442
|
1785 The full syntax of a CCL program in BNF notation:
|
|
428
|
1786
|
|
|
1787 @format
|
|
|
1788 CCL_PROGRAM :=
|
|
|
1789 (BUFFER_MAGNIFICATION
|
|
|
1790 CCL_MAIN_BLOCK
|
|
|
1791 [ CCL_EOF_BLOCK ])
|
|
|
1792
|
|
|
1793 BUFFER_MAGNIFICATION := integer
|
|
|
1794 CCL_MAIN_BLOCK := CCL_BLOCK
|
|
|
1795 CCL_EOF_BLOCK := CCL_BLOCK
|
|
|
1796
|
|
|
1797 CCL_BLOCK :=
|
|
|
1798 STATEMENT | (STATEMENT [STATEMENT ...])
|
|
|
1799 STATEMENT :=
|
|
2367
|
1800 SET | IF | BRANCH | LOOP | REPEAT | BREAK | READ | WRITE | CALL
|
|
|
1801 | TRANSLATE | MAP | END
|
|
428
|
1802
|
|
|
1803 SET :=
|
|
|
1804 (REG = EXPRESSION)
|
|
|
1805 | (REG ASSIGNMENT_OPERATOR EXPRESSION)
|
|
2367
|
1806 | INT-OR-CHAR
|
|
428
|
1807
|
|
|
1808 EXPRESSION := ARG | (EXPRESSION OPERATOR ARG)
|
|
|
1809
|
|
|
1810 IF := (if EXPRESSION CCL_BLOCK [CCL_BLOCK])
|
|
|
1811 BRANCH := (branch EXPRESSION CCL_BLOCK [CCL_BLOCK ...])
|
|
|
1812 LOOP := (loop STATEMENT [STATEMENT ...])
|
|
|
1813 BREAK := (break)
|
|
|
1814 REPEAT :=
|
|
|
1815 (repeat)
|
|
2367
|
1816 | (write-repeat [REG | INT-OR-CHAR | string])
|
|
|
1817 | (write-read-repeat REG [INT-OR-CHAR | ARRAY])
|
|
428
|
1818 READ :=
|
|
|
1819 (read REG ...)
|
|
2367
|
1820 | (read-if (REG OPERATOR ARG) CCL_BLOCK [CCL_BLOCK])
|
|
428
|
1821 | (read-branch REG CCL_BLOCK [CCL_BLOCK ...])
|
|
|
1822 WRITE :=
|
|
|
1823 (write REG ...)
|
|
|
1824 | (write EXPRESSION)
|
|
2367
|
1825 | (write INT-OR-CHAR) | (write string) | (write REG ARRAY)
|
|
428
|
1826 | string
|
|
|
1827 CALL := (call ccl-program-name)
|
|
3439
|
1828
|
|
|
1829
|
|
|
1830 TRANSLATE := ;; Not implemented under XEmacs, except mule-to-unicode and
|
|
|
1831 ;; unicode-to-mule.
|
|
|
1832 (translate-character REG(table) REG(charset) REG(codepoint))
|
|
|
1833 | (translate-character SYMBOL REG(charset) REG(codepoint))
|
|
|
1834 | (mule-to-unicode REG(charset) REG(codepoint))
|
|
|
1835 | (unicode-to-mule REG(unicode,code) REG(CHARSET))
|
|
|
1836
|
|
428
|
1837 END := (end)
|
|
|
1838
|
|
|
1839 REG := r0 | r1 | r2 | r3 | r4 | r5 | r6 | r7
|
|
2367
|
1840 ARG := REG | INT-OR-CHAR
|
|
428
|
1841 OPERATOR :=
|
|
|
1842 + | - | * | / | % | & | '|' | ^ | << | >> | <8 | >8 | //
|
|
|
1843 | < | > | == | <= | >= | != | de-sjis | en-sjis
|
|
|
1844 ASSIGNMENT_OPERATOR :=
|
|
|
1845 += | -= | *= | /= | %= | &= | '|=' | ^= | <<= | >>=
|
|
2367
|
1846 ARRAY := '[' INT-OR-CHAR ... ']'
|
|
|
1847 INT-OR-CHAR := integer | character
|
|
|
1848
|
|
428
|
1849 @end format
|
|
|
1850
|
|
|
1851 @node CCL Statements, CCL Expressions, CCL Syntax, CCL
|
|
|
1852 @comment Node, Next, Previous, Up
|
|
|
1853 @subsection CCL Statements
|
|
|
1854
|
|
442
|
1855 The Emacs Code Conversion Language provides the following statement
|
|
428
|
1856 types: @dfn{set}, @dfn{if}, @dfn{branch}, @dfn{loop}, @dfn{repeat},
|
|
3439
|
1857 @dfn{break}, @dfn{read}, @dfn{write}, @dfn{call}, @dfn{translate} and
|
|
|
1858 @dfn{end}.
|
|
428
|
1859
|
|
|
1860 @heading Set statement:
|
|
|
1861
|
|
442
|
1862 The @dfn{set} statement has three variants with the syntaxes
|
|
428
|
1863 @samp{(@var{reg} = @var{expression})},
|
|
|
1864 @samp{(@var{reg} @var{assignment_operator} @var{expression})}, and
|
|
|
1865 @samp{@var{integer}}. The assignment operator variation of the
|
|
|
1866 @dfn{set} statement works the same way as the corresponding C expression
|
|
|
1867 statement does. The assignment operators are @code{+=}, @code{-=},
|
|
|
1868 @code{*=}, @code{/=}, @code{%=}, @code{&=}, @code{|=}, @code{^=},
|
|
|
1869 @code{<<=}, and @code{>>=}, and they have the same meanings as in C. A
|
|
|
1870 "naked integer" @var{integer} is equivalent to a @var{set} statement of
|
|
|
1871 the form @code{(r0 = @var{integer})}.
|
|
|
1872
|
|
|
1873 @heading I/O statements:
|
|
|
1874
|
|
442
|
1875 The @dfn{read} statement takes one or more registers as arguments. It
|
|
444
|
1876 reads one byte (a C char) from the input into each register in turn.
|
|
428
|
1877
|
|
442
|
1878 The @dfn{write} takes several forms. In the form @samp{(write @var{reg}
|
|
428
|
1879 ...)} it takes one or more registers as arguments and writes each in
|
|
|
1880 turn to the output. The integer in a register (interpreted as an
|
|
2367
|
1881 Ichar) is encoded to multibyte form (ie, Ibytes) and written to the
|
|
428
|
1882 current output buffer. If it is less than 256, it is written as is.
|
|
|
1883 The forms @samp{(write @var{expression})} and @samp{(write
|
|
|
1884 @var{integer})} are treated analogously. The form @samp{(write
|
|
|
1885 @var{string})} writes the constant string to the output. A
|
|
|
1886 "naked string" @samp{@var{string}} is equivalent to the statement @samp{(write
|
|
|
1887 @var{string})}. The form @samp{(write @var{reg} @var{array})} writes
|
|
|
1888 the @var{reg}th element of the @var{array} to the output.
|
|
|
1889
|
|
|
1890 @heading Conditional statements:
|
|
|
1891
|
|
442
|
1892 The @dfn{if} statement takes an @var{expression}, a @var{CCL block}, and
|
|
428
|
1893 an optional @var{second CCL block} as arguments. If the
|
|
|
1894 @var{expression} evaluates to non-zero, the first @var{CCL block} is
|
|
|
1895 executed. Otherwise, if there is a @var{second CCL block}, it is
|
|
|
1896 executed.
|
|
|
1897
|
|
442
|
1898 The @dfn{read-if} variant of the @dfn{if} statement takes an
|
|
428
|
1899 @var{expression}, a @var{CCL block}, and an optional @var{second CCL
|
|
|
1900 block} as arguments. The @var{expression} must have the form
|
|
|
1901 @code{(@var{reg} @var{operator} @var{operand})} (where @var{operand} is
|
|
|
1902 a register or an integer). The @code{read-if} statement first reads
|
|
|
1903 from the input into the first register operand in the @var{expression},
|
|
|
1904 then conditionally executes a CCL block just as the @code{if} statement
|
|
|
1905 does.
|
|
|
1906
|
|
442
|
1907 The @dfn{branch} statement takes an @var{expression} and one or more CCL
|
|
428
|
1908 blocks as arguments. The CCL blocks are treated as a zero-indexed
|
|
|
1909 array, and the @code{branch} statement uses the @var{expression} as the
|
|
|
1910 index of the CCL block to execute. Null CCL blocks may be used as
|
|
|
1911 no-ops, continuing execution with the statement following the
|
|
|
1912 @code{branch} statement in the containing CCL block. Out-of-range
|
|
444
|
1913 values for the @var{expression} are also treated as no-ops.
|
|
428
|
1914
|
|
442
|
1915 The @dfn{read-branch} variant of the @dfn{branch} statement takes an
|
|
428
|
1916 @var{register}, a @var{CCL block}, and an optional @var{second CCL
|
|
|
1917 block} as arguments. The @code{read-branch} statement first reads from
|
|
|
1918 the input into the @var{register}, then conditionally executes a CCL
|
|
|
1919 block just as the @code{branch} statement does.
|
|
|
1920
|
|
|
1921 @heading Loop control statements:
|
|
|
1922
|
|
442
|
1923 The @dfn{loop} statement creates a block with an implied jump from the
|
|
444
|
1924 end of the block back to its head. The loop is exited on a @code{break}
|
|
428
|
1925 statement, and continued without executing the tail by a @code{repeat}
|
|
|
1926 statement.
|
|
|
1927
|
|
442
|
1928 The @dfn{break} statement, written @samp{(break)}, terminates the
|
|
428
|
1929 current loop and continues with the next statement in the current
|
|
444
|
1930 block.
|
|
428
|
1931
|
|
442
|
1932 The @dfn{repeat} statement has three variants, @code{repeat},
|
|
428
|
1933 @code{write-repeat}, and @code{write-read-repeat}. Each continues the
|
|
|
1934 current loop from its head, possibly after performing I/O.
|
|
|
1935 @code{repeat} takes no arguments and does no I/O before jumping.
|
|
444
|
1936 @code{write-repeat} takes a single argument (a register, an
|
|
428
|
1937 integer, or a string), writes it to the output, then jumps.
|
|
|
1938 @code{write-read-repeat} takes one or two arguments. The first must
|
|
|
1939 be a register. The second may be an integer or an array; if absent, it
|
|
|
1940 is implicitly set to the first (register) argument.
|
|
|
1941 @code{write-read-repeat} writes its second argument to the output, then
|
|
|
1942 reads from the input into the register, and finally jumps. See the
|
|
|
1943 @code{write} and @code{read} statements for the semantics of the I/O
|
|
|
1944 operations for each type of argument.
|
|
|
1945
|
|
3439
|
1946 @heading Other statements:
|
|
428
|
1947
|
|
442
|
1948 The @dfn{call} statement, written @samp{(call @var{ccl-program-name})},
|
|
428
|
1949 executes a CCL program as a subroutine. It does not return a value to
|
|
|
1950 the caller, but can modify the register status.
|
|
|
1951
|
|
3439
|
1952 The @dfn{mule-to-unicode} statement translates an XEmacs character into a
|
|
|
1953 UCS code point, using U+FFFD REPLACEMENT CHARACTER if the given XEmacs
|
|
|
1954 character has no known corresponding code point. It takes two
|
|
|
1955 arguments; the first is a register in which is stored the character set
|
|
|
1956 ID of the character to be translated, and into which the UCS code is
|
|
|
1957 stored. The second is a register which stores the XEmacs code of the
|
|
|
1958 character in question; if it is from a multidimensional character set,
|
|
|
1959 like most of the East Asian national sets, it's stored as @samp{((c1 <<
|
|
|
1960 8) & c2)}, where @samp{c1} is the first code, and @samp{c2} the second.
|
|
|
1961 (That is, as a single integer, the high-order eight bits of which encode
|
|
|
1962 the first position code, and the low order bits of which encode the
|
|
|
1963 second.)
|
|
|
1964
|
|
|
1965 The @dfn{unicode-to-mule} statement translates a Unicode code point
|
|
|
1966 (an integer) into an XEmacs character. Its first argument is a register
|
|
|
1967 containing the UCS code point; the code for the correspond character
|
|
|
1968 will be written into this register, in the same format as for
|
|
|
1969 @samp{mule-to-unicode} The second argument is a register into which will
|
|
|
1970 be written the character set ID of the converted character.
|
|
|
1971
|
|
442
|
1972 The @dfn{end} statement, written @samp{(end)}, terminates the CCL
|
|
428
|
1973 program successfully, and returns to caller (which may be a CCL
|
|
|
1974 program). It does not alter the status of the registers.
|
|
|
1975
|
|
|
1976 @node CCL Expressions, Calling CCL, CCL Statements, CCL
|
|
|
1977 @comment Node, Next, Previous, Up
|
|
|
1978 @subsection CCL Expressions
|
|
|
1979
|
|
442
|
1980 CCL, unlike Lisp, uses infix expressions. The simplest CCL expressions
|
|
428
|
1981 consist of a single @var{operand}, either a register (one of @code{r0},
|
|
|
1982 ..., @code{r0}) or an integer. Complex expressions are lists of the
|
|
|
1983 form @code{( @var{expression} @var{operator} @var{operand} )}. Unlike
|
|
|
1984 C, assignments are not expressions.
|
|
|
1985
|
|
442
|
1986 In the following table, @var{X} is the target resister for a @dfn{set}.
|
|
428
|
1987 In subexpressions, this is implicitly @code{r7}. This means that
|
|
|
1988 @code{>8}, @code{//}, @code{de-sjis}, and @code{en-sjis} cannot be used
|
|
|
1989 freely in subexpressions, since they return parts of their values in
|
|
|
1990 @code{r7}. @var{Y} may be an expression, register, or integer, while
|
|
|
1991 @var{Z} must be a register or an integer.
|
|
|
1992
|
|
|
1993 @multitable @columnfractions .22 .14 .09 .55
|
|
|
1994 @item Name @tab Operator @tab Code @tab C-like Description
|
|
|
1995 @item CCL_PLUS @tab @code{+} @tab 0x00 @tab X = Y + Z
|
|
|
1996 @item CCL_MINUS @tab @code{-} @tab 0x01 @tab X = Y - Z
|
|
|
1997 @item CCL_MUL @tab @code{*} @tab 0x02 @tab X = Y * Z
|
|
|
1998 @item CCL_DIV @tab @code{/} @tab 0x03 @tab X = Y / Z
|
|
|
1999 @item CCL_MOD @tab @code{%} @tab 0x04 @tab X = Y % Z
|
|
|
2000 @item CCL_AND @tab @code{&} @tab 0x05 @tab X = Y & Z
|
|
|
2001 @item CCL_OR @tab @code{|} @tab 0x06 @tab X = Y | Z
|
|
|
2002 @item CCL_XOR @tab @code{^} @tab 0x07 @tab X = Y ^ Z
|
|
|
2003 @item CCL_LSH @tab @code{<<} @tab 0x08 @tab X = Y << Z
|
|
|
2004 @item CCL_RSH @tab @code{>>} @tab 0x09 @tab X = Y >> Z
|
|
|
2005 @item CCL_LSH8 @tab @code{<8} @tab 0x0A @tab X = (Y << 8) | Z
|
|
|
2006 @item CCL_RSH8 @tab @code{>8} @tab 0x0B @tab X = Y >> 8, r[7] = Y & 0xFF
|
|
|
2007 @item CCL_DIVMOD @tab @code{//} @tab 0x0C @tab X = Y / Z, r[7] = Y % Z
|
|
|
2008 @item CCL_LS @tab @code{<} @tab 0x10 @tab X = (X < Y)
|
|
|
2009 @item CCL_GT @tab @code{>} @tab 0x11 @tab X = (X > Y)
|
|
|
2010 @item CCL_EQ @tab @code{==} @tab 0x12 @tab X = (X == Y)
|
|
|
2011 @item CCL_LE @tab @code{<=} @tab 0x13 @tab X = (X <= Y)
|
|
|
2012 @item CCL_GE @tab @code{>=} @tab 0x14 @tab X = (X >= Y)
|
|
|
2013 @item CCL_NE @tab @code{!=} @tab 0x15 @tab X = (X != Y)
|
|
|
2014 @item CCL_ENCODE_SJIS @tab @code{en-sjis} @tab 0x16 @tab X = HIGHER_BYTE (SJIS (Y, Z))
|
|
|
2015 @item @tab @tab @tab r[7] = LOWER_BYTE (SJIS (Y, Z)
|
|
|
2016 @item CCL_DECODE_SJIS @tab @code{de-sjis} @tab 0x17 @tab X = HIGHER_BYTE (DE-SJIS (Y, Z))
|
|
|
2017 @item @tab @tab @tab r[7] = LOWER_BYTE (DE-SJIS (Y, Z))
|
|
|
2018 @end multitable
|
|
|
2019
|
|
442
|
2020 The CCL operators are as in C, with the addition of CCL_LSH8, CCL_RSH8,
|
|
428
|
2021 CCL_DIVMOD, CCL_ENCODE_SJIS, and CCL_DECODE_SJIS. The CCL_ENCODE_SJIS
|
|
|
2022 and CCL_DECODE_SJIS treat their first and second bytes as the high and
|
|
|
2023 low bytes of a two-byte character code. (SJIS stands for Shift JIS, an
|
|
|
2024 encoding of Japanese characters used by Microsoft. CCL_ENCODE_SJIS is a
|
|
|
2025 complicated transformation of the Japanese standard JIS encoding to
|
|
|
2026 Shift JIS. CCL_DECODE_SJIS is its inverse.) It is somewhat odd to
|
|
|
2027 represent the SJIS operations in infix form.
|
|
|
2028
|
|
2640
|
2029 @node Calling CCL, CCL Example, CCL Expressions, CCL
|
|
428
|
2030 @comment Node, Next, Previous, Up
|
|
|
2031 @subsection Calling CCL
|
|
|
2032
|
|
442
|
2033 CCL programs are called automatically during Emacs buffer I/O when the
|
|
428
|
2034 external representation has a coding system type of @code{shift-jis},
|
|
|
2035 @code{big5}, or @code{ccl}. The program is specified by the coding
|
|
|
2036 system (@pxref{Coding Systems}). You can also call CCL programs from
|
|
|
2037 other CCL programs, and from Lisp using these functions:
|
|
|
2038
|
|
|
2039 @defun ccl-execute ccl-program status
|
|
|
2040 Execute @var{ccl-program} with registers initialized by
|
|
|
2041 @var{status}. @var{ccl-program} is a vector of compiled CCL code
|
|
444
|
2042 created by @code{ccl-compile}. It is an error for the program to try to
|
|
428
|
2043 execute a CCL I/O command. @var{status} must be a vector of nine
|
|
|
2044 values, specifying the initial value for the R0, R1 .. R7 registers and
|
|
|
2045 for the instruction counter IC. A @code{nil} value for a register
|
|
|
2046 initializer causes the register to be set to 0. A @code{nil} value for
|
|
|
2047 the IC initializer causes execution to start at the beginning of the
|
|
|
2048 program. When the program is done, @var{status} is modified (by
|
|
|
2049 side-effect) to contain the ending values for the corresponding
|
|
444
|
2050 registers and IC.
|
|
428
|
2051 @end defun
|
|
|
2052
|
|
444
|
2053 @defun ccl-execute-on-string ccl-program status string &optional continue
|
|
428
|
2054 Execute @var{ccl-program} with initial @var{status} on
|
|
|
2055 @var{string}. @var{ccl-program} is a vector of compiled CCL code
|
|
|
2056 created by @code{ccl-compile}. @var{status} must be a vector of nine
|
|
|
2057 values, specifying the initial value for the R0, R1 .. R7 registers and
|
|
|
2058 for the instruction counter IC. A @code{nil} value for a register
|
|
|
2059 initializer causes the register to be set to 0. A @code{nil} value for
|
|
|
2060 the IC initializer causes execution to start at the beginning of the
|
|
444
|
2061 program. An optional fourth argument @var{continue}, if non-@code{nil}, causes
|
|
428
|
2062 the IC to
|
|
|
2063 remain on the unsatisfied read operation if the program terminates due
|
|
|
2064 to exhaustion of the input buffer. Otherwise the IC is set to the end
|
|
444
|
2065 of the program. When the program is done, @var{status} is modified (by
|
|
428
|
2066 side-effect) to contain the ending values for the corresponding
|
|
|
2067 registers and IC. Returns the resulting string.
|
|
|
2068 @end defun
|
|
|
2069
|
|
442
|
2070 To call a CCL program from another CCL program, it must first be
|
|
428
|
2071 registered:
|
|
|
2072
|
|
|
2073 @defun register-ccl-program name ccl-program
|
|
444
|
2074 Register @var{name} for CCL program @var{ccl-program} in
|
|
|
2075 @code{ccl-program-table}. @var{ccl-program} should be the compiled form of
|
|
|
2076 a CCL program, or @code{nil}. Return index number of the registered CCL
|
|
428
|
2077 program.
|
|
|
2078 @end defun
|
|
|
2079
|
|
442
|
2080 Information about the processor time used by the CCL interpreter can be
|
|
428
|
2081 obtained using these functions:
|
|
|
2082
|
|
|
2083 @defun ccl-elapsed-time
|
|
|
2084 Returns the elapsed processor time of the CCL interpreter as cons of
|
|
|
2085 user and system time, as
|
|
|
2086 floating point numbers measured in seconds. If only one
|
|
|
2087 overall value can be determined, the return value will be a cons of that
|
|
|
2088 value and 0.
|
|
|
2089 @end defun
|
|
|
2090
|
|
|
2091 @defun ccl-reset-elapsed-time
|
|
|
2092 Resets the CCL interpreter's internal elapsed time registers.
|
|
|
2093 @end defun
|
|
|
2094
|
|
2640
|
2095 @node CCL Example, , Calling CCL, CCL
|
|
428
|
2096 @comment Node, Next, Previous, Up
|
|
2640
|
2097 @subsection CCL Example
|
|
|
2098
|
|
|
2099 In this section, we describe the implementation of a trivial coding
|
|
|
2100 system to transform from the Web's URL encoding to XEmacs' internal
|
|
|
2101 coding. Many people will have been first exposed to URL encoding when
|
|
|
2102 they saw ``%20'' where they expected a space in a file's name on their
|
|
|
2103 local hard disk; this can happen when a browser saves a file from the
|
|
|
2104 web and doesn't encode the name, as passed from the server, properly.
|
|
|
2105
|
|
|
2106 URL encoding itself is underspecified with regard to encodings beyond
|
|
|
2107 ASCII. The relevant document, RFC 1738, explicitly doesn't give any
|
|
|
2108 information on how to encode non-ASCII characters, and the ``obvious''
|
|
|
2109 way---use the %xx values for the octets of the eight bit MIME character
|
|
|
2110 set in which the page was served---breaks when a user types a character
|
|
|
2111 outside that character set. Best practice for web development is to
|
|
|
2112 serve all pages as UTF-8 and treat incoming form data as using that
|
|
|
2113 coding system. (Oh, and gamble that your clients won't ever want to
|
|
|
2114 type anything outside Unicode. But that's not so much of a gamble with
|
|
|
2115 today's client operating systems.) We don't treat non-ASCII in this
|
|
|
2116 example, as dealing with @samp{(read-multibyte-character ...)} and
|
|
|
2117 errors therewith would make it much harder to understand.
|
|
|
2118
|
|
|
2119 Since CCL isn't a very rich language, we move much of the logic that
|
|
|
2120 would ordinarily be computed from operations like @code{(member ..)},
|
|
|
2121 @code{(and ...)} and @code{(or ...)} into tables, from which register
|
|
|
2122 values are read and written, and on which @code{if} statements are
|
|
|
2123 predicated. Much more of the implementation of this coding system is
|
|
|
2124 occupied with constructing these tables---in normal Emacs Lisp---than it
|
|
|
2125 is with actual CCL code.
|
|
|
2126
|
|
|
2127 All the @code{defvar} statements we deal with in the next few sections
|
|
|
2128 are surrounded by a @code{(eval-and-compile ...)}, which means that the
|
|
|
2129 logic which initializes these variables executes at compile time, and if
|
|
|
2130 XEmacs loads the compiled version of the file, these variables are
|
|
|
2131 initialized as constants.
|
|
|
2132
|
|
|
2133 @menu
|
|
|
2134 * Four bits to ASCII:: Two tables used for getting hex digits from ASCII.
|
|
|
2135 * URI Encoding constants:: Useful predefined characters.
|
|
|
2136 * Numeric to ASCII-hexadecimal conversion:: Trivial in Lisp, not so in CCL.
|
|
|
2137 * Characters to be preserved:: No transformation needed for these characters.
|
|
|
2138 * The program to decode to internal format:: .
|
|
|
2139 * The program to encode from internal format:: .
|
|
2690
|
2140 * The actual coding system:: .
|
|
2640
|
2141 @end menu
|
|
|
2142
|
|
|
2143 @node Four bits to ASCII, URI Encoding constants, , CCL Example
|
|
|
2144 @subsubsection Four bits to ASCII
|
|
|
2145
|
|
|
2146 The first @code{defvar} is for
|
|
|
2147 @code{url-coding-high-order-nybble-as-ascii}, a 256-entry table that
|
|
|
2148 maps from an octet's value to the ASCII encoding for the hex value of
|
|
|
2149 its most significant four bits. That might sound complex, but it isn't;
|
|
|
2150 for decimal 65, hex value @samp{#x41}, the entry in the table is the
|
|
|
2151 ASCII encoding of `4'. For decimal 122, ASCII `z', hex value
|
|
|
2152 @code{#x7a}, @code{(elt url-coding-high-order-nybble-as-ascii #x7a)}
|
|
|
2153 after this file is loaded gives the ASCII encoding of 7.
|
|
|
2154
|
|
|
2155 @example
|
|
|
2156 (defvar url-coding-high-order-nybble-as-ascii
|
|
|
2157 (let ((val (make-vector 256 0))
|
|
|
2158 (i 0))
|
|
|
2159 (while (< i (length val))
|
|
2690
|
2160 (aset val i (char-to-int (aref (format "%02X" i) 0)))
|
|
2640
|
2161 (setq i (1+ i)))
|
|
|
2162 val)
|
|
|
2163 "Table to find an ASCII version of an octet's most significant 4 bits.")
|
|
|
2164 @end example
|
|
|
2165
|
|
|
2166 The next table, @code{url-coding-low-order-nybble-as-ascii} is almost
|
|
|
2167 the same thing, but this time it has a map for the hex encoding of the
|
|
2690
|
2168 low-order four bits. So the sixty-fifth entry (offset @samp{#x41}) is
|
|
2640
|
2169 the ASCII encoding of `1', the hundred-and-twenty-second (offset
|
|
|
2170 @samp{#x7a}) is the ASCII encoding of `A'.
|
|
|
2171
|
|
|
2172 @example
|
|
|
2173 (defvar url-coding-low-order-nybble-as-ascii
|
|
|
2174 (let ((val (make-vector 256 0))
|
|
|
2175 (i 0))
|
|
|
2176 (while (< i (length val))
|
|
2690
|
2177 (aset val i (char-to-int (aref (format "%02X" i) 1)))
|
|
2640
|
2178 (setq i (1+ i)))
|
|
|
2179 val)
|
|
|
2180 "Table to find an ASCII version of an octet's least significant 4 bits.")
|
|
|
2181 @end example
|
|
|
2182
|
|
|
2183 @node URI Encoding constants, Numeric to ASCII-hexadecimal conversion, Four bits to ASCII, CCL Example
|
|
|
2184 @subsubsection URI Encoding constants
|
|
|
2185
|
|
|
2186 Next, we have a couple of variables that make the CCL code more
|
|
|
2187 readable. The first is the ASCII encoding of the percentage sign; this
|
|
|
2188 character is used as an escape code, to start the encoding of a
|
|
|
2189 non-printable character. For historical reasons, URL encoding allows
|
|
|
2190 the space character to be encoded as a plus sign--it does make typing
|
|
|
2191 URLs like @samp{http://google.com/search?q=XEmacs+home+page} easier--and
|
|
|
2192 as such, we have to check when decoding for this value, and map it to
|
|
|
2193 the space character. When doing this in CCL, we use the
|
|
|
2194 @code{url-coding-escaped-space-code} variable.
|
|
|
2195
|
|
|
2196 @example
|
|
2690
|
2197 (defvar url-coding-escape-character-code (char-to-int ?%)
|
|
2640
|
2198 "The code point for the percentage sign, in ASCII.")
|
|
|
2199
|
|
2690
|
2200 (defvar url-coding-escaped-space-code (char-to-int ?+)
|
|
2640
|
2201 "The URL-encoded value of the space character, that is, +.")
|
|
|
2202 @end example
|
|
|
2203
|
|
2690
|
2204 @node Numeric to ASCII-hexadecimal conversion, Characters to be preserved, URI Encoding constants, CCL Example
|
|
2640
|
2205 @subsubsection Numeric to ASCII-hexadecimal conversion
|
|
|
2206
|
|
|
2207 Now, we have a couple of utility tables that wouldn't be necessary in
|
|
|
2208 a more expressive programming language than is CCL. The first is sixteen
|
|
|
2209 in length, and maps a hexadecimal number to the ASCII encoding of that
|
|
|
2210 number; so zero maps to ASCII `0', ten maps to ASCII `A.' The second
|
|
|
2211 does the reverse; that is, it maps an ASCII character to its value when
|
|
|
2212 interpreted as a hexadecimal digit. ('A' => 10, 'c' => 12, '2' => 2, as
|
|
|
2213 a few examples.)
|
|
|
2214
|
|
|
2215 @example
|
|
|
2216 (defvar url-coding-hex-digit-table
|
|
|
2217 (let ((i 0)
|
|
|
2218 (val (make-vector 16 0)))
|
|
|
2219 (while (< i 16)
|
|
2690
|
2220 (aset val i (char-to-int (aref (format "%X" i) 0)))
|
|
2640
|
2221 (setq i (1+ i)))
|
|
|
2222 val)
|
|
|
2223 "A map from a hexadecimal digit's numeric value to its encoding in ASCII.")
|
|
|
2224
|
|
|
2225 (defvar url-coding-latin-1-as-hex-table
|
|
|
2226 (let ((val (make-vector 256 0))
|
|
|
2227 (i 0))
|
|
|
2228 (while (< i (length val))
|
|
|
2229 ;; Get a hex val for this ASCII character.
|
|
|
2230 (aset val i (string-to-int (format "%c" i) 16))
|
|
|
2231 (setq i (1+ i)))
|
|
|
2232 val)
|
|
|
2233 "A map from Latin 1 code points to their values as hexadecimal digits.")
|
|
|
2234 @end example
|
|
|
2235
|
|
2690
|
2236 @node Characters to be preserved, The program to decode to internal format, Numeric to ASCII-hexadecimal conversion, CCL Example
|
|
2640
|
2237 @subsubsection Characters to be preserved
|
|
|
2238
|
|
|
2239 And finally, the last of these tables. URL encoding says that
|
|
|
2240 alphanumeric characters, the underscore, hyphen and the full stop
|
|
|
2241 @footnote{That's what the standards call it, though my North American
|
|
|
2242 readers will be more familiar with it as the period character.} retain
|
|
|
2243 their ASCII encoding, and don't undergo transformation.
|
|
|
2244 @code{url-coding-should-preserve-table} is an array in which the entries
|
|
|
2245 are one if the corresponding ASCII character should be left as-is, and
|
|
|
2246 zero if they should be transformed. So the entries for all the control
|
|
|
2247 and most of the punctuation charcters are zero. Lisp programmers will
|
|
|
2248 observe that this initialization is particularly inefficient, but
|
|
|
2249 they'll also be aware that this is a long way from an inner loop where
|
|
|
2250 every nanosecond counts.
|
|
|
2251
|
|
|
2252 @example
|
|
|
2253 (defvar url-coding-should-preserve-table
|
|
|
2254 (let ((preserve
|
|
|
2255 (list ?- ?_ ?. ?a ?b ?c ?d ?e ?f ?g ?h ?i ?j ?k ?l ?m ?n ?o
|
|
|
2256 ?p ?q ?r ?s ?t ?u ?v ?w ?x ?y ?z ?A ?B ?C ?D ?E ?F ?G
|
|
|
2257 ?H ?I ?J ?K ?L ?M ?N ?O ?P ?Q ?R ?S ?T ?U ?V ?W ?X ?Y
|
|
|
2258 ?Z ?0 ?1 ?2 ?3 ?4 ?5 ?6 ?7 ?8 ?9))
|
|
|
2259 (i 0)
|
|
|
2260 (res (make-vector 256 0)))
|
|
|
2261 (while (< i 256)
|
|
|
2262 (when (member (int-char i) preserve)
|
|
|
2263 (aset res i 1))
|
|
|
2264 (setq i (1+ i)))
|
|
|
2265 res)
|
|
|
2266 "A 256-entry array of flags, indicating whether or not to preserve an
|
|
|
2267 octet as its ASCII encoding.")
|
|
|
2268 @end example
|
|
|
2269
|
|
2690
|
2270 @node The program to decode to internal format, The program to encode from internal format, Characters to be preserved, CCL Example
|
|
2640
|
2271 @subsubsection The program to decode to internal format
|
|
|
2272
|
|
|
2273 After the almost interminable tables, we get to the CCL. The first
|
|
|
2274 CCL program, @code{ccl-decode-urlcoding} decodes from the URL coding to
|
|
|
2275 our internal format; since this version of CCL doesn't have support for
|
|
|
2276 error checking on the input, we don't do any verification on it.
|
|
|
2277
|
|
|
2278 The buffer magnification--approximate ratio of the size of the output
|
|
|
2279 buffer to the size of the input buffer--is declared as one, because
|
|
|
2280 fractional values aren't allowed. (Since all those %20's will map to
|
|
|
2281 ` ', the length of the output text will be less than that of the input
|
|
|
2282 text.)
|
|
|
2283
|
|
|
2284 So, first we read an octet from the input buffer into register
|
|
|
2285 @samp{r0}, to set up the loop. Next, we start the loop, with a
|
|
|
2286 @code{(loop ...)} statement, and we check if the value in @samp{r0} is a
|
|
|
2287 percentage sign. (Note the comma before
|
|
|
2288 @code{url-coding-escape-character-code}; since CCL is a Lisp macro
|
|
|
2289 language, we can break out of the macro evaluation with a comman, and as
|
|
|
2290 such, ``@code{,url-coding-escape-character-code}'' will be evaluated as a
|
|
|
2291 literal `37.')
|
|
|
2292
|
|
|
2293 If it is a percentage sign, we read the next two octets into @samp{r2}
|
|
|
2294 and @samp{r3}, and convert them into their hexadecimal numeric values,
|
|
|
2295 using the @code{url-coding-latin-1-as-hex-table} array declared above.
|
|
|
2296 (But again, it'll be interpreted as a literal array.) We then left
|
|
|
2297 shift the first by four bits, mask the two together, and write the
|
|
|
2298 result to the output buffer.
|
|
|
2299
|
|
|
2300 If it isn't a percentage sign, and it is a `+' sign, we write a
|
|
|
2301 space--hexadecimal 20--to the output buffer.
|
|
|
2302
|
|
|
2303 If none of those things are true, we pass the octet to the output buffer
|
|
|
2304 untransformed. (This could be a place to put error checking, in a more
|
|
|
2305 expressive language.) We then read one more octet from the input
|
|
|
2306 buffer, and move to the next iteration of the loop.
|
|
|
2307
|
|
|
2308 @example
|
|
|
2309 (define-ccl-program ccl-decode-urlcoding
|
|
|
2310 `(1
|
|
|
2311 ((read r0)
|
|
|
2312 (loop
|
|
|
2313 (if (r0 == ,url-coding-escape-character-code)
|
|
|
2314 ((read r2 r3)
|
|
|
2315 ;; Assign the value at offset r2 in the url-coding-hex-digit-table
|
|
|
2316 ;; to r3.
|
|
|
2317 (r2 = r2 ,url-coding-latin-1-as-hex-table)
|
|
|
2318 (r3 = r3 ,url-coding-latin-1-as-hex-table)
|
|
|
2319 (r2 <<= 4)
|
|
|
2320 (r3 |= r2)
|
|
|
2321 (write r3))
|
|
|
2322 (if (r0 == ,url-coding-escaped-space-code)
|
|
|
2323 (write #x20)
|
|
|
2324 (write r0)))
|
|
|
2325 (read r0)
|
|
|
2326 (repeat))))
|
|
|
2327 "CCL program to take URI-encoded ASCII text and transform it to our
|
|
|
2328 internal encoding. ")
|
|
|
2329 @end example
|
|
|
2330
|
|
2690
|
2331 @node The program to encode from internal format, The actual coding system, The program to decode to internal format, CCL Example
|
|
2640
|
2332 @subsubsection The program to encode from internal format
|
|
|
2333
|
|
|
2334 Next, we see the CCL program to encode ASCII text as URL coded text.
|
|
|
2335 Here, the buffer magnification is specified as three, to account for ` '
|
|
|
2336 mapping to %20, etc. As before, we read an octet from the input into
|
|
|
2337 @samp{r0}, and move into the body of the loop. Next, we check if we
|
|
|
2338 should preserve the value of this octet, by reading from offset
|
|
|
2339 @samp{r0} in the @code{url-coding-should-preserve-table} into @samp{r1}.
|
|
|
2340 Then we have an @samp{if} statement predicated on the value in
|
|
|
2341 @samp{r1}; for the true branch, we write the input octet directly. For
|
|
|
2342 the false branch, we write a percentage sign, the ASCII encoding of the
|
|
|
2343 high four bits in hex, and then the ASCII encoding of the low four bits
|
|
|
2344 in hex.
|
|
|
2345
|
|
|
2346 We then read an octet from the input into @samp{r0}, and repeat the loop.
|
|
|
2347
|
|
|
2348 @example
|
|
|
2349 (define-ccl-program ccl-encode-urlcoding
|
|
|
2350 `(3
|
|
|
2351 ((read r0)
|
|
|
2352 (loop
|
|
|
2353 (r1 = r0 ,url-coding-should-preserve-table)
|
|
|
2354 ;; If we should preserve the value, just write the octet directly.
|
|
|
2355 (if r1
|
|
|
2356 (write r0)
|
|
|
2357 ;; else, write a percentage sign, and the hex value of the octet, in
|
|
|
2358 ;; an ASCII-friendly format.
|
|
|
2359 ((write ,url-coding-escape-character-code)
|
|
|
2360 (write r0 ,url-coding-high-order-nybble-as-ascii)
|
|
|
2361 (write r0 ,url-coding-low-order-nybble-as-ascii)))
|
|
|
2362 (read r0)
|
|
|
2363 (repeat))))
|
|
|
2364 "CCL program to encode octets (almost) according to RFC 1738")
|
|
|
2365 @end example
|
|
428
|
2366
|
|
2690
|
2367 @node The actual coding system, , The program to encode from internal format, CCL Example
|
|
|
2368 @subsubsection The actual coding system
|
|
|
2369
|
|
|
2370 To actually create the coding system, we call
|
|
|
2371 @samp{make-coding-system}. The first argument is the symbol that is to
|
|
|
2372 be the name of the coding system, in our case @samp{url-coding}. The
|
|
|
2373 second specifies that the coding system is to be of type
|
|
|
2374 @samp{ccl}---there are several other coding system types available,
|
|
|
2375 including, see the documentation for @samp{make-coding-system} for the
|
|
|
2376 full list. Then there's a documentation string describing the wherefore
|
|
|
2377 and caveats of the coding system, and the final argument is a property
|
|
|
2378 list giving information about the CCL programs and the coding system's
|
|
|
2379 mnemonic.
|
|
|
2380
|
|
|
2381 @example
|
|
|
2382 (make-coding-system
|
|
|
2383 'url-coding 'ccl
|
|
|
2384 "The coding used by application/x-www-form-urlencoded HTTP applications.
|
|
|
2385 This coding form doesn't specify anything about non-ASCII characters, so
|
|
|
2386 make sure you've transformed to a seven-bit coding system first."
|
|
|
2387 '(decode ccl-decode-urlcoding
|
|
|
2388 encode ccl-encode-urlcoding
|
|
|
2389 mnemonic "URLenc"))
|
|
|
2390 @end example
|
|
|
2391
|
|
|
2392 If you're lucky, the @samp{url-coding} coding system describe here
|
|
|
2393 should be available in the XEmacs package system. Otherwise, downloading
|
|
|
2394 it from @samp{http://www.parhasard.net/url-coding.el} should work for
|
|
|
2395 the foreseeable future.
|
|
|
2396
|
|
775
|
2397 @node Category Tables, Unicode Support, CCL, MULE
|
|
428
|
2398 @section Category Tables
|
|
|
2399
|
|
|
2400 A category table is a type of char table used for keeping track of
|
|
|
2401 categories. Categories are used for classifying characters for use in
|
|
440
|
2402 regexps---you can refer to a category rather than having to use a
|
|
428
|
2403 complicated [] expression (and category lookups are significantly
|
|
|
2404 faster).
|
|
|
2405
|
|
|
2406 There are 95 different categories available, one for each printable
|
|
|
2407 character (including space) in the ASCII charset. Each category is
|
|
|
2408 designated by one such character, called a @dfn{category designator}.
|
|
|
2409 They are specified in a regexp using the syntax @samp{\cX}, where X is a
|
|
|
2410 category designator. (This is not yet implemented.)
|
|
|
2411
|
|
|
2412 A category table specifies, for each character, the categories that
|
|
|
2413 the character is in. Note that a character can be in more than one
|
|
|
2414 category. More specifically, a category table maps from a character to
|
|
|
2415 either the value @code{nil} (meaning the character is in no categories)
|
|
|
2416 or a 95-element bit vector, specifying for each of the 95 categories
|
|
|
2417 whether the character is in that category.
|
|
|
2418
|
|
|
2419 Special Lisp functions are provided that abstract this, so you do not
|
|
|
2420 have to directly manipulate bit vectors.
|
|
|
2421
|
|
444
|
2422 @defun category-table-p object
|
|
|
2423 This function returns @code{t} if @var{object} is a category table.
|
|
428
|
2424 @end defun
|
|
|
2425
|
|
|
2426 @defun category-table &optional buffer
|
|
|
2427 This function returns the current category table. This is the one
|
|
|
2428 specified by the current buffer, or by @var{buffer} if it is
|
|
|
2429 non-@code{nil}.
|
|
|
2430 @end defun
|
|
|
2431
|
|
|
2432 @defun standard-category-table
|
|
|
2433 This function returns the standard category table. This is the one used
|
|
|
2434 for new buffers.
|
|
|
2435 @end defun
|
|
|
2436
|
|
444
|
2437 @defun copy-category-table &optional category-table
|
|
|
2438 This function returns a new category table which is a copy of
|
|
|
2439 @var{category-table}, which defaults to the standard category table.
|
|
428
|
2440 @end defun
|
|
|
2441
|
|
444
|
2442 @defun set-category-table category-table &optional buffer
|
|
|
2443 This function selects @var{category-table} as the new category table for
|
|
|
2444 @var{buffer}. @var{buffer} defaults to the current buffer if omitted.
|
|
428
|
2445 @end defun
|
|
|
2446
|
|
444
|
2447 @defun category-designator-p object
|
|
|
2448 This function returns @code{t} if @var{object} is a category designator (a
|
|
428
|
2449 char in the range @samp{' '} to @samp{'~'}).
|
|
|
2450 @end defun
|
|
|
2451
|
|
444
|
2452 @defun category-table-value-p object
|
|
|
2453 This function returns @code{t} if @var{object} is a category table value.
|
|
428
|
2454 Valid values are @code{nil} or a bit vector of size 95.
|
|
|
2455 @end defun
|
|
|
2456
|
|
775
|
2457
|
|
|
2458 @c Added 2002-03-13 sjt
|
|
1183
|
2459 @node Unicode Support, Charset Unification, Category Tables, MULE
|
|
775
|
2460 @section Unicode Support
|
|
|
2461 @cindex unicode
|
|
|
2462 @cindex utf-8
|
|
|
2463 @cindex utf-16
|
|
|
2464 @cindex ucs-2
|
|
|
2465 @cindex ucs-4
|
|
|
2466 @cindex bmp
|
|
|
2467 @cindex basic multilingual plance
|
|
|
2468
|
|
|
2469 Unicode support was added by Ben Wing to XEmacs 21.5.6.
|
|
|
2470
|
|
|
2471 @defun set-language-unicode-precedence-list list
|
|
|
2472 Set the language-specific precedence list used for Unicode decoding.
|
|
|
2473 This is a list of charsets, which are consulted in order for a translation
|
|
|
2474 matching a given Unicode character. If no matches are found, the charsets
|
|
|
2475 in the default precedence list (see
|
|
|
2476 @code{set-default-unicode-precedence-list}) are consulted, and then all
|
|
|
2477 remaining charsets, in some arbitrary order.
|
|
|
2478
|
|
|
2479 The language-specific precedence list is meant to be set as part of the
|
|
|
2480 language environment initialization; the default precedence list is meant
|
|
|
2481 to be set by the user.
|
|
|
2482 @end defun
|
|
|
2483
|
|
|
2484 @defun language-unicode-precedence-list
|
|
|
2485 Return the language-specific precedence list used for Unicode decoding.
|
|
|
2486 See @code{set-language-unicode-precedence-list} for more information.
|
|
|
2487 @end defun
|
|
|
2488
|
|
|
2489 @defun set-default-unicode-precedence-list list
|
|
|
2490 Set the default precedence list used for Unicode decoding.
|
|
|
2491 This is meant to be set by the user. See
|
|
|
2492 `set-language-unicode-precedence-list' for more information.
|
|
|
2493 @end defun
|
|
|
2494
|
|
|
2495 @defun default-unicode-precedence-list
|
|
|
2496 Return the default precedence list used for Unicode decoding.
|
|
|
2497 See @code{set-language-unicode-precedence-list} for more information.
|
|
|
2498 @end defun
|
|
|
2499
|
|
|
2500 @defun set-unicode-conversion character code
|
|
|
2501 Add conversion information between Unicode codepoints and characters.
|
|
|
2502 @var{character} is one of the following:
|
|
|
2503
|
|
|
2504 @c #### fix this markup
|
|
|
2505 -- A character (in which case @var{code} must be a non-negative integer)
|
|
|
2506 -- A vector of characters (in which case @var{code} must be a vector of
|
|
|
2507 non-negative integers of the same length)
|
|
|
2508
|
|
|
2509 Values of @var{code} above 2^20 - 1 are allowed for the purpose of specifying
|
|
|
2510 private characters, but will cause errors when converted to UTF-16 or UTF-32.
|
|
|
2511 UCS-4 and UTF-8 can handle values to 2^31 - 1, but XEmacs Lisp integers top
|
|
|
2512 out at 2^30 - 1.
|
|
|
2513 @end defun
|
|
|
2514
|
|
|
2515 @defun character-to-unicode character
|
|
|
2516 Convert @var{character} to Unicode codepoint.
|
|
|
2517 When there is no international support (i.e. MULE is not defined),
|
|
|
2518 this function simply does @code{char-to-int}.
|
|
|
2519 @end defun
|
|
|
2520
|
|
|
2521 @defun unicode-to-character code [charsets]
|
|
|
2522 Convert Unicode codepoint @var{code} to character.
|
|
|
2523 @var{code} should be a non-negative integer.
|
|
|
2524 If @var{charsets} is given, it should be a list of charsets, and only those
|
|
|
2525 charsets will be consulted, in the given order, for a translation.
|
|
|
2526 Otherwise, the default ordering of all charsets will be given (see
|
|
|
2527 @code{set-unicode-charset-precedence}).
|
|
|
2528
|
|
|
2529 When there is no international support (i.e. MULE is not defined),
|
|
|
2530 this function simply does @code{int-to-char} and ignores the
|
|
|
2531 @var{charsets} argument.
|
|
|
2532 @end defun
|
|
|
2533
|
|
|
2534 @defun parse-unicode-translation-table filename charset start end offset flags
|
|
|
2535 Parse Unicode translation data in @var{filename} for MULE @var{charset}.
|
|
|
2536 Data is text, in the form of one translation per line -- charset
|
|
|
2537 codepoint followed by Unicode codepoint. Numbers are decimal or hex
|
|
|
2538 \(preceded by 0x). Comments are marked with a #. Charset codepoints
|
|
|
2539 for two-dimensional charsets should have the first octet stored in the
|
|
|
2540 high 8 bits of the hex number and the second in the low 8 bits.
|
|
|
2541
|
|
|
2542 If @var{start} and @var{end} are given, only charset codepoints within
|
|
|
2543 the given range will be processed. If @var{offset} is given, that value
|
|
|
2544 will be added to all charset codepoints in the file to obtain the
|
|
|
2545 internal charset codepoint. @var{start} and @var{end} apply to the
|
|
|
2546 codepoints in the file, before @var{offset} is applied.
|
|
|
2547
|
|
|
2548 (Note that, as usual, we assume that octets are in the range 32 to
|
|
|
2549 127 or 33 to 126. If you have a table in kuten form, with octets in
|
|
|
2550 the range 1 to 94, you will have to use an offset of 5140,
|
|
|
2551 i.e. 0x2020.)
|
|
|
2552
|
|
|
2553 @var{flags}, if specified, control further how the tables are interpreted
|
|
|
2554 and are used to special-case certain known table weirdnesses in the
|
|
|
2555 Unicode tables:
|
|
|
2556
|
|
|
2557 @table @code
|
|
|
2558 @item ignore-first-column'
|
|
|
2559 Exactly as it sounds. The JIS X 0208 tables have 3 columns of data instead
|
|
|
2560 of 2; the first is the Shift-JIS codepoint.
|
|
|
2561
|
|
|
2562 @item big5
|
|
|
2563 The charset codepoint is a Big Five codepoint; convert it to the
|
|
|
2564 proper hacked-up codepoint in `chinese-big5-1' or `chinese-big5-2'.
|
|
|
2565 @end table
|
|
|
2566 @end defun
|
|
|
2567
|
|
1183
|
2568
|
|
|
2569 @node Charset Unification, Charsets and Coding Systems, Unicode Support, MULE
|
|
|
2570 @section Character Set Unification
|
|
|
2571
|
|
|
2572 Mule suffers from a design defect that causes it to consider the ISO
|
|
|
2573 Latin character sets to be disjoint. This results in oddities such as
|
|
|
2574 files containing both ISO 8859/1 and ISO 8859/15 codes, and using ISO
|
|
|
2575 2022 control sequences to switch between them, as well as more plausible
|
|
|
2576 but often unnecessary combinations like ISO 8859/1 with ISO 8859/2.
|
|
|
2577 This can be very annoying when sending messages or even in simple
|
|
|
2578 editing on a single host. Unification works around the problem by
|
|
|
2579 converting as many characters as possible to use a single Latin coded
|
|
|
2580 character set before saving the buffer.
|
|
|
2581
|
|
|
2582 This node and its children were ripp'd untimely from
|
|
|
2583 @file{latin-unity.texi}, and have been quickly converted for use here.
|
|
|
2584 However as APIs are likely to diverge, beware of inaccuracies. Please
|
|
|
2585 report any you discover with @kbd{M-x report-xemacs-bug RET}, as well
|
|
|
2586 as any ambiguities or downright unintelligible passages.
|
|
|
2587
|
|
|
2588 A lot of the stuff here doesn't belong here; it belongs in the
|
|
|
2589 @ref{Top, , , xemacs, XEmacs User's Manual}. Report those as bugs,
|
|
|
2590 too, preferably with patches.
|
|
|
2591
|
|
|
2592 @menu
|
|
|
2593 * Overview:: Unification history and general information.
|
|
|
2594 * Usage:: An overview of the operation of Unification.
|
|
|
2595 * Configuration:: Configuring Unification for use.
|
|
|
2596 * Theory of Operation:: How Unification works.
|
|
|
2597 * What Unification Cannot Do for You:: Inherent problems of 8-bit charsets.
|
|
|
2598 * Charsets and Coding Systems:: Reference lists with annotations.
|
|
1188
|
2599 * Unification Internals:: Utilities and implementation details.
|
|
1183
|
2600 @end menu
|
|
|
2601
|
|
|
2602 @node Overview, Usage, Charset Unification, Charset Unification
|
|
|
2603 @subsection An Overview of Unification
|
|
|
2604
|
|
|
2605 Mule suffers from a design defect that causes it to consider the ISO
|
|
|
2606 Latin character sets to be disjoint. This manifests itself when a user
|
|
|
2607 enters characters using input methods associated with different coded
|
|
|
2608 character sets into a single buffer.
|
|
|
2609
|
|
|
2610 A very important example involves email. Many sites, especially in the
|
|
|
2611 U.S., default to use of the ISO 8859/1 coded character set (also called
|
|
|
2612 ``Latin 1,'' though these are somewhat different concepts). However,
|
|
|
2613 ISO 8859/1 provides a generic CURRENCY SIGN character. Now that the
|
|
|
2614 Euro has become the official currency of most countries in Europe, this
|
|
|
2615 is unsatisfactory (and in practice, useless). So Europeans generally
|
|
|
2616 use ISO 8859/15, which is nearly identical to ISO 8859/1 for most
|
|
|
2617 languages, except that it substitutes EURO SIGN for CURRENCY SIGN.
|
|
|
2618
|
|
|
2619 Suppose a European user yanks text from a post encoded in ISO 8859/1
|
|
|
2620 into a message composition buffer, and enters some text including the
|
|
|
2621 Euro sign. Then Mule will consider the buffer to contain both ISO
|
|
|
2622 8859/1 and ISO 8859/15 text, and MUAs such as Gnus will (if naively
|
|
|
2623 programmed) send the message as a multipart mixed MIME body!
|
|
|
2624
|
|
|
2625 This is clearly stupid. What is not as obvious is that, just as any
|
|
|
2626 European can include American English in their text because ASCII is a
|
|
|
2627 subset of ISO 8859/15, most European languages which use Latin
|
|
|
2628 characters (eg, German and Polish) can typically be mixed while using
|
|
|
2629 only one Latin coded character set (in this case, ISO 8859/2). However,
|
|
|
2630 this often depends on exactly what text is to be encoded.
|
|
|
2631
|
|
|
2632 Unification works around the problem by converting as many characters as
|
|
|
2633 possible to use a single Latin coded character set before saving the
|
|
|
2634 buffer.
|
|
|
2635
|
|
|
2636 @node Usage, Configuration, Overview, Charset Unification
|
|
|
2637 @subsection Operation of Unification
|
|
|
2638
|
|
|
2639 Normally, Unification works in the background by installing
|
|
|
2640 @code{unity-sanity-check} on @code{write-region-pre-hook}. This is
|
|
|
2641 done by default for the ISO 8859 Latin family of character sets. The
|
|
|
2642 user activates this functionality for other character set families by
|
|
|
2643 invoking @code{enable-unification}, either interactively or in her
|
|
|
2644 init file. @xref{Init File, , , xemacs}. Unification can be
|
|
|
2645 deactivated by invoking @code{disable-unification}.
|
|
|
2646
|
|
|
2647 Unification also provides a few functions for remapping or recoding the
|
|
|
2648 buffer by hand. To @dfn{remap} a character means to change the buffer
|
|
|
2649 representation of the character by using another coded character set.
|
|
|
2650 Remapping never changes the identity of the character, but may involve
|
|
|
2651 altering the code point of the character. To @dfn{recode} a character
|
|
|
2652 means to simply change the coded character set. Recoding never alters
|
|
|
2653 the code point of the character, but may change the identity of the
|
|
|
2654 character. @xref{Theory of Operation}.
|
|
|
2655
|
|
|
2656 There are a few variables which determine which coding systems are
|
|
|
2657 always acceptable to Unification: @code{unity-ucs-list},
|
|
|
2658 @code{unity-preferred-coding-system-list}, and
|
|
|
2659 @code{unity-preapproved-coding-system-list}. The latter two default
|
|
|
2660 to @code{()}, and should probably be avoided because they short-circuit
|
|
|
2661 the sanity check. If you find you need to use them, consider reporting
|
|
|
2662 it as a bug or request for enhancement. Because they seem unsafe, the
|
|
|
2663 recommended interface is likely to change.
|
|
|
2664
|
|
|
2665 @menu
|
|
|
2666 * Basic Functionality:: User interface and customization.
|
|
|
2667 * Interactive Usage:: Treating text by hand.
|
|
|
2668 Also documents the hook function(s).
|
|
|
2669 @end menu
|
|
|
2670
|
|
|
2671
|
|
|
2672 @node Basic Functionality, Interactive Usage, , Usage
|
|
|
2673 @section Basic Functionality
|
|
|
2674
|
|
|
2675 These functions and user options initialize and configure Unification.
|
|
|
2676 In normal use, none of these should be needed.
|
|
|
2677
|
|
|
2678 @strong{These APIs are certain to change.}
|
|
|
2679
|
|
|
2680 @defun enable-unification
|
|
|
2681 Set up hooks and initialize variables for latin-unity.
|
|
|
2682
|
|
|
2683 There are no arguments.
|
|
|
2684
|
|
|
2685 This function is idempotent. It will reinitialize any hooks or variables
|
|
|
2686 that are not in initial state.
|
|
|
2687 @end defun
|
|
|
2688
|
|
|
2689 @defun disable-unification
|
|
|
2690 There are no arguments.
|
|
|
2691
|
|
|
2692 Clean up hooks and void variables used by latin-unity.
|
|
|
2693 @end defun
|
|
|
2694
|
|
|
2695 @defopt unity-ucs-list
|
|
|
2696 List of coding systems considered to be universal.
|
|
|
2697
|
|
|
2698 The default value is @code{'(utf-8 iso-2022-7 ctext escape-quoted)}.
|
|
|
2699
|
|
|
2700 Order matters; coding systems earlier in the list will be preferred when
|
|
|
2701 recommending a coding system. These coding systems will not be used
|
|
|
2702 without querying the user (unless they are also present in
|
|
|
2703 @code{unity-preapproved-coding-system-list}), and follow the
|
|
|
2704 @code{unity-preferred-coding-system-list} in the list of suggested
|
|
|
2705 coding systems.
|
|
|
2706
|
|
|
2707 If none of the preferred coding systems are feasible, the first in
|
|
|
2708 this list will be the default.
|
|
|
2709
|
|
|
2710 Notes on certain coding systems: @code{escape-quoted} is a special
|
|
|
2711 coding system used for autosaves and compiled Lisp in Mule. You should
|
|
|
2712 @c #### fix in latin-unity.texi
|
|
|
2713 never delete this, although it is rare that a user would want to use it
|
|
|
2714 directly. Unification does not try to be \"smart\" about other general
|
|
|
2715 ISO 2022 coding systems, such as ISO-2022-JP. (They are not recognized
|
|
|
2716 as equivalent to @code{iso-2022-7}.) If your preferred coding system is
|
|
|
2717 one of these, you may consider adding it to @code{unity-ucs-list}.
|
|
|
2718 However, this will typically have the side effect that (eg) ISO 8859/1
|
|
|
2719 files will be saved in 7-bit form with ISO 2022 escape sequences.
|
|
|
2720 @end defopt
|
|
|
2721
|
|
|
2722 Coding systems which are not Latin and not in
|
|
|
2723 @code{unity-ucs-list} are handled by short circuiting checks of
|
|
|
2724 coding system against the next two variables.
|
|
|
2725
|
|
|
2726 @defopt unity-preapproved-coding-system-list
|
|
|
2727 List of coding systems used without querying the user if feasible.
|
|
|
2728
|
|
|
2729 The default value is @samp{(buffer-default preferred)}.
|
|
|
2730
|
|
|
2731 The first feasible coding system in this list is used. The special values
|
|
|
2732 @samp{preferred} and @samp{buffer-default} may be present:
|
|
|
2733
|
|
|
2734 @table @code
|
|
|
2735 @item buffer-default
|
|
|
2736 Use the coding system used by @samp{write-region}, if feasible.
|
|
|
2737
|
|
|
2738 @item preferred
|
|
|
2739 Use the coding system specified by @samp{prefer-coding-system} if feasible.
|
|
|
2740 @end table
|
|
|
2741
|
|
|
2742 "Feasible" means that all characters in the buffer can be represented by
|
|
|
2743 the coding system. Coding systems in @samp{unity-ucs-list} are
|
|
|
2744 always considered feasible. Other feasible coding systems are computed
|
|
|
2745 by @samp{unity-representations-feasible-region}.
|
|
|
2746
|
|
|
2747 Note that the first universal coding system in this list shadows all
|
|
|
2748 other coding systems. In particular, if your preferred coding system is
|
|
|
2749 a universal coding system, and @code{preferred} is a member of this
|
|
|
2750 list, unification will blithely convert all your files to that coding
|
|
|
2751 system. This is considered a feature, but it may surprise most users.
|
|
|
2752 Users who don't like this behavior should put @code{preferred} in
|
|
|
2753 @code{unity-preferred-coding-system-list}.
|
|
|
2754 @end defopt
|
|
|
2755
|
|
|
2756 @defopt unity-preferred-coding-system-list
|
|
|
2757 @c #### fix in latin-unity.texi
|
|
|
2758 List of coding systems suggested to the user if feasible.
|
|
|
2759
|
|
|
2760 The default value is @samp{(iso-8859-1 iso-8859-15 iso-8859-2 iso-8859-3
|
|
|
2761 iso-8859-4 iso-8859-9)}.
|
|
|
2762
|
|
|
2763 If none of the coding systems in
|
|
|
2764 @c #### fix in latin-unity.texi
|
|
|
2765 @code{unity-preapproved-coding-system-list} are feasible, this list
|
|
|
2766 will be recommended to the user, followed by the
|
|
|
2767 @code{unity-ucs-list}. The first coding system in this list is default. The
|
|
|
2768 special values @samp{preferred} and @samp{buffer-default} may be
|
|
|
2769 present:
|
|
|
2770
|
|
|
2771 @table @code
|
|
|
2772 @item buffer-default
|
|
|
2773 Use the coding system used by @samp{write-region}, if feasible.
|
|
|
2774
|
|
|
2775 @item preferred
|
|
|
2776 Use the coding system specified by @samp{prefer-coding-system} if feasible.
|
|
|
2777 @end table
|
|
|
2778
|
|
|
2779 "Feasible" means that all characters in the buffer can be represented by
|
|
|
2780 the coding system. Coding systems in @samp{unity-ucs-list} are
|
|
|
2781 always considered feasible. Other feasible coding systems are computed
|
|
|
2782 by @samp{unity-representations-feasible-region}.
|
|
|
2783 @end defopt
|
|
|
2784
|
|
|
2785
|
|
|
2786 @defvar unity-iso-8859-1-aliases
|
|
|
2787 List of coding systems to be treated as aliases of ISO 8859/1.
|
|
|
2788
|
|
|
2789 The default value is '(iso-8859-1).
|
|
|
2790
|
|
|
2791 This is not a user variable; to customize input of coding systems or
|
|
|
2792 charsets, @samp{unity-coding-system-alias-alist} or
|
|
|
2793 @samp{unity-charset-alias-alist}.
|
|
|
2794 @end defvar
|
|
|
2795
|
|
|
2796
|
|
|
2797 @node Interactive Usage, , Basic Functionality, Usage
|
|
|
2798 @section Interactive Usage
|
|
|
2799
|
|
|
2800 First, the hook function @code{unity-sanity-check} is documented.
|
|
|
2801 (It is placed here because it is not an interactive function, and there
|
|
|
2802 is not yet a programmer's section of the manual.)
|
|
|
2803
|
|
|
2804 These functions provide access to internal functionality (such as the
|
|
|
2805 remapping function) and to extra functionality (the recoding functions
|
|
|
2806 and the test function).
|
|
|
2807
|
|
|
2808
|
|
|
2809 @defun unity-sanity-check begin end filename append visit lockname &optional coding-system
|
|
|
2810
|
|
|
2811 Check if @var{coding-system} can represent all characters between
|
|
|
2812 @var{begin} and @var{end}.
|
|
|
2813
|
|
|
2814 For compatibility with old broken versions of @code{write-region},
|
|
|
2815 @var{coding-system} defaults to @code{buffer-file-coding-system}.
|
|
|
2816 @var{filename}, @var{append}, @var{visit}, and @var{lockname} are
|
|
|
2817 ignored.
|
|
|
2818
|
|
|
2819 Return nil if buffer-file-coding-system is not (ISO-2022-compatible)
|
|
|
2820 Latin. If @code{buffer-file-coding-system} is safe for the charsets
|
|
|
2821 actually present in the buffer, return it. Otherwise, ask the user to
|
|
|
2822 choose a coding system, and return that.
|
|
|
2823
|
|
|
2824 This function does @emph{not} do the safe thing when
|
|
|
2825 @code{buffer-file-coding-system} is nil (aka no-conversion). It
|
|
|
2826 considers that ``non-Latin,'' and passes it on to the Mule detection
|
|
|
2827 mechanism.
|
|
|
2828
|
|
|
2829 This function is intended for use as a @code{write-region-pre-hook}. It
|
|
|
2830 does nothing except return @var{coding-system} if @code{write-region}
|
|
|
2831 handlers are inhibited.
|
|
|
2832 @end defun
|
|
|
2833
|
|
|
2834 @defun unity-buffer-representations-feasible
|
|
|
2835
|
|
|
2836 There are no arguments.
|
|
|
2837
|
|
|
2838 Apply unity-region-representations-feasible to the current buffer.
|
|
|
2839 @end defun
|
|
|
2840
|
|
|
2841 @defun unity-region-representations-feasible begin end &optional buf
|
|
|
2842
|
|
|
2843 Return character sets that can represent the text from @var{begin} to @var{end} in @var{buf}.
|
|
|
2844
|
|
|
2845 @var{buf} defaults to the current buffer. Called interactively, will be
|
|
|
2846 applied to the region. Function assumes @var{begin} <= @var{end}.
|
|
|
2847
|
|
|
2848 The return value is a cons. The car is the list of character sets
|
|
|
2849 that can individually represent all of the non-ASCII portion of the
|
|
|
2850 buffer, and the cdr is the list of character sets that can
|
|
|
2851 individually represent all of the ASCII portion.
|
|
|
2852
|
|
|
2853 The following is taken from a comment in the source. Please refer to
|
|
|
2854 the source to be sure of an accurate description.
|
|
|
2855
|
|
|
2856 The basic algorithm is to map over the region, compute the set of
|
|
|
2857 charsets that can represent each character (the ``feasible charset''),
|
|
|
2858 and take the intersection of those sets.
|
|
|
2859
|
|
|
2860 The current implementation takes advantage of the fact that ASCII
|
|
|
2861 characters are common and cannot change asciisets. Then using
|
|
|
2862 skip-chars-forward makes motion over ASCII subregions very fast.
|
|
|
2863
|
|
|
2864 This same strategy could be applied generally by precomputing classes
|
|
|
2865 of characters equivalent according to their effect on latinsets, and
|
|
|
2866 adding a whole class to the skip-chars-forward string once a member is
|
|
|
2867 found.
|
|
|
2868
|
|
|
2869 Probably efficiency is a function of the number of characters matched,
|
|
|
2870 or maybe the length of the match string? With @code{skip-category-forward}
|
|
|
2871 over a precomputed category table it should be really fast. In practice
|
|
|
2872 for Latin character sets there are only 29 classes.
|
|
|
2873 @end defun
|
|
|
2874
|
|
|
2875 @defun unity-remap-region begin end character-set &optional coding-system
|
|
|
2876
|
|
|
2877 Remap characters between @var{begin} and @var{end} to equivalents in
|
|
|
2878 @var{character-set}. Optional argument @var{coding-system} may be a
|
|
|
2879 coding system name (a symbol) or nil. Characters with no equivalent are
|
|
|
2880 left as-is.
|
|
|
2881
|
|
|
2882 When called interactively, @var{begin} and @var{end} are set to the
|
|
|
2883 beginning and end, respectively, of the active region, and the function
|
|
|
2884 prompts for @var{character-set}. The function does completion, knows
|
|
|
2885 how to guess a character set name from a coding system name, and also
|
|
|
2886 provides some common aliases. See @code{unity-guess-charset}.
|
|
|
2887 There is no way to specify @var{coding-system}, as it has no useful
|
|
|
2888 function interactively.
|
|
|
2889
|
|
|
2890 Return @var{coding-system} if @var{coding-system} can encode all
|
|
|
2891 characters in the region, t if @var{coding-system} is nil and the coding
|
|
|
2892 system with G0 = 'ascii and G1 = @var{character-set} can encode all
|
|
|
2893 characters, and otherwise nil. Note that a non-null return does
|
|
|
2894 @emph{not} mean it is safe to write the file, only the specified region.
|
|
|
2895 (This behavior is useful for multipart MIME encoding and the like.)
|
|
|
2896
|
|
|
2897 Note: by default this function is quite fascist about universal coding
|
|
|
2898 systems. It only admits @samp{utf-8}, @samp{iso-2022-7}, and
|
|
|
2899 @samp{ctext}. Customize @code{unity-approved-ucs-list} to change
|
|
|
2900 this.
|
|
|
2901
|
|
|
2902 This function remaps characters that are artificially distinguished by Mule
|
|
|
2903 internal code. It may change the code point as well as the character set.
|
|
|
2904 To recode characters that were decoded in the wrong coding system, use
|
|
|
2905 @code{unity-recode-region}.
|
|
|
2906 @end defun
|
|
|
2907
|
|
|
2908 @defun unity-recode-region begin end wrong-cs right-cs
|
|
|
2909
|
|
|
2910 Recode characters between @var{begin} and @var{end} from @var{wrong-cs}
|
|
|
2911 to @var{right-cs}.
|
|
|
2912
|
|
|
2913 @var{wrong-cs} and @var{right-cs} are character sets. Characters retain
|
|
|
2914 the same code point but the character set is changed. Only characters
|
|
|
2915 from @var{wrong-cs} are changed to @var{right-cs}. The identity of the
|
|
|
2916 character may change. Note that this could be dangerous, if characters
|
|
|
2917 whose identities you do not want changed are included in the region.
|
|
|
2918 This function cannot guess which characters you want changed, and which
|
|
|
2919 should be left alone.
|
|
|
2920
|
|
|
2921 When called interactively, @var{begin} and @var{end} are set to the
|
|
|
2922 beginning and end, respectively, of the active region, and the function
|
|
|
2923 prompts for @var{wrong-cs} and @var{right-cs}. The function does
|
|
|
2924 completion, knows how to guess a character set name from a coding system
|
|
|
2925 name, and also provides some common aliases. See
|
|
|
2926 @code{unity-guess-charset}.
|
|
|
2927
|
|
|
2928 Another way to accomplish this, but using coding systems rather than
|
|
|
2929 character sets to specify the desired recoding, is
|
|
|
2930 @samp{unity-recode-coding-region}. That function may be faster
|
|
|
2931 but is somewhat more dangerous, because it may recode more than one
|
|
|
2932 character set.
|
|
|
2933
|
|
|
2934 To change from one Mule representation to another without changing identity
|
|
|
2935 of any characters, use @samp{unity-remap-region}.
|
|
|
2936 @end defun
|
|
|
2937
|
|
|
2938 @defun unity-recode-coding-region begin end wrong-cs right-cs
|
|
|
2939
|
|
|
2940 Recode text between @var{begin} and @var{end} from @var{wrong-cs} to
|
|
|
2941 @var{right-cs}.
|
|
|
2942
|
|
|
2943 @var{wrong-cs} and @var{right-cs} are coding systems. Characters retain
|
|
|
2944 the same code point but the character set is changed. The identity of
|
|
|
2945 characters may change. This is an inherently dangerous function;
|
|
|
2946 multilingual text may be recoded in unexpected ways. #### It's also
|
|
|
2947 dangerous because the coding systems are not sanity-checked in the
|
|
|
2948 current implementation.
|
|
|
2949
|
|
|
2950 When called interactively, @var{begin} and @var{end} are set to the
|
|
|
2951 beginning and end, respectively, of the active region, and the function
|
|
|
2952 prompts for @var{wrong-cs} and @var{right-cs}. The function does
|
|
|
2953 completion, knows how to guess a coding system name from a character set
|
|
|
2954 name, and also provides some common aliases. See
|
|
|
2955 @code{unity-guess-coding-system}.
|
|
|
2956
|
|
|
2957 Another, safer, way to accomplish this, using character sets rather
|
|
|
2958 than coding systems to specify the desired recoding, is to use
|
|
|
2959 @c #### fixme in latin-unity.texi
|
|
|
2960 @code{unity-recode-region}.
|
|
|
2961
|
|
|
2962 To change from one Mule representation to another without changing identity
|
|
|
2963 of any characters, use @code{unity-remap-region}.
|
|
|
2964 @end defun
|
|
|
2965
|
|
|
2966 Helper functions for input of coding system and character set names.
|
|
|
2967
|
|
|
2968 @defun unity-guess-charset candidate
|
|
|
2969 Guess a charset based on the symbol @var{candidate}.
|
|
|
2970
|
|
|
2971 @var{candidate} itself is not tried as the value.
|
|
|
2972
|
|
|
2973 Uses the natural mapping in @samp{unity-cset-codesys-alist}, and
|
|
|
2974 the values in @samp{unity-charset-alias-alist}."
|
|
|
2975 @end defun
|
|
|
2976
|
|
|
2977 @defun unity-guess-coding-system candidate
|
|
|
2978 Guess a coding system based on the symbol @var{candidate}.
|
|
|
2979
|
|
|
2980 @var{candidate} itself is not tried as the value.
|
|
|
2981
|
|
|
2982 Uses the natural mapping in @samp{unity-cset-codesys-alist}, and
|
|
|
2983 the values in @samp{unity-coding-system-alias-alist}."
|
|
|
2984 @end defun
|
|
|
2985
|
|
|
2986 @defun unity-example
|
|
|
2987
|
|
|
2988 A cheesy example for Unification.
|
|
|
2989
|
|
|
2990 At present it just makes a multilingual buffer. To test, setq
|
|
|
2991 buffer-file-coding-system to some value, make the buffer dirty (eg
|
|
|
2992 with RET BackSpace), and save.
|
|
|
2993 @end defun
|
|
|
2994
|
|
|
2995
|
|
|
2996 @node Configuration, Theory of Operation, Usage, Charset Unification
|
|
|
2997 @subsection Configuring Unification for Use
|
|
|
2998
|
|
|
2999 If you want Unification to be automatically initialized, invoke
|
|
|
3000 @samp{enable-unification} with no arguments in your init file.
|
|
|
3001 @xref{Init File, , , xemacs}. If you are using GNU Emacs or an XEmacs
|
|
|
3002 earlier than 21.1, you should also load @file{auto-autoloads} using the
|
|
|
3003 full path (@emph{never} @samp{require} @file{auto-autoloads} libraries).
|
|
|
3004
|
|
|
3005 You may wish to define aliases for commonly used character sets and
|
|
|
3006 coding systems for convenience in input.
|
|
|
3007
|
|
|
3008 @defopt unity-charset-alias-alist
|
|
|
3009 Alist mapping aliases to Mule charset names (symbols)."
|
|
|
3010
|
|
|
3011 The default value is
|
|
|
3012 @example
|
|
|
3013 ((latin-1 . latin-iso8859-1)
|
|
|
3014 (latin-2 . latin-iso8859-2)
|
|
|
3015 (latin-3 . latin-iso8859-3)
|
|
|
3016 (latin-4 . latin-iso8859-4)
|
|
|
3017 (latin-5 . latin-iso8859-9)
|
|
|
3018 (latin-9 . latin-iso8859-15)
|
|
|
3019 (latin-10 . latin-iso8859-16))
|
|
|
3020 @end example
|
|
|
3021
|
|
|
3022 If a charset does not exist on your system, it will not complete and you
|
|
|
3023 will not be able to enter it in response to prompts. A real charset
|
|
|
3024 with the same name as an alias in this list will shadow the alias.
|
|
|
3025 @end defopt
|
|
|
3026
|
|
|
3027 @defopt unity-coding-system-alias-alist nil
|
|
|
3028 Alist mapping aliases to Mule coding system names (symbols).
|
|
|
3029
|
|
|
3030 The default value is @samp{nil}.
|
|
|
3031 @end defopt
|
|
|
3032
|
|
|
3033
|
|
|
3034 @node Theory of Operation, What Unification Cannot Do for You, Configuration, Charset Unification
|
|
|
3035 @subsection Theory of Operation
|
|
|
3036
|
|
|
3037 Standard encodings suffer from the design defect that they do not
|
|
|
3038 provide a reliable way to recognize which coded character sets in use.
|
|
|
3039 @xref{What Unification Cannot Do for You}. There are scores of
|
|
|
3040 character sets which can be represented by a single octet (8-bit byte),
|
|
|
3041 whose union contains many hundreds of characters. Obviously this
|
|
|
3042 results in great confusion, since you can't tell the players without a
|
|
|
3043 scorecard, and there is no scorecard.
|
|
|
3044
|
|
|
3045 There are two ways to solve this problem. The first is to create a
|
|
|
3046 universal coded character set. This is the concept behind Unicode.
|
|
|
3047 However, there have been satisfactory (nearly) universal character sets
|
|
|
3048 for several decades, but even today many Westerners resist using Unicode
|
|
|
3049 because they consider its space requirements excessive. On the other
|
|
|
3050 hand, Asians dislike Unicode because they consider it to be incomplete.
|
|
|
3051 (This is partly, but not entirely, political.)
|
|
|
3052
|
|
|
3053 In any case, Unicode only solves the internal representation problem.
|
|
|
3054 Many data sets will contain files in ``legacy'' encodings, and Unicode
|
|
|
3055 does not help distinguish among them.
|
|
|
3056
|
|
|
3057 The second approach is to embed information about the encodings used in
|
|
|
3058 a document in its text. This approach is taken by the ISO 2022
|
|
|
3059 standard. This would solve the problem completely from the users' of
|
|
|
3060 view, except that ISO 2022 is basically not implemented at all, in the
|
|
|
3061 sense that few applications or systems implement more than a small
|
|
|
3062 subset of ISO 2022 functionality. This is due to the fact that
|
|
|
3063 mono-literate users object to the presence of escape sequences in their
|
|
|
3064 texts (which they, with some justification, consider data corruption).
|
|
|
3065 Programmers are more than willing to cater to these users, since
|
|
|
3066 implementing ISO 2022 is a painstaking task.
|
|
|
3067
|
|
|
3068 In fact, Emacs/Mule adopts both of these approaches. Internally it uses
|
|
|
3069 a universal character set, @dfn{Mule code}. Externally it uses ISO 2022
|
|
|
3070 techniques both to save files in forms robust to encoding issues, and as
|
|
|
3071 hints when attempting to ``guess'' an unknown encoding. However, Mule
|
|
|
3072 suffers from a design defect, namely it embeds the character set
|
|
|
3073 information that ISO 2022 attaches to runs of characters by introducing
|
|
|
3074 them with a control sequence in each character. That causes Mule to
|
|
|
3075 consider the ISO Latin character sets to be disjoint. This manifests
|
|
|
3076 itself when a user enters characters using input methods associated with
|
|
|
3077 different coded character sets into a single buffer.
|
|
|
3078
|
|
|
3079 There are two problems stemming from this design. First, Mule
|
|
1188
|
3080 represents the same character in different ways. Abstractly, 'ó'
|
|
1183
|
3081 (LATIN SMALL LETTER O WITH ACUTE) can get represented as
|
|
|
3082 [latin-iso8859-1 #x73] or as [latin-iso8859-2 #x73]. So what looks like
|
|
1188
|
3083 'óó' in the display might actually be represented [latin-iso8859-1
|
|
1183
|
3084 #x73][latin-iso8859-2 #x73] in the buffer, and saved as [#xF3 ESC - B
|
|
|
3085 #xF3 ESC - A] in the file. In some cases this treatment would be
|
|
|
3086 appropriate (consider HYPHEN, MINUS SIGN, EN DASH, EM DASH, and U+4E00
|
|
|
3087 (the CJK ideographic character meaning ``one'')), and although arguably
|
|
|
3088 incorrect it is convenient when mixing the CJK scripts. But in the case
|
|
|
3089 of the Latin scripts this is wrong.
|
|
|
3090
|
|
|
3091 Worse yet, it is very likely to occur when mixing ``different'' encodings
|
|
|
3092 (such as ISO 8859/1 and ISO 8859/15) that differ only in a few code
|
|
|
3093 points that are almost never used. A very important example involves
|
|
|
3094 email. Many sites, especially in the U.S., default to use of the ISO
|
|
|
3095 8859/1 coded character set (also called ``Latin 1,'' though these are
|
|
|
3096 somewhat different concepts). However, ISO 8859/1 provides a generic
|
|
|
3097 CURRENCY SIGN character. Now that the Euro has become the official
|
|
|
3098 currency of most countries in Europe, this is unsatisfactory (and in
|
|
|
3099 practice, useless). So Europeans generally use ISO 8859/15, which is
|
|
|
3100 nearly identical to ISO 8859/1 for most languages, except that it
|
|
|
3101 substitutes EURO SIGN for CURRENCY SIGN.
|
|
|
3102
|
|
|
3103 Suppose a European user yanks text from a post encoded in ISO 8859/1
|
|
|
3104 into a message composition buffer, and enters some text including the
|
|
|
3105 Euro sign. Then Mule will consider the buffer to contain both ISO
|
|
|
3106 8859/1 and ISO 8859/15 text, and MUAs such as Gnus will (if naively
|
|
|
3107 programmed) send the message as a multipart mixed MIME body!
|
|
|
3108
|
|
|
3109 This is clearly stupid. What is not as obvious is that, just as any
|
|
|
3110 European can include American English in their text because ASCII is a
|
|
|
3111 subset of ISO 8859/15, most European languages which use Latin
|
|
|
3112 characters (eg, German and Polish) can typically be mixed while using
|
|
|
3113 only one Latin coded character set (in the case of German and Polish,
|
|
|
3114 ISO 8859/2). However, this often depends on exactly what text is to be
|
|
|
3115 encoded (even for the same pair of languages).
|
|
|
3116
|
|
|
3117 Unification works around the problem by converting as many characters as
|
|
|
3118 possible to use a single Latin coded character set before saving the
|
|
|
3119 buffer.
|
|
|
3120
|
|
|
3121 Because the problem is rarely noticable in editing a buffer, but tends
|
|
|
3122 to manifest when that buffer is exported to a file or process, the
|
|
|
3123 Unification package uses the strategy of examining the buffer prior to
|
|
|
3124 export. If use of multiple Latin coded character sets is detected,
|
|
|
3125 Unification attempts to unify them by finding a single coded character
|
|
|
3126 set which contains all of the Latin characters in the buffer.
|
|
|
3127
|
|
|
3128 The primary purpose of Unification is to fix the problem by giving the
|
|
|
3129 user the choice to change the representation of all characters to one
|
|
|
3130 character set and give sensible recommendations based on context. In
|
|
1188
|
3131 the 'ó' example, either ISO 8859/1 or ISO 8859/2 is satisfactory, and
|
|
1183
|
3132 both will be suggested. In the EURO SIGN example, only ISO 8859/15
|
|
|
3133 makes sense, and that is what will be recommended. In both cases, the
|
|
|
3134 user will be reminded that there are universal encodings available.
|
|
|
3135
|
|
|
3136 I call this @dfn{remapping} (from the universal character set to a
|
|
|
3137 particular ISO 8859 coded character set). It is mere accident that this
|
|
|
3138 letter has the same code point in both character sets. (Not entirely,
|
|
|
3139 but there are many examples of Latin characters that have different code
|
|
|
3140 points in different Latin-X sets.)
|
|
|
3141
|
|
1188
|
3142 Note that, in the 'ó' example, that treating the buffer in this way will
|
|
1183
|
3143 result in a representation such as [latin-iso8859-2
|
|
|
3144 #x73][latin-iso8859-2 #x73], and the file will be saved as [#xF3 #xF3].
|
|
|
3145 This is guaranteed to occasionally result in the second problem you
|
|
|
3146 observed, to which we now turn.
|
|
|
3147
|
|
|
3148 This problem is that, although the file is intended to be an
|
|
|
3149 ISO-8859/2-encoded file, in an ISO 8859/1 locale Mule (and every POSIX
|
|
|
3150 compliant program---this is required by the standard, obvious if you
|
|
|
3151 think a bit, @pxref{What Unification Cannot Do for You}) will read that
|
|
|
3152 file as [latin-iso8859-1 #x73] [latin-iso8859-1 #x73]. Of course this
|
|
|
3153 is no problem if all of the characters in the file are contained in ISO
|
|
|
3154 8859/1, but suppose there are some which are not, but are contained in
|
|
|
3155 the (intended) ISO 8859/2.
|
|
|
3156
|
|
|
3157 You now want to fix this, but not by finding the same character in
|
|
|
3158 another set. Instead, you want to simply change the character set that
|
|
|
3159 Mule associates with that buffer position without changing the code.
|
|
|
3160 (This is conceptually somewhat distinct from the first problem, and
|
|
|
3161 logically ought to be handled in the code that defines coding systems.
|
|
|
3162 However, unification is not an unreasonable place for it.) Unification
|
|
|
3163 provides two functions (one fast and dangerous, the other slow and
|
|
|
3164 careful) to handle this. I call this @dfn{recoding}, because the
|
|
|
3165 transformation actually involves @emph{encoding} the buffer to file
|
|
|
3166 representation, then @emph{decoding} it to buffer representation (in a
|
|
|
3167 different character set). This cannot be done automatically because
|
|
|
3168 Mule can have no idea what the correct encoding is---after all, it
|
|
|
3169 already gave you its best guess. @xref{What Unification Cannot Do for
|
|
|
3170 You}. So these functions must be invoked by the user. @xref{Interactive
|
|
|
3171 Usage}.
|
|
|
3172
|
|
|
3173
|
|
|
3174 @node What Unification Cannot Do for You, Unification Internals, Theory of Operation, Charset Unification
|
|
|
3175 @subsection What Unification Cannot Do for You
|
|
|
3176
|
|
|
3177 Unification @strong{cannot} save you if you insist on exporting data in
|
|
|
3178 8-bit encodings in a multilingual environment. @emph{You will
|
|
|
3179 eventually corrupt data if you do this.} It is not Mule's, or any
|
|
|
3180 application's, fault. You will have only yourself to blame; consider
|
|
|
3181 yourself warned. (It is true that Mule has bugs, which make Mule
|
|
|
3182 somewhat more dangerous and inconvenient than some naive applications.
|
|
|
3183 We're working to address those, but no application can remedy the
|
|
|
3184 inherent defect of 8-bit encodings.)
|
|
|
3185
|
|
|
3186 Use standard universal encodings, preferably Unicode (UTF-8) unless
|
|
|
3187 applicable standards indicate otherwise. The most important such case
|
|
|
3188 is Internet messages, where MIME should be used, whether or not the
|
|
|
3189 subordinate encoding is a universal encoding. (Note that since one of
|
|
|
3190 the important provisions of MIME is the @samp{Content-Type} header,
|
|
|
3191 which has the charset parameter, MIME is to be considered a universal
|
|
|
3192 encoding for the purposes of this manual. Of course, technically
|
|
|
3193 speaking it's neither a coded character set nor a coding extension
|
|
|
3194 technique compliant with ISO 2022.)
|
|
|
3195
|
|
|
3196 As mentioned earlier, the problem is that standard encodings suffer from
|
|
|
3197 the design defect that they do not provide a reliable way to recognize
|
|
|
3198 which coded character sets are in use. There are scores of character
|
|
|
3199 sets which can be represented by a single octet (8-bit byte), whose
|
|
|
3200 union contains many hundreds of characters. Thus any 8-bit coded
|
|
|
3201 character set must contain characters that share code points used for
|
|
|
3202 different characters in other coded character sets.
|
|
|
3203
|
|
|
3204 This means that a given file's intended encoding cannot be identified
|
|
|
3205 with 100% reliability unless it contains encoding markers such as those
|
|
|
3206 provided by MIME or ISO 2022.
|
|
|
3207
|
|
|
3208 Unification actually makes it more likely that you will have problems of
|
|
|
3209 this kind. Traditionally Mule has been ``helpful'' by simply using an
|
|
|
3210 ISO 2022 universal coding system when the current buffer coding system
|
|
|
3211 cannot handle all the characters in the buffer. This has the effect
|
|
|
3212 that, because the file contains control sequences, it is not recognized
|
|
|
3213 as being in the locale's normal 8-bit encoding. It may be annoying if
|
|
|
3214 you are not a Mule expert, but your data is automatically recoverable
|
|
|
3215 with a tool you already have: Mule.
|
|
|
3216
|
|
|
3217 However, with unification, Mule converts to a single 8-bit character set
|
|
|
3218 when possible. But typically this will @emph{not} be in your usual
|
|
|
3219 locale. Ie, the times that an ISO 8859/1 user will need Unification is
|
|
|
3220 when there are ISO 8859/2 characters in the buffer. But then most
|
|
|
3221 likely the file will be saved in a pure 8-bit encoding that is not ISO
|
|
|
3222 8859/1, ie, ISO 8859/2. Mule's autorecognizer (which is probably the
|
|
|
3223 most sophisticated yet available) cannot tell the difference between ISO
|
|
|
3224 8859/1 and ISO 8859/2, and in a Western European locale will choose the
|
|
|
3225 former even though the latter was intended. Even the extension
|
|
|
3226 (``statistical recognition'') planned for XEmacs 22 is unlikely to be at
|
|
|
3227 all accurate in the case of mixed codes.
|
|
|
3228
|
|
|
3229 So now consider adding some additional ISO 8859/1 text to the buffer.
|
|
|
3230 If it includes any ISO 8859/1 codes that are used by different
|
|
|
3231 characters in ISO 8859/2, you now have a file that cannot be
|
|
|
3232 mechanically disentangled. You need a human being who can recognize
|
|
|
3233 that @emph{this is German and Swedish} and stays in Latin-1, while
|
|
|
3234 @emph{that is Polish} and needs to be recoded to Latin-2.
|
|
|
3235
|
|
|
3236 Moral: switch to a universal coded character set, preferably Unicode
|
|
|
3237 using the UTF-8 transformation format. If you really need the space,
|
|
|
3238 compress your files.
|
|
|
3239
|
|
|
3240
|
|
|
3241 @node Unification Internals, , What Unification Cannot Do for You, Charset Unification
|
|
|
3242 @subsection Internals
|
|
|
3243
|
|
|
3244 No internals documentation yet.
|
|
|
3245
|
|
|
3246 @file{unity-utils.el} provides one utility function.
|
|
|
3247
|
|
|
3248 @defun unity-dump-tables
|
|
|
3249
|
|
|
3250 Dump the temporary table created by loading @file{unity-utils.el}
|
|
|
3251 to @file{unity-tables.el}. Loading the latter file initializes
|
|
|
3252 @samp{unity-equivalences}.
|
|
|
3253 @end defun
|
|
|
3254
|
|
|
3255
|
|
|
3256 @node Charsets and Coding Systems, , Charset Unification, MULE
|
|
|
3257 @subsection Charsets and Coding Systems
|
|
|
3258
|
|
|
3259 This section provides reference lists of Mule charsets and coding
|
|
|
3260 systems. Mule charsets are typically named by character set and
|
|
|
3261 standard.
|
|
|
3262
|
|
|
3263 @table @strong
|
|
|
3264 @item ASCII variants
|
|
|
3265
|
|
|
3266 Identification of equivalent characters in these sets is not properly
|
|
|
3267 implemented. Unification does not distinguish the two charsets.
|
|
|
3268
|
|
|
3269 @samp{ascii} @samp{latin-jisx0201}
|
|
|
3270
|
|
|
3271 @item Extended Latin
|
|
|
3272
|
|
|
3273 Characters from the following ISO 2022 conformant charsets are
|
|
|
3274 identified with equivalents in other charsets in the group by
|
|
|
3275 Unification.
|
|
|
3276
|
|
|
3277 @samp{latin-iso8859-1} @samp{latin-iso8859-15} @samp{latin-iso8859-2}
|
|
|
3278 @samp{latin-iso8859-3} @samp{latin-iso8859-4} @samp{latin-iso8859-9}
|
|
|
3279 @samp{latin-iso8859-13} @samp{latin-iso8859-16}
|
|
|
3280
|
|
|
3281 The follow charsets are Latin variants which are not understood by
|
|
|
3282 Unification. In addition, many of the Asian language standards provide
|
|
|
3283 ASCII, at least, and sometimes other Latin characters. None of these
|
|
|
3284 are identified with their ISO 8859 equivalents.
|
|
|
3285
|
|
|
3286 @samp{vietnamese-viscii-lower}
|
|
|
3287 @samp{vietnamese-viscii-upper}
|
|
|
3288
|
|
|
3289 @item Other character sets
|
|
|
3290
|
|
|
3291 @samp{arabic-1-column}
|
|
|
3292 @samp{arabic-2-column}
|
|
|
3293 @samp{arabic-digit}
|
|
|
3294 @samp{arabic-iso8859-6}
|
|
|
3295 @samp{chinese-big5-1}
|
|
|
3296 @samp{chinese-big5-2}
|
|
|
3297 @samp{chinese-cns11643-1}
|
|
|
3298 @samp{chinese-cns11643-2}
|
|
|
3299 @samp{chinese-cns11643-3}
|
|
|
3300 @samp{chinese-cns11643-4}
|
|
|
3301 @samp{chinese-cns11643-5}
|
|
|
3302 @samp{chinese-cns11643-6}
|
|
|
3303 @samp{chinese-cns11643-7}
|
|
|
3304 @samp{chinese-gb2312}
|
|
|
3305 @samp{chinese-isoir165}
|
|
|
3306 @samp{cyrillic-iso8859-5}
|
|
|
3307 @samp{ethiopic}
|
|
|
3308 @samp{greek-iso8859-7}
|
|
|
3309 @samp{hebrew-iso8859-8}
|
|
|
3310 @samp{ipa}
|
|
|
3311 @samp{japanese-jisx0208}
|
|
|
3312 @samp{japanese-jisx0208-1978}
|
|
|
3313 @samp{japanese-jisx0212}
|
|
|
3314 @samp{katakana-jisx0201}
|
|
|
3315 @samp{korean-ksc5601}
|
|
|
3316 @samp{sisheng}
|
|
|
3317 @samp{thai-tis620}
|
|
|
3318 @samp{thai-xtis}
|
|
|
3319
|
|
|
3320 @item Non-graphic charsets
|
|
|
3321
|
|
|
3322 @samp{control-1}
|
|
|
3323 @end table
|
|
|
3324
|
|
|
3325 @table @strong
|
|
|
3326 @item No conversion
|
|
|
3327
|
|
|
3328 Some of these coding systems may specify EOL conventions. Note that
|
|
|
3329 @samp{iso-8859-1} is a no-conversion coding system, not an ISO 2022
|
|
|
3330 coding system. Although unification attempts to compensate for this, it
|
|
|
3331 is possible that the @samp{iso-8859-1} coding system will behave
|
|
|
3332 differently from other ISO 8859 coding systems.
|
|
|
3333
|
|
|
3334 @samp{binary} @samp{no-conversion} @samp{raw-text} @samp{iso-8859-1}
|
|
|
3335
|
|
|
3336 @item Latin coding systems
|
|
|
3337
|
|
|
3338 These coding systems are all single-byte, 8-bit ISO 2022 coding systems,
|
|
|
3339 combining ASCII in the GL register (bytes with high-bit clear) and an
|
|
|
3340 extended Latin character set in the GR register (bytes with high-bit set).
|
|
|
3341
|
|
|
3342 @samp{iso-8859-15} @samp{iso-8859-2} @samp{iso-8859-3} @samp{iso-8859-4}
|
|
|
3343 @samp{iso-8859-9} @samp{iso-8859-13} @samp{iso-8859-14} @samp{iso-8859-16}
|
|
|
3344
|
|
|
3345 These coding systems are single-byte, 8-bit coding systems that do not
|
|
|
3346 conform to international standards. They should be avoided in all
|
|
|
3347 potentially multilingual contexts, including any text distributed over
|
|
|
3348 the Internet and World Wide Web.
|
|
|
3349
|
|
|
3350 @samp{windows-1251}
|
|
|
3351
|
|
|
3352 @item Multilingual coding systems
|
|
|
3353
|
|
|
3354 The following ISO-2022-based coding systems are useful for multilingual
|
|
|
3355 text.
|
|
|
3356
|
|
|
3357 @samp{ctext} @samp{iso-2022-lock} @samp{iso-2022-7} @samp{iso-2022-7bit}
|
|
|
3358 @samp{iso-2022-7bit-ss2} @samp{iso-2022-8} @samp{iso-2022-8bit-ss2}
|
|
|
3359
|
|
|
3360 XEmacs also supports Unicode with the Mule-UCS package. These are the
|
|
|
3361 preferred coding systems for multilingual use. (There is a possible
|
|
|
3362 exception for texts that mix several Asian ideographic character sets.)
|
|
|
3363
|
|
|
3364 @samp{utf-16-be} @samp{utf-16-be-no-signature} @samp{utf-16-le}
|
|
|
3365 @samp{utf-16-le-no-signature} @samp{utf-7} @samp{utf-7-safe}
|
|
|
3366 @samp{utf-8} @samp{utf-8-ws}
|
|
|
3367
|
|
|
3368 Development versions of XEmacs (the 21.5 series) support Unicode
|
|
|
3369 internally, with (at least) the following coding systems implemented:
|
|
|
3370
|
|
|
3371 @samp{utf-16-be} @samp{utf-16-be-bom} @samp{utf-16-le}
|
|
|
3372 @samp{utf-16-le-bom} @samp{utf-8} @samp{utf-8-bom}
|
|
|
3373
|
|
|
3374 @item Asian ideographic languages
|
|
|
3375
|
|
|
3376 The following coding systems are based on ISO 2022, and are more or less
|
|
|
3377 suitable for encoding multilingual texts. They all can represent ASCII
|
|
|
3378 at least, and sometimes several other foreign character sets, without
|
|
|
3379 resort to arbitrary ISO 2022 designations. However, these subsets are
|
|
|
3380 not identified with the corresponding national standards in XEmacs Mule.
|
|
|
3381
|
|
|
3382 @samp{chinese-euc} @samp{cn-big5} @samp{cn-gb-2312} @samp{gb2312}
|
|
|
3383 @samp{hz} @samp{hz-gb-2312} @samp{old-jis} @samp{japanese-euc}
|
|
|
3384 @samp{junet} @samp{euc-japan} @samp{euc-jp} @samp{iso-2022-jp}
|
|
|
3385 @samp{iso-2022-jp-1978-irv} @samp{iso-2022-jp-2} @samp{euc-kr}
|
|
|
3386 @samp{korean-euc} @samp{iso-2022-kr} @samp{iso-2022-int-1}
|
|
|
3387
|
|
|
3388 The following coding systems cannot be used for general multilingual
|
|
|
3389 text and do not cooperate well with other coding systems.
|
|
|
3390
|
|
|
3391 @samp{big5} @samp{shift_jis}
|
|
|
3392
|
|
|
3393 @item Other languages
|
|
|
3394
|
|
|
3395 The following coding systems are based on ISO 2022. Though none of them
|
|
|
3396 provides any Latin characters beyond ASCII, XEmacs Mule allows (and up
|
|
|
3397 to 21.4 defaults to) use of ISO 2022 control sequences to designate
|
|
|
3398 other character sets for inclusion the text.
|
|
|
3399
|
|
|
3400 @samp{iso-8859-5} @samp{iso-8859-7} @samp{iso-8859-8}
|
|
|
3401 @samp{ctext-hebrew}
|
|
|
3402
|
|
|
3403 The following are character sets that do not conform to ISO 2022 and
|
|
|
3404 thus cannot be safely used in a multilingual context.
|
|
|
3405
|
|
|
3406 @samp{alternativnyj} @samp{koi8-r} @samp{tis-620} @samp{viqr}
|
|
|
3407 @samp{viscii} @samp{vscii}
|
|
|
3408
|
|
|
3409 @item Special coding systems
|
|
|
3410
|
|
|
3411 Mule uses the following coding systems for special purposes.
|
|
|
3412
|
|
|
3413 @samp{automatic-conversion} @samp{undecided} @samp{escape-quoted}
|
|
|
3414
|
|
|
3415 @samp{escape-quoted} is especially important, as it is used internally
|
|
|
3416 as the coding system for autosaved data.
|
|
|
3417
|
|
|
3418 The following coding systems are aliases for others, and are used for
|
|
|
3419 communication with the host operating system.
|
|
|
3420
|
|
|
3421 @samp{file-name} @samp{keyboard} @samp{terminal}
|
|
|
3422
|
|
|
3423 @end table
|
|
|
3424
|
|
|
3425 Mule detection of coding systems is actually limited to detection of
|
|
|
3426 classes of coding systems called @dfn{coding categories}. These coding
|
|
|
3427 categories are identified by the ISO 2022 control sequences they use, if
|
|
|
3428 any, by their conformance to ISO 2022 restrictions on code points that
|
|
|
3429 may be used, and by characteristic patterns of use of 8-bit code points.
|
|
|
3430
|
|
|
3431 @samp{no-conversion}
|
|
|
3432 @samp{utf-8}
|
|
|
3433 @samp{ucs-4}
|
|
|
3434 @samp{iso-7}
|
|
|
3435 @samp{iso-lock-shift}
|
|
|
3436 @samp{iso-8-1}
|
|
|
3437 @samp{iso-8-2}
|
|
|
3438 @samp{iso-8-designate}
|
|
|
3439 @samp{shift-jis}
|
|
|
3440 @samp{big5}
|
|
|
3441
|
|
|
3442
|
|
|
3443 @c end of mule.texi
|
|
|
3444
|
