@c -*-texinfo-*-
@c This is part of the XEmacs Lisp Reference Manual.
@c Copyright (C) 1996 Ben Wing, 2001-2002 Free Software Foundation.
@c See the file lispref.texi for copying conditions.
@setfilename ../../info/internationalization.info
@node MULE, Tips, Internationalization, top
@chapter MULE

  @dfn{MULE} is the name originally given to the version of GNU Emacs
extended for multi-lingual (and in particular Asian-language) support.
``MULE'' is short for ``MUlti-Lingual Emacs''.  It is an extension and
complete rewrite of Nemacs (``Nihon Emacs'' where ``Nihon'' is the
Japanese word for ``Japan''), which only provided support for Japanese.
XEmacs refers to its multi-lingual support as @dfn{MULE support} since
it is based on @dfn{MULE}.

@menu
* Internationalization Terminology::
                        Definition of various internationalization terms.
* Charsets::            Sets of related characters.
* MULE Characters::     Working with characters in XEmacs/MULE.
* Composite Characters:: Making new characters by overstriking other ones.
* Coding Systems::      Ways of representing a string of chars using integers.
* CCL::                 A special language for writing fast converters.
* Category Tables::     Subdividing charsets into groups.
* Unicode Support::     The universal coded character set.
* Charset Unification:: Handling overlapping character sets.
* Charsets and Coding Systems:: Tables and reference information.
@end menu

@node Internationalization Terminology, Charsets, , MULE
@section Internationalization Terminology

  In internationalization terminology, a string of text is divided up
into @dfn{characters}, which are the printable units that make up the
text.  A single character is (for example) a capital @samp{A}, the
number @samp{2}, a Katakana character, a Hangul character, a Kanji
ideograph (an @dfn{ideograph} is a ``picture'' character, such as is
used in Japanese Kanji, Chinese Hanzi, and Korean Hanja; typically there
are thousands of such ideographs in each language), etc.  The basic
property of a character is that it is the smallest unit of text with
semantic significance in text processing---i.e., characters are abstract
units defined by their meaning, not by their exact appearance.

  Human beings normally process text visually, so to a first approximation
a character may be identified with its shape.  Note that the same
character may be drawn by two different people (or in two different
fonts) in slightly different ways, although the "basic shape" will be the
same.  But consider the works of Scott Kim; human beings can recognize
hugely variant shapes as the "same" character.  Sometimes, especially
where characters are extremely complicated to write, completely
different shapes may be defined as the "same" character in national
standards.  The Taiwanese variant of Hanzi is generally the most
complicated; over the centuries, the Japanese, Koreans, and the People's
Republic of China have adopted simplifications of the shape, but the
line of descent from the original shape is recorded, and the meanings
and pronunciation of different forms of the same character are
considered to be identical within each language.  (Of course, it may
take a specialist to recognize the related form; the point is that the
relations are standardized, despite the differing shapes.)

  In some cases, the differences will be significant enough that it is
actually possible to identify two or more distinct shapes that both
represent the same character.  For example, the lowercase letters
@samp{a} and @samp{g} each have two distinct possible shapes---the
@samp{a} can optionally have a curved tail projecting off the top, and
the @samp{g} can be formed either of two loops, or of one loop and a
tail hanging off the bottom.  Such distinct possible shapes of a
character are called @dfn{glyphs}.  The important characteristic of two
glyphs making up the same character is that the choice between one or
the other is purely stylistic and has no linguistic effect on a word
(this is the reason why a capital @samp{A} and lowercase @samp{a}
are different characters rather than different glyphs---e.g.
@samp{Aspen} is a city while @samp{aspen} is a kind of tree).

  Note that @dfn{character} and @dfn{glyph} are used differently
here than elsewhere in XEmacs.

  A @dfn{character set} is essentially a set of related characters.  ASCII,
for example, is a set of 94 characters (or 128, if you count
non-printing characters).  Other character sets are ISO8859-1 (ASCII
plus various accented characters and other international symbols),
JIS X 0201 (ASCII, more or less, plus half-width Katakana), JIS X 0208
(Japanese Kanji), JIS X 0212 (a second set of less-used Japanese Kanji),
GB2312 (Mainland Chinese Hanzi), etc.

  The definition of a character set will implicitly or explicitly give
it an @dfn{ordering}, a way of assigning a number to each character in
the set.  For many character sets, there is a natural ordering, for
example the ``ABC'' ordering of the Roman letters.  But it is not clear
whether digits should come before or after the letters, and in fact
different European languages treat the ordering of accented characters
differently.  It is useful to use the natural order where available, of
course.  The number assigned to any particular character is called the
character's @dfn{code point}.  (Within a given character set, each
character has a unique code point.  Thus the word "set" is ill-chosen;
different orderings of the same characters are different character sets.
Identifying characters is simple enough for alphabetic character sets,
but the difference in ordering can cause great headaches when the same
thousands of characters are used by different cultures as in the Hanzi.)

  It's important to understand that a character is defined not by any
number attached to it, but by its meaning.  For example, ASCII and
EBCDIC are two charsets containing exactly the same characters
(lowercase and uppercase letters, numbers 0 through 9, particular
punctuation marks) but with different numberings.  The @samp{comma}
character in ASCII and EBCDIC, for instance, is the same character
despite having a different numbering.  Conversely, when comparing ASCII
and JIS-Roman, which look the same except that the latter has a yen sign
substituted for the backslash, we would say that the backslash and yen
sign are @emph{not} the same characters, despite having the same number
(95) and despite the fact that all other characters are present in both
charsets, with the same numbering.  ASCII and JIS-Roman, then, do
@emph{not} have exactly the same characters in them (ASCII has a
backslash character but no yen-sign character, and vice-versa for
JIS-Roman), unlike ASCII and EBCDIC, even though the numberings in ASCII
and JIS-Roman are closer.

  Sometimes, a code point is not a single number, but instead a group of
numbers, called @dfn{position codes}.  In such cases, the number of
position codes required to index a particular character in a character
set is called the @dfn{dimension} of the character set.  Character sets
indexed by more than one position code typically use byte-sized position
codes.  Small character sets, e.g. ASCII, invariably use a single
position code, but for larger character sets, the choice of whether to
use multiple position codes or a single large (16-bit or 32-bit) number
is arbitrary.  Unicode typically uses a single large number, but
language-specific or "national" character sets often use multiple
(usually two) position codes.  For example, JIS X 0208, i.e. Japanese
Kanji, has thousands of characters, and is of dimension two -- every
character is indexed by two position codes, each in the range 1 through
94.  (This number ``94'' is not a coincidence; it is the same as the
number of printable characters in ASCII, and was chosen so that JIS
characters could be directly encoded using two printable ASCII
characters.)  Note that the choice of the range here is somewhat
arbitrary -- it could just as easily be 0 through 93, 2 through 95, etc.
In fact, the range for JIS position codes (and for other character sets
modeled after it) is often given as range 33 through 126, so as to
directly match ASCII printing characters.

  An @dfn{encoding} is a way of numerically representing characters from
one or more character sets into a stream of like-sized numerical values
called @dfn{words} -- typically 8-bit bytes, but sometimes 16-bit or
32-bit quantities.  In a context where dealing with Japanese motivates
much of XEmacs' design in this area, it's important to clearly
distinguish between charsets and encodings.  For a simple charset like
ASCII, there is only one encoding normally used -- each character is
represented by a single byte, with the same value as its code point.
For more complicated charsets, however, or when a single encoding needs
to represent more than charset, things are not so obvious.  Unicode
version 2, for example, is a large charset with thousands of characters,
each indexed by a 16-bit number, often represented in hex, e.g. 0x05D0
for the Hebrew letter "aleph".  One obvious encoding (actually two
encodings, depending on which of the two possible byte orderings is
chosen) simply uses two bytes per character.  This encoding is
convenient for internal processing of Unicode text; however, it's
incompatible with ASCII, and thus external text (files, e-mail, etc.) 
that is encoded this way is completely uninterpretable by programs
lacking Unicode support.  For this reason, a different, ASCII-compatible
encoding, e.g. UTF-8, is usually used for external text.  UTF-8
represents Unicode characters with one to three bytes (often extended to
six bytes to handle characters with up to 31-bit indices).  Unicode
characters 00 to 7F (identical with ASCII) are directly represented with
one byte, and other characters with two or more bytes, each in the range
80 to FF.  Applications that don't understand Unicode will still be able
to process ASCII characters represented in UTF-8-encoded text, and will
typically ignore (and hopefully preserve) the high-bit characters.

Similarly, Shift-JIS and EUC-JP are different encodings normally used to
encode the same character set(s), these character sets being subsets of
Unicode.  However, the obvious approach of unifying XEmacs' internal
encoding across character sets, as was part of the motivation behind
Unicode, wasn't taken.  This means that characters in these character
sets that are identical to characters in other character sets---for
example, the Greek alphabet is in the large Japanese character sets and
at least one European character set--are unfortunately disjoint. 

  Naive use of code points is also not possible if more than one
character set is to be used in the encoding.  For example, printed
Japanese text typically requires characters from multiple character sets
-- ASCII, JIS X 0208, and JIS X 0212, to be specific.  Each of these is
indexed using one or more position codes in the range 1 through 94 (or
33 through 126), so the position codes could not be used directly or
there would be no way to tell which character was meant.  Different
Japanese encodings handle this differently -- JIS uses special escape
characters to denote different character sets; EUC sets the high bit of
the position codes for JIS X 0208 and JIS X 0212, and puts a special
extra byte before each JIS X 0212 character; etc.

  The encodings described above are all 7-bit or 8-bit encodings.  The
fixed-width Unicode encoding previous described, however, is sometimes
considered to be a 16-bit encoding, in which case the issue of byte
ordering does not come up. (Imagine, for example, that the text is
represented as an array of shorts.) Similarly, Unicode version 3 (which
has characters with indices above 0xFFFF), and other very large
character sets, may be represented internally as 32-bit encodings,
i.e. arrays of ints.  However, it does not make too much sense to talk
about 16-bit or 32-bit encodings for external data, since nowadays 8-bit
data is a universal standard -- the closest you can get is fixed-width
encodings using two or four bytes to encode 16-bit or 32-bit values. (A
"7-bit" encoding is used when it cannot be guaranteed that the high bit
of 8-bit data will be correctly preserved.  Some e-mail gateways, for
example, strip the high bit of text passing through them.  These same
gateways often handle non-printable characters incorrectly, and so 7-bit
encodings usually avoid using bytes with such values.)

  A general method of handling text using multiple character sets
(whether for multilingual text, or simply text in an extremely
complicated single language like Japanese) is defined in the
international standard ISO 2022.  ISO 2022 will be discussed in more
detail later (@pxref{ISO 2022}), but for now suffice it to say that text
needs control functions (at least spacing), and if escape sequences are
to be used, an escape sequence introducer.  It was decided to make all
text streams compatible with ASCII in the sense that the codes 0--31
(and 128-159) would always be control codes, never graphic characters,
and where defined by the character set the @samp{SPC} character would be
assigned code 32, and @samp{DEL} would be assigned 127.  Thus there are
94 code points remaining if 7 bits are used.  This is the reason that
most character sets are defined using position codes in the range 1
through 94.  Then ISO 2022 compatible encodings are produced by shifting
the position codes 1 to 94 into character codes 33 to 126, or (if 8 bit
codes are available) into character codes 161 to 254.

  Encodings are classified as either @dfn{modal} or @dfn{non-modal}.  In
a @dfn{modal encoding}, there are multiple states that the encoding can
be in, and the interpretation of the values in the stream depends on the
current global state of the encoding.  Special values in the encoding,
called @dfn{escape sequences}, are used to change the global state.
JIS, for example, is a modal encoding.  The bytes @samp{ESC $ B}
indicate that, from then on, bytes are to be interpreted as position
codes for JIS X 0208, rather than as ASCII.  This effect is cancelled
using the bytes @samp{ESC ( B}, which mean ``switch from whatever the
current state is to ASCII''.  To switch to JIS X 0212, the escape
sequence @samp{ESC $ ( D}. (Note that here, as is common, the escape
sequences do in fact begin with @samp{ESC}.  This is not necessarily the
case, however.  Some encodings use control characters called "locking
shifts" (effect persists until cancelled) to switch character sets.)

  A @dfn{non-modal encoding} has no global state that extends past the
character currently being interpreted.  EUC, for example, is a
non-modal encoding.  Characters in JIS X 0208 are encoded by setting
the high bit of the position codes, and characters in JIS X 0212 are
encoded by doing the same but also prefixing the character with the
byte 0x8F.

  The advantage of a modal encoding is that it is generally more
space-efficient, and is easily extendible because there are essentially
an arbitrary number of escape sequences that can be created.  The
disadvantage, however, is that it is much more difficult to work with
if it is not being processed in a sequential manner.  In the non-modal
EUC encoding, for example, the byte 0x41 always refers to the letter
@samp{A}; whereas in JIS, it could either be the letter @samp{A}, or
one of the two position codes in a JIS X 0208 character, or one of the
two position codes in a JIS X 0212 character.  Determining exactly which
one is meant could be difficult and time-consuming if the previous
bytes in the string have not already been processed, or impossible if
they are drawn from an external stream that cannot be rewound.

  Non-modal encodings are further divided into @dfn{fixed-width} and
@dfn{variable-width} formats.  A fixed-width encoding always uses
the same number of words per character, whereas a variable-width
encoding does not.  EUC is a good example of a variable-width
encoding: one to three bytes are used per character, depending on
the character set.  16-bit and 32-bit encodings are nearly always
fixed-width, and this is in fact one of the main reasons for using
an encoding with a larger word size.  The advantages of fixed-width
encodings should be obvious.  The advantages of variable-width
encodings are that they are generally more space-efficient and allow
for compatibility with existing 8-bit encodings such as ASCII.  (For
example, in Unicode ASCII characters are simply promoted to a 16-bit
representation.  That means that every ASCII character contains a
@samp{NUL} byte; evidently all of the standard string manipulation
functions will lose badly in a fixed-width Unicode environment.)

  The bytes in an 8-bit encoding are often referred to as @dfn{octets}
rather than simply as bytes.  This terminology dates back to the days
before 8-bit bytes were universal, when some computers had 9-bit bytes,
others had 10-bit bytes, etc.

@node Charsets, MULE Characters, Internationalization Terminology, MULE
@section Charsets

  A @dfn{charset} in MULE is an object that encapsulates a
particular character set as well as an ordering of those characters.
Charsets are permanent objects and are named using symbols, like
faces.

@defun charsetp object
This function returns non-@code{nil} if @var{object} is a charset.
@end defun

@menu
* Charset Properties::          Properties of a charset.
* Basic Charset Functions::     Functions for working with charsets.
* Charset Property Functions::  Functions for accessing charset properties.
* Predefined Charsets::         Predefined charset objects.
@end menu

@node Charset Properties, Basic Charset Functions, , Charsets
@subsection Charset Properties

  Charsets have the following properties:

@table @code
@item name
A symbol naming the charset.  Every charset must have a different name;
this allows a charset to be referred to using its name rather than
the actual charset object.
@item doc-string
A documentation string describing the charset.
@item registry
A regular expression matching the font registry field for this character
set.  For example, both the @code{ascii} and @code{latin-iso8859-1}
charsets use the registry @code{"ISO8859-1"}.  This field is used to
choose an appropriate font when the user gives a general font
specification such as @samp{-*-courier-medium-r-*-140-*}, i.e. a
14-point upright medium-weight Courier font.
@item dimension
Number of position codes used to index a character in the character set.
XEmacs/MULE can only handle character sets of dimension 1 or 2.
This property defaults to 1.
@item chars
Number of characters in each dimension.  In XEmacs/MULE, the only
allowed values are 94 or 96. (There are a couple of pre-defined
character sets, such as ASCII, that do not follow this, but you cannot
define new ones like this.) Defaults to 94.  Note that if the dimension
is 2, the character set thus described is 94x94 or 96x96.
@item columns
Number of columns used to display a character in this charset.
Only used in TTY mode. (Under X, the actual width of a character
can be derived from the font used to display the characters.)
If unspecified, defaults to the dimension. (This is almost
always the correct value, because character sets with dimension 2
are usually ideograph character sets, which need two columns to
display the intricate ideographs.)
@item direction
A symbol, either @code{l2r} (left-to-right) or @code{r2l}
(right-to-left).  Defaults to @code{l2r}.  This specifies the
direction that the text should be displayed in, and will be
left-to-right for most charsets but right-to-left for Hebrew
and Arabic. (Right-to-left display is not currently implemented.)
@item final
Final byte of the standard ISO 2022 escape sequence designating this
charset.  Must be supplied.  Each combination of (@var{dimension},
@var{chars}) defines a separate namespace for final bytes, and each
charset within a particular namespace must have a different final byte.
Note that ISO 2022 restricts the final byte to the range 0x30 - 0x7E if
dimension == 1, and 0x30 - 0x5F if dimension == 2.  Note also that final
bytes in the range 0x30 - 0x3F are reserved for user-defined (not
official) character sets.  For more information on ISO 2022, see @ref{Coding
Systems}.
@item graphic
0 (use left half of font on output) or 1 (use right half of font on
output).  Defaults to 0.  This specifies how to convert the position
codes that index a character in a character set into an index into the
font used to display the character set.  With @code{graphic} set to 0,
position codes 33 through 126 map to font indices 33 through 126; with
it set to 1, position codes 33 through 126 map to font indices 161
through 254 (i.e. the same number but with the high bit set).  For
example, for a font whose registry is ISO8859-1, the left half of the
font (octets 0x20 - 0x7F) is the @code{ascii} charset, while the right
half (octets 0xA0 - 0xFF) is the @code{latin-iso8859-1} charset.
@item ccl-program
A compiled CCL program used to convert a character in this charset into
an index into the font.  This is in addition to the @code{graphic}
property.  If a CCL program is defined, the position codes of a
character will first be processed according to @code{graphic} and
then passed through the CCL program, with the resulting values used
to index the font.

  This is used, for example, in the Big5 character set (used in Taiwan).
This character set is not ISO-2022-compliant, and its size (94x157) does
not fit within the maximum 96x96 size of ISO-2022-compliant character
sets.  As a result, XEmacs/MULE splits it (in a rather complex fashion,
so as to group the most commonly used characters together) into two
charset objects (@code{big5-1} and @code{big5-2}), each of size 94x94,
and each charset object uses a CCL program to convert the modified
position codes back into standard Big5 indices to retrieve a character
from a Big5 font.
@end table

  Most of the above properties can only be set when the charset is
initialized, and cannot be changed later.
@xref{Charset Property Functions}.

@node Basic Charset Functions, Charset Property Functions, Charset Properties, Charsets
@subsection Basic Charset Functions

@defun find-charset charset-or-name
This function retrieves the charset of the given name.  If
@var{charset-or-name} is a charset object, it is simply returned.
Otherwise, @var{charset-or-name} should be a symbol.  If there is no
such charset, @code{nil} is returned.  Otherwise the associated charset
object is returned.
@end defun

@defun get-charset name
This function retrieves the charset of the given name.  Same as
@code{find-charset} except an error is signalled if there is no such
charset instead of returning @code{nil}.
@end defun

@defun charset-list
This function returns a list of the names of all defined charsets.
@end defun

@defun make-charset name doc-string props
This function defines a new character set.  This function is for use
with MULE support.  @var{name} is a symbol, the name by which the
character set is normally referred.  @var{doc-string} is a string
describing the character set.  @var{props} is a property list,
describing the specific nature of the character set.  The recognized
properties are @code{registry}, @code{dimension}, @code{columns},
@code{chars}, @code{final}, @code{graphic}, @code{direction}, and
@code{ccl-program}, as previously described.
@end defun

@defun make-reverse-direction-charset charset new-name
This function makes a charset equivalent to @var{charset} but which goes
in the opposite direction.  @var{new-name} is the name of the new
charset.  The new charset is returned.
@end defun

@defun charset-from-attributes dimension chars final &optional direction
This function returns a charset with the given @var{dimension},
@var{chars}, @var{final}, and @var{direction}.  If @var{direction} is
omitted, both directions will be checked (left-to-right will be returned
if character sets exist for both directions).
@end defun

@defun charset-reverse-direction-charset charset
This function returns the charset (if any) with the same dimension,
number of characters, and final byte as @var{charset}, but which is
displayed in the opposite direction.
@end defun

@node Charset Property Functions, Predefined Charsets, Basic Charset Functions, Charsets
@subsection Charset Property Functions

  All of these functions accept either a charset name or charset object.

@defun charset-property charset prop
This function returns property @var{prop} of @var{charset}.
@xref{Charset Properties}.
@end defun

  Convenience functions are also provided for retrieving individual
properties of a charset.

@defun charset-name charset
This function returns the name of @var{charset}.  This will be a symbol.
@end defun

@defun charset-description charset
This function returns the documentation string of @var{charset}.
@end defun

@defun charset-registry charset
This function returns the registry of @var{charset}.
@end defun

@defun charset-dimension charset
This function returns the dimension of @var{charset}.
@end defun

@defun charset-chars charset
This function returns the number of characters per dimension of
@var{charset}.
@end defun

@defun charset-width charset
This function returns the number of display columns per character (in
TTY mode) of @var{charset}.
@end defun

@defun charset-direction charset
This function returns the display direction of @var{charset}---either
@code{l2r} or @code{r2l}.
@end defun

@defun charset-iso-final-char charset
This function returns the final byte of the ISO 2022 escape sequence
designating @var{charset}.
@end defun

@defun charset-iso-graphic-plane charset
This function returns either 0 or 1, depending on whether the position
codes of characters in @var{charset} map to the left or right half
of their font, respectively.
@end defun

@defun charset-ccl-program charset
This function returns the CCL program, if any, for converting
position codes of characters in @var{charset} into font indices.
@end defun

  The two properties of a charset that can currently be set after the
charset has been created are the CCL program and the font registry.

@defun set-charset-ccl-program charset ccl-program
This function sets the @code{ccl-program} property of @var{charset} to
@var{ccl-program}.
@end defun

@defun set-charset-registry charset registry
This function sets the @code{registry} property of @var{charset} to
@var{registry}.
@end defun

@node Predefined Charsets, , Charset Property Functions, Charsets
@subsection Predefined Charsets

  The following charsets are predefined in the C code.

@example
Name                    Type  Fi Gr Dir Registry
--------------------------------------------------------------
ascii                    94    B  0  l2r ISO8859-1
control-1                94       0  l2r ---
latin-iso8859-1          94    A  1  l2r ISO8859-1
latin-iso8859-2          96    B  1  l2r ISO8859-2
latin-iso8859-3          96    C  1  l2r ISO8859-3
latin-iso8859-4          96    D  1  l2r ISO8859-4
cyrillic-iso8859-5       96    L  1  l2r ISO8859-5
arabic-iso8859-6         96    G  1  r2l ISO8859-6
greek-iso8859-7          96    F  1  l2r ISO8859-7
hebrew-iso8859-8         96    H  1  r2l ISO8859-8
latin-iso8859-9          96    M  1  l2r ISO8859-9
thai-tis620              96    T  1  l2r TIS620
katakana-jisx0201        94    I  1  l2r JISX0201.1976
latin-jisx0201           94    J  0  l2r JISX0201.1976
japanese-jisx0208-1978   94x94 @@  0  l2r JISX0208.1978
japanese-jisx0208        94x94 B  0  l2r JISX0208.19(83|90)
japanese-jisx0212        94x94 D  0  l2r JISX0212
chinese-gb2312           94x94 A  0  l2r GB2312
chinese-cns11643-1       94x94 G  0  l2r CNS11643.1
chinese-cns11643-2       94x94 H  0  l2r CNS11643.2
chinese-big5-1           94x94 0  0  l2r Big5
chinese-big5-2           94x94 1  0  l2r Big5
korean-ksc5601           94x94 C  0  l2r KSC5601
composite                96x96    0  l2r ---
@end example

  The following charsets are predefined in the Lisp code.

@example
Name                     Type  Fi Gr Dir Registry
--------------------------------------------------------------
arabic-digit             94    2  0  l2r MuleArabic-0
arabic-1-column          94    3  0  r2l MuleArabic-1
arabic-2-column          94    4  0  r2l MuleArabic-2
sisheng                  94    0  0  l2r sisheng_cwnn\|OMRON_UDC_ZH
chinese-cns11643-3       94x94 I  0  l2r CNS11643.1
chinese-cns11643-4       94x94 J  0  l2r CNS11643.1
chinese-cns11643-5       94x94 K  0  l2r CNS11643.1
chinese-cns11643-6       94x94 L  0  l2r CNS11643.1
chinese-cns11643-7       94x94 M  0  l2r CNS11643.1
ethiopic                 94x94 2  0  l2r Ethio
ascii-r2l                94    B  0  r2l ISO8859-1
ipa                      96    0  1  l2r MuleIPA
vietnamese-viscii-lower  96    1  1  l2r VISCII1.1
vietnamese-viscii-upper  96    2  1  l2r VISCII1.1
@end example

For all of the above charsets, the dimension and number of columns are
the same.

  Note that ASCII, Control-1, and Composite are handled specially.
This is why some of the fields are blank; and some of the filled-in
fields (e.g. the type) are not really accurate.

@node MULE Characters, Composite Characters, Charsets, MULE
@section MULE Characters

@defun make-char charset arg1 &optional arg2
This function makes a multi-byte character from @var{charset} and octets
@var{arg1} and @var{arg2}.
@end defun

@defun char-charset character
This function returns the character set of char @var{character}.
@end defun

@defun char-octet character &optional n
This function returns the octet (i.e. position code) numbered @var{n}
(should be 0 or 1) of char @var{character}.  @var{n} defaults to 0 if omitted.
@end defun

@defun find-charset-region start end &optional buffer
This function returns a list of the charsets in the region between
@var{start} and @var{end}.  @var{buffer} defaults to the current buffer
if omitted.
@end defun

@defun find-charset-string string
This function returns a list of the charsets in @var{string}.
@end defun

@node Composite Characters, Coding Systems, MULE Characters, MULE
@section Composite Characters

  Composite characters are not yet completely implemented.

@defun make-composite-char string
This function converts a string into a single composite character.  The
character is the result of overstriking all the characters in the
string.
@end defun

@defun composite-char-string character
This function returns a string of the characters comprising a composite
character.
@end defun

@defun compose-region start end &optional buffer
This function composes the characters in the region from @var{start} to
@var{end} in @var{buffer} into one composite character.  The composite
character replaces the composed characters.  @var{buffer} defaults to
the current buffer if omitted.
@end defun

@defun decompose-region start end &optional buffer
This function decomposes any composite characters in the region from
@var{start} to @var{end} in @var{buffer}.  This converts each composite
character into one or more characters, the individual characters out of
which the composite character was formed.  Non-composite characters are
left as-is.  @var{buffer} defaults to the current buffer if omitted.
@end defun

@node Coding Systems, CCL, Composite Characters, MULE
@section Coding Systems

  A coding system is an object that defines how text containing multiple
character sets is encoded into a stream of (typically 8-bit) bytes.  The
coding system is used to decode the stream into a series of characters
(which may be from multiple charsets) when the text is read from a file
or process, and is used to encode the text back into the same format
when it is written out to a file or process.

  For example, many ISO-2022-compliant coding systems (such as Compound
Text, which is used for inter-client data under the X Window System) use
escape sequences to switch between different charsets -- Japanese Kanji,
for example, is invoked with @samp{ESC $ ( B}; ASCII is invoked with
@samp{ESC ( B}; and Cyrillic is invoked with @samp{ESC - L}.  See
@code{make-coding-system} for more information.

  Coding systems are normally identified using a symbol, and the symbol is
accepted in place of the actual coding system object whenever a coding
system is called for. (This is similar to how faces and charsets work.)

@defun coding-system-p object
This function returns non-@code{nil} if @var{object} is a coding system.
@end defun

@menu
* Coding System Types::               Classifying coding systems.
* ISO 2022::                          An international standard for
                                        charsets and encodings.
* EOL Conversion::                    Dealing with different ways of denoting
                                        the end of a line.
* Coding System Properties::          Properties of a coding system.
* Basic Coding System Functions::     Working with coding systems.
* Coding System Property Functions::  Retrieving a coding system's properties.
* Encoding and Decoding Text::        Encoding and decoding text.
* Detection of Textual Encoding::     Determining how text is encoded.
* Big5 and Shift-JIS Functions::      Special functions for these non-standard
                                        encodings.
* Predefined Coding Systems::         Coding systems implemented by MULE.
@end menu

@node Coding System Types, ISO 2022, , Coding Systems
@subsection Coding System Types

  The coding system type determines the basic algorithm XEmacs will use to
decode or encode a data stream.  Character encodings will be converted
to the MULE encoding, escape sequences processed, and newline sequences
converted to XEmacs's internal representation.  There are three basic
classes of coding system type: no-conversion, ISO-2022, and special.

  No conversion allows you to look at the file's internal representation.
Since XEmacs is basically a text editor, "no conversion" does convert
newline conventions by default.  (Use the 'binary coding-system if this
is not desired.)

  ISO 2022 (@pxref{ISO 2022}) is the basic international standard regulating
use of "coded character sets for the exchange of data", ie, text
streams.  ISO 2022 contains functions that make it possible to encode
text streams to comply with restrictions of the Internet mail system and
de facto restrictions of most file systems (eg, use of the separator
character in file names).  Coding systems which are not ISO 2022
conformant can be difficult to handle.  Perhaps more important, they are
not adaptable to multilingual information interchange, with the obvious
exception of ISO 10646 (Unicode).  (Unicode is partially supported by
XEmacs with the addition of the Lisp package ucs-conv.)

  The special class of coding systems includes automatic detection, CCL (a
"little language" embedded as an interpreter, useful for translating
between variants of a single character set), non-ISO-2022-conformant
encodings like Unicode, Shift JIS, and Big5, and MULE internal coding.
(NB: this list is based on XEmacs 21.2.  Terminology may vary slightly
for other versions of XEmacs and for GNU Emacs 20.)

@table @code
@item no-conversion
No conversion, for binary files, and a few special cases of non-ISO-2022
coding systems where conversion is done by hook functions (usually
implemented in CCL).  On output, graphic characters that are not in
ASCII or Latin-1 will be replaced by a @samp{?}. (For a
no-conversion-encoded buffer, these characters will only be present if
you explicitly insert them.)
@item iso2022
Any ISO-2022-compliant encoding.  Among others, this includes JIS (the
Japanese encoding commonly used for e-mail), national variants of EUC
(the standard Unix encoding for Japanese and other languages), and
Compound Text (an encoding used in X11).  You can specify more specific
information about the conversion with the @var{flags} argument.
@item ucs-4
ISO 10646 UCS-4 encoding.  A 31-bit fixed-width superset of Unicode.
@item utf-8
ISO 10646 UTF-8 encoding.  A ``file system safe'' transformation format
that can be used with both UCS-4 and Unicode.
@item undecided
Automatic conversion.  XEmacs attempts to detect the coding system used
in the file.
@item shift-jis
Shift-JIS (a Japanese encoding commonly used in PC operating systems).
@item big5
Big5 (the encoding commonly used for Taiwanese).
@item ccl
The conversion is performed using a user-written pseudo-code program.
CCL (Code Conversion Language) is the name of this pseudo-code.  For
example, CCL is used to map KOI8-R characters (an encoding for Russian
Cyrillic) to ISO8859-5 (the form used internally by MULE).
@item internal
Write out or read in the raw contents of the memory representing the
buffer's text.  This is primarily useful for debugging purposes, and is
only enabled when XEmacs has been compiled with @code{DEBUG_XEMACS} set
(the @samp{--debug} configure option).  @strong{Warning}: Reading in a
file using @code{internal} conversion can result in an internal
inconsistency in the memory representing a buffer's text, which will
produce unpredictable results and may cause XEmacs to crash.  Under
normal circumstances you should never use @code{internal} conversion.
@end table

@node ISO 2022, EOL Conversion, Coding System Types, Coding Systems
@section ISO 2022

  This section briefly describes the ISO 2022 encoding standard.  A more
thorough treatment is available in the original document of ISO
2022 as well as various national standards (such as JIS X 0202).

  Character sets (@dfn{charsets}) are classified into the following four
categories, according to the number of characters in the charset:
94-charset, 96-charset, 94x94-charset, and 96x96-charset.  This means
that although an ISO 2022 coding system may have variable width
characters, each charset used is fixed-width (in contrast to the MULE
character set and UTF-8, for example).

  ISO 2022 provides for switching between character sets via escape
sequences.  This switching is somewhat complicated, because ISO 2022
provides for both legacy applications like Internet mail that accept
only 7 significant bits in some contexts (RFC 822 headers, for example),
and more modern "8-bit clean" applications.  It also provides for
compact and transparent representation of languages like Japanese which
mix ASCII and a national script (even outside of computer programs).

  First, ISO 2022 codified prevailing practice by dividing the code space
into "control" and "graphic" regions.  The code points 0x00-0x1F and
0x80-0x9F are reserved for "control characters", while "graphic
characters" must be assigned to code points in the regions 0x20-0x7F and
0xA0-0xFF.  The positions 0x20 and 0x7F are special, and under some
circumstances must be assigned the graphic character "ASCII SPACE" and
the control character "ASCII DEL" respectively.

  The various regions are given the name C0 (0x00-0x1F), GL (0x20-0x7F),
C1 (0x80-0x9F), and GR (0xA0-0xFF).  GL and GR stand for "graphic left"
and "graphic right", respectively, because of the standard method of
displaying graphic character sets in tables with the high byte indexing
columns and the low byte indexing rows.  I don't find it very intuitive,
but these are called "registers".

  An ISO 2022-conformant encoding for a graphic character set must use a
fixed number of bytes per character, and the values must fit into a
single register; that is, each byte must range over either 0x20-0x7F, or
0xA0-0xFF.  It is not allowed to extend the range of the repertoire of a
character set by using both ranges at the same.  This is why a standard
character set such as ISO 8859-1 is actually considered by ISO 2022 to
be an aggregation of two character sets, ASCII and LATIN-1, and why it
is technically incorrect to refer to ISO 8859-1 as "Latin 1".  Also, a
single character's bytes must all be drawn from the same register; this
is why Shift JIS (for Japanese) and Big 5 (for Chinese) are not ISO
2022-compatible encodings.

  The reason for this restriction becomes clear when you attempt to define
an efficient, robust encoding for a language like Japanese.  Like ISO
8859, Japanese encodings are aggregations of several character sets.  In
practice, the vast majority of characters are drawn from the "JIS Roman"
character set (a derivative of ASCII; it won't hurt to think of it as
ASCII) and the JIS X 0208 standard "basic Japanese" character set
including not only ideographic characters ("kanji") but syllabic
Japanese characters ("kana"), a wide variety of symbols, and many
alphabetic characters (Roman, Greek, and Cyrillic) as well.  Although
JIS X 0208 includes the whole Roman alphabet, as a 2-byte code it is not
suited to programming; thus the inclusion of ASCII in the standard
Japanese encodings.

  For normal Japanese text such as in newspapers, a broad repertoire of
approximately 3000 characters is used.  Evidently this won't fit into
one byte; two must be used.  But much of the text processed by Japanese
computers is computer source code, nearly all of which is ASCII.  A not
insignificant portion of ordinary text is English (as such or as
borrowed Japanese vocabulary) or other languages which can represented
at least approximately in ASCII, as well.  It seems reasonable then to
represent ASCII in one byte, and JIS X 0208 in two.  And this is exactly
what the Extended Unix Code for Japanese (EUC-JP) does.  ASCII is
invoked to the GL register, and JIS X 0208 is invoked to the GR
register.  Thus, each byte can be tested for its character set by
looking at the high bit; if set, it is Japanese, if clear, it is ASCII.
Furthermore, since control characters like newline can never be part of
a graphic character, even in the case of corruption in transmission the
stream will be resynchronized at every line break, on the order of 60-80
bytes.  This coding system requires no escape sequences or special
control codes to represent 99.9% of all Japanese text.

  Note carefully the distinction between the character sets (ASCII and JIS
X 0208), the encoding (EUC-JP), and the coding system (ISO 2022).  The
JIS X 0208 character set is used in three different encodings for
Japanese, but in ISO-2022-JP it is invoked into GL (so the high bit is
always clear), in EUC-JP it is invoked into GR (setting the high bit in
the process), and in Shift JIS the high bit may be set or reset, and the
significant bits are shifted within the 16-bit character so that the two
main character sets can coexist with a third (the "halfwidth katakana"
of JIS X 0201).  As the name implies, the ISO-2022-JP encoding is also a
version of the ISO-2022 coding system.

  In order to systematically treat subsidiary character sets (like the
"halfwidth katakana" already mentioned, and the "supplementary kanji" of
JIS X 0212), four further registers are defined: G0, G1, G2, and G3.
Unlike GL and GR, they are not logically distinguished by internal
format.  Instead, the process of "invocation" mentioned earlier is
broken into two steps: first, a character set is @dfn{designated} to one
of the registers G0-G3 by use of an @dfn{escape sequence} of the form:

@example
        ESC [@var{I}] @var{I} @var{F}
@end example

where @var{I} is an intermediate character or characters in the range
0x20 - 0x3F, and @var{F}, from the range 0x30-0x7Fm is the final
character identifying this charset.  (Final characters in the range
0x30-0x3F are reserved for private use and will never have a publicly
registered meaning.)

  Then that register is @dfn{invoked} to either GL or GR, either
automatically (designations to G0 normally involve invocation to GL as
well), or by use of shifting (affecting only the following character in
the data stream) or locking (effective until the next designation or
locking) control sequences.  An encoding conformant to ISO 2022 is
typically defined by designating the initial contents of the G0-G3
registers, specifying a 7 or 8 bit environment, and specifying whether
further designations will be recognized.

  Some examples of character sets and the registered final characters
@var{F} used to designate them:

@need 1000
@table @asis
@item 94-charset
 ASCII (B), left (J) and right (I) half of JIS X 0201, ...
@item 96-charset
 Latin-1 (A), Latin-2 (B), Latin-3 (C), ...
@item 94x94-charset
 GB2312 (A), JIS X 0208 (B), KSC5601 (C), ...
@item 96x96-charset
 none for the moment
@end table

  The meanings of the various characters in these sequences, where not
specified by the ISO 2022 standard (such as the ESC character), are
assigned by @dfn{ECMA}, the European Computer Manufacturers Association.

  The meaning of intermediate characters are:

@example
@group
        $ [0x24]: indicate charset of dimension 2 (94x94 or 96x96).
        ( [0x28]: designate to G0 a 94-charset whose final byte is @var{F}.
        ) [0x29]: designate to G1 a 94-charset whose final byte is @var{F}.
        * [0x2A]: designate to G2 a 94-charset whose final byte is @var{F}.
        + [0x2B]: designate to G3 a 94-charset whose final byte is @var{F}.
        , [0x2C]: designate to G0 a 96-charset whose final byte is @var{F}.
        - [0x2D]: designate to G1 a 96-charset whose final byte is @var{F}.
        . [0x2E]: designate to G2 a 96-charset whose final byte is @var{F}.
        / [0x2F]: designate to G3 a 96-charset whose final byte is @var{F}.
@end group
@end example

  The comma may be used in files read and written only by MULE, as a MULE
extension, but this is illegal in ISO 2022.  (The reason is that in ISO
2022 G0 must be a 94-member character set, with 0x20 assigned the value
SPACE, and 0x7F assigned the value DEL.)

  Here are examples of designations:

@example
@group
        ESC ( B :              designate to G0 ASCII
        ESC - A :              designate to G1 Latin-1
        ESC $ ( A or ESC $ A : designate to G0 GB2312
        ESC $ ( B or ESC $ B : designate to G0 JISX0208
        ESC $ ) C :            designate to G1 KSC5601
@end group
@end example

(The short forms used to designate GB2312 and JIS X 0208 are for
backwards compatibility; the long forms are preferred.)

  To use a charset designated to G2 or G3, and to use a charset designated
to G1 in a 7-bit environment, you must explicitly invoke G1, G2, or G3
into GL.  There are two types of invocation, Locking Shift (forever) and
Single Shift (one character only).

  Locking Shift is done as follows:

@example
        LS0 or SI (0x0F): invoke G0 into GL
        LS1 or SO (0x0E): invoke G1 into GL
        LS2:  invoke G2 into GL
        LS3:  invoke G3 into GL
        LS1R: invoke G1 into GR
        LS2R: invoke G2 into GR
        LS3R: invoke G3 into GR
@end example

  Single Shift is done as follows:

@example
@group
        SS2 or ESC N: invoke G2 into GL
        SS3 or ESC O: invoke G3 into GL
@end group
@end example

  The shift functions (such as LS1R and SS3) are represented by control
characters (from C1) in 8 bit environments and by escape sequences in 7
bit environments.

(#### Ben says: I think the above is slightly incorrect.  It appears that
SS2 invokes G2 into GR and SS3 invokes G3 into GR, whereas ESC N and
ESC O behave as indicated.  The above definitions will not parse
EUC-encoded text correctly, and it looks like the code in mule-coding.c
has similar problems.)

  Evidently there are a lot of ISO-2022-compliant ways of encoding
multilingual text.  Now, in the world, there exist many coding systems
such as X11's Compound Text, Japanese JUNET code, and so-called EUC
(Extended UNIX Code); all of these are variants of ISO 2022.

  In MULE, we characterize a version of ISO 2022 by the following
attributes:

@enumerate
@item
The character sets initially designated to G0 thru G3.
@item
Whether short form designations are allowed for Japanese and Chinese.
@item
Whether ASCII should be designated to G0 before control characters.
@item
Whether ASCII should be designated to G0 at the end of line.
@item
7-bit environment or 8-bit environment.
@item
Whether Locking Shifts are used or not.
@item
Whether to use ASCII or the variant JIS X 0201-1976-Roman.
@item
Whether to use JIS X 0208-1983 or the older version JIS X 0208-1976.
@end enumerate

(The last two are only for Japanese.)

  By specifying these attributes, you can create any variant
of ISO 2022.

  Here are several examples:

@example
@group
ISO-2022-JP -- Coding system used in Japanese email (RFC 1463 #### check).
        1. G0 <- ASCII, G1..3 <- never used
        2. Yes.
        3. Yes.
        4. Yes.
        5. 7-bit environment
        6. No.
        7. Use ASCII
        8. Use JIS X 0208-1983
@end group

@group
ctext -- X11 Compound Text
        1. G0 <- ASCII, G1 <- Latin-1, G2,3 <- never used.
        2. No.
        3. No.
        4. Yes.
        5. 8-bit environment.
        6. No.
        7. Use ASCII.
        8. Use JIS X 0208-1983.
@end group

@group
euc-china -- Chinese EUC.  Often called the "GB encoding", but that is
technically incorrect.
        1. G0 <- ASCII, G1 <- GB 2312, G2,3 <- never used.
        2. No.
        3. Yes.
        4. Yes.
        5. 8-bit environment.
        6. No.
        7. Use ASCII.
        8. Use JIS X 0208-1983.
@end group

@group
ISO-2022-KR -- Coding system used in Korean email.
        1. G0 <- ASCII, G1 <- KSC 5601, G2,3 <- never used.
        2. No.
        3. Yes.
        4. Yes.
        5. 7-bit environment.
        6. Yes.
        7. Use ASCII.
        8. Use JIS X 0208-1983.
@end group
@end example

MULE creates all of these coding systems by default.

@node EOL Conversion, Coding System Properties, ISO 2022, Coding Systems
@subsection EOL Conversion

@table @code
@item nil
Automatically detect the end-of-line type (LF, CRLF, or CR).  Also
generate subsidiary coding systems named @code{@var{name}-unix},
@code{@var{name}-dos}, and @code{@var{name}-mac}, that are identical to
this coding system but have an EOL-TYPE value of @code{lf}, @code{crlf},
and @code{cr}, respectively.
@item lf
The end of a line is marked externally using ASCII LF.  Since this is
also the way that XEmacs represents an end-of-line internally,
specifying this option results in no end-of-line conversion.  This is
the standard format for Unix text files.
@item crlf
The end of a line is marked externally using ASCII CRLF.  This is the
standard format for MS-DOS text files.
@item cr
The end of a line is marked externally using ASCII CR.  This is the
standard format for Macintosh text files.
@item t
Automatically detect the end-of-line type but do not generate subsidiary
coding systems.  (This value is converted to @code{nil} when stored
internally, and @code{coding-system-property} will return @code{nil}.)
@end table

@node Coding System Properties, Basic Coding System Functions, EOL Conversion, Coding Systems
@subsection Coding System Properties

@table @code
@item mnemonic
String to be displayed in the modeline when this coding system is
active.

@item eol-type
End-of-line conversion to be used.  It should be one of the types
listed in @ref{EOL Conversion}.

@item eol-lf
The coding system which is the same as this one, except that it uses the
Unix line-breaking convention.

@item eol-crlf
The coding system which is the same as this one, except that it uses the
DOS line-breaking convention.

@item eol-cr
The coding system which is the same as this one, except that it uses the
Macintosh line-breaking convention.

@item post-read-conversion
Function called after a file has been read in, to perform the decoding.
Called with two arguments, @var{start} and @var{end}, denoting a region of
the current buffer to be decoded.

@item pre-write-conversion
Function called before a file is written out, to perform the encoding.
Called with two arguments, @var{start} and @var{end}, denoting a region of
the current buffer to be encoded.
@end table

  The following additional properties are recognized if @var{type} is
@code{iso2022}:

@table @code
@item charset-g0
@itemx charset-g1
@itemx charset-g2
@itemx charset-g3
The character set initially designated to the G0 - G3 registers.
The value should be one of

@itemize @bullet
@item
A charset object (designate that character set)
@item
@code{nil} (do not ever use this register)
@item
@code{t} (no character set is initially designated to the register, but
may be later on; this automatically sets the corresponding
@code{force-g*-on-output} property)
@end itemize

@item force-g0-on-output
@itemx force-g1-on-output
@itemx force-g2-on-output
@itemx force-g3-on-output
If non-@code{nil}, send an explicit designation sequence on output
before using the specified register.

@item short
If non-@code{nil}, use the short forms @samp{ESC $ @@}, @samp{ESC $ A},
and @samp{ESC $ B} on output in place of the full designation sequences
@samp{ESC $ ( @@}, @samp{ESC $ ( A}, and @samp{ESC $ ( B}.

@item no-ascii-eol
If non-@code{nil}, don't designate ASCII to G0 at each end of line on
output.  Setting this to non-@code{nil} also suppresses other
state-resetting that normally happens at the end of a line.

@item no-ascii-cntl
If non-@code{nil}, don't designate ASCII to G0 before control chars on
output.

@item seven
If non-@code{nil}, use 7-bit environment on output.  Otherwise, use 8-bit
environment.

@item lock-shift
If non-@code{nil}, use locking-shift (SO/SI) instead of single-shift or
designation by escape sequence.

@item no-iso6429
If non-@code{nil}, don't use ISO6429's direction specification.

@item escape-quoted
If non-@code{nil}, literal control characters that are the same as the
beginning of a recognized ISO 2022 or ISO 6429 escape sequence (in
particular, ESC (0x1B), SO (0x0E), SI (0x0F), SS2 (0x8E), SS3 (0x8F),
and CSI (0x9B)) are ``quoted'' with an escape character so that they can
be properly distinguished from an escape sequence.  (Note that doing
this results in a non-portable encoding.) This encoding flag is used for
byte-compiled files.  Note that ESC is a good choice for a quoting
character because there are no escape sequences whose second byte is a
character from the Control-0 or Control-1 character sets; this is
explicitly disallowed by the ISO 2022 standard.

@item input-charset-conversion
A list of conversion specifications, specifying conversion of characters
in one charset to another when decoding is performed.  Each
specification is a list of two elements: the source charset, and the
destination charset.

@item output-charset-conversion
A list of conversion specifications, specifying conversion of characters
in one charset to another when encoding is performed.  The form of each
specification is the same as for @code{input-charset-conversion}.
@end table

  The following additional properties are recognized (and required) if
@var{type} is @code{ccl}:

@table @code
@item decode
CCL program used for decoding (converting to internal format).

@item encode
CCL program used for encoding (converting to external format).
@end table

  The following properties are used internally:  @var{eol-cr},
@var{eol-crlf}, @var{eol-lf}, and @var{base}.

@node Basic Coding System Functions, Coding System Property Functions, Coding System Properties, Coding Systems
@subsection Basic Coding System Functions

@defun find-coding-system coding-system-or-name
This function retrieves the coding system of the given name.

  If @var{coding-system-or-name} is a coding-system object, it is simply
returned.  Otherwise, @var{coding-system-or-name} should be a symbol.
If there is no such coding system, @code{nil} is returned.  Otherwise
the associated coding system object is returned.
@end defun

@defun get-coding-system name
This function retrieves the coding system of the given name.  Same as
@code{find-coding-system} except an error is signalled if there is no
such coding system instead of returning @code{nil}.
@end defun

@defun coding-system-list
This function returns a list of the names of all defined coding systems.
@end defun

@defun coding-system-name coding-system
This function returns the name of the given coding system.
@end defun

@defun coding-system-base coding-system
Returns the base coding system (undecided EOL convention)
coding system.
@end defun

@defun make-coding-system name type &optional doc-string props
This function registers symbol @var{name} as a coding system.

@var{type} describes the conversion method used and should be one of
the types listed in @ref{Coding System Types}.

@var{doc-string} is a string describing the coding system.

@var{props} is a property list, describing the specific nature of the
character set.  Recognized properties are as in @ref{Coding System
Properties}.
@end defun

@defun copy-coding-system old-coding-system new-name
This function copies @var{old-coding-system} to @var{new-name}.  If
@var{new-name} does not name an existing coding system, a new one will
be created.
@end defun

@defun subsidiary-coding-system coding-system eol-type
This function returns the subsidiary coding system of
@var{coding-system} with eol type @var{eol-type}.
@end defun

@node Coding System Property Functions, Encoding and Decoding Text, Basic Coding System Functions, Coding Systems
@subsection Coding System Property Functions

@defun coding-system-doc-string coding-system
This function returns the doc string for @var{coding-system}.
@end defun

@defun coding-system-type coding-system
This function returns the type of @var{coding-system}.
@end defun

@defun coding-system-property coding-system prop
This function returns the @var{prop} property of @var{coding-system}.
@end defun

@node Encoding and Decoding Text, Detection of Textual Encoding, Coding System Property Functions, Coding Systems
@subsection Encoding and Decoding Text

@defun decode-coding-region start end coding-system &optional buffer
This function decodes the text between @var{start} and @var{end} which
is encoded in @var{coding-system}.  This is useful if you've read in
encoded text from a file without decoding it (e.g. you read in a
JIS-formatted file but used the @code{binary} or @code{no-conversion} coding
system, so that it shows up as @samp{^[$B!<!+^[(B}).  The length of the
encoded text is returned.  @var{buffer} defaults to the current buffer
if unspecified.
@end defun

@defun encode-coding-region start end coding-system &optional buffer
This function encodes the text between @var{start} and @var{end} using
@var{coding-system}.  This will, for example, convert Japanese
characters into stuff such as @samp{^[$B!<!+^[(B} if you use the JIS
encoding.  The length of the encoded text is returned.  @var{buffer}
defaults to the current buffer if unspecified.
@end defun

@node Detection of Textual Encoding, Big5 and Shift-JIS Functions, Encoding and Decoding Text, Coding Systems
@subsection Detection of Textual Encoding

@defun coding-category-list
This function returns a list of all recognized coding categories.
@end defun

@defun set-coding-priority-list list
This function changes the priority order of the coding categories.
@var{list} should be a list of coding categories, in descending order of
priority.  Unspecified coding categories will be lower in priority than
all specified ones, in the same relative order they were in previously.
@end defun

@defun coding-priority-list
This function returns a list of coding categories in descending order of
priority.
@end defun

@defun set-coding-category-system coding-category coding-system
This function changes the coding system associated with a coding category.
@end defun

@defun coding-category-system coding-category
This function returns the coding system associated with a coding category.
@end defun

@defun detect-coding-region start end &optional buffer
This function detects coding system of the text in the region between
@var{start} and @var{end}.  Returned value is a list of possible coding
systems ordered by priority.  If only ASCII characters are found, it
returns @code{autodetect} or one of its subsidiary coding systems
according to a detected end-of-line type.  Optional arg @var{buffer}
defaults to the current buffer.
@end defun

@node Big5 and Shift-JIS Functions, Predefined Coding Systems, Detection of Textual Encoding, Coding Systems
@subsection Big5 and Shift-JIS Functions

  These are special functions for working with the non-standard
Shift-JIS and Big5 encodings.

@defun decode-shift-jis-char code
This function decodes a JIS X 0208 character of Shift-JIS coding-system.
@var{code} is the character code in Shift-JIS as a cons of type bytes.
The corresponding character is returned.
@end defun

@defun encode-shift-jis-char character
This function encodes a JIS X 0208 character @var{character} to
SHIFT-JIS coding-system.  The corresponding character code in SHIFT-JIS
is returned as a cons of two bytes.
@end defun

@defun decode-big5-char code
This function decodes a Big5 character @var{code} of BIG5 coding-system.
@var{code} is the character code in BIG5.  The corresponding character
is returned.
@end defun

@defun encode-big5-char character
This function encodes the Big5 character @var{character} to BIG5
coding-system.  The corresponding character code in Big5 is returned.
@end defun

@node Predefined Coding Systems, , Big5 and Shift-JIS Functions, Coding Systems
@subsection Coding Systems Implemented

  MULE initializes most of the commonly used coding systems at XEmacs's
startup.  A few others are initialized only when the relevant language
environment is selected and support libraries are loaded.  (NB: The
following list is based on XEmacs 21.2.19, the development branch at the
time of writing.  The list may be somewhat different for other
versions.  Recent versions of GNU Emacs 20 implement a few more rare
coding systems; work is being done to port these to XEmacs.)

  Unfortunately, there is not a consistent naming convention for character
sets, and for practical purposes coding systems often take their name
from their principal character sets (ASCII, KOI8-R, Shift JIS).  Others
take their names from the coding system (ISO-2022-JP, EUC-KR), and a few
from their non-text usages (internal, binary).  To provide for this, and
for the fact that many coding systems have several common names, an
aliasing system is provided.  Finally, some effort has been made to use
names that are registered as MIME charsets (this is why the name
'shift_jis contains that un-Lisp-y underscore).

  There is a systematic naming convention regarding end-of-line (EOL)
conventions for different systems.  A coding system whose name ends in
"-unix" forces the assumptions that lines are broken by newlines (0x0A).
A coding system whose name ends in "-mac" forces the assumptions that
lines are broken by ASCII CRs (0x0D).  A coding system whose name ends
in "-dos" forces the assumptions that lines are broken by CRLF sequences
(0x0D 0x0A).  These subsidiary coding systems are automatically derived
from a base coding system.  Use of the base coding system implies
autodetection of the text file convention.  (The fact that the -unix,
-mac, and -dos are derived from a base system results in them showing up
as "aliases" in `list-coding-systems'.)  These subsidiaries have a
consistent modeline indicator as well.  "-dos" coding systems have ":T"
appended to their modeline indicator, while "-mac" coding systems have
":t" appended (eg, "ISO8:t" for iso-2022-8-mac).

  In the following table, each coding system is given with its mode line
indicator in parentheses.  Non-textual coding systems are listed first,
followed by textual coding systems and their aliases. (The coding system
subsidiary modeline indicators ":T" and ":t" will be omitted from the
table of coding systems.)

  ### SJT 1999-08-23 Maybe should order these by language?  Definitely
need language usage for the ISO-8859 family.

  Note that although true coding system aliases have been implemented for
XEmacs 21.2, the coding system initialization has not yet been converted
as of 21.2.19.  So coding systems described as aliases have the same
properties as the aliased coding system, but will not be equal as Lisp
objects.

@table @code

@item automatic-conversion
@itemx undecided
@itemx undecided-dos
@itemx undecided-mac
@itemx undecided-unix

Modeline indicator: @code{Auto}.  A type @code{undecided} coding system.
Attempts to determine an appropriate coding system from file contents or
the environment.

@item raw-text
@itemx no-conversion
@itemx raw-text-dos
@itemx raw-text-mac
@itemx raw-text-unix
@itemx no-conversion-dos
@itemx no-conversion-mac
@itemx no-conversion-unix

Modeline indicator: @code{Raw}.  A type @code{no-conversion} coding system,
which converts only line-break-codes.  An implementation quirk means
that this coding system is also used for ISO8859-1.

@item binary
Modeline indicator: @code{Binary}.  A type @code{no-conversion} coding
system which does no character coding or EOL conversions.  An alias for
@code{raw-text-unix}.

@item alternativnyj
@itemx alternativnyj-dos
@itemx alternativnyj-mac
@itemx alternativnyj-unix

Modeline indicator: @code{Cy.Alt}.  A type @code{ccl} coding system used for
Alternativnyj, an encoding of the Cyrillic alphabet.

@item big5
@itemx big5-dos
@itemx big5-mac
@itemx big5-unix

Modeline indicator: @code{Zh/Big5}.  A type @code{big5} coding system used for
BIG5, the most common encoding of traditional Chinese as used in Taiwan.

@item cn-gb-2312
@itemx cn-gb-2312-dos
@itemx cn-gb-2312-mac
@itemx cn-gb-2312-unix

Modeline indicator: @code{Zh-GB/EUC}.  A type @code{iso2022} coding system used
for simplified Chinese (as used in the People's Republic of China), with
the @code{ascii} (G0), @code{chinese-gb2312} (G1), and @code{sisheng}
(G2) character sets initially designated.  Chinese EUC (Extended Unix
Code).

@item ctext-hebrew
@itemx ctext-hebrew-dos
@itemx ctext-hebrew-mac
@itemx ctext-hebrew-unix

Modeline indicator: @code{CText/Hbrw}.  A type @code{iso2022} coding system
with the @code{ascii} (G0) and @code{hebrew-iso8859-8} (G1) character
sets initially designated for Hebrew.

@item ctext
@itemx ctext-dos
@itemx ctext-mac
@itemx ctext-unix

Modeline indicator: @code{CText}.  A type @code{iso2022} 8-bit coding system
with the @code{ascii} (G0) and @code{latin-iso8859-1} (G1) character
sets initially designated.  X11 Compound Text Encoding.  Often
mistakenly recognized instead of EUC encodings; usual cause is
inappropriate setting of @code{coding-priority-list}.

@item escape-quoted

Modeline indicator: @code{ESC/Quot}.  A type @code{iso2022} 8-bit coding
system with the @code{ascii} (G0) and @code{latin-iso8859-1} (G1)
character sets initially designated and escape quoting.  Unix EOL
conversion (ie, no conversion).  It is used for .ELC files.

@item euc-jp
@itemx euc-jp-dos
@itemx euc-jp-mac
@itemx euc-jp-unix

Modeline indicator: @code{Ja/EUC}.  A type @code{iso2022} 8-bit coding system
with @code{ascii} (G0), @code{japanese-jisx0208} (G1),
@code{katakana-jisx0201} (G2), and @code{japanese-jisx0212} (G3)
initially designated.  Japanese EUC (Extended Unix Code).

@item euc-kr
@itemx euc-kr-dos
@itemx euc-kr-mac
@itemx euc-kr-unix

Modeline indicator: @code{ko/EUC}.  A type @code{iso2022} 8-bit coding system
with @code{ascii} (G0) and @code{korean-ksc5601} (G1) initially
designated.  Korean EUC (Extended Unix Code).

@item hz-gb-2312
Modeline indicator: @code{Zh-GB/Hz}.  A type @code{no-conversion} coding
system with Unix EOL convention (ie, no conversion) using
post-read-decode and pre-write-encode functions to translate the Hz/ZW
coding system used for Chinese.

@item iso-2022-7bit
@itemx iso-2022-7bit-unix
@itemx iso-2022-7bit-dos
@itemx iso-2022-7bit-mac
@itemx iso-2022-7

Modeline indicator: @code{ISO7}.  A type @code{iso2022} 7-bit coding system
with @code{ascii} (G0) initially designated.  Other character sets must
be explicitly designated to be used.

@item iso-2022-7bit-ss2
@itemx iso-2022-7bit-ss2-dos
@itemx iso-2022-7bit-ss2-mac
@itemx iso-2022-7bit-ss2-unix

Modeline indicator: @code{ISO7/SS}.  A type @code{iso2022} 7-bit coding system
with @code{ascii} (G0) initially designated.  Other character sets must
be explicitly designated to be used.  SS2 is used to invoke a
96-charset, one character at a time.

@item iso-2022-8
@itemx iso-2022-8-dos
@itemx iso-2022-8-mac
@itemx iso-2022-8-unix

Modeline indicator: @code{ISO8}.  A type @code{iso2022} 8-bit coding system
with @code{ascii} (G0) and @code{latin-iso8859-1} (G1) initially
designated.  Other character sets must be explicitly designated to be
used.  No single-shift or locking-shift.

@item iso-2022-8bit-ss2
@itemx iso-2022-8bit-ss2-dos
@itemx iso-2022-8bit-ss2-mac
@itemx iso-2022-8bit-ss2-unix

Modeline indicator: @code{ISO8/SS}.  A type @code{iso2022} 8-bit coding system
with @code{ascii} (G0) and @code{latin-iso8859-1} (G1) initially
designated.  Other character sets must be explicitly designated to be
used.  SS2 is used to invoke a 96-charset, one character at a time.

@item iso-2022-int-1
@itemx iso-2022-int-1-dos
@itemx iso-2022-int-1-mac
@itemx iso-2022-int-1-unix

Modeline indicator: @code{INT-1}.  A type @code{iso2022} 7-bit coding system
with @code{ascii} (G0) and @code{korean-ksc5601} (G1) initially
designated.  ISO-2022-INT-1.

@item iso-2022-jp-1978-irv
@itemx iso-2022-jp-1978-irv-dos
@itemx iso-2022-jp-1978-irv-mac
@itemx iso-2022-jp-1978-irv-unix

Modeline indicator: @code{Ja-78/7bit}.  A type @code{iso2022} 7-bit coding
system.  For compatibility with old Japanese terminals; if you need to
know, look at the source.

@item iso-2022-jp
@itemx iso-2022-jp-2 (ISO7/SS)
@itemx iso-2022-jp-dos
@itemx iso-2022-jp-mac
@itemx iso-2022-jp-unix
@itemx iso-2022-jp-2-dos
@itemx iso-2022-jp-2-mac
@itemx iso-2022-jp-2-unix

Modeline indicator: @code{MULE/7bit}.  A type @code{iso2022} 7-bit coding
system with @code{ascii} (G0) initially designated, and complex
specifications to insure backward compatibility with old Japanese
systems.  Used for communication with mail and news in Japan.  The "-2"
versions also use SS2 to invoke a 96-charset one character at a time.

@item iso-2022-kr
Modeline indicator: @code{Ko/7bit}  A type @code{iso2022} 7-bit coding
system with @code{ascii} (G0) and @code{korean-ksc5601} (G1) initially
designated.  Used for e-mail in Korea.

@item iso-2022-lock
@itemx iso-2022-lock-dos
@itemx iso-2022-lock-mac
@itemx iso-2022-lock-unix

Modeline indicator: @code{ISO7/Lock}.  A type @code{iso2022} 7-bit coding
system with @code{ascii} (G0) initially designated, using Locking-Shift
to invoke a 96-charset.

@item iso-8859-1
@itemx iso-8859-1-dos
@itemx iso-8859-1-mac
@itemx iso-8859-1-unix

Due to implementation, this is not a type @code{iso2022} coding system,
but rather an alias for the @code{raw-text} coding system.

@item iso-8859-2
@itemx iso-8859-2-dos
@itemx iso-8859-2-mac
@itemx iso-8859-2-unix

Modeline indicator: @code{MIME/Ltn-2}.  A type @code{iso2022} coding
system with @code{ascii} (G0) and @code{latin-iso8859-2} (G1) initially
invoked.

@item iso-8859-3
@itemx iso-8859-3-dos
@itemx iso-8859-3-mac
@itemx iso-8859-3-unix

Modeline indicator: @code{MIME/Ltn-3}.  A type @code{iso2022} coding system
with @code{ascii} (G0) and @code{latin-iso8859-3} (G1) initially
invoked.

@item iso-8859-4
@itemx iso-8859-4-dos
@itemx iso-8859-4-mac
@itemx iso-8859-4-unix

Modeline indicator: @code{MIME/Ltn-4}.  A type @code{iso2022} coding system
with @code{ascii} (G0) and @code{latin-iso8859-4} (G1) initially
invoked.

@item iso-8859-5
@itemx iso-8859-5-dos
@itemx iso-8859-5-mac
@itemx iso-8859-5-unix

Modeline indicator: @code{ISO8/Cyr}.  A type @code{iso2022} coding system with
@code{ascii} (G0) and @code{cyrillic-iso8859-5} (G1) initially invoked.

@item iso-8859-7
@itemx iso-8859-7-dos
@itemx iso-8859-7-mac
@itemx iso-8859-7-unix

Modeline indicator: @code{Grk}.  A type @code{iso2022} coding system with
@code{ascii} (G0) and @code{greek-iso8859-7} (G1) initially invoked.

@item iso-8859-8
@itemx iso-8859-8-dos
@itemx iso-8859-8-mac
@itemx iso-8859-8-unix

Modeline indicator: @code{MIME/Hbrw}.  A type @code{iso2022} coding system with
@code{ascii} (G0) and @code{hebrew-iso8859-8} (G1) initially invoked.

@item iso-8859-9
@itemx iso-8859-9-dos
@itemx iso-8859-9-mac
@itemx iso-8859-9-unix

Modeline indicator: @code{MIME/Ltn-5}.  A type @code{iso2022} coding system
with @code{ascii} (G0) and @code{latin-iso8859-9} (G1) initially
invoked.

@item koi8-r
@itemx koi8-r-dos
@itemx koi8-r-mac
@itemx koi8-r-unix

Modeline indicator: @code{KOI8}.  A type @code{ccl} coding-system used for
KOI8-R, an encoding of the Cyrillic alphabet.

@item shift_jis
@itemx shift_jis-dos
@itemx shift_jis-mac
@itemx shift_jis-unix

Modeline indicator: @code{Ja/SJIS}.  A type @code{shift-jis} coding-system
implementing the Shift-JIS encoding for Japanese.  The underscore is to
conform to the MIME charset implementing this encoding.

@item tis-620
@itemx tis-620-dos
@itemx tis-620-mac
@itemx tis-620-unix

Modeline indicator: @code{TIS620}.  A type @code{ccl} encoding for Thai.  The
external encoding is defined by TIS620, the internal encoding is
peculiar to MULE, and called @code{thai-xtis}.

@item viqr

Modeline indicator: @code{VIQR}.  A type @code{no-conversion} coding
system with Unix EOL convention (ie, no conversion) using
post-read-decode and pre-write-encode functions to translate the VIQR
coding system for Vietnamese.

@item viscii
@itemx viscii-dos
@itemx viscii-mac
@itemx viscii-unix

Modeline indicator: @code{VISCII}.  A type @code{ccl} coding-system used
for VISCII 1.1 for Vietnamese.  Differs slightly from VSCII; VISCII is
given priority by XEmacs.

@item vscii
@itemx vscii-dos
@itemx vscii-mac
@itemx vscii-unix

Modeline indicator: @code{VSCII}.  A type @code{ccl} coding-system used
for VSCII 1.1 for Vietnamese.  Differs slightly from VISCII, which is
given priority by XEmacs.  Use
@code{(prefer-coding-system 'vietnamese-vscii)} to give priority to VSCII.

@end table

@node CCL, Category Tables, Coding Systems, MULE
@section CCL

  CCL (Code Conversion Language) is a simple structured programming
language designed for character coding conversions.  A CCL program is
compiled to CCL code (represented by a vector of integers) and executed
by the CCL interpreter embedded in Emacs.  The CCL interpreter
implements a virtual machine with 8 registers called @code{r0}, ...,
@code{r7}, a number of control structures, and some I/O operators.  Take
care when using registers @code{r0} (used in implicit @dfn{set}
statements) and especially @code{r7} (used internally by several
statements and operations, especially for multiple return values and I/O
operations).

  CCL is used for code conversion during process I/O and file I/O for
non-ISO2022 coding systems.  (It is the only way for a user to specify a
code conversion function.)  It is also used for calculating the code
point of an X11 font from a character code.  However, since CCL is
designed as a powerful programming language, it can be used for more
generic calculation where efficiency is demanded.  A combination of
three or more arithmetic operations can be calculated faster by CCL than
by Emacs Lisp.

  @strong{Warning:}  The code in @file{src/mule-ccl.c} and
@file{$packages/lisp/mule-base/mule-ccl.el} is the definitive
description of CCL's semantics.  The previous version of this section
contained several typos and obsolete names left from earlier versions of
MULE, and many may remain.  (I am not an experienced CCL programmer; the
few who know CCL well find writing English painful.)

  A CCL program transforms an input data stream into an output data
stream.  The input stream, held in a buffer of constant bytes, is left
unchanged.  The buffer may be filled by an external input operation,
taken from an Emacs buffer, or taken from a Lisp string.  The output
buffer is a dynamic array of bytes, which can be written by an external
output operation, inserted into an Emacs buffer, or returned as a Lisp
string.

  A CCL program is a (Lisp) list containing two or three members.  The
first member is the @dfn{buffer magnification}, which indicates the
required minimum size of the output buffer as a multiple of the input
buffer.  It is followed by the @dfn{main block} which executes while
there is input remaining, and an optional @dfn{EOF block} which is
executed when the input is exhausted.  Both the main block and the EOF
block are CCL blocks.

  A @dfn{CCL block} is either a CCL statement or list of CCL statements.
A @dfn{CCL statement} is either a @dfn{set statement} (either an integer
or an @dfn{assignment}, which is a list of a register to receive the
assignment, an assignment operator, and an expression) or a @dfn{control
statement} (a list starting with a keyword, whose allowable syntax
depends on the keyword).

@menu
* CCL Syntax::          CCL program syntax in BNF notation.
* CCL Statements::      Semantics of CCL statements.
* CCL Expressions::     Operators and expressions in CCL.
* Calling CCL::         Running CCL programs.
* CCL Example::         A trivial program to transform the Web's URL encoding.
@end menu

@node    CCL Syntax, CCL Statements, , CCL
@comment Node,       Next,           Previous,  Up
@subsection CCL Syntax

  The full syntax of a CCL program in BNF notation:

@format
CCL_PROGRAM :=
        (BUFFER_MAGNIFICATION
         CCL_MAIN_BLOCK
         [ CCL_EOF_BLOCK ])

BUFFER_MAGNIFICATION := integer
CCL_MAIN_BLOCK := CCL_BLOCK
CCL_EOF_BLOCK := CCL_BLOCK

CCL_BLOCK :=
        STATEMENT | (STATEMENT [STATEMENT ...])
STATEMENT :=
        SET | IF | BRANCH | LOOP | REPEAT | BREAK | READ | WRITE  | CALL
	| TRANSLATE | MAP | END

SET :=
        (REG = EXPRESSION)
        | (REG ASSIGNMENT_OPERATOR EXPRESSION)
        | INT-OR-CHAR

EXPRESSION := ARG | (EXPRESSION OPERATOR ARG)

IF := (if EXPRESSION CCL_BLOCK [CCL_BLOCK])
BRANCH := (branch EXPRESSION CCL_BLOCK [CCL_BLOCK ...])
LOOP := (loop STATEMENT [STATEMENT ...])
BREAK := (break)
REPEAT :=
        (repeat)
        | (write-repeat [REG | INT-OR-CHAR | string])
        | (write-read-repeat REG [INT-OR-CHAR | ARRAY])
READ :=
        (read REG ...)
        | (read-if (REG OPERATOR ARG) CCL_BLOCK [CCL_BLOCK])
        | (read-branch REG CCL_BLOCK [CCL_BLOCK ...])
WRITE :=
        (write REG ...)
        | (write EXPRESSION)
        | (write INT-OR-CHAR) | (write string) | (write REG ARRAY)
        | string
CALL := (call ccl-program-name)


TRANSLATE := ;; Not implemented under XEmacs, except mule-to-unicode and
	     ;; unicode-to-mule.
	     (translate-character REG(table) REG(charset) REG(codepoint)) 
	     | (translate-character SYMBOL REG(charset) REG(codepoint)) 
	     | (mule-to-unicode REG(charset) REG(codepoint))
	     | (unicode-to-mule REG(unicode,code) REG(CHARSET))

END := (end)

REG := r0 | r1 | r2 | r3 | r4 | r5 | r6 | r7
ARG := REG | INT-OR-CHAR
OPERATOR :=
        + | - | * | / | % | & | '|' | ^ | << | >> | <8 | >8 | //
        | < | > | == | <= | >= | != | de-sjis | en-sjis
ASSIGNMENT_OPERATOR :=
        += | -= | *= | /= | %= | &= | '|=' | ^= | <<= | >>=
ARRAY := '[' INT-OR-CHAR ... ']'
INT-OR-CHAR := integer | character

@end format

@node    CCL Statements, CCL Expressions, CCL Syntax, CCL
@comment Node,           Next,            Previous,   Up
@subsection CCL Statements

  The Emacs Code Conversion Language provides the following statement
types: @dfn{set}, @dfn{if}, @dfn{branch}, @dfn{loop}, @dfn{repeat},
@dfn{break}, @dfn{read}, @dfn{write}, @dfn{call}, @dfn{translate} and
@dfn{end}.

@heading Set statement:

  The @dfn{set} statement has three variants with the syntaxes
@samp{(@var{reg} = @var{expression})},
@samp{(@var{reg} @var{assignment_operator} @var{expression})}, and
@samp{@var{integer}}.  The assignment operator variation of the
@dfn{set} statement works the same way as the corresponding C expression
statement does.  The assignment operators are @code{+=}, @code{-=},
@code{*=}, @code{/=}, @code{%=}, @code{&=}, @code{|=}, @code{^=},
@code{<<=}, and @code{>>=}, and they have the same meanings as in C.  A
"naked integer" @var{integer} is equivalent to a @var{set} statement of
the form @code{(r0 = @var{integer})}.

@heading I/O statements:

  The @dfn{read} statement takes one or more registers as arguments.  It
reads one byte (a C char) from the input into each register in turn.

  The @dfn{write} takes several forms.  In the form @samp{(write @var{reg}
...)} it takes one or more registers as arguments and writes each in
turn to the output.  The integer in a register (interpreted as an
Ichar) is encoded to multibyte form (ie, Ibytes) and written to the
current output buffer.  If it is less than 256, it is written as is.
The forms @samp{(write @var{expression})} and @samp{(write
@var{integer})} are treated analogously.  The form @samp{(write
@var{string})} writes the constant string to the output.  A
"naked string" @samp{@var{string}} is equivalent to the statement @samp{(write
@var{string})}.  The form @samp{(write @var{reg} @var{array})} writes
the @var{reg}th element of the @var{array} to the output.

@heading Conditional statements:

  The @dfn{if} statement takes an @var{expression}, a @var{CCL block}, and
an optional @var{second CCL block} as arguments.  If the
@var{expression} evaluates to non-zero, the first @var{CCL block} is
executed.  Otherwise, if there is a @var{second CCL block}, it is
executed.

  The @dfn{read-if} variant of the @dfn{if} statement takes an
@var{expression}, a @var{CCL block}, and an optional @var{second CCL
block} as arguments.  The @var{expression} must have the form
@code{(@var{reg} @var{operator} @var{operand})} (where @var{operand} is
a register or an integer).  The @code{read-if} statement first reads
from the input into the first register operand in the @var{expression},
then conditionally executes a CCL block just as the @code{if} statement
does.

  The @dfn{branch} statement takes an @var{expression} and one or more CCL
blocks as arguments.  The CCL blocks are treated as a zero-indexed
array, and the @code{branch} statement uses the @var{expression} as the
index of the CCL block to execute.  Null CCL blocks may be used as
no-ops, continuing execution with the statement following the
@code{branch} statement in the containing CCL block.  Out-of-range
values for the @var{expression} are also treated as no-ops.

  The @dfn{read-branch} variant of the @dfn{branch} statement takes an
@var{register}, a @var{CCL block}, and an optional @var{second CCL
block} as arguments.  The @code{read-branch} statement first reads from
the input into the @var{register}, then conditionally executes a CCL
block just as the @code{branch} statement does.

@heading Loop control statements:

  The @dfn{loop} statement creates a block with an implied jump from the
end of the block back to its head.  The loop is exited on a @code{break}
statement, and continued without executing the tail by a @code{repeat}
statement.

  The @dfn{break} statement, written @samp{(break)}, terminates the
current loop and continues with the next statement in the current
block.

  The @dfn{repeat} statement has three variants, @code{repeat},
@code{write-repeat}, and @code{write-read-repeat}.  Each continues the
current loop from its head, possibly after performing I/O.
@code{repeat} takes no arguments and does no I/O before jumping.
@code{write-repeat} takes a single argument (a register, an
integer, or a string), writes it to the output, then jumps.
@code{write-read-repeat} takes one or two arguments.  The first must
be a register.  The second may be an integer or an array; if absent, it
is implicitly set to the first (register) argument.
@code{write-read-repeat} writes its second argument to the output, then
reads from the input into the register, and finally jumps.  See the
@code{write} and @code{read} statements for the semantics of the I/O
operations for each type of argument.

@heading Other statements:

  The @dfn{call} statement, written @samp{(call @var{ccl-program-name})},
executes a CCL program as a subroutine.  It does not return a value to
the caller, but can modify the register status.

  The @dfn{mule-to-unicode} statement translates an XEmacs character into a
UCS code point, using U+FFFD REPLACEMENT CHARACTER if the given XEmacs
character has no known corresponding code point.  It takes two
arguments; the first is a register in which is stored the character set
ID of the character to be translated, and into which the UCS code is
stored.  The second is a register which stores the XEmacs code of the
character in question; if it is from a multidimensional character set,
like most of the East Asian national sets, it's stored as @samp{((c1 <<
8) & c2)}, where @samp{c1} is the first code, and @samp{c2} the second.
(That is, as a single integer, the high-order eight bits of which encode
the first position code, and the low order bits of which encode the
second.)

  The @dfn{unicode-to-mule} statement translates a Unicode code point
(an integer) into an XEmacs character.  Its first argument is a register
containing the UCS code point; the code for the correspond character
will be written into this register, in the same format as for
@samp{mule-to-unicode} The second argument is a register into which will
be written the character set ID of the converted character. 

  The @dfn{end} statement, written @samp{(end)}, terminates the CCL
program successfully, and returns to caller (which may be a CCL
program).  It does not alter the status of the registers.

@node    CCL Expressions, Calling CCL, CCL Statements, CCL
@comment Node,            Next,        Previous,       Up
@subsection CCL Expressions

  CCL, unlike Lisp, uses infix expressions.  The simplest CCL expressions
consist of a single @var{operand}, either a register (one of @code{r0},
..., @code{r0}) or an integer.  Complex expressions are lists of the
form @code{( @var{expression} @var{operator} @var{operand} )}.  Unlike
C, assignments are not expressions.

  In the following table, @var{X} is the target resister for a @dfn{set}.
In subexpressions, this is implicitly @code{r7}.  This means that
@code{>8}, @code{//}, @code{de-sjis}, and @code{en-sjis} cannot be used
freely in subexpressions, since they return parts of their values in
@code{r7}.  @var{Y} may be an expression, register, or integer, while
@var{Z} must be a register or an integer.

@multitable @columnfractions .22 .14 .09 .55
@item Name @tab Operator @tab Code @tab C-like Description
@item CCL_PLUS @tab @code{+} @tab 0x00 @tab X = Y + Z
@item CCL_MINUS @tab @code{-} @tab 0x01 @tab X = Y - Z
@item CCL_MUL @tab @code{*} @tab 0x02 @tab X = Y * Z
@item CCL_DIV @tab @code{/} @tab 0x03 @tab X = Y / Z
@item CCL_MOD @tab @code{%} @tab 0x04 @tab X = Y % Z
@item CCL_AND @tab @code{&} @tab 0x05 @tab X = Y & Z
@item CCL_OR @tab @code{|} @tab 0x06 @tab X = Y | Z
@item CCL_XOR @tab @code{^} @tab 0x07 @tab X = Y ^ Z
@item CCL_LSH @tab @code{<<} @tab 0x08 @tab X = Y << Z
@item CCL_RSH @tab @code{>>} @tab 0x09 @tab X = Y >> Z
@item CCL_LSH8 @tab @code{<8} @tab 0x0A @tab X = (Y << 8) | Z
@item CCL_RSH8 @tab @code{>8} @tab 0x0B @tab X = Y >> 8, r[7] = Y & 0xFF
@item CCL_DIVMOD @tab @code{//} @tab 0x0C @tab X = Y / Z, r[7] = Y % Z
@item CCL_LS @tab @code{<} @tab 0x10 @tab X = (X < Y)
@item CCL_GT @tab @code{>} @tab 0x11 @tab X = (X > Y)
@item CCL_EQ @tab @code{==} @tab 0x12 @tab X = (X == Y)
@item CCL_LE @tab @code{<=} @tab 0x13 @tab X = (X <= Y)
@item CCL_GE @tab @code{>=} @tab 0x14 @tab X = (X >= Y)
@item CCL_NE @tab @code{!=} @tab 0x15 @tab X = (X != Y)
@item CCL_ENCODE_SJIS @tab @code{en-sjis} @tab 0x16 @tab X = HIGHER_BYTE (SJIS (Y, Z))
@item @tab @tab @tab r[7] = LOWER_BYTE (SJIS (Y, Z)
@item CCL_DECODE_SJIS @tab @code{de-sjis} @tab 0x17 @tab X = HIGHER_BYTE (DE-SJIS (Y, Z))
@item @tab @tab @tab r[7] = LOWER_BYTE (DE-SJIS (Y, Z))
@end multitable

  The CCL operators are as in C, with the addition of CCL_LSH8, CCL_RSH8,
CCL_DIVMOD, CCL_ENCODE_SJIS, and CCL_DECODE_SJIS.  The CCL_ENCODE_SJIS
and CCL_DECODE_SJIS treat their first and second bytes as the high and
low bytes of a two-byte character code.  (SJIS stands for Shift JIS, an
encoding of Japanese characters used by Microsoft.  CCL_ENCODE_SJIS is a
complicated transformation of the Japanese standard JIS encoding to
Shift JIS.  CCL_DECODE_SJIS is its inverse.)  It is somewhat odd to
represent the SJIS operations in infix form.

@node    Calling CCL, CCL Example, CCL Expressions, CCL
@comment Node,        Next,          Previous,        Up
@subsection Calling CCL

  CCL programs are called automatically during Emacs buffer I/O when the
external representation has a coding system type of @code{shift-jis},
@code{big5}, or @code{ccl}.  The program is specified by the coding
system (@pxref{Coding Systems}).  You can also call CCL programs from
other CCL programs, and from Lisp using these functions:

@defun ccl-execute ccl-program status
Execute @var{ccl-program} with registers initialized by
@var{status}.  @var{ccl-program} is a vector of compiled CCL code
created by @code{ccl-compile}.  It is an error for the program to try to
execute a CCL I/O command.  @var{status} must be a vector of nine
values, specifying the initial value for the R0, R1 .. R7 registers and
for the instruction counter IC.  A @code{nil} value for a register
initializer causes the register to be set to 0.  A @code{nil} value for
the IC initializer causes execution to start at the beginning of the
program.  When the program is done, @var{status} is modified (by
side-effect) to contain the ending values for the corresponding
registers and IC.
@end defun

@defun ccl-execute-on-string ccl-program status string &optional continue
Execute @var{ccl-program} with initial @var{status} on
@var{string}.  @var{ccl-program} is a vector of compiled CCL code
created by @code{ccl-compile}.  @var{status} must be a vector of nine
values, specifying the initial value for the R0, R1 .. R7 registers and
for the instruction counter IC.  A @code{nil} value for a register
initializer causes the register to be set to 0.  A @code{nil} value for
the IC initializer causes execution to start at the beginning of the
program.  An optional fourth argument @var{continue}, if non-@code{nil}, causes
the IC to
remain on the unsatisfied read operation if the program terminates due
to exhaustion of the input buffer.  Otherwise the IC is set to the end
of the program.  When the program is done, @var{status} is modified (by
side-effect) to contain the ending values for the corresponding
registers and IC.  Returns the resulting string.
@end defun

  To call a CCL program from another CCL program, it must first be
registered:

@defun register-ccl-program name ccl-program
Register @var{name} for CCL program @var{ccl-program} in
@code{ccl-program-table}.  @var{ccl-program} should be the compiled form of
a CCL program, or @code{nil}.  Return index number of the registered CCL
program.
@end defun

  Information about the processor time used by the CCL interpreter can be
obtained using these functions:

@defun ccl-elapsed-time
Returns the elapsed processor time of the CCL interpreter as cons of
user and system time, as
floating point numbers measured in seconds.  If only one
overall value can be determined, the return value will be a cons of that
value and 0.
@end defun

@defun ccl-reset-elapsed-time
Resets the CCL interpreter's internal elapsed time registers.
@end defun

@node    CCL Example, ,  Calling CCL, CCL
@comment Node,         Next, Previous,    Up
@subsection CCL Example

  In this section, we describe the implementation of a trivial coding
system to transform from the Web's URL encoding to XEmacs' internal
coding.  Many people will have been first exposed to URL encoding when
they saw ``%20'' where they expected a space in a file's name on their
local hard disk; this can happen when a browser saves a file from the
web and doesn't encode the name, as passed from the server, properly.

  URL encoding itself is underspecified with regard to encodings beyond
ASCII.  The relevant document, RFC 1738, explicitly doesn't give any
information on how to encode non-ASCII characters, and the ``obvious''
way---use the %xx values for the octets of the eight bit MIME character
set in which the page was served---breaks when a user types a character
outside that character set.  Best practice for web development is to
serve all pages as UTF-8 and treat incoming form data as using that
coding system.  (Oh, and gamble that your clients won't ever want to
type anything outside Unicode.  But that's not so much of a gamble with
today's client operating systems.)  We don't treat non-ASCII in this
example, as dealing with @samp{(read-multibyte-character ...)} and
errors therewith would make it much harder to understand.

  Since CCL isn't a very rich language, we move much of the logic that
would ordinarily be computed from operations like @code{(member ..)},
@code{(and ...)} and @code{(or ...)} into tables, from which register
values are read and written, and on which @code{if} statements are
predicated.  Much more of the implementation of this coding system is
occupied with constructing these tables---in normal Emacs Lisp---than it
is with actual CCL code.

  All the @code{defvar} statements we deal with in the next few sections
are surrounded by a @code{(eval-and-compile ...)}, which means that the
logic which initializes these variables executes at compile time, and if
XEmacs loads the compiled version of the file, these variables are
initialized as constants.

@menu
* Four bits to ASCII::  Two tables used for getting hex digits from ASCII.
* URI Encoding constants::  Useful predefined characters. 
* Numeric to ASCII-hexadecimal conversion:: Trivial in Lisp, not so in CCL.
* Characters to be preserved:: No transformation needed for these characters.
* The program to decode to internal format:: .
* The program to encode from internal format:: . 
* The actual coding system:: .
@end menu

@node Four bits to ASCII, URI Encoding constants, , CCL Example
@subsubsection Four bits to ASCII

  The first @code{defvar} is for
@code{url-coding-high-order-nybble-as-ascii}, a 256-entry table that
maps from an octet's value to the ASCII encoding for the hex value of
its most significant four bits.  That might sound complex, but it isn't;
for decimal 65, hex value @samp{#x41}, the entry in the table is the
ASCII encoding of `4'.  For decimal 122, ASCII `z', hex value
@code{#x7a}, @code{(elt url-coding-high-order-nybble-as-ascii #x7a)}
after this file is loaded gives the ASCII encoding of 7.

@example
(defvar url-coding-high-order-nybble-as-ascii
  (let ((val (make-vector 256 0))
	(i 0))
    (while (< i (length val))
      (aset val i (char-to-int (aref (format "%02X" i) 0)))
      (setq i (1+ i)))
    val)
  "Table to find an ASCII version of an octet's most significant 4 bits.")
@end example

  The next table, @code{url-coding-low-order-nybble-as-ascii} is almost
the same thing, but this time it has a map for the hex encoding of the
low-order four bits.  So the sixty-fifth entry (offset @samp{#x41}) is
the ASCII encoding of `1', the hundred-and-twenty-second (offset
@samp{#x7a}) is the ASCII encoding of `A'.

@example
(defvar url-coding-low-order-nybble-as-ascii 
  (let ((val (make-vector 256 0))
	(i 0))
    (while (< i (length val))
      (aset val i (char-to-int (aref (format "%02X" i) 1)))
      (setq i (1+ i)))
    val)
  "Table to find an ASCII version of an octet's least significant 4 bits.")
@end example

@node URI Encoding constants, Numeric to ASCII-hexadecimal conversion, Four bits to ASCII, CCL Example
@subsubsection URI Encoding constants

  Next, we have a couple of variables that make the CCL code more
readable.  The first is the ASCII encoding of the percentage sign; this
character is used as an escape code, to start the encoding of a
non-printable character.  For historical reasons, URL encoding allows
the space character to be encoded as a plus sign--it does make typing
URLs like @samp{http://google.com/search?q=XEmacs+home+page} easier--and
as such, we have to check when decoding for this value, and map it to
the space character.  When doing this in CCL, we use the
@code{url-coding-escaped-space-code} variable. 
  
@example
(defvar url-coding-escape-character-code (char-to-int ?%)
  "The code point for the percentage sign, in ASCII.")

(defvar url-coding-escaped-space-code (char-to-int ?+)
  "The URL-encoded value of the space character, that is, +.")
@end example

@node Numeric to ASCII-hexadecimal conversion, Characters to be preserved, URI Encoding constants, CCL Example
@subsubsection Numeric to ASCII-hexadecimal conversion

  Now, we have a couple of utility tables that wouldn't be necessary in
a more expressive programming language than is CCL. The first is sixteen
in length, and maps a hexadecimal number to the ASCII encoding of that
number; so zero maps to ASCII `0', ten maps to ASCII `A.' The second
does the reverse; that is, it maps an ASCII character to its value when
interpreted as a hexadecimal digit. ('A' => 10, 'c' => 12, '2' => 2, as
a few examples.)

@example
(defvar url-coding-hex-digit-table 
  (let ((i 0)
	(val (make-vector 16 0)))
    (while (< i 16)
      (aset val i (char-to-int (aref (format "%X" i) 0)))
      (setq i (1+ i)))
    val)
  "A map from a hexadecimal digit's numeric value to its encoding in ASCII.")

(defvar url-coding-latin-1-as-hex-table
  (let ((val (make-vector 256 0))
	(i 0))
    (while (< i (length val))
      ;; Get a hex val for this ASCII character.
      (aset val i (string-to-int (format "%c" i) 16))
      (setq i (1+ i)))
    val)
  "A map from Latin 1 code points to their values as hexadecimal digits.")
@end example

@node Characters to be preserved, The program to decode to internal format, Numeric to ASCII-hexadecimal conversion, CCL Example
@subsubsection Characters to be preserved

  And finally, the last of these tables.  URL encoding says that
alphanumeric characters, the underscore, hyphen and the full stop
@footnote{That's what the standards call it, though my North American
readers will be more familiar with it as the period character.} retain
their ASCII encoding, and don't undergo transformation.
@code{url-coding-should-preserve-table} is an array in which the entries
are one if the corresponding ASCII character should be left as-is, and
zero if they should be transformed.  So the entries for all the control
and most of the punctuation charcters are zero.  Lisp programmers will
observe that this initialization is particularly inefficient, but
they'll also be aware that this is a long way from an inner loop where
every nanosecond counts.

@example
(defvar url-coding-should-preserve-table 
  (let ((preserve 
	 (list ?- ?_ ?. ?a ?b ?c ?d ?e ?f ?g ?h ?i ?j ?k ?l ?m ?n ?o 
	       ?p ?q ?r ?s ?t ?u ?v ?w ?x ?y ?z ?A ?B ?C ?D ?E ?F ?G
	       ?H ?I ?J ?K ?L ?M ?N ?O ?P ?Q ?R ?S ?T ?U ?V ?W ?X ?Y
	       ?Z ?0 ?1 ?2 ?3 ?4 ?5 ?6 ?7 ?8 ?9))
	(i 0)
	(res (make-vector 256 0)))
    (while (< i 256)
      (when (member (int-char i) preserve)
	(aset res i 1))
      (setq i (1+ i)))
    res)
  "A 256-entry array of flags, indicating whether or not to preserve an
octet as its ASCII encoding.")
@end example

@node The program to decode to internal format, The program to encode from internal format, Characters to be preserved, CCL Example
@subsubsection The program to decode to internal format

  After the almost interminable tables, we get to the CCL.  The first
CCL program, @code{ccl-decode-urlcoding} decodes from the URL coding to
our internal format; since this version of CCL doesn't have support for
error checking on the input, we don't do any verification on it.

The buffer magnification--approximate ratio of the size of the output
buffer to the size of the input buffer--is declared as one, because
fractional values aren't allowed. (Since all those %20's will map to 
` ', the length of the output text will be less than that of the input
text.)  

So, first we read an octet from the input buffer into register
@samp{r0}, to set up the loop.  Next, we start the loop, with a
@code{(loop ...)} statement, and we check if the value in @samp{r0} is a
percentage sign.  (Note the comma before
@code{url-coding-escape-character-code}; since CCL is a Lisp macro
language, we can break out of the macro evaluation with a comman, and as
such, ``@code{,url-coding-escape-character-code}'' will be evaluated as a
literal `37.')  

If it is a percentage sign, we read the next two octets into @samp{r2}
and @samp{r3}, and convert them into their hexadecimal numeric values,
using the @code{url-coding-latin-1-as-hex-table} array declared above.
(But again, it'll be interpreted as a literal array.)  We then left
shift the first by four bits, mask the two together, and write the
result to the output buffer.

If it isn't a percentage sign, and it is a `+' sign, we write a
space--hexadecimal 20--to the output buffer. 

If none of those things are true, we pass the octet to the output buffer
untransformed.  (This could be a place to put error checking, in a more
expressive language.)  We then read one more octet from the input
buffer, and move to the next iteration of the loop. 

@example
(define-ccl-program ccl-decode-urlcoding
  `(1	
    ((read r0)
     (loop
       (if (r0 == ,url-coding-escape-character-code)
	   ((read r2 r3)
	    ;; Assign the value at offset r2 in the url-coding-hex-digit-table
	    ;; to r3.
	    (r2 = r2 ,url-coding-latin-1-as-hex-table)
	    (r3 = r3 ,url-coding-latin-1-as-hex-table)
	    (r2 <<= 4)
	    (r3 |= r2)
	    (write r3))
	 (if (r0 == ,url-coding-escaped-space-code)
	     (write #x20)
	   (write r0)))
       (read r0)
       (repeat))))
  "CCL program to take URI-encoded ASCII text and transform it to our
internal encoding. ")
@end example

@node The program to encode from internal format, The actual coding system, The program to decode to internal format, CCL Example
@subsubsection The program to encode from internal format

  Next, we see the CCL program to encode ASCII text as URL coded text.
Here, the buffer magnification is specified as three, to account for ` '
mapping to %20, etc.  As before, we read an octet from the input into
@samp{r0}, and move into the body of the loop.  Next, we check if we
should preserve the value of this octet, by reading from offset
@samp{r0} in the @code{url-coding-should-preserve-table} into @samp{r1}.
Then we have an @samp{if} statement predicated on the value in
@samp{r1}; for the true branch, we write the input octet directly.  For
the false branch, we write a percentage sign, the ASCII encoding of the
high four bits in hex, and then the ASCII encoding of the low four bits
in hex. 

We then read an octet from the input into @samp{r0}, and repeat the loop.

@example
(define-ccl-program ccl-encode-urlcoding
  `(3
    ((read r0)
     (loop
       (r1 = r0 ,url-coding-should-preserve-table)
       ;; If we should preserve the value, just write the octet directly.
       (if r1
	   (write r0)
	 ;; else, write a percentage sign, and the hex value of the octet, in
	 ;; an ASCII-friendly format.
	 ((write ,url-coding-escape-character-code)
	  (write r0 ,url-coding-high-order-nybble-as-ascii)
	  (write r0 ,url-coding-low-order-nybble-as-ascii)))
       (read r0)
       (repeat))))
  "CCL program to encode octets (almost) according to RFC 1738")
@end example

@node The actual coding system, , The program to encode from internal format, CCL Example
@subsubsection The actual coding system

To actually create the coding system, we call
@samp{make-coding-system}.  The first argument is the symbol that is to
be the name of the coding system, in our case @samp{url-coding}. The
second specifies that the coding system is to be of type
@samp{ccl}---there are several other coding system types available,
including, see the documentation for @samp{make-coding-system} for the
full list. Then there's a documentation string describing the wherefore
and caveats of the coding system, and the final argument is a property
list giving information about the CCL programs and the coding system's
mnemonic. 

@example
(make-coding-system 
 'url-coding 'ccl 
 "The coding used by application/x-www-form-urlencoded HTTP applications.
This coding form doesn't specify anything about non-ASCII characters, so
make sure you've transformed to a seven-bit coding system first."
 '(decode ccl-decode-urlcoding
   encode ccl-encode-urlcoding
   mnemonic "URLenc"))
@end example

If you're lucky, the @samp{url-coding} coding system describe here
should be available in the XEmacs package system. Otherwise, downloading
it from @samp{http://www.parhasard.net/url-coding.el} should work for
the foreseeable future. 

@node Category Tables, Unicode Support, CCL, MULE
@section Category Tables

  A category table is a type of char table used for keeping track of
categories.  Categories are used for classifying characters for use in
regexps---you can refer to a category rather than having to use a
complicated [] expression (and category lookups are significantly
faster).

  There are 95 different categories available, one for each printable
character (including space) in the ASCII charset.  Each category is
designated by one such character, called a @dfn{category designator}.
They are specified in a regexp using the syntax @samp{\cX}, where X is a
category designator. (This is not yet implemented.)

  A category table specifies, for each character, the categories that
the character is in.  Note that a character can be in more than one
category.  More specifically, a category table maps from a character to
either the value @code{nil} (meaning the character is in no categories)
or a 95-element bit vector, specifying for each of the 95 categories
whether the character is in that category.

  Special Lisp functions are provided that abstract this, so you do not
have to directly manipulate bit vectors.

@defun category-table-p object
This function returns @code{t} if @var{object} is a category table.
@end defun

@defun category-table &optional buffer
This function returns the current category table.  This is the one
specified by the current buffer, or by @var{buffer} if it is
non-@code{nil}.
@end defun

@defun standard-category-table
This function returns the standard category table.  This is the one used
for new buffers.
@end defun

@defun copy-category-table &optional category-table
This function returns a new category table which is a copy of
@var{category-table}, which defaults to the standard category table.
@end defun

@defun set-category-table category-table &optional buffer
This function selects @var{category-table} as the new category table for
@var{buffer}.  @var{buffer} defaults to the current buffer if omitted.
@end defun

@defun category-designator-p object
This function returns @code{t} if @var{object} is a category designator (a
char in the range @samp{' '} to @samp{'~'}).
@end defun

@defun category-table-value-p object
This function returns @code{t} if @var{object} is a category table value.
Valid values are @code{nil} or a bit vector of size 95.
@end defun


@c Added 2002-03-13 sjt
@node Unicode Support, Charset Unification, Category Tables, MULE
@section Unicode Support
@cindex unicode
@cindex utf-8
@cindex utf-16
@cindex ucs-2
@cindex ucs-4
@cindex bmp
@cindex basic multilingual plance

Unicode support was added by Ben Wing to XEmacs 21.5.6.

@defun set-language-unicode-precedence-list list
Set the language-specific precedence list used for Unicode decoding.
This is a list of charsets, which are consulted in order for a translation
matching a given Unicode character.  If no matches are found, the charsets
in the default precedence list (see
@code{set-default-unicode-precedence-list}) are consulted, and then all
remaining charsets, in some arbitrary order.

The language-specific precedence list is meant to be set as part of the
language environment initialization; the default precedence list is meant
to be set by the user.
@end defun

@defun language-unicode-precedence-list
Return the language-specific precedence list used for Unicode decoding.
See @code{set-language-unicode-precedence-list} for more information.
@end defun

@defun set-default-unicode-precedence-list list
Set the default precedence list used for Unicode decoding.
This is meant to be set by the user.  See
`set-language-unicode-precedence-list' for more information.
@end defun

@defun default-unicode-precedence-list
Return the default precedence list used for Unicode decoding.
See @code{set-language-unicode-precedence-list} for more information.
@end defun

@defun set-unicode-conversion character code
Add conversion information between Unicode codepoints and characters.
@var{character} is one of the following:

@c #### fix this markup
-- A character (in which case @var{code} must be a non-negative integer)
-- A vector of characters (in which case @var{code} must be a vector of
   non-negative integers of the same length)

Values of @var{code} above 2^20 - 1 are allowed for the purpose of specifying
private characters, but will cause errors when converted to UTF-16 or UTF-32.
UCS-4 and UTF-8 can handle values to 2^31 - 1, but XEmacs Lisp integers top
out at 2^30 - 1.
@end defun

@defun character-to-unicode character
Convert @var{character} to Unicode codepoint.
When there is no international support (i.e. MULE is not defined),
this function simply does @code{char-to-int}.
@end defun

@defun unicode-to-character code [charsets]
Convert Unicode codepoint @var{code} to character.
@var{code} should be a non-negative integer.
If @var{charsets} is given, it should be a list of charsets, and only those
charsets will be consulted, in the given order, for a translation.
Otherwise, the default ordering of all charsets will be given (see
@code{set-unicode-charset-precedence}).

When there is no international support (i.e. MULE is not defined),
this function simply does @code{int-to-char} and ignores the
@var{charsets} argument.
@end defun

@defun parse-unicode-translation-table filename charset start end offset flags
Parse Unicode translation data in @var{filename} for MULE @var{charset}.
Data is text, in the form of one translation per line -- charset
codepoint followed by Unicode codepoint.  Numbers are decimal or hex
\(preceded by 0x).  Comments are marked with a #.  Charset codepoints
for two-dimensional charsets should have the first octet stored in the
high 8 bits of the hex number and the second in the low 8 bits.

If @var{start} and @var{end} are given, only charset codepoints within
the given range will be processed.  If @var{offset} is given, that value
will be added to all charset codepoints in the file to obtain the
internal charset codepoint.  @var{start} and @var{end} apply to the
codepoints in the file, before @var{offset} is applied.

(Note that, as usual, we assume that octets are in the range 32 to
127 or 33 to 126.  If you have a table in kuten form, with octets in
the range 1 to 94, you will have to use an offset of 5140,
i.e. 0x2020.)

@var{flags}, if specified, control further how the tables are interpreted
and are used to special-case certain known table weirdnesses in the
Unicode tables:

@table @code
@item ignore-first-column'
Exactly as it sounds.  The JIS X 0208 tables have 3 columns of data instead
of 2; the first is the Shift-JIS codepoint.

@item big5
The charset codepoint is a Big Five codepoint; convert it to the
proper hacked-up codepoint in `chinese-big5-1' or `chinese-big5-2'.
@end table
@end defun


@node Charset Unification, Charsets and Coding Systems, Unicode Support, MULE
@section Character Set Unification

Mule suffers from a design defect that causes it to consider the ISO
Latin character sets to be disjoint.  This results in oddities such as
files containing both ISO 8859/1 and ISO 8859/15 codes, and using ISO
2022 control sequences to switch between them, as well as more plausible
but often unnecessary combinations like ISO 8859/1 with ISO 8859/2.
This can be very annoying when sending messages or even in simple
editing on a single host.  Unification works around the problem by
converting as many characters as possible to use a single Latin coded
character set before saving the buffer.

This node and its children were ripp'd untimely from
@file{latin-unity.texi}, and have been quickly converted for use here.
However as APIs are likely to diverge, beware of inaccuracies.  Please
report any you discover with @kbd{M-x report-xemacs-bug RET}, as well
as any ambiguities or downright unintelligible passages.

A lot of the stuff here doesn't belong here; it belongs in the
@ref{Top, , , xemacs, XEmacs User's Manual}.  Report those as bugs,
too, preferably with patches.

@menu
* Overview::                    Unification history and general information.
* Usage::                       An overview of the operation of Unification.
* Configuration::               Configuring Unification for use.
* Theory of Operation::         How Unification works.
* What Unification Cannot Do for You::  Inherent problems of 8-bit charsets.
* Charsets and Coding Systems:: Reference lists with annotations.
* Unification Internals::       Utilities and implementation details.
@end menu

@node Overview, Usage, Charset Unification, Charset Unification
@subsection An Overview of Unification

Mule suffers from a design defect that causes it to consider the ISO
Latin character sets to be disjoint.  This manifests itself when a user
enters characters using input methods associated with different coded
character sets into a single buffer.

A very important example involves email.  Many sites, especially in the
U.S., default to use of the ISO 8859/1 coded character set (also called
``Latin 1,'' though these are somewhat different concepts).  However,
ISO 8859/1 provides a generic CURRENCY SIGN character.  Now that the
Euro has become the official currency of most countries in Europe, this
is unsatisfactory (and in practice, useless).  So Europeans generally
use ISO 8859/15, which is nearly identical to ISO 8859/1 for most
languages, except that it substitutes EURO SIGN for CURRENCY SIGN.

Suppose a European user yanks text from a post encoded in ISO 8859/1
into a message composition buffer, and enters some text including the
Euro sign.  Then Mule will consider the buffer to contain both ISO
8859/1 and ISO 8859/15 text, and MUAs such as Gnus will (if naively
programmed) send the message as a multipart mixed MIME body!

This is clearly stupid.  What is not as obvious is that, just as any
European can include American English in their text because ASCII is a
subset of ISO 8859/15, most European languages which use Latin
characters (eg, German and Polish) can typically be mixed while using
only one Latin coded character set (in this case, ISO 8859/2).  However,
this often depends on exactly what text is to be encoded.

Unification works around the problem by converting as many characters as
possible to use a single Latin coded character set before saving the
buffer.

@node Usage, Configuration, Overview, Charset Unification
@subsection Operation of Unification

Normally, Unification works in the background by installing
@code{unity-sanity-check} on @code{write-region-pre-hook}.  This is
done by default for the ISO 8859 Latin family of character sets.  The
user activates this functionality for other character set families by
invoking @code{enable-unification}, either interactively or in her
init file.  @xref{Init File, , , xemacs}.  Unification can be
deactivated by invoking @code{disable-unification}.

Unification also provides a few functions for remapping or recoding the
buffer by hand.  To @dfn{remap} a character means to change the buffer
representation of the character by using another coded character set.
Remapping never changes the identity of the character, but may involve
altering the code point of the character.  To @dfn{recode} a character
means to simply change the coded character set.  Recoding never alters
the code point of the character, but may change the identity of the
character.  @xref{Theory of Operation}.

There are a few variables which determine which coding systems are
always acceptable to Unification:  @code{unity-ucs-list},
@code{unity-preferred-coding-system-list}, and
@code{unity-preapproved-coding-system-list}.  The latter two default
to @code{()}, and should probably be avoided because they short-circuit
the sanity check.  If you find you need to use them, consider reporting
it as a bug or request for enhancement.  Because they seem unsafe, the
recommended interface is likely to change.

@menu
* Basic Functionality::            User interface and customization.
* Interactive Usage::              Treating text by hand.
                                   Also documents the hook function(s).
@end menu


@node Basic Functionality, Interactive Usage, , Usage
@section Basic Functionality

These functions and user options initialize and configure Unification.
In normal use, none of these should be needed.

@strong{These APIs are certain to change.}

@defun enable-unification
Set up hooks and initialize variables for latin-unity.

There are no arguments.

This function is idempotent.  It will reinitialize any hooks or variables
that are not in initial state.
@end defun

@defun disable-unification
There are no arguments.

Clean up hooks and void variables used by latin-unity.
@end defun

@defopt unity-ucs-list
List of coding systems considered to be universal.

The default value is @code{'(utf-8 iso-2022-7 ctext escape-quoted)}.

Order matters; coding systems earlier in the list will be preferred when
recommending a coding system.  These coding systems will not be used
without querying the user (unless they are also present in
@code{unity-preapproved-coding-system-list}), and follow the
@code{unity-preferred-coding-system-list} in the list of suggested
coding systems.

If none of the preferred coding systems are feasible, the first in
this list will be the default.

Notes on certain coding systems:  @code{escape-quoted} is a special
coding system used for autosaves and compiled Lisp in Mule.  You should
@c #### fix in latin-unity.texi
never delete this, although it is rare that a user would want to use it
directly.  Unification does not try to be \"smart\" about other general
ISO 2022 coding systems, such as ISO-2022-JP.  (They are not recognized
as equivalent to @code{iso-2022-7}.)  If your preferred coding system is
one of these, you may consider adding it to @code{unity-ucs-list}.
However, this will typically have the side effect that (eg) ISO 8859/1
files will be saved in 7-bit form with ISO 2022 escape sequences.
@end defopt

Coding systems which are not Latin and not in
@code{unity-ucs-list} are handled by short circuiting checks of
coding system against the next two variables.

@defopt unity-preapproved-coding-system-list
List of coding systems used without querying the user if feasible.

The default value is @samp{(buffer-default preferred)}.

The first feasible coding system in this list is used.  The special values
@samp{preferred} and @samp{buffer-default} may be present:

@table @code
@item buffer-default
Use the coding system used by @samp{write-region}, if feasible.

@item preferred
Use the coding system specified by @samp{prefer-coding-system} if feasible.
@end table

"Feasible" means that all characters in the buffer can be represented by
the coding system.  Coding systems in @samp{unity-ucs-list} are
always considered feasible.  Other feasible coding systems are computed
by @samp{unity-representations-feasible-region}.

Note that the first universal coding system in this list shadows all
other coding systems.  In particular, if your preferred coding system is
a universal coding system, and @code{preferred} is a member of this
list, unification will blithely convert all your files to that coding
system.  This is considered a feature, but it may surprise most users.
Users who don't like this behavior should put @code{preferred} in
@code{unity-preferred-coding-system-list}.
@end defopt

@defopt unity-preferred-coding-system-list
@c #### fix in latin-unity.texi
List of coding systems suggested to the user if feasible.

The default value is @samp{(iso-8859-1 iso-8859-15 iso-8859-2 iso-8859-3
iso-8859-4 iso-8859-9)}.

If none of the coding systems in
@c #### fix in latin-unity.texi
@code{unity-preapproved-coding-system-list} are feasible, this list
will be recommended to the user, followed by the
@code{unity-ucs-list}.  The first coding system in this list is default.  The
special values @samp{preferred} and @samp{buffer-default} may be
present:

@table @code
@item buffer-default
Use the coding system used by @samp{write-region}, if feasible.

@item preferred
Use the coding system specified by @samp{prefer-coding-system} if feasible.
@end table

"Feasible" means that all characters in the buffer can be represented by
the coding system.  Coding systems in @samp{unity-ucs-list} are
always considered feasible.  Other feasible coding systems are computed
by @samp{unity-representations-feasible-region}.
@end defopt


@defvar unity-iso-8859-1-aliases
List of coding systems to be treated as aliases of ISO 8859/1.

The default value is '(iso-8859-1).

This is not a user variable; to customize input of coding systems or
charsets, @samp{unity-coding-system-alias-alist} or
@samp{unity-charset-alias-alist}.
@end defvar


@node Interactive Usage, , Basic Functionality, Usage
@section Interactive Usage

First, the hook function @code{unity-sanity-check} is documented.
(It is placed here because it is not an interactive function, and there
is not yet a programmer's section of the manual.)

These functions provide access to internal functionality (such as the
remapping function) and to extra functionality (the recoding functions
and the test function).


@defun unity-sanity-check begin end filename append visit lockname &optional coding-system

Check if @var{coding-system} can represent all characters between
@var{begin} and @var{end}.

For compatibility with old broken versions of @code{write-region},
@var{coding-system} defaults to @code{buffer-file-coding-system}.
@var{filename}, @var{append}, @var{visit}, and @var{lockname} are
ignored.

Return nil if buffer-file-coding-system is not (ISO-2022-compatible)
Latin.  If @code{buffer-file-coding-system} is safe for the charsets
actually present in the buffer, return it.  Otherwise, ask the user to
choose a coding system, and return that.

This function does @emph{not} do the safe thing when
@code{buffer-file-coding-system} is nil (aka no-conversion).  It
considers that ``non-Latin,'' and passes it on to the Mule detection
mechanism.

This function is intended for use as a @code{write-region-pre-hook}.  It
does nothing except return @var{coding-system} if @code{write-region}
handlers are inhibited.
@end defun

@defun unity-buffer-representations-feasible

There are no arguments.

Apply unity-region-representations-feasible to the current buffer.
@end defun

@defun unity-region-representations-feasible begin end &optional buf

Return character sets that can represent the text from @var{begin} to @var{end} in @var{buf}.

@var{buf} defaults to the current buffer.  Called interactively, will be
applied to the region.  Function assumes @var{begin} <= @var{end}.

The return value is a cons.  The car is the list of character sets
that can individually represent all of the non-ASCII portion of the
buffer, and the cdr is the list of character sets that can
individually represent all of the ASCII portion.

The following is taken from a comment in the source.  Please refer to
the source to be sure of an accurate description.

The basic algorithm is to map over the region, compute the set of
charsets that can represent each character (the ``feasible charset''),
and take the intersection of those sets.

The current implementation takes advantage of the fact that ASCII
characters are common and cannot change asciisets.  Then using
skip-chars-forward makes motion over ASCII subregions very fast.

This same strategy could be applied generally by precomputing classes
of characters equivalent according to their effect on latinsets, and
adding a whole class to the skip-chars-forward string once a member is
found.

Probably efficiency is a function of the number of characters matched,
or maybe the length of the match string?  With @code{skip-category-forward}
over a precomputed category table it should be really fast.  In practice
for Latin character sets there are only 29 classes.
@end defun

@defun unity-remap-region begin end character-set &optional coding-system

Remap characters between @var{begin} and @var{end} to equivalents in
@var{character-set}.  Optional argument @var{coding-system} may be a
coding system name (a symbol) or nil.  Characters with no equivalent are
left as-is.

When called interactively, @var{begin} and @var{end} are set to the
beginning and end, respectively, of the active region, and the function
prompts for @var{character-set}.  The function does completion, knows
how to guess a character set name from a coding system name, and also
provides some common aliases.  See @code{unity-guess-charset}.
There is no way to specify @var{coding-system}, as it has no useful
function interactively.

Return @var{coding-system} if @var{coding-system} can encode all
characters in the region, t if @var{coding-system} is nil and the coding
system with G0 = 'ascii and G1 = @var{character-set} can encode all
characters, and otherwise nil.  Note that a non-null return does
@emph{not} mean it is safe to write the file, only the specified region.
(This behavior is useful for multipart MIME encoding and the like.)

Note:  by default this function is quite fascist about universal coding
systems.  It only admits @samp{utf-8}, @samp{iso-2022-7}, and
@samp{ctext}.  Customize @code{unity-approved-ucs-list} to change
this.

This function remaps characters that are artificially distinguished by Mule
internal code.  It may change the code point as well as the character set.
To recode characters that were decoded in the wrong coding system, use
@code{unity-recode-region}.
@end defun

@defun unity-recode-region begin end wrong-cs right-cs

Recode characters between @var{begin} and @var{end} from @var{wrong-cs}
to @var{right-cs}.

@var{wrong-cs} and @var{right-cs} are character sets.  Characters retain
the same code point but the character set is changed.  Only characters
from @var{wrong-cs} are changed to @var{right-cs}.  The identity of the
character may change.  Note that this could be dangerous, if characters
whose identities you do not want changed are included in the region.
This function cannot guess which characters you want changed, and which
should be left alone.

When called interactively, @var{begin} and @var{end} are set to the
beginning and end, respectively, of the active region, and the function
prompts for @var{wrong-cs} and @var{right-cs}.  The function does
completion, knows how to guess a character set name from a coding system
name, and also provides some common aliases.  See
@code{unity-guess-charset}.

Another way to accomplish this, but using coding systems rather than
character sets to specify the desired recoding, is
@samp{unity-recode-coding-region}.  That function may be faster
but is somewhat more dangerous, because it may recode more than one
character set.

To change from one Mule representation to another without changing identity
of any characters, use @samp{unity-remap-region}.
@end defun

@defun unity-recode-coding-region begin end wrong-cs right-cs

Recode text between @var{begin} and @var{end} from @var{wrong-cs} to
@var{right-cs}.

@var{wrong-cs} and @var{right-cs} are coding systems.  Characters retain
the same code point but the character set is changed.  The identity of
characters may change.  This is an inherently dangerous function;
multilingual text may be recoded in unexpected ways.  #### It's also
dangerous because the coding systems are not sanity-checked in the
current implementation.

When called interactively, @var{begin} and @var{end} are set to the
beginning and end, respectively, of the active region, and the function
prompts for @var{wrong-cs} and @var{right-cs}.  The function does
completion, knows how to guess a coding system name from a character set
name, and also provides some common aliases.  See
@code{unity-guess-coding-system}.

Another, safer, way to accomplish this, using character sets rather
than coding systems to specify the desired recoding, is to use
@c #### fixme in latin-unity.texi
@code{unity-recode-region}.

To change from one Mule representation to another without changing identity
of any characters, use @code{unity-remap-region}.
@end defun

Helper functions for input of coding system and character set names.

@defun unity-guess-charset candidate
Guess a charset based on the symbol @var{candidate}.

@var{candidate} itself is not tried as the value.

Uses the natural mapping in @samp{unity-cset-codesys-alist}, and
the values in @samp{unity-charset-alias-alist}."
@end defun

@defun unity-guess-coding-system candidate
Guess a coding system based on the symbol @var{candidate}.

@var{candidate} itself is not tried as the value.

Uses the natural mapping in @samp{unity-cset-codesys-alist}, and
the values in @samp{unity-coding-system-alias-alist}."
@end defun

@defun unity-example

A cheesy example for Unification.

At present it just makes a multilingual buffer.  To test, setq
buffer-file-coding-system to some value, make the buffer dirty (eg
with RET BackSpace), and save.
@end defun


@node Configuration, Theory of Operation, Usage, Charset Unification
@subsection Configuring Unification for Use

If you want Unification to be automatically initialized, invoke
@samp{enable-unification} with no arguments in your init file.
@xref{Init File, , , xemacs}.  If you are using GNU Emacs or an XEmacs
earlier than 21.1, you should also load @file{auto-autoloads} using the
full path (@emph{never} @samp{require} @file{auto-autoloads} libraries).

You may wish to define aliases for commonly used character sets and
coding systems for convenience in input.

@defopt unity-charset-alias-alist
Alist mapping aliases to Mule charset names (symbols)."

The default value is
@example
   ((latin-1 . latin-iso8859-1)
    (latin-2 . latin-iso8859-2)
    (latin-3 . latin-iso8859-3)
    (latin-4 . latin-iso8859-4)
    (latin-5 . latin-iso8859-9)
    (latin-9 . latin-iso8859-15)
    (latin-10 . latin-iso8859-16))
@end example

If a charset does not exist on your system, it will not complete and you
will not be able to enter it in response to prompts.  A real charset
with the same name as an alias in this list will shadow the alias.
@end defopt

@defopt unity-coding-system-alias-alist nil
Alist mapping aliases to Mule coding system names (symbols).

The default value is @samp{nil}.
@end defopt


@node Theory of Operation, What Unification Cannot Do for You, Configuration, Charset Unification
@subsection Theory of Operation

Standard encodings suffer from the design defect that they do not
provide a reliable way to recognize which coded character sets in use.
@xref{What Unification Cannot Do for You}.  There are scores of
character sets which can be represented by a single octet (8-bit byte),
whose union contains many hundreds of characters.  Obviously this
results in great confusion, since you can't tell the players without a
scorecard, and there is no scorecard.

There are two ways to solve this problem.  The first is to create a
universal coded character set.  This is the concept behind Unicode.
However, there have been satisfactory (nearly) universal character sets
for several decades, but even today many Westerners resist using Unicode
because they consider its space requirements excessive.  On the other
hand, Asians dislike Unicode because they consider it to be incomplete.
(This is partly, but not entirely, political.)

In any case, Unicode only solves the internal representation problem.
Many data sets will contain files in ``legacy'' encodings, and Unicode
does not help distinguish among them.

The second approach is to embed information about the encodings used in
a document in its text.  This approach is taken by the ISO 2022
standard.  This would solve the problem completely from the users' of
view, except that ISO 2022 is basically not implemented at all, in the
sense that few applications or systems implement more than a small
subset of ISO 2022 functionality.  This is due to the fact that
mono-literate users object to the presence of escape sequences in their
texts (which they, with some justification, consider data corruption).
Programmers are more than willing to cater to these users, since
implementing ISO 2022 is a painstaking task.

In fact, Emacs/Mule adopts both of these approaches.  Internally it uses
a universal character set, @dfn{Mule code}.  Externally it uses ISO 2022
techniques both to save files in forms robust to encoding issues, and as
hints when attempting to ``guess'' an unknown encoding.  However, Mule
suffers from a design defect, namely it embeds the character set
information that ISO 2022 attaches to runs of characters by introducing
them with a control sequence in each character.  That causes Mule to
consider the ISO Latin character sets to be disjoint.  This manifests
itself when a user enters characters using input methods associated with
different coded character sets into a single buffer.

There are two problems stemming from this design.  First, Mule
represents the same character in different ways.  Abstractly, 'ó'
(LATIN SMALL LETTER O WITH ACUTE) can get represented as
[latin-iso8859-1 #x73] or as [latin-iso8859-2 #x73].  So what looks like
'óó' in the display might actually be represented [latin-iso8859-1
#x73][latin-iso8859-2 #x73] in the buffer, and saved as [#xF3 ESC - B
#xF3 ESC - A] in the file.  In some cases this treatment would be
appropriate (consider HYPHEN, MINUS SIGN, EN DASH, EM DASH, and U+4E00
(the CJK ideographic character meaning ``one'')), and although arguably
incorrect it is convenient when mixing the CJK scripts.  But in the case
of the Latin scripts this is wrong.

Worse yet, it is very likely to occur when mixing ``different'' encodings
(such as ISO 8859/1 and ISO 8859/15) that differ only in a few code
points that are almost never used.  A very important example involves
email.  Many sites, especially in the U.S., default to use of the ISO
8859/1 coded character set (also called ``Latin 1,'' though these are
somewhat different concepts).  However, ISO 8859/1 provides a generic
CURRENCY SIGN character.  Now that the Euro has become the official
currency of most countries in Europe, this is unsatisfactory (and in
practice, useless).  So Europeans generally use ISO 8859/15, which is
nearly identical to ISO 8859/1 for most languages, except that it
substitutes EURO SIGN for CURRENCY SIGN.

Suppose a European user yanks text from a post encoded in ISO 8859/1
into a message composition buffer, and enters some text including the
Euro sign.  Then Mule will consider the buffer to contain both ISO
8859/1 and ISO 8859/15 text, and MUAs such as Gnus will (if naively
programmed) send the message as a multipart mixed MIME body!

This is clearly stupid.  What is not as obvious is that, just as any
European can include American English in their text because ASCII is a
subset of ISO 8859/15, most European languages which use Latin
characters (eg, German and Polish) can typically be mixed while using
only one Latin coded character set (in the case of German and Polish,
ISO 8859/2).  However, this often depends on exactly what text is to be
encoded (even for the same pair of languages).

Unification works around the problem by converting as many characters as
possible to use a single Latin coded character set before saving the
buffer.

Because the problem is rarely noticeable in editing a buffer, but tends
to manifest when that buffer is exported to a file or process, the
Unification package uses the strategy of examining the buffer prior to
export.  If use of multiple Latin coded character sets is detected,
Unification attempts to unify them by finding a single coded character
set which contains all of the Latin characters in the buffer.

The primary purpose of Unification is to fix the problem by giving the
user the choice to change the representation of all characters to one
character set and give sensible recommendations based on context.  In
the 'ó' example, either ISO 8859/1 or ISO 8859/2 is satisfactory, and
both will be suggested.  In the EURO SIGN example, only ISO 8859/15
makes sense, and that is what will be recommended.  In both cases, the
user will be reminded that there are universal encodings available.

I call this @dfn{remapping} (from the universal character set to a
particular ISO 8859 coded character set).  It is mere accident that this
letter has the same code point in both character sets.  (Not entirely,
but there are many examples of Latin characters that have different code
points in different Latin-X sets.)

Note that, in the 'ó' example, that treating the buffer in this way will
result in a representation such as [latin-iso8859-2
#x73][latin-iso8859-2 #x73], and the file will be saved as [#xF3 #xF3].
This is guaranteed to occasionally result in the second problem you
observed, to which we now turn.

This problem is that, although the file is intended to be an
ISO-8859/2-encoded file, in an ISO 8859/1 locale Mule (and every POSIX
compliant program---this is required by the standard, obvious if you
think a bit, @pxref{What Unification Cannot Do for You}) will read that
file as [latin-iso8859-1 #x73] [latin-iso8859-1 #x73].  Of course this
is no problem if all of the characters in the file are contained in ISO
8859/1, but suppose there are some which are not, but are contained in
the (intended) ISO 8859/2.

You now want to fix this, but not by finding the same character in
another set.  Instead, you want to simply change the character set that
Mule associates with that buffer position without changing the code.
(This is conceptually somewhat distinct from the first problem, and
logically ought to be handled in the code that defines coding systems.
However, unification is not an unreasonable place for it.)  Unification
provides two functions (one fast and dangerous, the other slow and
careful) to handle this.  I call this @dfn{recoding}, because the
transformation actually involves @emph{encoding} the buffer to file
representation, then @emph{decoding} it to buffer representation (in a
different character set).  This cannot be done automatically because
Mule can have no idea what the correct encoding is---after all, it
already gave you its best guess.  @xref{What Unification Cannot Do for
You}.  So these functions must be invoked by the user.  @xref{Interactive
Usage}.


@node What Unification Cannot Do for You, Unification Internals, Theory of Operation, Charset Unification
@subsection What Unification Cannot Do for You

Unification @strong{cannot} save you if you insist on exporting data in
8-bit encodings in a multilingual environment.  @emph{You will
eventually corrupt data if you do this.}  It is not Mule's, or any
application's, fault.  You will have only yourself to blame; consider
yourself warned.  (It is true that Mule has bugs, which make Mule
somewhat more dangerous and inconvenient than some naive applications.
We're working to address those, but no application can remedy the
inherent defect of 8-bit encodings.)

Use standard universal encodings, preferably Unicode (UTF-8) unless
applicable standards indicate otherwise.  The most important such case
is Internet messages, where MIME should be used, whether or not the
subordinate encoding is a universal encoding.  (Note that since one of
the important provisions of MIME is the @samp{Content-Type} header,
which has the charset parameter, MIME is to be considered a universal
encoding for the purposes of this manual.  Of course, technically
speaking it's neither a coded character set nor a coding extension
technique compliant with ISO 2022.)

As mentioned earlier, the problem is that standard encodings suffer from
the design defect that they do not provide a reliable way to recognize
which coded character sets are in use.  There are scores of character
sets which can be represented by a single octet (8-bit byte), whose
union contains many hundreds of characters.  Thus any 8-bit coded
character set must contain characters that share code points used for
different characters in other coded character sets.

This means that a given file's intended encoding cannot be identified
with 100% reliability unless it contains encoding markers such as those
provided by MIME or ISO 2022.

Unification actually makes it more likely that you will have problems of
this kind.  Traditionally Mule has been ``helpful'' by simply using an
ISO 2022 universal coding system when the current buffer coding system
cannot handle all the characters in the buffer.  This has the effect
that, because the file contains control sequences, it is not recognized
as being in the locale's normal 8-bit encoding.  It may be annoying if
you are not a Mule expert, but your data is automatically recoverable
with a tool you already have: Mule.

However, with unification, Mule converts to a single 8-bit character set
when possible.  But typically this will @emph{not} be in your usual
locale.  Ie, the times that an ISO 8859/1 user will need Unification is
when there are ISO 8859/2 characters in the buffer.  But then most
likely the file will be saved in a pure 8-bit encoding that is not ISO
8859/1, ie, ISO 8859/2.  Mule's autorecognizer (which is probably the
most sophisticated yet available) cannot tell the difference between ISO
8859/1 and ISO 8859/2, and in a Western European locale will choose the
former even though the latter was intended.  Even the extension
(``statistical recognition'') planned for XEmacs 22 is unlikely to be at
all accurate in the case of mixed codes.

So now consider adding some additional ISO 8859/1 text to the buffer.
If it includes any ISO 8859/1 codes that are used by different
characters in ISO 8859/2, you now have a file that cannot be
mechanically disentangled.  You need a human being who can recognize
that @emph{this is German and Swedish} and stays in Latin-1, while
@emph{that is Polish} and needs to be recoded to Latin-2.

Moral: switch to a universal coded character set, preferably Unicode
using the UTF-8 transformation format.  If you really need the space,
compress your files.


@node Unification Internals, , What Unification Cannot Do for You, Charset Unification
@subsection Internals

No internals documentation yet.

@file{unity-utils.el} provides one utility function.

@defun unity-dump-tables

Dump the temporary table created by loading @file{unity-utils.el}
to @file{unity-tables.el}.  Loading the latter file initializes
@samp{unity-equivalences}.
@end defun


@node Charsets and Coding Systems, , Charset Unification, MULE
@subsection Charsets and Coding Systems

This section provides reference lists of Mule charsets and coding
systems.  Mule charsets are typically named by character set and
standard.

@table @strong
@item ASCII variants

Identification of equivalent characters in these sets is not properly
implemented.  Unification does not distinguish the two charsets.

@samp{ascii} @samp{latin-jisx0201}

@item Extended Latin

Characters from the following ISO 2022 conformant charsets are
identified with equivalents in other charsets in the group by
Unification.

@samp{latin-iso8859-1} @samp{latin-iso8859-15} @samp{latin-iso8859-2}
@samp{latin-iso8859-3} @samp{latin-iso8859-4} @samp{latin-iso8859-9}
@samp{latin-iso8859-13} @samp{latin-iso8859-16}

The follow charsets are Latin variants which are not understood by
Unification.  In addition, many of the Asian language standards provide
ASCII, at least, and sometimes other Latin characters.  None of these
are identified with their ISO 8859 equivalents.

@samp{vietnamese-viscii-lower}
@samp{vietnamese-viscii-upper}

@item Other character sets

@samp{arabic-1-column}
@samp{arabic-2-column}
@samp{arabic-digit}
@samp{arabic-iso8859-6}
@samp{chinese-big5-1}
@samp{chinese-big5-2}
@samp{chinese-cns11643-1}
@samp{chinese-cns11643-2}
@samp{chinese-cns11643-3}
@samp{chinese-cns11643-4}
@samp{chinese-cns11643-5}
@samp{chinese-cns11643-6}
@samp{chinese-cns11643-7}
@samp{chinese-gb2312}
@samp{chinese-isoir165}
@samp{cyrillic-iso8859-5}
@samp{ethiopic}
@samp{greek-iso8859-7}
@samp{hebrew-iso8859-8}
@samp{ipa}
@samp{japanese-jisx0208}
@samp{japanese-jisx0208-1978}
@samp{japanese-jisx0212}
@samp{katakana-jisx0201}
@samp{korean-ksc5601}
@samp{sisheng}
@samp{thai-tis620}
@samp{thai-xtis}

@item Non-graphic charsets

@samp{control-1}
@end table

@table @strong
@item No conversion

Some of these coding systems may specify EOL conventions.  Note that
@samp{iso-8859-1} is a no-conversion coding system, not an ISO 2022
coding system.  Although unification attempts to compensate for this, it
is possible that the @samp{iso-8859-1} coding system will behave
differently from other ISO 8859 coding systems.

@samp{binary} @samp{no-conversion} @samp{raw-text} @samp{iso-8859-1}

@item Latin coding systems

These coding systems are all single-byte, 8-bit ISO 2022 coding systems,
combining ASCII in the GL register (bytes with high-bit clear) and an
extended Latin character set in the GR register (bytes with high-bit set).

@samp{iso-8859-15} @samp{iso-8859-2} @samp{iso-8859-3} @samp{iso-8859-4}
@samp{iso-8859-9} @samp{iso-8859-13} @samp{iso-8859-14} @samp{iso-8859-16}

These coding systems are single-byte, 8-bit coding systems that do not
conform to international standards.  They should be avoided in all
potentially multilingual contexts, including any text distributed over
the Internet and World Wide Web.

@samp{windows-1251}

@item Multilingual coding systems

The following ISO-2022-based coding systems are useful for multilingual
text.

@samp{ctext} @samp{iso-2022-lock} @samp{iso-2022-7} @samp{iso-2022-7bit}
@samp{iso-2022-7bit-ss2} @samp{iso-2022-8} @samp{iso-2022-8bit-ss2}

XEmacs also supports Unicode with the Mule-UCS package.  These are the
preferred coding systems for multilingual use.  (There is a possible
exception for texts that mix several Asian ideographic character sets.)

@samp{utf-16-be} @samp{utf-16-be-no-signature} @samp{utf-16-le}
@samp{utf-16-le-no-signature} @samp{utf-7} @samp{utf-7-safe}
@samp{utf-8} @samp{utf-8-ws}

Development versions of XEmacs (the 21.5 series) support Unicode
internally, with (at least) the following coding systems implemented:

@samp{utf-16-be} @samp{utf-16-be-bom} @samp{utf-16-le}
@samp{utf-16-le-bom} @samp{utf-8} @samp{utf-8-bom}

@item Asian ideographic languages

The following coding systems are based on ISO 2022, and are more or less
suitable for encoding multilingual texts.  They all can represent ASCII
at least, and sometimes several other foreign character sets, without
resort to arbitrary ISO 2022 designations.  However, these subsets are
not identified with the corresponding national standards in XEmacs Mule.

@samp{chinese-euc} @samp{cn-big5} @samp{cn-gb-2312} @samp{gb2312}
@samp{hz} @samp{hz-gb-2312} @samp{old-jis} @samp{japanese-euc}
@samp{junet} @samp{euc-japan} @samp{euc-jp} @samp{iso-2022-jp}
@samp{iso-2022-jp-1978-irv} @samp{iso-2022-jp-2} @samp{euc-kr}
@samp{korean-euc} @samp{iso-2022-kr} @samp{iso-2022-int-1}

The following coding systems cannot be used for general multilingual
text and do not cooperate well with other coding systems.

@samp{big5} @samp{shift_jis}

@item Other languages

The following coding systems are based on ISO 2022.  Though none of them
provides any Latin characters beyond ASCII, XEmacs Mule allows (and up
to 21.4 defaults to) use of ISO 2022 control sequences to designate
other character sets for inclusion the text.

@samp{iso-8859-5} @samp{iso-8859-7} @samp{iso-8859-8}
@samp{ctext-hebrew}

The following are character sets that do not conform to ISO 2022 and
thus cannot be safely used in a multilingual context.

@samp{alternativnyj} @samp{koi8-r} @samp{tis-620} @samp{viqr}
@samp{viscii} @samp{vscii}

@item Special coding systems

Mule uses the following coding systems for special purposes.

@samp{automatic-conversion} @samp{undecided} @samp{escape-quoted}

@samp{escape-quoted} is especially important, as it is used internally
as the coding system for autosaved data.

The following coding systems are aliases for others, and are used for
communication with the host operating system.

@samp{file-name} @samp{keyboard} @samp{terminal}

@end table

Mule detection of coding systems is actually limited to detection of
classes of coding systems called @dfn{coding categories}.  These coding
categories are identified by the ISO 2022 control sequences they use, if
any, by their conformance to ISO 2022 restrictions on code points that
may be used, and by characteristic patterns of use of 8-bit code points.

@samp{no-conversion}
@samp{utf-8}
@samp{ucs-4}
@samp{iso-7}
@samp{iso-lock-shift}
@samp{iso-8-1}
@samp{iso-8-2}
@samp{iso-8-designate}
@samp{shift-jis}
@samp{big5}


@c end of mule.texi