Mercurial > hg > xemacs-beta
annotate lisp/unicode.el @ 2367:ecf1ebac70d8
[xemacs-hg @ 2004-11-04 23:05:23 by ben]
commit mega-patch
configure.in: Turn off -Winline and -Wchar-subscripts.
Use the right set of cflags when compiling modules.
Rewrite ldap configuration to separate the inclusion of lber
(needed in recent Cygwin) from the basic checks for the
needed libraries.
add a function for MAKE_JUNK_C; initially code was added to
generate xemacs.def using this, but it will need to be rewritten.
add an rm -f for junk.c to avoid weird Cygwin bug with cp -f onto
an existing file.
Sort list of auto-detected functions and eliminate unused checks for
stpcpy, setlocale and getwd.
Add autodetection of Cygwin scanf problems
BETA: Rewrite section on configure to indicate what flags are important
and what not.
digest-doc.c, make-dump-id.c, profile.c, sorted-doc.c: Add proper decls for main().
make-msgfile.c: Document that this is old junk.
Move proposal to text.c.
make-msgfile.lex: Move proposal to text.c.
make-mswin-unicode.pl: Convert error-generating code so that the entire message will
be seen as a single unrecognized token.
mule/mule-ccl.el: Update docs.
lispref/mule.texi: Update CCL docs.
ldap/eldap.c: Mule-ize.
Use EXTERNAL_LIST_LOOP_2 instead of deleted EXTERNAL_LIST_LOOP.
* XEmacs 21.5.18 "chestnut" is released.
---------------------------------------------------------------
MULE-RELATED WORK:
---------------------------------------------------------------
---------------------------
byte-char conversion
---------------------------
buffer.c, buffer.h, insdel.c, text.c: Port FSF algorithm for byte-char conversion, replacing broken
previous version. Track the char position of the gap. Add
functions to do char-byte conversion downwards as well as upwards.
Move comments about algorithm workings to internals manual.
---------------------------
work on types
---------------------------
alloc.c, console-x-impl.h, dump-data.c, dump-data.h, dumper.c, dialog-msw.c, dired-msw.c, doc.c, editfns.c, esd.c, event-gtk.h, event-msw.c, events.c, file-coding.c, file-coding.h, fns.c, glyphs-eimage.c, glyphs-gtk.c, glyphs-msw.c, glyphs-shared.c, glyphs-x.c, glyphs.c, glyphs.h, gui.c, hpplay.c, imgproc.c, intl-win32.c, lrecord.h, lstream.c, keymap.c, lisp.h, libsst.c, linuxplay.c, miscplay.c, miscplay.h, mule-coding.c, nas.c, nt.c, ntheap.c, ntplay.c, objects-msw.c, objects-tty.c, objects-x.c, print.c, process-nt.c, process.c, redisplay.h, select-common.h, select-gtk.c, select-x.c, sgiplay.c, sound.c, sound.h, sunplay.c, sysfile.h, sysdep.c, syswindows.h, text.c, unexnt.c, win32.c, xgccache.c: Further work on types. This creates a full set of types for all
the basic semantics of `char' that I have so far identified, so that
its semantics can always be identified for the purposes of proper
Mule-safe code, and the raw use of `char' always avoided.
(1) More type renaming, for consistency of naming.
Char_ASCII -> Ascbyte
UChar_ASCII -> UAscbyte
Char_Binary -> CBinbyte
UChar_Binary -> Binbyte
SChar_Binary -> SBinbyte
(2) Introduce Rawbyte, CRawbyte, Boolbyte, Chbyte, UChbyte, and
Bitbyte and use them.
(3) New types Itext, Wexttext and Textcount for separating out
the concepts of bytes and textual units (different under UTF-16
and UTF-32, which are potential internal encodings).
(4) qxestr*_c -> qxestr*_ascii.
lisp.h: New; goes with other qxe() functions. #### Maybe goes in a
different section.
lisp.h: Group generic int-type defs together with EMACS_INT defs.
lisp.h: * lisp.h (WEXTTEXT_IS_WIDE)
New defns.
lisp.h: New type to replace places where int occurs as a boolean.
It's signed because occasionally people may want to use -1 as
an error value, and because unsigned ints are viral -- see comments
in the internals manual against using them.
dynarr.c: int -> Bytecount.
---------------------------
Mule-izing
---------------------------
device-x.c: Partially Mule-ize.
dumper.c, dumper.h: Mule-ize. Use Rawbyte. Use stderr_out not printf. Use wext_*().
sysdep.c, syswindows.h, text.c: New Wexttext API for manipulation of external text that may be
Unicode (e.g. startup code under Windows).
emacs.c: Mule-ize. Properly deal with argv in external encoding.
Use wext_*() and Wexttext. Use Rawbyte.
#if 0 some old junk on SCO that is unlikely to be correct.
Rewrite allocation code in run-temacs.
emacs.c, symsinit.h, win32.c: Rename win32 init function and call it even earlier, to
initialize mswindows_9x_p even earlier, for use in startup code
(XEUNICODE_P).
process.c: Use _wenviron not environ under Windows, to get Unicode environment
variables.
event-Xt.c: Mule-ize drag-n-drop related stuff.
dragdrop.c, dragdrop.h, frame-x.c: Mule-ize.
text.h: Add some more stand-in defines for particular kinds of conversion;
use in Mule-ization work in frame-x.c etc.
---------------------------
Freshening
---------------------------
intl-auto-encap-win32.c, intl-auto-encap-win32.h: Regenerate.
---------------------------
Unicode-work
---------------------------
intl-win32.c, syswindows.h: Factor out common options to MultiByteToWideChar and
WideCharToMultiByte. Add convert_unicode_to_multibyte_malloc()
and convert_unicode_to_multibyte_dynarr() and use. Add stuff for
alloca() conversion of multibyte/unicode.
alloc.c: Use dfc_external_data_len() in case of unicode coding system.
alloc.c, mule-charset.c: Don't zero out and reinit charset Unicode tables. This fucks up
dump-time loading. Anyway, either we load them at dump time or
run time, never both.
unicode.c: Dump the blank tables as well.
---------------------------------------------------------------
DOCUMENTATION, MOSTLY MULE-RELATED:
---------------------------------------------------------------
EmacsFrame.c, emodules.c, event-Xt.c, fileio.c, input-method-xlib.c, mule-wnnfns.c, redisplay-gtk.c, redisplay-tty.c, redisplay-x.c, regex.c, sysdep.c: Add comment about Mule work needed.
text.h: Add more documentation describing why DFC routines were not written
to return their value. Add some other DFC documentation.
console-msw.c, console-msw.h: Add pointer to docs in win32.c.
emacs.c: Add comments on sources of doc info.
text.c, charset.h, unicode.c, intl-win32.c, intl-encap-win32.c, text.h, file-coding.c, mule-coding.c: Collect background comments and related to text matters and
internationalization, and proposals for work to be done, in text.c
or Internals manual, stuff related to specific textual API's in
text.h, and stuff related to internal implementation of Unicode
conversion in unicode.c. Put lots of pointers to the comments to
make them easier to find.
s/mingw32.h, s/win32-common.h, s/win32-native.h, s/windowsnt.h, win32.c: Add bunches of new documentation on the different kinds of
builds and environments under Windows and how they work.
Collect this info in win32.c. Add pointers to these docs in
the relevant s/* files.
emacs.c: Document places with long comments.
Remove comment about exiting, move to internals manual, put
in pointer.
event-stream.c: Move docs about event queues and focus to internals manual, put
in pointer.
events.h: Move docs about event stream callbacks to internals manual, put
in pointer.
profile.c, redisplay.c, signal.c: Move documentation to the Internals manual.
process-nt.c: Add pointer to comment in win32-native.el.
lisp.h: Add comments about some comment conventions.
lisp.h: Add comment about the second argument.
device-msw.c, redisplay-msw.c: @@#### comments are out-of-date.
---------------------------------------------------------------
PDUMP WORK (MOTIVATED BY UNICODE CHANGES)
---------------------------------------------------------------
alloc.c, buffer.c, bytecode.c, console-impl.h, console.c, device.c, dumper.c, lrecord.h, elhash.c, emodules.h, events.c, extents.c, frame.c, glyphs.c, glyphs.h, mule-charset.c, mule-coding.c, objects.c, profile.c, rangetab.c, redisplay.c, specifier.c, specifier.h, window.c, lstream.c, file-coding.h, file-coding.c: PDUMP:
Properly implement dump_add_root_block(), which never worked before,
and is necessary for dumping Unicode tables.
Pdump name changes for accuracy:
XD_STRUCT_PTR -> XD_BLOCK_PTR.
XD_STRUCT_ARRAY -> XD_BLOCK_ARRAY.
XD_C_STRING -> XD_ASCII_STRING.
*_structure_* -> *_block_*.
lrecord.h: some comments added about
dump_add_root_block() vs dump_add_root_block_ptr().
extents.c: remove incorrect comment about pdump problems with gap array.
---------------------------------------------------------------
ALLOCATION
---------------------------------------------------------------
abbrev.c, alloc.c, bytecode.c, casefiddle.c, device-msw.c, device-x.c, dired-msw.c, doc.c, doprnt.c, dragdrop.c, editfns.c, emodules.c, file-coding.c, fileio.c, filelock.c, fns.c, glyphs-eimage.c, glyphs-gtk.c, glyphs-msw.c, glyphs-x.c, gui-msw.c, gui-x.c, imgproc.c, intl-win32.c, lread.c, menubar-gtk.c, menubar.c, nt.c, objects-msw.c, objects-x.c, print.c, process-nt.c, process-unix.c, process.c, realpath.c, redisplay.c, search.c, select-common.c, symbols.c, sysdep.c, syswindows.h, text.c, text.h, ui-byhand.c: New macros {alloca,xnew}_{itext,{i,ext,raw,bin,asc}bytes} for
more convenient allocation of these commonly requested items.
Modify functions to use alloca_ibytes, alloca_array, alloca_extbytes,
xnew_ibytes, etc. also XREALLOC_ARRAY, xnew.
alloc.c: Rewrite the allocation functions to factor out repeated code.
Add assertions for freeing dumped data.
lisp.h: Moved down and consolidated with other allocation stuff.
lisp.h, dynarr.c: New functions for allocation that's very efficient when mostly in
LIFO order.
lisp.h, text.c, text.h: Factor out some stuff for general use by alloca()-conversion funs.
text.h, lisp.h: Fill out convenience routines for allocating various kinds of
bytes and put them in lisp.h. Use them in place of xmalloc(),
ALLOCA().
text.h: Fill out the convenience functions so the _MALLOC() kinds match
the alloca() kinds.
---------------------------------------------------------------
ERROR-CHECKING
---------------------------------------------------------------
text.h: Create ASSERT_ASCTEXT_ASCII() and ASSERT_ASCTEXT_ASCII_LEN()
from similar Eistring checkers and change the Eistring checkers to
use them instead.
---------------------------------------------------------------
MACROS IN LISP.H
---------------------------------------------------------------
lisp.h: Redo GCPRO declarations. Create a "base" set of functions that can
be used to generate any kind of gcpro sets -- regular, ngcpro,
nngcpro, private ones used in GC_EXTERNAL_LIST_LOOP_2.
buffer.c, callint.c, chartab.c, console-msw.c, device-x.c, dialog-msw.c, dired.c, extents.c, ui-gtk.c, rangetab.c, nt.c, mule-coding.c, minibuf.c, menubar-msw.c, menubar.c, menubar-gtk.c, lread.c, lisp.h, gutter.c, glyphs.c, glyphs-widget.c, fns.c, fileio.c, file-coding.c, specifier.c: Eliminate EXTERNAL_LIST_LOOP, which does not check for circularities.
Use EXTERNAL_LIST_LOOP_2 instead or EXTERNAL_LIST_LOOP_3
or EXTERNAL_PROPERTY_LIST_LOOP_3 or GC_EXTERNAL_LIST_LOOP_2
(new macro). Removed/redid comments on EXTERNAL_LIST_LOOP.
---------------------------------------------------------------
SPACING FIXES
---------------------------------------------------------------
callint.c, hftctl.c, number-gmp.c, process-unix.c: Spacing fixes.
---------------------------------------------------------------
FIX FOR GEOMETRY PROBLEM IN FIRST FRAME
---------------------------------------------------------------
unicode.c: Add workaround for newlib bug in sscanf() [should be fixed by
release 1.5.12 of Cygwin].
toolbar.c: bug fix for problem of initial frame being 77 chars wide on Windows.
will be overridden by my other ws.
---------------------------------------------------------------
FIX FOR LEAKING PROCESS HANDLES:
---------------------------------------------------------------
process-nt.c: Fixes for leaking handles. Inspired by work done by Adrian Aichner
<adrian@xemacs.org>.
---------------------------------------------------------------
FIX FOR CYGWIN BUG (Unicode-related):
---------------------------------------------------------------
unicode.c: Add workaround for newlib bug in sscanf() [should be fixed by
release 1.5.12 of Cygwin].
---------------------------------------------------------------
WARNING FIXES:
---------------------------------------------------------------
console-stream.c: `reinit' is unused.
compiler.h, event-msw.c, frame-msw.c, intl-encap-win32.c, text.h: Add stuff to deal with ANSI-aliasing warnings I got.
regex.c: Gather includes together to avoid warning.
---------------------------------------------------------------
CHANGES TO INITIALIZATION ROUTINES:
---------------------------------------------------------------
buffer.c, emacs.c, console.c, debug.c, device-x.c, device.c, dragdrop.c, emodules.c, eval.c, event-Xt.c, event-gtk.c, event-msw.c, event-stream.c, event-tty.c, events.c, extents.c, faces.c, file-coding.c, fileio.c, font-lock.c, frame-msw.c, glyphs-widget.c, glyphs.c, gui-x.c, insdel.c, lread.c, lstream.c, menubar-gtk.c, menubar-x.c, minibuf.c, mule-wnnfns.c, objects-msw.c, objects.c, print.c, scrollbar-x.c, search.c, select-x.c, text.c, undo.c, unicode.c, window.c, symsinit.h: Call reinit_*() functions directly from emacs.c, for clarity.
Factor out some redundant init code. Move disallowed stuff
that had crept into vars_of_glyphs() into complex_vars_of_glyphs().
Call init_eval_semi_early() from eval.c not in the middle of
vars_of_() in emacs.c since there should be no order dependency
in the latter calls.
---------------------------------------------------------------
ARMAGEDDON:
---------------------------------------------------------------
alloc.c, emacs.c, lisp.h, print.c: Rename inhibit_non_essential_printing_operations to
inhibit_non_essential_conversion_operations.
text.c: Assert on !inhibit_non_essential_conversion_operations.
console-msw.c, print.c: Don't do conversion in SetConsoleTitle or FindWindow to avoid
problems during armageddon. Put #errors for NON_ASCII_INTERNAL_FORMAT
in places where problems would arise.
---------------------------------------------------------------
CHANGES TO THE BUILD PROCEDURE:
---------------------------------------------------------------
config.h.in, s/cxux.h, s/usg5-4-2.h, m/powerpc.h: Add comment about correct ordering of this file.
Rearrange everything to follow this -- put all #undefs together
and before the s&m files. Add undefs for HAVE_ALLOCA, C_ALLOCA,
BROKEN_ALLOCA_IN_FUNCTION_CALLS, STACK_DIRECTION. Remove unused
HAVE_STPCPY, HAVE_GETWD, HAVE_SETLOCALE.
m/gec63.h: Deleted; totally broken, not used at all, not in FSF.
m/7300.h, m/acorn.h, m/alliant-2800.h, m/alliant.h, m/altos.h, m/amdahl.h, m/apollo.h, m/att3b.h, m/aviion.h, m/celerity.h, m/clipper.h, m/cnvrgnt.h, m/convex.h, m/cydra5.h, m/delta.h, m/delta88k.h, m/dpx2.h, m/elxsi.h, m/ews4800r.h, m/gould.h, m/hp300bsd.h, m/hp800.h, m/hp9000s300.h, m/i860.h, m/ibmps2-aix.h, m/ibmrs6000.h, m/ibmrt-aix.h, m/ibmrt.h, m/intel386.h, m/iris4d.h, m/iris5d.h, m/iris6d.h, m/irist.h, m/isi-ov.h, m/luna88k.h, m/m68k.h, m/masscomp.h, m/mg1.h, m/mips-nec.h, m/mips-siemens.h, m/mips.h, m/news.h, m/nh3000.h, m/nh4000.h, m/ns32000.h, m/orion105.h, m/pfa50.h, m/plexus.h, m/pmax.h, m/powerpc.h, m/pyrmips.h, m/sequent-ptx.h, m/sequent.h, m/sgi-challenge.h, m/symmetry.h, m/tad68k.h, m/tahoe.h, m/targon31.h, m/tekxd88.h, m/template.h, m/tower32.h, m/tower32v3.h, m/ustation.h, m/vax.h, m/wicat.h, m/xps100.h: Delete C_ALLOCA, HAVE_ALLOCA, STACK_DIRECTION,
BROKEN_ALLOCA_IN_FUNCTION_CALLS. All of this is auto-detected.
When in doubt, I followed recent FSF sources, which also have
these things deleted.
| author | ben |
|---|---|
| date | Thu, 04 Nov 2004 23:08:28 +0000 |
| parents | 13a418960a88 |
| children | 5e2653bc0ab0 |
| rev | line source |
|---|---|
| 771 | 1 ;;; unicode.el --- Unicode support -*- coding: iso-2022-7bit; -*- |
| 2 | |
| 778 | 3 ;; Copyright (C) 2001, 2002 Ben Wing. |
| 771 | 4 |
| 5 ;; Keywords: multilingual, Unicode | |
| 6 | |
| 7 ;; This file is part of XEmacs. | |
| 8 | |
| 9 ;; XEmacs is free software; you can redistribute it and/or modify it | |
| 10 ;; under the terms of the GNU General Public License as published by | |
| 11 ;; the Free Software Foundation; either version 2, or (at your option) | |
| 12 ;; any later version. | |
| 13 | |
| 14 ;; XEmacs is distributed in the hope that it will be useful, but | |
| 15 ;; WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 16 ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU | |
| 17 ;; General Public License for more details. | |
| 18 | |
| 19 ;; You should have received a copy of the GNU General Public License | |
| 20 ;; along with XEmacs; see the file COPYING. If not, write to the Free | |
| 21 ;; Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA | |
| 22 ;; 02111-1307, USA. | |
| 23 | |
| 24 ;;; Synched up with: Not in FSF. | |
| 25 | |
| 26 ;;; Commentary: | |
| 27 | |
| 28 ;; Lisp support for Unicode, e.g. initialize the translation tables. | |
| 29 | |
| 30 ;;; Code: | |
| 31 | |
| 778 | 32 ; ;; Subsets of Unicode. |
| 33 | |
| 2297 | 34 ; #### what is this bogosity ... "chars 96, final ?2" !!?! |
| 778 | 35 ; (make-charset 'mule-unicode-2500-33ff |
| 36 ; "Unicode characters of the range U+2500..U+33FF." | |
| 37 ; '(dimension | |
| 38 ; 2 | |
| 39 ; registry "ISO10646-1" | |
| 40 ; chars 96 | |
| 41 ; columns 1 | |
| 42 ; direction l2r | |
| 43 ; final ?2 | |
| 44 ; graphic 0 | |
| 45 ; short-name "Unicode subset 2" | |
| 46 ; long-name "Unicode subset (U+2500..U+33FF)" | |
| 47 ; )) | |
| 48 | |
| 49 | |
| 50 ; (make-charset 'mule-unicode-e000-ffff | |
| 51 ; "Unicode characters of the range U+E000..U+FFFF." | |
| 52 ; '(dimension | |
| 53 ; 2 | |
| 54 ; registry "ISO10646-1" | |
| 55 ; chars 96 | |
| 56 ; columns 1 | |
| 57 ; direction l2r | |
| 58 ; final ?3 | |
| 59 ; graphic 0 | |
| 60 ; short-name "Unicode subset 3" | |
| 61 ; long-name "Unicode subset (U+E000+FFFF)" | |
| 62 ; )) | |
| 63 | |
| 64 | |
| 65 ; (make-charset 'mule-unicode-0100-24ff | |
| 66 ; "Unicode characters of the range U+0100..U+24FF." | |
| 67 ; '(dimension | |
| 68 ; 2 | |
| 69 ; registry "ISO10646-1" | |
| 70 ; chars 96 | |
| 71 ; columns 1 | |
| 72 ; direction l2r | |
| 73 ; final ?1 | |
| 74 ; graphic 0 | |
| 75 ; short-name "Unicode subset" | |
| 76 ; long-name "Unicode subset (U+0100..U+24FF)" | |
| 77 ; )) | |
| 78 | |
| 79 | |
| 2367 | 80 ;; accessed in loadup.el, mule-cmds.el; see discussion in unicode.c |
| 81 (defvar load-unicode-tables-at-dump-time (eq system-type 'windows-nt) | |
| 82 "[INTERNAL] Whether to load the Unicode tables at dump time. | |
| 83 Setting this at run-time does nothing.") | |
| 84 | |
| 771 | 85 ;; NOTE: This takes only a fraction of a second on my Pentium III |
| 86 ;; 700Mhz even with a totally optimization-disabled XEmacs. | |
| 87 (defun load-unicode-tables () | |
| 88 "Initialize the Unicode translation tables for all standard charsets." | |
| 780 | 89 (let ((parse-args |
| 90 '(("unicode/unicode-consortium" | |
| 877 | 91 ;; Due to the braindamaged way Mule treats the ASCII and Control-1 |
| 92 ;; charsets' types, trying to load them results in out-of-range | |
| 93 ;; warnings at unicode.c:1439. They're no-ops anyway, they're | |
| 94 ;; hardwired in unicode.c (unicode_to_ichar, ichar_to_unicode). | |
| 95 ;; ("8859-1.TXT" ascii #x00 #x7F #x0) | |
| 96 ;; ("8859-1.TXT" control-1 #x80 #x9F #x-80) | |
| 97 ;; The 8859-1.TXT G1 assignments are half no-ops, hardwired in | |
| 98 ;; unicode.c ichar_to_unicode, but not in unicode_to_ichar. | |
| 780 | 99 ("8859-1.TXT" latin-iso8859-1 #xA0 #xFF #x-80) |
| 100 ;; "8859-10.TXT" | |
| 101 ;; "8859-13.TXT" | |
| 102 ("8859-14.TXT" latin-iso8859-14 #xA0 #xFF #x-80) | |
| 103 ("8859-15.TXT" latin-iso8859-15 #xA0 #xFF #x-80) | |
| 104 ("8859-2.TXT" latin-iso8859-2 #xA0 #xFF #x-80) | |
| 105 ("8859-3.TXT" latin-iso8859-3 #xA0 #xFF #x-80) | |
| 106 ("8859-4.TXT" latin-iso8859-4 #xA0 #xFF #x-80) | |
| 107 ("8859-5.TXT" cyrillic-iso8859-5 #xA0 #xFF #x-80) | |
| 108 ("8859-6.TXT" arabic-iso8859-6 #xA0 #xFF #x-80) | |
| 109 ("8859-7.TXT" greek-iso8859-7 #xA0 #xFF #x-80) | |
| 110 ("8859-8.TXT" hebrew-iso8859-8 #xA0 #xFF #x-80) | |
| 111 ("8859-9.TXT" latin-iso8859-9 #xA0 #xFF #x-80) | |
| 112 ;; charset for Big5 does not matter; specifying `big5' will | |
| 113 ;; automatically make the right thing happen | |
| 114 ("BIG5.TXT" chinese-big5-1 nil nil nil big5) | |
| 115 ("CNS11643.TXT" chinese-cns11643-1 #x10000 #x1FFFF #x-10000) | |
| 116 ("CNS11643.TXT" chinese-cns11643-2 #x20000 #x2FFFF #x-20000) | |
| 117 ;; "CP1250.TXT" | |
| 118 ;; "CP1251.TXT" | |
| 119 ;; "CP1252.TXT" | |
| 120 ;; "CP1253.TXT" | |
| 121 ;; "CP1254.TXT" | |
| 122 ;; "CP1255.TXT" | |
| 123 ;; "CP1256.TXT" | |
| 124 ;; "CP1257.TXT" | |
| 125 ;; "CP1258.TXT" | |
| 126 ;; "CP874.TXT" | |
| 127 ;; "CP932.TXT" | |
| 128 ;; "CP936.TXT" | |
| 129 ;; "CP949.TXT" | |
| 130 ;; "CP950.TXT" | |
| 131 ;; "GB12345.TXT" | |
| 132 ("GB2312.TXT" chinese-gb2312) | |
| 2297 | 133 ;; "HANGUL.TXT" |
| 134 ;; #### shouldn't JIS X 0201's upper limit be 7f? | |
| 780 | 135 ("JIS0201.TXT" latin-jisx0201 #x21 #x80) |
| 136 ("JIS0201.TXT" katakana-jisx0201 #xA0 #xFF #x-80) | |
| 137 ("JIS0208.TXT" japanese-jisx0208 nil nil nil ignore-first-column) | |
| 138 ("JIS0212.TXT" japanese-jisx0212) | |
| 139 ;; "JOHAB.TXT" | |
| 140 ;; "KOI8-R.TXT" | |
| 141 ;; "KSC5601.TXT" | |
| 142 ;; note that KSC5601.TXT as currently distributed is NOT what | |
| 143 ;; it claims to be! see comments in KSX1001.TXT. | |
| 144 ("KSX1001.TXT" korean-ksc5601) | |
| 145 ;; "OLD5601.TXT" | |
| 146 ;; "SHIFTJIS.TXT" | |
| 147 ) | |
| 148 ("unicode/mule-ucs" | |
| 2297 | 149 ;; #### we don't support surrogates?!?? |
| 780 | 150 ;; use these instead of the above ones once we support surrogates |
| 151 ;;("chinese-cns11643-1.txt" chinese-cns11643-1) | |
| 152 ;;("chinese-cns11643-2.txt" chinese-cns11643-2) | |
| 153 ;;("chinese-cns11643-3.txt" chinese-cns11643-3) | |
| 154 ;;("chinese-cns11643-4.txt" chinese-cns11643-4) | |
| 155 ;;("chinese-cns11643-5.txt" chinese-cns11643-5) | |
| 156 ;;("chinese-cns11643-6.txt" chinese-cns11643-6) | |
| 157 ;;("chinese-cns11643-7.txt" chinese-cns11643-7) | |
| 158 ("chinese-sisheng.txt" chinese-sisheng) | |
| 159 ("ethiopic.txt" ethiopic) | |
| 160 ("indian-is13194.txt" indian-is13194) | |
| 161 ("ipa.txt" ipa) | |
| 162 ("thai-tis620.txt" thai-tis620) | |
| 163 ("tibetan.txt" tibetan) | |
| 164 ("vietnamese-viscii-lower.txt" vietnamese-viscii-lower) | |
| 165 ("vietnamese-viscii-upper.txt" vietnamese-viscii-upper) | |
| 166 ) | |
| 167 ("unicode/other" | |
| 168 ("lao.txt" lao) | |
| 169 ) | |
| 771 | 170 ))) |
| 780 | 171 (mapcar #'(lambda (tables) |
| 172 (let ((undir | |
| 173 (expand-file-name (car tables) data-directory))) | |
| 174 (mapcar #'(lambda (args) | |
| 1318 | 175 (apply 'load-unicode-mapping-table |
| 780 | 176 (expand-file-name (car args) undir) |
| 177 (cdr args))) | |
| 178 (cdr tables)))) | |
| 771 | 179 parse-args))) |
| 180 | |
| 181 (make-coding-system | |
| 182 'utf-16 'unicode | |
| 183 "UTF-16" | |
| 184 '(mnemonic "UTF-16" | |
| 185 documentation | |
| 186 "UTF-16 Unicode encoding -- the standard (almost-) fixed-width | |
| 187 two-byte encoding, with surrogates. It will be fixed-width if all | |
| 188 characters are in the BMP (Basic Multilingual Plane -- first 65536 | |
| 189 codepoints). Cannot represent characters with codepoints above | |
| 190 0x10FFFF (a little more than 1,000,000). Unicode and ISO guarantee | |
| 191 never to encode any characters outside this range -- all the rest are | |
| 192 for private, corporate or internal use." | |
| 193 type utf-16)) | |
| 194 | |
| 195 (make-coding-system | |
| 196 'utf-16-bom 'unicode | |
| 197 "UTF-16 w/BOM" | |
| 198 '(mnemonic "UTF16-BOM" | |
| 199 documentation | |
| 200 "UTF-16 Unicode encoding with byte order mark (BOM) at the beginning. | |
| 201 The BOM is Unicode character U+FEFF -- i.e. the first two bytes are | |
| 202 0xFE and 0xFF, respectively, or reversed in a little-endian | |
| 203 representation. It has been sanctioned by the Unicode Consortium for | |
| 204 use at the beginning of a Unicode stream as a marker of the byte order | |
| 205 of the stream, and commonly appears in Unicode files under Microsoft | |
| 206 Windows, where it also functions as a magic cookie identifying a | |
| 207 Unicode file. The character is called \"ZERO WIDTH NO-BREAK SPACE\" | |
| 208 and is suitable as a byte-order marker because: | |
| 209 | |
| 210 -- it has no displayable representation | |
| 211 -- due to its semantics it never normally appears at the beginning | |
| 212 of a stream | |
| 213 -- its reverse U+FFFE is not a legal Unicode character | |
| 214 -- neither byte sequence is at all likely in any other standard | |
| 215 encoding, particularly at the beginning of a stream | |
| 216 | |
| 217 This coding system will insert a BOM at the beginning of a stream when | |
| 218 writing and strip it off when reading." | |
| 219 type utf-16 | |
| 220 need-bom t)) | |
| 221 | |
| 222 (make-coding-system | |
| 223 'utf-16-little-endian 'unicode | |
| 224 "UTF-16 Little Endian" | |
| 225 '(mnemonic "UTF16-LE" | |
| 226 documentation | |
| 227 "Little-endian version of UTF-16 Unicode encoding. | |
| 228 See `utf-16' coding system." | |
| 229 type utf-16 | |
| 230 little-endian t)) | |
| 231 | |
| 232 (make-coding-system | |
| 233 'utf-16-little-endian-bom 'unicode | |
| 234 "UTF-16 Little Endian w/BOM" | |
| 235 '(mnemonic "MSW-Unicode" | |
| 236 documentation | |
| 237 "Little-endian version of UTF-16 Unicode encoding, with byte order mark. | |
| 238 Standard encoding for representing Unicode under MS Windows. See | |
| 239 `utf-16-bom' coding system." | |
| 240 type utf-16 | |
| 241 little-endian t | |
| 242 need-bom t)) | |
| 243 | |
| 244 (make-coding-system | |
| 245 'ucs-4 'unicode | |
| 246 "UCS-4" | |
| 247 '(mnemonic "UCS4" | |
| 248 documentation | |
| 249 "UCS-4 Unicode encoding -- fully fixed-width four-byte encoding." | |
| 250 type ucs-4)) | |
| 251 | |
| 252 (make-coding-system | |
| 253 'ucs-4-little-endian 'unicode | |
| 254 "UCS-4 Little Endian" | |
| 255 '(mnemonic "UCS4-LE" | |
| 256 documentation | |
| 2297 | 257 ;; #### I don't think this is permitted by ISO 10646, only Unicode. |
| 258 ;; Call it UTF-32 instead? | |
| 771 | 259 "Little-endian version of UCS-4 Unicode encoding. See `ucs-4' coding system." |
| 260 type ucs-4 | |
| 261 little-endian t)) | |
| 262 | |
| 263 (make-coding-system | |
| 264 'utf-8 'unicode | |
| 265 "UTF-8" | |
| 266 '(mnemonic "UTF8" | |
| 267 documentation | |
| 268 "UTF-8 Unicode encoding -- ASCII-compatible 8-bit variable-width encoding | |
| 2297 | 269 sharing the following principles with the Mule-internal encoding: |
| 771 | 270 |
| 271 -- All ASCII characters (codepoints 0 through 127) are represented | |
| 272 by themselves (i.e. using one byte, with the same value as the | |
| 273 ASCII codepoint), and these bytes are disjoint from bytes | |
| 274 representing non-ASCII characters. | |
| 275 | |
| 276 This means that any 8-bit clean application can safely process | |
| 277 UTF-8-encoded text as it were ASCII, with no corruption (e.g. a | |
| 278 '/' byte is always a slash character, never the second byte of | |
| 279 some other character, as with Big5, so a pathname encoded in | |
| 280 UTF-8 can safely be split up into components and reassembled | |
| 281 again using standard ASCII processes). | |
| 282 | |
| 283 -- Leading bytes and non-leading bytes in the encoding of a | |
| 284 character are disjoint, so moving backwards is easy. | |
| 285 | |
| 286 -- Given only the leading byte, you know how many following bytes | |
| 287 are present. | |
| 288 " | |
| 289 type utf-8)) | |
| 290 | |
| 985 | 291 (make-coding-system |
| 292 'utf-8-bom 'unicode | |
| 293 "UTF-8 w/BOM" | |
| 294 '(mnemonic "MSW-UTF8" | |
| 295 documentation | |
| 296 "UTF-8 Unicode encoding, with byte order mark. | |
| 297 Standard encoding for representing UTF-8 under MS Windows." | |
| 298 type utf-8 | |
| 299 little-endian t | |
| 300 need-bom t)) | |
| 301 | |
| 771 | 302 ;; #### UTF-7 is not yet implemented, and it's tricky to do. There's |
| 303 ;; an implementation in appendix A.1 of the Unicode Standard, Version | |
| 304 ;; 2.0, but I don't know its licensing characteristics. | |
| 305 | |
| 306 ; (make-coding-system | |
| 307 ; 'utf-7 'unicode | |
| 308 ; "UTF-7" | |
| 309 ; '(mnemonic "UTF7" | |
| 310 ; documentation | |
| 311 ; "UTF-7 Unicode encoding -- 7-bit-ASCII modal Internet-mail-compatible | |
| 312 ; encoding especially designed for headers, with the following | |
| 313 ; properties: | |
| 314 | |
| 315 ; -- Only characters that are considered safe for passing through any mail | |
| 316 ; gateway without damage are used. | |
| 317 | |
| 318 ; -- This is a modal encoding, with two states. The first, default | |
| 319 ; state encodes the most common Unicode characters (upper and | |
| 320 ; lowercase letters, digits, and 9 common punctuation marks) as | |
| 321 ; themselves, and the second state, entered using '+' and | |
| 322 ; terminated with '-' or any character disallowed in state 2, | |
| 323 ; encodes any Unicode characters by first converting to UTF-16, | |
| 324 ; most significant byte first, and then to a slightly modified | |
| 325 ; Base64 encoding. (Thus, UTF-7 has the same limitations on the | |
| 326 ; characters it can encode as UTF-16.) | |
| 327 | |
| 328 ; -- The modified Base64 encoding deviates from standard Base64 in | |
| 329 ; that it omits the `=' pad character. This is eliminated so as to | |
| 330 ; avoid conflicts with the use of `=' as an escape in the | |
| 331 ; Quoted-Printable encoding and the related Q encoding for headers: | |
| 332 ; With this modification, non-whitespace chars in UTF-7 will be | |
| 333 ; represented in Quoted-Printable and in Q as-is, with no further | |
| 334 ; encoding. | |
| 335 | |
| 336 ; For more information, see Appendix A.1 of The Unicode Standard 2.0, or | |
| 337 ; wherever it is in v3.0." | |
| 338 ; type utf-7)) |