annotate lisp/unicode.el @ 4108:5da4cc7d5968

[xemacs-hg @ 2007-08-09 06:22:51 by michaels] 2007-08-07 Mike Sperber <mike@xemacs.org> * setup-paths.el (paths-find-doc-directory): (paths-find-exec-directory): (paths-find-lisp-directory): (paths-find-mule-lisp-directory): (paths-construct-info-path): (paths-find-data-directory): * packages.el (packages-find-installation-package-directories): * find-paths.el (paths-for-each-emacs-directory): (paths-find-emacs-directories): (paths-find-emacs-directory): (paths-for-each-site-directory): (paths-find-site-directory): (paths-find-site-directories): (paths-for-each-version-directory): (paths-find-version-directories): (paths-find-version-directory): Generalize to multiple bases. (paths-find-architecture-directory): Use above to give roots precedence over bases. This means, for example, that a directory in an in-place root will always get precedence over an installed root.
author michaels
date Thu, 09 Aug 2007 06:22:53 +0000
parents 1abf84db2c7f
children edb00a8b4eff
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
1 ;;; unicode.el --- Unicode support -*- coding: iso-2022-7bit; -*-
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
2
778
2923009caf47 [xemacs-hg @ 2002-03-16 10:38:59 by ben]
ben
parents: 771
diff changeset
3 ;; Copyright (C) 2001, 2002 Ben Wing.
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
4
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
5 ;; Keywords: multilingual, Unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
6
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
7 ;; This file is part of XEmacs.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
8
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
9 ;; XEmacs is free software; you can redistribute it and/or modify it
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
10 ;; under the terms of the GNU General Public License as published by
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
11 ;; the Free Software Foundation; either version 2, or (at your option)
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
12 ;; any later version.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
13
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
14 ;; XEmacs is distributed in the hope that it will be useful, but
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
15 ;; WITHOUT ANY WARRANTY; without even the implied warranty of
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
16 ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
17 ;; General Public License for more details.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
18
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
19 ;; You should have received a copy of the GNU General Public License
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
20 ;; along with XEmacs; see the file COPYING. If not, write to the Free
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
21 ;; Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
22 ;; 02111-1307, USA.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
23
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
24 ;;; Synched up with: Not in FSF.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
25
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
26 ;;; Commentary:
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
27
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
28 ;; Lisp support for Unicode, e.g. initialize the translation tables.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
29
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
30 ;;; Code:
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
31
3659
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan]
aidan
parents: 3506
diff changeset
32 ;; GNU Emacs has the charsets:
778
2923009caf47 [xemacs-hg @ 2002-03-16 10:38:59 by ben]
ben
parents: 771
diff changeset
33
3659
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan]
aidan
parents: 3506
diff changeset
34 ;; mule-unicode-2500-33ff
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan]
aidan
parents: 3506
diff changeset
35 ;; mule-unicode-e000-ffff
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan]
aidan
parents: 3506
diff changeset
36 ;; mule-unicode-0100-24ff
778
2923009caf47 [xemacs-hg @ 2002-03-16 10:38:59 by ben]
ben
parents: 771
diff changeset
37
3659
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan]
aidan
parents: 3506
diff changeset
38 ;; built-in. This is hack--and an incomplete hack at that--against the
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan]
aidan
parents: 3506
diff changeset
39 ;; spirit and the letter of standard ISO 2022 character sets. Instead of
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan]
aidan
parents: 3506
diff changeset
40 ;; this, we have the jit-ucs-charset-N Mule character sets, created in
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan]
aidan
parents: 3506
diff changeset
41 ;; unicode.c on encountering a Unicode code point that we don't recognise,
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan]
aidan
parents: 3506
diff changeset
42 ;; and saved in ISO 2022 coding systems using the UTF-8 escape described in
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan]
aidan
parents: 3506
diff changeset
43 ;; ISO-IR 196.
778
2923009caf47 [xemacs-hg @ 2002-03-16 10:38:59 by ben]
ben
parents: 771
diff changeset
44
4083
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
45 (eval-when-compile (when (featurep 'mule) (require 'ccl)))
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
46
2367
ecf1ebac70d8 [xemacs-hg @ 2004-11-04 23:05:23 by ben]
ben
parents: 2297
diff changeset
47 ;; accessed in loadup.el, mule-cmds.el; see discussion in unicode.c
ecf1ebac70d8 [xemacs-hg @ 2004-11-04 23:05:23 by ben]
ben
parents: 2297
diff changeset
48 (defvar load-unicode-tables-at-dump-time (eq system-type 'windows-nt)
ecf1ebac70d8 [xemacs-hg @ 2004-11-04 23:05:23 by ben]
ben
parents: 2297
diff changeset
49 "[INTERNAL] Whether to load the Unicode tables at dump time.
ecf1ebac70d8 [xemacs-hg @ 2004-11-04 23:05:23 by ben]
ben
parents: 2297
diff changeset
50 Setting this at run-time does nothing.")
ecf1ebac70d8 [xemacs-hg @ 2004-11-04 23:05:23 by ben]
ben
parents: 2297
diff changeset
51
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
52 ;; NOTE: This takes only a fraction of a second on my Pentium III
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
53 ;; 700Mhz even with a totally optimization-disabled XEmacs.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
54 (defun load-unicode-tables ()
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
55 "Initialize the Unicode translation tables for all standard charsets."
780
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
56 (let ((parse-args
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
57 '(("unicode/unicode-consortium"
877
e54d47b2d736 [xemacs-hg @ 2002-06-23 09:54:35 by stephent]
stephent
parents: 780
diff changeset
58 ;; Due to the braindamaged way Mule treats the ASCII and Control-1
e54d47b2d736 [xemacs-hg @ 2002-06-23 09:54:35 by stephent]
stephent
parents: 780
diff changeset
59 ;; charsets' types, trying to load them results in out-of-range
e54d47b2d736 [xemacs-hg @ 2002-06-23 09:54:35 by stephent]
stephent
parents: 780
diff changeset
60 ;; warnings at unicode.c:1439. They're no-ops anyway, they're
e54d47b2d736 [xemacs-hg @ 2002-06-23 09:54:35 by stephent]
stephent
parents: 780
diff changeset
61 ;; hardwired in unicode.c (unicode_to_ichar, ichar_to_unicode).
e54d47b2d736 [xemacs-hg @ 2002-06-23 09:54:35 by stephent]
stephent
parents: 780
diff changeset
62 ;; ("8859-1.TXT" ascii #x00 #x7F #x0)
e54d47b2d736 [xemacs-hg @ 2002-06-23 09:54:35 by stephent]
stephent
parents: 780
diff changeset
63 ;; ("8859-1.TXT" control-1 #x80 #x9F #x-80)
e54d47b2d736 [xemacs-hg @ 2002-06-23 09:54:35 by stephent]
stephent
parents: 780
diff changeset
64 ;; The 8859-1.TXT G1 assignments are half no-ops, hardwired in
e54d47b2d736 [xemacs-hg @ 2002-06-23 09:54:35 by stephent]
stephent
parents: 780
diff changeset
65 ;; unicode.c ichar_to_unicode, but not in unicode_to_ichar.
780
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
66 ("8859-1.TXT" latin-iso8859-1 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
67 ;; "8859-10.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
68 ;; "8859-13.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
69 ("8859-14.TXT" latin-iso8859-14 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
70 ("8859-15.TXT" latin-iso8859-15 #xA0 #xFF #x-80)
2575
e71117a6ddac [xemacs-hg @ 2005-02-09 15:29:07 by aidan]
aidan
parents: 2574
diff changeset
71 ("8859-16.TXT" latin-iso8859-16 #xA0 #xFF #x-80)
780
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
72 ("8859-2.TXT" latin-iso8859-2 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
73 ("8859-3.TXT" latin-iso8859-3 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
74 ("8859-4.TXT" latin-iso8859-4 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
75 ("8859-5.TXT" cyrillic-iso8859-5 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
76 ("8859-6.TXT" arabic-iso8859-6 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
77 ("8859-7.TXT" greek-iso8859-7 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
78 ("8859-8.TXT" hebrew-iso8859-8 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
79 ("8859-9.TXT" latin-iso8859-9 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
80 ;; charset for Big5 does not matter; specifying `big5' will
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
81 ;; automatically make the right thing happen
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
82 ("BIG5.TXT" chinese-big5-1 nil nil nil big5)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
83 ("CNS11643.TXT" chinese-cns11643-1 #x10000 #x1FFFF #x-10000)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
84 ("CNS11643.TXT" chinese-cns11643-2 #x20000 #x2FFFF #x-20000)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
85 ;; "CP1250.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
86 ;; "CP1251.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
87 ;; "CP1252.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
88 ;; "CP1253.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
89 ;; "CP1254.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
90 ;; "CP1255.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
91 ;; "CP1256.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
92 ;; "CP1257.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
93 ;; "CP1258.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
94 ;; "CP874.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
95 ;; "CP932.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
96 ;; "CP936.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
97 ;; "CP949.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
98 ;; "CP950.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
99 ;; "GB12345.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
100 ("GB2312.TXT" chinese-gb2312)
2297
13a418960a88 [xemacs-hg @ 2004-09-22 02:05:42 by stephent]
stephent
parents: 1318
diff changeset
101 ;; "HANGUL.TXT"
13a418960a88 [xemacs-hg @ 2004-09-22 02:05:42 by stephent]
stephent
parents: 1318
diff changeset
102 ;; #### shouldn't JIS X 0201's upper limit be 7f?
780
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
103 ("JIS0201.TXT" latin-jisx0201 #x21 #x80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
104 ("JIS0201.TXT" katakana-jisx0201 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
105 ("JIS0208.TXT" japanese-jisx0208 nil nil nil ignore-first-column)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
106 ("JIS0212.TXT" japanese-jisx0212)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
107 ;; "JOHAB.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
108 ;; "KOI8-R.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
109 ;; "KSC5601.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
110 ;; note that KSC5601.TXT as currently distributed is NOT what
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
111 ;; it claims to be! see comments in KSX1001.TXT.
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
112 ("KSX1001.TXT" korean-ksc5601)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
113 ;; "OLD5601.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
114 ;; "SHIFTJIS.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
115 )
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
116 ("unicode/mule-ucs"
2297
13a418960a88 [xemacs-hg @ 2004-09-22 02:05:42 by stephent]
stephent
parents: 1318
diff changeset
117 ;; #### we don't support surrogates?!??
780
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
118 ;; use these instead of the above ones once we support surrogates
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
119 ;;("chinese-cns11643-1.txt" chinese-cns11643-1)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
120 ;;("chinese-cns11643-2.txt" chinese-cns11643-2)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
121 ;;("chinese-cns11643-3.txt" chinese-cns11643-3)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
122 ;;("chinese-cns11643-4.txt" chinese-cns11643-4)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
123 ;;("chinese-cns11643-5.txt" chinese-cns11643-5)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
124 ;;("chinese-cns11643-6.txt" chinese-cns11643-6)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
125 ;;("chinese-cns11643-7.txt" chinese-cns11643-7)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
126 ("chinese-sisheng.txt" chinese-sisheng)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
127 ("ethiopic.txt" ethiopic)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
128 ("indian-is13194.txt" indian-is13194)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
129 ("ipa.txt" ipa)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
130 ("thai-tis620.txt" thai-tis620)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
131 ("tibetan.txt" tibetan)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
132 ("vietnamese-viscii-lower.txt" vietnamese-viscii-lower)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
133 ("vietnamese-viscii-upper.txt" vietnamese-viscii-upper)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
134 )
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
135 ("unicode/other"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
136 ("lao.txt" lao)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
137 )
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
138 )))
780
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
139 (mapcar #'(lambda (tables)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
140 (let ((undir
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
141 (expand-file-name (car tables) data-directory)))
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
142 (mapcar #'(lambda (args)
1318
b531bf8658e9 [xemacs-hg @ 2003-02-21 06:56:46 by ben]
ben
parents: 985
diff changeset
143 (apply 'load-unicode-mapping-table
780
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
144 (expand-file-name (car args) undir)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
145 (cdr args)))
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben]
ben
parents: 778
diff changeset
146 (cdr tables))))
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
147 parse-args)))
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
148
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
149 (make-coding-system
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
150 'utf-16 'unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
151 "UTF-16"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
152 '(mnemonic "UTF-16"
3767
6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan]
aidan
parents: 3667
diff changeset
153 documentation
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
154 "UTF-16 Unicode encoding -- the standard (almost-) fixed-width
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
155 two-byte encoding, with surrogates. It will be fixed-width if all
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
156 characters are in the BMP (Basic Multilingual Plane -- first 65536
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
157 codepoints). Cannot represent characters with codepoints above
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
158 0x10FFFF (a little more than 1,000,000). Unicode and ISO guarantee
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
159 never to encode any characters outside this range -- all the rest are
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
160 for private, corporate or internal use."
3767
6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan]
aidan
parents: 3667
diff changeset
161 unicode-type utf-16))
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
162
2574
5e2653bc0ab0 [xemacs-hg @ 2005-02-08 23:59:50 by aidan]
aidan
parents: 2367
diff changeset
163 (define-coding-system-alias 'utf-16-be 'utf-16)
5e2653bc0ab0 [xemacs-hg @ 2005-02-08 23:59:50 by aidan]
aidan
parents: 2367
diff changeset
164
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
165 (make-coding-system
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
166 'utf-16-bom 'unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
167 "UTF-16 w/BOM"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
168 '(mnemonic "UTF16-BOM"
3767
6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan]
aidan
parents: 3667
diff changeset
169 documentation
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
170 "UTF-16 Unicode encoding with byte order mark (BOM) at the beginning.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
171 The BOM is Unicode character U+FEFF -- i.e. the first two bytes are
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
172 0xFE and 0xFF, respectively, or reversed in a little-endian
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
173 representation. It has been sanctioned by the Unicode Consortium for
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
174 use at the beginning of a Unicode stream as a marker of the byte order
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
175 of the stream, and commonly appears in Unicode files under Microsoft
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
176 Windows, where it also functions as a magic cookie identifying a
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
177 Unicode file. The character is called \"ZERO WIDTH NO-BREAK SPACE\"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
178 and is suitable as a byte-order marker because:
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
179
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
180 -- it has no displayable representation
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
181 -- due to its semantics it never normally appears at the beginning
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
182 of a stream
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
183 -- its reverse U+FFFE is not a legal Unicode character
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
184 -- neither byte sequence is at all likely in any other standard
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
185 encoding, particularly at the beginning of a stream
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
186
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
187 This coding system will insert a BOM at the beginning of a stream when
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
188 writing and strip it off when reading."
3767
6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan]
aidan
parents: 3667
diff changeset
189 unicode-type utf-16
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
190 need-bom t))
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
191
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
192 (make-coding-system
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
193 'utf-16-little-endian 'unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
194 "UTF-16 Little Endian"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
195 '(mnemonic "UTF16-LE"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
196 documentation
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
197 "Little-endian version of UTF-16 Unicode encoding.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
198 See `utf-16' coding system."
3767
6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan]
aidan
parents: 3667
diff changeset
199 unicode-type utf-16
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
200 little-endian t))
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
201
2574
5e2653bc0ab0 [xemacs-hg @ 2005-02-08 23:59:50 by aidan]
aidan
parents: 2367
diff changeset
202 (define-coding-system-alias 'utf-16-le 'utf-16-little-endian)
5e2653bc0ab0 [xemacs-hg @ 2005-02-08 23:59:50 by aidan]
aidan
parents: 2367
diff changeset
203
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
204 (make-coding-system
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
205 'utf-16-little-endian-bom 'unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
206 "UTF-16 Little Endian w/BOM"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
207 '(mnemonic "MSW-Unicode"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
208 documentation
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
209 "Little-endian version of UTF-16 Unicode encoding, with byte order mark.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
210 Standard encoding for representing Unicode under MS Windows. See
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
211 `utf-16-bom' coding system."
3767
6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan]
aidan
parents: 3667
diff changeset
212 unicode-type utf-16
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
213 little-endian t
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
214 need-bom t))
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
215
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
216 (make-coding-system
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
217 'ucs-4 'unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
218 "UCS-4"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
219 '(mnemonic "UCS4"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
220 documentation
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
221 "UCS-4 Unicode encoding -- fully fixed-width four-byte encoding."
3767
6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan]
aidan
parents: 3667
diff changeset
222 unicode-type ucs-4))
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
223
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
224 (make-coding-system
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
225 'ucs-4-little-endian 'unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
226 "UCS-4 Little Endian"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
227 '(mnemonic "UCS4-LE"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
228 documentation
2297
13a418960a88 [xemacs-hg @ 2004-09-22 02:05:42 by stephent]
stephent
parents: 1318
diff changeset
229 ;; #### I don't think this is permitted by ISO 10646, only Unicode.
13a418960a88 [xemacs-hg @ 2004-09-22 02:05:42 by stephent]
stephent
parents: 1318
diff changeset
230 ;; Call it UTF-32 instead?
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
231 "Little-endian version of UCS-4 Unicode encoding. See `ucs-4' coding system."
3767
6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan]
aidan
parents: 3667
diff changeset
232 unicode-type ucs-4
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
233 little-endian t))
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
234
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
235 (make-coding-system
4096
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
236 'utf-32 'unicode
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
237 "UTF-32"
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
238 '(mnemonic "UTF32"
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
239 documentation
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
240 "UTF-32 Unicode encoding -- fixed-width four-byte encoding,
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
241 characters less than #x10FFFF are not supported. "
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
242 unicode-type utf-32))
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
243
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
244 (make-coding-system
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
245 'utf-32-little-endian 'unicode
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
246 "UTF-32 Little Endian"
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
247 '(mnemonic "UTF32-LE"
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
248 documentation
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
249 "Little-endian version of UTF-32 Unicode encoding.
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
250
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
251 A fixed-width four-byte encoding, characters less than #x10FFFF are not
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
252 supported. "
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
253 unicode-type ucs-4 little-endian t))
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
254
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
255 (make-coding-system
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
256 'utf-8 'unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
257 "UTF-8"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
258 '(mnemonic "UTF8"
3767
6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan]
aidan
parents: 3667
diff changeset
259 documentation "
6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan]
aidan
parents: 3667
diff changeset
260 UTF-8 Unicode encoding -- ASCII-compatible 8-bit variable-width encoding
2297
13a418960a88 [xemacs-hg @ 2004-09-22 02:05:42 by stephent]
stephent
parents: 1318
diff changeset
261 sharing the following principles with the Mule-internal encoding:
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
262
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
263 -- All ASCII characters (codepoints 0 through 127) are represented
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
264 by themselves (i.e. using one byte, with the same value as the
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
265 ASCII codepoint), and these bytes are disjoint from bytes
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
266 representing non-ASCII characters.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
267
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
268 This means that any 8-bit clean application can safely process
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
269 UTF-8-encoded text as it were ASCII, with no corruption (e.g. a
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
270 '/' byte is always a slash character, never the second byte of
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
271 some other character, as with Big5, so a pathname encoded in
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
272 UTF-8 can safely be split up into components and reassembled
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
273 again using standard ASCII processes).
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
274
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
275 -- Leading bytes and non-leading bytes in the encoding of a
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
276 character are disjoint, so moving backwards is easy.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
277
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
278 -- Given only the leading byte, you know how many following bytes
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
279 are present.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
280 "
3767
6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan]
aidan
parents: 3667
diff changeset
281 unicode-type utf-8))
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
282
985
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs]
youngs
parents: 877
diff changeset
283 (make-coding-system
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs]
youngs
parents: 877
diff changeset
284 'utf-8-bom 'unicode
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs]
youngs
parents: 877
diff changeset
285 "UTF-8 w/BOM"
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs]
youngs
parents: 877
diff changeset
286 '(mnemonic "MSW-UTF8"
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs]
youngs
parents: 877
diff changeset
287 documentation
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs]
youngs
parents: 877
diff changeset
288 "UTF-8 Unicode encoding, with byte order mark.
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs]
youngs
parents: 877
diff changeset
289 Standard encoding for representing UTF-8 under MS Windows."
3767
6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan]
aidan
parents: 3667
diff changeset
290 unicode-type utf-8
985
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs]
youngs
parents: 877
diff changeset
291 little-endian t
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs]
youngs
parents: 877
diff changeset
292 need-bom t))
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs]
youngs
parents: 877
diff changeset
293
2633
b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan]
aidan
parents: 2575
diff changeset
294 (defun decode-char (quote-ucs code &optional restriction)
3659
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan]
aidan
parents: 3506
diff changeset
295 "FSF compatibility--return Mule character with Unicode codepoint CODE.
2633
b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan]
aidan
parents: 2575
diff changeset
296 The second argument must be 'ucs, the third argument is ignored. "
4096
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
297 ;; We're prepared to accept invalid Unicode in unicode-to-char, but not in
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
298 ;; this function, which is the API that should actually be used, since
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
299 ;; it's available in GNU and in Mule-UCS.
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan]
aidan
parents: 4083
diff changeset
300 (check-argument-range code #x0 #x10FFFF)
3506
bcc2611d4cfc [xemacs-hg @ 2006-07-13 20:45:48 by aidan]
aidan
parents: 3439
diff changeset
301 (assert (eq quote-ucs 'ucs) t
2633
b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan]
aidan
parents: 2575
diff changeset
302 "Sorry, decode-char doesn't yet support anything but the UCS. ")
b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan]
aidan
parents: 2575
diff changeset
303 (unicode-to-char code))
b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan]
aidan
parents: 2575
diff changeset
304
b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan]
aidan
parents: 2575
diff changeset
305 (defun encode-char (char quote-ucs &optional restriction)
3659
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan]
aidan
parents: 3506
diff changeset
306 "FSF compatibility--return the Unicode code point of CHAR.
2633
b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan]
aidan
parents: 2575
diff changeset
307 The second argument must be 'ucs, the third argument is ignored. "
3506
bcc2611d4cfc [xemacs-hg @ 2006-07-13 20:45:48 by aidan]
aidan
parents: 3439
diff changeset
308 (assert (eq quote-ucs 'ucs) t
2633
b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan]
aidan
parents: 2575
diff changeset
309 "Sorry, encode-char doesn't yet support anything but the UCS. ")
b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan]
aidan
parents: 2575
diff changeset
310 (char-to-unicode char))
b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan]
aidan
parents: 2575
diff changeset
311
3667
4c8ad140bcec [xemacs-hg @ 2006-11-07 18:51:21 by aidan]
aidan
parents: 3666
diff changeset
312 (when (featurep 'mule)
4083
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
313 (let ((prog
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
314 (eval-when-compile
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
315 (let ((pre-existing
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
316 ;; This is the compiled CCL program from the assert
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
317 ;; below. Since this file is dumped and ccl.el isn't (and
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
318 ;; even when it was, it was dumped much later than this
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
319 ;; one), we can't compile the program at dump time. We can
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
320 ;; check at byte compile time that the program is as
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
321 ;; expected, though.
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
322 [1 16 131127 7 98872 65823 1307 5 -65536 65313 64833 1028 147513
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
323 8 82009 255 22]))
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
324 (when (featurep 'mule)
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
325 ;; Check that the pre-existing constant reflects the intended
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
326 ;; CCL program.
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
327 (assert
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
328 (equal pre-existing
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
329 (ccl-compile
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
330 `(1
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
331 (;; mule-to-unicode's first argument is the
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
332 ;; charset ID, the second its first byte
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
333 ;; left shifted by 7 bits masked with its
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
334 ;; second byte.
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
335 (r1 = (r1 << 7))
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
336 (r1 = (r1 | r2))
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
337 (mule-to-unicode r0 r1)
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
338 (if (r0 & ,(lognot #xFFFF))
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
339 ;; Redisplay looks in r1 and r2 for the first
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
340 ;; and second bytes of the X11 font,
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
341 ;; respectively. For non-BMP characters we
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
342 ;; display U+FFFD.
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
343 ((r1 = #xFF)
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
344 (r2 = #xFD))
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
345 ((r1 = (r0 >> 8))
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
346 (r2 = (r0 & #xFF))))))))
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
347 nil
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
348 "The pre-compiled CCL program appears broken. "))
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
349 pre-existing))))
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
350 (defconst ccl-encode-to-ucs-2 prog
3667
4c8ad140bcec [xemacs-hg @ 2006-11-07 18:51:21 by aidan]
aidan
parents: 3666
diff changeset
351 "CCL program to transform Mule characters to UCS-2.")
4083
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
352 (put 'ccl-encode-to-ucs-2 'ccl-program-idx
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan]
aidan
parents: 4072
diff changeset
353 (register-ccl-program 'ccl-encode-to-ucs-2 prog))))
3667
4c8ad140bcec [xemacs-hg @ 2006-11-07 18:51:21 by aidan]
aidan
parents: 3666
diff changeset
354
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
355 ;; #### UTF-7 is not yet implemented, and it's tricky to do. There's
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
356 ;; an implementation in appendix A.1 of the Unicode Standard, Version
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
357 ;; 2.0, but I don't know its licensing characteristics.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
358
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
359 ; (make-coding-system
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
360 ; 'utf-7 'unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
361 ; "UTF-7"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
362 ; '(mnemonic "UTF7"
3659
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan]
aidan
parents: 3506
diff changeset
363 ; documentation; "UTF-7 Unicode encoding -- 7-bit-ASCII modal Internet-mail-compatible
771
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
364 ; encoding especially designed for headers, with the following
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
365 ; properties:
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
366
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
367 ; -- Only characters that are considered safe for passing through any mail
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
368 ; gateway without damage are used.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
369
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
370 ; -- This is a modal encoding, with two states. The first, default
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
371 ; state encodes the most common Unicode characters (upper and
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
372 ; lowercase letters, digits, and 9 common punctuation marks) as
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
373 ; themselves, and the second state, entered using '+' and
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
374 ; terminated with '-' or any character disallowed in state 2,
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
375 ; encodes any Unicode characters by first converting to UTF-16,
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
376 ; most significant byte first, and then to a slightly modified
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
377 ; Base64 encoding. (Thus, UTF-7 has the same limitations on the
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
378 ; characters it can encode as UTF-16.)
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
379
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
380 ; -- The modified Base64 encoding deviates from standard Base64 in
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
381 ; that it omits the `=' pad character. This is eliminated so as to
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
382 ; avoid conflicts with the use of `=' as an escape in the
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
383 ; Quoted-Printable encoding and the related Q encoding for headers:
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
384 ; With this modification, non-whitespace chars in UTF-7 will be
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
385 ; represented in Quoted-Printable and in Q as-is, with no further
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
386 ; encoding.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
387
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
388 ; For more information, see Appendix A.1 of The Unicode Standard 2.0, or
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben]
ben
parents:
diff changeset
389 ; wherever it is in v3.0."
3767
6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan]
aidan
parents: 3667
diff changeset
390 ; unicode-type utf-7))