xemacs-beta: lisp/unicode.el annotate

annotate lisp/unicode.el @ 4570:e6a7054a9c30

Add check-coding-systems-region, test it and others, fix some bugs. tests/ChangeLog addition: 2008-12-28 Aidan Kehoe <kehoea@parhasard.net> * automated/query-coding-tests.el: Add tests for #'unencodable-char-position, #'check-coding-systems-region, #'encode-coding-char. Remove some debugging statements. lisp/ChangeLog addition: 2008-12-28 Aidan Kehoe <kehoea@parhasard.net> * coding.el (query-coding-region): (query-coding-string): Make these defsubsts, they're short enough and they're called explicitly rarely enough that it make some sense. The alternative would be compiler macros that avoid the binding of the arguments. (unencodable-char-position): Document where the docstring and API are from. Correct a special case for zero--check-argument-type returns nil when it succeeds, we can't usefully chain its result in an and here. (check-coding-systems-region): New. API taken from GNU; docstring and implementation are independent. (encode-coding-char): Add an optional third argument, as used by recent GNU. Document the origen of the docstring. (default-query-coding-region): Add a short docstring to the non-Mule implementation of this function. * unicode.el: Don't set the query-coding-function property for unicode coding systems if we're on non-mule. Unintern unicode-query-coding-region, unicode-query-coding-skip-chars-arg in the same context.

author	Aidan Kehoe <kehoea@parhasard.net>
date	Sun, 28 Dec 2008 22:51:14 +0000
parents	1d74a1d115ee
children	e0a8715fdb1f

rev	line source
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	1 ;;; unicode.el --- Unicode support -- coding: iso-2022-7bit; --
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	2
778 2923009caf47 [xemacs-hg @ 2002-03-16 10:38:59 by ben] ben parents: 771 diff changeset	3 ;; Copyright (C) 2001, 2002 Ben Wing.
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	4
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	5 ;; Keywords: multilingual, Unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	6
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	7 ;; This file is part of XEmacs.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	8
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	9 ;; XEmacs is free software; you can redistribute it and/or modify it
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	10 ;; under the terms of the GNU General Public License as published by
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	11 ;; the Free Software Foundation; either version 2, or (at your option)
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	12 ;; any later version.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	13
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	14 ;; XEmacs is distributed in the hope that it will be useful, but
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	15 ;; WITHOUT ANY WARRANTY; without even the implied warranty of
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	16 ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	17 ;; General Public License for more details.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	18
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	19 ;; You should have received a copy of the GNU General Public License
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	20 ;; along with XEmacs; see the file COPYING. If not, write to the Free
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	21 ;; Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	22 ;; 02111-1307, USA.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	23
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	24 ;;; Synched up with: Not in FSF.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	25
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	26 ;;; Commentary:
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	27
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	28 ;; Lisp support for Unicode, e.g. initialize the translation tables.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	29
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	30 ;;; Code:
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	31
3659 98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan] aidan parents: 3506 diff changeset	32 ;; GNU Emacs has the charsets:
778 2923009caf47 [xemacs-hg @ 2002-03-16 10:38:59 by ben] ben parents: 771 diff changeset	33
3659 98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan] aidan parents: 3506 diff changeset	34 ;; mule-unicode-2500-33ff
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan] aidan parents: 3506 diff changeset	35 ;; mule-unicode-e000-ffff
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan] aidan parents: 3506 diff changeset	36 ;; mule-unicode-0100-24ff
778 2923009caf47 [xemacs-hg @ 2002-03-16 10:38:59 by ben] ben parents: 771 diff changeset	37
3659 98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan] aidan parents: 3506 diff changeset	38 ;; built-in. This is hack--and an incomplete hack at that--against the
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan] aidan parents: 3506 diff changeset	39 ;; spirit and the letter of standard ISO 2022 character sets. Instead of
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan] aidan parents: 3506 diff changeset	40 ;; this, we have the jit-ucs-charset-N Mule character sets, created in
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan] aidan parents: 3506 diff changeset	41 ;; unicode.c on encountering a Unicode code point that we don't recognise,
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan] aidan parents: 3506 diff changeset	42 ;; and saved in ISO 2022 coding systems using the UTF-8 escape described in
98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan] aidan parents: 3506 diff changeset	43 ;; ISO-IR 196.
778 2923009caf47 [xemacs-hg @ 2002-03-16 10:38:59 by ben] ben parents: 771 diff changeset	44
4083 a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan] aidan parents: 4072 diff changeset	45 (eval-when-compile (when (featurep 'mule) (require 'ccl)))
a3f8bb07ab38 [xemacs-hg @ 2007-07-28 08:02:15 by aidan] aidan parents: 4072 diff changeset	46
2367 ecf1ebac70d8 [xemacs-hg @ 2004-11-04 23:05:23 by ben] ben parents: 2297 diff changeset	47 ;; accessed in loadup.el, mule-cmds.el; see discussion in unicode.c
ecf1ebac70d8 [xemacs-hg @ 2004-11-04 23:05:23 by ben] ben parents: 2297 diff changeset	48 (defvar load-unicode-tables-at-dump-time (eq system-type 'windows-nt)
ecf1ebac70d8 [xemacs-hg @ 2004-11-04 23:05:23 by ben] ben parents: 2297 diff changeset	49 "[INTERNAL] Whether to load the Unicode tables at dump time.
ecf1ebac70d8 [xemacs-hg @ 2004-11-04 23:05:23 by ben] ben parents: 2297 diff changeset	50 Setting this at run-time does nothing.")
ecf1ebac70d8 [xemacs-hg @ 2004-11-04 23:05:23 by ben] ben parents: 2297 diff changeset	51
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	52 ;; NOTE: This takes only a fraction of a second on my Pentium III
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	53 ;; 700Mhz even with a totally optimization-disabled XEmacs.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	54 (defun load-unicode-tables ()
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	55 "Initialize the Unicode translation tables for all standard charsets."
780 578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	56 (let ((parse-args
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	57 '(("unicode/unicode-consortium"
877 e54d47b2d736 [xemacs-hg @ 2002-06-23 09:54:35 by stephent] stephent parents: 780 diff changeset	58 ;; Due to the braindamaged way Mule treats the ASCII and Control-1
e54d47b2d736 [xemacs-hg @ 2002-06-23 09:54:35 by stephent] stephent parents: 780 diff changeset	59 ;; charsets' types, trying to load them results in out-of-range
e54d47b2d736 [xemacs-hg @ 2002-06-23 09:54:35 by stephent] stephent parents: 780 diff changeset	60 ;; warnings at unicode.c:1439. They're no-ops anyway, they're
e54d47b2d736 [xemacs-hg @ 2002-06-23 09:54:35 by stephent] stephent parents: 780 diff changeset	61 ;; hardwired in unicode.c (unicode_to_ichar, ichar_to_unicode).
e54d47b2d736 [xemacs-hg @ 2002-06-23 09:54:35 by stephent] stephent parents: 780 diff changeset	62 ;; ("8859-1.TXT" ascii #x00 #x7F #x0)
e54d47b2d736 [xemacs-hg @ 2002-06-23 09:54:35 by stephent] stephent parents: 780 diff changeset	63 ;; ("8859-1.TXT" control-1 #x80 #x9F #x-80)
e54d47b2d736 [xemacs-hg @ 2002-06-23 09:54:35 by stephent] stephent parents: 780 diff changeset	64 ;; The 8859-1.TXT G1 assignments are half no-ops, hardwired in
e54d47b2d736 [xemacs-hg @ 2002-06-23 09:54:35 by stephent] stephent parents: 780 diff changeset	65 ;; unicode.c ichar_to_unicode, but not in unicode_to_ichar.
780 578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	66 ("8859-1.TXT" latin-iso8859-1 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	67 ;; "8859-10.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	68 ;; "8859-13.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	69 ("8859-14.TXT" latin-iso8859-14 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	70 ("8859-15.TXT" latin-iso8859-15 #xA0 #xFF #x-80)
2575 e71117a6ddac [xemacs-hg @ 2005-02-09 15:29:07 by aidan] aidan parents: 2574 diff changeset	71 ("8859-16.TXT" latin-iso8859-16 #xA0 #xFF #x-80)
780 578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	72 ("8859-2.TXT" latin-iso8859-2 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	73 ("8859-3.TXT" latin-iso8859-3 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	74 ("8859-4.TXT" latin-iso8859-4 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	75 ("8859-5.TXT" cyrillic-iso8859-5 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	76 ("8859-7.TXT" greek-iso8859-7 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	77 ("8859-8.TXT" hebrew-iso8859-8 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	78 ("8859-9.TXT" latin-iso8859-9 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	79 ;; charset for Big5 does not matter; specifying `big5' will
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	80 ;; automatically make the right thing happen
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	81 ("BIG5.TXT" chinese-big5-1 nil nil nil big5)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	82 ("CNS11643.TXT" chinese-cns11643-1 #x10000 #x1FFFF #x-10000)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	83 ("CNS11643.TXT" chinese-cns11643-2 #x20000 #x2FFFF #x-20000)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	84 ;; "CP1250.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	85 ;; "CP1251.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	86 ;; "CP1252.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	87 ;; "CP1253.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	88 ;; "CP1254.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	89 ;; "CP1255.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	90 ;; "CP1256.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	91 ;; "CP1257.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	92 ;; "CP1258.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	93 ;; "CP874.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	94 ;; "CP932.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	95 ;; "CP936.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	96 ;; "CP949.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	97 ;; "CP950.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	98 ;; "GB12345.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	99 ("GB2312.TXT" chinese-gb2312)
2297 13a418960a88 [xemacs-hg @ 2004-09-22 02:05:42 by stephent] stephent parents: 1318 diff changeset	100 ;; "HANGUL.TXT"
13a418960a88 [xemacs-hg @ 2004-09-22 02:05:42 by stephent] stephent parents: 1318 diff changeset	101 ;; #### shouldn't JIS X 0201's upper limit be 7f?
780 578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	102 ("JIS0201.TXT" latin-jisx0201 #x21 #x80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	103 ("JIS0201.TXT" katakana-jisx0201 #xA0 #xFF #x-80)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	104 ("JIS0208.TXT" japanese-jisx0208 nil nil nil ignore-first-column)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	105 ("JIS0212.TXT" japanese-jisx0212)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	106 ;; "JOHAB.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	107 ;; "KOI8-R.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	108 ;; "KSC5601.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	109 ;; note that KSC5601.TXT as currently distributed is NOT what
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	110 ;; it claims to be! see comments in KSX1001.TXT.
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	111 ("KSX1001.TXT" korean-ksc5601)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	112 ;; "OLD5601.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	113 ;; "SHIFTJIS.TXT"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	114 )
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	115 ("unicode/mule-ucs"
2297 13a418960a88 [xemacs-hg @ 2004-09-22 02:05:42 by stephent] stephent parents: 1318 diff changeset	116 ;; #### we don't support surrogates?!??
780 578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	117 ;; use these instead of the above ones once we support surrogates
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	118 ;;("chinese-cns11643-1.txt" chinese-cns11643-1)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	119 ;;("chinese-cns11643-2.txt" chinese-cns11643-2)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	120 ;;("chinese-cns11643-3.txt" chinese-cns11643-3)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	121 ;;("chinese-cns11643-4.txt" chinese-cns11643-4)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	122 ;;("chinese-cns11643-5.txt" chinese-cns11643-5)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	123 ;;("chinese-cns11643-6.txt" chinese-cns11643-6)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	124 ;;("chinese-cns11643-7.txt" chinese-cns11643-7)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	125 ("chinese-sisheng.txt" chinese-sisheng)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	126 ("ethiopic.txt" ethiopic)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	127 ("indian-is13194.txt" indian-is13194)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	128 ("ipa.txt" ipa)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	129 ("thai-tis620.txt" thai-tis620)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	130 ("tibetan.txt" tibetan)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	131 ("vietnamese-viscii-lower.txt" vietnamese-viscii-lower)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	132 ("vietnamese-viscii-upper.txt" vietnamese-viscii-upper)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	133 )
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	134 ("unicode/other"
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	135 ("lao.txt" lao)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	136 )
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	137 )))
780 578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	138 (mapcar #'(lambda (tables)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	139 (let ((undir
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	140 (expand-file-name (car tables) data-directory)))
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	141 (mapcar #'(lambda (args)
1318 b531bf8658e9 [xemacs-hg @ 2003-02-21 06:56:46 by ben] ben parents: 985 diff changeset	142 (apply 'load-unicode-mapping-table
780 578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	143 (expand-file-name (car args) undir)
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	144 (cdr args)))
578cb2932d72 [xemacs-hg @ 2002-03-18 10:07:30 by ben] ben parents: 778 diff changeset	145 (cdr tables))))
4145 edb00a8b4eff [xemacs-hg @ 2007-08-26 20:00:29 by aidan] aidan parents: 4096 diff changeset	146 parse-args)
edb00a8b4eff [xemacs-hg @ 2007-08-26 20:00:29 by aidan] aidan parents: 4096 diff changeset	147 ;; The default-unicode-precedence-list. We set this here to default to
edb00a8b4eff [xemacs-hg @ 2007-08-26 20:00:29 by aidan] aidan parents: 4096 diff changeset	148 ;; not mapping various European characters to East Asian characters;
edb00a8b4eff [xemacs-hg @ 2007-08-26 20:00:29 by aidan] aidan parents: 4096 diff changeset	149 ;; otherwise the default-unicode-precedence-list is numerically ordered
edb00a8b4eff [xemacs-hg @ 2007-08-26 20:00:29 by aidan] aidan parents: 4096 diff changeset	150 ;; by charset ID.
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	151 (declare-fboundp
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	152 (set-default-unicode-precedence-list
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	153 '(ascii control-1 latin-iso8859-1 latin-iso8859-2 latin-iso8859-15
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	154 greek-iso8859-7 hebrew-iso8859-8 ipa cyrillic-iso8859-5
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	155 latin-iso8859-16 latin-iso8859-3 latin-iso8859-4 latin-iso8859-9
4491 d402d7b18bd8 Revamp the Arabic support. Create greek-iso-8bit-with-esc, arabic-iso-8bit-with-esc. Aidan Kehoe <kehoea@parhasard.net> parents: 4468 diff changeset	156 vietnamese-viscii-lower vietnamese-viscii-upper
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	157 jit-ucs-charset-0 japanese-jisx0208 japanese-jisx0208-1978
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	158 japanese-jisx0212 japanese-jisx0213-1 japanese-jisx0213-2
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	159 chinese-gb2312 chinese-sisheng chinese-big5-1 chinese-big5-2
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	160 indian-is13194 korean-ksc5601 chinese-cns11643-1 chinese-cns11643-2
4491 d402d7b18bd8 Revamp the Arabic support. Create greek-iso-8bit-with-esc, arabic-iso-8bit-with-esc. Aidan Kehoe <kehoea@parhasard.net> parents: 4468 diff changeset	161 chinese-isoir165
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	162 composite ethiopic indian-1-column indian-2-column jit-ucs-charset-0
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	163 katakana-jisx0201 lao thai-tis620 thai-xtis tibetan tibetan-1-column
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	164 latin-jisx0201 chinese-cns11643-3 chinese-cns11643-4
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	165 chinese-cns11643-5 chinese-cns11643-6 chinese-cns11643-7)))))
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	166
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	167 (make-coding-system
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	168 'utf-16 'unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	169 "UTF-16"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	170 '(mnemonic "UTF-16"
3767 6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan] aidan parents: 3667 diff changeset	171 documentation
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	172 "UTF-16 Unicode encoding -- the standard (almost-) fixed-width
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	173 two-byte encoding, with surrogates. It will be fixed-width if all
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	174 characters are in the BMP (Basic Multilingual Plane -- first 65536
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	175 codepoints). Cannot represent characters with codepoints above
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	176 0x10FFFF (a little more than 1,000,000). Unicode and ISO guarantee
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	177 never to encode any characters outside this range -- all the rest are
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	178 for private, corporate or internal use."
3767 6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan] aidan parents: 3667 diff changeset	179 unicode-type utf-16))
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	180
2574 5e2653bc0ab0 [xemacs-hg @ 2005-02-08 23:59:50 by aidan] aidan parents: 2367 diff changeset	181 (define-coding-system-alias 'utf-16-be 'utf-16)
5e2653bc0ab0 [xemacs-hg @ 2005-02-08 23:59:50 by aidan] aidan parents: 2367 diff changeset	182
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	183 (make-coding-system
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	184 'utf-16-bom 'unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	185 "UTF-16 w/BOM"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	186 '(mnemonic "UTF16-BOM"
3767 6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan] aidan parents: 3667 diff changeset	187 documentation
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	188 "UTF-16 Unicode encoding with byte order mark (BOM) at the beginning.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	189 The BOM is Unicode character U+FEFF -- i.e. the first two bytes are
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	190 0xFE and 0xFF, respectively, or reversed in a little-endian
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	191 representation. It has been sanctioned by the Unicode Consortium for
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	192 use at the beginning of a Unicode stream as a marker of the byte order
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	193 of the stream, and commonly appears in Unicode files under Microsoft
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	194 Windows, where it also functions as a magic cookie identifying a
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	195 Unicode file. The character is called \"ZERO WIDTH NO-BREAK SPACE\"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	196 and is suitable as a byte-order marker because:
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	197
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	198 -- it has no displayable representation
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	199 -- due to its semantics it never normally appears at the beginning
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	200 of a stream
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	201 -- its reverse U+FFFE is not a legal Unicode character
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	202 -- neither byte sequence is at all likely in any other standard
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	203 encoding, particularly at the beginning of a stream
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	204
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	205 This coding system will insert a BOM at the beginning of a stream when
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	206 writing and strip it off when reading."
3767 6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan] aidan parents: 3667 diff changeset	207 unicode-type utf-16
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	208 need-bom t))
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	209
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	210 (make-coding-system
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	211 'utf-16-little-endian 'unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	212 "UTF-16 Little Endian"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	213 '(mnemonic "UTF16-LE"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	214 documentation
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	215 "Little-endian version of UTF-16 Unicode encoding.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	216 See `utf-16' coding system."
3767 6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan] aidan parents: 3667 diff changeset	217 unicode-type utf-16
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	218 little-endian t))
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	219
2574 5e2653bc0ab0 [xemacs-hg @ 2005-02-08 23:59:50 by aidan] aidan parents: 2367 diff changeset	220 (define-coding-system-alias 'utf-16-le 'utf-16-little-endian)
5e2653bc0ab0 [xemacs-hg @ 2005-02-08 23:59:50 by aidan] aidan parents: 2367 diff changeset	221
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	222 (make-coding-system
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	223 'utf-16-little-endian-bom 'unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	224 "UTF-16 Little Endian w/BOM"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	225 '(mnemonic "MSW-Unicode"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	226 documentation
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	227 "Little-endian version of UTF-16 Unicode encoding, with byte order mark.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	228 Standard encoding for representing Unicode under MS Windows. See
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	229 `utf-16-bom' coding system."
3767 6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan] aidan parents: 3667 diff changeset	230 unicode-type utf-16
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	231 little-endian t
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	232 need-bom t))
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	233
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	234 (make-coding-system
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	235 'ucs-4 'unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	236 "UCS-4"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	237 '(mnemonic "UCS4"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	238 documentation
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	239 "UCS-4 Unicode encoding -- fully fixed-width four-byte encoding."
3767 6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan] aidan parents: 3667 diff changeset	240 unicode-type ucs-4))
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	241
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	242 (make-coding-system
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	243 'ucs-4-little-endian 'unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	244 "UCS-4 Little Endian"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	245 '(mnemonic "UCS4-LE"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	246 documentation
2297 13a418960a88 [xemacs-hg @ 2004-09-22 02:05:42 by stephent] stephent parents: 1318 diff changeset	247 ;; #### I don't think this is permitted by ISO 10646, only Unicode.
13a418960a88 [xemacs-hg @ 2004-09-22 02:05:42 by stephent] stephent parents: 1318 diff changeset	248 ;; Call it UTF-32 instead?
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	249 "Little-endian version of UCS-4 Unicode encoding. See `ucs-4' coding system."
3767 6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan] aidan parents: 3667 diff changeset	250 unicode-type ucs-4
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	251 little-endian t))
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	252
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	253 (make-coding-system
4096 1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	254 'utf-32 'unicode
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	255 "UTF-32"
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	256 '(mnemonic "UTF32"
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	257 documentation
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	258 "UTF-32 Unicode encoding -- fixed-width four-byte encoding,
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	259 characters less than #x10FFFF are not supported. "
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	260 unicode-type utf-32))
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	261
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	262 (make-coding-system
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	263 'utf-32-little-endian 'unicode
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	264 "UTF-32 Little Endian"
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	265 '(mnemonic "UTF32-LE"
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	266 documentation
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	267 "Little-endian version of UTF-32 Unicode encoding.
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	268
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	269 A fixed-width four-byte encoding, characters less than #x10FFFF are not
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	270 supported. "
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	271 unicode-type ucs-4 little-endian t))
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	272
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	273 (make-coding-system
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	274 'utf-8 'unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	275 "UTF-8"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	276 '(mnemonic "UTF8"
3767 6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan] aidan parents: 3667 diff changeset	277 documentation "
6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan] aidan parents: 3667 diff changeset	278 UTF-8 Unicode encoding -- ASCII-compatible 8-bit variable-width encoding
2297 13a418960a88 [xemacs-hg @ 2004-09-22 02:05:42 by stephent] stephent parents: 1318 diff changeset	279 sharing the following principles with the Mule-internal encoding:
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	280
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	281 -- All ASCII characters (codepoints 0 through 127) are represented
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	282 by themselves (i.e. using one byte, with the same value as the
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	283 ASCII codepoint), and these bytes are disjoint from bytes
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	284 representing non-ASCII characters.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	285
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	286 This means that any 8-bit clean application can safely process
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	287 UTF-8-encoded text as it were ASCII, with no corruption (e.g. a
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	288 '/' byte is always a slash character, never the second byte of
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	289 some other character, as with Big5, so a pathname encoded in
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	290 UTF-8 can safely be split up into components and reassembled
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	291 again using standard ASCII processes).
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	292
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	293 -- Leading bytes and non-leading bytes in the encoding of a
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	294 character are disjoint, so moving backwards is easy.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	295
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	296 -- Given only the leading byte, you know how many following bytes
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	297 are present.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	298 "
3767 6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan] aidan parents: 3667 diff changeset	299 unicode-type utf-8))
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	300
985 7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs] youngs parents: 877 diff changeset	301 (make-coding-system
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs] youngs parents: 877 diff changeset	302 'utf-8-bom 'unicode
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs] youngs parents: 877 diff changeset	303 "UTF-8 w/BOM"
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs] youngs parents: 877 diff changeset	304 '(mnemonic "MSW-UTF8"
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs] youngs parents: 877 diff changeset	305 documentation
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs] youngs parents: 877 diff changeset	306 "UTF-8 Unicode encoding, with byte order mark.
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs] youngs parents: 877 diff changeset	307 Standard encoding for representing UTF-8 under MS Windows."
3767 6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan] aidan parents: 3667 diff changeset	308 unicode-type utf-8
985 7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs] youngs parents: 877 diff changeset	309 little-endian t
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs] youngs parents: 877 diff changeset	310 need-bom t))
7f62a956b825 [xemacs-hg @ 2002-09-01 06:41:40 by youngs] youngs parents: 877 diff changeset	311
2633 b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan] aidan parents: 2575 diff changeset	312 (defun decode-char (quote-ucs code &optional restriction)
3659 98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan] aidan parents: 3506 diff changeset	313 "FSF compatibility--return Mule character with Unicode codepoint CODE.
2633 b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan] aidan parents: 2575 diff changeset	314 The second argument must be 'ucs, the third argument is ignored. "
4096 1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	315 ;; We're prepared to accept invalid Unicode in unicode-to-char, but not in
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	316 ;; this function, which is the API that should actually be used, since
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	317 ;; it's available in GNU and in Mule-UCS.
1abf84db2c7f [xemacs-hg @ 2007-08-04 20:00:10 by aidan] aidan parents: 4083 diff changeset	318 (check-argument-range code #x0 #x10FFFF)
3506 bcc2611d4cfc [xemacs-hg @ 2006-07-13 20:45:48 by aidan] aidan parents: 3439 diff changeset	319 (assert (eq quote-ucs 'ucs) t
2633 b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan] aidan parents: 2575 diff changeset	320 "Sorry, decode-char doesn't yet support anything but the UCS. ")
b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan] aidan parents: 2575 diff changeset	321 (unicode-to-char code))
b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan] aidan parents: 2575 diff changeset	322
b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan] aidan parents: 2575 diff changeset	323 (defun encode-char (char quote-ucs &optional restriction)
3659 98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan] aidan parents: 3506 diff changeset	324 "FSF compatibility--return the Unicode code point of CHAR.
2633 b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan] aidan parents: 2575 diff changeset	325 The second argument must be 'ucs, the third argument is ignored. "
3506 bcc2611d4cfc [xemacs-hg @ 2006-07-13 20:45:48 by aidan] aidan parents: 3439 diff changeset	326 (assert (eq quote-ucs 'ucs) t
2633 b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan] aidan parents: 2575 diff changeset	327 "Sorry, encode-char doesn't yet support anything but the UCS. ")
b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan] aidan parents: 2575 diff changeset	328 (char-to-unicode char))
b921e3d0ac3e [xemacs-hg @ 2005-03-04 21:59:42 by aidan] aidan parents: 2575 diff changeset	329
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	330 (defconst ccl-encode-to-ucs-2
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	331 (eval-when-compile
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	332 (let ((pre-existing
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	333 ;; This is the compiled CCL program from the assert
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	334 ;; below. Since this file is dumped and ccl.el isn't (and
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	335 ;; even when it was, it was dumped much later than this
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	336 ;; one), we can't compile the program at dump time. We can
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	337 ;; check at byte compile time that the program is as
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	338 ;; expected, though.
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	339 [1 16 131127 7 98872 65823 1307 5 -65536 65313 64833 1028
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	340 147513 8 82009 255 22]))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	341 (when (featurep 'mule)
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	342 ;; Check that the pre-existing constant reflects the intended
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	343 ;; CCL program.
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	344 (assert
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	345 (equal pre-existing
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	346 (ccl-compile
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	347 `(1
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	348 ( ;; mule-to-unicode's first argument is the
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	349 ;; charset ID, the second its first byte
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	350 ;; left shifted by 7 bits masked with its
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	351 ;; second byte.
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	352 (r1 = (r1 << 7))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	353 (r1 = (r1 \| r2))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	354 (mule-to-unicode r0 r1)
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	355 (if (r0 & ,(lognot #xFFFF))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	356 ;; Redisplay looks in r1 and r2 for the first
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	357 ;; and second bytes of the X11 font,
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	358 ;; respectively. For non-BMP characters we
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	359 ;; display U+FFFD.
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	360 ((r1 = #xFF)
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	361 (r2 = #xFD))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	362 ((r1 = (r0 >> 8))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	363 (r2 = (r0 & #xFF))))))))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	364 nil
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	365 "The pre-compiled CCL program appears broken. "))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	366 pre-existing))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	367 "CCL program to transform Mule characters to UCS-2.")
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	368
3667 4c8ad140bcec [xemacs-hg @ 2006-11-07 18:51:21 by aidan] aidan parents: 3666 diff changeset	369 (when (featurep 'mule)
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	370 (put 'ccl-encode-to-ucs-2 'ccl-program-idx
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	371 (declare-fboundp
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	372 (register-ccl-program 'ccl-encode-to-ucs-2 ccl-encode-to-ucs-2))))
4145 edb00a8b4eff [xemacs-hg @ 2007-08-26 20:00:29 by aidan] aidan parents: 4096 diff changeset	373
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	374 ;; Now, create jit-ucs-charset-0 entries for those characters in Windows
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	375 ;; Glyph List 4 that would otherwise end up in East Asian character sets.
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	376 ;;
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	377 ;; WGL4 is a character repertoire from Microsoft that gives a guideline
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	378 ;; for font implementors as to what characters are sufficient for
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	379 ;; pan-European support. The intention of this code is to avoid the
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	380 ;; situation where these characters end up mapping to East Asian XEmacs
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	381 ;; characters, which generally clash strongly with European characters
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	382 ;; both in font choice and character width; jit-ucs-charset-0 is a
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	383 ;; single-width character set which comes before the East Asian character
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	384 ;; sets in the default-unicode-precedence-list above.
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	385 (loop for (ucs ascii-or-latin-1)
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	386 in '((#x2013 ?-) ;; U+2013 EN DASH
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	387 (#x2014 ?-) ;; U+2014 EM DASH
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	388 (#x2105 ?%) ;; U+2105 CARE OF
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	389 (#x203e ?-) ;; U+203E OVERLINE
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	390 (#x221f ?\|) ;; U+221F RIGHT ANGLE
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	391 (#x2584 ?\|) ;; U+2584 LOWER HALF BLOCK
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	392 (#x2588 ?\|) ;; U+2588 FULL BLOCK
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	393 (#x258c ?\|) ;; U+258C LEFT HALF BLOCK
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	394 (#x2550 ?\|) ;; U+2550 BOX DRAWINGS DOUBLE HORIZONTAL
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	395 (#x255e ?\|) ;; U+255E BOX DRAWINGS VERTICAL SINGLE AND RIGHT DOUBLE
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	396 (#x256a ?\|) ;; U+256A BOX DRAWINGS VERTICAL SINGLE & HORIZONTAL DOUBLE
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	397 (#x2561 ?\|) ;; U+2561 BOX DRAWINGS VERTICAL SINGLE AND LEFT DOUBLE
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	398 (#x2215 ?/) ;; U+2215 DIVISION SLASH
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	399 (#x02c9 ?`) ;; U+02C9 MODIFIER LETTER MACRON
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	400 (#x2211 ?s) ;; U+2211 N-ARY SUMMATION
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	401 (#x220f ?s) ;; U+220F N-ARY PRODUCT
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	402 (#x2248 ?=) ;; U+2248 ALMOST EQUAL TO
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	403 (#x2264 ?=) ;; U+2264 LESS-THAN OR EQUAL TO
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	404 (#x2265 ?=) ;; U+2265 GREATER-THAN OR EQUAL TO
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	405 (#x201c ?') ;; U+201C LEFT DOUBLE QUOTATION MARK
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	406 (#x2026 ?.) ;; U+2026 HORIZONTAL ELLIPSIS
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	407 (#x2212 ?-) ;; U+2212 MINUS SIGN
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	408 (#x2260 ?=) ;; U+2260 NOT EQUAL TO
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	409 (#x221e ?=) ;; U+221E INFINITY
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	410 (#x2642 ?=) ;; U+2642 MALE SIGN
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	411 (#x2640 ?=) ;; U+2640 FEMALE SIGN
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	412 (#x2032 ?=) ;; U+2032 PRIME
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	413 (#x2033 ?=) ;; U+2033 DOUBLE PRIME
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	414 (#x25cb ?=) ;; U+25CB WHITE CIRCLE
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	415 (#x25cf ?=) ;; U+25CF BLACK CIRCLE
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	416 (#x25a1 ?=) ;; U+25A1 WHITE SQUARE
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	417 (#x25a0 ?=) ;; U+25A0 BLACK SQUARE
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	418 (#x25b2 ?=) ;; U+25B2 BLACK UP-POINTING TRIANGLE
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	419 (#x25bc ?=) ;; U+25BC BLACK DOWN-POINTING TRIANGLE
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	420 (#x2192 ?=) ;; U+2192 RIGHTWARDS ARROW
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	421 (#x2190 ?=) ;; U+2190 LEFTWARDS ARROW
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	422 (#x2191 ?=) ;; U+2191 UPWARDS ARROW
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	423 (#x2193 ?=) ;; U+2193 DOWNWARDS ARROW
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	424 (#x2229 ?=) ;; U+2229 INTERSECTION
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	425 (#x2202 ?=) ;; U+2202 PARTIAL DIFFERENTIAL
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	426 (#x2261 ?=) ;; U+2261 IDENTICAL TO
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	427 (#x221a ?=) ;; U+221A SQUARE ROOT
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	428 (#x222b ?=) ;; U+222B INTEGRAL
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	429 (#x2030 ?=) ;; U+2030 PER MILLE SIGN
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	430 (#x266a ?=) ;; U+266A EIGHTH NOTE
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	431 (#x2020 ?*) ;; U+2020 DAGGER
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	432 (#x2021 ?*) ;; U+2021 DOUBLE DAGGER
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	433 (#x2500 ?\|) ;; U+2500 BOX DRAWINGS LIGHT HORIZONTAL
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	434 (#x2502 ?\|) ;; U+2502 BOX DRAWINGS LIGHT VERTICAL
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	435 (#x250c ?\|) ;; U+250C BOX DRAWINGS LIGHT DOWN AND RIGHT
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	436 (#x2510 ?\|) ;; U+2510 BOX DRAWINGS LIGHT DOWN AND LEFT
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	437 (#x2518 ?\|) ;; U+2518 BOX DRAWINGS LIGHT UP AND LEFT
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	438 (#x2514 ?\|) ;; U+2514 BOX DRAWINGS LIGHT UP AND RIGHT
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	439 (#x251c ?\|) ;; U+251C BOX DRAWINGS LIGHT VERTICAL AND RIGHT
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	440 (#x252c ?\|) ;; U+252C BOX DRAWINGS LIGHT DOWN AND HORIZONTAL
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	441 (#x2524 ?\|) ;; U+2524 BOX DRAWINGS LIGHT VERTICAL AND LEFT
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	442 (#x2534 ?\|) ;; U+2534 BOX DRAWINGS LIGHT UP AND HORIZONTAL
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	443 (#x253c ?\|) ;; U+253C BOX DRAWINGS LIGHT VERTICAL AND HORIZONTAL
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	444 (#x02da ?^) ;; U+02DA RING ABOVE
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	445 (#x2122 ?\xa9) ;; U+2122 TRADE MARK SIGN, ?,A)(B
4145 edb00a8b4eff [xemacs-hg @ 2007-08-26 20:00:29 by aidan] aidan parents: 4096 diff changeset	446
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	447 (#x0132 ?\xe6) ;; U+0132 LATIN CAPITAL LIGATURE IJ, ?,Af(B
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	448 (#x013f ?\xe6) ;; U+013F LATIN CAPITAL LETTER L WITH MIDDLE DOT, ?,Af(B
4145 edb00a8b4eff [xemacs-hg @ 2007-08-26 20:00:29 by aidan] aidan parents: 4096 diff changeset	449
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	450 (#x0133 ?\xe6) ;; U+0133 LATIN SMALL LIGATURE IJ, ?,Af(B
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	451 (#x0140 ?\xe6) ;; U+0140 LATIN SMALL LETTER L WITH MIDDLE DOT, ?,Af(B
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	452 (#x0149 ?\xe6) ;; U+0149 LATIN SMALL LETTER N PRECEDED BY APOSTROPH,?,Af(B
4145 edb00a8b4eff [xemacs-hg @ 2007-08-26 20:00:29 by aidan] aidan parents: 4096 diff changeset	453
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	454 (#x2194 ?\|) ;; U+2194 LEFT RIGHT ARROW
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	455 (#x2660 ?*) ;; U+2660 BLACK SPADE SUIT
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	456 (#x2665 ?*) ;; U+2665 BLACK HEART SUIT
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	457 (#x2663 ?*) ;; U+2663 BLACK CLUB SUIT
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	458 (#x2592 ?\|) ;; U+2592 MEDIUM SHADE
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	459 (#x2195 ?\|) ;; U+2195 UP DOWN ARROW
4145 edb00a8b4eff [xemacs-hg @ 2007-08-26 20:00:29 by aidan] aidan parents: 4096 diff changeset	460
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	461 (#x2113 ?\xb9) ;; U+2113 SCRIPT SMALL L, ?,A9(B
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	462 (#x215b ?\xbe) ;; U+215B VULGAR FRACTION ONE EIGHTH, ?,A>(B
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	463 (#x215c ?\xbe) ;; U+215C VULGAR FRACTION THREE EIGHTHS, ?,A>(B
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	464 (#x215d ?\xbe) ;; U+215D VULGAR FRACTION FIVE EIGHTHS, ?,A>(B
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	465 (#x215e ?\xbe) ;; U+215E VULGAR FRACTION SEVEN EIGHTHS, ?,A>(B
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	466 (#x207f ?\xbe) ;; U+207F SUPERSCRIPT LATIN SMALL LETTER N, ?,A>(B
4145 edb00a8b4eff [xemacs-hg @ 2007-08-26 20:00:29 by aidan] aidan parents: 4096 diff changeset	467
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	468 ;; These are not in WGL 4, but are IPA characters that should not
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	469 ;; be double width. They are the only IPA characters that both
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	470 ;; occur in packages/mule-packages/leim/ipa.el and end up in East
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	471 ;; Asian character sets when that file is loaded in an XEmacs
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	472 ;; without packages.
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	473 (#x2197 ?\|) ;; U+2197 NORTH EAST ARROW
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	474 (#x2199 ?\|) ;; U+2199 SOUTH WEST ARROW
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	475 (#x2191 ?\|) ;; U+2191 UPWARDS ARROW
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	476 (#x207f ?\xb9)) ;; U+207F SUPERSCRIPT LATIN SMALL LETTER N, ?,A9(B
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	477 with decoded = nil
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	478 with syntax-table = (standard-syntax-table)
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	479 initially (unless (featurep 'mule) (return))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	480 ;; This creates jit-ucs-charset-0 entries because:
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	481 ;;
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	482 ;; 1. If the tables are dumped, it is run at dump time before they are
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	483 ;; dumped, and as such before the relevant conversions are available
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	484 ;; (they are made available in mule/general-late.el).
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	485 ;;
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	486 ;; 2. If the tables are not dumped, it is run at dump time, long before
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	487 ;; any of the other mappings are available.
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	488 ;;
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	489 do
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	490 (setq decoded (decode-char 'ucs ucs))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	491 (assert (eq (declare-fboundp (char-charset decoded))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	492 'jit-ucs-charset-0) nil
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	493 "Unexpected Unicode decoding behavior. ")
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	494 (modify-syntax-entry decoded
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	495 (string
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	496 (char-syntax ascii-or-latin-1))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	497 syntax-table))
4145 edb00a8b4eff [xemacs-hg @ 2007-08-26 20:00:29 by aidan] aidan parents: 4096 diff changeset	498
4268 75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	499 ;; Sigh, declarations needs to be at the start of the line to be picked up
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	500 ;; by make-docfile. Not so much an issue with ccl-encode-to-ucs-2, which we
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	501 ;; don't necessarily want to advertise, but the following are important.
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	502
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	503 ;; Create all the Unicode error sequences, normally as jit-ucs-charset-0
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	504 ;; characters starting at U+200000 (which isn't a valid Unicode code
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	505 ;; point). Make them available to user code.
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	506 (defvar unicode-error-default-translation-table
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	507 (loop
4468 a78d697ccd2c Import and extend GNU's descr-text.el, supporting prefix argument for C-x = Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	508 with char-table = (make-char-table 'generic)
4268 75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	509 for i from ?\x00 to ?\xFF
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	510 initially (unless (featurep 'mule) (return))
4268 75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	511 do
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	512 (put-char-table (aref
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	513 ;; #xd800 is the first leading surrogate;
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	514 ;; trailing surrogates must be in the range
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	515 ;; #xdc00-#xdfff. These examples are not, so we
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	516 ;; intentionally provoke an error sequence.
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	517 (decode-coding-string (format "\xd8\x00\x00%c" i)
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	518 'utf-16-be)
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	519 3)
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	520 i
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	521 char-table)
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	522 finally return char-table)
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	523 "Translation table mapping Unicode error sequences to Latin-1 chars.
4145 edb00a8b4eff [xemacs-hg @ 2007-08-26 20:00:29 by aidan] aidan parents: 4096 diff changeset	524
4202 a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	525 To transform XEmacs Unicode error sequences to the Latin-1 characters that
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	526 correspond to the octets on disk, you can use this variable. ")
4145 edb00a8b4eff [xemacs-hg @ 2007-08-26 20:00:29 by aidan] aidan parents: 4096 diff changeset	527
4490 67fbcaf3dbdc error-sequence -> invalid-sequence Aidan Kehoe <kehoea@parhasard.net> parents: 4489 diff changeset	528 (defvar unicode-invalid-sequence-regexp-range
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	529 (and (featurep 'mule)
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	530 (format "%c%c-%c"
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	531 (aref (decode-coding-string "\xd8\x00\x00\x00" 'utf-16-be) 0)
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	532 (aref (decode-coding-string "\xd8\x00\x00\x00" 'utf-16-be) 3)
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	533 (aref (decode-coding-string "\xd8\x00\x00\xFF" 'utf-16-be) 3)))
4268 75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	534 "Regular expression range to match Unicode error sequences in XEmacs.
4145 edb00a8b4eff [xemacs-hg @ 2007-08-26 20:00:29 by aidan] aidan parents: 4096 diff changeset	535
4202 a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	536 Invalid Unicode sequences on input are represented as XEmacs
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	537 characters with values stored as the keys in
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	538 `unicode-error-default-translation-table', one character for each
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	539 invalid octet. You can use this variable (with `re-search-forward' or
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	540 `skip-chars-forward') to search for such characters; see also
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	541 `unicode-error-translate-region'. ")
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	542
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	543 ;; Check that the lookup table is correct, and that all the actual error
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	544 ;; sequences are caught by the regexp.
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	545 (with-temp-buffer
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	546 (loop
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	547 for i from ?\x00 to ?\xFF
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	548 with to-check = (make-string 20 ?\x20)
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	549 initially (unless (featurep 'mule) (return))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	550 do
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	551 (delete-region (point-min) (point-max))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	552 (insert to-check)
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	553 (goto-char 10)
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	554 (insert (decode-coding-string (format "\xd8\x00\x00%c" i)
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	555 'utf-16-be))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	556 (backward-char)
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	557 (assert (= i (get-char-table (char-after (point))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	558 unicode-error-default-translation-table))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	559 (format "Char ?\\x%x not the expected error sequence!"
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	560 i))
4202 a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	561
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	562 (goto-char (point-min))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	563 ;; Comment out until the issue in
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	564 ;; 18179.49815.622843.336527@parhasard.net is fixed.
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	565 (assert t ; (re-search-forward (concat "["
4490 67fbcaf3dbdc error-sequence -> invalid-sequence Aidan Kehoe <kehoea@parhasard.net> parents: 4489 diff changeset	566 ; unicode-invalid-sequence-regexp-range
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	567 ; "]"))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	568 nil
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	569 (format "Could not find char ?\\x%x in buffer" i))))
4202 a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	570
4268 75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	571 (defun frob-unicode-errors-region (frob-function begin end &optional buffer)
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	572 "Call FROB-FUNCTION on the Unicode error sequences between BEGIN and END.
4202 a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	573
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	574 Optional argument BUFFER specifies the buffer that should be examined for
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	575 such sequences. "
4268 75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	576 (check-argument-type #'functionp frob-function)
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	577 (check-argument-range begin (point-min buffer) (point-max buffer))
75d0292c1bff [xemacs-hg @ 2007-11-14 19:41:04 by aidan] aidan parents: 4222 diff changeset	578 (check-argument-range end (point-min buffer) (point-max buffer))
4202 a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	579 (save-excursion
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	580 (save-restriction
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	581 (if buffer (set-buffer buffer))
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	582 (narrow-to-region begin end)
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	583 (goto-char (point-min))
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	584 (while end
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	585 (setq begin
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	586 (progn
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	587 (skip-chars-forward
4490 67fbcaf3dbdc error-sequence -> invalid-sequence Aidan Kehoe <kehoea@parhasard.net> parents: 4489 diff changeset	588 (concat "^" unicode-invalid-sequence-regexp-range))
4202 a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	589 (point))
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	590 end (and (not (= (point) (point-max)))
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	591 (progn
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	592 (skip-chars-forward
4490 67fbcaf3dbdc error-sequence -> invalid-sequence Aidan Kehoe <kehoea@parhasard.net> parents: 4489 diff changeset	593 unicode-invalid-sequence-regexp-range)
4202 a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	594 (point))))
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	595 (if end
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	596 (funcall frob-function begin end))))))
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	597
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	598 (defun unicode-error-translate-region (begin end &optional buffer table)
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	599 "Translate the Unicode error sequences in BUFFER between BEGIN and END.
4202 a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	600
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	601 The error sequences are transformed, by default, into the ASCII,
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	602 control-1 and latin-iso8859-1 characters with the numeric values
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	603 corresponding to the incorrect octets encountered. This is achieved
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	604 by using `unicode-error-default-translation-table' (which see) for
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	605 TABLE; you can change this by supplying another character table,
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	606 mapping from the error sequences to the desired characters. "
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	607 (unless table (setq table unicode-error-default-translation-table))
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	608 (frob-unicode-errors-region
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	609 (lambda (start finish)
a7c5de5b9880 [xemacs-hg @ 2007-10-02 10:33:04 by aidan] aidan parents: 4145 diff changeset	610 (translate-region start finish table))
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	611 begin end buffer))
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	612
4489 b75b075a9041 Support displaying invalid UTF-8 in language-environment-specific ways. Aidan Kehoe <kehoea@parhasard.net> parents: 4468 diff changeset	613 ;; Sure would be nice to be able to use defface here.
4490 67fbcaf3dbdc error-sequence -> invalid-sequence Aidan Kehoe <kehoea@parhasard.net> parents: 4489 diff changeset	614 (copy-face 'highlight 'unicode-invalid-sequence-warning-face)
4489 b75b075a9041 Support displaying invalid UTF-8 in language-environment-specific ways. Aidan Kehoe <kehoea@parhasard.net> parents: 4468 diff changeset	615
4549 68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	616 (defvar unicode-query-coding-skip-chars-arg nil ;; Set in general-late.el
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	617 "Used by `unicode-query-coding-region' to skip chars with known mappings.")
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	618
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	619 (defun unicode-query-coding-region (begin end coding-system
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	620 &optional buffer errorp highlightp)
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	621 "The `query-coding-region' implementation for Unicode coding systems."
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	622 (check-argument-type #'coding-system-p
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	623 (setq coding-system (find-coding-system coding-system)))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	624 (check-argument-type #'integer-or-marker-p begin)
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	625 (check-argument-type #'integer-or-marker-p end)
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	626 (let* ((skip-chars-arg unicode-query-coding-skip-chars-arg)
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	627 (ranges (make-range-table))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	628 (looking-at-arg (concat "[" skip-chars-arg "]"))
4568 1d74a1d115ee Add #'query-coding-region tests; do the work necessary to get them running. Aidan Kehoe <kehoea@parhasard.net> parents: 4566 diff changeset	629 fail-range-start fail-range-end char-after failed
4551 6812571bfcb9 Fix some bugs. Aidan Kehoe <kehoea@parhasard.net> parents: 4549 diff changeset	630 extent)
4549 68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	631 (save-excursion
4551 6812571bfcb9 Fix some bugs. Aidan Kehoe <kehoea@parhasard.net> parents: 4549 diff changeset	632 (when highlightp
6812571bfcb9 Fix some bugs. Aidan Kehoe <kehoea@parhasard.net> parents: 4549 diff changeset	633 (map-extents #'(lambda (extent ignored-arg)
6812571bfcb9 Fix some bugs. Aidan Kehoe <kehoea@parhasard.net> parents: 4549 diff changeset	634 (when (eq 'query-coding-warning-face
6812571bfcb9 Fix some bugs. Aidan Kehoe <kehoea@parhasard.net> parents: 4549 diff changeset	635 (extent-face extent))
6812571bfcb9 Fix some bugs. Aidan Kehoe <kehoea@parhasard.net> parents: 4549 diff changeset	636 (delete-extent extent))) buffer begin end))
4549 68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	637 (goto-char begin buffer)
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	638 (skip-chars-forward skip-chars-arg end buffer)
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	639 (while (< (point buffer) end)
4551 6812571bfcb9 Fix some bugs. Aidan Kehoe <kehoea@parhasard.net> parents: 4549 diff changeset	640 ; (message
4568 1d74a1d115ee Add #'query-coding-region tests; do the work necessary to get them running. Aidan Kehoe <kehoea@parhasard.net> parents: 4566 diff changeset	641 ; "fail-range-start is %S, point is %S, end is %S"
1d74a1d115ee Add #'query-coding-region tests; do the work necessary to get them running. Aidan Kehoe <kehoea@parhasard.net> parents: 4566 diff changeset	642 ; fail-range-start (point buffer) end)
4549 68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	643 (setq char-after (char-after (point buffer) buffer)
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	644 fail-range-start (point buffer))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	645 (while (and
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	646 (< (point buffer) end)
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	647 (not (looking-at looking-at-arg))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	648 (= -1 (char-to-unicode char-after)))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	649 (forward-char 1 buffer)
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	650 (setq char-after (char-after (point buffer) buffer)
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	651 failed t))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	652 (if (= fail-range-start (point buffer))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	653 ;; The character can actually be encoded by the coding
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	654 ;; system; check the characters past it.
4551 6812571bfcb9 Fix some bugs. Aidan Kehoe <kehoea@parhasard.net> parents: 4549 diff changeset	655 (forward-char 1 buffer)
4549 68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	656 ;; Can't be encoded; note this.
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	657 (when errorp
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	658 (error 'text-conversion-error
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	659 (format "Cannot encode %s using coding system"
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	660 (buffer-substring fail-range-start (point buffer)
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	661 buffer))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	662 (coding-system-name coding-system)))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	663 (put-range-table fail-range-start
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	664 ;; If char-after is non-nil, we're not at
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	665 ;; the end of the buffer.
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	666 (setq fail-range-end (if char-after
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	667 (point buffer)
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	668 (point-max buffer)))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	669 t ranges)
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	670 (when highlightp
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	671 (setq extent (make-extent fail-range-start fail-range-end buffer))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	672 (set-extent-priority extent (+ mouse-highlight-priority 2))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	673 (set-extent-face extent 'query-coding-warning-face)))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	674 (skip-chars-forward skip-chars-arg end buffer))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	675 (if failed
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	676 (values nil ranges)
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	677 (values t nil)))))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	678
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	679 (loop
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	680 for coding-system in (coding-system-list)
4570 e6a7054a9c30 Add check-coding-systems-region, test it and others, fix some bugs. Aidan Kehoe <kehoea@parhasard.net> parents: 4568 diff changeset	681 initially (unless (featurep 'mule) (return))
4549 68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	682 do (when (eq 'unicode (coding-system-type coding-system))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	683 (coding-system-put coding-system 'query-coding-function
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	684 #'unicode-query-coding-region)))
68d1ca56cffa First part of interactive checks that coding systems encode regions. Aidan Kehoe <kehoea@parhasard.net> parents: 4317 diff changeset	685
4317 15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	686 (unless (featurep 'mule)
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	687 ;; We do this in such a roundabout way--instead of having the above defun
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	688 ;; and defvar calls inside a (when (featurep 'mule) ...) form--to have
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	689 ;; make-docfile.c pick up symbol and function documentation correctly. An
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	690 ;; alternative approach would be to fix make-docfile.c to be able to read
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	691 ;; Lisp.
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	692 (mapcar #'unintern
15d36164ebd7 Eliminate lost docstring warnings on 21.5. Aidan Kehoe <kehoea@parhasard.net> parents: 4268 diff changeset	693 '(ccl-encode-to-ucs-2 unicode-error-default-translation-table
4490 67fbcaf3dbdc error-sequence -> invalid-sequence Aidan Kehoe <kehoea@parhasard.net> parents: 4489 diff changeset	694 unicode-invalid-regexp-range frob-unicode-errors-region
4570 e6a7054a9c30 Add check-coding-systems-region, test it and others, fix some bugs. Aidan Kehoe <kehoea@parhasard.net> parents: 4568 diff changeset	695 unicode-error-translate-region unicode-query-coding-region
e6a7054a9c30 Add check-coding-systems-region, test it and others, fix some bugs. Aidan Kehoe <kehoea@parhasard.net> parents: 4568 diff changeset	696 unicode-query-coding-skip-chars-arg)))
3667 4c8ad140bcec [xemacs-hg @ 2006-11-07 18:51:21 by aidan] aidan parents: 3666 diff changeset	697
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	698 ;; #### UTF-7 is not yet implemented, and it's tricky to do. There's
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	699 ;; an implementation in appendix A.1 of the Unicode Standard, Version
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	700 ;; 2.0, but I don't know its licensing characteristics.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	701
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	702 ; (make-coding-system
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	703 ; 'utf-7 'unicode
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	704 ; "UTF-7"
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	705 ; '(mnemonic "UTF7"
3659 98af8a976fc3 [xemacs-hg @ 2006-11-05 22:31:31 by aidan] aidan parents: 3506 diff changeset	706 ; documentation; "UTF-7 Unicode encoding -- 7-bit-ASCII modal Internet-mail-compatible
771 943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	707 ; encoding especially designed for headers, with the following
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	708 ; properties:
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	709
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	710 ; -- Only characters that are considered safe for passing through any mail
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	711 ; gateway without damage are used.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	712
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	713 ; -- This is a modal encoding, with two states. The first, default
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	714 ; state encodes the most common Unicode characters (upper and
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	715 ; lowercase letters, digits, and 9 common punctuation marks) as
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	716 ; themselves, and the second state, entered using '+' and
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	717 ; terminated with '-' or any character disallowed in state 2,
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	718 ; encodes any Unicode characters by first converting to UTF-16,
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	719 ; most significant byte first, and then to a slightly modified
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	720 ; Base64 encoding. (Thus, UTF-7 has the same limitations on the
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	721 ; characters it can encode as UTF-16.)
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	722
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	723 ; -- The modified Base64 encoding deviates from standard Base64 in
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	724 ; that it omits the `=' pad character. This is eliminated so as to
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	725 ; avoid conflicts with the use of `=' as an escape in the
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	726 ; Quoted-Printable encoding and the related Q encoding for headers:
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	727 ; With this modification, non-whitespace chars in UTF-7 will be
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	728 ; represented in Quoted-Printable and in Q as-is, with no further
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	729 ; encoding.
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	730
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	731 ; For more information, see Appendix A.1 of The Unicode Standard 2.0, or
943eaba38521 [xemacs-hg @ 2002-03-13 08:51:24 by ben] ben parents: diff changeset	732 ; wherever it is in v3.0."
3767 6b2ef948e140 [xemacs-hg @ 2006-12-29 18:09:38 by aidan] aidan parents: 3667 diff changeset	733 ; unicode-type utf-7))

Mercurial > hg > xemacs-beta

annotate lisp/unicode.el @ 4570:e6a7054a9c30