Move the #'query-coding-region implementation to C. This is necessary because there is no reasonable way to access the corresponding mswindows-multibyte functionality from Lisp, and we need such functionality if we're going to have a reliable and portable #'query-coding-region implementation. However, this change doesn't yet provide #'query-coding-region for the mswindow-multibyte coding systems, there should be no functional differences between an XEmacs with this change and one without it. src/ChangeLog addition: 2009-09-19 Aidan Kehoe <kehoea@parhasard.net> Move the #'query-coding-region implementation to C. This is necessary because there is no reasonable way to access the corresponding mswindows-multibyte functionality from Lisp, and we need such functionality if we're going to have a reliable and portable #'query-coding-region implementation. However, this change doesn't yet provide #'query-coding-region for the mswindow-multibyte coding systems, there should be no functional differences between an XEmacs with this change and one without it. * mule-coding.c (struct fixed_width_coding_system): Add a new coding system type, fixed_width, and implement it. It uses the CCL infrastructure but has a much simpler creation API, and its own query_method, formerly in lisp/mule/mule-coding.el. * unicode.c: Move the Unicode query method implementation here from unicode.el. * lisp.h: Declare Fmake_coding_system_internal, Fcopy_range_table here. * intl-win32.c (complex_vars_of_intl_win32): Use Fmake_coding_system_internal, not Fmake_coding_system. * general-slots.h: Add Qsucceeded, Qunencodable, Qinvalid_sequence here. * file-coding.h (enum coding_system_variant): Add fixed_width_coding_system here. (struct coding_system_methods): Add query_method and query_lstream_method to the coding system methods. Provide flags for the query methods. Declare the default query method; initialise it correctly in INITIALIZE_CODING_SYSTEM_TYPE. * file-coding.c (default_query_method): New function, the default query method for coding systems that do not set it. Moved from coding.el. (make_coding_system_1): Accept new elements in PROPS in #'make-coding-system; aliases, a list of aliases; safe-chars and safe-charsets (these were previously accepted but not saved); and category. (Fmake_coding_system_internal): New function, what used to be #'make-coding-system--on Mule builds, we've now moved some of the functionality of this to Lisp. (Fcoding_system_canonical_name_p): Move this earlier in the file, since it's now called from within make_coding_system_1. (Fquery_coding_region): Move the implementation of this here, from coding.el. (complex_vars_of_file_coding): Call Fmake_coding_system_internal, not Fmake_coding_system; specify safe-charsets properties when we're a mule build. * extents.h (mouse_highlight_priority, Fset_extent_priority, Fset_extent_face, Fmap_extents): Make these available to other C files. lisp/ChangeLog addition: 2009-09-19 Aidan Kehoe <kehoea@parhasard.net> Move the #'query-coding-region implementation to C. * coding.el: Consolidate code that depends on the presence or absence of Mule at the end of this file. (default-query-coding-region, query-coding-region): Move these functions to C. (default-query-coding-region-safe-charset-skip-chars-map): Remove this variable, the corresponding C variable is Vdefault_query_coding_region_chartab_cache in file-coding.c. (query-coding-string): Update docstring to reflect actual multiple values, be more careful about not modifying a range table that we're currently mapping over. (encode-coding-char): Make the implementation of this simpler. (featurep 'mule): Autoload #'make-coding-system from mule/make-coding-system.el if we're a mule build; provide an appropriate compiler macro. Do various non-mule compatibility things if we're not a mule build. * update-elc.el (additional-dump-dependencies): Add mule/make-coding-system as a dump time dependency if we're a mule build. * unicode.el (ccl-encode-to-ucs-2): (decode-char): (encode-char): Move these earlier in the file, for the sake of some byte compile warnings. (unicode-query-coding-region): Move this to unicode.c * mule/make-coding-system.el: New file, not dumped. Contains the functionality to rework the arguments necessary for fixed-width coding systems, and contains the implementation of #'make-coding-system, which now calls #'make-coding-system-internal. * mule/vietnamese.el (viscii): * mule/latin.el (iso-8859-2): (windows-1250): (iso-8859-3): (iso-8859-4): (iso-8859-14): (iso-8859-15): (iso-8859-16): (iso-8859-9): (macintosh): (windows-1252): * mule/hebrew.el (iso-8859-8): * mule/greek.el (iso-8859-7): (windows-1253): * mule/cyrillic.el (iso-8859-5): (koi8-r): (koi8-u): (windows-1251): (alternativnyj): (koi8-ru): (koi8-t): (koi8-c): (koi8-o): * mule/arabic.el (iso-8859-6): (windows-1256): Move all these coding systems to being of type fixed-width, not of type CCL. This allows the distinct query-coding-region for them to be in C, something which will eventually allow us to implement query-coding-region for the mswindows-multibyte coding systems. * mule/general-late.el (posix-charset-to-coding-system-hash): Document why we're pre-emptively persuading the byte compiler that the ELC for this file needs to be written using escape-quoted. Call #'set-unicode-query-skip-chars-args, now the Unicode query-coding-region implementation is in C. * mule/thai-xtis.el (tis-620): Don't bother checking whether we're XEmacs or not here. * mule/mule-coding.el: Move the eight bit fixed-width functionality from this file to make-coding-system.el. tests/ChangeLog addition: 2009-09-19 Aidan Kehoe <kehoea@parhasard.net> * automated/mule-tests.el: Check a coding system's type, not an 8-bit-fixed property, for whether that coding system should be treated as a fixed-width coding system. * automated/query-coding-tests.el: Don't test the query coding functionality for mswindows-multibyte coding systems, it's not yet implemented.

--- a/src/file-coding.h	Sat Sep 19 17:56:23 2009 +0200
+++ b/src/file-coding.h	Sat Sep 19 22:53:13 2009 +0100
@@ -230,7 +230,8 @@
   ccl_coding_system,
   shift_jis_coding_system,
   big5_coding_system,
-  unicode_coding_system
+  unicode_coding_system,
+  fixed_width_coding_system
 };
 
 struct coding_system_methods
@@ -318,6 +319,28 @@
 				  const unsigned char *src,
 				  unsigned_char_dynarr *dst, Bytecount n);
 
+  /* Query method: Check whether the buffer text between point and END
+     can be encoded by this coding system. Returns
+     either nil (meaning the text can be encoded by the coding system) or a
+     range table object describing the stretches that the coding system
+     cannot encode.
+     
+     Possible values for flags are below, search for
+     QUERY_METHOD_IGNORE_INVALID_SEQUENCES.
+
+     Coding systems are expected to be able to behave sensibly with all
+     possible octets on decoding, which is why this method is only available
+     for encoding. */
+  Lisp_Object (*query_method) (Lisp_Object coding_system, struct buffer *buf,
+                               Charbpos end, int flags);
+
+  /* Same as the previous method, but this works in the context of
+     lstreams. (Where the data do need to be copied, unfortunately.)  The
+     intention is to implement the query method for the mswindows-multibyte
+     coding systems in terms of a query_lstream method. */
+  Lisp_Object (*query_lstream_method) (struct coding_stream *str,
+                                       const Ibyte *start, Bytecount n);
+
   /* Coding mark method: Mark any Lisp objects in the type-specific data
      attached to `struct coding_stream'.  Optional. */
   void (*mark_coding_stream_method) (struct coding_stream *str);
@@ -387,6 +410,24 @@
   int coding_data_size;
 };
 
+/* Values for flags, as passed to query_method. */
+
+#define QUERY_METHOD_IGNORE_INVALID_SEQUENCES 0x0001
+#define QUERY_METHOD_ERRORP                   0x0002
+#define QUERY_METHOD_HIGHLIGHT                0x0004
+
+enum query_coding_failure_reasons
+  {
+    query_coding_succeeded = 0,
+    query_coding_unencodable = 1, 
+    query_coding_invalid_sequence = 2
+  }; 
+
+extern Lisp_Object Qquery_coding_warning_face;
+
+Lisp_Object default_query_method (Lisp_Object, struct buffer *, Charbpos,
+                                  int);
+
 /***** Calling a coding-system method *****/
 
 #define RAW_CODESYSMETH(cs, m) ((cs)->methods->m##_method)
@@ -413,7 +454,6 @@
 #define XCODESYSMETH_OR_GIVEN(cs, m, args, given) \
   CODESYSMETH_OR_GIVEN (XCODING_SYSTEM (cs), m, args, given)
 
-
 /***** Defining new coding-system types *****/
 
 extern const struct sized_memory_description coding_system_empty_extra_description;
@@ -477,6 +517,7 @@
   ty##_coding_system_methods->extra_description =			\
     &coding_system_empty_extra_description;				\
   ty##_coding_system_methods->enumtype = ty##_coding_system;		\
+  ty##_coding_system_methods->query_method = default_query_method;      \
   defsymbol_nodump (&ty##_coding_system_methods->predicate_symbol,	\
                     pred_sym);						\
   add_entry_to_coding_system_type_list (ty##_coding_system_methods);	\
@@ -1029,6 +1070,7 @@
 #ifdef MULE
 DECLARE_CODING_SYSTEM_TYPE (iso2022);
 DECLARE_CODING_SYSTEM_TYPE (ccl);
+DECLARE_CODING_SYSTEM_TYPE (fixed_width);
 DECLARE_CODING_SYSTEM_TYPE (shift_jis);
 DECLARE_CODING_SYSTEM_TYPE (big5);
 #endif