[xemacs-hg @ 2004-11-04 23:05:23 by ben] commit mega-patch configure.in: Turn off -Winline and -Wchar-subscripts. Use the right set of cflags when compiling modules. Rewrite ldap configuration to separate the inclusion of lber (needed in recent Cygwin) from the basic checks for the needed libraries. add a function for MAKE_JUNK_C; initially code was added to generate xemacs.def using this, but it will need to be rewritten. add an rm -f for junk.c to avoid weird Cygwin bug with cp -f onto an existing file. Sort list of auto-detected functions and eliminate unused checks for stpcpy, setlocale and getwd. Add autodetection of Cygwin scanf problems BETA: Rewrite section on configure to indicate what flags are important and what not. digest-doc.c, make-dump-id.c, profile.c, sorted-doc.c: Add proper decls for main(). make-msgfile.c: Document that this is old junk. Move proposal to text.c. make-msgfile.lex: Move proposal to text.c. make-mswin-unicode.pl: Convert error-generating code so that the entire message will be seen as a single unrecognized token. mule/mule-ccl.el: Update docs. lispref/mule.texi: Update CCL docs. ldap/eldap.c: Mule-ize. Use EXTERNAL_LIST_LOOP_2 instead of deleted EXTERNAL_LIST_LOOP. * XEmacs 21.5.18 "chestnut" is released. --------------------------------------------------------------- MULE-RELATED WORK: --------------------------------------------------------------- --------------------------- byte-char conversion --------------------------- buffer.c, buffer.h, insdel.c, text.c: Port FSF algorithm for byte-char conversion, replacing broken previous version. Track the char position of the gap. Add functions to do char-byte conversion downwards as well as upwards. Move comments about algorithm workings to internals manual. --------------------------- work on types --------------------------- alloc.c, console-x-impl.h, dump-data.c, dump-data.h, dumper.c, dialog-msw.c, dired-msw.c, doc.c, editfns.c, esd.c, event-gtk.h, event-msw.c, events.c, file-coding.c, file-coding.h, fns.c, glyphs-eimage.c, glyphs-gtk.c, glyphs-msw.c, glyphs-shared.c, glyphs-x.c, glyphs.c, glyphs.h, gui.c, hpplay.c, imgproc.c, intl-win32.c, lrecord.h, lstream.c, keymap.c, lisp.h, libsst.c, linuxplay.c, miscplay.c, miscplay.h, mule-coding.c, nas.c, nt.c, ntheap.c, ntplay.c, objects-msw.c, objects-tty.c, objects-x.c, print.c, process-nt.c, process.c, redisplay.h, select-common.h, select-gtk.c, select-x.c, sgiplay.c, sound.c, sound.h, sunplay.c, sysfile.h, sysdep.c, syswindows.h, text.c, unexnt.c, win32.c, xgccache.c: Further work on types. This creates a full set of types for all the basic semantics of `char' that I have so far identified, so that its semantics can always be identified for the purposes of proper Mule-safe code, and the raw use of `char' always avoided. (1) More type renaming, for consistency of naming. Char_ASCII -> Ascbyte UChar_ASCII -> UAscbyte Char_Binary -> CBinbyte UChar_Binary -> Binbyte SChar_Binary -> SBinbyte (2) Introduce Rawbyte, CRawbyte, Boolbyte, Chbyte, UChbyte, and Bitbyte and use them. (3) New types Itext, Wexttext and Textcount for separating out the concepts of bytes and textual units (different under UTF-16 and UTF-32, which are potential internal encodings). (4) qxestr*_c -> qxestr*_ascii. lisp.h: New; goes with other qxe() functions. #### Maybe goes in a different section. lisp.h: Group generic int-type defs together with EMACS_INT defs. lisp.h: * lisp.h (WEXTTEXT_IS_WIDE) New defns. lisp.h: New type to replace places where int occurs as a boolean. It's signed because occasionally people may want to use -1 as an error value, and because unsigned ints are viral -- see comments in the internals manual against using them. dynarr.c: int -> Bytecount. --------------------------- Mule-izing --------------------------- device-x.c: Partially Mule-ize. dumper.c, dumper.h: Mule-ize. Use Rawbyte. Use stderr_out not printf. Use wext_*(). sysdep.c, syswindows.h, text.c: New Wexttext API for manipulation of external text that may be Unicode (e.g. startup code under Windows). emacs.c: Mule-ize. Properly deal with argv in external encoding. Use wext_*() and Wexttext. Use Rawbyte. #if 0 some old junk on SCO that is unlikely to be correct. Rewrite allocation code in run-temacs. emacs.c, symsinit.h, win32.c: Rename win32 init function and call it even earlier, to initialize mswindows_9x_p even earlier, for use in startup code (XEUNICODE_P). process.c: Use _wenviron not environ under Windows, to get Unicode environment variables. event-Xt.c: Mule-ize drag-n-drop related stuff. dragdrop.c, dragdrop.h, frame-x.c: Mule-ize. text.h: Add some more stand-in defines for particular kinds of conversion; use in Mule-ization work in frame-x.c etc. --------------------------- Freshening --------------------------- intl-auto-encap-win32.c, intl-auto-encap-win32.h: Regenerate. --------------------------- Unicode-work --------------------------- intl-win32.c, syswindows.h: Factor out common options to MultiByteToWideChar and WideCharToMultiByte. Add convert_unicode_to_multibyte_malloc() and convert_unicode_to_multibyte_dynarr() and use. Add stuff for alloca() conversion of multibyte/unicode. alloc.c: Use dfc_external_data_len() in case of unicode coding system. alloc.c, mule-charset.c: Don't zero out and reinit charset Unicode tables. This fucks up dump-time loading. Anyway, either we load them at dump time or run time, never both. unicode.c: Dump the blank tables as well. --------------------------------------------------------------- DOCUMENTATION, MOSTLY MULE-RELATED: --------------------------------------------------------------- EmacsFrame.c, emodules.c, event-Xt.c, fileio.c, input-method-xlib.c, mule-wnnfns.c, redisplay-gtk.c, redisplay-tty.c, redisplay-x.c, regex.c, sysdep.c: Add comment about Mule work needed. text.h: Add more documentation describing why DFC routines were not written to return their value. Add some other DFC documentation. console-msw.c, console-msw.h: Add pointer to docs in win32.c. emacs.c: Add comments on sources of doc info. text.c, charset.h, unicode.c, intl-win32.c, intl-encap-win32.c, text.h, file-coding.c, mule-coding.c: Collect background comments and related to text matters and internationalization, and proposals for work to be done, in text.c or Internals manual, stuff related to specific textual API's in text.h, and stuff related to internal implementation of Unicode conversion in unicode.c. Put lots of pointers to the comments to make them easier to find. s/mingw32.h, s/win32-common.h, s/win32-native.h, s/windowsnt.h, win32.c: Add bunches of new documentation on the different kinds of builds and environments under Windows and how they work. Collect this info in win32.c. Add pointers to these docs in the relevant s/* files. emacs.c: Document places with long comments. Remove comment about exiting, move to internals manual, put in pointer. event-stream.c: Move docs about event queues and focus to internals manual, put in pointer. events.h: Move docs about event stream callbacks to internals manual, put in pointer. profile.c, redisplay.c, signal.c: Move documentation to the Internals manual. process-nt.c: Add pointer to comment in win32-native.el. lisp.h: Add comments about some comment conventions. lisp.h: Add comment about the second argument. device-msw.c, redisplay-msw.c: @@#### comments are out-of-date. --------------------------------------------------------------- PDUMP WORK (MOTIVATED BY UNICODE CHANGES) --------------------------------------------------------------- alloc.c, buffer.c, bytecode.c, console-impl.h, console.c, device.c, dumper.c, lrecord.h, elhash.c, emodules.h, events.c, extents.c, frame.c, glyphs.c, glyphs.h, mule-charset.c, mule-coding.c, objects.c, profile.c, rangetab.c, redisplay.c, specifier.c, specifier.h, window.c, lstream.c, file-coding.h, file-coding.c: PDUMP: Properly implement dump_add_root_block(), which never worked before, and is necessary for dumping Unicode tables. Pdump name changes for accuracy: XD_STRUCT_PTR -> XD_BLOCK_PTR. XD_STRUCT_ARRAY -> XD_BLOCK_ARRAY. XD_C_STRING -> XD_ASCII_STRING. *_structure_* -> *_block_*. lrecord.h: some comments added about dump_add_root_block() vs dump_add_root_block_ptr(). extents.c: remove incorrect comment about pdump problems with gap array. --------------------------------------------------------------- ALLOCATION --------------------------------------------------------------- abbrev.c, alloc.c, bytecode.c, casefiddle.c, device-msw.c, device-x.c, dired-msw.c, doc.c, doprnt.c, dragdrop.c, editfns.c, emodules.c, file-coding.c, fileio.c, filelock.c, fns.c, glyphs-eimage.c, glyphs-gtk.c, glyphs-msw.c, glyphs-x.c, gui-msw.c, gui-x.c, imgproc.c, intl-win32.c, lread.c, menubar-gtk.c, menubar.c, nt.c, objects-msw.c, objects-x.c, print.c, process-nt.c, process-unix.c, process.c, realpath.c, redisplay.c, search.c, select-common.c, symbols.c, sysdep.c, syswindows.h, text.c, text.h, ui-byhand.c: New macros {alloca,xnew}_{itext,{i,ext,raw,bin,asc}bytes} for more convenient allocation of these commonly requested items. Modify functions to use alloca_ibytes, alloca_array, alloca_extbytes, xnew_ibytes, etc. also XREALLOC_ARRAY, xnew. alloc.c: Rewrite the allocation functions to factor out repeated code. Add assertions for freeing dumped data. lisp.h: Moved down and consolidated with other allocation stuff. lisp.h, dynarr.c: New functions for allocation that's very efficient when mostly in LIFO order. lisp.h, text.c, text.h: Factor out some stuff for general use by alloca()-conversion funs. text.h, lisp.h: Fill out convenience routines for allocating various kinds of bytes and put them in lisp.h. Use them in place of xmalloc(), ALLOCA(). text.h: Fill out the convenience functions so the _MALLOC() kinds match the alloca() kinds. --------------------------------------------------------------- ERROR-CHECKING --------------------------------------------------------------- text.h: Create ASSERT_ASCTEXT_ASCII() and ASSERT_ASCTEXT_ASCII_LEN() from similar Eistring checkers and change the Eistring checkers to use them instead. --------------------------------------------------------------- MACROS IN LISP.H --------------------------------------------------------------- lisp.h: Redo GCPRO declarations. Create a "base" set of functions that can be used to generate any kind of gcpro sets -- regular, ngcpro, nngcpro, private ones used in GC_EXTERNAL_LIST_LOOP_2. buffer.c, callint.c, chartab.c, console-msw.c, device-x.c, dialog-msw.c, dired.c, extents.c, ui-gtk.c, rangetab.c, nt.c, mule-coding.c, minibuf.c, menubar-msw.c, menubar.c, menubar-gtk.c, lread.c, lisp.h, gutter.c, glyphs.c, glyphs-widget.c, fns.c, fileio.c, file-coding.c, specifier.c: Eliminate EXTERNAL_LIST_LOOP, which does not check for circularities. Use EXTERNAL_LIST_LOOP_2 instead or EXTERNAL_LIST_LOOP_3 or EXTERNAL_PROPERTY_LIST_LOOP_3 or GC_EXTERNAL_LIST_LOOP_2 (new macro). Removed/redid comments on EXTERNAL_LIST_LOOP. --------------------------------------------------------------- SPACING FIXES --------------------------------------------------------------- callint.c, hftctl.c, number-gmp.c, process-unix.c: Spacing fixes. --------------------------------------------------------------- FIX FOR GEOMETRY PROBLEM IN FIRST FRAME --------------------------------------------------------------- unicode.c: Add workaround for newlib bug in sscanf() [should be fixed by release 1.5.12 of Cygwin]. toolbar.c: bug fix for problem of initial frame being 77 chars wide on Windows. will be overridden by my other ws. --------------------------------------------------------------- FIX FOR LEAKING PROCESS HANDLES: --------------------------------------------------------------- process-nt.c: Fixes for leaking handles. Inspired by work done by Adrian Aichner <adrian@xemacs.org>. --------------------------------------------------------------- FIX FOR CYGWIN BUG (Unicode-related): --------------------------------------------------------------- unicode.c: Add workaround for newlib bug in sscanf() [should be fixed by release 1.5.12 of Cygwin]. --------------------------------------------------------------- WARNING FIXES: --------------------------------------------------------------- console-stream.c: `reinit' is unused. compiler.h, event-msw.c, frame-msw.c, intl-encap-win32.c, text.h: Add stuff to deal with ANSI-aliasing warnings I got. regex.c: Gather includes together to avoid warning. --------------------------------------------------------------- CHANGES TO INITIALIZATION ROUTINES: --------------------------------------------------------------- buffer.c, emacs.c, console.c, debug.c, device-x.c, device.c, dragdrop.c, emodules.c, eval.c, event-Xt.c, event-gtk.c, event-msw.c, event-stream.c, event-tty.c, events.c, extents.c, faces.c, file-coding.c, fileio.c, font-lock.c, frame-msw.c, glyphs-widget.c, glyphs.c, gui-x.c, insdel.c, lread.c, lstream.c, menubar-gtk.c, menubar-x.c, minibuf.c, mule-wnnfns.c, objects-msw.c, objects.c, print.c, scrollbar-x.c, search.c, select-x.c, text.c, undo.c, unicode.c, window.c, symsinit.h: Call reinit_*() functions directly from emacs.c, for clarity. Factor out some redundant init code. Move disallowed stuff that had crept into vars_of_glyphs() into complex_vars_of_glyphs(). Call init_eval_semi_early() from eval.c not in the middle of vars_of_() in emacs.c since there should be no order dependency in the latter calls. --------------------------------------------------------------- ARMAGEDDON: --------------------------------------------------------------- alloc.c, emacs.c, lisp.h, print.c: Rename inhibit_non_essential_printing_operations to inhibit_non_essential_conversion_operations. text.c: Assert on !inhibit_non_essential_conversion_operations. console-msw.c, print.c: Don't do conversion in SetConsoleTitle or FindWindow to avoid problems during armageddon. Put #errors for NON_ASCII_INTERNAL_FORMAT in places where problems would arise. --------------------------------------------------------------- CHANGES TO THE BUILD PROCEDURE: --------------------------------------------------------------- config.h.in, s/cxux.h, s/usg5-4-2.h, m/powerpc.h: Add comment about correct ordering of this file. Rearrange everything to follow this -- put all #undefs together and before the s&m files. Add undefs for HAVE_ALLOCA, C_ALLOCA, BROKEN_ALLOCA_IN_FUNCTION_CALLS, STACK_DIRECTION. Remove unused HAVE_STPCPY, HAVE_GETWD, HAVE_SETLOCALE. m/gec63.h: Deleted; totally broken, not used at all, not in FSF. m/7300.h, m/acorn.h, m/alliant-2800.h, m/alliant.h, m/altos.h, m/amdahl.h, m/apollo.h, m/att3b.h, m/aviion.h, m/celerity.h, m/clipper.h, m/cnvrgnt.h, m/convex.h, m/cydra5.h, m/delta.h, m/delta88k.h, m/dpx2.h, m/elxsi.h, m/ews4800r.h, m/gould.h, m/hp300bsd.h, m/hp800.h, m/hp9000s300.h, m/i860.h, m/ibmps2-aix.h, m/ibmrs6000.h, m/ibmrt-aix.h, m/ibmrt.h, m/intel386.h, m/iris4d.h, m/iris5d.h, m/iris6d.h, m/irist.h, m/isi-ov.h, m/luna88k.h, m/m68k.h, m/masscomp.h, m/mg1.h, m/mips-nec.h, m/mips-siemens.h, m/mips.h, m/news.h, m/nh3000.h, m/nh4000.h, m/ns32000.h, m/orion105.h, m/pfa50.h, m/plexus.h, m/pmax.h, m/powerpc.h, m/pyrmips.h, m/sequent-ptx.h, m/sequent.h, m/sgi-challenge.h, m/symmetry.h, m/tad68k.h, m/tahoe.h, m/targon31.h, m/tekxd88.h, m/template.h, m/tower32.h, m/tower32v3.h, m/ustation.h, m/vax.h, m/wicat.h, m/xps100.h: Delete C_ALLOCA, HAVE_ALLOCA, STACK_DIRECTION, BROKEN_ALLOCA_IN_FUNCTION_CALLS. All of this is auto-detected. When in doubt, I followed recent FSF sources, which also have these things deleted.

--- a/src/intl-win32.c	Thu Nov 04 22:51:31 2004 +0000
+++ b/src/intl-win32.c	Thu Nov 04 23:08:28 2004 +0000
@@ -1,5 +1,5 @@
 /* Win32 internationalization functions.
-   Copyright (C) 2000, 2001, 2002 Ben Wing.
+   Copyright (C) 2000, 2001, 2002, 2004 Ben Wing.
    Copyright (C) 2000 IKEYAMA Tomonori.
 
 This file is part of XEmacs.
@@ -53,6 +53,15 @@
 # define NO_EXT_MULTIBYTE_FEATURES
 #endif
 
+
+
+/*
+
+Detailed info on Microsoft Windows-Related Multilingual Issues:
+
+  (Info-goto-node "(internals)Microsoft Windows-Related Multilingual Issues")
+*/
+
 Lisp_Object Qmswindows_multibyte, Qmswindows_multibyte_to_unicode;
 Lisp_Object Qmswindows_tstr, Qmswindows_unicode;
 Lisp_Object Qmswindows_multibyte_system_default;
@@ -73,74 +82,6 @@
 /*            Language/locale/code page conversion functions            */
 /************************************************************************/
 
-/* There are various different ways of representing the vague concept
-   of "language", and it can be very confusing.  So:
-
-   -- The C library has the concept of "locale", which is a
-   combination of language and country, and which controls the way
-   currency and dates are displayed, the encoding of data, etc.
-
-   -- XEmacs has the concept of "language environment", more or less
-   like a locale; although currently in most cases it just refers to
-   the language, and no sub-language distinctions are
-   made. (Exceptions are with Chinese, which has different language
-   environments for Taiwan and mainland China, due to the different
-   encodings and writing systems.)
-
-   -- Windows has a number of different language concepts:
-
-   1. There are "languages" and "sublanguages", which correspond to
-   the languages and countries of the C library -- e.g. LANG_ENGLISH
-   and SUBLANG_ENGLISH_US.  These are identified by 8-bit integers,
-   called the "primary language identifier" and "sublanguage
-   identifier", respectively.  These are combined into a 16-bit
-   integer or "language identifier" by MAKELANGID().
-
-   2. The language identifier in turn is combined with a "sort
-   identifier" (and optionally a "sort version") to yield a 32-bit
-   integer called a "locale identifier" (type LCID), which identifies
-   locales -- the primary means of distinguishing language/regional
-   settings and similar to C library locales.
-
-   3. "Code pages" identify different text encodings (i.e. a set of
-   supported characters, an enumeration of those characters [i.e. an
-   association of character with number or number pair; there may be
-   disjoint ranges of numbers supported]), and a way of encoding a
-   stream of those characters into an 8-bit data stream).  All of the
-   encodings are either one-byte or mixed one-byte/two-byte encodings,
-   all non-modal; in the mixed encodings, two-byte characters have the
-   first byte >= 128, although the second byte may or may not be
-   restricted to this range, depending on the encoding.  Code pages
-   are similar to XEmacs "charsets"; the latter also define a set of
-   supported characters and an enumeration of those characters (but
-   code pages in additionally define an encoding, which charsets don't
-   do).  Code pages often function in Windows like charsets in XEmacs.
-
-   4. Every Windows locale has a specific code page associated with
-   it; more than one locale can share a code page -- e.g. all the
-   Western European languages, including English, do.
-
-   5. Windows also has an "input locale identifier" (aka "keyboard
-   layout id") or HKL, which is a 32-bit integer composed of the
-   16-bit language identifier and a 16-bit "device identifier", which
-   originally specified a particular keyboard layout (e.g. the locale
-   "US English" can have the QWERTY layout, the Dvorak layout, etc.),
-   but has been expanded to include speech-to-text converters and
-   other non-keyboard ways of inputting text.  Note that both the HKL
-   and LCID share the language identifier in the lower 16 bits, and in
-   both cases a 0 in the upper 16 bits means "default" (sort order or
-   device), providing a way to convert between HKL's, LCID's, and
-   language identifiers (i.e. language/sublanguage pairs).  The
-   default keyboard layout for a language is (as far as I can
-   determine) established using the Regional Settings control panel
-   applet, where you can add input locales as combinations of language
-   (actually language/sublanguage) and layout; presumably if you list
-   only one input locale with a particular language, the corresponding
-   layout is the default for that language.  But what if you list more
-   than one?  You can specify a single default input locale, but there
-   appears to be no way to do so on a per-language basis.
-*/   
-
 struct lang_to_string
 {
   int code;
@@ -675,7 +616,7 @@
   int i;
 
   for (i = 0; i < table_size; i++)
-    if (!strcmp ((char *) XSTRING_DATA (lang), table[i].string))
+    if (!qxestrcmp_ascii (XSTRING_DATA (lang), table[i].string))
       return table[i].code;
   return -1;
 }
@@ -688,14 +629,14 @@
 
   for (i = 0; i < table_size; i++)
     if (table[i].string &&
-	!strcmp ((char *) XSTRING_DATA (lang), table[i].string))
+	!qxestrcmp_ascii (XSTRING_DATA (lang), table[i].string))
       return table[i].code;
 
-  if (!strcmp ((char *) XSTRING_DATA (lang), "NEUTRAL"))
+  if (!qxestrcmp_ascii (XSTRING_DATA (lang), "NEUTRAL"))
     return SUBLANG_NEUTRAL;
-  if (!strcmp ((char *) XSTRING_DATA (lang), "DEFAULT"))
+  if (!qxestrcmp_ascii (XSTRING_DATA (lang), "DEFAULT"))
     return SUBLANG_DEFAULT;
-  if (!strcmp ((char *) XSTRING_DATA (lang), "SYS_DEFAULT"))
+  if (!qxestrcmp_ascii (XSTRING_DATA (lang), "SYS_DEFAULT"))
     return SUBLANG_SYS_DEFAULT;
 
   return -1;
@@ -1093,11 +1034,11 @@
 }
 
 static DWORD
-int_from_hex (Char_ASCII *s)
+int_from_hex (Ascbyte *s)
 {
   DWORD val = 0;
-  static Char_ASCII hex[] = "0123456789abcdefABCDEF";
-  Char_ASCII *p;
+  static Ascbyte hex[] = "0123456789abcdefABCDEF";
+  Ascbyte *p;
 
   while (*s && (p = strchr (hex, *s)) != NULL)
     {
@@ -1115,7 +1056,7 @@
 static Lisp_Object Vmswindows_valid_locales;
 
 static BOOL CALLBACK
-enum_locale_fn (Char_ASCII *localeNum)
+enum_locale_fn (Ascbyte *localeNum)
 {
   DWORD id = int_from_hex (localeNum);
   Vmswindows_valid_locales =
@@ -1485,7 +1426,7 @@
        ())
 {
   int num_layouts = GetKeyboardLayoutList (0, NULL);
-  HKL * layouts = (HKL *) ALLOCA (num_layouts * sizeof (HKL));
+  HKL *layouts = alloca_array (HKL, num_layouts);
   Lisp_Object obj = Qnil;
 
   if (GetKeyboardLayoutList (num_layouts, layouts) == num_layouts)
@@ -1682,7 +1623,7 @@
 wcsdup (const wchar_t *str)
 {
   int len = wcslen (str) + 1;
-  void *val = xmalloc (len * sizeof (wchar_t));
+  wchar_t *val = xnew_array (wchar_t, len);
 
   if (val == 0) return 0;
   return (wchar_t *) memcpy (val, str, len * sizeof (wchar_t));
@@ -1924,6 +1865,10 @@
   write_c_string (printcharfun, ")");
 }
 
+/* ----------------------------------------------------------------------- */
+/*                      Unicode/Multibyte converters                       */
+/* ----------------------------------------------------------------------- */
+
 /* Convert multibyte to Unicode according to the specified code page
    and return the value as a malloc()ed string.  This currently exists
    because the TO_INTERNAL_FORMAT() mechanism -- the normal way to do
@@ -1934,21 +1879,40 @@
    called recursively merely when I'm doing a simple conversion
    operation?  It turns out this can and does happen, consistently, as
    a result of calling QUIT -- it happens consistently for complicated
-   reasons outlined in event-msw.c, WM_KEYDOWN handling.) */
+   reasons outlined in event-msw.c, WM_KEYDOWN handling.)
+
+   NOTE: These functions are also used at startup. */
 
 Extbyte *
 convert_multibyte_to_unicode_malloc (const Extbyte *src, Bytecount n,
 				     int cp, Bytecount *size_out)
 {
-  Bytecount nout = MultiByteToWideChar (cp, 0, src, n, 0, 0);
+  Charcount nout = MultiByteToWideChar (cp, MBTOWC_OPTIONS, src, n, 0, 0);
   Extbyte *outp = xnew_array (Extbyte, nout * sizeof (WCHAR));
 
-  MultiByteToWideChar (cp, 0, src, n, (LPWSTR) outp, nout);
+  MultiByteToWideChar (cp, MBTOWC_OPTIONS, src, n, (LPWSTR) outp, nout);
   if (size_out)
     *size_out = nout * sizeof (WCHAR);
   return outp;
 }
 
+Extbyte *
+convert_unicode_to_multibyte_malloc (const Extbyte *src, Bytecount n,
+				     int cp, Bytecount *size_out)
+{
+  Bytecount nout = WideCharToMultiByte (cp, WCTOMB_OPTIONS, (LPWSTR) src,
+					n / sizeof (WCHAR),
+					0, 0, WCTOMB_INVALID_STRING, 0);
+  Extbyte *outp = xnew_array (Extbyte, nout);
+
+  WideCharToMultiByte (cp, WCTOMB_OPTIONS, (LPWSTR) src,
+		       n / sizeof (WCHAR), (LPSTR) outp, nout,
+		       WCTOMB_INVALID_STRING, 0);
+  if (size_out)
+    *size_out = nout;
+  return outp;
+}
+
 /* Convert MS Windows multibyte to internal, with specified code page.
    See above for why this exists, and the TO_INTERNAL_FORMAT() macros
    aren't just used. */
@@ -1972,6 +1936,69 @@
   return intdata;
 }
 
+static void
+unicode_multibyte_convert_now_damn_it (const void *src, Bytecount src_size,
+				       int to_unicode, void **dst,
+				       Bytecount *dst_size, int cp)
+{
+  if (to_unicode)
+    {
+      Charcount nout = MultiByteToWideChar (cp, MBTOWC_OPTIONS, (LPSTR) src,
+					    src_size, 0, 0);
+      Extbyte *outp = (Extbyte *) stack_like_malloc (nout * sizeof (WCHAR));
+
+      MultiByteToWideChar (cp, MBTOWC_OPTIONS, (LPSTR) src, src_size,
+			   (LPWSTR) outp, nout);
+      *dst = (void *) outp;
+      *dst_size = nout * sizeof (WCHAR);
+    }
+  else
+    {
+      Bytecount nout = WideCharToMultiByte (cp, WCTOMB_OPTIONS, (LPWSTR) src,
+					    src_size / sizeof (WCHAR),
+					    0, 0, WCTOMB_INVALID_STRING, 0);
+      Extbyte *outp = (Extbyte *) stack_like_malloc (nout);
+
+      WideCharToMultiByte (cp, WCTOMB_OPTIONS, (LPWSTR) src,
+			   src_size / sizeof (WCHAR), (LPSTR) outp, nout,
+			   WCTOMB_INVALID_STRING, 0);
+      *dst = (void *) outp;
+      *dst_size = nout;
+    }
+}
+
+Bytecount
+unicode_multibyte_convert_size (const char *srctext, const void *src,
+				Bytecount src_size, int to_unicode, int cp)
+{
+  alloca_convert_vals vals;
+
+  assert (find_pos_of_existing_active_alloca_convert (srctext) < 0);
+
+  vals.srctext = srctext;
+
+  unicode_multibyte_convert_now_damn_it (src, src_size, to_unicode, &vals.dst,
+					 &vals.dst_size, cp);
+
+  Dynarr_add (active_alloca_convert, vals);
+  return vals.dst_size;
+}
+
+void *
+unicode_multibyte_convert_copy_data (const char *srctext, void *alloca_data)
+{
+  alloca_convert_vals *vals;
+  int i = find_pos_of_existing_active_alloca_convert (srctext);
+
+  assert (i >= 0);
+  vals = Dynarr_atp (active_alloca_convert, i);
+  assert (alloca_data);
+  memcpy (alloca_data, vals->dst, vals->dst_size);
+  stack_like_free (vals->dst);
+  Dynarr_delete (active_alloca_convert, i);
+  return alloca_data;
+}
+
 /* Convert multibyte to Unicode according to the specified code page
    and append the results onto the specified Dynarr.  See above. */
 
@@ -1979,15 +2006,33 @@
 convert_multibyte_to_unicode_dynarr (const Extbyte *src, Bytecount n,
 				     int cp, unsigned_char_dynarr *dst)
 {
-  Bytecount nout = MultiByteToWideChar (cp, 0, src, n, 0, 0);
+  Charcount nout = MultiByteToWideChar (cp, MBTOWC_OPTIONS, src, n, 0, 0);
   void *outp;
 
   Dynarr_add_many (dst, 0, nout * sizeof (WCHAR));
   /* dynarr's buffer may be realloc()ed by call above, so access it after */
   outp = Dynarr_atp (dst, Dynarr_length (dst) - nout * sizeof (WCHAR));
-  MultiByteToWideChar (cp, 0, src, n, (LPWSTR) outp, nout);
+  MultiByteToWideChar (cp, MBTOWC_OPTIONS, src, n, (LPWSTR) outp, nout);
 }
 
+void
+convert_unicode_to_multibyte_dynarr (const Extbyte *src, Bytecount n,
+				     int cp, unsigned_char_dynarr *dst)
+{
+  Bytecount nout = WideCharToMultiByte (cp, WCTOMB_OPTIONS, (LPWSTR) src,
+					n / sizeof (WCHAR),
+					0, 0, WCTOMB_INVALID_STRING, 0);
+  void *outp;
+
+  Dynarr_add_many (dst, 0, nout);
+  /* dynarr's buffer may be realloc()ed by call above, so access it after */
+  outp = Dynarr_atp (dst, Dynarr_length (dst) - nout);
+  WideCharToMultiByte (cp, WCTOMB_OPTIONS, (LPWSTR) src,
+		       n / sizeof (WCHAR), (LPSTR) outp, nout,
+		       WCTOMB_INVALID_STRING, 0);
+}
+
+
 /* Convert MS Windows multibyte to Unicode. */
 
 static Bytecount
@@ -2054,20 +2099,8 @@
       else
 	data->partial_byte_present = 0;
 
-      {
-	int nout = WideCharToMultiByte (data->cp, WC_COMPOSITECHECK,
-					(LPWSTR) new_src, n / sizeof (WCHAR),
-					0, 0, "~", 0);
-	void *outp;
-
-	Dynarr_add_many (dst, 0, nout);
-	/* dynarr's buffer may be realloc()ed by call above, so access it
-           after */
-	outp = Dynarr_atp (dst, Dynarr_length (dst) - nout);
-	WideCharToMultiByte (data->cp, WC_COMPOSITECHECK, (LPWSTR) new_src,
-			     n / sizeof (WCHAR),
-			     (LPSTR) outp, nout, "~", 0);
-      }
+      convert_unicode_to_multibyte_dynarr ((Extbyte *) new_src, n, data->cp,
+					   dst);
     }
   return orign;
 }