--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/README.ben-mule-21-5	Wed Mar 13 08:54:06 2002 +0000
@@ -0,0 +1,1587 @@
+oct 27, 2001:
+
+-------- proposal for better buffer-switching commands:
+
+implement what VC++ currently has.  you have a single "switch" command like
+CTRL-TAB, which as long as you hold the CTRL button down, brings successive
+buffers that are "next in line" into the current position, bumping the rest
+forward.  once you release the CTRL key, the chain is broken, and further
+CTRL-TABs will start from the beginning again.  this way, frequently used
+buffers naturally move toward the front of the chain, and you can switch
+back and forth between two buffers using CTRL-TAB.  the only thing about
+CTRL-TAB is it's a bit awkward.  the way to implement is to have
+modifier-up strokes fire off a hook, like modifier-up-hook.  this is driven
+by event dispatch, so there are no synchronization issues.  when C-tab is
+pressed, the binding function does something like set a one-shot handler on
+the modifier-up-hook (perhaps separate hooks for separate modifiers?).
+
+to do this, we'd also want to change the buffer tabs so that they maintain
+their own order.  in particular, they start out synched to the regular
+order, but as you make changes, you don't want the tabs to change
+order. (in fact, they may already do this.) selecting a particular buffer
+from the buffer tabs DOES make the buffer go to the head of the line.  the
+invariant is that if the tabs are displaying X items, those X items are the
+first X items in the standard buffer list, but may be in a different
+order. (it looks like the tabs may already implement all of this.)
+
+oct 26, 2001:
+
+necessary testing/changes:
+
+- test all eol detection stuff under windows w/ and w/o mule, unix w/ and
+  w/o mule. (test configure flag, command-line flag, menu option) may need
+  a way of pretending to be unix under cygwin.
+- test under windows w/ and w/o mule, cygwin w/ and w/o mule, cygwin x
+  windows w/ and w/o mule.
+- test undecided-dos/unix/mac.
+- check ESC ESC works as isearch-quit under TTY's.
+- test coding-system-base and all its uses (grep for them).
+- menu item to revert to most recent auto save.
+- consider renaming build_string -> build_intstring and build_c_string to
+  build_string. (consistent with build_msg_string et al; many more
+  build_c_string than build_string)
+
+oct 20, 2001:
+
+fixed problem causing crash due to invalid internal-format data, fixed an
+existing bug in valid_char_p, and added checks to more quickly catch when
+invalid chars are generated.  still need to investigate why
+mswindows-multibyte is being detected.
+
+i now see why -- we only process 65536 bytes due to a constant
+MAX_BYTES_PROCESSED_FOR_DETECTION.  instead, we should have no limit as
+long as we have a seekable stream.  we also need to write
+stderr_out_lisp(), used in the debug info routines i wrote.
+
+check once more about DEBUG_XEMACS.  i think debugging info should be
+ON by default.  make sure it is.  check that nothing untoward will result
+in a production system, e.g. presumably assert()s should not really abort().
+(!! Actually, this should be runtime settable!  Use a variable for this, and
+it can be set using the same XEMACSDEBUG method.  In fact, now that I think
+of it, I'm sure that debugging info should be on always, with runtime ways
+of turning on or off any funny behavior.)
+
+oct 19, 2001:
+
+fixed various bugs preventing packages from being able to be built.  still
+another bug, with psgml/etc/cdtd/docbook, which contains some strange
+characters starting around char pos 110,000.  It gets detected as
+mswindows-multibyte (wrong! why?) and then invalid internal-format data is
+generated.  need to fix mswindows-multibyte (and possibly add something
+that signals an error as well; need to work on this error-signalling
+mechanism) and figure out why it's getting detected as such.  what i should
+do is add a debug var that outputs blow-by-blow info of the detection
+process.
+
+oct 9, 2001:
+
+the stuff with global-window-system-map doesn't appear to work.  in any
+case it needs better documentation. [DONE]
+
+M-home, M-end do work, but cause cl-macs to get loaded.  why?
+
+oct 8, 2001:
+
+finished the coding system changes and they finally work!
+
+need to implement undecided-unix/dos/mac.  they should be easy to do; it
+should be enough to specify an eol-type but not do-eol, but check this.
+
+consider making the standard naming be foo-lf/crlf/cr, with unix/dos/mac as
+aliases.
+
+print methods for coding systems should include some of the generic
+properties. (also then fix print_..._within_print_method). [DONE]
+
+in a little while, go back and delete the text-file-wrapper-coding-system
+code. (it'll be in CVS if necessary to get at it.) [DONE]
+
+need to verify at some point that non-text-file coding systems work
+properly when specified.  when gzip is working, this would be a good test
+case. (and consider creating base64 as well!)
+
+remove extra crap from coding-system-category that checks for chain coding
+systems. [DONE]
+
+perhaps make a primitive that gets at coding-system-canonical. [DONE]
+
+need to test cygwin, compiling the mule packages, get unix-eol stuff
+working.  frank from germany says he doesn't see a lisp backtrace when he
+gets an error during temacs?  verify that this actually gets outputted.
+
+consider putting the current language on the modeline, mousable so it can
+be switched.  also consider making the coding system be mousable and the
+line number (pick a line) and the percentage (pick a percentage).
+
+oct 6, 2001:
+
+added code so that debug_print() will output a newline to the mswindows
+debugging output, not just the console.  need to test. [DONE]
+
+working on problem where all files are being detected as binary.  the
+problem may be that the undecided coding system is getting wrapped with an
+auto-eol coding system, which it shouldn't be -- but even in this
+situation, we should get the right results!  check the
+canonicalize-after-coding methods.  also, determine_real_coding_system
+appears to be getting called even when we're not detecting encoding.  also,
+undecided needs a print method to show its params, and chain needs to be
+updated to show canonicalize_after_coding.  check others as well. [DONE]
+
+oct 5, 2001:
+
+finished up coding system changes, testing.
+
+errors byte-compiling files in iso-2022-7-bit.  perhaps it's not correctly
+detecting the encoding?
+
+noticed a problem in the dfc macros: we call
+get_coding_system_for_text_file with eol_wrap == 1, to allow for
+auto-detection of the eol type; but this defeats the check and
+short-circuit for unicode.
+
+still need to implement calling determine_real_coding_system() for
+non-seekable streams.  to implement correctly, we need to do our own
+buffering. [DONE, BUT WITHOUT BUFFERING]
+
+oct 4, 2001:
+
+implemented most stuff below.
+
+need to finish up changes to make_coding_system_1. (i changed the way
+internal coding systems were handled; i need to create subsidiaries for all
+types of coding systems, not just text ones.) there's a nasty xfree() crash
+i was hitting; perhaps it'll go away once all stuff has been rewritten.
+
+check under cygwin to make sure that when an error occurs during loadup, a
+backtrace is output.
+
+as soon as andy releases his new setup, we should put it onto various
+standard windows software repositories.
+
+oct 3, 2001:
+
+added global-tty-map and global-window-system-map.  add some stuff to the
+maps, e.g. C-x ESC for repeat vs. C-x ESC ESC on TTY's, and of course ESC
+ESC on window systems vs. ESC ESC ESC on TTY's. [TEST]
+
+was working on integrating the two help-for-tutorial versions (mule,
+non-mule). [DONE, but test under non-Mule]
+
+was working on the file-coding changes.  need to think more about
+text-file-wrapper.  conclusion i think is that
+get_coding_system_for_text_file should wrap using a special coding system
+type called a text-file-wrapper, which inherits from chain, and implements
+canonicalize-after-decoding to just return the unwrapped coding system.  We
+need to implement inheritance of coding systems, which will certainly come
+in extremely useful when coding systems get implemented in Lisp, which
+should happen at some point. (see existing docs about this.)  essentially,
+we have a way of declaring that we inherit from some system, and the
+appropriate data structures get created, perhaps just an extra inheritance
+pointer.  but when we create the coding system, the extra data needs to be
+a stretchy array of offsets, pointing to the type-specific data for the
+coding system type and all its parents.  that means that in the methods
+structure for a coding system (which perhaps should be expanded beyond
+method, it's just a "class structure") is the index in these arrays of
+offsets.  CODING_SYSTEM_DATA() can take any of the coding system classes
+(rename type to class!) that make up this class.  similarly, a coding
+system class inherits its methods from the class above unless specifying
+its own method, and can call the superclass method at any point by either
+just invoking its name, or conceivably by some macro like
+
+CALL_SUPER (method, (args))
+
+similar mods would have to be made to coding stream structures.
+
+perhaps for the immediate we can just sort of fake things like we currently
+do with undecided calling some stuff from chain.
+
+oct 2, 2001:
+
+need to implement support for iso-8859-15, i.e. iso-8859-1 + euro symbol.
+figure out how to fall back to iso-8859-1 as necessary.
+
+leave the current bindings the way they are for the moment, but bump off
+M-home and M-end (hardly used), and substitute my buffer movement stuff
+there. [DONE, but test]
+
+there's something to be said for combining block of 6 and paragraph,
+esp. if we make the definition of "paragraph" be so that it skips by 6 when
+within code.  hmm.
+
+eliminate advertised-undo crap, and similar hacks. [DONE]
+
+think about obsolete stuff to be eliminated.  think about eliminating or
+dimming obsolete items from hyper-apropos and something similar in
+completion buffers.
+
+sep 30, 2001:
+
+synched up the tutorials with FSF 21.0.105.  was rewriting them to favor
+the cursor keys over the older C-p, etc. keys.
+
+Got thinking about key bindings again.
+
+(1) I think that M-up/down and M-C-up/down should be reversed.  I use
+    scroll-up/down much more often than motion by paragraph.
+
+(2) Should we eliminate move by block (of 6) and subsitute it for
+    paragraph?  This would have the advantage that I could make bindings
+    for buffer change (forward/back buffer, perhaps M-C-up/down.  with
+    shift, M-C-S-up/down only goes within the same type (C files, etc.).
+    alternatively, just bump off beginning-of-defun from C-M-home, since
+    it's on C-M-a already.
+
+need someone to go over the other tutorials (five new ones, from FSF
+21.0.105) and fix them up to correspond to the english one.
+
+shouldn't shift-motion work with C-a and such as well as arrows?
+
+sep 29, 2001:
+
+charcount_to_bytecount can also be made to scream -- as can scan_buffer,
+buffer_mule_signal_inserted_region, others?  we should start profiling
+though before going too far down this line.
+
+Debug code that causes no slowdown should in general remain in the
+executable even in the release version because it may be useful (e.g. for
+people to see the event output).  so DEBUG_XEMACS should be rethought.
+things like use of msvcrtd.dll should be controlled by error_checking on.
+maybe DEBUG_XEMACS controls general debug code (e.g. use of msvcrtd.dll,
+asserts abort, error checking), and the actual debugging code should remain
+always, or be conditonalized on something else
+(e.g. DEBUGGING_FUNS_PRESENT).
+
+doc strings in dumped files are displayed with an extra blank line between
+each line.  presumably this is recent?  i assume either the change to
+detect-coding-region or the double-wrapping mentioned below.
+
+error with coding-system-property on iso-2022-jp-dos.  problem is that that
+coding system is wrapped, so its type shows up as chain, not iso-2022.
+this is a general problem, and i think the way to fix it is to in essence
+do late canonicalization -- similar in spirit to what was done long ago,
+canonicalize_when_code, except that the new coding system (the wrapper) is
+created only once, either when the original cs is created or when first
+needed.  this way, operations on the coding system work like expected, and
+you get the same results as currently when decoding/encoding.  the only
+thing tricky is handling canonicalize-after-coding and the ever-tricky
+double-wrapping problem mentioned below.  i think the proper solution is to
+move the autodetection of eol into the main autodetect type.  it can be
+asked to autodetect eol, coding, or both.  for just coding, it does like it
+currently does.  for just eol, it does similar to what it currently does
+but runs the detection code that convert-eol currently does, and selects
+the appropriate convert-eol system.  when it does both eol and coding, it
+does something on the order of creating two more autodetect coding systems,
+one for eol only and one for coding only, and chains them together.  when
+each has detected the appropriate value, the results are combined.  this
+automatically eliminates the double-wrapping problem, removes the need for
+complicated canonicalize-after-coding stuff in chain, and fixes the problem
+of autodetect not having a seekable stream because hidden inside of a
+chain. (we presume that in the both-eol-and-coding case, the various
+autodetect coding streams can communicate with each other appropriately.)
+
+also, we should solve the problem of internal coding systems floating
+around and clogging up the list simply by having an "internal" property on
+cs's and an internal param to coding-system-list (optional; if not given,
+you don't get the internal ones). [DONE]
+
+we should try to reduce the size of the from-unicode tables (the dominant
+memory hog in the tables).  one obvious thing is to not store a whole
+emchar as the mapped-to value, but a short that encodes the octets. [DONE]
+
+sep 28, 2001:
+
+need to merge up to latest in trunk.
+
+add unicode charsets for all non-translatable unicode chars; probably want
+to extend the concept of charsets to allow for dimension 3 and dimension 4
+charsets.  for the moment we should stick with just dimension 3 charsets;
+otherwise we run past the current maximum of 4 bytes per emchar. (most code
+would work automatically since it uses MAX_EMCHAR_LEN; the trickiness is in
+certain code that has intimate knowledge of the representation.
+e.g. bufpos_to_bytind() has to multiply or divide by 1, 2, 3, or 4,
+and has special ways of handling each number.  with 5 or 6 bytes per char,
+we'd have to change that code in various ways.) 96x96x96 = 884,000 or so,
+so with two 96x96x96 charsets, we could tackle all Unicode values
+representable by UTF-16 and then some -- and only these codepoints will
+ever have assigned chars, as far as we know.  
+
+need an easy way of showing the current language environment.  some menus
+need to have the current one checked or whatever. [DONE]
+
+implement unicode surrogates.
+
+implement buffer-file-coding-system-when-loaded -- make sure find-file,
+revert-file, etc. set the coding system [DONE]
+
+verify all the menu stuff [DONE]
+
+implemented the entirely-ascii check in buffers.  not sure how much gain
+it'll get us as we already have a known range inside of which is constant
+time, and with pure-ascii files the known range spans the whole buffer.
+improved the comment about how bufpos-to-bytind and vice-versa work. [DONE]
+
+fix double-wrapping of convert-eol: when undecided converts itself to
+something with a non-autodetect eol, it needs to tell the adjacent
+convert-eol to reduce itself to nothing.
+
+need menu item for find file with specified encoding. [DONE]
+
+renamed coding systems mswindows-### to windows-### to follow the standard
+in rfc1345. [DONE]
+
+implemented coding-system-subsidiary-parent [DONE]
+HAVE_MULE -> MULE in files in nt/ so that depend checking works [DONE]
+
+need to take the smarter search-all-files-in-dir stuff from my sample init
+file and put it on the grep menu [DONE]
+
+added item for revert w/specified encoding; mostly works, but needs fixes.
+in particular, you get the correct results, but buffer-file-coding-system
+does not reflect things right.  also, there are too many entries.  need to
+split into submenus.  there is already split code out there; see if it's
+generalized and if not make it so.  it should only split when there's more
+than a specified number, and when splitting, split into groups of a
+specified size, not into a specified number of groups. [DONE]
+
+too many entries in the langenv menus; need to split. [DONE]
+
+sep 27, 2001:
+
+NOTE: M-x grep for make-string causes crash now.  something definitely to
+do with string changes.  check very carefully the diffs and put in those
+sledgehammer checks. [DONE]
+
+fix font-lock bug i introduced. [DONE]
+
+added optimization to strings (keeps track of # of bytes of ascii at the
+beginning of a string).  perhaps should also keep an all-ascii flag to deal
+with really large (> 2 MB) strings.  rewrite code to count ascii-begin to
+use the 4-or-8-at-a-time stuff in bytecount_to_charcount.
+
+Error: M-q is causing Invalid Regexp error on the above paragraph.  It's
+not in working.  I assume it's a side effect of the string stuff.  VERIFY!
+Write sledgehammer checks for strings. [DONE]
+
+revamped the locale/init stuff so that it tries much harder to get things
+right.  should test a bit more.  in particular, test out Describe Language
+on the various created environments and make sure everything looks right.
+
+should change the menus: move the submenus on Edit->Mule directly under
+Edit.  add a menu entry on File to say "Reload with specified encoding ->".
+[DONE]
+
+Also Find File with specified encoding -> Also entry to change the EOL
+settings for Unix, and implement it.
+
+decode-coding-region isn't working because it needs to insert a binary
+(char->byte) converter. [DONE]
+
+chain should be rearranged to be in decoding order; similar for
+source/sink-type, other things?
+
+the detector should check for a magic cookie even without a seekable input.
+(currently its input is not seekable, because it's hidden within a chain.
+#### See what we can do about this.)
+
+provide a way to display various settings, e.g. the current category
+mappings and priority (see mule-diag; get this working so it's in the
+path); also a way to print out the likeliness results from a detection,
+perhaps a debug flag.
+
+problem with `env', which causes path issues due to `env' in packages.
+move env code to process, sync with fsf 21.0.105, check that the autoloads
+in `env' don't cause problems. [DONE]
+
+8-bit iso2022 detection appears broken; or at least, mule-canna.c is not so
+detected.
+
+sep 25, 2001:
+
+something else to do is review the font selection and fix it so that (e.g.) 
+JISX-0212 can be displayed.
+
+also, text in widgets needs to be drawn by us so that the correct fonts
+will be displayed even in multi-lingual text.
+
+sep 24, 2001:
+
+the detection system is now properly abstracted.  the detectors have been
+rewritten to include multiple levels of abstraction.  now we just need
+detectors for ascii, binary, and latin-x, as well as more sophisticated
+detectors in general and further review of the general algorithm for doing
+detection. (#### Is this written up anywhere?) after that, consider adding
+error-checking to decoding (VERY IMPORTANT) and verifying the binary
+correctness of things under unix no-mule.
+
+sep 23, 2001:
+
+began to fix the detection system -- adding multiple levels of likelihood
+and properly abstracting the detectors.  the system is in place except for
+the abstraction of the detector-specific data out of the struct
+detection_state.  we should get things working first before tackling that
+(which should not be too hard).  i'm rewriting algorithms here rather than
+just converting code, so it's harder.  mostly done with everything, but i
+need to review all detectors except iso2022 and make them properly follow
+the new way.  also write a no-conversion detector.  also need to look into
+the `recode' package and see how (if?) they handle detection, and maybe
+copy some of the algorithms.  also look at recent FSF 21.0 and see if their
+algorithms have improved.
+
+sep 22, 2001:
+
+fixed gc bugs from yesterday.
+fixed truename bug.
+close/finalize stuff works.
+eliminated notyet stuff in syswindows.h.
+eliminated special code in tstr_to_c_string.
+fixed pdump problems. (many of them, mostly latent bugs, ugh)
+fixed cygwin sscanf problems in parse-unicode-translation-table. (NOT a
+sscanf bug, but subtly different behavior w.r.t. whitespace in the format
+string, combined with a debugger that sucks ROCKS!! and consistently
+outputs garbage for variable values.)
+main stuff to test is the handling of EOF recognition vs. binary
+(i.e. check what the default settings are under Unix).  then we may have
+something that WORKS on all platforms!!!  (Also need to test Windows
+non-Mule)
+
+sep 21, 2001:
+
+finished redoing the close/finalize stuff in the lstream code.  but i
+encountered again the nasty bug mentioned on sep 15 that disappeared on its
+own then.  the problem seems to be that the finalize method of some of the
+lstreams is calling Lstream_delete(), which calls free_managed_lcrecord(),
+which is a no-no when we're inside of garbage-collection and the object
+passed to free_managed_lcrecord() is unmarked, and about to be released by
+the gc mechanism -- the free lists will end up with xfree()d objects on
+them, which is very bad.  we need to modify free_managed_lcrecord() to
+check if we're in gc and the object is unmarked, and ignore it rather than
+move it to the free list. [DONE]
+
+(#### What we really need to do is do what Java and C# do w.r.t. their
+finalize methods: For objects with finalizers, when they're about to be
+freed, leave them marked, run the finalizer, and set another bit on them
+indicating that the finalizer has run.  Next GC cycle, the objects will
+again come up for freeing, and this time the sweeper notices that the
+finalize method has already been called, and frees them for good (provided
+that a finalize method didn't do something to make the object alive
+again).)
+
+sep 20, 2001:
+
+redid the lstream code so there is only one coding stream.  combined the
+various doubled coding stream methods into one; i'm a little bit unsure of
+this last part, though, as the results of combining the two together seem
+unclean.  got it to compile, but it crashes in loadup.  need to go through
+and rehash the close vs. finalize stuff, as the problem was stuff getting
+freed too quickly, before the canonicalize-after-decoding was run.  should
+eliminate entirely CODING_STATE_END and use a different method (close
+coding stream).  rewrite to use these two.  make sure they're called in the
+right places.  Lstream_close on a stream should *NOT* do finalizing.
+finalize only on delete. [DONE]
+
+in general i'd like to see the flags eliminated and converted to
+bit-fields.  also, rewriting the methods to take advantage of rejecting
+should make it possible to eliminate much of the state in the various
+methods, esp. including the flags.  need to test this is working, though --
+reduce the buffer size down very low and try files with only CRLF's in
+them, with one offset by a byte from the other, and see if we correctly
+handle rejection.
+
+still have the problem with incorrectly truenaming files.
+
+
+sep 19, 2001:
+
+bug reported: crash while closing lstreams.
+
+the lstream/coding system close code needs revamping.  we need to document
+that order of closing lstreams is very important, and make sure we're
+consistent.  furthermore, chain and undecided lstreams need to close their
+underneath lstreams when they receive the EOF signal (there may be data in
+the underneath streams waiting to come out), not when they themselves are
+closed. [DONE]
+
+(if only we had proper inheritance.  i think in any case we should
+simulate it for the chain coding stream -- write things in such a way that
+undecided can use the chain coding stream and not have to duplicate
+anything itself.)
+
+in general we need to carefully think through the closing process to make
+sure everything always works correctly and in the right order.  also check
+very carefully to make sure there are no dangling pointers to deleted
+objects floating around.
+
+move the docs for the lstream functions to the functions themselves, not
+the header files.  document more carefully what exactly Lstream_delete()
+means and how it's used, what the connections are between Lstream_close(),
+Lstream_delete(), Lstream_flush(), lstream_finalize, etc. [DONE]
+
+additional error-checking: consider deadbeefing the memory in objects
+stored in lcrecord free lists; furthermore, consider whether lifo or fifo
+is correct; under error-checking, we should perhaps be doing fifo, and
+setting a minimum number of objects on the lists that's quite large so that
+it's highly likely that any erroneous accesses to freed objects will go
+into such deadbeefed memory and cause crashes.  also, at the earliest
+available opportunity, go through all freed memory and check for any
+consistency failures (overwrites of the deadbeef), crashing if so.  perhaps
+we could have some sort of id for each block, to easier trace where the
+offending block came from. (all of these ideas are present in the debug
+system malloc from VC++, plus more stuff.) there's similar code i wrote
+sitting somewhere (in free-hook.c? doesn't appear so. we need to delete the
+blocking stuff out of there!).  also look into using the debug system
+malloc from VC++, which has lots of cool stuff in it. we even have the
+sources.  that means compiling under pdump, which would be a good idea
+anyway.  set it as the default. (but then, we need to remove the
+requirement that Xpm be a DLL, which is extremely annoying.  look into
+this.)
+
+test the windows code page coding systems recently created.
+
+problems reading my mail files -- 1personal appears to hang, others come up
+with lots of ^M's.  investigate.
+
+test the enum functions i just wrote, and finish them.
+
+still pdump problems.
+
+sep 18, 2001:
+
+critical-quit broken sometime after aug 25.
+
+-- fixed critical quit.
+-- fixed process problems.
+-- print routines work. (no routine for ccl, though)
+-- can read and write unicode files, and they can still be read by some
+   other program
+-- defaults should come up correctly -- mswindows-multibyte is general.
+
+still need to test matej's stuff.
+seems ok with multibyte stuff but needs more testing.
+
+sep 17, 2001:
+
+!!!!! something broken with processes !!!!! cannot send mail anymore.  must
+investigate.
+
+sep 17, 2001:
+
+on mon/wed nights, stop *BEFORE* 11pm.  Otherwise i just start getting
+woozy and can't concentrate.
+
+just finished getting assorted fixups to the main branch committed, so it
+will compile under C++ (Andy committed some code that broke C++ builds).
+cup'd the code into the fixtypes workspace, updated the tags appropriately.
+i've created the appropriate log message, sitting in fixtypes.txt in
+/src/xemacs; perhaps it should go into a README.  now i just have to build
+on everything (it's currently building), verify it's ok, run patcher-mail,
+commit, send.
+
+my mule ws is also very close.  need to:
+
+-- test the new print routines.
+-- test it can read and write unicode files, and they can still be read by
+   some other program.
+-- try to see if unicode can be auto-detected properly.
+-- test it can read and write multibyte files in a few different formats.
+   currently can't recognize them, but if you set the cs right, it should
+   work.
+-- examine the test files sent by matej and see if we can handle them.
+
+sep 15, 2001:
+
+more eol fixing.  this stuff is utter crap.
+
+currently we wrap coding systems with convert-eol-autodetect when we create
+them in make_coding_system_1.  i had a feeling that this would be a
+problem, and indeed it is -- when autodetecting with `undecided', for
+example, we end up with multiple layers of eol conversion.  to avoid this,
+we need to do the eol wrapping *ONLY* when we actually retrieve a coding
+system in places such as insert-file-contents.  these places are
+insert-file-contents, load, process input, call-process-internal,
+encode/decode/detect-coding-region, database input, ...
+
+(later) it's fixed, and things basically work.  NOTE: for some reason,
+adding code to wrap coding systems with convert-eol-lf when eol-type == lf
+results in crashing during garbage collection in some pretty obscure place
+-- an lstream is free when it shouldn't be.  this is a bad sign.  i guess
+something might be getting initialized too early?
+
+we still need to fix the canonicalization-after-decoding code to avoid
+problems with coding systems like `internal-7' showing up.  basically, when
+eol==lf is detected, nil should be returned, and the callers should handle
+it appropriately, eliding when necessary.  chain needs to recognize when
+it's got only one (or even 0) items in the chain, and elide out the chain.
+
+sep 11, 2001: the day that will live in infamy.
+
+rewrite of sep 9 entry about formats:
+
+when calling make-coding-system, the name can be a cons of (format1 .
+format2), specifying that it decodes format1->format2 and encodes the other
+way.  if only one name is given, that is assumed to be format1, and the
+other is either `external' or `internal' depending on the end type.
+normally the user when decoding gives the decoding order in formats, but
+can leave off the last one, `internal', which is assumed.  a multichain
+might look like gzip|multibyte|unicode, using the coding systems named
+`gzip', `(unicode . multibyte)' and `unicode'.  the way this actually works
+is by searching for gzip->multibyte; if not found, look for gzip->external
+or gzip->internal. (In general we automatically do conversion between
+internal and external as necessary: thus gzip|crlf does the expected, and
+maps to gzip->external, external->internal, crlf->internal, which when
+fully specified would be gzip|external:external|internal:crlf|internal --
+see below.)  To forcibly fit together two converters that have explicitly
+specified and incompatible names (say you have unicode->multibyte and
+iso8859-1->ebcdic and you know that the multibyte and iso8859-1 in this
+case are compatible), you can force-cast using :, like this:
+ebcdic|iso8859-1:multibyte|unicode. (again, if you force-cast between
+internal and external formats, the conversion happens automatically.)
+
+
+sep 10, 2001:
+
+moved the autodetection stuff (both codesys and eol) into particular coding
+systems -- `undecided' and `convert-eol' (type == `autodetect').  needs
+lots of work.  still need to search through the rest of the code and find
+any remaining auto-detect code and move it into the undecided coding
+system.  need to modify make-coding-system so that it spits out
+auto-detecting versions of all text-file coding systems unless we say not
+to.  need eliminate entirely the EOF flag from both the stream info and the
+coding system; have only the original-eof flag.  in
+coding_system_from_mask, need to check that the returned value is not of
+type `undecided', falling back to no-conversion if so.  also need to make
+sure we wrap everything appropriate for text-files -- i removed the
+wrapping on set-coding-category-list or whatever (need to check all those
+files to make sure all wrapping is removed).  need to review carefully the
+new code in `undecided' to make sure it works are preserves the same logic
+as previously.  need to review the closing and rewinding behavior of chain
+and undecided (same -- should really consolidate into helper routines, so
+that any coding system can embed a chain in it) -- make sure the dynarr's
+are getting their data flushed out as necessary, rewound/closed in the
+right order, no missing steps, etc.
+
+also split out mule stuff into mule-coding.c.  work done on
+configure/xemacs.mak/Makefiles not done yet.  work on emacs.c/symsinit.h to
+interface with the new init functions not done yet.
+
+also put in a few declarations of the way i think the abstracted detection
+stuff ought to go.  DON'T WORK ON THIS MORE UNTIL THE REST IS DEALT WITH
+AND WE HAVE A WORKING XEMACS AGAIN WITH ALL EOL ISSUES NAILED.
+
+really need a version of cvs-mods that reports only the current directory.
+WRITE THIS!  use it to implement a better cvs-checkin.
+
+sep 9, 2001:
+
+implemented a gzip coding system.  unfortunately, doesn't quite work right
+because it doesn't handle the gzip headers -- it just reads and writes raw
+zlib data.  there's no function in the library to skip past the header, but
+we do have some code out of the library that we can snarf that implements
+header parsing.  we need to snarf that, store it, and output it again at
+the beginning when encoding.  in the process, we should create a "get next
+byte" macro that bails out when there are no more.  using this, we set up a
+nice way of doing most stuff statelessly -- if we have to bail, we reject
+everything back to the sync point.  also need to fix up the autodetection
+of zlib in configure.in.
+
+BIG problems with eol.  finished up everything i thought i would need to
+get eol stuff working, but no -- when you have mswindows-unicode, with its
+eol set to autodetect, the detection routines themselves do the autodetect
+(first), and fail (they report CR on CRLF because of the NULL byte between
+the CR and the LF) since they're not looking at ascii data.  with a chain
+it's similarly bad. for mswindows-multibyte, for example, which is a chain
+unicode->unicode-to-multibyte, autodetection happens inside of the chain,
+both when unicode and unicode-to-multibyte are active.  we could twiddle
+around with the eol flags to try to deal with this, but it's gonna be a big
+mess, which is exactly what we're trying to avoid.  what we basically want
+is to entirely rip out all EOL settings from either the coding system or
+the stream (yes, there are two!  one might saw autodetect, and then the
+stream contains the actual detected value).  instead, we simply create an
+eol-autodetect coding system -- or rather, it's part of the convert-eol
+coding system.  convert-eol, type = autodetect, does autodetection the
+first time it gets data sent to it to decode, and thereafter sets a stream
+parameter indicating the actual eol type for this stream.  this means that
+all autodetect coding systems, as created by `make-coding-system', really
+are chains with a convert-eol at the beginning.  only subsidiary xxx-unix
+has no wrapping at all.  this should allow eof detection of gzip, unicode,
+etc.  for that matter, general autodetection should be entirely
+encapsulated inside of the `autodetect' coding system, with no
+eol-autodetection -- the chain becomes convert-eol (autodetect) ->
+autodetect or perhaps backwards.  the generic autodetect similarly has a
+coding-system in its stream methods, and needs somehow or other to insert
+the detected coding-system into the chain.  either it contains a chain
+inside of it (perhaps it *IS* a chain), or there's some magic involving
+canonicalization-type switcherooing in the middle of a decode.  either way,
+once everything is good and done and we want to save the coding system so
+it can be used later, we need to do another sort of canonicalization --
+converting auto-detect-type coding systems into the detected systems.
+again, a coding-system method, with some magic currently so that
+subsidiaries get properly used rather than something that's new but
+equivalent to subsidiaries. (#### perhaps we could use a hash table to
+avoid recreating coding systems when not necessary.  but that would require
+that coding systems be immutable from external, and i'm not sure that's the
+case.)
+
+i really think, after all, that i should reverse the naming of everything
+in chain and source-sink-type -- they should be decoding-centric.  later
+on, if/when we come up with the proper way to make it totally symmetrical,
+we'll be fine whether before then we were encoding or decoding centric.
+
+
+sep 9, 2001:
+
+investigated eol parameter.
+implemented handling in make-coding-system of eol-cr and eol-crlf.
+fixed calls everywhere to Fget_coding_system / Ffind_coding_system to
+reject non-char->byte coding systems.
+
+still need to handle "query eol type using coding-system-property" so it
+magically returns the right type by parsing the chain.
+
+no work done on formats, as mentioned below.  we should consider using :
+instead of || to indicate casting.
+
+early sep 9, 2001:
+
+renamed some codesys properties: `list' in chain -> chain; `subtype' in
+unicode -> type.  everything compiles again and sort of works; some CRLF
+problems that may resolve themselves when i finish the convert-eol stuff.
+the stuff to create subsidiaries has been rewritten to use chains; but i
+still need to investigate how the EOL type parameter is used.  also, still
+need to implement this: when a coding system is created, and its eol type
+is not autodetect or lf, a chain needs to be created and returned.  i think
+that what needs to happen is that the eol type can only be set to
+autodetect or lf; later on this should be changed to simply be either
+autodetect or not (but that would require ripping out the eol converting
+stuff in the various coding systems), and eventually we will do the work on
+the detection mechanism so it can do chain detection; then we won't need an
+eol autodetect setting at all.  i think there's a way to query the eol type
+of a coding system; this should check to see if the coding system is a
+chain and there's a convert-eol at the front; if so, the eol type comes
+from the type of the convert-eol.
+
+also check out everywhere that Fget_coding_system or Ffind_coding_system is
+called, and see whether anything but a char->byte system can be tolerated.
+create a new function for all the places that only want char->byte,
+something like get_coding_system_char_to_byte_only.
+
+think about specifying formats in make-coding-system.  perhaps the name can
+be a cons of (format1, format2), specifying that it encodes
+format1->format2 and decodes the other way.  if only one name is given,
+that is assumed to be format2, and the other is either `byte' or `char'
+depending on the end type.  normally the user when decoding gives the
+decoding order in formats, but can leave off the last one, `char', which is
+assumed.  perhaps we should say `internal' instead of `char' and `external'
+instead of byte.  a multichain might look like gzip|multibyte|unicode,
+using the coding systems named `gzip', `(unicode . multibyte)' and
+`unicode'.  we would have to allow something where one format is given only
+as generic byte/char or internal/external to fit with any of the same
+byte/char type.  when forcibly fitting together two converters that have
+explicitly specified and incompatible names (say you have
+unicode->multibyte and iso8859-1->ebcdic and you know that the multibyte
+and iso8859-1 in this case are compatible), you can force-cast using ||,
+like this: ebcdic|iso8859-1||multibyte|unicode.  this will also force
+external->internal translation as necessary:
+unicode|multibyte||crlf|internal does unicode->multibyte,
+external->internal, crlf->internal.  perhaps you'd need to put in the
+internal translation, like this: unicode|multibyte|internal||crlf|internal,
+which means unicode->multibyte, external->internal (multibyte is compatible
+with external); force-cast to crlf format and convert crlf->internal.
+
+even later: Sep 8, 2001:
+
+chain doesn't need to set character mode, that happens automatically when
+the coding systems are created.  fixed chain to return correct source/sink
+type for itself and to check the compatibility of source/sink types in its
+chain.  fixed decode/encode-coding-region to check the source and sink
+types of the coding system performing the conversion and insert appropriate
+byte->char/char->byte converters (aka "binary" coding system).  fixed
+set-coding-category-system to only accept the traditional
+encode-char-to-byte types of coding systems.
+
+still need to extend chain to specify the parameters mentioned below,
+esp. "reverse".  also need to extend the print mechanism for chain so it
+prints out the chain.  probably this should be general: have a new method
+to return all properties, and output those properties.  you could also
+implement a read syntax for coding systems this way.
+
+still need to implement convert-eol and finish up the rest of the eol stuff
+mentioned below.
+
+later September 7, 2001: (more like Sep 8)
+
+moved many Lisp_Coding_System * params to Lisp_Object.  In general this is
+the way to go, and if we ever implement a copying GC, we will never want to
+be passing direct pointers around.  With no error-checking, we lose no
+cycles using Lisp_Objects in place of pointers -- the Lisp_Object itself is
+nothing but a pointer, and so all the casts and "dereferences" boil down to
+nothing.
+
+Clarified and cleaned up the "character mode" on streams, and documented
+who (caller or object itself) has the right to be setting character mode on
+a stream, depending on whether it's a read or write stream.  changed
+conversion_end_type method and enum source_sink_type to return
+encoding-centric values, rather than decoding-centric.  for the moment,
+we're going to be entirely encoding-centric in everything; we can rethink
+later.  fixed coding systems so that the decode and encode methods are
+guaranteed to receive only full characters, if that's the source type of
+the data, as per conversion_end_type.
+
+still need to fix the chain method so that it correctly sets the character
+mode on all the lstreams in it and checks the source/sink types to be
+compatible.  also fix decode-coding-string and friends to put the
+appropriate byte->character (i.e. no-conversion) coding systems on the ends
+as necessary so that the final ends are both character.  also add to chain
+a parameter giving the ability to switch the direction of conversion of any
+particular item in the chain (i.e. swap encoding and decoding).  i think
+what we really want to do is allow for arbitrary parameters to be put onto
+a particular coding system in the chain, of which the only one so far is
+swap-encode-decode.  don't need too much codage here for that, but make the
+design extendable.
+
+
+
+September 7, 2001:
+
+just added a return value from the decode and encode methods of a coding
+system, so that some of the data can get rejected.  fixed the calling
+routines to handle this.  need to investigate when and whether the coding
+lstream is set to character mode, so that the decode/encode methods only
+get whole characters.  if not, we should do so, according to the source
+type of these methods.  also need to implement the convert_eol coding
+system, and fix the subsidiary coding systems (and in general, any coding
+system where the eol type is specified and is not LF) to be chains
+involving convert_eol.
+
+after everything is working, need to remove eol handling from encode/decode
+methods and eventually consider rewriting (simplifying) them given the
+reject ability.
+
+September 5, 2001:
+
+-- need to organize this.  get everything below into the TODO list.
+   CVS the TODO list frequently so i can delete old stuff.  prioritize
+   it!!!!!!!!!
+
+-- move README.ben-mule... to STATUS.ben-mule...; use README for
+   intro, overview of what's new, what's broken, how to use the
+   features, etc.
+
+-- need a global and local coding-category-precedence list, which get
+   merged.
+
+-- finished the BOM support.  also finished something not listed
+   below, expansion to the auto-generator of Unicode-encapsulation to
+   support bracketing code with #if ... #endif, for Cygwin and MINGW
+   problems, e.g.  This is tested; appears to work.
+
+-- need to add more multibyte coding systems now that we have various
+   properties to specify them.  need to add DEFUN's for mac-code-page
+   and ebcdic-code-page for completeness.  need to rethink the whole
+   way that the priority list works.  it will continue to be total
+   junk until multiple levels of likeliness get implemented.
+
+-- need to finish up the stuff about the various defaults. [need to
+   investigate more generally where all the different default values
+   are that control encoding. (there are six places or so.) need to
+   list them in make-coding-system docs and put pointers
+   elsewhere. [[[[#### what interface to specify that this default
+   should be unicode?  a "Unicode" language environment seems too
+   drastic, as the language environment controls much more.]]]] even
+   skipping the Unicode stuff here, we need to survey and list the
+   variables that control coding page behavior and determine how they
+   need to be set for various possible scenarios:
+
+  -- total binary: no detection at all.
+  -- raw-text only: wants only autodetection of line endings, nothing else.
+  -- "standard Windows environment": tries for Unicode, falls back on
+     code page encoding.
+  -- some sort of East European environment, and Russian.
+  -- some sort of standard Japanese Windows environment.
+  -- standard Chinese Windows environments (traditional and simplified)
+  -- various Unix environments (European, Japanese, Russian, etc.)
+  -- Unicode support in all of these when it's reasonable
+
+These really require multiple likelihood levels to be fully
+implementable.  We should see what can be done ("gracefully fall
+back") with single likelihood level.  need lots of testing.
+
+-- need to fix the truename problem.
+
+-- lots of testing: need to test all of the stuff above and below that's recently been implemented.
+
+
+
+September 4, 2001:
+
+mostly everything compiles.  currently there is a crash in
+parse-unicode-translation-table, and Cygwin/Mule won't run.  it may
+well be a bug in the sscanf() in Cygwin.
+
+working on today:
+
+-- adding BOM support for Unicode coding systems.  mostly there, but
+   need to finish adding BOM support to the detection routines.  then test.
+-- adding properties to unicode-to-multibyte to specify the coding
+   system in various flexible ways, e.g. directly specified code page
+   or ansi or oem code page of specified locale, current locale,
+   user-default or system-default locale.  need to test.
+-- creating a `multibyte' coding system, with the same parameters as
+   unicode-to-multibyte and which resolves at coding-system-creation
+   time to the appropriate chain.  creating the underlying mechanism
+   to allow such under-the-scenes switcheroo.  need to test.
+-- set default-value of buffer-file-coding-system to
+   mswindows-multibyte, as Matej said it should be.  need to test.
+   need to investigate more generally where all the different default
+   values are that control encoding. (there are six places or so.) 
+   need to list them in make-coding-system docs and put pointers
+   elsewhere. #### what interface to specify that this default should
+   be unicode?  a "Unicode" language environment seems too drastic, as
+   the language environment controls much more.
+-- thinking about adding multiple levels of certainty to the detection
+   schemes, instead of just a mask.  eventually, we need to totally
+   abstract things, but that can easier be done in many steps. (we
+   need multiple levels of likelihood to more reasonably support a
+   Windows environment with code-page type files.  currently, in order
+   to get them detected, we have to put them first, because they can
+   look like lots of other things; but then, other encodings don't get
+   detected.  with multiple levels of likelihood, we still put the
+   code-page categories first, but they will return low levels of
+   likelihood.  Lower-down encodings may be able to return higher
+   levels of likelihood, and will get taken preferentially.)
+-- making it so you cannot disable file-coding, but you get an
+   equivalent default on Unix non-Mule systems where all defaults are
+   `binary'.  need to test!!!!!!!!!
+
+Matej (mostly, + some others) notes the following problems, and here
+are possible solutions:
+
+-- he wants the defaults to work right. [figure out what those
+   defaults are.  i presume they are auto-detection of data in current
+   code page and in unicode, and new files have current code page set
+   as their output encoding.]
+
+-- too easy to lose data with incorrect encodings. [need to set up an
+   error system for encoding/decoding.  extremely important but a
+   little tricky to implement so let's deal with other issues now.]
+
+-- EOL isn't always detected correctly. [#### ?? need examples]
+
+-- truename isn't working: c:\t.txt and c:\tmp.txt have the same truename.
+   [should be easy to fix]
+
+-- unicode files lose the BOM mark. [working on this]
+
+-- command-line utilities use OEM. [actually it seems more
+   complicated.  it seems they use the codepage of the console.  we
+   may be able to set that, e.g. to UTF8, before we invoke a command.
+   need to investigate.]
+
+-- no way to handle unicode characters not recognized as charsets. [we
+   need to create something like 8 private 2-dimensional charsets to
+   handle all BMP Unicode chars.  Obviously this is a stopgap
+   solution.  Switching to Unicode internal will ultimately make life
+   far easier and remove the BMP limitation.  but for now it will
+   work.  we translate all characters where we have charsets into
+   chars in those charsets, and the remainder in a unicode charset.
+   that way we can save them out again and guarantee no data loss with
+   unicode.  this creates font problems, though ...]
+
+-- problems with xemacs font handling. [xemacs font handling is not
+   sophisticated enough.  it goes on a charset granularity basis and
+   only looks for a font whose name contains the corresponding windows
+   charset in it.  with unicode this fails in various ways.  for one
+   the granularity needs to be single character, so that those unicode
+   charsets mentioned above work; and it needs to query the font to
+   see what unicode ranges it supports, rather than just looking at
+   the charset ending.]
+
+
+
+August 28, 2001:
+
+working on getting everything to compile again: Cygwin, non-MULE,
+pdump.  not there yet.
+
+mswindows-multibyte is now defined using chain, and works.  removed
+most vestiges of the mswindows-multibyte coding system type.
+
+file-coding is on by default; should default to binary only on Unix.
+Need to test. (Needs to compile first :-)
+
+August 26, 2001:
+
+I've fixed the issue of inputting non-ASCII text under -nuni, and done
+some of the work on the Russian C-x problem -- we now compute the
+other possibilities.  We still need to fix the key-lookup code,
+though, and that code is unfortunately a bit ugly.  the best way, it
+seems, is to expand the command-builder structure so you can specify
+different interpretations for keys. (if we do find an alternative
+binding, though, we need to mess with both the command builder and
+this-command-keys, as does the function-key stuff.  probably need to
+abstract that munging code.)
+
+high-priority:
+
+[currently doing]
+
+-- support for WM_IME_CHAR.  IME input can work under -nuni if we use
+   WM_IME_CHAR.  probably we should always be using this, instead of
+   snarfing input using WM_COMPOSITION.  i'll check this out.
+-- Russian C-x problem.  see above.
+
+[clean-up]
+
+-- make sure it compiles and runs under non-mule.  remember that some
+   code needs the unicode support, or at least a simple version of it.
+-- make sure it compiles and runs under pdump.  see below.
+-- clean up mswindows-multibyte, TSTR_TO_C_STRING.  see below. [DONE]
+-- eliminate last vestiges of codepage<->charset conversion and similar stuff.
+
+[other]
+-- cut and paste.  see below.
+-- misc issues with handling lang environments.  see also August 25,
+   "finally: working on the C-x in ...".
+   -- when switching lang env, needs to set keyboard layout.
+   -- user var to control whether, when moving into text of a
+      particular language, we set the appropriate keyboard layout.  we
+      would need to have a lisp api for retrieving and setting the
+      keyboard layout, set text properties to indicate the layout of
+      text, and have a way of dealing with text with no property on
+      it. (e.g. saved text has no text properties on it.) basically,
+      we need to get a keyboard layout from a charset; getting a
+      language would do.  Perhaps we need a table that maps charsets
+      to language environments.
+   -- test that the lang env is properly set at startup.  test that
+      switching the lang env properly sets the C locale (call
+      setlocale(), set LANG, etc.) -- a spawned subprogram should have
+      the new locale in its environment.
+-- look through everything below and see if anything is missed in this
+   priority list, and if so add it.  create a separate file for the
+   priority list, so it can be updated as appropriate.
+
+
+mid-priority:
+
+-- clean up the chain coding system.  its list should specify decode
+   order, not encode; i now think this way is more logical.  it should
+   check the endpoints to make sure they make sense.  it should also
+   allow for the specification of "reverse-direction coding systems":
+   use the specified coding system, but invert the sense of decode and
+   encode.
+
+-- along with that, places that take an arbitrary coding system and
+   expect the ends to be anything specific need to check this, and add
+   the appropriate conversions from byte->char or char->byte.
+
+-- get some support for arabic, thai, vietnamese, japanese jisx 0212:
+   at least get the unicode information in place and make sure we have
+   things tied together so that we can display them.  worry about r2l
+   some other time.
+
+August 25, 2001:
+
+There is actually more non-Unicode-ized stuff, but it's basically
+inconsequential. (See previous note.) You can check using the file
+nmkun.txt (#### RENAME), which is just a list of all the routines that
+have been split. (It was generated from the output of `nmake
+unicode-encapsulate', after removing everything from the output but
+the function names.) Use something like
+
+fgrep -f ../nmkun.txt -w [a-hj-z]*.[ch]  |m
+
+in the source directory, which does a word match and skips
+intl-unicode-win32.[ch] and intl-win32.[ch], which have a whole lot of
+references to these, unavoidably.  It effectively detects what needs
+to be changed because changed versions either begin qxe... or end with
+A or W, and in each case there's no whole-word match.
+
+The nasty bug has been fixed below.  The -nuni option now works -- all
+specially-written code to handle the encapsulation has been tested by
+some operation (fonts by loadup and checking the output of (list-fonts
+""); devmode by printing; dragdrop tests other stuff).
+
+NOTE: for -nuni (Win 95), areas need work:
+
+-- cut and paste.  we should be able to receive Unicode text if it's
+   there, and we should be able to receive it even in Win 95 or -nuni.
+   we should just check in all circumstances.  also, under 95, when we
+   put some text in the clipboard, it may or may not also be
+   automatically enumerated as unicode.  we need to test this out
+   and/or just go ahead and manually do the unicode enumeration.
+
+-- receiving keyboard input.  we get only a single byte, but we should
+   be able to correlate the language of the keyboard layout to a
+   particular code page, so we can then decode it correctly.
+
+-- mswindows-multibyte.  still implemented as its own thing.  should
+   be done as a chain of (encoding) unicode | unicode-to-multibyte.
+   need to turn this on, get it working, and look into optimizations
+   in the dfc stuff. (#### perhaps there's a general way to do these
+   optimizations???  something like having a method on a coding system
+   that can specify whether a pure-ASCII string gets rendered as
+   pure-ASCII bytes and vice-versa.)
+
+
+ALSO:
+
+-- we have special macros TSTR_TO_C_STRING and such because formerly
+   the DFC macros didn't know about external stuff that was Unicode
+   encoded and would call strlen() on them.  this is fixed, so now we
+   should undo the special macros, make em normal, removal the
+   comments about this, and make sure it works. [DONE]
+
+
+-- finally: working on the C-x in Russian key layout problem.  in the
+   process will probably end up doing work on cleaning up the handling
+   of keyboard layouts, integrating or deleting the FSF stuff, adding
+   code to change the keyboard layout as we move in and out of text in
+   different languages (implemented as a post-command-hook; we need
+   something like internal-post-command-hook if not already there, for
+   internal stuff that doesn't want to get mixed up with the regular
+   post-command-hook; similar for pre-command-hook).  also, when
+   langenv changes, ways to set the keyboard layout appropriately.
+
+-- i think the stuff above is higher priority than the other stuff
+   mentioned below.  what i'm aiming for is to be able to input and
+   work with multiple languages without weird glitches, both under 95
+   and NT.  the problems above are all basic impediments to such work.
+   we assume for the moment that the user can make use of the existing
+   file i/o conversion stuff, and put that lower in priority, after
+   the basic input is working.
+
+-- i should get my modem connected and write up what's going on and
+   send it to the lists; also cvs commit my workspaces and get more
+   testers.
+
+August 24, 2001:
+
+All code has been Unicode-ized except for some stuff in console-msw.c
+that deals with console output.  Much of the Unicode-encapsulation
+stuff, particularly the hand-written stuff, really needs testing.  I
+added a new command-line option, -nuni, to force use of all ANSI calls
+-- XE_UNICODEP evaluates to false in this case.
+
+There is a nasty bug that appeared recently, probably when the event
+code got Unicode-ized -- bad interactions with OS sticky modifiers.
+Hold the shift key down and release it, then instead of affecting the
+next char only, it gets permanently stuck on (until you do a regular
+shift+char stroke).  This needs to be debugged.
+
+Other things on agenda:
+
+-- go through and prioritize what's listed below.
+
+-- make sure the pdump code can compile and work.  for the moment we
+   just don't try to dump any Unicode tables and load them up each
+   time.  this is certainly fast but ...
+
+-- there's the problem that XEmacs can't be run in a directory with
+   non-ASCII/Latin-1 chars in it, since it will be doing Unicode
+   processing before we've had a chance to load the tables.  In fact,
+   even finding the tables in such a situation is problematic using
+   the normal commands.  my idea is to eventually load the stuff
+   extremely extremely early, at the same time as the pdump data gets
+   loaded.  in fact, the unicode table data (stored in an efficient
+   binary format) can even be stuck into the pdump file (which would
+   mean as a resource to the executable, for windows).  we'd need to
+   extend pdump a bit: to allow for attaching extra data to the pdump
+   file. (something like pdump_attach_extra_data (addr, length)
+   returns a number of some sort, an index into the file, which you
+   can then retrieve with pdump_load_extra_data(), which returns an
+   addr (mmap()ed or loaded), and later you pdump_unload_extra_data()
+   when finished.  we'd probably also need
+   pdump_attach_extra_data_append(), which appends data to the data
+   just written out with pdump_attach_extra_data().  this way,
+   multiple tables in memory can be written out into one contiguous
+   table. (we'd use the tar-like trick of allowing new blocks to be
+   written without going back to change the old blocks -- we just rely
+   on the end of file/end of memory.) this same mechanism could be
+   extracted out of pdump and used to handle the non-pdump situation
+   (or alternatively, we could just dump either the memory image of
+   the tables themselves or the compressed binary version).  in the
+   case of extra unicode tables not known about at compile time that
+   get loaded before dumping, we either just dump them into the image
+   (pdump and all) or extract them into the compressed binary format,
+   free the original tables, and treat them like all other tables.
+
+-- `C-x b' when using a Russian keyboard layout.  XEmacs currently
+   tries to interpret C+cyrillic char, which causes an error.  We want
+   C-x b to still work even when the keyboard normally generates
+   Cyrillic.  What we should do is expand the keyboard event structure
+   so that it contains not only the actual char, but what the char
+   would have been in various other keyboard layouts, and in contexts
+   where only certain keystrokes make sense (creating control chars,
+   and looking up in keymaps), we proceed in order, processing each of
+   them until we get something.  order should be something like:
+   current keyboard layout; layout of the current language
+   environment; layout of the user's default language; layout of the
+   system default language; layout of US English.
+
+-- reading and writing Unicode files.  multiple problems:
+
+   -- EOL's aren't handled right.  for the moment, just fix the
+      Unicode coding systems; later on, create EOL-only coding
+      systems:
+
+      1. they would be character->character and operate next to the
+         internal data; this means that coding systems need to be able
+         to handle ends of lines that are either CR, LF, or CRLF.
+         usually this isn't a problem, as they are just characters
+         like any other and get encoded appropriately.  however,
+         coding systems that are line-oriented need to recognize any
+         of the three as line endings.
+
+      2. we'd also have to complete the stuff that handles coding
+         systems where either end can be byte or char (four
+         possibilities total; use a single enum such as
+         ENCODES_CHAR_TO_BYTE, ENCODES_BYTE_TO_BYTE, etc.).
+
+      3. we'd need ways of specifying the chaining of coding systems.
+         e.g. when reading a coding system, a user can specify more
+         than one with a | symbol between them.  when a context calls
+         for a coding system and a chain is needed, the `chain' coding
+         system is useful; but we should really expand the contexts
+         where a list of coding systems can be given, and whenever
+         possible try to inline the chain instead of using a
+         surrounding `chain' coding system.
+
+      4. the `chain' needs some work so that it passes all sorts of
+         lstream commands down to the chain inside it -- it should be
+         entirely transparent and the fact that there's actually a
+         surrounding coding system should be invisible.  more general
+         coding system methods might need to be created.
+
+      5. important: we need a way of specifying how detecting works
+         when we have more than one coding system.  we might need more
+         than a single priority list.  need to think about this.
+
+   -- Unicode files beginning with the BOM are not recognized as such.
+      we need to fix this; but to make things sensible, we really need
+      to add the idea of different levels of confidence regarding
+      what's detected.  otherwise, Unicode says "yes this is me" but
+      others higher up do too.  in the process we should probably
+      finish abstracting the detection system and fix up some
+      stupidities in it.
+
+   -- When writing a file, we need error detection; otherwise somebody
+      will create a Unicode file without realizing the coding system
+      of the buffer is Raw, and then lose all the non-ASCII/Latin-1
+      text when it's written out.  We need two levels
+
+      1. first, a "safe-charset" level that checks before any actual
+         encoding to see if all characters in the document can safely
+         be represented using the given coding system.  FSF has a
+         "safe-charset" property of coding systems, but it's stupid
+         because this information can be automatically derived from
+         the coding system, at least the vast majority of the time.
+         What we need is some sort of
+         alternative-coding-system-precedence-list, langenv-specific,
+         where everything on it can be checked for safe charsets and
+         then the user given a list of possibilities.  When the user
+         does "save with specified encoding", they should see the same
+         precedence list.  Again like with other precedence lists,
+         there's also a global one, and presumably all coding systems
+         not on other list get appended to the end (and perhaps not
+         checked at all when doing safe-checking?).  safe-checking
+         should work something like this: compile a list of all
+         charsets used in the buffer, along with a count of chars
+         used.  that way, "slightly unsafe" charsets can perhaps be
+         presented at the end, which will lose only a few characters
+         and are perhaps what the users were looking for.
+
+      2. when actually writing out, we need error checking in case an
+         individual char in a charset can't be written even though the
+         charsets are safe.  again, the user gets the choice of other
+         reasonable coding systems.
+
+      3. same thing (error checking, list of alternatives, etc.) needs
+         to happen when reading!  all of this will be a lot of work!
+
+
+
+Announcement, August 20, 2001:
+
+I'm looking for testers.  There is a complete and fast implementation
+in C of Unicode conversion, translations for almost all of the
+standardly-defined charsets that load up automatically and
+instantaneously at runtime, coding systems supporting the common
+external representations of Unicode [utf-16, ucs-4, utf-8,
+little-endian versions of utf-16 and ucs-4; utf-7 is sitting there
+with abort[]s where the coding routines should go, just waiting for
+somebody to implement], and a nice set of primitives for translating
+characters<->codepoints and setting the priority lists used to control
+codepoint->char lookup.
+
+It's so far hooked into one place: the Windows IME.  Currently I can
+select the Japanese IME from the thing on my tray pad in the lower
+right corner of the screen, and type Japanese into XEmacs, and you get
+Japanese in XEmacs -- regardless of whether you set either your
+current or global system locale to Japanese,and regardless of whether
+you set your XEmacs lang env as Japanese.  This should work for many
+other languages, too -- Cyrillic, Chinese either Traditional or
+Simplified, and many others, but YMMV.  There may be some lurking
+bugs (hardly surprising for something so raw).
+
+To get at this, checkout using `ben-mule-21-5', NOT the simpler
+*`mule-21-5'.  For example
+
+cvs -d :pserver:xemacs@cvs.xemacs.org:/usr/CVSroot checkout -r ben-mule-21-5  xemacs
+
+or you get the idea.  the `-r ben-mule-21-5' is important.
+
+I keep track of my progress in a file called README.ben-mule-21-5 in
+the root directory of the source tree.
+
+WARNING: Pdump might not work. Will be fixed rsn.
+
+August 20, 2001:
+
+-- still need to sort out demand loading, binary format, etc.  figure
+   out what the goals are and how we're going to achieve them.  for
+   the moment let's just say that running XEmacs in a directory with
+   Japanese or other weird characters in the name is likely to cause
+   problems under MS Windows, but once XEmacs is initialized (and
+   before processing init files), all Unicode support is there.
+
+-- wrote the size computation routines, although not yet tested.
+
+-- lots more abstraction of coding systems; almost done.
+
+-- UNICODE WORKS!!!!!
+
+
+August 19, 2001:
+
+Still needed on the Unicode support:
+
+-- demand loading: load the Unicode table data the first time a
+   conversion needs to be done.
+
+-- maybe: table size computation: figure out how big the in-memory
+   tables actually are.
+
+-- maybe: create a space-efficient binary format for the data, and a
+   way to dump out an existing charset's data into this binary format.
+   it should allow for many such groups of data to be appended
+   together in one file, such that you can just append the new data
+   onto the end and not have to go back and modify anything
+   previously. (like how tar archives work, and how the UFS? for
+   CD-R's and CD-RW's works.)
+
+-- maybe: figure out how to be able to access the Unicode tables at
+   init_intl() time, before we know how to get at data-directory; that
+   way we can handle the need for unicode conversions that come up
+   very early, for example if XEmacs is run from a directory
+   containing Japanese in it.  Presumably we'd want to generalize the
+   stuff in pdump.c that deals with the dumper file, so that it can
+   handle other files -- putting the file either in the directory of
+   the executable or in a resource, maybe actually attached to the
+   pdump file itself -- or maybe we just dump the data into the actual
+   executable.  With pdump we could extend pdump to allow for data
+   that's in the pdump file but not actually mapped at startup,
+   separate from the data that does get mapped -- and then at runtime
+   the pointer gets restored not with a real pointer but an offset
+   into the file; another pdump call and we get some way to access the
+   data. (tricky because it might be in a resource, not a file.  we
+   might have to just tell pdump to mmap or whatever the data in, and
+   then tell pdump to release it.)
+
+-- fix multibyte to use unicode.  at first, just reverse
+   mswindows-multibyte-to-unicode to be unicode-to-multibyte; later
+   implement something in chain to allow for reversal, for declaring
+   the ends of the coding systems, etc.
+
+-- actually make sure that the IME stuff is working!!!
+
+Other things before announcing:
+
+-- change so that the Unicode tables are not pdumped.  This means we
+   need to free any table data out there.  Make sure that pdump
+   compiles and try to finish the pretty-much-already-done stuff
+   already with XD_STRUCT_ARRAY and dynamic size computation; just
+   need to see what's going on with LO_LINK.
+
+August 14, 2001:
+
+To do a diff between this workspace and the mainline, use the most recent sync tags, currently:
+
+cvs diff -r main-branch-ben-mule-21-5-aug-11-2001-sync -r ben-mule-21-5-post-aug-11-2001-sync
+
+Unicode support:
+
+Unicode support is important for supporting many languages under
+Windows, such as Cyrillic, without resorting to translation tables for
+particular Windows-specific code pages.  Internally, all characters in
+Windows can be represented in two encodings: code pages and Unicode.
+With Unicode support, we can seamlessly support all Windows
+characters.  Currently, the test in the drive to support Unicode is if
+IME input works properly, since it is being converted from Unicode.
+
+Unicode support also requires that the various Windows API's be
+"Unicode-encapsulated", so that they automatically call the ANSI or
+Unicode version of the API call appropriately and handle the size
+differences in structures.  What this means is:
+
+-- first, note that Windows already provides a sort of encapsulation
+   of all API's that deal with text.  All such API's are underlyingly
+   provided in two versions, with an A or W suffix (ANSI or "wide"
+   i.e. Unicode), and the compile-time constant UNICODE controls which
+   is selected by the unsuffixed API.  Same thing happens with
+   structures.  Unfortunately, this is compile-time only, not
+   run-time, so not sufficient. (Creating the necessary run-time
+   encoding is not conceptually difficult, but very time-consuming to
+   write.  It adds no significant overhead, and the only reason it's
+   not standard in Windows is conscious marketing attempts by
+   Microsoft to cripple Windows 95.  FUCK MICROSOFT!  They even
+   describe in a KnowledgeBase article exactly how to create such an
+   API [although we don't exactly follow their procedure], and point
+   out its usefulness; the procedure is also described more generally
+   in Nadine Kano's book on Win32 internationalization -- written SIX
+   YEARS AGO!  Obviously Microsoft has such an API available
+   internally.)
+
+-- what we do is provide an encapsulation of each standard Windows API
+   call that is split into A and W versions.  current theory is to
+   avoid all preprocessor games; so we name the function with a prefix
+   -- "qxe" currently -- and require callers to use the prefixed name.
+   Callers need to explicitly use the W version of all structures, and
+   convert text themselves using Qmswindows_tstr.  the qxe
+   encapsulated version will automatically call the appropriate A or W
+   version depending on whether we're running on 9x or NT, and copy
+   data between W and A versions of the structures as necessary.
+
+-- We require the caller to handle the actual translation of text to
+   avoid possible overflow when dealing with fixed-size Windows
+   structures.  There are no such problems when copying data between
+   the A and W versions because ANSI text is never larger than its
+   equivalent Unicode representation.
+
+-- We allow for incremental creation of the encapsulated routines by
+   using the coding system Qmswindows_tstr_notyet.  This is an alias
+   for Qmswindows_multibyte, i.e. it always converts to ANSI; but it
+   indicates that it will be changed to Qmswindows_tstr when we have a
+   qxe version of the API call that the data is being passed to and
+   change the code to use the new function.
+
+Besides creating the encapsulation, the following needs to be done for
+Unicode support:
+
+-- No actual translation tables are fed into XEmacs.  We need to
+   provide glue code to read the tables in etc/unicode.  See
+   etc/unicode/README for the interface to implement.
+
+-- Fix pdump.  The translation tables for Unicode characters function
+   as unions of structures with different numbers of indirection
+   levels, in order to be efficient.  pdump doesn't yet support such
+   unions.  charset.h has a general description of how the translation
+   tables work, and the pdump code has constants added for the new
+   required data types, and descriptions of how these should work.
+
+-- ultimately, there's no end to additional work (composition, bidi
+   reordering, glyph shaping/ordering, etc.), but the above is enough
+   to get basic translation working.
+
+Merging this workspace into the trunk requires some work.  ChangeLogs
+have not yet been created.  Also, there is a lot of additional code in
+this workspace other than just Windows and Unicode stuff.  Some of the
+changes have been somewhat disruptive to the code base, in particular:
+
+-- the code that handles the details of processing multilingual text
+   has been consolidated to make it easier to extend it.  it has been
+   yanked out of various files (buffer.h, mule-charset.h, lisp.h,
+   insdel.c, fns.c, file-coding.c, etc.) and put into text.c and
+   text.h.  mule-charset.h has also been renamed charset.h.  all long
+   comments concerning the representations and their processing have
+   been consolidated into text.c.
+
+-- nt/config.h has been eliminated and everything in it merged into
+   config.h.in and s/windowsnt.h.  see config.h.in for more info.
+
+-- s/windowsnt.h has been completely rewritten, and s/cygwin32.h and
+   s/mingw32.h have been largely rewritten.  tons of dead weight has
+   been removed, and stuff common to more than one file has been
+   isolated into s/win32-common.h and s/win32-native.h, similar to
+   what's already done for usg variants.
+
+-- large amounts of code throughout the code base have been Mule-ized,
+   not just Windows code.
+
+-- file-coding.c/.h have been largely rewritten (although still mostly
+   syncable); see below.
+
+
+
+June 26, 2001:
+
+-- ben-mule-21-5
+
+this contains all the mule work i've been doing.  this includes mostly
+work done to get mule working under ms windows, but in the process
+i've [of course] fixed a whole lot of other things as well, mostly
+mule issues.  the specifics:
+
+- it compiles and runs under windows and should basically work.  the
+  stuff remaining to do is (a) improved unicode support (see below)
+  and (b) smarter handling of keyboard layouts.  in particular, it
+  should (1) set the right keyboard layout when you change your
+  language environment; (2) optionally (a user var) set the
+  appropriate keyboard layout as you move the cursor into text in a
+  particular language.
+
+- i added a bunch of code to better support OS locales.  it tries to
+  notice your locale at startup and set the language environment
+  accordingly (this more or less works), and call setlocale() and set
+  LANG when you change the language environment (may or may not work).
+
+- major rewriting of file-coding.  it's mostly abstracted into coding
+  systems that are defined by methods (similar to devices and
+  specifiers), with the ultimate aim being to allow non-i18n coding
+  systems such as gzip.  there is a "chain" coding system that allows
+  multiple coding systems to be chained together. (it doesn't yet
+  have the concept that either end of a coding system can be bytes or
+  chars; this needs to be added.)
+
+- unicode support.  very raw.  a few days ago i wrote a complete and
+  efficient implementation of unicode translation.  it should be very
+  fast, and fairly memory-efficient in its tables.  it allows for
+  charset priority lists, which should be language-environment
+  specific (but i haven't yet written the glue code).  it works in
+  preliminary testing, but obviously needs more testing and work.
+  as of yet there is no translation data added for the standard charsets.
+  the tables are in etc/unicode, and all we need is a bit of glue code
+  to process them.  see etc/unicode/README for the interface to
+  implement.
+
+- support for unicode in windows is partly there.  this will work even
+  on windows 95.  the basic model is implemented but it needs finishing
+  up.
+
+- there is a preliminary implementation of windows ime support courtesy
+  of ikeyama.
+
+- if you want to get cyrillic working under windows (it appears to "work"
+  but the wrong chars currently appear), the best way is to add unicode
+  support for iso-8859-5 and use it in redisplay-msw.c.  we are already
+  passing unicode codepoints to the text-draw routine (ExtTextOutW).
+  (ExtTextOutW and GetTextExtentPoint32W are implemented on both 95 and NT.)
+
+- i fixed the iso2022 handling so it will correctly read in files
+  containing unknown charsets, creating a "temporary" charset which
+  can later be overwritten by the real charset when it's defined.
+  this allows iso2022 elisp files with literals in strange languages
+  to compile correctly under mule.  i also added a hack that will
+  correctly read in and write out the emacs-specific "composition"
+  escape sequences, i.e. ESC 0 through ESC 4.  this means that my
+  workspace correctly compiles the new file devanagari.el that i added
+  (see below).
+
+- i copied the remaining language-specific files from fsf.  i made
+  some minor changes in certain cases but for the most part the stuff
+  was just copied and may not work.
+
+- i fixed post-read-conversion in coding systems to follow fsf
+  conventions. (i also support our convention, for the moment.  a
+  kludge, of course.)
+
+- make-coding-system accepts (but ignores) the additional properties
+  present in the fsf version, for compatibility.