xemacs-beta: README.ben-mule-21-5 comparison

comparison README.ben-mule-21-5 @ 771:943eaba38521

[xemacs-hg @ 2002-03-13 08:51:24 by ben] The big ben-mule-21-5 check-in! Various files were added and deleted. See CHANGES-ben-mule. There are still some test suite failures. No crashes, though. Many of the failures have to do with problems in the test suite itself rather than in the actual code. I'll be addressing these in the next day or so -- none of the test suite failures are at all critical. Meanwhile I'll be trying to address the biggest issues -- i.e. build or run failures, which will almost certainly happen on various platforms. All comments should be sent to ben@xemacs.org -- use a Cc: if necessary when sending to mailing lists. There will be pre- and post- tags, something like pre-ben-mule-21-5-merge-in, and post-ben-mule-21-5-merge-in.

author	ben
date	Wed, 13 Mar 2002 08:54:06 +0000
parents
children

comparison

equal deleted inserted replaced

-:336a418893b5
+:943eaba38521
+oct 27, 2001:
+-------- proposal for better buffer-switching commands:
+implement what VC++ currently has.  you have a single "switch" command like
+CTRL-TAB, which as long as you hold the CTRL button down, brings successive
+buffers that are "next in line" into the current position, bumping the rest
+forward.  once you release the CTRL key, the chain is broken, and further
+CTRL-TABs will start from the beginning again.  this way, frequently used
+buffers naturally move toward the front of the chain, and you can switch
+back and forth between two buffers using CTRL-TAB.  the only thing about
+CTRL-TAB is it's a bit awkward.  the way to implement is to have
+modifier-up strokes fire off a hook, like modifier-up-hook.  this is driven
+by event dispatch, so there are no synchronization issues.  when C-tab is
+pressed, the binding function does something like set a one-shot handler on
+the modifier-up-hook (perhaps separate hooks for separate modifiers?).
+to do this, we'd also want to change the buffer tabs so that they maintain
+their own order.  in particular, they start out synched to the regular
+order, but as you make changes, you don't want the tabs to change
+order. (in fact, they may already do this.) selecting a particular buffer
+from the buffer tabs DOES make the buffer go to the head of the line.  the
+invariant is that if the tabs are displaying X items, those X items are the
+first X items in the standard buffer list, but may be in a different
+order. (it looks like the tabs may already implement all of this.)
+oct 26, 2001:
+necessary testing/changes:
+- test all eol detection stuff under windows w/ and w/o mule, unix w/ and
+w/o mule. (test configure flag, command-line flag, menu option) may need
+a way of pretending to be unix under cygwin.
+- test under windows w/ and w/o mule, cygwin w/ and w/o mule, cygwin x
+windows w/ and w/o mule.
+- test undecided-dos/unix/mac.
+- check ESC ESC works as isearch-quit under TTY's.
+- test coding-system-base and all its uses (grep for them).
+- menu item to revert to most recent auto save.
+- consider renaming build_string -> build_intstring and build_c_string to
+build_string. (consistent with build_msg_string et al; many more
+build_c_string than build_string)
+oct 20, 2001:
+fixed problem causing crash due to invalid internal-format data, fixed an
+existing bug in valid_char_p, and added checks to more quickly catch when
+invalid chars are generated.  still need to investigate why
+mswindows-multibyte is being detected.
+i now see why -- we only process 65536 bytes due to a constant
+MAX_BYTES_PROCESSED_FOR_DETECTION.  instead, we should have no limit as
+long as we have a seekable stream.  we also need to write
+stderr_out_lisp(), used in the debug info routines i wrote.
+check once more about DEBUG_XEMACS.  i think debugging info should be
+ON by default.  make sure it is.  check that nothing untoward will result
+in a production system, e.g. presumably assert()s should not really abort().
+(!! Actually, this should be runtime settable!  Use a variable for this, and
+it can be set using the same XEMACSDEBUG method.  In fact, now that I think
+of it, I'm sure that debugging info should be on always, with runtime ways
+of turning on or off any funny behavior.)
+oct 19, 2001:
+fixed various bugs preventing packages from being able to be built.  still
+another bug, with psgml/etc/cdtd/docbook, which contains some strange
+characters starting around char pos 110,000.  It gets detected as
+mswindows-multibyte (wrong! why?) and then invalid internal-format data is
+generated.  need to fix mswindows-multibyte (and possibly add something
+that signals an error as well; need to work on this error-signalling
+mechanism) and figure out why it's getting detected as such.  what i should
+do is add a debug var that outputs blow-by-blow info of the detection
+process.
+oct 9, 2001:
+the stuff with global-window-system-map doesn't appear to work.  in any
+case it needs better documentation. [DONE]
+M-home, M-end do work, but cause cl-macs to get loaded.  why?
+oct 8, 2001:
+finished the coding system changes and they finally work!
+need to implement undecided-unix/dos/mac.  they should be easy to do; it
+should be enough to specify an eol-type but not do-eol, but check this.
+consider making the standard naming be foo-lf/crlf/cr, with unix/dos/mac as
+aliases.
+print methods for coding systems should include some of the generic
+properties. (also then fix print_..._within_print_method). [DONE]
+in a little while, go back and delete the text-file-wrapper-coding-system
+code. (it'll be in CVS if necessary to get at it.) [DONE]
+need to verify at some point that non-text-file coding systems work
+properly when specified.  when gzip is working, this would be a good test
+case. (and consider creating base64 as well!)
+remove extra crap from coding-system-category that checks for chain coding
+systems. [DONE]
+perhaps make a primitive that gets at coding-system-canonical. [DONE]
+need to test cygwin, compiling the mule packages, get unix-eol stuff
+working.  frank from germany says he doesn't see a lisp backtrace when he
+gets an error during temacs?  verify that this actually gets outputted.
+consider putting the current language on the modeline, mousable so it can
+be switched.  also consider making the coding system be mousable and the
+line number (pick a line) and the percentage (pick a percentage).
+oct 6, 2001:
+added code so that debug_print() will output a newline to the mswindows
+debugging output, not just the console.  need to test. [DONE]
+working on problem where all files are being detected as binary.  the
+problem may be that the undecided coding system is getting wrapped with an
+auto-eol coding system, which it shouldn't be -- but even in this
+situation, we should get the right results!  check the
+canonicalize-after-coding methods.  also, determine_real_coding_system
+appears to be getting called even when we're not detecting encoding.  also,
+undecided needs a print method to show its params, and chain needs to be
+updated to show canonicalize_after_coding.  check others as well. [DONE]
+oct 5, 2001:
+finished up coding system changes, testing.
+errors byte-compiling files in iso-2022-7-bit.  perhaps it's not correctly
+detecting the encoding?
+noticed a problem in the dfc macros: we call
+get_coding_system_for_text_file with eol_wrap == 1, to allow for
+auto-detection of the eol type; but this defeats the check and
+short-circuit for unicode.
+still need to implement calling determine_real_coding_system() for
+non-seekable streams.  to implement correctly, we need to do our own
+buffering. [DONE, BUT WITHOUT BUFFERING]
+oct 4, 2001:
+implemented most stuff below.
+need to finish up changes to make_coding_system_1. (i changed the way
+internal coding systems were handled; i need to create subsidiaries for all
+types of coding systems, not just text ones.) there's a nasty xfree() crash
+i was hitting; perhaps it'll go away once all stuff has been rewritten.
+check under cygwin to make sure that when an error occurs during loadup, a
+backtrace is output.
+as soon as andy releases his new setup, we should put it onto various
+standard windows software repositories.
+oct 3, 2001:
+added global-tty-map and global-window-system-map.  add some stuff to the
+maps, e.g. C-x ESC for repeat vs. C-x ESC ESC on TTY's, and of course ESC
+ESC on window systems vs. ESC ESC ESC on TTY's. [TEST]
+was working on integrating the two help-for-tutorial versions (mule,
+non-mule). [DONE, but test under non-Mule]
+was working on the file-coding changes.  need to think more about
+text-file-wrapper.  conclusion i think is that
+get_coding_system_for_text_file should wrap using a special coding system
+type called a text-file-wrapper, which inherits from chain, and implements
+canonicalize-after-decoding to just return the unwrapped coding system.  We
+need to implement inheritance of coding systems, which will certainly come
+in extremely useful when coding systems get implemented in Lisp, which
+should happen at some point. (see existing docs about this.)  essentially,
+we have a way of declaring that we inherit from some system, and the
+appropriate data structures get created, perhaps just an extra inheritance
+pointer.  but when we create the coding system, the extra data needs to be
+a stretchy array of offsets, pointing to the type-specific data for the
+coding system type and all its parents.  that means that in the methods
+structure for a coding system (which perhaps should be expanded beyond
+method, it's just a "class structure") is the index in these arrays of
+offsets.  CODING_SYSTEM_DATA() can take any of the coding system classes
+(rename type to class!) that make up this class.  similarly, a coding
+system class inherits its methods from the class above unless specifying
+its own method, and can call the superclass method at any point by either
+just invoking its name, or conceivably by some macro like
+CALL_SUPER (method, (args))
+similar mods would have to be made to coding stream structures.
+perhaps for the immediate we can just sort of fake things like we currently
+do with undecided calling some stuff from chain.
+oct 2, 2001:
+need to implement support for iso-8859-15, i.e. iso-8859-1 + euro symbol.
+figure out how to fall back to iso-8859-1 as necessary.
+leave the current bindings the way they are for the moment, but bump off
+M-home and M-end (hardly used), and substitute my buffer movement stuff
+there. [DONE, but test]
+there's something to be said for combining block of 6 and paragraph,
+esp. if we make the definition of "paragraph" be so that it skips by 6 when
+within code.  hmm.
+eliminate advertised-undo crap, and similar hacks. [DONE]
+think about obsolete stuff to be eliminated.  think about eliminating or
+dimming obsolete items from hyper-apropos and something similar in
+completion buffers.
+sep 30, 2001:
+synched up the tutorials with FSF 21.0.105.  was rewriting them to favor
+the cursor keys over the older C-p, etc. keys.
+Got thinking about key bindings again.
+(1) I think that M-up/down and M-C-up/down should be reversed.  I use
+scroll-up/down much more often than motion by paragraph.
+(2) Should we eliminate move by block (of 6) and subsitute it for
+paragraph?  This would have the advantage that I could make bindings
+for buffer change (forward/back buffer, perhaps M-C-up/down.  with
+shift, M-C-S-up/down only goes within the same type (C files, etc.).
+alternatively, just bump off beginning-of-defun from C-M-home, since
+it's on C-M-a already.
+need someone to go over the other tutorials (five new ones, from FSF
+21.0.105) and fix them up to correspond to the english one.
+shouldn't shift-motion work with C-a and such as well as arrows?
+sep 29, 2001:
+charcount_to_bytecount can also be made to scream -- as can scan_buffer,
+buffer_mule_signal_inserted_region, others?  we should start profiling
+though before going too far down this line.
+Debug code that causes no slowdown should in general remain in the
+executable even in the release version because it may be useful (e.g. for
+people to see the event output).  so DEBUG_XEMACS should be rethought.
+things like use of msvcrtd.dll should be controlled by error_checking on.
+maybe DEBUG_XEMACS controls general debug code (e.g. use of msvcrtd.dll,
+asserts abort, error checking), and the actual debugging code should remain
+always, or be conditonalized on something else
+(e.g. DEBUGGING_FUNS_PRESENT).
+doc strings in dumped files are displayed with an extra blank line between
+each line.  presumably this is recent?  i assume either the change to
+detect-coding-region or the double-wrapping mentioned below.
+error with coding-system-property on iso-2022-jp-dos.  problem is that that
+coding system is wrapped, so its type shows up as chain, not iso-2022.
+this is a general problem, and i think the way to fix it is to in essence
+do late canonicalization -- similar in spirit to what was done long ago,
+canonicalize_when_code, except that the new coding system (the wrapper) is
+created only once, either when the original cs is created or when first
+needed.  this way, operations on the coding system work like expected, and
+you get the same results as currently when decoding/encoding.  the only
+thing tricky is handling canonicalize-after-coding and the ever-tricky
+double-wrapping problem mentioned below.  i think the proper solution is to
+move the autodetection of eol into the main autodetect type.  it can be
+asked to autodetect eol, coding, or both.  for just coding, it does like it
+currently does.  for just eol, it does similar to what it currently does
+but runs the detection code that convert-eol currently does, and selects
+the appropriate convert-eol system.  when it does both eol and coding, it
+does something on the order of creating two more autodetect coding systems,
+one for eol only and one for coding only, and chains them together.  when
+each has detected the appropriate value, the results are combined.  this
+automatically eliminates the double-wrapping problem, removes the need for
+complicated canonicalize-after-coding stuff in chain, and fixes the problem
+of autodetect not having a seekable stream because hidden inside of a
+chain. (we presume that in the both-eol-and-coding case, the various
+autodetect coding streams can communicate with each other appropriately.)
+also, we should solve the problem of internal coding systems floating
+around and clogging up the list simply by having an "internal" property on
+cs's and an internal param to coding-system-list (optional; if not given,
+you don't get the internal ones). [DONE]
+we should try to reduce the size of the from-unicode tables (the dominant
+memory hog in the tables).  one obvious thing is to not store a whole
+emchar as the mapped-to value, but a short that encodes the octets. [DONE]
+sep 28, 2001:
+need to merge up to latest in trunk.
+add unicode charsets for all non-translatable unicode chars; probably want
+to extend the concept of charsets to allow for dimension 3 and dimension 4
+charsets.  for the moment we should stick with just dimension 3 charsets;
+otherwise we run past the current maximum of 4 bytes per emchar. (most code
+would work automatically since it uses MAX_EMCHAR_LEN; the trickiness is in
+certain code that has intimate knowledge of the representation.
+e.g. bufpos_to_bytind() has to multiply or divide by 1, 2, 3, or 4,
+and has special ways of handling each number.  with 5 or 6 bytes per char,
+we'd have to change that code in various ways.) 96x96x96 = 884,000 or so,
+so with two 96x96x96 charsets, we could tackle all Unicode values
+representable by UTF-16 and then some -- and only these codepoints will
+ever have assigned chars, as far as we know.
+need an easy way of showing the current language environment.  some menus
+need to have the current one checked or whatever. [DONE]
+implement unicode surrogates.
+implement buffer-file-coding-system-when-loaded -- make sure find-file,
+revert-file, etc. set the coding system [DONE]
+verify all the menu stuff [DONE]
+implemented the entirely-ascii check in buffers.  not sure how much gain
+it'll get us as we already have a known range inside of which is constant
+time, and with pure-ascii files the known range spans the whole buffer.
+improved the comment about how bufpos-to-bytind and vice-versa work. [DONE]
+fix double-wrapping of convert-eol: when undecided converts itself to
+something with a non-autodetect eol, it needs to tell the adjacent
+convert-eol to reduce itself to nothing.
+need menu item for find file with specified encoding. [DONE]
+renamed coding systems mswindows-### to windows-### to follow the standard
+in rfc1345. [DONE]
+implemented coding-system-subsidiary-parent [DONE]
+HAVE_MULE -> MULE in files in nt/ so that depend checking works [DONE]
+need to take the smarter search-all-files-in-dir stuff from my sample init
+file and put it on the grep menu [DONE]
+added item for revert w/specified encoding; mostly works, but needs fixes.
+in particular, you get the correct results, but buffer-file-coding-system
+does not reflect things right.  also, there are too many entries.  need to
+split into submenus.  there is already split code out there; see if it's
+generalized and if not make it so.  it should only split when there's more
+than a specified number, and when splitting, split into groups of a
+specified size, not into a specified number of groups. [DONE]
+too many entries in the langenv menus; need to split. [DONE]
+sep 27, 2001:
+NOTE: M-x grep for make-string causes crash now.  something definitely to
+do with string changes.  check very carefully the diffs and put in those
+sledgehammer checks. [DONE]
+fix font-lock bug i introduced. [DONE]
+added optimization to strings (keeps track of # of bytes of ascii at the
+beginning of a string).  perhaps should also keep an all-ascii flag to deal
+with really large (> 2 MB) strings.  rewrite code to count ascii-begin to
+use the 4-or-8-at-a-time stuff in bytecount_to_charcount.
+Error: M-q is causing Invalid Regexp error on the above paragraph.  It's
+not in working.  I assume it's a side effect of the string stuff.  VERIFY!
+Write sledgehammer checks for strings. [DONE]
+revamped the locale/init stuff so that it tries much harder to get things
+right.  should test a bit more.  in particular, test out Describe Language
+on the various created environments and make sure everything looks right.
+should change the menus: move the submenus on Edit->Mule directly under
+Edit.  add a menu entry on File to say "Reload with specified encoding ->".
+[DONE]
+Also Find File with specified encoding -> Also entry to change the EOL
+settings for Unix, and implement it.
+decode-coding-region isn't working because it needs to insert a binary
+(char->byte) converter. [DONE]
+chain should be rearranged to be in decoding order; similar for
+source/sink-type, other things?
+the detector should check for a magic cookie even without a seekable input.
+(currently its input is not seekable, because it's hidden within a chain.
+#### See what we can do about this.)
+provide a way to display various settings, e.g. the current category
+mappings and priority (see mule-diag; get this working so it's in the
+path); also a way to print out the likeliness results from a detection,
+perhaps a debug flag.
+problem with `env', which causes path issues due to `env' in packages.
+move env code to process, sync with fsf 21.0.105, check that the autoloads
+in `env' don't cause problems. [DONE]
+8-bit iso2022 detection appears broken; or at least, mule-canna.c is not so
+detected.
+sep 25, 2001:
+something else to do is review the font selection and fix it so that (e.g.)
+JISX-0212 can be displayed.
+also, text in widgets needs to be drawn by us so that the correct fonts
+will be displayed even in multi-lingual text.
+sep 24, 2001:
+the detection system is now properly abstracted.  the detectors have been
+rewritten to include multiple levels of abstraction.  now we just need
+detectors for ascii, binary, and latin-x, as well as more sophisticated
+detectors in general and further review of the general algorithm for doing
+detection. (#### Is this written up anywhere?) after that, consider adding
+error-checking to decoding (VERY IMPORTANT) and verifying the binary
+correctness of things under unix no-mule.
+sep 23, 2001:
+began to fix the detection system -- adding multiple levels of likelihood
+and properly abstracting the detectors.  the system is in place except for
+the abstraction of the detector-specific data out of the struct
+detection_state.  we should get things working first before tackling that
+(which should not be too hard).  i'm rewriting algorithms here rather than
+just converting code, so it's harder.  mostly done with everything, but i
+need to review all detectors except iso2022 and make them properly follow
+the new way.  also write a no-conversion detector.  also need to look into
+the `recode' package and see how (if?) they handle detection, and maybe
+copy some of the algorithms.  also look at recent FSF 21.0 and see if their
+algorithms have improved.
+sep 22, 2001:
+fixed gc bugs from yesterday.
+fixed truename bug.
+close/finalize stuff works.
+eliminated notyet stuff in syswindows.h.
+eliminated special code in tstr_to_c_string.
+fixed pdump problems. (many of them, mostly latent bugs, ugh)
+fixed cygwin sscanf problems in parse-unicode-translation-table. (NOT a
+sscanf bug, but subtly different behavior w.r.t. whitespace in the format
+string, combined with a debugger that sucks ROCKS!! and consistently
+outputs garbage for variable values.)
+main stuff to test is the handling of EOF recognition vs. binary
+(i.e. check what the default settings are under Unix).  then we may have
+something that WORKS on all platforms!!!  (Also need to test Windows
+non-Mule)
+sep 21, 2001:
+finished redoing the close/finalize stuff in the lstream code.  but i
+encountered again the nasty bug mentioned on sep 15 that disappeared on its
+own then.  the problem seems to be that the finalize method of some of the
+lstreams is calling Lstream_delete(), which calls free_managed_lcrecord(),
+which is a no-no when we're inside of garbage-collection and the object
+passed to free_managed_lcrecord() is unmarked, and about to be released by
+the gc mechanism -- the free lists will end up with xfree()d objects on
+them, which is very bad.  we need to modify free_managed_lcrecord() to
+check if we're in gc and the object is unmarked, and ignore it rather than
+move it to the free list. [DONE]
+(#### What we really need to do is do what Java and C# do w.r.t. their
+finalize methods: For objects with finalizers, when they're about to be
+freed, leave them marked, run the finalizer, and set another bit on them
+indicating that the finalizer has run.  Next GC cycle, the objects will
+again come up for freeing, and this time the sweeper notices that the
+finalize method has already been called, and frees them for good (provided
+that a finalize method didn't do something to make the object alive
+again).)
+sep 20, 2001:
+redid the lstream code so there is only one coding stream.  combined the
+various doubled coding stream methods into one; i'm a little bit unsure of
+this last part, though, as the results of combining the two together seem
+unclean.  got it to compile, but it crashes in loadup.  need to go through
+and rehash the close vs. finalize stuff, as the problem was stuff getting
+freed too quickly, before the canonicalize-after-decoding was run.  should
+eliminate entirely CODING_STATE_END and use a different method (close
+coding stream).  rewrite to use these two.  make sure they're called in the
+right places.  Lstream_close on a stream should *NOT* do finalizing.
+finalize only on delete. [DONE]
+in general i'd like to see the flags eliminated and converted to
+bit-fields.  also, rewriting the methods to take advantage of rejecting
+should make it possible to eliminate much of the state in the various
+methods, esp. including the flags.  need to test this is working, though --
+reduce the buffer size down very low and try files with only CRLF's in
+them, with one offset by a byte from the other, and see if we correctly
+handle rejection.
+still have the problem with incorrectly truenaming files.
+sep 19, 2001:
+bug reported: crash while closing lstreams.
+the lstream/coding system close code needs revamping.  we need to document
+that order of closing lstreams is very important, and make sure we're
+consistent.  furthermore, chain and undecided lstreams need to close their
+underneath lstreams when they receive the EOF signal (there may be data in
+the underneath streams waiting to come out), not when they themselves are
+closed. [DONE]
+(if only we had proper inheritance.  i think in any case we should
+simulate it for the chain coding stream -- write things in such a way that
+undecided can use the chain coding stream and not have to duplicate
+anything itself.)
+in general we need to carefully think through the closing process to make
+sure everything always works correctly and in the right order.  also check
+very carefully to make sure there are no dangling pointers to deleted
+objects floating around.
+move the docs for the lstream functions to the functions themselves, not
+the header files.  document more carefully what exactly Lstream_delete()
+means and how it's used, what the connections are between Lstream_close(),
+Lstream_delete(), Lstream_flush(), lstream_finalize, etc. [DONE]
+additional error-checking: consider deadbeefing the memory in objects
+stored in lcrecord free lists; furthermore, consider whether lifo or fifo
+is correct; under error-checking, we should perhaps be doing fifo, and
+setting a minimum number of objects on the lists that's quite large so that
+it's highly likely that any erroneous accesses to freed objects will go
+into such deadbeefed memory and cause crashes.  also, at the earliest
+available opportunity, go through all freed memory and check for any
+consistency failures (overwrites of the deadbeef), crashing if so.  perhaps
+we could have some sort of id for each block, to easier trace where the
+offending block came from. (all of these ideas are present in the debug
+system malloc from VC++, plus more stuff.) there's similar code i wrote
+sitting somewhere (in free-hook.c? doesn't appear so. we need to delete the
+blocking stuff out of there!).  also look into using the debug system
+malloc from VC++, which has lots of cool stuff in it. we even have the
+sources.  that means compiling under pdump, which would be a good idea
+anyway.  set it as the default. (but then, we need to remove the
+requirement that Xpm be a DLL, which is extremely annoying.  look into
+this.)
+test the windows code page coding systems recently created.
+problems reading my mail files -- 1personal appears to hang, others come up
+with lots of ^M's.  investigate.
+test the enum functions i just wrote, and finish them.
+still pdump problems.
+sep 18, 2001:
+critical-quit broken sometime after aug 25.
+-- fixed critical quit.
+-- fixed process problems.
+-- print routines work. (no routine for ccl, though)
+-- can read and write unicode files, and they can still be read by some
+other program
+-- defaults should come up correctly -- mswindows-multibyte is general.
+still need to test matej's stuff.
+seems ok with multibyte stuff but needs more testing.
+sep 17, 2001:
+!!!!! something broken with processes !!!!! cannot send mail anymore.  must
+investigate.
+sep 17, 2001:
+on mon/wed nights, stop *BEFORE* 11pm.  Otherwise i just start getting
+woozy and can't concentrate.
+just finished getting assorted fixups to the main branch committed, so it
+will compile under C++ (Andy committed some code that broke C++ builds).
+cup'd the code into the fixtypes workspace, updated the tags appropriately.
+i've created the appropriate log message, sitting in fixtypes.txt in
+/src/xemacs; perhaps it should go into a README.  now i just have to build
+on everything (it's currently building), verify it's ok, run patcher-mail,
+commit, send.
+my mule ws is also very close.  need to:
+-- test the new print routines.
+-- test it can read and write unicode files, and they can still be read by
+some other program.
+-- try to see if unicode can be auto-detected properly.
+-- test it can read and write multibyte files in a few different formats.
+currently can't recognize them, but if you set the cs right, it should
+work.
+-- examine the test files sent by matej and see if we can handle them.
+sep 15, 2001:
+more eol fixing.  this stuff is utter crap.
+currently we wrap coding systems with convert-eol-autodetect when we create
+them in make_coding_system_1.  i had a feeling that this would be a
+problem, and indeed it is -- when autodetecting with `undecided', for
+example, we end up with multiple layers of eol conversion.  to avoid this,
+we need to do the eol wrapping *ONLY* when we actually retrieve a coding
+system in places such as insert-file-contents.  these places are
+insert-file-contents, load, process input, call-process-internal,
+encode/decode/detect-coding-region, database input, ...
+(later) it's fixed, and things basically work.  NOTE: for some reason,
+adding code to wrap coding systems with convert-eol-lf when eol-type == lf
+results in crashing during garbage collection in some pretty obscure place
+-- an lstream is free when it shouldn't be.  this is a bad sign.  i guess
+something might be getting initialized too early?
+we still need to fix the canonicalization-after-decoding code to avoid
+problems with coding systems like `internal-7' showing up.  basically, when
+eol==lf is detected, nil should be returned, and the callers should handle
+it appropriately, eliding when necessary.  chain needs to recognize when
+it's got only one (or even 0) items in the chain, and elide out the chain.
+sep 11, 2001: the day that will live in infamy.
+rewrite of sep 9 entry about formats:
+when calling make-coding-system, the name can be a cons of (format1 .
+format2), specifying that it decodes format1->format2 and encodes the other
+way.  if only one name is given, that is assumed to be format1, and the
+other is either `external' or `internal' depending on the end type.
+normally the user when decoding gives the decoding order in formats, but
+can leave off the last one, `internal', which is assumed.  a multichain
+might look like gzip|multibyte|unicode, using the coding systems named
+`gzip', `(unicode . multibyte)' and `unicode'.  the way this actually works
+is by searching for gzip->multibyte; if not found, look for gzip->external
+or gzip->internal. (In general we automatically do conversion between
+internal and external as necessary: thus gzip|crlf does the expected, and
+maps to gzip->external, external->internal, crlf->internal, which when
+fully specified would be gzip|external:external|internal:crlf|internal --
+see below.)  To forcibly fit together two converters that have explicitly
+specified and incompatible names (say you have unicode->multibyte and
+iso8859-1->ebcdic and you know that the multibyte and iso8859-1 in this
+case are compatible), you can force-cast using :, like this:
+ebcdic|iso8859-1:multibyte|unicode. (again, if you force-cast between
+internal and external formats, the conversion happens automatically.)
+sep 10, 2001:
+moved the autodetection stuff (both codesys and eol) into particular coding
+systems -- `undecided' and `convert-eol' (type == `autodetect').  needs
+lots of work.  still need to search through the rest of the code and find
+any remaining auto-detect code and move it into the undecided coding
+system.  need to modify make-coding-system so that it spits out
+auto-detecting versions of all text-file coding systems unless we say not
+to.  need eliminate entirely the EOF flag from both the stream info and the
+coding system; have only the original-eof flag.  in
+coding_system_from_mask, need to check that the returned value is not of
+type `undecided', falling back to no-conversion if so.  also need to make
+sure we wrap everything appropriate for text-files -- i removed the
+wrapping on set-coding-category-list or whatever (need to check all those
+files to make sure all wrapping is removed).  need to review carefully the
+new code in `undecided' to make sure it works are preserves the same logic
+as previously.  need to review the closing and rewinding behavior of chain
+and undecided (same -- should really consolidate into helper routines, so
+that any coding system can embed a chain in it) -- make sure the dynarr's
+are getting their data flushed out as necessary, rewound/closed in the
+right order, no missing steps, etc.
+also split out mule stuff into mule-coding.c.  work done on
+configure/xemacs.mak/Makefiles not done yet.  work on emacs.c/symsinit.h to
+interface with the new init functions not done yet.
+also put in a few declarations of the way i think the abstracted detection
+stuff ought to go.  DON'T WORK ON THIS MORE UNTIL THE REST IS DEALT WITH
+AND WE HAVE A WORKING XEMACS AGAIN WITH ALL EOL ISSUES NAILED.
+really need a version of cvs-mods that reports only the current directory.
+WRITE THIS!  use it to implement a better cvs-checkin.
+sep 9, 2001:
+implemented a gzip coding system.  unfortunately, doesn't quite work right
+because it doesn't handle the gzip headers -- it just reads and writes raw
+zlib data.  there's no function in the library to skip past the header, but
+we do have some code out of the library that we can snarf that implements
+header parsing.  we need to snarf that, store it, and output it again at
+the beginning when encoding.  in the process, we should create a "get next
+byte" macro that bails out when there are no more.  using this, we set up a
+nice way of doing most stuff statelessly -- if we have to bail, we reject
+everything back to the sync point.  also need to fix up the autodetection
+of zlib in configure.in.
+BIG problems with eol.  finished up everything i thought i would need to
+get eol stuff working, but no -- when you have mswindows-unicode, with its
+eol set to autodetect, the detection routines themselves do the autodetect
+(first), and fail (they report CR on CRLF because of the NULL byte between
+the CR and the LF) since they're not looking at ascii data.  with a chain
+it's similarly bad. for mswindows-multibyte, for example, which is a chain
+unicode->unicode-to-multibyte, autodetection happens inside of the chain,
+both when unicode and unicode-to-multibyte are active.  we could twiddle
+around with the eol flags to try to deal with this, but it's gonna be a big
+mess, which is exactly what we're trying to avoid.  what we basically want
+is to entirely rip out all EOL settings from either the coding system or
+the stream (yes, there are two!  one might saw autodetect, and then the
+stream contains the actual detected value).  instead, we simply create an
+eol-autodetect coding system -- or rather, it's part of the convert-eol
+coding system.  convert-eol, type = autodetect, does autodetection the
+first time it gets data sent to it to decode, and thereafter sets a stream
+parameter indicating the actual eol type for this stream.  this means that
+all autodetect coding systems, as created by `make-coding-system', really
+are chains with a convert-eol at the beginning.  only subsidiary xxx-unix
+has no wrapping at all.  this should allow eof detection of gzip, unicode,
+etc.  for that matter, general autodetection should be entirely
+encapsulated inside of the `autodetect' coding system, with no
+eol-autodetection -- the chain becomes convert-eol (autodetect) ->
+autodetect or perhaps backwards.  the generic autodetect similarly has a
+coding-system in its stream methods, and needs somehow or other to insert
+the detected coding-system into the chain.  either it contains a chain
+inside of it (perhaps it *IS* a chain), or there's some magic involving
+canonicalization-type switcherooing in the middle of a decode.  either way,
+once everything is good and done and we want to save the coding system so
+it can be used later, we need to do another sort of canonicalization --
+converting auto-detect-type coding systems into the detected systems.
+again, a coding-system method, with some magic currently so that
+subsidiaries get properly used rather than something that's new but
+equivalent to subsidiaries. (#### perhaps we could use a hash table to
+avoid recreating coding systems when not necessary.  but that would require
+that coding systems be immutable from external, and i'm not sure that's the
+case.)
+i really think, after all, that i should reverse the naming of everything
+in chain and source-sink-type -- they should be decoding-centric.  later
+on, if/when we come up with the proper way to make it totally symmetrical,
+we'll be fine whether before then we were encoding or decoding centric.
+sep 9, 2001:
+investigated eol parameter.
+implemented handling in make-coding-system of eol-cr and eol-crlf.
+fixed calls everywhere to Fget_coding_system / Ffind_coding_system to
+reject non-char->byte coding systems.
+still need to handle "query eol type using coding-system-property" so it
+magically returns the right type by parsing the chain.
+no work done on formats, as mentioned below.  we should consider using :
+instead of || to indicate casting.
+early sep 9, 2001:
+renamed some codesys properties: `list' in chain -> chain; `subtype' in
+unicode -> type.  everything compiles again and sort of works; some CRLF
+problems that may resolve themselves when i finish the convert-eol stuff.
+the stuff to create subsidiaries has been rewritten to use chains; but i
+still need to investigate how the EOL type parameter is used.  also, still
+need to implement this: when a coding system is created, and its eol type
+is not autodetect or lf, a chain needs to be created and returned.  i think
+that what needs to happen is that the eol type can only be set to
+autodetect or lf; later on this should be changed to simply be either
+autodetect or not (but that would require ripping out the eol converting
+stuff in the various coding systems), and eventually we will do the work on
+the detection mechanism so it can do chain detection; then we won't need an
+eol autodetect setting at all.  i think there's a way to query the eol type
+of a coding system; this should check to see if the coding system is a
+chain and there's a convert-eol at the front; if so, the eol type comes
+from the type of the convert-eol.
+also check out everywhere that Fget_coding_system or Ffind_coding_system is
+called, and see whether anything but a char->byte system can be tolerated.
+create a new function for all the places that only want char->byte,
+something like get_coding_system_char_to_byte_only.
+think about specifying formats in make-coding-system.  perhaps the name can
+be a cons of (format1, format2), specifying that it encodes
+format1->format2 and decodes the other way.  if only one name is given,
+that is assumed to be format2, and the other is either `byte' or `char'
+depending on the end type.  normally the user when decoding gives the
+decoding order in formats, but can leave off the last one, `char', which is
+assumed.  perhaps we should say `internal' instead of `char' and `external'
+instead of byte.  a multichain might look like gzip|multibyte|unicode,
+using the coding systems named `gzip', `(unicode . multibyte)' and
+`unicode'.  we would have to allow something where one format is given only
+as generic byte/char or internal/external to fit with any of the same
+byte/char type.  when forcibly fitting together two converters that have
+explicitly specified and incompatible names (say you have
+unicode->multibyte and iso8859-1->ebcdic and you know that the multibyte
+and iso8859-1 in this case are compatible), you can force-cast using ||,
+like this: ebcdic|iso8859-1||multibyte|unicode.  this will also force
+external->internal translation as necessary:
+unicode|multibyte||crlf|internal does unicode->multibyte,
+external->internal, crlf->internal.  perhaps you'd need to put in the
+internal translation, like this: unicode|multibyte|internal||crlf|internal,
+which means unicode->multibyte, external->internal (multibyte is compatible
+with external); force-cast to crlf format and convert crlf->internal.
+even later: Sep 8, 2001:
+chain doesn't need to set character mode, that happens automatically when
+the coding systems are created.  fixed chain to return correct source/sink
+type for itself and to check the compatibility of source/sink types in its
+chain.  fixed decode/encode-coding-region to check the source and sink
+types of the coding system performing the conversion and insert appropriate
+byte->char/char->byte converters (aka "binary" coding system).  fixed
+set-coding-category-system to only accept the traditional
+encode-char-to-byte types of coding systems.
+still need to extend chain to specify the parameters mentioned below,
+esp. "reverse".  also need to extend the print mechanism for chain so it
+prints out the chain.  probably this should be general: have a new method
+to return all properties, and output those properties.  you could also
+implement a read syntax for coding systems this way.
+still need to implement convert-eol and finish up the rest of the eol stuff
+mentioned below.
+later September 7, 2001: (more like Sep 8)
+moved many Lisp_Coding_System * params to Lisp_Object.  In general this is
+the way to go, and if we ever implement a copying GC, we will never want to
+be passing direct pointers around.  With no error-checking, we lose no
+cycles using Lisp_Objects in place of pointers -- the Lisp_Object itself is
+nothing but a pointer, and so all the casts and "dereferences" boil down to
+nothing.
+Clarified and cleaned up the "character mode" on streams, and documented
+who (caller or object itself) has the right to be setting character mode on
+a stream, depending on whether it's a read or write stream.  changed
+conversion_end_type method and enum source_sink_type to return
+encoding-centric values, rather than decoding-centric.  for the moment,
+we're going to be entirely encoding-centric in everything; we can rethink
+later.  fixed coding systems so that the decode and encode methods are
+guaranteed to receive only full characters, if that's the source type of
+the data, as per conversion_end_type.
+still need to fix the chain method so that it correctly sets the character
+mode on all the lstreams in it and checks the source/sink types to be
+compatible.  also fix decode-coding-string and friends to put the
+appropriate byte->character (i.e. no-conversion) coding systems on the ends
+as necessary so that the final ends are both character.  also add to chain
+a parameter giving the ability to switch the direction of conversion of any
+particular item in the chain (i.e. swap encoding and decoding).  i think
+what we really want to do is allow for arbitrary parameters to be put onto
+a particular coding system in the chain, of which the only one so far is
+swap-encode-decode.  don't need too much codage here for that, but make the
+design extendable.
+September 7, 2001:
+just added a return value from the decode and encode methods of a coding
+system, so that some of the data can get rejected.  fixed the calling
+routines to handle this.  need to investigate when and whether the coding
+lstream is set to character mode, so that the decode/encode methods only
+get whole characters.  if not, we should do so, according to the source
+type of these methods.  also need to implement the convert_eol coding
+system, and fix the subsidiary coding systems (and in general, any coding
+system where the eol type is specified and is not LF) to be chains
+involving convert_eol.
+after everything is working, need to remove eol handling from encode/decode
+methods and eventually consider rewriting (simplifying) them given the
+reject ability.
+September 5, 2001:
+-- need to organize this.  get everything below into the TODO list.
+CVS the TODO list frequently so i can delete old stuff.  prioritize
+it!!!!!!!!!
+-- move README.ben-mule... to STATUS.ben-mule...; use README for
+intro, overview of what's new, what's broken, how to use the
+features, etc.
+-- need a global and local coding-category-precedence list, which get
+merged.
+-- finished the BOM support.  also finished something not listed
+below, expansion to the auto-generator of Unicode-encapsulation to
+support bracketing code with #if ... #endif, for Cygwin and MINGW
+problems, e.g.  This is tested; appears to work.
+-- need to add more multibyte coding systems now that we have various
+properties to specify them.  need to add DEFUN's for mac-code-page
+and ebcdic-code-page for completeness.  need to rethink the whole
+way that the priority list works.  it will continue to be total
+junk until multiple levels of likeliness get implemented.
+-- need to finish up the stuff about the various defaults. [need to
+investigate more generally where all the different default values
+are that control encoding. (there are six places or so.) need to
+list them in make-coding-system docs and put pointers
+elsewhere. [[[[#### what interface to specify that this default
+should be unicode?  a "Unicode" language environment seems too
+drastic, as the language environment controls much more.]]]] even
+skipping the Unicode stuff here, we need to survey and list the
+variables that control coding page behavior and determine how they
+need to be set for various possible scenarios:
+-- total binary: no detection at all.
+-- raw-text only: wants only autodetection of line endings, nothing else.
+-- "standard Windows environment": tries for Unicode, falls back on
+code page encoding.
+-- some sort of East European environment, and Russian.
+-- some sort of standard Japanese Windows environment.
+-- standard Chinese Windows environments (traditional and simplified)
+-- various Unix environments (European, Japanese, Russian, etc.)
+-- Unicode support in all of these when it's reasonable
+These really require multiple likelihood levels to be fully
+implementable.  We should see what can be done ("gracefully fall
+back") with single likelihood level.  need lots of testing.
+-- need to fix the truename problem.
+-- lots of testing: need to test all of the stuff above and below that's recently been implemented.
+September 4, 2001:
+mostly everything compiles.  currently there is a crash in
+parse-unicode-translation-table, and Cygwin/Mule won't run.  it may
+well be a bug in the sscanf() in Cygwin.
+working on today:
+-- adding BOM support for Unicode coding systems.  mostly there, but
+need to finish adding BOM support to the detection routines.  then test.
+-- adding properties to unicode-to-multibyte to specify the coding
+system in various flexible ways, e.g. directly specified code page
+or ansi or oem code page of specified locale, current locale,
+user-default or system-default locale.  need to test.
+-- creating a `multibyte' coding system, with the same parameters as
+unicode-to-multibyte and which resolves at coding-system-creation
+time to the appropriate chain.  creating the underlying mechanism
+to allow such under-the-scenes switcheroo.  need to test.
+-- set default-value of buffer-file-coding-system to
+mswindows-multibyte, as Matej said it should be.  need to test.
+need to investigate more generally where all the different default
+values are that control encoding. (there are six places or so.)
+need to list them in make-coding-system docs and put pointers
+elsewhere. #### what interface to specify that this default should
+be unicode?  a "Unicode" language environment seems too drastic, as
+the language environment controls much more.
+-- thinking about adding multiple levels of certainty to the detection
+schemes, instead of just a mask.  eventually, we need to totally
+abstract things, but that can easier be done in many steps. (we
+need multiple levels of likelihood to more reasonably support a
+Windows environment with code-page type files.  currently, in order
+to get them detected, we have to put them first, because they can
+look like lots of other things; but then, other encodings don't get
+detected.  with multiple levels of likelihood, we still put the
+code-page categories first, but they will return low levels of
+likelihood.  Lower-down encodings may be able to return higher
+levels of likelihood, and will get taken preferentially.)
+-- making it so you cannot disable file-coding, but you get an
+equivalent default on Unix non-Mule systems where all defaults are
+`binary'.  need to test!!!!!!!!!
+Matej (mostly, + some others) notes the following problems, and here
+are possible solutions:
+-- he wants the defaults to work right. [figure out what those
+defaults are.  i presume they are auto-detection of data in current
+code page and in unicode, and new files have current code page set
+as their output encoding.]
+-- too easy to lose data with incorrect encodings. [need to set up an
+error system for encoding/decoding.  extremely important but a
+little tricky to implement so let's deal with other issues now.]
+-- EOL isn't always detected correctly. [#### ?? need examples]
+-- truename isn't working: c:\t.txt and c:\tmp.txt have the same truename.
+[should be easy to fix]
+-- unicode files lose the BOM mark. [working on this]
+-- command-line utilities use OEM. [actually it seems more
+complicated.  it seems they use the codepage of the console.  we
+may be able to set that, e.g. to UTF8, before we invoke a command.
+need to investigate.]
+-- no way to handle unicode characters not recognized as charsets. [we
+need to create something like 8 private 2-dimensional charsets to
+handle all BMP Unicode chars.  Obviously this is a stopgap
+solution.  Switching to Unicode internal will ultimately make life
+far easier and remove the BMP limitation.  but for now it will
+work.  we translate all characters where we have charsets into
+chars in those charsets, and the remainder in a unicode charset.
+that way we can save them out again and guarantee no data loss with
+unicode.  this creates font problems, though ...]
+-- problems with xemacs font handling. [xemacs font handling is not
+sophisticated enough.  it goes on a charset granularity basis and
+only looks for a font whose name contains the corresponding windows
+charset in it.  with unicode this fails in various ways.  for one
+the granularity needs to be single character, so that those unicode
+charsets mentioned above work; and it needs to query the font to
+see what unicode ranges it supports, rather than just looking at
+the charset ending.]
+August 28, 2001:
+working on getting everything to compile again: Cygwin, non-MULE,
+pdump.  not there yet.
+mswindows-multibyte is now defined using chain, and works.  removed
+most vestiges of the mswindows-multibyte coding system type.
+file-coding is on by default; should default to binary only on Unix.
+Need to test. (Needs to compile first :-)
+August 26, 2001:
+I've fixed the issue of inputting non-ASCII text under -nuni, and done
+some of the work on the Russian C-x problem -- we now compute the
+other possibilities.  We still need to fix the key-lookup code,
+though, and that code is unfortunately a bit ugly.  the best way, it
+seems, is to expand the command-builder structure so you can specify
+different interpretations for keys. (if we do find an alternative
+binding, though, we need to mess with both the command builder and
+this-command-keys, as does the function-key stuff.  probably need to
+abstract that munging code.)
+high-priority:
+[currently doing]
+-- support for WM_IME_CHAR.  IME input can work under -nuni if we use
+WM_IME_CHAR.  probably we should always be using this, instead of
+snarfing input using WM_COMPOSITION.  i'll check this out.
+-- Russian C-x problem.  see above.
+[clean-up]
+-- make sure it compiles and runs under non-mule.  remember that some
+code needs the unicode support, or at least a simple version of it.
+-- make sure it compiles and runs under pdump.  see below.
+-- clean up mswindows-multibyte, TSTR_TO_C_STRING.  see below. [DONE]
+-- eliminate last vestiges of codepage<->charset conversion and similar stuff.
+[other]
+-- cut and paste.  see below.
+-- misc issues with handling lang environments.  see also August 25,
+"finally: working on the C-x in ...".
+-- when switching lang env, needs to set keyboard layout.
+-- user var to control whether, when moving into text of a
+particular language, we set the appropriate keyboard layout.  we
+would need to have a lisp api for retrieving and setting the
+keyboard layout, set text properties to indicate the layout of
+text, and have a way of dealing with text with no property on
+it. (e.g. saved text has no text properties on it.) basically,
+we need to get a keyboard layout from a charset; getting a
+language would do.  Perhaps we need a table that maps charsets
+to language environments.
+-- test that the lang env is properly set at startup.  test that
+switching the lang env properly sets the C locale (call
+setlocale(), set LANG, etc.) -- a spawned subprogram should have
+the new locale in its environment.
+-- look through everything below and see if anything is missed in this
+priority list, and if so add it.  create a separate file for the
+priority list, so it can be updated as appropriate.
+mid-priority:
+-- clean up the chain coding system.  its list should specify decode
+order, not encode; i now think this way is more logical.  it should
+check the endpoints to make sure they make sense.  it should also
+allow for the specification of "reverse-direction coding systems":
+use the specified coding system, but invert the sense of decode and
+encode.
+-- along with that, places that take an arbitrary coding system and
+expect the ends to be anything specific need to check this, and add
+the appropriate conversions from byte->char or char->byte.
+-- get some support for arabic, thai, vietnamese, japanese jisx 0212:
+at least get the unicode information in place and make sure we have
+things tied together so that we can display them.  worry about r2l
+some other time.
+August 25, 2001:
+There is actually more non-Unicode-ized stuff, but it's basically
+inconsequential. (See previous note.) You can check using the file
+nmkun.txt (#### RENAME), which is just a list of all the routines that
+have been split. (It was generated from the output of `nmake
+unicode-encapsulate', after removing everything from the output but
+the function names.) Use something like
+fgrep -f ../nmkun.txt -w [a-hj-z]*.[ch]  |m
+in the source directory, which does a word match and skips
+intl-unicode-win32.[ch] and intl-win32.[ch], which have a whole lot of
+references to these, unavoidably.  It effectively detects what needs
+to be changed because changed versions either begin qxe... or end with
+A or W, and in each case there's no whole-word match.
+The nasty bug has been fixed below.  The -nuni option now works -- all
+specially-written code to handle the encapsulation has been tested by
+some operation (fonts by loadup and checking the output of (list-fonts
+""); devmode by printing; dragdrop tests other stuff).
+NOTE: for -nuni (Win 95), areas need work:
+-- cut and paste.  we should be able to receive Unicode text if it's
+there, and we should be able to receive it even in Win 95 or -nuni.
+we should just check in all circumstances.  also, under 95, when we
+put some text in the clipboard, it may or may not also be
+automatically enumerated as unicode.  we need to test this out
+and/or just go ahead and manually do the unicode enumeration.
+-- receiving keyboard input.  we get only a single byte, but we should
+be able to correlate the language of the keyboard layout to a
+particular code page, so we can then decode it correctly.
+-- mswindows-multibyte.  still implemented as its own thing.  should
+be done as a chain of (encoding) unicode | unicode-to-multibyte.
+need to turn this on, get it working, and look into optimizations
+in the dfc stuff. (#### perhaps there's a general way to do these
+optimizations???  something like having a method on a coding system
+that can specify whether a pure-ASCII string gets rendered as
+pure-ASCII bytes and vice-versa.)
+ALSO:
+-- we have special macros TSTR_TO_C_STRING and such because formerly
+the DFC macros didn't know about external stuff that was Unicode
+encoded and would call strlen() on them.  this is fixed, so now we
+should undo the special macros, make em normal, removal the
+comments about this, and make sure it works. [DONE]
+-- finally: working on the C-x in Russian key layout problem.  in the
+process will probably end up doing work on cleaning up the handling
+of keyboard layouts, integrating or deleting the FSF stuff, adding
+code to change the keyboard layout as we move in and out of text in
+different languages (implemented as a post-command-hook; we need
+something like internal-post-command-hook if not already there, for
+internal stuff that doesn't want to get mixed up with the regular
+post-command-hook; similar for pre-command-hook).  also, when
+langenv changes, ways to set the keyboard layout appropriately.
+-- i think the stuff above is higher priority than the other stuff
+mentioned below.  what i'm aiming for is to be able to input and
+work with multiple languages without weird glitches, both under 95
+and NT.  the problems above are all basic impediments to such work.
+we assume for the moment that the user can make use of the existing
+file i/o conversion stuff, and put that lower in priority, after
+the basic input is working.
+-- i should get my modem connected and write up what's going on and
+send it to the lists; also cvs commit my workspaces and get more
+testers.
+August 24, 2001:
+All code has been Unicode-ized except for some stuff in console-msw.c
+that deals with console output.  Much of the Unicode-encapsulation
+stuff, particularly the hand-written stuff, really needs testing.  I
+added a new command-line option, -nuni, to force use of all ANSI calls
+-- XE_UNICODEP evaluates to false in this case.
+There is a nasty bug that appeared recently, probably when the event
+code got Unicode-ized -- bad interactions with OS sticky modifiers.
+Hold the shift key down and release it, then instead of affecting the
+next char only, it gets permanently stuck on (until you do a regular
+shift+char stroke).  This needs to be debugged.
+Other things on agenda:
+-- go through and prioritize what's listed below.
+-- make sure the pdump code can compile and work.  for the moment we
+just don't try to dump any Unicode tables and load them up each
+time.  this is certainly fast but ...
+-- there's the problem that XEmacs can't be run in a directory with
+non-ASCII/Latin-1 chars in it, since it will be doing Unicode
+processing before we've had a chance to load the tables.  In fact,
+even finding the tables in such a situation is problematic using
+the normal commands.  my idea is to eventually load the stuff
+extremely extremely early, at the same time as the pdump data gets
+loaded.  in fact, the unicode table data (stored in an efficient
+binary format) can even be stuck into the pdump file (which would
+mean as a resource to the executable, for windows).  we'd need to
+extend pdump a bit: to allow for attaching extra data to the pdump
+file. (something like pdump_attach_extra_data (addr, length)
+returns a number of some sort, an index into the file, which you
+can then retrieve with pdump_load_extra_data(), which returns an
+addr (mmap()ed or loaded), and later you pdump_unload_extra_data()
+when finished.  we'd probably also need
+pdump_attach_extra_data_append(), which appends data to the data
+just written out with pdump_attach_extra_data().  this way,
+multiple tables in memory can be written out into one contiguous
+table. (we'd use the tar-like trick of allowing new blocks to be
+written without going back to change the old blocks -- we just rely
+on the end of file/end of memory.) this same mechanism could be
+extracted out of pdump and used to handle the non-pdump situation
+(or alternatively, we could just dump either the memory image of
+the tables themselves or the compressed binary version).  in the
+case of extra unicode tables not known about at compile time that
+get loaded before dumping, we either just dump them into the image
+(pdump and all) or extract them into the compressed binary format,
+free the original tables, and treat them like all other tables.
+-- `C-x b' when using a Russian keyboard layout.  XEmacs currently
+tries to interpret C+cyrillic char, which causes an error.  We want
+C-x b to still work even when the keyboard normally generates
+Cyrillic.  What we should do is expand the keyboard event structure
+so that it contains not only the actual char, but what the char
+would have been in various other keyboard layouts, and in contexts
+where only certain keystrokes make sense (creating control chars,
+and looking up in keymaps), we proceed in order, processing each of
+them until we get something.  order should be something like:
+current keyboard layout; layout of the current language
+environment; layout of the user's default language; layout of the
+system default language; layout of US English.
+-- reading and writing Unicode files.  multiple problems:
+-- EOL's aren't handled right.  for the moment, just fix the
+Unicode coding systems; later on, create EOL-only coding
+systems:
+1. they would be character->character and operate next to the
+internal data; this means that coding systems need to be able
+to handle ends of lines that are either CR, LF, or CRLF.
+usually this isn't a problem, as they are just characters
+like any other and get encoded appropriately.  however,
+coding systems that are line-oriented need to recognize any
+of the three as line endings.
+2. we'd also have to complete the stuff that handles coding
+systems where either end can be byte or char (four
+possibilities total; use a single enum such as
+ENCODES_CHAR_TO_BYTE, ENCODES_BYTE_TO_BYTE, etc.).
+3. we'd need ways of specifying the chaining of coding systems.
+e.g. when reading a coding system, a user can specify more
+than one with a | symbol between them.  when a context calls
+for a coding system and a chain is needed, the `chain' coding
+system is useful; but we should really expand the contexts
+where a list of coding systems can be given, and whenever
+possible try to inline the chain instead of using a
+surrounding `chain' coding system.
+4. the `chain' needs some work so that it passes all sorts of
+lstream commands down to the chain inside it -- it should be
+entirely transparent and the fact that there's actually a
+surrounding coding system should be invisible.  more general
+coding system methods might need to be created.
+5. important: we need a way of specifying how detecting works
+when we have more than one coding system.  we might need more
+than a single priority list.  need to think about this.
+-- Unicode files beginning with the BOM are not recognized as such.
+we need to fix this; but to make things sensible, we really need
+to add the idea of different levels of confidence regarding
+what's detected.  otherwise, Unicode says "yes this is me" but
+others higher up do too.  in the process we should probably
+finish abstracting the detection system and fix up some
+stupidities in it.
+-- When writing a file, we need error detection; otherwise somebody
+will create a Unicode file without realizing the coding system
+of the buffer is Raw, and then lose all the non-ASCII/Latin-1
+text when it's written out.  We need two levels
+1. first, a "safe-charset" level that checks before any actual
+encoding to see if all characters in the document can safely
+be represented using the given coding system.  FSF has a
+"safe-charset" property of coding systems, but it's stupid
+because this information can be automatically derived from
+the coding system, at least the vast majority of the time.
+What we need is some sort of
+alternative-coding-system-precedence-list, langenv-specific,
+where everything on it can be checked for safe charsets and
+then the user given a list of possibilities.  When the user
+does "save with specified encoding", they should see the same
+precedence list.  Again like with other precedence lists,
+there's also a global one, and presumably all coding systems
+not on other list get appended to the end (and perhaps not
+checked at all when doing safe-checking?).  safe-checking
+should work something like this: compile a list of all
+charsets used in the buffer, along with a count of chars
+used.  that way, "slightly unsafe" charsets can perhaps be
+presented at the end, which will lose only a few characters
+and are perhaps what the users were looking for.
+2. when actually writing out, we need error checking in case an
+individual char in a charset can't be written even though the
+charsets are safe.  again, the user gets the choice of other
+reasonable coding systems.
+3. same thing (error checking, list of alternatives, etc.) needs
+to happen when reading!  all of this will be a lot of work!
+Announcement, August 20, 2001:
+I'm looking for testers.  There is a complete and fast implementation
+in C of Unicode conversion, translations for almost all of the
+standardly-defined charsets that load up automatically and
+instantaneously at runtime, coding systems supporting the common
+external representations of Unicode [utf-16, ucs-4, utf-8,
+little-endian versions of utf-16 and ucs-4; utf-7 is sitting there
+with abort[]s where the coding routines should go, just waiting for
+somebody to implement], and a nice set of primitives for translating
+characters<->codepoints and setting the priority lists used to control
+codepoint->char lookup.
+It's so far hooked into one place: the Windows IME.  Currently I can
+select the Japanese IME from the thing on my tray pad in the lower
+right corner of the screen, and type Japanese into XEmacs, and you get
+Japanese in XEmacs -- regardless of whether you set either your
+current or global system locale to Japanese,and regardless of whether
+you set your XEmacs lang env as Japanese.  This should work for many
+other languages, too -- Cyrillic, Chinese either Traditional or
+Simplified, and many others, but YMMV.  There may be some lurking
+bugs (hardly surprising for something so raw).
+To get at this, checkout using `ben-mule-21-5', NOT the simpler
+*`mule-21-5'.  For example
+cvs -d :pserver:xemacs@cvs.xemacs.org:/usr/CVSroot checkout -r ben-mule-21-5  xemacs
+or you get the idea.  the `-r ben-mule-21-5' is important.
+I keep track of my progress in a file called README.ben-mule-21-5 in
+the root directory of the source tree.
+WARNING: Pdump might not work. Will be fixed rsn.
+August 20, 2001:
+-- still need to sort out demand loading, binary format, etc.  figure
+out what the goals are and how we're going to achieve them.  for
+the moment let's just say that running XEmacs in a directory with
+Japanese or other weird characters in the name is likely to cause
+problems under MS Windows, but once XEmacs is initialized (and
+before processing init files), all Unicode support is there.
+-- wrote the size computation routines, although not yet tested.
+-- lots more abstraction of coding systems; almost done.
+-- UNICODE WORKS!!!!!
+August 19, 2001:
+Still needed on the Unicode support:
+-- demand loading: load the Unicode table data the first time a
+conversion needs to be done.
+-- maybe: table size computation: figure out how big the in-memory
+tables actually are.
+-- maybe: create a space-efficient binary format for the data, and a
+way to dump out an existing charset's data into this binary format.
+it should allow for many such groups of data to be appended
+together in one file, such that you can just append the new data
+onto the end and not have to go back and modify anything
+previously. (like how tar archives work, and how the UFS? for
+CD-R's and CD-RW's works.)
+-- maybe: figure out how to be able to access the Unicode tables at
+init_intl() time, before we know how to get at data-directory; that
+way we can handle the need for unicode conversions that come up
+very early, for example if XEmacs is run from a directory
+containing Japanese in it.  Presumably we'd want to generalize the
+stuff in pdump.c that deals with the dumper file, so that it can
+handle other files -- putting the file either in the directory of
+the executable or in a resource, maybe actually attached to the
+pdump file itself -- or maybe we just dump the data into the actual
+executable.  With pdump we could extend pdump to allow for data
+that's in the pdump file but not actually mapped at startup,
+separate from the data that does get mapped -- and then at runtime
+the pointer gets restored not with a real pointer but an offset
+into the file; another pdump call and we get some way to access the
+data. (tricky because it might be in a resource, not a file.  we
+might have to just tell pdump to mmap or whatever the data in, and
+then tell pdump to release it.)
+-- fix multibyte to use unicode.  at first, just reverse
+mswindows-multibyte-to-unicode to be unicode-to-multibyte; later
+implement something in chain to allow for reversal, for declaring
+the ends of the coding systems, etc.
+-- actually make sure that the IME stuff is working!!!
+Other things before announcing:
+-- change so that the Unicode tables are not pdumped.  This means we
+need to free any table data out there.  Make sure that pdump
+compiles and try to finish the pretty-much-already-done stuff
+already with XD_STRUCT_ARRAY and dynamic size computation; just
+need to see what's going on with LO_LINK.
+August 14, 2001:
+To do a diff between this workspace and the mainline, use the most recent sync tags, currently:
+cvs diff -r main-branch-ben-mule-21-5-aug-11-2001-sync -r ben-mule-21-5-post-aug-11-2001-sync
+Unicode support:
+Unicode support is important for supporting many languages under
+Windows, such as Cyrillic, without resorting to translation tables for
+particular Windows-specific code pages.  Internally, all characters in
+Windows can be represented in two encodings: code pages and Unicode.
+With Unicode support, we can seamlessly support all Windows
+characters.  Currently, the test in the drive to support Unicode is if
+IME input works properly, since it is being converted from Unicode.
+Unicode support also requires that the various Windows API's be
+"Unicode-encapsulated", so that they automatically call the ANSI or
+Unicode version of the API call appropriately and handle the size
+differences in structures.  What this means is:
+-- first, note that Windows already provides a sort of encapsulation
+of all API's that deal with text.  All such API's are underlyingly
+provided in two versions, with an A or W suffix (ANSI or "wide"
+i.e. Unicode), and the compile-time constant UNICODE controls which
+is selected by the unsuffixed API.  Same thing happens with
+structures.  Unfortunately, this is compile-time only, not
+run-time, so not sufficient. (Creating the necessary run-time
+encoding is not conceptually difficult, but very time-consuming to
+write.  It adds no significant overhead, and the only reason it's
+not standard in Windows is conscious marketing attempts by
+Microsoft to cripple Windows 95.  FUCK MICROSOFT!  They even
+describe in a KnowledgeBase article exactly how to create such an
+API [although we don't exactly follow their procedure], and point
+out its usefulness; the procedure is also described more generally
+in Nadine Kano's book on Win32 internationalization -- written SIX
+YEARS AGO!  Obviously Microsoft has such an API available
+internally.)
+-- what we do is provide an encapsulation of each standard Windows API
+call that is split into A and W versions.  current theory is to
+avoid all preprocessor games; so we name the function with a prefix
+-- "qxe" currently -- and require callers to use the prefixed name.
+Callers need to explicitly use the W version of all structures, and
+convert text themselves using Qmswindows_tstr.  the qxe
+encapsulated version will automatically call the appropriate A or W
+version depending on whether we're running on 9x or NT, and copy
+data between W and A versions of the structures as necessary.
+-- We require the caller to handle the actual translation of text to
+avoid possible overflow when dealing with fixed-size Windows
+structures.  There are no such problems when copying data between
+the A and W versions because ANSI text is never larger than its
+equivalent Unicode representation.
+-- We allow for incremental creation of the encapsulated routines by
+using the coding system Qmswindows_tstr_notyet.  This is an alias
+for Qmswindows_multibyte, i.e. it always converts to ANSI; but it
+indicates that it will be changed to Qmswindows_tstr when we have a
+qxe version of the API call that the data is being passed to and
+change the code to use the new function.
+Besides creating the encapsulation, the following needs to be done for
+Unicode support:
+-- No actual translation tables are fed into XEmacs.  We need to
+provide glue code to read the tables in etc/unicode.  See
+etc/unicode/README for the interface to implement.
+-- Fix pdump.  The translation tables for Unicode characters function
+as unions of structures with different numbers of indirection
+levels, in order to be efficient.  pdump doesn't yet support such
+unions.  charset.h has a general description of how the translation
+tables work, and the pdump code has constants added for the new
+required data types, and descriptions of how these should work.
+-- ultimately, there's no end to additional work (composition, bidi
+reordering, glyph shaping/ordering, etc.), but the above is enough
+to get basic translation working.
+Merging this workspace into the trunk requires some work.  ChangeLogs
+have not yet been created.  Also, there is a lot of additional code in
+this workspace other than just Windows and Unicode stuff.  Some of the
+changes have been somewhat disruptive to the code base, in particular:
+-- the code that handles the details of processing multilingual text
+has been consolidated to make it easier to extend it.  it has been
+yanked out of various files (buffer.h, mule-charset.h, lisp.h,
+insdel.c, fns.c, file-coding.c, etc.) and put into text.c and
+text.h.  mule-charset.h has also been renamed charset.h.  all long
+comments concerning the representations and their processing have
+been consolidated into text.c.
+-- nt/config.h has been eliminated and everything in it merged into
+config.h.in and s/windowsnt.h.  see config.h.in for more info.
+-- s/windowsnt.h has been completely rewritten, and s/cygwin32.h and
+s/mingw32.h have been largely rewritten.  tons of dead weight has
+been removed, and stuff common to more than one file has been
+isolated into s/win32-common.h and s/win32-native.h, similar to
+what's already done for usg variants.
+-- large amounts of code throughout the code base have been Mule-ized,
+not just Windows code.
+-- file-coding.c/.h have been largely rewritten (although still mostly
+syncable); see below.
+June 26, 2001:
+-- ben-mule-21-5
+this contains all the mule work i've been doing.  this includes mostly
+work done to get mule working under ms windows, but in the process
+i've [of course] fixed a whole lot of other things as well, mostly
+mule issues.  the specifics:
+- it compiles and runs under windows and should basically work.  the
+stuff remaining to do is (a) improved unicode support (see below)
+and (b) smarter handling of keyboard layouts.  in particular, it
+should (1) set the right keyboard layout when you change your
+language environment; (2) optionally (a user var) set the
+appropriate keyboard layout as you move the cursor into text in a
+particular language.
+- i added a bunch of code to better support OS locales.  it tries to
+notice your locale at startup and set the language environment
+accordingly (this more or less works), and call setlocale() and set
+LANG when you change the language environment (may or may not work).
+- major rewriting of file-coding.  it's mostly abstracted into coding
+systems that are defined by methods (similar to devices and
+specifiers), with the ultimate aim being to allow non-i18n coding
+systems such as gzip.  there is a "chain" coding system that allows
+multiple coding systems to be chained together. (it doesn't yet
+have the concept that either end of a coding system can be bytes or
+chars; this needs to be added.)
+- unicode support.  very raw.  a few days ago i wrote a complete and
+efficient implementation of unicode translation.  it should be very
+fast, and fairly memory-efficient in its tables.  it allows for
+charset priority lists, which should be language-environment
+specific (but i haven't yet written the glue code).  it works in
+preliminary testing, but obviously needs more testing and work.
+as of yet there is no translation data added for the standard charsets.
+the tables are in etc/unicode, and all we need is a bit of glue code
+to process them.  see etc/unicode/README for the interface to
+implement.
+- support for unicode in windows is partly there.  this will work even
+on windows 95.  the basic model is implemented but it needs finishing
+up.
+- there is a preliminary implementation of windows ime support courtesy
+of ikeyama.
+- if you want to get cyrillic working under windows (it appears to "work"
+but the wrong chars currently appear), the best way is to add unicode
+support for iso-8859-5 and use it in redisplay-msw.c.  we are already
+passing unicode codepoints to the text-draw routine (ExtTextOutW).
+(ExtTextOutW and GetTextExtentPoint32W are implemented on both 95 and NT.)
+- i fixed the iso2022 handling so it will correctly read in files
+containing unknown charsets, creating a "temporary" charset which
+can later be overwritten by the real charset when it's defined.
+this allows iso2022 elisp files with literals in strange languages
+to compile correctly under mule.  i also added a hack that will
+correctly read in and write out the emacs-specific "composition"
+escape sequences, i.e. ESC 0 through ESC 4.  this means that my
+workspace correctly compiles the new file devanagari.el that i added
+(see below).
+- i copied the remaining language-specific files from fsf.  i made
+some minor changes in certain cases but for the most part the stuff
+was just copied and may not work.
+- i fixed post-read-conversion in coding systems to follow fsf
+conventions. (i also support our convention, for the moment.  a
+kludge, of course.)
+- make-coding-system accepts (but ignores) the additional properties
+present in the fsf version, for compatibility.

Mercurial > hg > xemacs-beta

comparison README.ben-mule-21-5 @ 771:943eaba38521