xemacs-beta: man/internals/internals.texi comparison

comparison man/internals/internals.texi @ 400:a86b2b5e0111 r21-2-30

Import from CVS: tag r21-2-30

author	cvs
date	Mon, 13 Aug 2007 11:14:34 +0200
parents	74fd4e045ea6
children	2f8bb876ab1d

comparison

equal deleted inserted replaced

-:376370fb5946
+:a86b2b5e0111
 * Menus::
 * Subprocesses::
 * Interface to X Windows::
 * Index::
-@detailmenu --- The Detailed Node Listing ---
+@detailmenu
+--- The Detailed Node Listing ---
 A History of Emacs
 * Through Version 18::          Unification prevails.
 * Lucid Emacs::                 One version 19 Emacs.
 * Garbage Collection - Step by Step::
 * Integers and Characters::
 * Allocation from Frob Blocks::
 * lrecords::
 * Low-level allocation::
-* Pure Space::
 * Cons::
 * Vector::
 * Bit Vector::
 * Symbol::
 * Marker::
 Java, which is inexcusable.
 @end enumerate
 Unfortunately, there is no perfect language.  Static typing allows a
 compiler to catch programmer errors and produce more efficient code, but
-makes programming more tedious and less fun.  For the forseeable future,
+makes programming more tedious and less fun.  For the foreseeable future,
 an Ideal Editing and Programming Environment (and that is what XEmacs
 aspires to) will be programmable in multiple languages: high level ones
 like Lisp for user customization and prototyping, and lower level ones
 for infrastructure and industrial strength applications.  If I had my
 way, XEmacs would be friendly towards the Python, Scheme, C++, ML,
 @example
 [ 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 ]
 [ 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 ]
-<---> ^ <------------------------------------------------------>
+<---------------------------------------------------------> <->
-tag  |       a pointer to a structure, or an integer
+a pointer to a structure, or an integer            tag
-|
+@end example
-mark bit
-@end example
+A tag of 00 is used for all pointer object types, a tag of 10 is used
+for characters, and the other two tags 01 and 11 are joined together to
-The tag describes the type of the Lisp object.  For integers and chars,
+form the integer object type.  This representation gives us 31 bits
-the lower 28 bits contain the value of the integer or char; for all
+integers, 30 bits characters and pointers are represented directly
-others, the lower 28 bits contain a pointer.  The mark bit is used
+without any bit masking.  This representation, though, assumes that
-during garbage-collection, and is always 0 when garbage collection is
+pointers to structs are always aligned to multiples of 4, so the lower 2
-not happening. (The way that garbage collection works, basically, is that it
+bits are always zero.
-loops over all places where Lisp objects could exist---this includes
-all global variables in C that contain Lisp objects [including
-@code{Vobarray}, the C equivalent of @code{obarray}; through this, all
-Lisp variables will get marked], plus various other places---and
-recursively scans through the Lisp objects, marking each object it finds
-by setting the mark bit.  Then it goes through the lists of all objects
-allocated, freeing the ones that are not marked and turning off the mark
-bit of the ones that are marked.)
 Lisp objects use the typedef @code{Lisp_Object}, but the actual C type
 used for the Lisp object can vary.  It can be either a simple type
 (@code{long} on the DEC Alpha, @code{int} on other machines) or a
 structure whose fields are bit fields that line up properly (actually, a
 object is desired, you get a compile error), and it makes it easier to
 decode Lisp objects when debugging.  The choice of which type to use is
 determined by the preprocessor constant @code{USE_UNION_TYPE} which is
 defined via the @code{--use-union-type} option to @code{configure}.
-@cindex record type
-Note that there are only eight types that the tag can represent, but
-many more actual types than this.  This is handled by having one of the
-tag types specify a meta-type called a @dfn{record}; for all such
-objects, the first four bytes of the pointed-to structure indicate what
-the actual type is.
-Note also that having 28 bits for pointers and integers restricts a lot
-of things to 256 megabytes of memory. (Basically, enough pointers and
-indices and whatnot get stuffed into Lisp objects that the total amount
-of memory used by XEmacs can't grow above 256 megabytes.  In older
-versions of XEmacs and GNU Emacs, the tag was 5 bits wide, allowing for
-32 types, which was more than the actual number of types that existed at
-the time, and no ``record'' type was necessary.  However, this limited
-the editor to 64 megabytes total, which some users who edited large
-files might conceivably exceed.)
-Also, note that there is an implicit assumption here that all pointers
-are low enough that the top bits are all zero and can just be chopped
-off.  On standard machines that allocate memory from the bottom up (and
-give each process its own address space), this works fine.  Some
-machines, however, put the data space somewhere else in memory
-(e.g. beginning at 0x80000000).  Those machines cope by defining
-@code{DATA_SEG_BITS} in the corresponding @file{m/} or @file{s/} file to
-the proper mask.  Then, pointers retrieved from Lisp objects are
-automatically OR'ed with this value prior to being used.
-A corollary of the previous paragraph is that @strong{(pointers to)
-stack-allocated structures cannot be put into Lisp objects}.  The stack
-is generally located near the top of memory; if you put such a pointer
-into a Lisp object, it will get its top bits chopped off, and you will
-lose.
-Actually, there's an alternative representation of a @code{Lisp_Object},
-invented by Kyle Jones, that is used when the
-@code{--use-minimal-tagbits} option to @code{configure} is used.  In
-this case the 2 lower bits are used for the tag bits.  This
-representation assumes that pointers to structs are always aligned to
-multiples of 4, so the lower 2 bits are always zero.
-@example
-[ 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 ]
-[ 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 ]
-<---------------------------------------------------------> <->
-a pointer to a structure, or an integer            tag
-@end example
-A tag of 00 is used for all pointer object types, a tag of 10 is used
-for characters, and the other two tags 01 and 11 are joined together to
-form the integer object type.  The markbit is moved to part of the
-structure being pointed at (integers and chars do not need to be marked,
-since no memory is allocated).  This representation has these
-advantages:
-@enumerate
-@item
-31 bits can be used for Lisp Integers.
-@item
-@emph{Any} pointer can be represented directly, and no bit masking
-operations are necessary.
-@end enumerate
-The disadvantages are:
-@enumerate
-@item
-An extra level of indirection is needed when accessing the object types
-that were not record types.  So checking whether a Lisp object is a cons
-cell becomes a slower operation.
-@item
-Mark bits can no longer be stored directly in Lisp objects, so another
-place for them must be found.  This means that a cons cell requires more
-memory than merely room for 2 lisp objects, leading to extra memory use.
-@end enumerate
 Various macros are used to construct Lisp objects and extract the
 components.  Macros of the form @code{XINT()}, @code{XCHAR()},
-@code{XSTRING()}, @code{XSYMBOL()}, etc. mask out the pointer/integer
+@code{XSTRING()}, @code{XSYMBOL()}, etc. shift out the tag field if
-field and cast it to the appropriate type.  All of the macros that
+needed cast it to the appropriate type.  @code{XINT()} needs to be a bit
-construct pointers will @code{OR} with @code{DATA_SEG_BITS} if
+tricky so that negative numbers are properly sign-extended.  Since
-necessary.  @code{XINT()} needs to be a bit tricky so that negative
+integers are stored left-shifted, if the right-shift operator does an
-numbers are properly sign-extended: Usually it does this by shifting the
+arithmetic shift (i.e. it leaves the most-significant bit as-is rather
-number four bits to the left and then four bits to the right.  This
+than shifting in a zero, so that it mimics a divide-by-two even for
-assumes that the right-shift operator does an arithmetic shift (i.e. it
+negative numbers) the shift to remove the tag bit is enough.  This is
-leaves the most-significant bit as-is rather than shifting in a zero, so
+the case on all the systems we support.
-that it mimics a divide-by-two even for negative numbers).  Not all
-machines/compilers do this, and on the ones that don't, a more
-complicated definition is selected by defining
-@code{EXPLICIT_SIGN_EXTEND}.
 Note that when @code{ERROR_CHECK_TYPECHECK} is defined, the extractor
 macros become more complicated---they check the tag bits and/or the
 type field in the first four bytes of a record type to ensure that the
 object is really of the correct type.  This is great for catching places
 some_variable = 0;}.  The reason for this has to do with some kludges
 done during the dumping process: If possible, the initialized data
 segment is re-mapped so that it becomes part of the (unmodifiable) code
 segment in the dumped executable.  This allows this memory to be shared
 among multiple running XEmacs processes.  XEmacs is careful to place as
-much constant data as possible into initialized variables (in
+much constant data as possible into initialized variables during the
-particular, into what's called the @dfn{pure space}---see below) during
+@file{temacs} phase.
-the @file{temacs} phase.
 @cindex copy-on-write
 @strong{Please note:} This kludge only works on a few systems nowadays,
 and is rapidly becoming irrelevant because most modern operating systems
 provide @dfn{copy-on-write} semantics.  All data is initially shared
 @item Bufbyte
 @cindex Bufbyte
 The data representing the text in a buffer or string is logically a set
 of @code{Bufbyte}s.
-XEmacs does not work with character formats all the time; when reading
+XEmacs does not work with the same character formats all the time; when
-characters from the outside, it decodes them to an internal format, and
+reading characters from the outside, it decodes them to an internal
-likewise encodes them when writing.  @code{Bufbyte} (in fact
+format, and likewise encodes them when writing.  @code{Bufbyte} (in fact
 @code{unsigned char}) is the basic unit of XEmacs internal buffers and
-strings format.
+strings format.  A @code{Bufbyte *} is the type that points at text
+encoded in the variable-width internal encoding.
 One character can correspond to one or more @code{Bufbyte}s.  In the
-current implementation, an ASCII character is represented by the same
+current Mule implementation, an ASCII character is represented by the
-@code{Bufbyte}, and extended characters are represented by a sequence of
+same @code{Bufbyte}, and other characters are represented by a sequence
-@code{Bufbyte}s.
+of two or more @code{Bufbyte}s.
-Without Mule support, a @code{Bufbyte} is equivalent to an
+Without Mule support, there are exactly 256 characters, implicitly
-@code{Emchar}.
+Latin-1, and each character is represented using one @code{Bufbyte}, and
+there is a one-to-one correspondence between @code{Bufbyte}s and
+@code{Emchar}s.
 @item Bufpos
 @itemx Charcount
 @cindex Bufpos
 @cindex Charcount
 A @code{Bufpos} represents a character position in a buffer or string.
 A @code{Charcount} represents a number (count) of characters.
 Logically, subtracting two @code{Bufpos} values yields a
 @code{Charcount} value.  Although all of these are @code{typedef}ed to
-@code{int}, we use them in preference to @code{int} to make it clear
+@code{EMACS_INT}, we use them in preference to @code{EMACS_INT} to make
-what sort of position is being used.
+it clear what sort of position is being used.
 @code{Bufpos} and @code{Charcount} values are the only ones that are
 ever visible to Lisp.
 @item Bytind
 @itemx Bytecount
 @cindex Bytind
 @cindex Bytecount
 A @code{Bytind} represents a byte position in a buffer or string.  A
-@code{Bytecount} represents the distance between two positions in bytes.
+@code{Bytecount} represents the distance between two positions, in bytes.
 The relationship between @code{Bytind} and @code{Bytecount} is the same
 as the relationship between @code{Bufpos} and @code{Charcount}.
 @item Extbyte
 @itemx Extcount
 learn about them.
 @table @code
 @item MAX_EMCHAR_LEN
 @cindex MAX_EMCHAR_LEN
-This preprocessor constant is the maximum number of buffer bytes per
+This preprocessor constant is the maximum number of buffer bytes to
-Emacs character, i.e. the byte length of an @code{Emchar}.  It is useful
+represent an Emacs character in the variable width internal encoding.
-when allocating temporary strings to keep a known number of characters.
+It is useful when allocating temporary strings to keep a known number of
-For instance:
+characters.  For instance:
 @example
 @group
 @{
 Charcount cclen;
 always convert it to an appropriate external encoding, lest the internal
 stuff (such as the infamous \201 characters) leak out.
 The interface to conversion between the internal and external
 representations of text are the numerous conversion macros defined in
-@file{buffer.h}.  Before looking at them, we'll look at the external
+@file{buffer.h}.  There used to be a fixed set of external formats
-formats supported by these macros.
+supported by these macros, but now any coding system can be used with
+these macros.  The coding system alias mechanism is used to create the
-Currently meaningful formats are @code{FORMAT_BINARY},
+following logical coding systems, which replace the fixed external
-@code{FORMAT_FILENAME}, @code{FORMAT_OS}, and @code{FORMAT_CTEXT}.  Here
+formats.  The (dontusethis-set-symbol-value-handler) mechanism was
-is a description of these.
+enhanced to make this possible (more work on that is needed - like
+remove the @code{dontusethis-} prefix).
 @table @code
-@item FORMAT_BINARY
+@item Qbinary
-Binary format.  This is the simplest format and is what we use in the
+This is the simplest format and is what we use in the absence of a more
-absence of a more appropriate format.  This converts according to the
+appropriate format.  This converts according to the @code{binary} coding
-@code{binary} coding system:
+system:
 @enumerate a
 @item
-On input, bytes 0--255 are converted into characters 0--255.
+On input, bytes 0--255 are converted into (implicitly Latin-1)
+characters 0--255.  A non-Mule xemacs doesn't really know about
+different character sets and the fonts to display them, so the bytes can
+be treated as text in different 1-byte encodings by simply setting the
+appropriate fonts.  So in a sense, non-Mule xemacs is a multi-lingual
+editor if, for example, different fonts are used to display text in
+different buffers, faces, or windows.  The specifier mechanism gives the
+user complete control over this kind of behavior.
 @item
 On output, characters 0--255 are converted into bytes 0--255 and other
-characters are converted into `X'.
+characters are converted into `~'.
 @end enumerate
-@item FORMAT_FILENAME
+@item Qfile_name
-Format used for filenames.  In the original Mule, this is user-definable
+Format used for filenames.  This is user-definable via either the
-with the @code{pathname-coding-system} variable.  For the moment, we
+@code{file-name-coding-system} or @code{pathname-coding-system} (now
-just use the @code{binary} coding system.
+obsolete) variables.
-@item FORMAT_OS
+@item Qnative
 Format used for the external Unix environment---@code{argv[]}, stuff
 from @code{getenv()}, stuff from the @file{/etc/passwd} file, etc.
+Currently this is the same as Qfile_name.  The two should be
-Perhaps should be the same as FORMAT_FILENAME.
+distinguished for clarity and possible future separation.
-@item FORMAT_CTEXT
+@item Qctext
-Compound--text format.  This is the standard X format used for data
+Compound--text format.  This is the standard X11 format used for data
 stored in properties, selections, and the like.  This is an 8-bit
-no-lock-shift ISO2022 coding system.
+no-lock-shift ISO2022 coding system.  This is a real coding system,
+unlike Qfile_name, which is user-definable.
 @end table
-The macros to convert between these formats and the internal format, and
+There are two fundamental macros to convert between external and
-vice versa, follow.
+internal format.
+@code{TO_INTERNAL_FORMAT} converts external data to internal format, and
+@code{TO_EXTERNAL_FORMAT} converts the other way around.  The arguments
+each of these receives are a source type, a source, a sink type, a sink,
+and a coding system (or a symbol naming a coding system).
+A typical call looks like
+@example
+TO_EXTERNAL_FORMAT (LISP_STRING, str, C_STRING_MALLOC, ptr, Qfile_name);
+@end example
+which means that the contents of the lisp string @code{str} are written
+to a malloc'ed memory area which will be pointed to by @code{ptr}, after
+the function returns.  The conversion will be done using the
+@code{file-name} coding system, which will be controlled by the user
+indirectly by setting or binding the variable
+@code{file-name-coding-system}.
+Some sources and sinks require two C variables to specify.  We use some
+preprocessor magic to allow different source and sink types, and even
+different numbers of arguments to specify different types of sources and
+sinks.
+So we can have a call that looks like
+@example
+TO_INTERNAL_FORMAT (DATA, (ptr, len),
+MALLOC, (ptr, len),
+coding_system);
+@end example
+The parenthesized argument pairs are required to make the preprocessor
+magic work.
+Here are the different source and sink types:
 @table @code
-@item GET_CHARPTR_INT_DATA_ALLOCA
+@item @code{DATA, (ptr, len),}
-@itemx GET_CHARPTR_EXT_DATA_ALLOCA
+input data is a fixed buffer of size @var{len} at address @var{ptr}
-These two are the most basic conversion macros.
+@item @code{ALLOCA, (ptr, len),}
-@code{GET_CHARPTR_INT_DATA_ALLOCA} converts external data to internal
+output data is placed in an alloca()ed buffer of size @var{len} pointed to by @var{ptr}
-format, and @code{GET_CHARPTR_EXT_DATA_ALLOCA} converts the other way
+@item @code{MALLOC, (ptr, len),}
-around.  The arguments each of these receives are @var{ptr} (pointer to
+output data is in a malloc()ed buffer of size @var{len} pointed to by @var{ptr}
-the text in external format), @var{len} (length of texts in bytes),
+@item @code{C_STRING_ALLOCA, ptr,}
-@var{fmt} (format of the external text), @var{ptr_out} (lvalue to which
+equivalent to @code{ALLOCA (ptr, len_ignored)} on output.
-new text should be copied), and @var{len_out} (lvalue which will be
+@item @code{C_STRING_MALLOC, ptr,}
-assigned the length of the internal text in bytes).  The resulting text
+equivalent to @code{MALLOC (ptr, len_ignored)} on output
-is stored to a stack-allocated buffer.  If the text doesn't need
+@item @code{C_STRING, ptr,}
-changing, these macros will do nothing, except for setting
+equivalent to @code{DATA, (ptr, strlen (ptr) + 1)} on input
-@var{len_out}.
+@item @code{LISP_STRING, string,}
+input or output is a Lisp_Object of type string
-The macros above take many arguments which makes them unwieldy.  For
+@item @code{LISP_BUFFER, buffer,}
-this reason, a number of convenience macros are defined with obvious
+output is written to @code{(point)} in lisp buffer @var{buffer}
-functionality, but accepting less arguments.  The general rule is that
+@item @code{LISP_LSTREAM, lstream,}
-macros with @samp{INT} in their name convert text to internal Emacs
+input or output is a Lisp_Object of type lstream
-representation, whereas the @samp{EXT} macros convert to external
+@item @code{LISP_OPAQUE, object,}
-representation.
+input or output is a Lisp_Object of type opaque
-@item GET_C_CHARPTR_INT_DATA_ALLOCA
-@itemx GET_C_CHARPTR_EXT_DATA_ALLOCA
-As their names imply, these macros work on C char pointers, which are
-zero-terminated, and thus do not need @var{len} or @var{len_out}
-parameters.
-@item GET_STRING_EXT_DATA_ALLOCA
-@itemx GET_C_STRING_EXT_DATA_ALLOCA
-These two macros convert a Lisp string into an external representation.
-The difference between them is that @code{GET_STRING_EXT_DATA_ALLOCA}
-stores its output to a generic string, providing @var{len_out}, the
-length of the resulting external string.  On the other hand,
-@code{GET_C_STRING_EXT_DATA_ALLOCA} assumes that the caller will be
-satisfied with output string being zero-terminated.
-Note that for Lisp strings only one conversion direction makes sense.
-@item GET_C_CHARPTR_EXT_BINARY_DATA_ALLOCA
-@itemx GET_CHARPTR_EXT_BINARY_DATA_ALLOCA
-@itemx GET_STRING_BINARY_DATA_ALLOCA
-@itemx GET_C_STRING_BINARY_DATA_ALLOCA
-@itemx GET_C_CHARPTR_EXT_FILENAME_DATA_ALLOCA
-@itemx ...
-These macros convert internal text to a specific external
-representation, with the external format being encoded into the name of
-the macro.  Note that the @code{GET_STRING_...} and
-@code{GET_C_STRING...}  macros lack the @samp{EXT} tag, because they
-only make sense in that direction.
-@item GET_C_CHARPTR_INT_BINARY_DATA_ALLOCA
-@itemx GET_CHARPTR_INT_BINARY_DATA_ALLOCA
-@itemx GET_C_CHARPTR_INT_FILENAME_DATA_ALLOCA
-@itemx ...
-These macros convert external text of a specific format to its internal
-representation, with the external format being incoded into the name of
-the macro.
 @end table
+Often, the data is being converted to a '\0'-byte-terminated string,
+which is the format required by many external system C APIs.  For these
+purposes, a source type of @code{C_STRING} or a sink type of
+@code{C_STRING_ALLOCA} or @code{C_STRING_MALLOC} is appropriate.
+Otherwise, we should try to keep XEmacs '\0'-byte-clean, which means
+using (ptr, len) pairs.
+The sinks to be specified must be lvalues, unless they are the lisp
+object types @code{LISP_LSTREAM} or @code{LISP_BUFFER}.
+For the sink types @code{ALLOCA} and @code{C_STRING_ALLOCA}, the
+resulting text is stored in a stack-allocated buffer, which is
+automatically freed on returning from the function.  However, the sink
+types @code{MALLOC} and @code{C_STRING_MALLOC} return @code{xmalloc()}ed
+memory.  The caller is responsible for freeing this memory using
+@code{xfree()}.
+Note that it doesn't make sense for @code{LISP_STRING} to be a source
+for @code{TO_INTERNAL_FORMAT} or a sink for @code{TO_EXTERNAL_FORMAT}.
+You'll get an assertion failure if you try.
 @node General Guidelines for Writing Mule-Aware Code, An Example of Mule-Aware Code, Conversion to and from External Data, Coding for Mule
 @subsection General Guidelines for Writing Mule-Aware Code
 This section contains some general guidance on how to write Mule-aware
 It is extremely important to always convert external data, because
 XEmacs can crash if unexpected 8bit sequences are copied to its internal
 buffers literally.
 This means that when a system function, such as @code{readdir}, returns
-a string, you need to convert it using one of the conversion macros
+a string, you may need to convert it using one of the conversion macros
 described in the previous chapter, before passing it further to Lisp.
-In the case of @code{readdir}, you would use the
-@code{GET_C_CHARPTR_INT_FILENAME_DATA_ALLOCA} macro.
+Actually, most of the basic system functions that accept '\0'-terminated
+string arguments, like @code{stat()} and @code{open()}, have been
+@strong{encapsulated} so that they are they @code{always} do internal to
+external conversion themselves.  This means you must pass internally
+encoded data, typically the @code{XSTRING_DATA} of a Lisp_String to
+these functions.  This is actually a design bug, since it unexpectedly
+changes the semantics of the system functions.  A better design would be
+to provide separate versions of these system functions that accepted
+Lisp_Objects which were lisp strings in place of their current
+@code{char *} arguments.
+@example
+int stat_lisp (Lisp_Object path, struct stat *buf); /* Implement me */
+@end example
 Also note that many internal functions, such as @code{make_string},
 accept Bufbytes, which removes the need for them to convert the data
 they receive.  This increases efficiency because that way external data
 needs to be decoded only once, when it is read.  After that, it is
 @end table
 @node An Example of Mule-Aware Code,  , General Guidelines for Writing Mule-Aware Code, Coding for Mule
 @subsection An Example of Mule-Aware Code
-As an example of Mule-aware code, we shall will analyze the
+As an example of Mule-aware code, we will analyze the @code{string}
-@code{string} function, which conses up a Lisp string from the character
+function, which conses up a Lisp string from the character arguments it
-arguments it receives.  Here is the definition, pasted from
+receives.  Here is the definition, pasted from @code{alloc.c}:
-@code{alloc.c}:
 @example
 @group
 DEFUN ("string", Fstring, 0, MANY, 0, /*
 Concatenate all the argument characters and make the result a string.
 @item
 Generated header files should be included using the @code{#include <...>} syntax,
 not the @code{#include "..."} syntax.  The generated headers are:
-@file{config.h puresize-adjust.h sheap-adjust.h paths.h Emacs.ad.h}
+@file{config.h sheap-adjust.h paths.h Emacs.ad.h}
 The basic rule is that you should assume builds using @code{--srcdir}
 and the @code{#include <...>} syntax needs to be used when the
 to-be-included generated file is in a potentially different directory
 @emph{at compile time}.  The non-obvious C rule is that @code{#include "..."}
 @example
 make check
 @end example
 @end itemize
+Here is a checklist of things to do when creating a new lisp object type
+named @var{foo}:
+@enumerate
+@item
+create @var{foo}.h
+@item
+create @var{foo}.c
+@item
+add definitions of syms_of_@var{foo}, etc. to @var{foo}.c
+@item
+add declarations of syms_of_@var{foo}, etc. to symsinit.h
+@item
+add calls to syms_of_@var{foo}, etc. to emacs.c(main_1)
+@item
+add definitions of macros like CHECK_FOO and FOOP to @var{foo}.h
+@item
+add the new type index to enum lrecord_type
+@item
+add DEFINE_LRECORD_IMPLEMENTATION call to @var{foo}.c
+@end enumerate
 @node A Summary of the Various XEmacs Modules, Allocation of Objects in XEmacs Lisp, Rules When Writing New C Code, Top
 @chapter A Summary of the Various XEmacs Modules
 This is accurate as of XEmacs 20.0.
 @example
 alloc.c
-pure.c
-puresize.h
 @end example
 The large module @file{alloc.c} implements all of the basic allocation and
 garbage collection for Lisp objects.  The most commonly used Lisp
 objects are allocated in chunks, similar to the Blocktype data type
 Because the different subsystems are divided into general and specific
 code, adding a new subtype within a subsystem will in general not
 require changes to the generic subsystem code or affect any of the other
 subtypes in the subsystem; this provides a great deal of robustness to
 the XEmacs code.
-@cindex pure space
-@file{pure.c} contains the declaration of the @dfn{purespace} array.
-Pure space is a hack used to place some constant Lisp data into the code
-segment of the XEmacs executable, even though the data needs to be
-initialized through function calls.  (See above in section VIII for more
-info about this.)  During startup, certain sorts of data is
-automatically copied into pure space, and other data is copied manually
-in some of the basic Lisp files by calling the function @code{purecopy},
-which copies the object if possible (this only works in temacs, of
-course) and returns the new object.  In particular, while temacs is
-executing, the Lisp reader automatically copies all compiled-function
-objects that it reads into pure space.  Since compiled-function objects
-are large, are never modified, and typically comprise the majority of
-the contents of a compiled-Lisp file, this works well.  While XEmacs is
-running, any attempt to modify an object that resides in pure space
-causes an error.  Objects in pure space are never garbage collected --
-almost all of the time, they're intended to be permanent, and in any
-case you can't write into pure space to set the mark bits.
-@file{puresize.h} contains the declaration of the size of the pure space
-array.  This depends on the optional features that are compiled in, any
-extra purespace requested by the user at compile time, and certain other
-factors (e.g. 64-bit machines need more pure space because their Lisp
-objects are larger).  The smallest size that suffices should be used, so
-that there's no wasted space.  If there's not enough pure space, you
-will get an error during the build process, specifying how much more
-pure space is needed.
 @example
 eval.c
 backtrace.h
 * Garbage Collection - Step by Step::
 * Integers and Characters::
 * Allocation from Frob Blocks::
 * lrecords::
 * Low-level allocation::
-* Pure Space::
 * Cons::
 * Vector::
 * Bit Vector::
 * Symbol::
 * Marker::
 symbols, the primitives are @code{make-symbol} and @code{intern}; etc.
 Some Lisp objects, especially those that are primarily used internally,
 have no corresponding Lisp primitives.  Every Lisp object, though,
 has at least one C primitive for creating it.
-Recall from section (VII) that a Lisp object, as stored in a 32-bit
+Recall from section (VII) that a Lisp object, as stored in a 32-bit or
-or 64-bit word, has a mark bit, a few tag bits, and a ``value'' that
+64-bit word, has a few tag bits, and a ``value'' that occupies the
-occupies the remainder of the bits.  We can separate the different
+remainder of the bits.  We can separate the different Lisp object types
-Lisp object types into four broad categories:
+into three broad categories:
 @itemize @bullet
 @item
 (a) Those for whom the value directly represents the contents of the
 Lisp object.  Only two types are in this category: integers and
 characters.  No special allocation or garbage collection is necessary
 for such objects.  Lisp objects of these types do not need to be
 @code{GCPRO}ed.
 @end itemize
-In the remaining three categories, the value is a pointer to a
-structure.
-@itemize @bullet
-@item
-@cindex frob block
-(b) Those for whom the tag directly specifies the type.  Recall that
-there are only three tag bits; this means that at most five types can be
-specified this way.  The most commonly-used types are stored in this
-format; this includes conses, strings, vectors, and sometimes symbols.
-With the exception of vectors, objects in this category are allocated in
-@dfn{frob blocks}, i.e. large blocks of memory that are subdivided into
-individual objects.  This saves a lot on malloc overhead, since there
-are typically quite a lot of these objects around, and the objects are
-small.  (A cons, for example, occupies 8 bytes on 32-bit machines---4
-bytes for each of the two objects it contains.) Vectors are individually
-@code{malloc()}ed since they are of variable size.  (It would be
-possible, and desirable, to allocate vectors of certain small sizes out
-of frob blocks, but it isn't currently done.) Strings are handled
-specially: Each string is allocated in two parts, a fixed size structure
-containing a length and a data pointer, and the actual data of the
-string.  The former structure is allocated in frob blocks as usual, and
-the latter data is stored in @dfn{string chars blocks} and is relocated
-during garbage collection to eliminate holes.
-@end itemize
 In the remaining two categories, the type is stored in the object
 itself.  The tag for all such objects is the generic @dfn{lrecord}
-(Lisp_Record) tag.  The first four bytes (or eight, for 64-bit machines)
+(Lisp_Type_Record) tag.  The first bytes of the object's structure are an
-of the object's structure are a pointer to a structure that describes
+integer (actually a char) characterising the object's type and some
-the object's type, which includes method pointers and a pointer to a
+flags, in particular the mark bit used for garbage collection.  A
-string naming the type.  Note that it's possible to save some space by
+structure describing the type is accessible thru the
-using a one- or two-byte tag, rather than a four- or eight-byte pointer
+lrecord_implementation_table indexed with said integer.  This structure
-to store the type, but it's not clear it's worth making the change.
+includes the method pointers and a pointer to a string naming the type.
 @itemize @bullet
 @item
-(c) Those lrecords that are allocated in frob blocks (see above).  This
+(b) Those lrecords that are allocated in frob blocks (see above).  This
 includes the objects that are most common and relatively small, and
-includes floats, compiled functions, symbols (when not in category (b)),
+includes conses, strings, subrs, floats, compiled functions, symbols,
 extents, events, and markers.  With the cleanup of frob blocks done in
 19.12, it's not terribly hard to add more objects to this category, but
-it's a bit trickier than adding an object type to type (d) (esp. if the
+it's a bit trickier than adding an object type to type (c) (esp. if the
 object needs a finalization method), and is not likely to save much
 space unless the object is small and there are many of them. (In fact,
 if there are very few of them, it might actually waste space.)
 @item
-(d) Those lrecords that are individually @code{malloc()}ed.  These are
+(c) Those lrecords that are individually @code{malloc()}ed.  These are
 called @dfn{lcrecords}.  All other types are in this category.  Adding a
 new type to this category is comparatively easy, and all types added
 since 19.8 (when the current allocation scheme was devised, by Richard
 Mlynarik), with the exception of the character type, have been in this
 category.
 @end itemize
 Note that bit vectors are a bit of a special case.  They are
-simple lrecords as in category (c), but are individually @code{malloc()}ed
+simple lrecords as in category (b), but are individually @code{malloc()}ed
 like vectors.  You can basically view them as exactly like vectors
 except that their type is stored in lrecord fashion rather than
 in directly-tagged fashion.
-Note that FSF Emacs redesigned their object system in 19.29 to follow
-a similar scheme.  However, given RMS's expressed dislike for data
-abstraction, the FSF scheme is not nearly as clean or as easy to
-extend. (FSF calls items of type (c) @code{Lisp_Misc} and items of type
-(d) @code{Lisp_Vectorlike}, with separate tags for each, although
-@code{Lisp_Vectorlike} is also used for vectors.)
 @node Garbage Collection, GCPROing, Introduction to Allocation, Allocation of Objects in XEmacs Lisp
 @section Garbage Collection
 @cindex garbage collection
 that ``all of memory'' means all currently allocated objects.
 Traversing all these objects means traversing all frob blocks,
 all vectors (which are chained in one big list), and all
 lcrecords (which are likewise chained).
-Note that, when an object is marked, the mark has to occur
+Garbage collection can be invoked explicitly by calling
-inside of the object's structure, rather than in the 32-bit
+@code{garbage-collect} but is also called automatically by @code{eval},
-@code{Lisp_Object} holding the object's pointer; i.e. you can't just
+once a certain amount of memory has been allocated since the last
-set the pointer's mark bit.  This is because there may be many
+garbage collection (according to @code{gc-cons-threshold}).
-pointers to the same object.  This means that the method of
-marking an object can differ depending on the type.  The
-different marking methods are approximately as follows:
-@enumerate
-@item
-For conses, the mark bit of the car is set.
-@item
-For strings, the mark bit of the string's plist is set.
-@item
-For symbols when not lrecords, the mark bit of the
-symbol's plist is set.
-@item
-For vectors, the length is negated after adding 1.
-@item
-For lrecords, the pointer to the structure describing
-the type is changed (see below).
-@item
-Integers and characters do not need to be marked, since
-no allocation occurs for them.
-@end enumerate
-The details of this are in the @code{mark_object()} function.
-Note that any code that operates during garbage collection has
-to be especially careful because of the fact that some objects
-may be marked and as such may not look like they normally do.
-In particular:
-@itemize @bullet
-Some object pointers may have their mark bit set.  This will make
-@code{FOOBARP()} predicates fail.  Use @code{GC_FOOBARP()} to deal with
-this.
-@item
-Even if you clear the mark bit, @code{FOOBARP()} will still fail
-for lrecords because the implementation pointer has been
-changed (see below).  @code{GC_FOOBARP()} will correctly deal with
-this.
-@item
-Vectors have their size field munged, so anything that
-looks at this field will fail.
-@item
-Note that @code{XFOOBAR()} macros @emph{will} work correctly on object
-pointers with their mark bit set, because the logical shift operations
-that remove the tag also remove the mark bit.
-@end itemize
-Finally, note that garbage collection can be invoked explicitly
-by calling @code{garbage-collect} but is also called automatically
-by @code{eval}, once a certain amount of memory has been allocated
-since the last garbage collection (according to @code{gc-cons-threshold}).
 @node GCPROing, Garbage Collection - Step by Step, Garbage Collection, Allocation of Objects in XEmacs Lisp
 @section @code{GCPRO}ing
 @code{GCPRO}ing is one of the ugliest and trickiest parts of Emacs
 other from one of the roots of accessibility.  The roots
 of accessibility are:
 @enumerate
 @item
-All objects that have been @code{staticpro()}d.  This is used for
+All objects that have been @code{staticpro()}d or
-any global C variables that hold Lisp objects.  A call to
+@code{staticpro_nodump()}ed.  This is used for any global C variables
-@code{staticpro()} happens implicitly as a result of any symbols
+that hold Lisp objects.  A call to @code{staticpro()} happens implicitly
-declared with @code{defsymbol()} and any variables declared with
+as a result of any symbols declared with @code{defsymbol()} and any
-@code{DEFVAR_FOO()}.  You need to explicitly call @code{staticpro()}
+variables declared with @code{DEFVAR_FOO()}.  You need to explicitly
-(in the @code{vars_of_foo()} method of a module) for other global
+call @code{staticpro()} (in the @code{vars_of_foo()} method of a module)
-C variables holding Lisp objects. (This typically includes
+for other global C variables holding Lisp objects. (This typically
-internal lists and such things.)
+includes internal lists and such things.).  Use
+@code{staticpro_nodump()} only in the rare cases when you do not want
+the pointed variable to be saved at dump time but rather recompute it at
+startup.
 Note that @code{obarray} is one of the @code{staticpro()}d things.
 Therefore, all functions and variables get marked through this.
 @item
 Any shadowed bindings that are sitting on the @code{specpdl} stack.
 @code{Feval}.
 @end enumerate
 The upshot is that garbage collection can basically occur everywhere
 @code{Feval}, respectively @code{Ffuncall}, is used - either directly or
-through another function. Since calls to these two functions are
+through another function. Since calls to these two functions are hidden
-hidden in various other functions, many calls to
+in various other functions, many calls to @code{garbage_collect_1} are
-@code{garabge_collect_1} are not obviously foreseeable, and therefore
+not obviously foreseeable, and therefore unexpected. Instances where
-unexpected. Instances where they are used that are worth remembering are
+they are used that are worth remembering are various elisp commands, as
-various elisp commands, as for example @code{or},
+for example @code{or}, @code{and}, @code{if}, @code{cond}, @code{while},
-@code{and}, @code{if}, @code{cond}, @code{while}, @code{setq}, etc.,
+@code{setq}, etc., miscellaneous @code{gui_item_...} functions,
-miscellaneous @code{gui_item_...} functions, everything related to
+everything related to @code{eval} (@code{Feval_buffer}, @code{call0},
-@code{eval} (@code{Feval_buffer}, @code{call0}, ...) and inside
+...) and inside @code{Fsignal}. The latter is used to handle signals, as
-@code{Fsignal}. The latter is used to handle signals, as for example the
+for example the ones raised by every @code{QUITE}-macro triggered after
-ones raised by every @code{QUITE}-macro triggered after pressing Ctrl-g.
+pressing Ctrl-g.
 @node garbage_collect_1, mark_object, Invocation, Garbage Collection - Step by Step
 @subsection @code{garbage_collect_1}
 @cindex @code{garbage_collect_1}
 @item
 Next the correct frame in which to put
 all the output occurring during garbage collecting is determined. In
 order to be able to restore the old display's state after displaying the
 message, some data about the current cursor position has to be
-saved. The variables @code{pre_gc_curser} and @code{cursor_changed} take
+saved. The variables @code{pre_gc_cursor} and @code{cursor_changed} take
 care of that.
 @item
 The state of @code{gc_currently_forbidden} must be restored after
 the garbage collection, no matter what happens during the process. We
 accomplish this by @code{record_unwind_protect}ing the suitable function
 @code{Vall_weak_lists} immediately. A marked list is treated more
 carefully by going over it and removing just the unmarked pairs.
 @item
 The function @code{prune_specifiers} checks all listed specifiers held
-in @code{Vall_speficiers} and removes the ones from the lists that are
+in @code{Vall_specifiers} and removes the ones from the lists that are
 unmarked.
 @item
 All syntax tables are stored in a list called
 @code{Vall_syntax_tables}. The function @code{prune_syntax_tables} walks
 @section lrecords
 [see @file{lrecord.h}]
 All lrecords have at the beginning of their structure a @code{struct
-lrecord_header}.  This just contains a pointer to a @code{struct
+lrecord_header}.  This just contains a type number and some flags,
+including the mark bit.  The type number, thru the
+@code{lrecord_implementation_table}, gives access to a @code{struct
 lrecord_implementation}, which is a structure containing method pointers
 and such.  There is one of these for each type, and it is a global,
 constant, statically-declared structure that is declared in the
-@code{DEFINE_LRECORD_IMPLEMENTATION()} macro. (This macro actually
+@code{DEFINE_LRECORD_IMPLEMENTATION()} macro.
-declares an array of two @code{struct lrecord_implementation}
-structures.  The first one contains all the standard method pointers,
+Simple lrecords (of type (b) above) just have a @code{struct
-and is used in all normal circumstances.  During garbage collection,
-however, the lrecord is @dfn{marked} by bumping its implementation
-pointer by one, so that it points to the second structure in the array.
-This structure contains a special indication in it that it's a
-@dfn{marked-object} structure: the finalize method is the special
-function @code{this_marks_a_marked_record()}, and all other methods are
-null pointers.  At the end of garbage collection, all lrecords will
-either be reclaimed or unmarked by decrementing their implementation
-pointers, so this second structure pointer will never remain past
-garbage collection.
-Simple lrecords (of type (c) above) just have a @code{struct
 lrecord_header} at their beginning.  lcrecords, however, actually have a
 @code{struct lcrecord_header}.  This, in turn, has a @code{struct
 lrecord_header} at its beginning, so sanity is preserved; but it also
 has a pointer used to chain all lcrecords together, and a special ID
 field used to distinguish one lcrecord from another. (This field is used
 (i.e. declared with a @code{_SEQUENCE_IMPLEMENTATION} macro.)  This should
 simply return the object's size in bytes, exactly as you might expect.
 For an example, see the methods for window configurations and opaques.
 @end enumerate
-@node Low-level allocation, Pure Space, lrecords, Allocation of Objects in XEmacs Lisp
+@node Low-level allocation, Cons, lrecords, Allocation of Objects in XEmacs Lisp
 @section Low-level allocation
 Memory that you want to allocate directly should be allocated using
 @code{xmalloc()} rather than @code{malloc()}.  This implements
 error-checking on the return value, and once upon a time did some more
 routines.  These routines also call @code{INCREMENT_CONS_COUNTER()} at the
 appropriate times; this keeps statistics on how much memory is
 allocated, so that garbage-collection can be invoked when the
 threshold is reached.
-@node Pure Space, Cons, Low-level allocation, Allocation of Objects in XEmacs Lisp
+@node Cons, Vector, Low-level allocation, Allocation of Objects in XEmacs Lisp
-@section Pure Space
-Not yet documented.
-@node Cons, Vector, Pure Space, Allocation of Objects in XEmacs Lisp
 @section Cons
 Conses are allocated in standard frob blocks.  The only thing to
 note is that conses can be explicitly freed using @code{free_cons()}
 and associated functions @code{free_list()} and @code{free_alist()}.  This
 ``vector''.)
 @node Symbol, Marker, Bit Vector, Allocation of Objects in XEmacs Lisp
 @section Symbol
-Symbols are also allocated in frob blocks.  Note that the code
+Symbols are also allocated in frob blocks.  Symbols in the awful
-exists for symbols to be either lrecords (category (c) above)
+horrible obarray structure are chained through their @code{next} field.
-or simple types (category (b) above), and are lrecords by
-default (I think), although there is no good reason for this.
-Note that symbols in the awful horrible obarray structure are
-chained through their @code{next} field.
 Remember that @code{intern} looks up a symbol in an obarray, creating
 one if necessary.
 @node Marker, String, Symbol, Allocation of Objects in XEmacs Lisp
 A bunch of tables needed to reassign properly the global pointers are
 then written.  They are:
 @enumerate
-@item the staticpro array
+@item
-@item the dumpstruct array
+the staticpro array
-@item the lrecord_implementation_table array
+@item
-@item a vector of all the offsets to the objects in the file that include a
+the dumpstruct array
+@item
+the lrecord_implementation_table array
+@item
+a vector of all the offsets to the objects in the file that include a
 description (for faster relocation at reload time)
-@item the pdump_wired and pdump_wired_list arrays
+@item
+the pdump_wired and pdump_wired_list arrays
 @end enumerate
 For each of the arrays we write both the pointer to the variables and
 the relocated offset of the object they point to.  Since these variables
 are global, the pointers are still valid when restarting the program and
 @code{funcall_compiled_function()} calls the real byte-code interpreter
 @code{execute_optimized_program()} on the byte-code instructions, which
 are converted into an internal form for faster execution.
 When a compiled function is executed for the first time by
-@code{funcall_compiled_function()}, or when it is @code{Fpurecopy()}ed
+@code{funcall_compiled_function()}, or during the dump phase of building
-during the dump phase of building XEmacs, the byte-code instructions are
+XEmacs, the byte-code instructions are converted from a
-converted from a @code{Lisp_String} (which is inefficient to access,
+@code{Lisp_String} (which is inefficient to access, especially in the
-especially in the presence of MULE) into a @code{Lisp_Opaque} object
+presence of MULE) into a @code{Lisp_Opaque} object containing an array
-containing an array of unsigned char, which can be directly executed by
+of unsigned char, which can be directly executed by the byte-code
-the byte-code interpreter.  At this time the byte code is also analyzed
+interpreter.  At this time the byte code is also analyzed for validity
-for validity and transformed into a more optimized form, so that
+and transformed into a more optimized form, so that
 @code{execute_optimized_program()} can really fly.
 Here are some of the optimizations performed by the internal byte-code
 transformer:
 @enumerate
 References to the @code{constants} array that will be used as a Lisp
 variable are checked for being correct non-constant (i.e. not @code{t},
 @code{nil}, or @code{keywordp}) symbols, so that the byte interpreter
 doesn't have to.
 @item
-The maxiumum number of variable bindings in the byte-code is
+The maximum number of variable bindings in the byte-code is
 pre-computed, so that space on the @code{specpdl} stack can be
 pre-reserved once for the whole function execution.
 @item
 All byte-code jumps are relative to the current program counter instead
 of the start of the program, thereby saving a register.
 All of these are very simple and work as expected, calling
 @code{Feval()} or @code{Fprogn()} as necessary and (in the case of
 @code{let} and @code{let*}) using @code{specbind()} to create bindings
 and @code{unbind_to()} to undo the bindings when finished.
-Note that, with the exeption of @code{Fprogn}, these functions are
+Note that, with the exception of @code{Fprogn}, these functions are
 typically called in real life only in interpreted code, since the byte
 compiler knows how to convert calls to these functions directly into
 byte code.
 @node Catch and Throw,  , Simple Special Forms, Evaluation; Stack Frames; Bindings
 There is a separate Lisp object type for each of these four concepts.
 Furthermore, there is logically a @dfn{selected console},
 @dfn{selected display}, @dfn{selected frame}, and @dfn{selected window}.
 Each of these objects is distinguished in various ways, such as being the
 default object for various functions that act on objects of that type.
-Note that every containing object rememembers the ``selected'' object
+Note that every containing object remembers the ``selected'' object
 among the objects that it contains: e.g. not only is there a selected
 window, but every frame remembers the last window in it that was
 selected, and changing the selected frame causes the remembered window
 within it to become the selected window.  Similar relationships apply
 for consoles to devices and devices to frames.
 @item
 Output changes Implemented by @code{redisplay-output.c},
 @code{redisplay-x.c}, @code{redisplay-msw.c} and @code{redisplay-tty.c}
 @end enumerate
-Steps 1 and 2 are device-independant and relatively complex.  Step 3 is
+Steps 1 and 2 are device-independent and relatively complex.  Step 3 is
 mostly device-dependent.
 Determining the desired display
 Display attributes are stored in @code{display_line} structures. Each
 @code{display_line} consists of a set of @code{display_block}'s and each
 @code{display_block} contains a number of @code{rune}'s. Generally
 dynarr's of @code{display_line}'s are held by each window representing
 the current display and the desired display.
-The @code{display_line} structures are tighly tied to buffers which
+The @code{display_line} structures are tightly tied to buffers which
 presents a problem for redisplay as this connection is bogus for the
 modeline. Hence the @code{display_line} generation routines are
 duplicated for generating the modeline. This means that the modeline
 display code has many bugs that the standard redisplay code does not.
 caching is done by @code{image_instantiate} and is necessary because it
 is generally possible to display an image-instance in multiple
 domains. For instance if we create a Pixmap, we can actually display
 this on multiple windows - even though we only need a single Pixmap
 instance to do this. If caching wasn't done then it would be necessary
-to create image-instances for every displayable occurrance of a glyph -
+to create image-instances for every displayable occurrence of a glyph -
 and every usage - and this would be extremely memory and cpu intensive.
 Widget-glyphs (a.k.a native widgets) are not cached in this way. This is
 because widget-glyph image-instances on screen are toolkit windows, and
 thus cannot be reused in multiple XEmacs domains. Thus widget-glyphs are
 tree recursively.
 This has desirable properties such as lw_modify_all_widgets which is
 called from glyphs-x.c and updates all the properties of a widget
 without having to know what the widget is or what toolkit it is from.
-Unfortunately this also has hairy properrties such as making the lwlib
+Unfortunately this also has hairy properties such as making the lwlib
 code quite complex. And of course lwlib has to know at some level what
 the widget is and how to set its properties.
 @node Specifiers, Menus, Glyphs, Top
 @chapter Specifiers

Mercurial > hg > xemacs-beta

comparison man/internals/internals.texi @ 400:a86b2b5e0111 r21-2-30