The XEmacs Packaging System
***************************

The XEmacs distribution, starting with version 21, comes only with a very basic set of built-in modes and libraries.  Most of the libraries that were part of the distribution of earlier versions of XEmacs are now available separately.  The user as well as the system administrator can choose which packages to install; the actual installation process is easy.  This gives an installer the ability to tailor an XEmacs installation for local needs with safe removal of unnecessary code.

This chapter describes how to package Lisp libraries for use with the XEmacs Packaging System.

Please note carefully that the term package as used in XEmacs refers to an aggregation of Lisp code and/or data distributed as a unit.  It does not, as it does in many Lisps, refer to a way of creating separate name spaces.  XEmacs has no facility for providing separate name spaces.  (If we ever do get separate name spaces, we'll probably regret overloading the nomenclature in this way, but it's become established.)

An overview of the XEmacs Packaging System
******************************************

The XEmacs Packaging System is a system for administering the installation, upgrade, and removal of Lisp libraries.  For the end user, it provides facilities for determining availability of packages and which versions at remote sites.  It will download and automatically install a package, ensuring that any old files from previous versions of the package are removed first.  By providing a standard set of hierarchies for installation, it makes configuration of XEmacs simpler.  Furthermore, packages normally provide ancillary auto-autoloads and custom-loads libraries, which are automatically detected and loaded by XEmacs upon startup.  This means that once installed, all facilities of package, including autoloading the library upon invocation of a command provided by the library and convenient configuration and customization, are automatically available to the user.  There is no need to add autoloads or keybindings to in the init file, and structured configuration of the package is available through the Customize system even before the libraries are loaded.

All of this convenience comes at a cost.  The cost of administration at the package level is negligible compared to the benefits, of course. However, the requirement that XEmacs find and load auto-autoloads and custom-loads libraries comes at a fairly large cost in startup time.  In order to reduce this cost, XEmacs imposes fairly strict conditions on the structure of an installed package.

Meeting these requirements, as well as simply providing the auto-autoloads and the information about availability and so on does impose some costs on the library maintainer.  The XEmacs Packaging System also provides structure and utilities to the library maintainer to make these tasks easier.  This manual documents the requirements and the tools that the XEmacs Packaging System provides to ensure that a package satisfies them.

The User View
=============

N.B.  Much of the discussion in this section undoubtedly belongs elsewhere, Packages (xemacs).

From the user's point of view, an XEmacs binary package is simply a standard tarball (usually gzipped) containing Lisp sources, compiled Lisp, documentation, and possibly data files or supporting executables. The tarball is unpacked using standard tools such as GNU tar and gzip. The package system does impose certain requirements for automatic configuration to work.

Here the main consideration is that the tarball ``expects'' to be unpacked from the top of a package hierarchy.  A package hierarchy is basically an image of a classic Emacs ``run-in-place'' tree, with lisp, etc, info, man, lib-src, and pkginfo subdirectories of the top.  The pkginfo subdirectory is for use by the XEmacs Packaging System administration tools, and currently contains a MANIFEST.PACKAGE-NAME file for each package to ensure that no cruft remains when a package is removed or updated.  The lisp, etc, and lib-src subdirectories are further subdivided, with a subdirectory for each package.  The info directory obeys the usual conventions. I.e., the info directory is flat with a(n) (optional) dir file and one (set of) info file(s) per package.  The man subdirectory typically contains documentation sources, separated by package.  (It does not contain man(1) pages, as Emacs provides very few of them.)

There are several standard package hierarchies, and administrators can configure others at build time, while users can configure others at run time.  The standard system hierarchies are all subdirectories of an XEmacs installation root, typically /usr/local/lib/xemacs/. These are the xemacs-packages, mule-packages, infodock-packages, and site-packages hierarchies.  Each has the structure described above, but the purposes differ.  The xemacs-packages is the normal place for installing ``official'' packages and many third-party libraries.  Unfortunately, it is not yet quite possible to read libraries containing international characters with a non-Mule XEmacs, so such libraries are sequestered in the mule-packages hierarchy.  Some packages are compatible only with the Infodock development environment, and they will be installed in the infodock-packages hierarchy.  The site-packages hierarchy is for packages not distributed by XEmacs.org, typically locally developed.

Packages are in principle supposed to be XEmacs version-independent, but if such dependencies are unavoidable, additional standard package hierarchies may be installed under version directories, e.g. /usr/local/lib/xemacs-21.4.6/.

Users who do not have sufficient privilege to install packages in the system hierarchies may install package hierarchies under ~/.xemacs.  At present only the xemacs-packages and mule-packages hierarchies are supported, but it might make sense to extend this to support infodock-packages and site-packages hierarchies in the future.

The package hierarchies are not searched directly for libraries to be loaded; this would be very costly.  Instead, the hierarchies are ordered according to certain rules, and searched for package lisp directories at invocation.  These directories are added to the general load-path.  As usual, it is load-path that is searched at run-time.  This approach is somewhat costly at initialization, but results in a very ``clean'' load-path.

The order of search can be changed at build time by specifying the --package-path option to configure, or at run-time by specifying the EMACSPACKAGEPATH environment variable. See Packages (xemacs).

The default order of search is hierarchically determined.  First, the roots are ordered.  The early roots are the user-specific roots, typically ~/.xemacs.  The late roots are the system roots, typically /usr/local/lib/xemacs-21.4.6 and /usr/local/lib/xemacs, in that order.  All hierarchies for a given root are searched for package Lisp directories, which are appended to load-path in the order found.  Then the search proceeds to the next root, whose results will be appended to the load-path generated by previous roots.

Second, the hierarchies below each root are searched in the order site-packages, infodock-packages, mule-packages, then xemacs-packages.

In each hierarchy there should be a lisp subdirectory, containing directories named for the packages.  Each package's Lisp libraries thus are contained in a directory of the form ROOT/HIERARCHY/lisp/PACKAGE/.

With such a complex search algorithm, the possibility of libraries being shadowed by another library with the same name is quite real.  There are two considerations here.  First, every XEmacs package contains certain libraries with constant names.  These are

_pkg.el
     Lisp code to inform the package administration system about the package

auto-autoloads.el
     Lisp code to set up autoloaded functions and variables that may be needed at load time

custom-load.el
     definitions of configuration variables for use with the Customize system.

They are special-cased, because the way they are used prevents shadowing from being an issue.

Second, it is possible that multiple copies of some library, or different libraries with the same name, are installed in various places in the hierarchies.  To detect such shadows, use list-load-path-shadows.

Finally, note that most basic Emacs functionality, including most of the Lisp API, is implemented in Lisp libraries.  Because they use internal reserved APIs that are subject to change according the needs of the developers, these libraries are distributed with the XEmacs binary, and are called core Lisp libraries.  Most core Lisp libraries are ``preloaded'' into the Emacs binary and in normal usage are never explicitly loaded.  However, they can be explicitly loaded, and if so they are searched on load-path. Furthermore, functions such as locate-library will also search on the load-path.  The searching takes place under somewhat different rules from those used for packaged Lisp.  It is probably easiest to think of the package hierarchy searching algorithm as receiving a load-path initialized to the core Lisp directories.

The Library Maintainer View
===========================

From the library maintainer's viewpoint, the advantages to the XEmacs Packaging System stem from the convenience to the user of installation and upgrade. Since an installed package automatically registers its entry points via autoload and its configuration variables with the Customize system, configuration FAQs are reduced.  When it's easy to upgrade, users learn to try Tools | Packages | Update Installed Packages before posting a FAQ whose answer is ``long since fixed, please upgrade.''

This comes at some cost, as the library maintainer needs to arrange that the package be installed in a directory structure that satisfies the requirements of the XEmacs Packaging System.  Autoload cookies and defcustoms must also be added to existing libraries.  The XEmacs Packaging System provides infrastructure to assure that all of these annoyances need only be dealt with once.  The autoload cookies and defcustoms are beyond the scope of this chapter, but most maintainers of modern packages are already familiar with these mechanisms.

The XEmacs Packaging System may be divided into the infrastructure common to all packages, and the package-specific control files.  The infrastructure supports global builds, installation, and generation of the ``sumo'' bundles of packages, as well as generation of individual packages.  The package control files describe the structure of the package's source tree and provide administrative information.

Infrastructure
--------------

In order to get the greatest benefit from the XEmacs Packaging System, a library maintainer should place the package sources in an appropriate place in the XEmacs source package hierarchy, and arrange to have the source package imported into the XEmacs CVS repository. (We realize that the latter requirement can be quite burdensome.  We are working on ways to remove this requirement, but for the present it remains necessary.)  The library maintainer must also keep sources for any packages his/her package requires.  This requirement is somewhat burdensome, but unlikely to be relaxed because of the implementation of compilation of macros in Emacs Lisp.  Macros cannot be called by compiled Lisp (the macro expansion, which is always known at compile time, is inlined), so the source of the macro must be loaded before compiling the called function.

The source package hierarchy may be rooted anywhere.  The CVS module is called ``packages,'' so we will refer to the top directory of the source package hierarchy as ``the packages directory.''  The packages directory contains two source subdirectories, xemacs-packages and mule-packages (for convenience in segregating the packages which depend on Mule, as they will cause load-time errors in a non-Mule XEmacs).  Each subdirectory contains many package source directories, whose internal structure is not specified. That structure is left up to the convenience of the library maintainers. The requirements on the top directory of an individual package source tree are given below, Control Files.

The packages directory contains some auxiliary Lisp libraries used in the compilation and packaging process.  The content of these libraries is of interest primarily to the packaging engineers, The Package Release Engineer View.

Finally, the packages, packages/xemacs-packages, and packages/mule-packages directories contain Makefiles and include files to control the package creation process.  The Makefiles in packages/xemacs-packages and packages/mule-packages simply define the default sets of known packages and include ../iterate.rules, which implements recursive building of all target packages.

The make infrastructure in packages includes

Makefile
     controls building of individual packages, local installation, and bundling of ``sumo'' tarballs

iterate.rules
     controls recursive builds of multiple packages

XEmacs.rules
     provides the rules for building and packaging.  Included by all package Makefiles.

Local.rules
     provides local configuration, such as installation targets and staging directories, as well as a number of kludges (many now obsolete) required for building packages on the Windows platform.

Local.rules.template
     a template for Local.rules, liberally commented

Local.rules.mk
     consistency checking for Local.rules, included by both the top-level Makefile and by XEmacs.rules.

package-compile.el
     compile environment (e.g., load-path) setup.  It is very bogus that this is here, an alternative mechanism is likely to be provided.

Of these, only Local.rules and package-compile.el need to be modified by the library maintainer.  The changes to Local.rules affect only your environment.  This should need to be done only once when first preparing the source environment.  The necessary modifications to package-compile.el need to be done for each package and are discussed in the next section, Control Files.

Control Files
-------------

Each package source must contain a number of control files in the top-level directory.  These files in general can be created and then ignored, except for a few variables that need to be updated when new versions are released.  In most cases even adding, renaming, and removing library source files can be handled by generic rules.

The package control files include

Makefile
     Must set a few make variables used by the administrative utilities, and defines a couple of package-building targets to depend on appropriate targets defined generically in XEmacs.rules.  It may also provide various variables and rules to transform the source tree structure into that expected by the run-time system.

package-info.in
     Provides a template for package information to be provided to the administrative utilities.  Static variables that are rarely changed (such as the package's name) are entered as literals.  Some variables are generated by the build process (build dates and MD5 checksums) and are automatically filled in.  Finally, some variables that change irregularly (dependences and even version numbers) are set as make variables in the Makefile.

ChangeLog
     Not strictly required, but normally a ChangeLog will be added by the XEmacs package maintainer if different from the upstream maintainer.

package-compile.el
     compile environment (e.g., load-path) setup.  It is very bogus that this is here, an alternative mechanism is likely to be provided.

_pkg.el
     Generated.  Simply does a package-provide for the package.

_auto-autoloads.el
     Generated.  Read when XEmacs is initialized, and provides autoloads for all defuns and other specially-marked forms in the sources.

custom-loads.el
     Generated.  Read when XEmacs is initialized, and informs the Customize subsystem how to find the defcustom forms needed to create Customization forms for the usre configuration variables of the package.

Obtaining the XEmacs Packaging System and Required Utilities
------------------------------------------------------------

Currently both the infrastructure for creating XEmacs packages and the package sources themselves are available only by CVS.  See http://www.xemacs.org/Develop/cvsaccess.html for more intformation.

The XEmacs Packaging System currently requires GNU make, and probably XEmacs, to build packages.

The Package Release Engineer View
---------------------------------

The XEmacs Package Release Engineer is responsible for keeping the system coherent.  The changes to packages/package-compile.el and packages/xemacs-packages/Makefile required to make the package available to others, and for building SUMO tarballs, etc, are done by the Package Release Engineer, not individual library maintainers.

The Package Release Engineer also maintains assorted infrastructure for actually making releases.  These are generally available for inspection in the xemacs-builds module in the CVS repository.

Package Terminology:
====================

Libraries and Packages
----------------------

A Lisp library is a single loadable file containing Lisp code.  It may be in source or byte-compiled form.  A Lisp package is a set of one or more libraries, usually related to each other in some way, bundled with administrative information for convenient distribution.

Package Flavors
---------------

There are two main flavors of packages.

Regular Packages
     A regular package is a set of Lisp libraries design to cooperate with one another.  A very complex example is Gnus.  One may not in general safely remove any of the component libraries.

Single-File Packages
     A single-file package is a collection of thematically related but otherwise independent Lisp libraries.  These libraries are bundled together for convenience of the maintainers.  Usually individual libraries may be deleted at will without any loss of functionality of other libraries in the package.  However, we would recommend that you follow this rule of thumb: "When in doubt, don't delete".  If it's really that big a deal, request that the maintainers split the package into smaller aggregations.

Package Distributions
---------------------

XEmacs Lisp packages are distributed in two ways.  Binary packages are used by system administrators and end users.  They are packaged in a form convenient for direct installation into an XEmacs package hierarchy.  Source packages are for developers and include all files necessary for rebuilding byte-compiled lisp and creating tarballs for distribution or installation.  This is all of the package author's source code plus all of the files necessary to build distribution tarballs (Unix Tar format files, gzipped for space savings). (Occasionally sources that are not relevant to XEmacs are removed.)

Currently, source packages are only available via CVS.  See http://www.xemacs.org/Develop/cvsaccess.html for details.

The package distributions are also split according to major features required in XEmacs to support them.  At present there are generic packages, which can be loaded by any XEmacs, and Mule packages, which require Mule support or they will cause errors when loaded.  Note that there is no guarantee that a generic package will have any useful functionality in a minimally configured XEmacs.  As long as any XEmacs can successfully load the package's libraries (perhaps given other required Lisp libraries), it will be classified as generic.  At the present time only Mule packages need be treated specially, and even those only if they contain multibyte characters.

Building Packages:
==================

Currently, source packages are only available via anonymous CVS.  See http://www.xemacs.org/Develop/cvsaccess.html for details of checking out the packages module.

Prerequisites for Building Source Packages
------------------------------------------

GNU cp

GNU install
     (or a BSD compatible install program).

GNU make
     (3.75 or later preferred).

makeinfo
     (1.68 from texinfo-3.11 or later required, 1.69 from Texinfo 4 preferred).

GNU tar
     (or equivalent).

GNU gzip
     (or equivalent).

A properly configured Local.rules file.
     Local.rules File.

And of course, XEmacs, 21.0 or higher.

What You Can Do With Source Packages
------------------------------------

The packages CVS sources are most useful for creating XEmacs package tarballs for installation into your own XEmacs installations or for distributing to others.

The supported make targets are:

all
     Bytecompile all files, build and bytecompile byproduct files like auto-autoloads.el and custom-load.el.  Create info version of TeXinfo documentation if present.

bindist
     Does a make all as well as create a binary package tarball in the staging directory.

install
     Bytecompile all files, build and bytecompile byproduct files like auto-autoloads.el and custom-load.el.  Create info version of TeXinfo documentation if present.  And install everything into the staging directory.

srckit
     Usually simply depends on srckit-std, with no actions.  This does a make distclean and creates a package source tarball in the staging directory.  This is generally only of use for package maintainers.

binkit
     May depend on binkit-sourceonly, binkit-sourceinfo, binkit-sourcedata, or binkit-sourcedatainfo, with no actions. sourceonly indicates there is nothing to install in a data directory or info directory.  sourceinfo indicates that source and info files are to be installed.  sourcedata indicates that source and etc (data) files are to be installed. sourcedatainfo indicates source, etc (data), and info files are to be installed.  A few packages have needs beyond the basic templates so this is not yet complete.

dist
     Runs the rules srckit followed by binkit.  This is primarily of use by XEmacs maintainers producing files for distribution.

clean
     Remove all built files except auto-autoloads.el and custom-load.el.

distclean
     Remove all created files.

The Local.rules File:
=====================

This file in packages provides the XEmacs Packaging System with information about the local configuration and environment.  To create Local.rules, simply copy Local.rules.template from that directory to Local.rules and edit it to suit your needs.

These are the variables in Local.rules that you will need to provide values for.  The following variables control which packages will be built:

XEMACS_PACKAGES
     The default is xemacs-packages, which results in the set in the xemacs-packages/Makefile PACKAGES variable.

     Otherwise, it should be a list of package source directories prefixed by xemacs-packages:

          XEMACS_PACKAGES = xemacs-packages/xemacs-base xemacs-packages/bbdb

BUILD_WITHOUT_MULE
     The default is the empty value.

     Building from CVS defaults to building the Mule packages.  Set this to 't' if you don't want/have Mule.

MULE_PACKAGES
     The default is mule-packages, which results in the set in the mule-packages/Makefile PACKAGES variable.

     Otherwise, it should be a list of package source directories prefixed by mule-packages:

          MULE_PACKAGES = mule-packages/mule-base mule-packages/skk

PACKAGE_INDEX
     The default is package-index.

     If you want the package index file to have a different name, change this.  This is probably a bad idea unless you are a packages release engineer, as it will confuse the package administration tools.

The following variables determine where files are installed and how they are installed.  Several of the defaults use the variable XEMACS_PACKAGES_BASE.  Never set this variable in Local.rules; it is automatically set in XEmacs.rules.

XEMACS_STAGING
     The default is ${XEMACS_PACKAGES_BASE}/../xemacs-packages.

     Generic packages will be installed here.  This can be the final destination for files or symlinks (if the packages are being installed locally), or a clean staging area for building tarballs.

     N.B.  make bindist ignores this variable.  It should be handled by the administration utilities, but currently isn't.

MULE_STAGING

     The default is ${XEMACS_PACKAGES_BASE}/../mule-packages.

     Packages requiring Mule to load correctly will be installed here.  This can be the final destination for files or symlinks (if the packages are being installed locally), or a clean staging area for building tarballs.

     N.B.  make bindist ignores this variable.  It should be handled by the administration utilities, but currently isn't.

symlink
     The default is the empty value.

     Set this to 't' if you want to simulate ``running in place.''  It is currently not possible to ask XEmacs to use any package source tree as an automatically configured member of load-path, and it is unlikely that complex trees such as that of the Gnus package will ever be able to ``run in place.''  This variable, when set to `t', causes the build process to create a symlink farm otherwise identical to an installed tree of binary packages.  Thus it is purely a space optimization.

     Setting this is incompatible with make bindist.

The following variables determine how packages are made.

XEMACS
     The default is xemacs.

     The path to the XEmacs executable you wish to use to compile the packages and execute Lisp build scripts.

XEMACS_NATIVE_NT
     The default is the empty value.

     Set this to 't' if you are building on WinNT.  It controls hairy shell quoting in the Makefiles.

INSTALL
     The default is install -c.

     The path to your BSD compatible install program.

TAR
     The default is tar.

     The path to your tar program.

BZIP2
     The default is the empty value.

     If unset, bzipped tarballs will not be built.  If this is set to something that resolves to a bzip2 executable, bzip2 tarballs will be built in addition to gzip tarballs.

MAKEINFO
     The default is makeinfo.

     The path to your makeinfo program

Creating Packages:
==================

Creating a package from an existing Lisp library is not very difficult.

In addition to the Lisp libraries themselves, you need a package-info.in file and a simple Makefile.  The rest is done by XEmacs.rules, part of the packaging system infrastructure.

package-info.in contains a single Lisp form like this:

     (NAME                               ; your package's name
       (standards-version 1.1
        version VERSION                  ; Makefile
        author-version AUTHOR_VERSION    ; Makefile
        date DATE                        ; Makefile
        build-date BUILD_DATE            ; generated
        maintainer MAINTAINER            ; Makefile
        distribution DISTRIBUTION        ; "mule" if MULE is needed,
                                         ; else "xemacs"
        priority high
        category CATEGORY                ; Makefile
        dump nil
        description "description"        ; a one-line description string
        filename FILENAME                ; obsolete
        md5sum MD5SUM                    ; generated
        size SIZE                        ; generated
        provides (FEATURE ...)           ; one for every `provides' form
        requires (REQUIRES)              ; Makefile
                                         ; NOT run-time dependencies!  These
                                         ; are files that provide macros or
                                         ; defsubsts that must be inlined.
        type regular
     ))

You should replace NAME, DISTRIBUTION, DESCRIPTION, and FEATURE ... with appropriate values, according to the comments.  Fields marked as obsolete can be ignored.  Fields marked as generated are generated by the package construction process, and will be filled in automatically.  Fields marked as Makefile should be set as variables in the Makefile.

The provides can be done automatically, but currently aren't.  It would probably be a good idea to set them in the Makefile (they do change, fairly often, but at present they aren't.

The Makefile is quite stylized.  The idea is similar to an Imakefile or an automake file: the complexity is hidden in generic rules files, in this case the XEmacs.rules include file in the top directory of the packages hierarchy.

It is important to note that the XEmacs used to compile packages is the bare minimum:  it is called with the -no-user-file -no-site-file -no-autoloads.  This means that anything not dumped into XEmacs by default needs to be specified in the REQUIRES variable (for packaged Lisp) or in some cases the PRELOADS (autoloads used in libraries mentioned in PRELOADS).

An XEmacs Packaging System Makefile has three components.  First, there is a variable definition section.  The standard XEmacs Packaging System make variables must be defined here for use by the XEmacs.rules include file. Second, the file ../../XEmacs.rules is included.  Finally, the make rules are defined, possibly including additional variable definitions for use by the Makefile.  These always include rules for the targets all, binkit, and srckit.

Although a number of facilities are available for complex libraries, most simple packages' Makefiles contain a copyright notice, the variable definitions mentioned above, and some boilerplate.

     # Makefile for apackage's lisp code

     # This file is part of XEmacs.

     # XEmacs is free software; you can redistribute it and/or modify it
     # under the terms of the GNU General Public License as published by the
     # Free Software Foundation; either version 2, or (at your option) any
     # later version.

     # XEmacs is distributed in the hope that it will be useful, but WITHOUT
     # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
     # FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
     # for more details.

     # You should have received a copy of the GNU General Public License
     # along with XEmacs; see the file COPYING.  If not, write to
     # the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
     # Boston, MA 02111-1307, USA.

     VERSION = 0.00
     AUTHOR_VERSION = 0.00
     MAINTAINER = A. M. Aintainer <ama@not.a.doc>
     PACKAGE = apackage
     PKG_TYPE = regular
     REQUIRES = xemacs-base
     CATEGORY = standard

     # All .els should be compiled and packaged.
     ELS = $(wildcard *.el)
     ELCS = $(ELS:.el=.elc)

     include ../../XEmacs.rules

     all:: $(ELCS) auto-autoloads.elc custom-load.elc

     srckit: srckit-std

     binkit: binkit-common

The XEmacs Packaging System does not automatically become aware of your package simply because there is a new subtree.  If any package, including your own, requires any of your files, it must be explicitly added to the compile environment or loads/requires that search load-path will fail.  The changes that need to be made are

an entry in package-directory-map
     This tells the XEmacs Packaging System which distribution (currently xemacs-packages or mule-packages) your package is found in.  It then looks in the distribution subdirectory whose name is the same as the package's.

an entry in the cond in package-name-to-directory
     This is optional; it is necessary only if you keep your Lisp code somewhere other than the top-level directory of the package's source tree, eg, in packages/xemacs-packages/PACKAGE/lisp.

This only needs to be done once, when the package is first added to the XEmacs Packaging System.  (Well, when you randomly change the subdirectory layout, too.) Your changes to package-compile.el must be cleared and checked in by the XEmacs Package Release Engineer before your package will build correctly from a fresh checkout.

This is unfortunate; it works pretty well once set up, but can cause confusion when first building a package in the XEmacs Packaging System context.  In particular, if the package-directory-map entry for a required package is not found, the necessary requires will not be executed by package-compile.el.  If required functions are executed (under eval-when-compile), they won't be found and the compile will fail.  If required function is actually a macro, the byte compiler will not recognize that, compile a function call to the macro.  This will cause a run-time error because the byte-code interpreter does not know how to execute macros.  (Macros can always be expanded at compile-time, and this is more efficient.)

If your package keeps some or all Lisp code somewhere other than the top directory, then an entry in package-name-to-directory is also necessary, or requires will fail, leading to the problems just described.

The package-info.in structure is simply Lisp data, to be read by a Lisp script, have values substituted for variables, and then written out (appropriately quoted) into a loadable Lisp file, to be consed into the package-index.el list at the FTP archives.  That list is structured as an alist with package names as keys.  The package data is a plist.  Do not rely on this, as it may change.  If you have a good reason for relying on it, let the maintainers know and we may incorporate it in a future revision of the XEmacs Packaging System standard.

There are several kinds of fields, distinguished by how they get their values.  There are literals written into package-info.in by the package maintainer.  There are variables substituted in by the build process, some computed, and others written as values of make variables in the Makefile by the package maintainer.  There are a few implementation constants, some of which are simply the default value for obsolete fields.

The package-info.in literals provided by the maintainer generally should not change over the life of the package.  (The exception is the provides field, which should be generated, but isn't yet.) Values described as ``literal'' below are unquoted literal text.  These are normally interpreted as symbols by the package build process.  The maintainer literals are

PACKAGE_NAME
     A literal.  The only unnamed ``field,'' the name of the package.

distribution
     A literal, either xemacs (for generic packages) or mule (for packages requiring Mule).  See Package Terminology.

description
     A Lisp string containing a one-line text description for use in package listings.

provides
     A (Lisp) list of features provided by the libraries in the package.  All of the features provided by libraries in your package should be elements of this list.

type
     A literal, either regular or single-file.  For practical purposes, regular should be considered an implementation constant.

Values which are expected to change regularly as the package is enhanced are implemented as make variables.  You should not change them in the package-info.in file; they are automatically filled in by the build process.

The corresponding field name is given in parentheses.  These include

VERSION
     (version) The version of the XEmacs package, a numeric literal (a decimal fixed-point number with two-places of precision).

AUTHOR_VERSION
     (author-version) The upstream author's version, an unintepreted literal.

DATE
     (date) Date of release of the upstream version.

MAINTAINER
     (maintainer) A literal containing the XEmacs package's maintainer and his/her email address.

CATEGORY
     (category) A literal, either standard or mule.  Probably redundant.

REQUIRES
     (requires) A list of packages required to correctly build this package.

     Note that the usual form in package-info.in already has the parentheses, so the make variable should be set to a space-separated list of package names not enclosed in parentheses.

     The list is of packages, not libraries, as would ordinarily be provided to the Lisp require function.

     REQUIRES cannot be correctly computed from the calls to require in the package's library sources.  REQUIRES is used to ensure that all macro and defstruct definitions used by the package are available at build time.  This is not merely a matter of efficiency, to get the expansions inlined.  In fact, it is impossible to call a macro by name in byte-compiled Emacs Lisp code.  Thus, if the macro expansion is not inlined, the call will result in an error at run-time!  Thus, packages providing libraries that would be loaded because of autoload definitions must also be included.

     On the other hand, if a package provides no macros to this package, it is preferable not to include it in REQUIRES, because it is not uncommon that if the developer doesn't normally use the required package, he will never use the functionality in the package being built, either.  In that case it would be preferable to not require the developer to have source for the dependencies.  That said, of course it is safe to put too many packages in REQUIRES.

Values for the following fields are automatically generated by the build process.

build-date
     The date the package tarball was generated.

md5sum
     An MD5 checksum for the package tarball, as gzipped.

size
     The size of the package tarball, as gzipped.

It is not clear that either md5sum or size works correctly if the BZIP2 variable in Local.rules is set.

The implementation constants are

standards-version
     Currently 1.1.  Defines the format of the package-info.in file and the Makefile.  A true implementation constant.

priority
     An unimplemented and underspecified feature.  Suggestions for specification and implementation welcome.

dump
     An obsolete feature, superseded by the site-load.el mechanism. The value should always be nil.

filename
     An obsolete feature, completely ignored.  Don't even think about doing anything useful with it.

A number of make variables are defined by the XEmacs Packaging System.  Some are required, others are optional.  Of course your Makefile may define other variables for private use, but you should be careful not to choose names that conflict with variables defined and used by the XEmacs Packaging System.

The required variables are described in the table below. The corresponding field names for package-info.in, where relevant, are given in parentheses.

VERSION
     (version) The version of the XEmacs package, a numeric literal (a decimal fixed-point number with two-places of precision).

AUTHOR_VERSION
     (author-version) The upstream author's version, an unintepreted literal.

DATE
     (date) Date of release of the upstream version.

MAINTAINER
     (maintainer) A literal containing the XEmacs package's maintainer and his/her email address.

CATEGORY
     (category) A literal, either standard or mule.  Probably redundant.

REQUIRES
     (requires) A list of packages required to correctly build this package.

     Note that the usual form in package-info.in already has the parentheses, so the make variable should be set to a space-separated list of package names not enclosed in parentheses.

     The list is of packages, not libraries, as would ordinarily be provided to the Lisp require function.

     REQUIRES cannot be correctly computed from the calls to require in the package's library sources.  REQUIRES is used to ensure that all macro and defstruct definitions used by the package are available at build time.  This is not merely a matter of efficiency, to get the expansions inlined.  In fact, it is impossible to call a macro by name in byte-compiled Emacs Lisp code.  Thus, if the macro expansion is not inlined, the call will result in an error at run-time!  Thus, packages providing libraries that would be loaded because of autoload definitions must also be included.

     On the other hand, if a package provides no macros to this package, it is preferable not to include it in REQUIRES, because it is not uncommon that if the developer doesn't normally use the required package, he will never use the functionality in the package being built, either.  In that case it would be preferable to not require the developer to have source for the dependencies.  That said, of course it is safe to put too many packages in REQUIRES.

ELCS
     The list of the byte-compiled Lisp files used by the package.  These files and their .el versions will be included in the binary package.  This variable determines which libraries will be byte-compiled.  These libraries are also deleted by make clean.

     Note there is no sanity-checking done on this variable.  If you put .el files in here, they will not be compiled and they will be deleted by make clean.  You would surely be very distressed if that happened, so be very careful.  If this variable is left empty, none of your Lisp code will be compiled or packaged.  This would be a less than amusing surprise, too.

     We don't consider this a feature, of course.  Please do submit code to do sanity checking to xemacs-patches@xemacs.org.

Optional, but very commonly used variables are explained below.

     item EXTRA_SOURCES Other files (such as extra Lisp sources or an upstream Makefile) that are normally placed in the installed Lisp directory, but not byte-compiled.  These files are preserved by the clean targets.

EXTRA_OBJS
     Other files (such as compiled autoload or concatenated .elc libraries) which are normally placed in the installed Lisp directory, but do not have corresponding source files and should be deleted by the clean targets.  Some of these (such as package-specific autoload setups) can and probably should be replaced by XEmacs Packaging System solutions such as auto-autoloads.el, but many cannot.

PRELOADS
     A specification for loading libraries containing macros before compiling the Lisp in the package.  This is spliced directly into the invocation of XEmacs for byte-compilation, so it must contain the -l flag for XEmacs:

          PRELOADS=-l ./apackage-macros.el -l ../bpackage/lisp/bpackage-macros.el

     Preloads are loaded before package-compile.el, so the LOAD-PATH is minimal.  Therefore PRELOADS must specify a full path to packaged Lisp.  The base LOAD-PATH does include the core Lisp directory, so core libraries are found.

INFO_FILES
     Any Info file(s) generated by the package.  These must be paths relative to the root of the package's source tree.

TEXI_FILES
     The Texinfo source file(s).  These must be paths relative to the root of the package's source tree.

MANUAL
     The name to be used for Info files and man pages.

DATA_FILES
     Any data files, such as pixmaps, READMEs, and ChangeLogs.  These must be paths relative to the root of the package's source tree.  These files will be copied to $(DATA_DEST) for installation.  Any directory component of the path for a file will be stripped, so that the file ends up in $(DATA_DEST), not in a subdiredtory.  If there are several destination directories, the forms $(DATA_FILES_N) and $(DATA_DEST_N) (N may take values in 1-40) must be used.  The XEmacs Packaging System doesn't make provision for copying trees at this time.

DATA_DEST
     The installation location for data files, relative to the etc/ directory of the package hierarchy.  The default is currently empty, but the normal value is simply $(PACKAGE).  Leaving it empty (i.e., put it directly under etc/) will probably work, but is subject to name conflicts with other packages.  If there are several destination directories, the form $(DATA_DEST_N) must be used.  The XEmacs Packaging System doesn't make provision for copying trees at this time.

The following variables are defaulted by Local.rules or XEmacs.rules.  These variables often should be added to using += rather than set by = or :=.

GENERATED
     A list of .elc files compiled from .el files generated by the package build process.  These files and the corresponding .el files are copied to the installed lisp directory.  Defaults to ${AUTOLOAD_PATH}/auto-autoloads.elc.  Typical usage is

          GENERATED += ${AUTOLOAD_PATH}/custom-load.elc

Rarely used variables.

The standard targets that need to be defined in your Makefile follow.  These normally should not have an action.  All of the work should be done by dependent targets, usually having standard definitions in the XEmacs Packaging System.

all
     A list of generated files, usually byte-compiled Lisp libraries, to be bundled in the package.  The typical dependencies are

          $(ELCS) auto-autoloads.elc custom-load.elc

     Other targets (such as Info files) may need to be added as dependencies for the all target.

srckit
     The target for generating a source package.  Not implemented.  If it were, the normal dependency would be srckit-std.

binkit
     The target for creating a ``master'' installation.  Binary packages are actually generated by the bindist target.  See Building Packages.

Standard dependencies for srckit and binkit are defined in XEmacs.rules.  The most useful of these values are given in the following table.

srckit-std
     Build a standard source kit.  Not fully implemented.

binkit-sourceonly
     The binkit target need only install source and compiled Lisp in the staging area.  There is nothing to install in a data directory or info directory.

binkit-sourceinfo
     Both source and info files are to be installed in the staging area.

binkit-sourcedata
     Both source and etc (data) files are to be installed in the staging area.

binkit-sourcedatainfo
     Source, etc (data), and info files all are present and need to be installed in the staging area.

binkit-common
     A dependency for all the above.  (In fact in the current implementation binkit-common does all the work for all of the binkit targets.)

Data files include things like pixmaps for a package-specific toolbar, and are normally installed in etc/PACKAGE_NAME.  A few packages have needs beyond the basic templates.  See XEmacs.rules or a future revision of this manual for details.

Documenting Packages:
=====================

Some random notes on documenting your package.

Do write a Texinfo file.  It's not that hard to do basically, and even using the more advanced features of Texinfo soon become natural.  For a start, just grab the template Samples/package.texi from the XEmacs Packaging System source tree, and drop your current README into the Top node.  At least this way your documentation will be accessible from the standard Info readers.  Next, try to add lots of cross-referencing and logical markup, and then node structure.

Address both end users and developer issues.  You may not be the maintainer forever.

If you are maintaining a package that is part of the GNU Emacs distribution, you'll likely find that you occasionally synchronize your package with the GNU Emacs sources.  When you synch a file, conventionally you should place a comment just above the standard ;;; Code comment that looks like this:

     ;; Synched with:
     ;; GNU Emacs 21.1, 2002-02-08, Stephen Turnbull <stephen@xemacs.org>

This comment is a status flag; the ChangeLog doesn't really give the same information.

Do maintain a detailed ChangeLog.

Issues
======

To be completed.