The XEmacs Packaging System
***************************

The XEmacs distribution, starting with version 21, comes only with a very basic set of built-in modes and libraries.  Most of the libraries that were part of the distribution of earlier versions of XEmacs are now available separately.  The user as well as the system administrator can choose which packages to install; the actual installation process is easy.  This gives an installer the ability to tailor an XEmacs installation for local needs with safe removal of unnecessary code.

This chapter describes how to package Lisp libraries for use with the XEmacs Packaging System.

Please note carefully that the term package as used in XEmacs refers to an aggregation of Lisp code and/or data distributed as a unit.  It does not, as it does in many Lisps, refer to a way of creating separate name spaces.  XEmacs has no facility for providing separate name spaces.  (If we ever do get separate name spaces, we'll probably regret overloading the nomenclature in this way, but it's become established.)

An overview of the XEmacs Packaging System
******************************************

The XEmacs Packaging System is a system for administering the installation, upgrade, and removal of Lisp libraries.  For the end user, it provides facilities for determining availability of packages and which versions at remote sites.  It will download and automatically install a package, ensuring that any old files from previous versions of the package are removed first.  By providing a standard set of hierarchies for installation, it makes configuration of XEmacs simpler.  Furthermore, packages normally provide ancillary auto-autoloads and custom-loads libraries, which are automatically detected and loaded by XEmacs upon startup.  This means that once installed, all facilities of package, including autoloading the library upon invocation of a command provided by the library and convenient configuration and customization, are automatically available to the user.  There is no need to add autoloads or keybindings to in the init file, and structured configuration of the package is available through the Customize system even before the libraries are loaded.

All of this convenience comes at a cost.  The cost of administration at the package level is negligible compared to the benefits, of course. However, the requirement that XEmacs find and load auto-autoloads and custom-loads libraries comes at a fairly large cost in startup time.  In order to reduce this cost, XEmacs imposes fairly strict conditions on the structure of an installed package.

Meeting these requirements, as well as simply providing the auto-autoloads and the information about availability and so on does impose some costs on the library maintainer.  The XEmacs Packaging System also provides structure and utilities to the library maintainer to make these tasks easier.  This manual documents the requirements and the tools that the XEmacs Packaging System provides to ensure that a package satisfies them.

The User View
=============

N.B.  Much of the discussion in this section undoubtedly belongs elsewhere, Packages (xemacs).

From the user's point of view, an XEmacs binary package is simply a standard tarball (usually gzipped) containing Lisp sources, compiled Lisp, documentation, and possibly data files or supporting executables. The tarball is unpacked using standard tools such as GNU tar and gzip. The package system does impose certain requirements for automatic configuration to work.

Here the main consideration is that the tarball ``expects'' to be unpacked from the top of a package hierarchy.  A package hierarchy is basically an image of a classic Emacs ``run-in-place'' tree, with lisp, etc, info, man, lib-src, and pkginfo subdirectories of the top.  The pkginfo subdirectory is for use by the XEmacs Packaging System administration tools, and currently contains a MANIFEST.PACKAGE-NAME file for each package to ensure that no cruft remains when a package is removed or updated.  The lisp, etc, and lib-src subdirectories are further subdivided, with a subdirectory for each package.  The info directory obeys the usual conventions. I.e., the info directory is flat with a(n) (optional) dir file and one (set of) info file(s) per package.  The man subdirectory typically contains documentation sources, separated by package.  (It does not contain man(1) pages, as Emacs provides very few of them.)

There are several standard package hierarchies, and administrators can configure others at build time, while users can configure others at run time.  The standard system hierarchies are all subdirectories of an XEmacs installation root, typically /usr/local/lib/xemacs/. These are the xemacs-packages, mule-packages, infodock-packages, and site-packages hierarchies.  Each has the structure described above, but the purposes differ.  The xemacs-packages is the normal place for installing ``official'' packages and many third-party libraries.  Unfortunately, it is not yet quite possible to read libraries containing international characters with a non-Mule XEmacs, so such libraries are sequestered in the mule-packages hierarchy.  Some packages are compatible only with the Infodock development environment, and they will be installed in the infodock-packages hierarchy.  The site-packages hierarchy is for packages not distributed by XEmacs.org, typically locally developed.

Packages are in principle supposed to be XEmacs version-independent, but if such dependencies are unavoidable, additional standard package hierarchies may be installed under version directories, e.g. /usr/local/lib/xemacs-21.4.6/.

Users who do not have sufficient privilege to install packages in the system hierarchies may install package hierarchies under ~/.xemacs. At present only the xemacs-packages, mule-packages, and site-packages hierarchies are supported, but it might make sense to extend this to support infodock-packages hierarchies in the future.

The package hierarchies are not searched directly for libraries to be loaded; this would be very costly.  Instead, the hierarchies are ordered according to certain rules, and searched for package lisp directories at invocation.  These directories are added to the general load-path.  As usual, it is load-path that is searched at run-time.  This approach is somewhat costly at initialization, but results in a very ``clean'' load-path.

The order of search can be changed at build time by specifying the --with-user-packages (an alias for --with-early-packages), --with-system-packages (an alias for --with-late-packages), and --with-legacy-packages (an alias for --with-last-packages) options to configure, or at run-time by specifying the EMACSEARLYPACKAGES, EMACSLATEPACKAGES, EMACSLASTPACKAGES environment variables.  See Packages (xemacs).

The default order of search is hierarchically determined.  First, the roots are ordered.  The early roots are the user-specific roots, typically ~/.xemacs.  The late roots are the system roots, typically /usr/local/lib/xemacs-21.4.6 and /usr/local/lib/xemacs, in that order.  All hierarchies for a given root are searched for package Lisp directories, which are appended to load-path in the order found.  Then the search proceeds to the next root, whose results will be appended to the load-path generated by previous roots.

Second, the hierarchies below each root are searched in the order site-packages, infodock-packages, mule-packages, then xemacs-packages.

In each hierarchy there should be a lisp subdirectory, containing directories named for the packages.  Each package's Lisp libraries thus are contained in a directory of the form ROOT/HIERARCHY/lisp/PACKAGE/.

With such a complex search algorithm, the possibility of libraries being shadowed by another library with the same name is quite real.  There are two considerations here.  First, every XEmacs package contains certain libraries with constant names.  These are

_pkg.el
     Lisp code to inform the package administration system about the package

auto-autoloads.el
     Lisp code to set up autoloaded functions and variables that may be needed at load time

custom-load.el
     definitions of configuration variables for use with the Customize system.

They are special-cased, because the way they are used prevents shadowing from being an issue.

Second, it is possible that multiple copies of some library, or different libraries with the same name, are installed in various places in the hierarchies.  To detect such shadows, use list-load-path-shadows.

Finally, note that most basic Emacs functionality, including most of the Lisp API, is implemented in Lisp libraries.  Because they use internal reserved APIs that are subject to change according the needs of the developers, these libraries are distributed with the XEmacs binary, and are called core Lisp libraries.  Most core Lisp libraries are ``preloaded'' into the Emacs binary and in normal usage are never explicitly loaded.  However, they can be explicitly loaded, and if so they are searched on load-path. Furthermore, functions such as locate-library will also search on the load-path.  The searching takes place under somewhat different rules from those used for packaged Lisp.  It is probably easiest to think of the package hierarchy searching algorithm as receiving a load-path initialized to the core Lisp directories.

The Library Maintainer View
===========================

From the library maintainer's viewpoint, the advantages to the XEmacs Packaging System stem from the convenience to the user of installation and upgrade. Since an installed package automatically registers its entry points via autoload and its configuration variables with the Customize system, configuration FAQs are reduced.  When it's easy to upgrade, users learn to try Tools | Packages | Update Installed Packages before posting a FAQ whose answer is ``long since fixed, please upgrade.''

This comes at some cost, as the library maintainer needs to arrange that the package be installed in a directory structure that satisfies the requirements of the XEmacs Packaging System.  Autoload cookies and defcustoms must also be added to existing libraries.  The XEmacs Packaging System provides infrastructure to assure that all of these annoyances need only be dealt with once.  The autoload cookies and defcustoms are beyond the scope of this chapter, but most maintainers of modern packages are already familiar with these mechanisms.

The XEmacs Packaging System may be divided into the infrastructure common to all packages, and the package-specific control files.  The infrastructure supports global builds, installation, and generation of the ``sumo'' bundles of packages, as well as generation of individual packages.  The package control files describe the structure of the package's source tree and provide administrative information.

Infrastructure
--------------

In order to get the greatest benefit from the XEmacs Packaging System, a library maintainer should place the package sources in an appropriate place in the XEmacs source package hierarchy, and arrange to have the source package imported into the XEmacs CVS repository. (We realize that the latter requirement can be quite burdensome.  We are working on ways to remove this requirement, but for the present it remains necessary.)  The library maintainer must also keep sources for any packages his/her package requires.  This requirement is somewhat burdensome, but unlikely to be relaxed because of the implementation of compilation of macros in Emacs Lisp.  Macros cannot be called by compiled Lisp (the macro expansion, which is always known at compile time, is inlined), so the source of the macro must be loaded before compiling the called function.

The source package hierarchy may be rooted anywhere.  The CVS module is called ``packages,'' so we will refer to the top directory of the source package hierarchy as ``the packages directory.''  The packages directory contains two source subdirectories, xemacs-packages and mule-packages (for convenience in segregating the packages which depend on Mule, as they will cause load-time errors in a non-Mule XEmacs).  Each subdirectory contains many package source directories, whose internal structure is not specified. That structure is left up to the convenience of the library maintainers. The requirements on the top directory of an individual package source tree are given below, Control Files.

The packages directory contains some auxiliary Lisp libraries used in the compilation and packaging process.  The content of these libraries is of interest primarily to the packaging engineers, The Package Release Engineer View.

Finally, the packages, packages/xemacs-packages, and packages/mule-packages directories contain Makefiles and include files to control the package creation process.  The Makefiles in packages/xemacs-packages and packages/mule-packages simply define the default sets of known packages and include ../iterate.rules, which implements recursive building of all target packages.

The make infrastructure in packages includes

Makefile
     controls building of individual packages, local installation, and bundling of ``sumo'' tarballs

iterate.rules
     controls recursive builds of multiple packages

meta-iterate.rules
     This is used by higher-level subdirectories that do not directly contain packages.  Subdirectories directly containing packages should use iterate.rules instead.

XEmacs.rules
     provides the rules for building and packaging.  Included by all package Makefiles.

Local.rules
     provides local configuration, such as installation targets and staging directories, as well as a number of kludges (many now obsolete) required for building packages on the Windows platform.

Local.rules.template
     a template for Local.rules, liberally commented

Local.rules.mk
     consistency checking for Local.rules, included by both the top-level Makefile and by XEmacs.rules.

Local.rules.inc
     a file to include in package Makefiles to be able to get at variables in Local.rules before including XEmacs.rules.

package-compile.el
     compile environment (e.g., load-path) setup.

Of these, only Local.rules and package-compile.el need to be modified by the library maintainer.  The changes to Local.rules affect only your environment.  This should need to be done only once when first preparing the source environment.  The necessary modifications to package-compile.el need to be done for each package and are discussed in the next section, Control Files.

Control Files
-------------

Each package source must contain a number of control files in the top-level directory.  These files in general can be created and then ignored, except for a few variables that need to be updated when new versions are released.  In most cases even adding, renaming, and removing library source files can be handled by generic rules.

The package control files include

Makefile
     Must set a few make variables used by the administrative utilities, and defines a couple of package-building targets to depend on appropriate targets defined generically in XEmacs.rules.  It may also provide various variables and rules to transform the source tree structure into that expected by the run-time system.

package-info.in
     Provides a template for package information to be provided to the administrative utilities.  Static variables that are rarely changed (such as the package's name) are entered as literals.  Some variables are generated by the build process (build dates and MD5 checksums) and are automatically filled in.  Finally, some variables that change irregularly (dependences and even version numbers) are set as make variables in the Makefile.

ChangeLog
     Not strictly required, but normally a ChangeLog will be added by the XEmacs package maintainer if different from the upstream maintainer.

_pkg.el
     Generated.  Simply does a package-provide for the package.

auto-autoloads.el
     Generated.  Read when XEmacs is initialized, and provides autoloads for defuns and other forms in the sources that are marked with an autoload cookie (;;;###autoload.

custom-loads.el
     Generated.  Read when XEmacs is initialized, and informs the Customize subsystem how to find the defcustom forms needed to create Customization forms for the usre configuration variables of the package.

Obtaining the XEmacs Packaging System and Required Utilities
------------------------------------------------------------

Currently both the infrastructure for creating XEmacs packages and the package sources themselves are available only by CVS.  See http://www.xemacs.org/Develop/cvsaccess.html for more intformation.

The XEmacs Packaging System currently requires GNU make, and XEmacs, to build packages.

The Package Release Engineer View
---------------------------------

The XEmacs Package Release Engineer is responsible for keeping the system coherent.  The changes to packages/package-compile.el and packages/xemacs-packages/Makefile required to make the package available to others, and for building SUMO tarballs, etc, are done by the Package Release Engineer, not individual library maintainers.

The Package Release Engineer also maintains assorted infrastructure for actually making releases.  These are generally available for inspection in the xemacs-builds module in the CVS repository.

Package Terminology:
====================

Libraries and Packages
----------------------

A Lisp library is a single loadable file containing Lisp code.  It may be in source or byte-compiled form.  A Lisp package is a set of one or more libraries, usually related to each other in some way, bundled with administrative information for convenient distribution.

Package Flavors
---------------

There are two main flavors of packages.

Regular Packages
     A regular package is a set of Lisp libraries design to cooperate with one another.  A very complex example is Gnus.  One may not in general safely remove any of the component libraries.

Single-File Packages
     A single-file package is a collection of thematically related but otherwise independent Lisp libraries.  These libraries are bundled together for convenience of the maintainers.  Usually individual libraries may be deleted at will without any loss of functionality of other libraries in the package.  However, we would recommend that you follow this rule of thumb: "When in doubt, don't delete".  If it's really that big a deal, request that the maintainers split the package into smaller aggregations.

Package Distributions
---------------------

XEmacs Lisp packages are distributed in two ways.  Binary packages are used by system administrators and end users.  They are packaged in a form convenient for direct installation into an XEmacs package hierarchy.  Source packages are for developers and include all files necessary for rebuilding byte-compiled lisp and creating tarballs for distribution or installation.  This is all of the package author's source code plus all of the files necessary to build distribution tarballs (Unix Tar format files, gzipped for space savings). (Occasionally sources that are not relevant to XEmacs are usually renamed to file.upstream.)

Currently, source packages are only available via CVS.  See http://www.xemacs.org/Develop/cvsaccess.html for details.

The package distributions are also split according to major features required in XEmacs to support them.  At present there are generic packages, which can be loaded by any XEmacs, and Mule packages, which require Mule support or they will cause errors when loaded.  Note that there is no guarantee that a generic package will have any useful functionality in a minimally configured XEmacs.  As long as any XEmacs can successfully load the package's libraries (perhaps given other required Lisp libraries), it will be classified as generic.  At the present time only Mule packages need be treated specially, and even those only if they contain multibyte characters.

Building Packages:
==================

Currently, source packages are only available via anonymous CVS.  See http://www.xemacs.org/Develop/cvsaccess.html for details of checking out the packages module.

Prerequisites for Building Source Packages
------------------------------------------

GNU cp

GNU install
     (or a BSD compatible install program).

GNU make
     (3.79 or later preferred).

makeinfo
     (4.2 from texinfo-4.2)

GNU tar
     (or equivalent).

GNU gzip
     (or equivalent).

A properly configured Local.rules file.
     Local.rules File.

And of course, XEmacs, 21.0 or higher.

What You Can Do With Source Packages
====================================

The packages CVS sources are most useful for creating XEmacs package tarballs for installation into your own XEmacs installations or for distributing to others.

It should be noted that most of the package Makefiles do not need to contain any target rules.  Everything is handled from the XEmacs.rules file, located in the toplevel directory of the packages source tree.

Makefile targets
****************

The following targets can be used when running make to build the packages:

mostlyclean
     Removes any documentation files that have been processed by TeX.

clean
     Does a mostlyclean, plus removes generated postscript and dvi files.  Also removes any generated .elc files, along with the normal .elc files in the package and HTML and .info files.

distclean
     Use this when preparing a distribution.  It kills anything that can be rebuilt.

extraclean
     Does a distclean and also removes any backup files (*~) and core files.

package-info
     Creates the package-info file from the package-info.in and writes an entry in the package-index file.

bindist
     Builds the package, including any Texinfo documentation (info format), writes an entry into the package-index file and builds a tarball of the package.  Also writes an entry into setup-packages.ini which is later used in the creation of netinstaller's setup.ini.

install
     Builds and installs a package

install-only
     Doesn't build anything, just installs it.

autoloads
     Generate the package's auto-autoloads.el file.

binkit
     Creates the directories needed for installation and copies the files there.  Basically this is an alias for install-only.

html
     Builds the HTML versions of the documentation.

compile
     Does most of the work.  Builds the elcs, infos at a minimum.

The targets that most people would be interested in would be:
=============================================================

  * all

  * bindist

  * html

  * install

  * install-only

  * clean

  * distclean

The Local.rules File:
=====================

This file in packages provides the XEmacs Packaging System with information about the local configuration and environment.  To create Local.rules, simply copy Local.rules.template from that directory to Local.rules and edit it to suit your needs.

These are the variables in Local.rules that you may need to provide values for:

XEMACS
     The name (and path if needed) of the XEmacs binary to use for building the packages.  The default is xemacs.

XEMACS_21_5
     This will enable some, as yet, unimplemented features in XEmacs 21.5 and above.  For now leave this blank (the default) regardless of the XEmacs version you are using.

BUILD_WITHOUT_MULE
     Set this to t if you are using a non-Mule XEmacs.  The default is that this variable is not set (blank) which means to build with Mule.

XEMACS_NATIVE_NT
     Set this to t if you are using a native Microsoft Windows build of XEmacs (not a Cygwin build) to build the packages. N.B. To Windows users, you still need the Cygwin environment to actually build the packages.

XEMACS_INSTALLED_PACKAGES_ROOT
     Set this to the root of where you want the packages to be installed. Under this directory will hang xemacs-packages and mule-packages.  See NONMULE_INSTALLED_PACKAGES_ROOT and MULE_INSTALLED_PACKAGES_ROOT.  The default for this is /usr/local/lib/xemacs.  Which may not be what you want if you are developing XEmacs.  To quote the comments in Local.rules.template:

          If you are developing XEmacs, you probably don't want to install the packages under /usr/local, which is where the stable, released version of XEmacs goes.  Instead, we suggest a layout as described in the base README file of recent versions of XEmacs.  In a nutshell, we suggest you put your source under /src/xemacs, and under this put the package sources in package-src/, and the installed packages in xemacs-packages/ and mule-packages/.  If you do everything this way, you might want to set things as follows:

          XEMACS_INSTALLED_PACKAGES_ROOT = ${XEMACS_PACKAGES_BASE}/..

          which puts the xemacs-packages/ and mule-packages/ directories as sisters of the package-src/ directory, and you have to tell configure the location of the installed packages using `--package-path', something like

          configure --package-path=/src/xemacs/xemacs-packages;/src/xemacs/mule-packages

symlink
     The default is unset (blank).  If you set this to t then make install will create a symlink farm of the installed packages under XEMACS_INSTALLED_PACKAGES_ROOT.  Obviously, for this to work, your system has to support symbolic links.  This is as close as you can get to running in place for the packages.

NONMULE_INSTALLED_PACKAGES_ROOT
     This is where the non-Mule packages get installed to.  The default is ${XEMACS_INSTALLED_PACKAGES_ROOT}/xemacs-packages.

MULE_INSTALLED_PACKAGES_ROOT
     This is where the Mule packages get installed to.  The default is ${XEMACS_INSTALLED_PACKAGES_ROOT}/mule-packages.

NONMULE_PACKAGES
     A whitespace separated list of non-Mule packages to build/install.

          NONMULE_PACKAGES = bbdb gnus xemacs-base prog-modes

     The value for this variable can also be the symbol xemacs-packages, which means to build/install all of the non-Mule packages.  The default is xemacs-packages.

MULE_PACKAGES
     A whitespace separated list of Mule packages to build/install.

          MULE_PACKAGES = mule-base leim locale

     The value for this variable can also be the symbol mule-packages, which means to build/install all of the Mule packages.  The default is mule-packages.

PACKAGE_INDEX
     The name of the package-index file.  The default is package-index and you probably don't need to worry about changing it.

INSTALL
     The path to a BSD compatible install program.  The default is install -c.

TAR
     The path to GNU/tar.  The default is tar.

BZIP2
     The path to the bzip2 compression program.  The default is unset (blank).  If this is set .tar.bz2 archives will be built in addition to the .tar.gz archives.

EXCLUDES
     For things that you don't want to go into the package tarballs. It takes the same format as GNU/tar's --exclude option.  The default is:

          EXCLUDES =					\
          	--exclude 'CVS'				\
          	--exclude 'RCS'				\
          	--exclude 'SCCS'			\
          	--exclude '*~'				\
          	--exclude '*.orig'			\
          	--exclude '*.rej'			\
          	--exclude '.\#*'

VANILLA
     Set to the XEmacs command line option that forces running in vanilla mode.  The default is -vanilla.  You wouldn't ever need to alter this.

BATCH
     How to put XEmacs into batch mode.  It also sets a couple of other things and in the normal course of events you wouldn't need to alter this from the default which is:

          BATCH = $(VANILLA) -batch -eval \
                  '(setq stack-trace-on-error t \
                         load-always-display-messages t \
                         load-ignore-out-of-date-elc-files t \
                         load-show-full-path-in-messages t)'

MAKEINFO
     The path to makeinfo.  The default is makeinfo

INSTALL_HTML
     Set this to t if you want to install HTML versions of the Texinfo documentation.  The default is unset (blank).

TEXI2HTML
     The path to the program that can convert Texinfo source to HTML.  The default is texi2html.

TEXI2DVI
     The path to the program that can convert Texinfo source to DVI.  The default is texi2dvi

DVIPS
     The path to the program that can convert DVI to Postscript.  The default is dvips

TEXI2PDF
     The path to the program that can convert Texinfo source to PDF format. The default is texi2pdf.

TEX
     The path to TeX.  The default is tex

MSGFMT
     The path to msgfmt.  The default is msgfmt

RCOPY
     The path to your copy command (GNU cp).  The default is dependent on whether or not SYMLINK is set (t).

     If SYMLINK is unset (blank), RCOPY's default is cp -af.  If SYMLINK is set (t), RCOPY's default is cp --force --recursive --symbolic-link.

It should be noted that in most cases the defaults should be fine.  Most people will probably only need to alter:

  * XEMACS_INSTALLED_PACKAGES_ROOT

  * NONMULE_INSTALLED_PACKAGES_ROOT

  * MULE_INSTALLED_PACKAGES_ROOT

  * NONMULE_PACKAGES

  * MULE_PACKAGES

Creating Packages:
******************

Creating a package from an existing Lisp library is not very difficult.

In addition to the Lisp libraries themselves, you need a package-info.in file and a simple Makefile.  The rest is done by XEmacs.rules, part of the packaging system infrastructure.

package-info.in
***************

package-info.in contains information that gets injected into the package-index file when make bindist is run.  Here is a real world example from the xemacs-base package (a description of each field follows the example):

     (xemacs-base
       (standards-version 1.1
        version VERSION
        author-version AUTHOR_VERSION
        date DATE
        build-date BUILD_DATE
        maintainer MAINTAINER
        distribution xemacs
        priority high
        category CATEGORY
        dump nil
        description "Fundamental XEmacs support, you almost certainly need this."
        filename FILENAME
        md5sum MD5SUM
        size SIZE
        provides (add-log advice-preload advice annotations assoc case-table chistory comint-xemacs comint compile debug ebuff-menu echistory edmacro ehelp electric enriched env facemenu ffap helper imenu iso-syntax macros novice outline passwd pp regexp-opt regi ring shell skeleton sort thing time-stamp timezone tq xbm-button xpm-button)
        requires (REQUIRES)
        type regular
     ))

Description of the Fields in package-info.in:
---------------------------------------------

NAME
     The name of the package.  In the case of the example it is xemacs-base.

standards-version
     Part of the internal package infrastructure, its value should always be 1.1.  Do not change this.

version
     This is the XEmacs package version number of the package.  It is set from the Makefile variable VERSION.  This is something that the XEmacs Package Release Engineer deals with so there is no need for a package maintainer to touch it.  In package-info.in just put the place-marker, VERSION here.

author-version
     This is the package's internal, or upstream version number if it has one.  It is set from the Makefile variable AUTHOR_VERSION.

date
     This is the date of the last change made to the package.  It is auto-generated at build time, taken from the package's toplevel ChangeLog.

build-date
     The date the package was built.  It is auto-generated.

maintainer
     This is the name and email address of the package's maintainer.  It is taken from the Makefile variable MAINTAINER.

distribution
     An unused field, leave as xemacs

priority
     An unused field, can be any of high, medium, or low.

category
     The category of the package.  It is taken from the Makefile variable CATEGORY and can be either standard for non-Mule packages, or mule for Mule packages.  The is also provision for unsupported in this field which would be for packages that XEmacs.org do not distribute.

     N.B. As yet, the XEmacs Packaging System does not support this type of package.  It will in the future.

dump
     Unused.  Always nil

description
     A free form short description of the package.

filename
     The file name of the package's binary tarball.  It is generated at build time by make bindist.

md5sum
     The MD5 message digest of the package's binary tarball.  Generated at build time by make bindist.

size
     The size in bytes of the package's binary tarball.  Generated at build time.

provides
     A whitespace separated list of all the features the package provides.  Surround the list with parens.

requires
     Taken from the Makefile variable REQUIRES.  It is a list of all the package's dependencies, including any macros and defstructs that need to be inlined.

     REQUIRES cannot be correctly computed from the calls to require in the package's library sources.  REQUIRES is used to ensure that all macro and defstruct definitions used by the package are available at build time.  This is not merely a matter of efficiency, to get the expansions inlined.  In fact, it is impossible to call a macro by name in byte-compiled Emacs Lisp code.  Thus, if the macro expansion is not inlined, the call will result in an error at run-time!  Thus, packages providing libraries that would be loaded because of autoload definitions must also be included.

type
     Can either be regular for a regular package, or single-file for a single file package.

     N.B. This doesn't refer to the number of lisp files in a package.  A single-file package can have multiple lisp files in it. See Package Terminology.

The fields in package-info.in that need to be changed directly are:

  * NAME

  * description

  * provides

  * type

Everything else is either set from the appropriate Makefile variable, is auto-generated at build time, or is static.

Makefile
********

The Makefile is quite stylized.  The idea is similar to an Imakefile or an automake file: the complexity is hidden in generic rules files, in this case the XEmacs.rules include file in the top directory of the packages hierarchy.

It is important to note that the XEmacs used to compile packages is the bare minimum: it is called with the -no-autoloads.  This means that anything not dumped into XEmacs by default needs to be specified in the REQUIRES variable (for packaged Lisp) or in some cases the PRELOADS (autoloads used in libraries mentioned in PRELOADS).

There isn't much to an XEmacs Packaging System Makefile, basically it just contains a few Makefile variables and that's it.  See the example.

Here is a real world example, from the build package:

     # Makefile for build lisp code

     # This file is part of XEmacs.
     #
     # XEmacs is free software: you can redistribute it and/or modify it
     # under the terms of the GNU General Public License as published by the
     # Free Software Foundation, either version 3 of the License, or (at your
     # option) any later version.
     #
     # XEmacs is distributed in the hope that it will be useful, but WITHOUT
     # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
     # FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
     # for more details.
     #
     # You should have received a copy of the GNU General Public License
     # along with XEmacs.  If not, see <http://www.gnu.org/licenses/>.

     # For the time being, remove MULE_ELCS from the all dependencies if
     # building without Mule.

     VERSION = 1.10
     AUTHOR_VERSION = 2.02
     MAINTAINER = Adrian Aichner <adrian@xemacs.org>
     PACKAGE = build
     PKG_TYPE = regular
     REQUIRES = xemacs-base pcl-cvs dired w3 prog-modes
     CATEGORY = standard

     ELCS = build.elc build-report.elc

     STANDARD_DOCS = t

     include ../../XEmacs.rules

Most packages don't need any more than what you see above.  It is usually not necessary to specify any special Makefile rules.  Everything is handled from the *.rules files in the toplevel of the package source hierarchy.

Of course, with that said, there are always exceptions to the rule.  If you think that your package will need some special Makefile hackery contact the XEmacs developers. We distribute over 100 packages so the chances are good that you won't be the first to need such hackery and it is probably already catered for.

Makefile Variables Explained:
-----------------------------

A number of make variables are defined by the XEmacs Packaging System.  Some are required, others are optional.  Of course your Makefile may define other variables for private use, but you should be careful not to choose names that conflict with variables defined and used by the XEmacs Packaging System.

The required variables are described in the table below. The corresponding field names for package-info.in, where relevant, are given in parentheses.

VERSION
     (version) The version of the XEmacs package, a numeric literal (a decimal fixed-point number with two-places of precision).  The only person who ever needs to touch this is the XEmacs Packages Release Engineer.

AUTHOR_VERSION
     (author-version) The upstream author's version, an uninterpreted literal.

MAINTAINER
     (maintainer) A literal containing the XEmacs package's maintainer and his/her email address.

PACKAGE
     The name of the package, a literal

PKG_TYPE
     The type of package, a literal containing either regular for regular packages, or single-file for single-file packages.  This should feed the type field in package-info.in, but currently it doesn't.

     N.B. single-file here does not refer to the number of lisp files in a package. See Package Terminology.

CATEGORY
     (category) A literal, either standard or mule.  The non-Mule packages are standard and the Mule packages are, you guessed it, mule.  This field is used at package installation time as part of the process of determining where a package should be installed to.

REQUIRES
     (requires) A list of packages required to correctly build this package.

     Note that the usual form in package-info.in already has the parentheses, so the make variable should be set to a space-separated list of package names not enclosed in parentheses.

     The list is of packages, not libraries, as would ordinarily be provided to the Lisp require function.

     REQUIRES cannot be correctly computed from the calls to require in the package's library sources.  REQUIRES is used to ensure that all macro and defstruct definitions used by the package are available at build time.  This is not merely a matter of efficiency, to get the expansions inlined.  In fact, it is impossible to call a macro by name in byte-compiled Emacs Lisp code.  Thus, if the macro expansion is not inlined, the call will result in an error at run-time!  Thus, packages providing libraries that would be loaded because of autoload definitions must also be included.

ELCS
     The list of the byte-compiled Lisp files used by the package.  These files and their .el versions will be included in the binary package.  This variable determines which libraries will be byte-compiled.  These libraries are also deleted by make clean.

     Note there is no sanity-checking done on this variable.  If you put .el files in here, they will not be compiled and they will be deleted by make clean.  You would surely be very distressed if that happened, so be very careful.  If this variable is left empty, none of your Lisp code will be compiled or packaged.  This would be a less than amusing surprise, too.

     We don't consider this a feature, of course.  Please do submit code to do sanity checking to xemacs-patches@xemacs.org.

Optional, but commonly used variables are explained below.

ELCS_1
     A list of extra byte-compiled Lisp files used by the package to be installed in a subdirectory of the package's lisp directory.  The same care should be taken with this as with ELCS in regard to make clean.

ELCS_1_DEST
     The name of the subdirectory for the ELCS_1 files to be installed to.  Be sure to include $(PACKAGE)/ as part of the name.

          ELCS_1_DEST = $(PACKAGE)/extra

     Would put the ELCS_1 files for the package, foo into xemacs-packages/lisp/foo/extra/.

EARLY_GENERATED_LISP
     For additional .el files that will be generated before any byte-compiling happens.  Use this for autoload-type files.  You must write Makefile rules to build these files.

GENERATED_LISP
     For additional .el files that will be generated at byte-compilation time.  You must write Makefile rules to build these files.

PRELOADS
     This is used if you need to pass extra command line arguments to XEmacs to build the package.  For instance, a specification for loading libraries containing macros before compiling the Lisp in the package.  This is spliced directly into the invocation of XEmacs for byte-compilation, so it must contain the -l flag for XEmacs:

          PRELOADS=-l ./apackage-macros.el -l ../bpackage/lisp/bpackage-macros.el

     Preloads are loaded before package-compile.el, so the load-path is minimal.  Therefore PRELOADS must specify a full path to packaged Lisp.  The base load-path does include the core Lisp directory, so core libraries are found.

AUTOLOAD_PATH
     The subdirectory in the package's source tree where the .el files reside.  This is where the auto-autoloads.el file will be placed.

     N.B. There is no need to use this variable if the .el files are in the package's toplevel directory.  AUTOLOAD_PATH defaults to ..

PACKAGE_SUPPRESS
     Place calls to package-suppress here to indicate Lisp libraries that should only be available to particular versions of XEmacs.  For example:

          PACKAGE_SUPPRESS = \
           (package-suppress 'xemacs-base \"regexp-opt\" '(emacs-version>= 21 5 11)) \
           (package-suppress 'xemacs-base \"easy-mmode\" '(emacs-version>= 21 5 11))

     N.B. This feature has not yet been implemented in XEmacs yet. It will appear in an upcoming version of XEmacs 21.5.

STANDARD_DOCS
     Set this to t if your package's Texinfo source file is located in the package's toplevel directory and is named $(PACKAGE).texi.

EXPLICIT_DOCS
     Use this to explicitly list Texinfo sources that aren't in the package's toplevel directory.  For example:

          EXPLICIT_DOCS = texi/$(PACKAGE).texi

     See DOCS_TXI_EXTENSION and DOCS_TEXINFO_EXTENSION if you don't use the .texi file extension on your Texinfo sources.

EXTRA_TEXI_FILES
     List here extra Texinfo source files needed to build your documentation.  Whatever is listed here is passed on to makeinfo as a dependency.

EXTRA_HTML_FILES
     Use this to specify extra .html files to output.

DOCS_TEXINFO_EXTENSION
     Set this to t if your Texinfo source files have a .texinfo extension.

DOCS_TXI_EXTENSION
     Set this to t if your Texinfo source files have a .txi extension.

EXTRA_DOC_FILES
     Files listed here will be installed to .../man/$(PACKAGE)/.  For example, you might want to list TeX files or .eps files here.

EXTRA_SOURCES
     Other files (such as extra Lisp sources or an upstream Makefile) that are normally placed in the installed Lisp directory, but not byte-compiled.  These files are preserved by the clean targets.

LIBSRC_FILES
     For files that need to be installed to lib-src/$(PACKAGE)/.  If the files listed here need to be built you will have to write Makefile rules to do so.

DATA_FILES
     Any data files, such as pixmaps, READMEs, and ChangeLogs.  These must be paths relative to the root of the package's source tree.  These files will be copied to $(DATA_DEST) for installation.  Any directory component of the path for a file will be stripped, so that the file ends up in $(DATA_DEST), not in a subdiredtory.

DATA_DEST
     The directory where the files in DATA_FILES are installed to.  It is a subdirectory of the installed etc/ directory.  Be sure to prefix this value with $(PACKAGE), for example:

          DATA_DEST = $(PACKAGE)/foo

     Would put files into .../etc/$(PACKAGE)/foo/.

DATA_1_FILES ... DATA_35_FILES
     For data files that need to go into a different directory from DATA_DEST.

DATA_1_DEST ... DATA_35_DEST
     The name of the subdirectory for files specified in DATA_N_FILES.  And like DATA_DEST, be sure to prefix $(PACKAGE) to the value of these variables.

EXTRA_DEPENDENCIES
     For additional files to build that aren't appropriate to place in any other Makefile variable.  You will need to write Makefile rules to build these files.

package-compile.el
==================

The XEmacs Packaging System does not automatically become aware of your package simply because there is a new subtree.  If any package, including your own, requires any of your files, it must be explicitly added to the compile environment or loads/requires that search load-path will fail.  The changes that need to be made are

an entry in package-directory-map
     This tells the XEmacs Packaging System which distribution (currently xemacs-packages or mule-packages) your package is found in.  It then looks in the distribution subdirectory whose name is the same as the package's.

an entry in the cond in package-name-to-directory
     This is optional; it is necessary only if you keep your Lisp code somewhere other than the top-level directory of the package's source tree, eg, in packages/xemacs-packages/$(PACKAGE)/lisp.

This only needs to be done once, when the package is first added to the XEmacs Packaging System.  (Well, when you randomly change the subdirectory layout, too.) Your changes to package-compile.el must be cleared and checked in by the XEmacs Package Release Engineer before your package will build correctly from a fresh checkout.

This is unfortunate; it works pretty well once set up, but can cause confusion when first building a package in the XEmacs Packaging System context.  In particular, if the package-directory-map entry for a required package, including the package itself, is not found, the necessary requires will not be executed by package-compile.el.  If required functions are executed (under eval-when-compile), they won't be found and the compile will fail.  If required function is actually a macro, the byte compiler will not recognize that, compile a function call to the macro.  This will cause a run-time error because the byte-code interpreter does not know how to execute macros.  (Macros can always be expanded at compile-time, and this is more efficient.)

If your package keeps some or all Lisp code somewhere other than the top directory, then an entry in package-name-to-directory is also necessary, or requires will fail, leading to the problems just described.

Documenting Packages:
=====================

Some random notes on documenting your package.

Do write a Texinfo file.  It's not that hard to do basically, and even using the more advanced features of Texinfo soon become natural.  For a start, just grab the template Samples/package.texi from the XEmacs Packaging System source tree, and drop your current README into the Top node.  At least this way your documentation will be accessible from the standard Info readers.  Next, try to add lots of cross-referencing and logical markup, and then node structure.

Address both end users and developer issues.  You may not be the maintainer forever.

If you are maintaining a package that is part of the GNU Emacs distribution, you'll likely find that you occasionally synchronize your package with the GNU Emacs sources.  When you synch a file, conventionally you should place a comment just above the standard ;;; Code comment that looks like this:

     ;; Synched with:
     ;; GNU Emacs 21.1, 2002-02-08, Stephen Turnbull <stephen@xemacs.org>

This comment is a status flag; the ChangeLog doesn't really give the same information.

Do maintain a detailed ChangeLog.

Issues
======

To be completed.