@c -*-texinfo-*-
@c This is part of the XEmacs Lisp Reference Manual.
@c Copyright (C) 2001 Free Software Foundation, Inc.
@c See the file lispref.texi for copying conditions.

@setfilename ../../info/packaging.info

@c Macro to make formatting of the XEmacs pms name consistent.
@c Maybe @sc looks OK in HTML?  If so, condition on Info.
@iftex
@macro xpms
XE@sc{macs} Packaging System
@end macro
@end iftex
@ifnottex
@macro xpms
XEmacs Packaging System
@end macro
@end ifnottex

@node Packaging, Lisp Data Types, Introduction, Top
@chapter The @xpms{}
@cindex package
@cindex packaging

The XEmacs distribution, starting with version 21, comes only with a
very basic set of built-in modes and libraries.  Most of the libraries
that were part of the distribution of earlier versions of XEmacs are now
available separately.  The user as well as the system administrator can
choose which packages to install; the actual installation process is
easy.  This gives an installer the ability to tailor an XEmacs
installation for local needs with safe removal of unnecessary code.

This chapter describes how to package Lisp libraries for use with the
@xpms{}.

@emph{Please note carefully} that the term @dfn{package} as used in
XEmacs refers to an aggregation of Lisp code and/or data distributed as
a unit.  It does not, as it does in many Lisps, refer to a way of
creating separate name spaces.  XEmacs has no facility for providing
separate name spaces.  (If we ever do get separate name spaces, we'll
probably regret overloading the nomenclature in this way, but it's
become established.)

@menu
Introduction:
* Package Overview::       Lisp Libraries and Packages.

Packaging Lisp Libraries:
* Package Terminology::    Basic stuff.
* Building Packages::      Turn packaged source into a tarball.
* Local.rules File::       Tell the @xpms{} about your host.
* Creating Packages::      Tell the @xpms{} about your package.
@c * History::                     History of the @xpms{}
@c * Installation::                Installing the @xpms{} with your (X)Emacs.
@c * Configuration::               Configuring the @xpms{} for use.
@c * Usage::                       An overview of the operation of the @xpms{}.
@c * Bug Reports::                 Reporting Bugs and Problems
@c * Frequently Asked Questions::  Questions and answers from the mailing list.

Internals and Package Release Engineering:
* Issues::                      
@end menu

@node Package Overview, Package Terminology, , Packaging
@chapter An overview of the @xpms{}

The @xpms{} is a system for administering the installation, upgrade, and
removal of Lisp libraries.  For the end user, it provides facilities for
determining availability of packages and which versions at remote
sites.  It will download and automatically install a package, ensuring
that any old files from previous versions of the package are removed
first.  By providing a standard set of hierarchies for installation, it
makes configuration of XEmacs simpler.  Furthermore, packages normally
provide ancillary auto-autoloads and custom-loads libraries, which are
automatically detected and loaded by XEmacs upon startup.  This means
that once installed, all facilities of package, including autoloading
the library upon invocation of a command provided by the library and
convenient configuration and customization, are automatically available
to the user.  There is no need to add autoloads or keybindings to in the
init file, and structured configuration of the package is available
through the Customize system even before the libraries are loaded.

All of this convenience comes at a cost.  The cost of administration at
the package level is negligible compared to the benefits, of course.
However, the requirement that XEmacs find and load auto-autoloads and
custom-loads libraries comes at a fairly large cost in startup time.  In
order to reduce this cost, XEmacs imposes fairly strict conditions on
the structure of an installed package.

Meeting these requirements, as well as simply providing the
auto-autoloads and the information about availability and so on does
impose some costs on the library maintainer.  The @xpms{} also provides
structure and utilities to the library maintainer to make these tasks
easier.  This manual documents the requirements and the tools that the
@xpms{} provides to ensure that a package satisfies them.

@menu
* The User's View::
* The Library Maintainer's View::
* The Package Release Engineer's View::
@end menu


@node The User's View, The Library Maintainer's View, , Package Overview
@section The User's View

@strong{N.B.}  Much of the discussion in this section undoubtedly
belongs elsewhere, @ref{Packages,,,xemacs}.

From the user's point of view, an XEmacs binary package is simply a
standard tarball (usually gzipped) containing Lisp sources, compiled
Lisp, documentation, and possibly data files or supporting executables.
The tarball is unpacked using standard tools such as GNU tar and gzip.
The package system does impose certain requirements for automatic
configuration to work.

Here the main consideration is that the tarball ``expects'' to be
unpacked from the top of a package hierarchy.  A @dfn{package hierarchy}
is basically an image of a classic Emacs ``run-in-place'' tree, with
@file{lisp}, @file{etc}, @file{info}, @file{man}, @file{lib-src}, and
@file{pkginfo} subdirectories of the top.  The @file{pkginfo}
subdirectory is for use by the @xpms{} administration tools, and
currently contains a @file{MANIFEST.@var{package-name}} file for each
package to ensure that no cruft remains when a package is removed or
updated.  The @file{lisp}, @file{etc}, and @file{lib-src} subdirectories
are further subdivided, with a subdirectory for each package.  The
@file{info} directory obeys the usual conventions.
@emph{I.e.}, the @file{info} directory is flat
with a(n) (optional) @file{dir} file and one (set of) info file(s) per
package.  The @file{man} subdirectory typically contains documentation
sources, separated by package.  (It does not contain @file{man(1)}
pages, as Emacs provides very few of them.)

There are several standard package hierarchies, and administrators can
configure others at build time, while users can configure others at run
time.  The standard system hierarchies are all subdirectories of an
@c #### This is possibly incorrect usage of "installation root."
XEmacs installation root, typically @file{/usr/local/lib/xemacs/}.
These are the @file{xemacs-packages}, @file{mule-packages},
@file{infodock-packages}, and @file{site-packages} hierarchies.  Each
has the structure described above, but the purposes differ.  The
@file{xemacs-packages} is the normal place for installing ``official''
packages and many third-party libraries.  Unfortunately, it is not yet
quite possible to read libraries containing international characters
with a non-Mule XEmacs, so such libraries are sequestered in the
@file{mule-packages} hierarchy.  Some packages are compatible only with
the Infodock development environment, and they will be installed in the
@file{infodock-packages} hierarchy.  The @file{site-packages} hierarchy
is for packages not distributed by XEmacs.org, typically locally
developed.

Packages are in principle supposed to be XEmacs version-independent, but
if such dependencies are unavoidable, additional standard package
hierarchies may be installed under version directories, @emph{e.g.}
@file{/usr/local/lib/xemacs-21.4.6/}.

Users who do not have sufficient privilege to install packages in the
system hierarchies may install package hierarchies under
@file{~/.xemacs}.  At present only the @file{xemacs-packages} and
@file{mule-packages} hierarchies are supported, but it might make sense
to extend this to support @file{infodock-packages} and
@file{site-packages} hierarchies in the future.

The package hierarchies are not searched directly for libraries to be
loaded; this would be very costly.  Instead, the hierarchies are ordered
according to certain rules, and searched for package lisp directories at
invocation.  These directories are added to the general
@code{load-path}.  As usual, it is @code{load-path} that is searched at
run-time.  This approach is somewhat costly at initialization, but
results in a very ``clean'' @code{load-path}.

The order of search can be changed at build time by specifying the
@samp{--package-path} option to @file{configure}, or at run-time by
specifying the @code{EMACSPACKAGEPATH} environment variable.
@xref{Packages,,,xemacs}.

@c #### The following description is quite possibly inaccurate.
@c Please, Michael, write some specs up!
The default order of search is hierarchically determined.  First, the
roots are ordered.  The @dfn{early} roots are the user-specific roots,
typically @file{~/.xemacs}.  The @dfn{late} roots are the system roots,
typically @file{/usr/local/lib/xemacs-21.4.6} and
@file{/usr/local/lib/xemacs}, in that order.  All hierarchies for a
given root are searched for package Lisp directories, which are appended
to @code{load-path} in the order found.  Then the search proceeds to the
next root, whose results will be appended to the @code{load-path}
generated by previous roots.

Second, the hierarchies below each root are searched in the order
@file{site-packages}, @file{infodock-packages}, @file{mule-packages},
then @file{xemacs-packages}.

In each hierarchy there should be a @file{lisp} subdirectory, containing
directories named for the packages.  Each package's Lisp libraries thus
are contained in a directory of the form
@var{root}/@var{hierarchy}/lisp/@var{package}/.

With such a complex search algorithm, the possibility of libraries being
shadowed by another library with the same name is quite real.  There are
two considerations here.  First, every XEmacs package contains certain
libraries with constant names.  These are

@table @file
@item _pkg.el
Lisp code to inform the package administration system about the package

@item auto-autoloads.el
Lisp code to set up autoloaded functions and variables that may be
needed at load time

@item custom-load.el
definitions of configuration variables for use with the Customize
system.
@end table

They are special-cased, because the way they are used prevents shadowing
from being an issue.

Second, it is possible that multiple copies of some library, or
different libraries with the same name, are installed in various places
in the hierarchies.  To detect such shadows, use
@code{list-load-path-shadows}.

Finally, note that most basic Emacs functionality, including most of the
Lisp API, is implemented in Lisp libraries.  Because they use internal
reserved APIs that are subject to change according the needs of the
developers, these libraries are distributed with the XEmacs binary, and
are called @dfn{core Lisp libraries}.  Most core Lisp libraries are
``preloaded'' into the Emacs binary and in normal usage are never
explicitly loaded.  However, they can be explicitly loaded, and if so
they are searched on @code{load-path}.
@c #### Is this correct?  It is not for C-h f, for example.
Furthermore, functions such as @code{locate-library} will also search on
the @code{load-path}.  The searching takes place under somewhat
different rules from those used for packaged Lisp.  It is probably
easiest to think of the package hierarchy searching algorithm as
receiving a @code{load-path} initialized to the core Lisp directories.


@node The Library Maintainer's View, The Package Release Engineer's View, The User's View, Package Overview
@section The Library Maintainer's View

From the library maintainer's viewpoint, the advantages to the @xpms{}
stem from the convenience to the user of installation and upgrade.
Since an installed package automatically registers its entry points via
autoload and its configuration variables with the Customize system,
configuration FAQs are reduced.  When it's easy to upgrade, users learn
to try @samp{Tools | Packages | Update Installed Packages} before
posting a FAQ whose answer is ``long since fixed, please upgrade.''

This comes at some cost, as the library maintainer needs to arrange that
the package be installed in a directory structure that satisfies the
requirements of the @xpms{}.  Autoload cookies and defcustoms must also
be added to existing libraries.  The @xpms{} provides infrastructure to
assure that all of these annoyances need only be dealt with once.  The
autoload cookies and defcustoms are beyond the scope of this chapter, but
most maintainers of modern packages are already familiar with these
mechanisms.

The @xpms{} may be divided into the @dfn{infrastructure} common to all
packages, and the package-specific @dfn{control files}.  The
infrastructure supports global builds, installation, and generation of
the ``sumo'' bundles of packages, as well as generation of individual
packages.  The package control files describe the structure of the
package's source tree and provide administrative information.

@menu
* Infrastructure::    Global Makefiles and common rules.
* Control Files::     Package-specific Makefiles and administrative files.
* Obtaining::         Obtaining the @xpms{} and required utilities.
@end menu

@node Infrastructure, Control Files, , The Library Maintainer's View
@subsection Infrastructure

In order to get the greatest benefit from the @xpms{}, a library
maintainer should place the package sources in an appropriate place in
the XEmacs source package hierarchy, and arrange to have the source
package imported into the XEmacs CVS repository.
@c #### the parenthetical remark should go to "issues."
(We realize that the
latter requirement can be quite burdensome.  We are working on ways to
remove this requirement, but for the present it remains necessary.)  The
library maintainer must also keep sources for any packages his/her
package requires.  This requirement is somewhat burdensome, but unlikely
to be relaxed because of the implementation of compilation of macros in
Emacs Lisp.  Macros cannot be called by compiled Lisp (the macro
expansion, which is always known at compile time, is inlined), so the
source of the macro must be loaded before compiling the called function.

The source package hierarchy may be rooted anywhere.  The CVS module is
called ``packages,'' so we will refer to the top directory of the source
package hierarchy as ``the @file{packages} directory.''  The
@file{packages} directory contains two source subdirectories,
@file{xemacs-packages} and @file{mule-packages} (for convenience in
segregating the packages which depend on Mule, as they will cause
load-time errors in a non-Mule XEmacs).  Each subdirectory contains many
package source directories, whose internal structure is not specified.
That structure is left up to the convenience of the library maintainers.
The requirements on the top directory of an individual package source
tree are given below, @ref{Control Files}.

The @file{packages} directory contains some auxiliary Lisp libraries
used in the compilation and packaging process.  The content of these
libraries is of interest primarily to the packaging engineers, @ref{The
Package Release Engineer's View}.

Finally, the @file{packages}, @file{packages/xemacs-packages}, and
@file{packages/mule-packages} directories contain @file{Makefile}s and
include files to control the package creation process.  The
@file{Makefile}s in @file{packages/xemacs-packages} and
@file{packages/mule-packages} simply define the default sets of known
packages and include @file{../iterate.rules}, which implements recursive
building of all target packages.

The @samp{make} infrastructure in @file{packages} includes

@table @file
@item Makefile
controls building of individual packages, local installation, and
bundling of ``sumo'' tarballs

@item iterate.rules
controls recursive builds of multiple packages

@item XEmacs.rules
provides the rules for building and packaging.  Included by all package
@file{Makefile}s.

@item Local.rules
provides local configuration, such as installation targets and staging
directories, as well as a number of kludges (many now obsolete) required
for building packages on the Windows platform.

@item Local.rules.template
a template for Local.rules, liberally commented

@item Local.rules.mk
consistency checking for @file{Local.rules}, included by both the
top-level @file{Makefile} and by @file{XEmacs.rules}.

@c #### Add to "issues"
@item package-compile.el
compile environment (@emph{e.g.}, load-path) setup.  It is very bogus
that this is here, an alternative mechanism is likely to be provided.
@end table

Of these, only @file{Local.rules} and @file{package-compile.el} need to
be modified by the library maintainer.  The changes to Local.rules
affect only your environment.  This should need to be done only once
when first preparing the source environment.  The necessary
modifications to @file{package-compile.el} need to be done for each
package and are discussed in the next section, @ref{Control Files}.


@node Control Files, Obtaining, Infrastructure, The Library Maintainer's View
@subsection Control Files

Each package source must contain a number of control files in the
top-level directory.  These files in general can be created and then
ignored, except for a few variables that need to be updated when new
versions are released.  In most cases even adding, renaming, and
removing library source files can be handled by generic rules.

The package control files include

@table @file
@item Makefile
Must set a few @file{make} variables used by the administrative
utilities, and defines a couple of package-building targets to depend on
appropriate targets defined generically in @file{XEmacs.rules}.  It may
also provide various variables and rules to transform the source tree
structure into that expected by the run-time system.

@item package-info.in
Provides a template for package information to be provided to the
administrative utilities.  Static variables that are rarely changed
(such as the package's name) are entered as literals.  Some variables
are generated by the build process (build dates and MD5 checksums) and
are automatically filled in.  Finally, some variables that change
irregularly (dependences and even version numbers) are set as
@file{make} variables in the @file{Makefile}.

@item ChangeLog
Not strictly required, but normally a ChangeLog will be added by the
XEmacs package maintainer if different from the upstream maintainer.

@item package-compile.el
compile environment (@emph{e.g.}, load-path) setup.  It is very bogus
that this is here, an alternative mechanism is likely to be provided.

@item _pkg.el
Generated.  Simply does a @code{package-provide} for the package.

@item _auto-autoloads.el
Generated.  Read when XEmacs is initialized, and provides autoloads for
all defuns and other specially-marked forms in the sources.

@item custom-loads.el
Generated.  Read when XEmacs is initialized, and informs the Customize
subsystem how to find the defcustom forms needed to create Customization
forms for the usre configuration variables of the package.
@end table


@node Obtaining, , Control Files, The Library Maintainer's View
@subsection Obtaining the @xpms{} and Required Utilities

Currently both the infrastructure for creating XEmacs packages and the
package sources themselves are available only by CVS.  See
@uref{http://www.xemacs.org/Develop/cvsaccess.html} for more
intformation.

The @xpms{} currently requires GNU @file{make}, and probably XEmacs, to
build packages.


@node The Package Release Engineer's View, , The Library Maintainer's View, Package Overview
@subsection The Package Release Engineer's View

The XEmacs Package Release Engineer is responsible for keeping the
system coherent.  The changes to @file{packages/package-compile.el} and
@file{packages/xemacs-packages/Makefile} required to make the package
available to others, and for building SUMO tarballs, @emph{etc}, are
done by the Package Release Engineer, not individual library
maintainers.

The Package Release Engineer also maintains assorted infrastructure for
actually making releases.  These are generally available for inspection
in the @code{xemacs-builds} module in the CVS repository.

@c #### To be completed.


@c #### The following section is lifted verbatim from the XEmacs User's
@c      Manual, file packages.texi.
@node Package Terminology, Building Packages, Package Overview, Packaging
@comment  node-name,  next,  previous,  up
@heading Package Terminology:

@subsection Libraries and Packages
@cindex library
@cindex package

A Lisp @dfn{library} is a single loadable file containing Lisp code.  It
may be in source or byte-compiled form.  A Lisp @dfn{package} is a set
of one or more libraries, usually related to each other in some way,
bundled with administrative information for convenient distribution.

@subsection Package Flavors

There are two main flavors of packages.

@table @strong
@item Regular Packages
@cindex regular package
A regular package is a set of Lisp libraries design to cooperate with
one another.  A very complex example is Gnus.  One may not in general
safely remove any of the component libraries.

@item Single-File Packages
@cindex single-file package
A single-file package is an collection of thematically related but
otherwise independent Lisp libraries.  These libraries are bundled
together for convenience of the maintainers.  Usually individual
libraries may be deleted at will without any loss of functionality of
other libraries in the package.  However, we would recommend that you
follow this rule of thumb: "When in doubt, don't delete".  If it's
really that big a deal, request that the maintainers split the package
into smaller aggregations.
@end table

@subsection Package Distributions
@cindex package distributions
@cindex binary packages
@cindex source packages
XEmacs Lisp packages are distributed in two ways.  @dfn{Binary packages}
are used by system administrators and end users.  They are packaged in a
form convenient for direct installation into an XEmacs package
hierarchy.  @dfn{Source packages} are for developers and include all
files necessary for rebuilding byte-compiled lisp and creating tarballs
for distribution or installation.  This is all of the package author's
source code plus all of the files necessary to build distribution
tarballs (Unix Tar format files, gzipped for space savings).
@c #### This next is an Evile Practice and should be discontinued.
(Occasionally sources that are not relevant to XEmacs are removed.)

Currently, source packages are only available via CVS.  See
@url{http://www.xemacs.org/Develop/cvsaccess.html} for details.

The package distributions are also split according to major features
required in XEmacs to support them.  At present there are @dfn{generic}
packages, which can be loaded by @emph{any} XEmacs, and @dfn{Mule}
packages, which @emph{require} Mule support or they will cause errors
when loaded.  Note that there is no guarantee that a generic package
will have any useful functionality in a minimally configured XEmacs.  As
long as any XEmacs can successfully load the package's libraries
(perhaps given other required Lisp libraries), it will be classified as
generic.  At the present time only Mule packages need be treated
specially, and even those only if they contain multibyte characters.


@c #### The following section is lifted verbatim from the XEmacs User's
@c      Manual, file packages.texi.
@node Building Packages, Local.rules File, Package Terminology, Packaging
@comment  node-name,  next,  previous,  up
@cindex building packages
@cindex package building
@heading Building Packages:
Currently, source packages are only available via anonymous CVS.  See
@url{http://www.xemacs.org/Develop/cvsaccess.html} for details of
checking out the @file{packages} module.

@subsection Prerequisites for Building Source Packages

@table @code
@item GNU cp
@item GNU install 
(or a BSD compatible install program).
@item GNU make 
(3.75 or later preferred).
@item makeinfo 
(1.68 from texinfo-3.11 or later required, 1.69 from Texinfo 4 preferred).
@item GNU tar
(or equivalent).
@item GNU gzip
(or equivalent).
@item A properly configured @file{Local.rules} file.
@ref{Local.rules File}.
@end table

And of course, XEmacs, 21.0 or higher.

@subsection What You Can Do With Source Packages

The packages CVS sources are most useful for creating XEmacs package
tarballs for installation into your own XEmacs installations or for
distributing to others.

The supported @file{make} targets are:

@table @code
@item all
Bytecompile all files, build and bytecompile byproduct files like
@file{auto-autoloads.el} and @file{custom-load.el}.  Create info version
of TeXinfo documentation if present.

@c #### Why do we need this _and_ the binkit target?
@item bindist
Does a @code{make all} as well as create a binary package tarball in the
staging directory.

@item install
Bytecompile all files, build and bytecompile byproduct files like
@file{auto-autoloads.el} and @file{custom-load.el}.  Create info version
of TeXinfo documentation if present.  And install everything into the
staging directory.

@item srckit
Usually simply depends on @code{srckit-std}, with no actions.  This does
a @code{make distclean} and creates a package source tarball in the
staging directory.  This is generally only of use for package
maintainers.

@item binkit
May depend on @code{binkit-sourceonly}, @code{binkit-sourceinfo},
@code{binkit-sourcedata}, or @code{binkit-sourcedatainfo}, with no
actions. @code{sourceonly} indicates there is nothing to install in a
data directory or info directory.  @code{sourceinfo} indicates that
source and info files are to be installed.  @code{sourcedata} indicates
that source and etc (data) files are to be installed.
@code{sourcedatainfo} indicates source, etc (data), and info files are
to be installed.  A few packages have needs beyond the basic templates
so this is not yet complete.

@item dist
Runs the rules @code{srckit} followed by @code{binkit}.  This is
primarily of use by XEmacs maintainers producing files for distribution.

@item clean
Remove all built files except @file{auto-autoloads.el} and
@file{custom-load.el}.

@item distclean
Remove all created files.
@end table

@c #### The following section is lifted verbatim from the XEmacs User's
@c      Manual, file packages.texi.
@node Local.rules File, Creating Packages, Building Packages, Packaging
@comment  node-name,  next,  previous,  up
@cindex local.rules
@heading The Local.rules File:
This file in @file{packages} provides the @xpms{} with information about
the local configuration and environment.  To create @file{Local.rules},
simply copy @file{Local.rules.template} from that directory to
@file{Local.rules} and edit it to suit your needs.

These are the variables in @file{Local.rules} that you will need to
provide values for.  The following variables control which packages will
be built:

@table @var
@item XEMACS_PACKAGES
The default is @samp{xemacs-packages}, which results in the set in
the @file{xemacs-packages/Makefile} @code{PACKAGES} variable.

Otherwise, it should be a list of package source directories prefixed by
@samp{xemacs-packages}:

@example
XEMACS_PACKAGES = xemacs-packages/xemacs-base xemacs-packages/bbdb
@end example

@item BUILD_WITHOUT_MULE
The default is the empty value.

Building from CVS defaults to building the Mule
packages.  Set this to 't' if you don't want/have Mule.

@item MULE_PACKAGES
The default is @samp{mule-packages}, which results in the set in
the @file{mule-packages/Makefile} @code{PACKAGES} variable.

Otherwise, it should be a list of package source directories prefixed by
@samp{mule-packages}:

@example
MULE_PACKAGES = mule-packages/mule-base mule-packages/skk
@end example

@item PACKAGE_INDEX
The default is @file{package-index}.

If you want the package index file to have a different name, change
this.  This is probably a bad idea unless you are a packages release
engineer, as it will confuse the package administration tools.
@end table

The following variables determine where files are installed and how they
are installed.  Several of the defaults use the variable
@var{XEMACS_PACKAGES_BASE}.  Never set this variable in
@file{Local.rules}; it is automatically set in @file{XEmacs.rules}.

@table @asis
@item @var{XEMACS_STAGING}
The default is @file{$@{XEMACS_PACKAGES_BASE@}/../xemacs-packages}.

Generic packages will be installed here.  This can be the final
destination for files or symlinks (if the packages are being installed
locally), or a clean staging area for building tarballs.

@strong{N.B.}  @samp{make bindist} ignores this variable.  It should be
handled by the administration utilities, but currently isn't.

@item @var{MULE_STAGING}

The default is @file{$@{XEMACS_PACKAGES_BASE@}/../mule-packages}.

Packages requiring Mule to load correctly will be installed here.  This
can be the final destination for files or symlinks (if the packages are
being installed locally), or a clean staging area for building tarballs.

@strong{N.B.}  @samp{make bindist} ignores this variable.  It should be
handled by the administration utilities, but currently isn't.

@item symlink
The default is the empty value.

Set this to 't' if you want to simulate ``running in place.''  It is
currently not possible to ask XEmacs to use any package source tree as
an automatically configured member of @code{load-path}, and it is
unlikely that complex trees such as that of the Gnus package will ever
be able to ``run in place.''  This variable, when set to `t', causes the
build process to create a symlink farm otherwise identical to an
installed tree of binary packages.  Thus it is purely a space
optimization.

Setting this is incompatible with @samp{make bindist}.
@end table

The following variables determine how packages are made.

@table @var
@item XEMACS
The default is @samp{xemacs}.

The path to the XEmacs executable you wish to use to compile the
packages and execute Lisp build scripts.

@item XEMACS_NATIVE_NT
The default is the empty value.

Set this to 't' if you are building on WinNT.  It controls hairy shell
quoting in the @file{Makefile}s.

@item INSTALL
The default is @samp{install -c}.

The path to your BSD compatible install program.

@item TAR
The default is @samp{tar}.

The path to your tar program.

@item BZIP2
The default is the empty value.

If unset, bzipped tarballs will not be built.  If this is set to
something that resolves to a @samp{bzip2} executable, bzip2 tarballs
will be built @emph{in addition to} @samp{gzip} tarballs.

@item MAKEINFO
The default is @samp{makeinfo}.

The path to your @file{makeinfo} program
@end table


@c #### The following section is lifted verbatim from the XEmacs User's
@c      Manual, file packages.texi.
@node Creating Packages, Issues, Local.rules File, Packaging
@comment  node-name,  next,  previous,  up
@cindex creating packages
@heading Creating Packages:
Creating a package from an existing Lisp library is not very difficult.

In addition to the Lisp libraries themselves, you need a
@file{package-info.in} file and a simple @file{Makefile}.  The rest is
done by @file{XEmacs.rules}, part of the packaging system
infrastructure.

@file{package-info.in} contains a single Lisp form like this:

@example
(NAME                               ; your package's name
  (standards-version 1.1
   version VERSION                  ; Makefile
   author-version AUTHOR_VERSION    ; Makefile
   date DATE                        ; Makefile
   build-date BUILD_DATE            ; generated
   maintainer MAINTAINER            ; Makefile
   distribution DISTRIBUTION        ; "mule" if MULE is needed,
                                    ; else "xemacs"
   priority high
   category CATEGORY                ; Makefile
   dump nil
   description "description"        ; a one-line description string
   filename FILENAME                ; obsolete
   md5sum MD5SUM                    ; generated
   size SIZE                        ; generated
   provides (FEATURE ...)           ; one for every `provides' form
   requires (REQUIRES)              ; Makefile
                                    ; NOT run-time dependencies!  These
                                    ; are files that provide macros or
                                    ; defsubsts that must be inlined.
   type regular
))
@end example

You should replace NAME, DISTRIBUTION, DESCRIPTION, and FEATURE ... with
appropriate values, according to the comments.  Fields marked as
@samp{obsolete} can be ignored.  Fields marked as @samp{generated} are
generated by the package construction process, and will be filled in
automatically.  Fields marked as @samp{Makefile} should be set as
variables in the @file{Makefile}.

The @samp{provides} can be done automatically, but currently aren't.  It
would probably be a good idea to set them in the @file{Makefile} (they
do change, fairly often, but at present they aren't.

The @file{Makefile} is quite stylized.  The idea is similar to an
@file{Imakefile} or an @code{automake} file: the complexity is hidden in
generic rules files, in this case the @file{XEmacs.rules} include file
in the top directory of the packages hierarchy.

An @xpms{} @file{Makefile} has three components.  First, there is a
variable definition section.  The standard @xpms{} @file{make} variables
must be defined here for use by the @file{XEmacs.rules} include file.
Second, the file @file{../../XEmacs.rules} is included.  Finally, the
@file{make} rules are defined, possibly including additional variable
definitions for use by the @file{Makefile}.  These always include rules
for the targets @samp{all}, @samp{binkit}, and @file{srckit}.

Although a number of facilities are available for complex libraries,
most simple packages' @file{Makefile}s contain a copyright notice, the
variable definitions mentioned above, and some boilerplate.

@example
# Makefile for apackage's lisp code

# This file is part of XEmacs.

# XEmacs is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation; either version 2, or (at your option) any
# later version.

# XEmacs is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
# for more details.

# You should have received a copy of the GNU General Public License
# along with XEmacs; see the file COPYING.  If not, write to
# the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
# Boston, MA 02111-1307, USA.

VERSION = 0.00
AUTHOR_VERSION = 0.00
MAINTAINER = A. M. Aintainer <ama@@not.a.doc>
PACKAGE = apackage
PKG_TYPE = regular
REQUIRES = xemacs-base
CATEGORY = standard

# All .els should be compiled and packaged.
ELS = $(wildcard *.el)
ELCS = $(ELS:.el=.elc)

include ../../XEmacs.rules

all:: $(ELCS) auto-autoloads.elc custom-load.elc

srckit: srckit-std

binkit: binkit-common
@end example

@menu
* package-compile.el::
* package-info.in Fields::
* Makefile Variables::
* Makefile Targets::
@end menu


@node package-compile.el, package-info.in Fields, , Creating Packages

The @xpms{} does not automatically become aware of your package simply
because there is a new subtree.  If any package, including your own,
requires any of your files, it must be explicitly added to the compile
environment or loads/requires that search load-path will fail.  The
changes that need to be made are

@table @strong
@item an entry in @code{package-directory-map}
This tells the @xpms{} which distribution (currently
@samp{xemacs-packages} or @samp{mule-packages}) your package is found
in.  It then looks in the distribution subdirectory whose name is the
same as the package's.

@item an entry in the @code{cond} in @code{package-name-to-directory}
This is optional; it is necessary only if you keep your Lisp code
somewhere other than the top-level directory of the package's source
tree, eg, in @file{packages/xemacs-packages/@var{PACKAGE}/lisp}.
@end table

This only needs to be done once, when the package is first added to the
@xpms{}.  (Well, when you randomly change the subdirectory layout, too.)
Your changes to @file{package-compile.el} must be cleared and checked in
by the XEmacs Package Release Engineer before your package will build
correctly from a fresh checkout.

This is unfortunate; it works pretty well once set up, but can cause
confusion when first building a package in the @xpms{} context.  In
particular, if the @code{package-directory-map} entry for a required
package
@c #### including the package itself?
is not found, the necessary requires will not be executed by
@file{package-compile.el}.  If required functions are executed (under
@code{eval-when-compile}), they won't be found and the compile will
fail.  If required function is actually a macro, the byte compiler will
not recognize that, compile a function call to the macro.  This will
cause a run-time error because the byte-code interpreter does not know
how to execute macros.  (Macros can always be expanded at compile-time,
and this is more efficient.)

If your package keeps some or all Lisp code somewhere other than the top
directory, then an entry in @code{package-name-to-directory} is also
necessary, or requires will fail, leading to the problems just described.


@node package-info.in Fields, Makefile Variables, package-compile.el, Creating Packages

The @file{package-info.in} structure is simply Lisp data, to be read by
a Lisp script, have values substituted for variables, and then written
out (appropriately quoted) into a loadable Lisp file, to be consed into
the @file{package-index.el} list at the FTP archives.  That list is
structured as an alist with package names as keys.  The package data is
a plist.  Do not rely on this, as it may change.  If you have a good
reason for relying on it, let the maintainers know and we may
incorporate it in a future revision of the @xpms{} standard.

There are several kinds of fields, distinguished by how they get their
values.  There are literals written into @file{package-info.in} by the
package maintainer.  There are variables substituted in by the build
process, some computed, and others written as values of @file{make}
variables in the @file{Makefile} by the package maintainer.  There are a
few implementation constants, some of which are simply the default value
for obsolete fields.

The @file{package-info.in} literals provided by the maintainer generally
should not change over the life of the package.  (The exception is the
@samp{provides} field, which should be generated, but isn't yet.)
Values described as ``literal'' below are unquoted literal test.  These
are normally interpreted as symbols by the package build process.  The
maintainer literals are

@table @asis
@item @var{package_name}
A literal.  The only unnamed ``field,'' the name of the package.

@item distribution
A literal, either @samp{xemacs} (for generic packages) or @samp{mule}
(for packages requiring Mule).  @xref{Package Terminology}.

@item description
A Lisp string containing a one-line text description for use in package
listings.

@item provides
A (Lisp) list of features provided by the libraries in the package.  All
of the features provided by libraries in your package should be elements
of this list.

@item type
A literal, either @samp{regular} or @samp{single-file}.  For practical
purposes, @samp{regular} should be considered an implementation constant.
@end table

@c #### The following should be rewritten to @xref the make variables
@c node, and simply associate the field names to the make variables with
@c one line of description.
Values which are expected to change regularly as the package is enhanced
are implemented as @file{make} variables.  You should not change them in
the @file{package-info.in} file; they are automatically filled in by the
build process.

The corresponding field name is given in parentheses.  These include

@table @code
@item VERSION
(version)
The version of the XEmacs package, a numeric literal (a decimal
fixed-point number with two-places of precision).

@item AUTHOR_VERSION
(author-version)
The upstream author's version, an unintepreted literal.

@item DATE
(date)
Date of release of the upstream version.

@item MAINTAINER
(maintainer)
A literal containing the XEmacs package's maintainer and his/her email
address.

@item CATEGORY
(category)
A literal, either @samp{standard} or @samp{mule}.  Probably redundant.

@item REQUIRES
(requires)
A list of packages required to correctly build this package.

Note that the usual form in @file{package-info.in} already has the
parentheses, so the @file{make} variable should be set to a
space-separated list of package names @emph{not} enclosed in
parentheses.

The list is of @emph{packages}, not @emph{libraries}, as would
ordinarily be provided to the Lisp @code{require} function.

@samp{REQUIRES} cannot be correctly computed from the calls to
@code{require} in the package's library sources.  @samp{REQUIRES} is
used to ensure that all macro and defstruct definitions used by the
package are available at build time.  This is not merely a matter of
efficiency, to get the expansions inlined.  In fact, it is
@emph{impossible} to call a macro by name in byte-compiled Emacs Lisp
code.  Thus, if the macro expansion is not inlined, the call will result
in an error at run-time!  Thus, packages providing libraries that would
be loaded because of autoload definitions must also be included.

On the other hand, if a package provides no macros to this package, it
is preferable @emph{not} to include it in @samp{REQUIRES}, because it is
not uncommon that if the developer doesn't normally use the required
package, he will never use the functionality in the package being built,
either.  In that case it would be preferable to not require the
developer to have source for the dependencies.  That said, of course it
is safe to put too many packages in @samp{REQUIRES}.
@end table

Values for the following fields are automatically generated by the build
process.

@table @asis
@item build-date
The date the package tarball was generated.

@item md5sum
An MD5 checksum for the package tarball, as gzipped.

@item size
The size of the package tarball, as gzipped.
@end table

It is not clear that either md5sum or size works correctly if the
@samp{BZIP2} variable in @file{Local.rules} is set.

The implementation constants are

@table @asis
@item standards-version
Currently 1.1.  Defines the format of the @file{package-info.in} file
and the @file{Makefile}.  A true implementation constant.

@item priority
An unimplemented and underspecified feature.  Suggestions for
specification and implementation welcome.

@item dump
An obsolete feature, superseded by the @file{site-load.el} mechanism.
The value should always be nil.

@item filename
An obsolete feature, completely ignored.  Don't even think about doing
anything useful with it.
@end table


@node Makefile Variables, Makefile Targets, package-info.in Fields, Creating Packages

A number of @file{make} variables are defined by the @xpms{}.  Some are
required, others are optional.  Of course your @file{Makefile} may
define other variables for private use, but you should be careful not to
choose names that conflict with variables defined and used by the
@xpms{}.

The required variables are described in the table below.
The corresponding field names for @file{package-info.in}, where
relevant, are given in parentheses.

@c #### This is the canonical place for this information.  If there is
@c unnecessary duplication with package-info.in documentation, shorten
@c that and leave this full-length.
@table @code
@item VERSION
(version)
The version of the XEmacs package, a numeric literal (a decimal
fixed-point number with two-places of precision).

@item AUTHOR_VERSION
(author-version)
The upstream author's version, an unintepreted literal.

@item DATE
(date)
Date of release of the upstream version.

@item MAINTAINER
(maintainer)
A literal containing the XEmacs package's maintainer and his/her email
address.

@item CATEGORY
(category)
A literal, either @samp{standard} or @samp{mule}.  Probably redundant.

@item REQUIRES
(requires)
A list of packages required to correctly build this package.

Note that the usual form in @file{package-info.in} already has the
parentheses, so the @file{make} variable should be set to a
space-separated list of package names @emph{not} enclosed in
parentheses.

The list is of @emph{packages}, not @emph{libraries}, as would
ordinarily be provided to the Lisp @code{require} function.

@samp{REQUIRES} cannot be correctly computed from the calls to
@code{require} in the package's library sources.  @samp{REQUIRES} is
used to ensure that all macro and defstruct definitions used by the
package are available at build time.  This is not merely a matter of
efficiency, to get the expansions inlined.  In fact, it is
@emph{impossible} to call a macro by name in byte-compiled Emacs Lisp
code.  Thus, if the macro expansion is not inlined, the call will result
in an error at run-time!  Thus, packages providing libraries that would
be loaded because of autoload definitions must also be included.

On the other hand, if a package provides no macros to this package, it
is preferable @emph{not} to include it in @samp{REQUIRES}, because it is
not uncommon that if the developer doesn't normally use the required
package, he will never use the functionality in the package being built,
either.  In that case it would be preferable to not require the
developer to have source for the dependencies.  That said, of course it
is safe to put too many packages in @samp{REQUIRES}.

@item ELCS
The list of the byte-compiled Lisp files used by the package.  These
files and their @file{.el} versions will be included in the binary
package.  This variable determines which libraries will be
byte-compiled.  These libraries are also deleted by @samp{make clean}.

Note there is no sanity-checking done on this variable.  If you put
@samp{.el} files in here, they will not be compiled and they @emph{will}
be deleted by @samp{make clean}.  You would surely be very distressed if
that happened, so be very careful.  If this variable is left empty, none
of your Lisp code will be compiled or packaged.  This would be a less
than amusing surprise, too.

We don't consider this a feature, of course.  Please do submit code to
do sanity checking to @email{xemacs-patches@@xemacs.org}.
@end table

Optional, but very commonly used variables include:

@table @code
item EXTRA_SOURCES
Other files (such as extra Lisp sources or an upstream @file{Makefile})
that are normally placed in the installed Lisp directory, but not
byte-compiled.  These files are @emph{preserved} by the @samp{clean}
targets.

@item EXTRA_OBJS
Other files (such as compiled autoload or concatenated @file{.elc}
libraries) which are normally placed in the installed Lisp directory,
but do @emph{not} have corresponding source files and @emph{should} be
deleted by the @samp{clean} targets.  Some of these (such as
package-specific autoload setups) can and probably should be replaced by
@xpms{} solutions such as @file{auto-autoloads.el}, but many cannot.

@item PRELOADS
A specification for loading libraries containing macros before compiling
the Lisp in the package.  This is spliced directly into the invocation
of XEmacs for byte-compilation, so it must contain the @samp{-l} flag
for XEmacs:

@example
PRELOADS=-l ./apackage-macros.el -l ../bpackage/lisp/bpackage-macros.el
@end example

@item INFO_FILES
Any Info file(s) generated by the package.  These must be paths relative
to the root of the package's source tree.

@item TEXI_FILES
The Texinfo source file(s).  These must be paths relative
to the root of the package's source tree.

@item MANUAL
The name to be used for Info files and man pages.

@item DATA_FILES
Any data files, such as pixmaps, READMEs, and ChangeLogs.  These must be
paths relative to the root of the package's source tree.

@item DATA_DEST
The installation location for data files, relative to the @file{etc/}
directory of the package hierarchy.  The normal value is simply
$(PACKAGE).  Leaving it empty (@emph{i.e.}, put it directly under
@file{etc/}) will probably work, but is subject to name conflicts with
other packages.
@end table

Rarely used variables.

@c @table @code
@c @item
@c @end table

@node Makefile Targets, , Makefile Variables, Creating Packages

The standard targets that need to be defined in your @file{Makefile}
follow.  These normally should @emph{not} have an action.  All of the
work should be done by dependent targets, usually having standard
definitions in the @xpms{}.

@table @samp
@item all
A list of generated files, usually byte-compiled Lisp libraries, to be
bundled in the package.  The typical dependencies are

@example
$(ELCS) auto-autoloads.elc custom-load.elc
@end example

Other targets (such as Info files) may need to be added as dependencies
for the @code{all} target.

@item srckit
The target for generating a source package.  Not implemented.  If it
were, the normal dependency would be @samp{srckit-std}.

@item binkit
The target for creating a ``master'' installation.  Binary packages are
actually generated by the @samp{bindist} target.  @xref{Building Packages}.
@end table

Standard dependencies for @code{srckit} and @code{binkit} are defined in
@file{XEmacs.rules}.  The most useful of these values are given in the
following table.

@table @samp
@item srckit-std
Build a standard source kit.  Not fully implemented.

@item binkit-sourceonly
The @samp{binkit} target need only install source and compiled Lisp in
the staging area.  There is nothing to install in a data directory or
info directory.

@item binkit-sourceinfo
Both source and info files are to be installed in the staging area.

@item binkit-sourcedata
Both source and etc (data) files are to be installed in the staging
area.

@item binkit-sourcedatainfo
Source, etc (data), and info files all are present and need to be
installed in the staging area.

@item binkit-common
A dependency for all the above.  (In fact in the current implementation
@samp{binkit-common} does all the work for all of the @samp{binkit}
targets.)
@end table

Data files include things like pixmaps for a package-specific toolbar,
and are normally installed in @file{etc/@var{PACKAGE_NAME}}.  A few
packages have needs beyond the basic templates.  See @file{XEmacs.rules}
or a future revision of this manual for details.


@node Issues, , Creating Packages, Packaging
@section Issues

To be completed.