EFS
***

This file documents EFS, a system for transparent file-transfer between remote hosts using the FTP protocol within Emacs.

This info is for version 1.15 of EFS.

Documentation version: 1.0

Copyright © 1991, 1992 Free Software Foundation, Inc.

Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies.

Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one.

Introducing EFS
***************

EFS is a system for transparent file-transfer between remote UNIX, Guardian, DOS, Macintosh, KA9Q, Netware, NOS/VE, Plan 9, TI Explorer, Twenex, TOPS 20, VOS, MPE, MVS, VMS, CMS or MTS hosts using FTP. This means that you can edit, copy and otherwise manipulate files on any machine you have access to from within Emacs as if it were a local file. EFS works by introducing an extended filename syntax, and overloading functions such as insert-file-contents so that accessing a remote file causes appropriate commands to be sent to an FTP process. EFS includes and enhanced version of Dired to facilitate directory browsing and multiple file transfer from remote hosts.

The authors of EFS are Sandy Rutherford (sandy@math.ubc.ca) and Andy (Ange) Norman (ange@hplb.hpl.hp.com).  EFS is partly based on an earlier package called ange-ftp.  The integration of EFS 1.15 into XEmacs and numerous bug fixes were done by Mike Sperber (sperber@informatik.uni-tuebingen.de).

Many people have sent in enhancements for ange-ftp and EFS. Members of the ange-ftp and EFS Hall of Fame include:

  * Many thanks to Roland McGrath for improving the filename syntax handling, for suggesting many enhancements and for numerous cleanups to the code.

  * Thanks to Jamie Zawinski for bugfixes and for ideas such as gateways.

  * Thanks to Ken Laprade for improved .netrc parsing and password reading, and Dired/shell autoloading.

  * Thanks to Sebastian Kremer for tree dired support and for many ideas and bugfixes.

  * Thanks to Joe Wells for bugfixes, non-UNIX system support, VOS support, and hostname completion.

  * Thanks to Nakagawa Takayuki for many good ideas, filename-completion, help with file-name expansion, efficiency worries, stylistic concerns and many bugfixes.

  * Also, thanks to Rob Austein, Doug Bagley, Andy Caiger, Jim Franklin, Noah Friedman, Aksnes Knut-Havard, Elmar Heeb, John Interrante, Roland McGrath, Jeff Morgenthaler, Mike Northam, Jens Petersen, Jack Repenning, Joerg-Martin Schwarz, Michael Sperber, Svein Tjemsland, Andy Whitcroft, Raymond A. Wiker and many others whose names have been forgotten who have helped to debug and fix problems.

Finally, this info file was put together by Mike Sperber (sperber@informatik.uni-tuebingen.de) from an ange-ftp info file written by Dave Smith (dsmith@stats.adelaide.edu.au) and the EFS source code.

Installing EFS
**************

At the time of writing of this documentation, the version of EFS distributed with XEmacs (this means the EFS running on XEmacs 19.15/20.1) is the latest version available.  It includes many bugfixes and some enhancements over the last separately released version of EFS, 1.15.  This situation will change once a new version of EFS is available.

Generally, EFS is pretty easy to get hold of. FTP is the probably the simplest method, but other options such as mail are available.

Once you have the Emacs-Lisp source, there are a few customisations you might need to make. The ideal configuration is to have the FTP process running on the same machine as you are running Emacs on, but this is not always possible since some machines cannot access hosts outside the local network. In this case, the FTP process needs to be run on a machine which does have access to the local world --- this is called the gateway host. EFS has facilities to make use of a gateway host when accessing remote hosts.

How to get the EFS source code
==============================

The latest separately distributed version of EFS should always be available from Andy Norman's home page at

     http://www-uk.hpl.hp.com/people/ange/efs

There are also some ftp locations:

Switzerland

          /anonymous@itp.ethz.ch:/sandy/efs/

Massachusetts, USA

          /anonymous@alpha.gnu.ai.mit.edu:/efs/

California, USA

          /anonymous@ftp.hmc.edu:/pub/emacs/packages/efs/

Failing these, someone on the EFS mailing list (see Getting help) may be able to help you find the latest version.

Installing the source
=====================

For byte compiling the EFS package, a Makefile is provided.  You should follow the instructions at the top of the Makefile.  If you have any problems, please let us know so that we can fix them for other users. Don't even consider using eFS without byte compiling it. It will be far too slow.

If you decide to byte compile efs by hand, it is important that the file efs-defun.el be byte compiled first, followed by efs.el. The other files may be byte compiled in any order.

To use EFS, simply put the byte compiled files in your load path and add

     (require 'efs)

in your .emacs file.  Note this takes awhile, and some users have found this to be unbearably slow.  Therefore ...

If you would like efs to be autoloaded when you attempt to access a remote file, put

     (require 'efs-auto)

in your .emacs file. Note that there are some limitations associated with autoloading EFS. A discussion of them is given at the top of efs-auto.el.

Note that, in XEmacs, EFS automatically loads efs-auto when the user accesses a remote file.  Therefore, no additional require statements should be necessary to use EFS.  Just fire away ...

The above instructions should allow you to access all hosts that your local machine can access. If your local host has limited access, however, you may wish to have EFS working through a gateway machine. If so, read on. Otherwise, see Using EFS to get started using EFS.

Using a gateway
===============

Sometimes it is necessary for the FTP process to be run on a different machine than the machine running Emacs.  This can happen when the local machine has restrictions on what hosts it can access.

Suppose you are running Emacs (and EFS, of course) on a machine X (let's call it the `local host') and you want to access a file on a machine Z (which we will call the `remote host'). Unfortunately, X does not have FTP access to Z: when you try a manual FTP something like the following happens:

     X$ ftp Z.foo.bar.com
     ftp: connect: Host is unreachable

However, X does have access to a machine Y (the `gateway machine') which can access Z. Fortunately, you have an account on the gateway machine, and so the solution is to login to Y, ftp to Z, download the file you want from Z to Y, and then copy it from Y to the local host, X. This can get a bit tedious, to say the least, but fortunately EFS can do all the hard work for you.

Setting up a gateway
====================

 1. Set the variable efs-gateway-host to the name of a machine that doesn't have the access restrictions.  If you need to use a nonstandard port to access this host for gateway use, then specify efs-gateway-host as <hostname>#<port>.

 2. Set the variable efs-ftp-local-host-regexp to a regular expression that matches hosts that can be contacted from running a local ftp process, but fails to match hosts that can't be accessed locally.

     For example:

               "\\.hp\\.com$\\|^[^.]*$"

     will match all hosts that are in the .hp.com domain, or don't have an explicit domain in their name, but will fail to match hosts with explicit domains or that are specified by their ip address.

 3. Set the variable efs-local-host-regexp to machines that you have direct TCP/IP access.  In other words, you must be able to ping these hosts.  Usually, efs-ftp-local-host-regexp and efs-local-host-regexp will be the same.  However, they will differ for so-called transparent gateways.  See below for more details.

 4. Set the variable efs-gateway-tmp-name-template to the name of a directory plus an identifying filename prefix for making temporary files on the gateway.  For example: "/tmp/hplose/ange/efs"

 5. If the gateway and the local host share cross-mounted directories, set the value of efs-gateway-mounted-dirs-alist accordingly. It is particularly useful, but not mandatory, that the directory of efs-gateway-tmp-name-template be cross-mounted.

 6. Set the variable efs-gateway-type to the type gateway that you have. This variable is a list, the first element of which is a symbol denoting the type of gateway.

Supported gateway types
=======================

local
     This means that your local host is itself the gateway.  However, it is necessary to use a different FTP client to gain access to the outside world.  If the name of the FTP client were xftp, you might set efs-gateway-type to

          (list 'local "xftp" efs-ftp-program-args)

     If xftp required special arguments, then give them in place of efs-ftp-program-args.

proxy

     This indicates that your gateway works by first FTP'ing to it, and then issuing a USER command of the form

          USER <username>@<host>

     In this case, you might set efs-gateway-type to

          (list 'proxy "ftp" efs-ftp-program-args)

     If you need to use a nonstandard client, such as iftp, give this instead of ftp.  If this client needs to take special arguments, give them instead of efs-ftp-program-args.

remsh

     For this type of gateway, you need to start a remote shell on your gateway, using either remsh or rsh.  You should set EFS-GATEWAY-TYPE to something like

          (list 'remsh "remsh" nil "ftp" efs-ftp-program-args)

     If you use rsh instead of remsh, change the second element from "remsh" to "rsh".  Note that the symbol indicating the gateway type should still be 'remsh.  If you want to pass arguments to the remsh program, give them as the third element.  For example, if you need to specify a user, make this (list "-l" "sandy"). If you need to use a nonstandard FTP client, specify that as the fourth element.  If your FTP client needs to be given special arguments, give them instead of efs-ftp-program-args.

interactive

     This indicates that you need to establish a login on the gateway, using either telnet or rlogin. You should set efs-gateway-type to something like

          (list 'interactive "rlogin" nil "exec ftp" efs-ftp-program-args)

     If you need to use telnet, then give "telnet" in place of the second element "rlogin".  If your login program needs to be given arguments, then they should be given in the third slot.  The fourth element is for the name of the FTP client program.  Giving this as "exec ftp", instead of "ftp", ensures that you are logged out if the FTP client dies.  If the FTP client takes special arguments, give these instead of efs-ftp-program-args.  Furthermore, you should see the documentation at the top of efs-gwp.el.  You may need to set the variables efs-gwp-setup-term-command, and efs-gwp-prompt-pattern.

raptor
     This is a type of gateway where efs is expected to specify a gateway user, and send a password for this user using the ACCOUNT command. For example, to log in to foobar.edu as sandy, while using the account ange on the gateway, the following commands would be sent:

          open raptorgate.com
          quote USER sandy@foobar.edu ange
          quote pass <sandy's password on foobar>
          quote account <ange's password on raptorgate>

     For such a gateway, you would set efs-gateway-type to

          (list 'raptor efs-ftp-program efs-ftp-program-args <GATEWAY USER>)

     where <GATEWAY USER> is the name of your account on the gateway.  In the above example, this would be "ange".  You can set your gateway password by simply setting an account password for the gateway host. This can be done with either efs-set-account, or within your .netrc file.  If no password is set, you will be prompted for one.

interlock
     This is a type of gateway where you are expected to send a PASS command after opening the connection to the gateway. The precise login sequence is

          open interlockgate
          quote PASS <sandy's password on interlockgate>
          quote USER sandy@foobar.edu
          quote PASS <sandy's password on foobar.edu>

     For such a gateway, you should set efs-gateway-type to

          (list 'interlock efs-ftp-program efs-ftp-program-args)

     If you need to use a nonstandard name for your FTP client, then replace efs-ftp-program with this name.  If your FTP client needs to take nonstandard arguments, then replace efs-ftp-program-args with these arguments.

     If your gateway returns both a 220 code and a 331 code to the "open interlockgate" command, then you should add a regular expression to efs-skip-msgs that matches the 220 response. Returning two response codes to a single FTP command is not permitted in RFC 959.  It is not possible for efs to ignore the 220 by default, because than it would hang for interlock installations which do not require a password.

kerberos
     With this gateway, you need to authenticate yourself by getting a kerberos "ticket" first.  Usually, this is done with the kinit program. Once authenticated, you connect to foobar.com as user sandy with the sequence: (Note that the "-n" argument inhibits automatic login. Although, in manual use you probably don't use it, EFS always uses it.)

          iftp -n
          open foobar.com
          user sandy@foobar.com

     You should set efs-gateway-type to something like

          (list 'kerberos "iftp" efs-ftp-program-args "kinit" <KINIT-ARGS>)

     If you use an FTP client other than iftp, insert its name instead of "iftp" above.  If your FTP client needs special arguments, give them as a list of strings in place of efs-ftp-program-args.  If the program that you use to collect a ticket in not called "kinit", then give its name in place of "kinit" above. <KINIT-ARGS> should be any arguments that you need to pass to your kinit program, given as a list of strings.  Most likely, you will give this as nil.

     See the file efs-kerberos.el for more configuration variables.  If you need to adjust any of these variables, please report this to us so that we can fix them for other users.

     If EFS detects that you are not authenticated to use the gateway, it will run the kinit program automatically, prompting you for a password. If you give a password in your .netrc file for login the value of efs-gateway-host and user kerberos, then EFS will use this to obtain gateway authentication.

Transparent gateways

     If your gateway is completely transparent (for example it uses socks), then you should set efs-gateway-type to nil.  Also, set efs-ftp-local-host-regexp to ".*".  However, efs-local-host-regexp, must still be set to a regular expression matching hosts in your local domain.  EFS uses this to determine which machines that it can open-network-stream to.  Furthermore, you should still set efs-gateway-host to the name of your gateway machine. That way EFS will know that this is a special machine having direct TCP/IP access to both hosts in the outside world, and hosts in your local domain.

Common Problems with Gateways
=============================

Spurious 220 responses
----------------------

Some proxy-style gateways (eg gateway type 'proxy or 'raptor), return two 3-digit FTP reply codes to the USER command. For example:

     open gateway.weird
     220 Connected to gateway.weird
     quote USER sandy@foobar
     220 Connected to foobar
     331 Password required for sandy

This is wrong, according to the FTP Protocol.  Each command must return exactly one 3-digit reply code.  It may be preceded by continuation lines.  What should really be returned is:

     quote USER sandy@foobar
     331-Connected to foobar.
     331 Password required for sandy.

or even

     quote USER sandy@foobar
     331-220 Connected to foobar.
     331 Password required for sandy.

Even though the "331-220" looks strange, it is correct protocol, and EFS will parse it properly.

If your gateway is returning a spurious 220 to USER, a work-around is to add a regular expression to efs-skip-msgs that matches this line.  It must not match the 220 line returned to the open command.  This work-around may not work, as some system FTP clients also get confused by the spurious 220.  In this case, the only solution is to patch the gateway server.  In either case, please send a bug report to the author of your gateway software.

Case-sensitive parsing of FTP commands
--------------------------------------

Some gateway servers seem to treat FTP commands case-sensitively. This is incorrect, as RFC 959 clearly states that FTP commands are always to be case-insensitive.  If this is a problem with your gateway server, you should send a bug report to its author. If efs is using a case for FTP commands that does not suit your server, a possible work-around is to edit the efs source so that the required case is used.  However, we will not be making any changes to the standard EFS distribution to support this type of server behaviour. If you need help changing the efs source, you should enquire with the efs-help mailing list.

Other user options
==================

Here are other user options available in EFS:

efs-netrc-filename: The name of a file in netrc(5) format that EFS will use to match hostnames, users and their respective passwords.  Hostnames specified here are also used for hostname completion. The default is "~/.netrc".

efs-default-user: If this is a string, it is the username to use when none is specified in a filename. If nil, then the name under which the user is logged in is used. If non-nil but not a string, the user is prompted for the name. The default is nil.

efs-default-password: The password to use when the user is the same as efs-default-user. The default is nil.

efs-default-account: Account password to use when the user is the same as efs-default-user. The default is nil.

efs-dumb-unix-host-regexp: The FTP servers on some machines have problems if the ls command is used. The usual indication that something is wrong is when EFS erroneously thinks that a directory is just a plain file. The routine efs-add-host can be called to tell EFS to limit itself to the DIR command and not ls for a given host (but this change will take effect for the current Emacs session only) when called like this:

     (efs-add-host 'dumb-unix "hostname")

If a large number of machines with similar hostnames have this problem then it is easier to change the value of this variable to a regexp which matches hostnames which have this problem, particularly since EFS cannot automatically detect such hosts. The default is nil.

efs-binary-file-name-regexp: By default EFS will transfer files in ASCII mode. If a file being transferred matches the value of this regexp then the FTP process will be toggled into BINARY mode before the transfer and back to ASCII mode after the transfer. The default is:

       (concat "\\." ; the dot
     	  ;; extensions
     	  "\\([zZ]\\|t?gz\\|lzh\\|arc\\|zip\\|zoo\\|ta[rz]\\|dvi\\|sit\\|"
     	  "ps\\|elc\\|gif\\|Z-part-..\\|tpz\\|exe\\|[jm]pg\\|TZ[a-z]?\\|lib\\)"
     	  "\\(~\\|~[0-9]+~\\)?$" ; backups
     	  "\\|"
     	  ;; UPPER CASE LAND
     	  "\\."
     	  "\\(ARC\\|ELC\\|TAGS\\|EXE\\|ZIP\\|DVI\|ZOO\\|GIF\\|T?GZ\\|"
     	  "[JM]PG\\)"
     	  "\\([.#;][0-9]+\\)?$" ; versions
     	  )

efs-hash-mark-size: EFS by default requests that the FTP process sends hash marks (just # characters) during transfers to keep track of how much data has been sent or received. This variable, if non-nil, should be the number of kilobytes represented by the FTP client's hash mark. The default value of 1 doesn't work for me --- I use 2 instead.

efs-verbose: If this is t then EFS will be chatty about interaction with the FTP process. The default is t.

efs-ftp-program-name: This should be the name of the FTP program to run on the local host. The default value of "ftp" should be fine for most systems.

efs-make-backup-files: A list of operating systems for which EFS will make Emacs backup files on the remote host. For example, '(unix) makes sense, but '(unix vms) or '(vms) would be silly, since VMS makes its own backups.  The host type is determined by the function efs-host-type.  Possible host types are: dumb-unix; vos; vms; mts; and unix. The default of nil means make no backups on remote hosts.

efs-skip-msgs: A regular expression matching messages from the ftp process that can be ignored. The default is

       (concat
        "^110 \\|" ; Restart marker reply.
        "^125 \\|" ; Data connection already open; transfer starting.
        "^150 ") ; File status OK; about to open connection.

but you might need to tweak it if EFS is giving up when it shouldn't.

efs-fatal-msgs: A regular expression matching messages from the FTP process that indicate something has gone drastically wrong attempting the action that was initiated and that the FTP process should (or already has) been killed. The default is

     (concat
        ;; RFC959 codes
        "^221 \\|" ; Service closing control connection.
        "^421 \\|" ; Service not available.
        "^425 \\|" ; Can't open data connection.
        "^426 \\|" ; Connection closed, transfer aborted.
        "^451 \\|" ; Requested action aborted, local error in processing.
        ;; RFC959 non-compliant codes
        "^552 Maximum Idle Time Exceded\\.$\\|" ; Hellsoft server uses this to
     					   ; indicate a timeout. 552 is
     					   ; supposed to be used for exceeded
     					   ; storage allocation. Note that
     					   ; they also misspelled the error
     					   ; message.
        ;; client problems
        "^ftp: \\|^Not connected\\|^rcmd: \\|^No control connection\\|"
        "^unknown host\\|: unknown host$\\|^lost connection\\|"
        "^[Ss]egmentation fault\\|"
        ;; Make sure that the "local: " isn't just a message about a file.
        "^local: [^/]\\|"
        ;; Gateways
        "^iftp: cannot authenticate to server\\b"
        )

efs-gateway-fatal-msgs: Regular expression matching messages from the rlogin / telnet process that indicates that logging in to the gateway machine has gone wrong. The default is

     "No route to host\\|Connection closed\\|No such host\\|Login incorrect"

efs-tmp-name-template: This should be a directory and a filename prefix indicating where EFS should make temporary files. The default of "/tmp/efs" should be fine for most systems.

efs-retry-time: Number of seconds to wait before retrying if a file or listing doesn't arrive. For slow connections, you might get a ``listing unreadable'' error messages or an empty buffer for a file that you know has something in it.  The solution is to increase the value of efs-retry-time.  Its default value is 5 which is plenty for reasonable connections.  However, for some transatlantic connections 20 might be a better value.

Using EFS
*********

Once installed, efs operates largely transparently. All files normally accessible to you on the internet, become part of a large virtual file system. These files are accessed using an extended file name syntax. To access file <path> on remote host <host> by logging in as user <user>, you simply specify the full path of the file as /<user>@<host>:<path>. Nearly all Emacs file handling functions work for remote files. It is not possible to access remote files using shell commands in an emacs *shell* buffer, as such commands are passed directly to the shell, and not handled by emacs.

FTP is the underlying utility that efs uses to operate on remote files.

For example, if find-file is given a filename of:

     /ange@anorman:/tmp/notes

then EFS will spawn an FTP process, connect to the host 'anorman' as user 'ange', get the file /tmp/notes and pop up a buffer containing the contents of that file as if it were on the local file system.  If efs needed a password to connect then it would prompt the user in the minibuffer. For further discussion of the EFS path syntax, see the paragraph on extended file name syntax Remote filenames.

Full file-name completion is supported on every type of remote host.  To do filename completion, EFS needs a listing from the remote host. Therefore, for very slow connections, it might not save any time. However, the listing is cached, so subsequent uses of file-name completion will be just as fast as for local file names.

Using nonstandard ports
=======================

EFS supports the use of nonstandard ports on remote hosts.  To specify that port <port> should be used, give the host name as host#<port>. Host names may be given in this form anywhere that efs normally expects a host name. This includes in the .netrc file. Logically, EFS treats different ports to correspond to different remote hosts.

Extended filename syntax
========================

The default full EFS path syntax is

     /<user>@<host>#<port>:<path>

Both the #<port>'and <user>@ may be omitted.

If the #<port> is omitted, then the default port is taken to be 21, the usual FTP port. For most users, the port syntax will only very rarely be necessary.

If the <user>@ is omitted, then EFS will use a default user.  If a login token is specified in your .netrc file, then this will be used as the default user for <host>.  Otherwise, it is determined based on the value of the variable efs-default-user.

This EFS path syntax can be customised to a certain extent by changing a number of variables.  To undertake such a customization requires some knowledge about the internal workings of EFS.

Passwords
=========

A password is required for each host / user pair.  This will be prompted for when needed, unless already set by calling efs-set-passwd, or specified in a valid ~/.netrc file.

When EFS prompts for a password, it provides defaults from its cache of currently known passwords.  The defaults are ordered such that passwords for accounts which have the same user name as the login which is currently underway have priority. You can cycle through your list of defaults with C-n to cycle forwards and C-p to cycle backwards. The list is circular.

Passwords for anonymous user
----------------------------

Passwords for the user anonymous (or ftp) are handled specially. The variable efs-generate-anonymous-password controls what \vindex efs-generate-anonymous-password happens. If the value of this variable is a string, then this is used as the password; if non-nil, then a password is created from the name of the user and the hostname of the machine on which Emacs is running; if nil (the default) then the user is prompted for a password as normal.

Account passwords
-----------------

Some FTP servers require an additional password which is sent by the ACCOUNT command.  EFS will detect this and prompt the user for an account password if the server expects one.  Also, an account password can be set by calling efs-set-account, or by specifying an account token in the .netrc file.

Some operating systems, such as CMS, require that ACCOUNT be used to give a write access password for minidisks.  efs-set-account can be used to set a write password for a specific minidisk. Also, tokens of the form

     minidisk <minidisk name> <password>

may be added to host lines in your .netrc file. Minidisk tokens must be at the end of the host line, however there may be an arbitrary number of them for any given host.

Using Dired
===========

This feature of EFS is particularly useful when file transfers, as opposed to file editing, are the order of the day. Simply run find-file on a directory to get a listing of the files in that directory. For example, you might run find-file on

     /anonymous@archive.site.com:pub

to see what's in the pub directory of your favourite archive site. This brings up a Dired buffer of all the files in that directory. The f command is useful for looking at README files --- if you then decide to save it C-x C-w is useful. You can also use this method to copy files, but the c command is easier. The f command can also be used to descend the directory tree by applying it to directories.

You can also use Dired to refresh EFS's internal cache. If you (or anybody else) has changed a remote directory since you first accessed it with EFS, completion is not provided on any new files that EFS does not know about. If you have (or create) a Dired buffer which contains the modified directory, executing revert-buffer with a prefix argument (C-u g in the Dired buffer) will force a refresh of both the the buffer and also EFS's internal cache. If you find that filename completion isn't working on a file that you know is there, this is how to fix the problem.

Dired provides facilities for maintaining an entire directory tree in a Dired buffer, for marking files which match a certain regexp (or you can select files interactively) and then copying all those files to your local host (or even a different remote host). Another useful feature is Virtual Dired, which allows you to save Dired buffers of remote hosts, allowing you to browse them at a later date without actually needing to connect to the host.

Using a .netrc file
===================

Being prompted for passwords all the time can get rather annoying, but there is a way to fix the problem --- a .netrc (but see Other options and efs-netrc-filename if you want another filename) file in your home directory. Basically, this is a file (in the format of Unix netrc(5)) which contains the names of all the machines you regularly login to, as well as the username and password you use for that machine. You can also supply an account password, if required.

Your .netrc file consists of lines of the form

     machine <machine-name> login <user-name> password <password>

It doesn't all have to be on the one line, though: any login or password commands in the file refer to the previous machine command. You can also have account <account-passwd> commands if you need special account passwords.

For example, you might have the following line in your .netrc:

     machine Y.local.lan.edu login myname password secret

Then if you run find-file on the file /Y.local.lan.edu:somefile you will automatically be logged in as user myname with password secret. You can still login under another name and password, if you so desire: just include the user@ part of the filename.

You may also include a default option, as follows:

     default login <user-name> password <password>

which applies to any other machines not mentioned elsewhere in your .netrc. A particularly useful application of this is with anonymous logins:

     default login myname password myname@myhost.edu

so that accessing /anyhost:anyfile will automatically log you in anonymously, provided the host is not mentioned in the .netrc. Note also that if the value of efs-default-user is non-nil, its value will have precedence over the username supplied in the default option of the .netrc.

The .netrc file is also useful in another regard: machines included in it are provided with hostname completion. That is, for any machine in the .netrc, you need only type a slash and the first few characters of its name and then press TAB to be logged in automatically with a username and password from the .netrc file. So it's a good idea to put hosts you use regularly in your .netrc as well:

     machine archive.site.com login anonymous password myname@X.local.lan.edu

EFS commands
============

EFS supplies a few interactive commands to make connecting with hosts a little easier.

Command efs-set-user: Prompts for a hostname and a username. Next time access to the host is attempted, EFS will attempt to log in again with the new username.

Command efs-set-passwd: Prompts for a hostname, user and password. Future logins to that host as that user will use the given password.

Command efs-set-account: Prompts for a hostname, user and account. Future logins to that host as that user will use the given account.

Note that the effects of the above three commands only last the duration of the current Emacs session. To make their effects permanent, you may include them as lisp code in your .emacs:

     (efs-set-user HOST USER)
     (efs-set-password HOST USER PASSWORD)
     (efs-set-account HOST USER ACCOUNT)

This is an alternative to using a .netrc; see Using a .netrc.

Command efs-kill-ftp-process: kill the FTP process associated with a given buffer's filename (by default the current buffer). This is an easy way to achieve a resynch: any future accesses to the remote host will cause the FTP process to be recreated.

FTP processes
=============

When EFS starts up an FTP process, it leaves it running for speed purposes.  Some FTP servers will close the connection after a period of time, but EFS should be able to quietly reconnect the next time that the process is needed.

The FTP process will be killed should the associated *ftp user@host* buffer be deleted.  This should not cause efs any grief.

Showing background FTP activity on the mode-line
------------------------------------------------

After EFS is loaded, the command efs-display-ftp-activity will cause background FTP activity to be displayed on the mode line. The variable efs-mode-line-format is used to determine how this data is displayed. efs does not continuously track the number of active sessions, as this would cause the display to change too rapidly. Rather, it uses a heuristic algorithm to determine when there is a significant change in FTP activity.

File types
----------

By default EFS will assume that all files are ASCII. If a file being transferred matches the value of efs-binary-file-name-regexp then the file will be assumed to be a binary file, and EFS will transfer it using "type image". ASCII files will be transferred using a transfer type which efs computes to be correct according to its knowledge of the file system of the remote host. The command efs-prompt-for-transfer-type toggles the variable efs-prompt-for-transfer-type. When this variable is non-nil, EFS will prompt the user for the transfer type to use for every FTP transfer.  Having this set all the time is annoying, but it is useful to give special treatment to a small set of files.  There is also a variable efs-text-file-name-regexp.  This is tested before efs-binary-file-name-regexp, so if you set efs-text-file-name-regexp to a non-trivial regular expression, and efs-binary-file-name-regexp to ".*", the result will to make image the default tranfer type.

Also, if you set efs-treat-crlf-as-nl, then EFS will use type image to transfer files between hosts whose file system differ only in that one specifies end of line as CR-LF, and the other as NL.  This is useful if you are transferring files between UNIX and DOS machines, and have a package such as dos-mode.el, that handles the extra ^M's.

Status reports
--------------

Most EFS commands that talk to the FTP process output a status message on what they are doing.  In addition, efs can take advantage of the FTP client's HASH command to display the status of transferring files and listing directories.  See the documentation for the variables efs-hash-mark-size, efs-send-hash and efs-verbose for more details.

Caching of directory information
--------------------------------

EFS keeps an internal cache of file listings from remote hosts. If this cache gets out of synch, it can be renewed by reverting a dired buffer for the appropriate directory (dired-revert is usually bound to g).

Alternatively, you can add the following two lines to your .emacs file if you want C-r to refresh EFS's cache whilst doing filename completion.

     (define-key minibuffer-local-completion-map "\C-r" 'efs-re-read-dir)
     (define-key minibuffer-local-must-match-map "\C-r" 'efs-re-read-dir)

Tips for using EFS
==================

 1. Beware of compressing files on non-UNIX hosts. EFS will do it by copying the file to the local machine, compressing it there, and then sending it back. Binary file transfers between machines of different architectures can be a risky business. Test things out first on some test files. See Bugs Also, note that EFS sometimes copies files by moving them through the local machine. Again, be careful when doing this with binary files on non-Unix machines.

 2. Beware that dired over ftp will use your setting of dired-no-confirm (list of dired commands for which confirmation is not asked). You might want to reconsider your setting of this variable, because you might want confirmation for more commands on remote direds than on local direds. For example, I strongly recommend that you not include compress in this list. If there is enough demand it might be a good idea to have an alist efs-dired-no-confirm of pairs ( TYPE . LIST ), where TYPE is an operating system type and LIST is a list of commands for which confirmation would be suppressed.  Then remote dired listings would take their (buffer-local) value of dired-no-confirm from this alist. Who votes for this?

 3. Some combinations of FTP clients and servers break and get out of sync when asked to list a non-existent directory.  Some of the ai.mit.edu machines cause this problem for some FTP clients. Using efs-kill-ftp-process can be used to restart the ftp process, which should get things back in synch.

 4. Some ftp servers impose a length limit on the password that can be sent. If this limit is exceeded they may bomb in an incomprehensible way. This sort of behaviour is common with MVS servers. Therefore, you should beware of this possibility if you are generating a long password (like an email address) with efs-generate-anonymous-password.

 5. Some antiquated FTP servers hang when asked for an RNFR command. EFS sometimes uses this to test whether its local cache is stale. If your server for HOST hangs when asked for this command, put

          (efs-set-host-property HOST 'rnfr-failed t)

     in your efs-ftp-startup-function-alist entry for HOST.

 6. The FTP servers on some Unix machines have problems if the ls command is used.  EFS will try to correct for this automatically, and send the dir command instead.  If it fails, you can call the function efs-add-host, and give the host type as dumb-unix.  Note that this change will take effect for the current Emacs session only. To make this specification for future emacs sessions, put

          (efs-add-host 'dumb-unix "hostname")

     in your .emacs file. Also, please report any failure to automatically recognize dumb unix to the "bugs" address given below, so that we can fix the auto recognition code.

Descriptive directory listings
==============================

Some hosts (such as cs.uwp.edu) now use descriptive directory listings (which in fact contain less information than the standard listing!) when issued the ls command, and EFS has been modified to cope with this. EFS can detect such listings, but if you regularly use a remote host which uses this extended listing format you should set the variable efs-dl-dir-regexp to a regular expression which matches directories using the extended listing format. You shouldn't anchor the regexp with $ -- that way the regexp will match subdirectories as well.  Alternatively, you can use the interactive command efs-add-dl-dir to temporarily add a remote directory for this Emacs session only.

Dired has been modified to work with such descriptive listings.

Using EFS with non-Unix hosts
=============================

EFS also works with some non-Unix hosts, although not necessarily with all the features available with Unix hosts. VMS, CMS, and MTS systems will all now work with EFS and Dired.  It also works with a whole bunch of others, but documentation for that has not been written yet. This section was taken straight from the ange-ftp manual, and is therefore in all likelihood out-of-date.

EFS should be able to automatically detect which type of host you are using (VMS, CMS or MTS), but if it is unable to do so you can fix the problem by setting the appropriate efs-TYPE-host-regexp variable (where TYPE is one of vms, cms or mts) -- see below. If EFS is unable to automatically detect any VMS, CMS or MTS host, please report this as a bug: see Bugs.

In all cases the file-name conventions of the remote host are converted to a UNIX-ish format, and this is the format you should use to find files on such hosts.

VMS support
-----------

VMS filenames are of the form FILE.TYPE;##, where both FILE and TYPE can be up to 39 characters long, and ## is an integer version number between 1 and 32,767. Valid characters in filenames are A-Z, 0-9, _, - and $, however $ cannot begin a filename and - cannot be used as the first or last character.

Directories in VMS are converted to the standard UNIX / notation. For example, the VMS filename

     PUB$:[ANONYMOUS.SDSCPUB.NEXT]README.TXT;1

would be entered as

     /PUB$$:/ANONYMOUS/SDSCPUB/NEXT/README.TXT;1

(The double $ is required to prevent Emacs from attempting to expand an environment variable.)  Similarly, to anonymously FTP the file [.CSV.POLICY]RULES.MEM;1 from ymir.claremont.edu you would type C-x C-f /anonymous@ymir.claremont.edu:CSV/POLICY/RULES.MEM;1. You can always drop off the ;## part at the end of the filename to get the latest version.

Sandy Rutherford provides some tips for using VMS hosts:

  * Although VMS is not case sensitive, EMACS running under UNIX is. Therefore, to access a VMS file, you must enter the filename with upper case letters.

  * To access the latest version of file under VMS, you use the filename without the ; and version number. You should always edit the latest version of a file. If you want to edit an earlier version, copy it to a new file first. This has nothing to do with EFS, but is simply good VMS operating practice. Therefore, to edit FILE.TXT;3 (say 3 is latest version), do C-x C-f /ymir.claremont.edu:FILE.TXT. If you inadvertently do

          C-x C-f /ymir.claremont.edu:FILE.TXT;3

     you will find that VMS will not allow you to save the file because it will refuse to overwrite FILE.TXT;3, but instead will want to create FILE.TXT;4, and attach the buffer to this file. To get out of this situation, M-x write-file /ymir.claremont.edu:FILE.TXT will attach the buffer to latest version of the file. For this reason, in Dired f (dired-find-file), always loads the file sans version, whereas v, (dired-view-file), always loads the explicit version number. The reasoning being that it reasonable to view old versions of a file, but not to edit them.

  * VMS filenames often contain $ characters: make sure you always quote these as $$ and watch out for the Emacs bug which fails to quote $'s when defaults are presented in the minibuffer: see Bugs.

EFS should automatically detect that you are using a VMS host. If it fails to do so (which should be reported as a bug) you can use the command efs-add-vms-host to inform EFS manually. For a more permanent effect, or if you use a VMS host regularly, it's a good idea to set efs-vms-host-regexp to a regular expression matching that host's name. For instance, if use use ymir.claremont.edu a lot, place the following in your .emacs:

     (setq efs-vms-host-regexp "^ymir.claremont.edu$")

CMS support
-----------

EFS has full support, including Dired support, for hosts running CMS.

CMS filenames are entered in a UNIX-y way. Minidisks are treated as UNIX directories; for example to access the file READ.ME in minidisk *.311 on cuvmb.cc.columbia.edu, you would enter

     /anonymous@cuvmb.cc.columbia.edu:/*.311/READ.ME

If *.301 is the default minidisk for this account, you could access FOO.BAR on this minidisk as

     /anonymous@cuvmb.cc.columbia.edu:FOO.BAR

CMS filenames are of the form FILE.TYPE, where both FILE and TYPE can be up to 8 characters. Again, beware that CMS filenames are always upper case, and hence must be entered as such.

Sandy Rutherford provides some tips on using CMS hosts:

  * CMS machines, with the exception of anonymous accounts, nearly always need an account password. To have EFS send an account password, you can either include it in your .netrc (see Using a .netrc), or use efs-set-account.

  * EFS cannot send ``write passwords'' for a minidisk. Hopefully, we can fix this.

EFS should automatically detect that you are using a CMS host. If it fails to do so (which should be reported as a bug) you can use the command efs-add-cms-host to inform EFS manually. For a more permanent effect, or if you use a CMS host regularly, it's a good idea to set efs-cms-host-regexp to a regular expression matching that host's name.

MTS support
-----------

EFS has full support, including Dired support, for hosts running the Michigan terminal system, and should be able to automatically recognise any MTS machine.

MTS filenames are entered in a UNIX-y way. For example, if your account was YYYY, the file FILE in the account XXXX: on mtsg.ubc.ca would be entered as

     /YYYY@mtsg.ubc.ca:/XXXX:/FILE

In other words, MTS accounts are treated as UNIX directories. Of course, to access a file in another account, you must have access permission for it.  If FILE were in your own account, then you could enter it in a relative path fashion as

     /YYYY@mtsg.ubc.ca:FILE

MTS filenames can be up to 12 characters. Like UNIX, the structure of the filename does not contain a type (i.e. it can have as many .'s as you like.) MTS filenames are always in upper case, and hence be sure to enter them as such! MTS is not case sensitive, but an EMACS running under UNIX is.

EFS should automatically detect that you are using an MTS host. If it fails to do so (which should be reported as a bug) you can use the command efs-add-mts-host to inform EFS manually. For a more permanent effect, or if you use an MTS host regularly, it's a good idea to set efs-mts-host-regexp to a regular expression matching that host's name.

File- and host-name completion
==============================

Full filename completion is supported on all remote UNIX hosts and some non-Unix hosts.  Hostnames also have completion if they are mentioned in the .netrc and no username is specified. However using the filename completion feature can be a bit of a two edged sword.

To understand why, we need to discuss how EFS works. Whenever EFS is asked to find a remote file (or directory) an ls command is sent to the FTP process to list all the files in the directory. This list is maintained in an internal cache, to provide filename completion for later requests on that directory. EFS keeps this cache up-to-date by monitoring Emacs commands which affect files and directories, but if a process outside Emacs (such as another user) changes a directory (e.g. a new file is added) completion won't work on that file since EFS doesn't know about it yet. The solution if to force EFS to reread the directory and update it's cache, and the easiest way to do that is with Dired --- see Using Dired to see how.

Another problem is that the ls command can take a long time, especially when dealing with distant hosts over slow links. So if you're after a file in the pub/images directory but nothing else, it's a better idea to type pub/images/file TAB than pub/im TAB which will force a read of the pub directory (since EFS needs to know how to complete im). A little extra typing can often save a lot of waiting. Don't be afraid to use the TAB key once the directory is cached, though.

Accessing the FTP process buffer
================================

The FTP process used to access the remote files is available for access if you wish. It will be in a buffer called "*ftp REMOTE-FILE-NAME*", i.e. if you found the file

     /anonymous@archive.site.com:pub/README

there will be a buffer

     *ftp anonymous@archive.site.com*

where all the transfers are taking place. You can have a look at the buffer using C-x b as usual, and even type in commands to the FTP process under an interface very much like shell-mode. There are two instances when doing this can be very useful: one is accessing non-UNIX hosts, where Dired and filename completion may not work (if EFS even works at all).  If you are going to use mget or mput, make sure you type glob first: EFS turns globbing off by default. Don't be afraid of changing directories, either --- EFS always uses absolute pathnames when communicating with the FTP process.

You can kill the FTP process at any time simply by killing this buffer. You can also call efs-kill-ftp-process. This won't cause EFS any grief whatsoever --- if you later make another request to that host, EFS will simply fire up another process and create a new buffer to hold it.

Getting help
************

EFS has its own mailing list called efs-help.  All users of EFS are welcome to subscribe (see below) and to discuss aspects of EFS.

To [un]subscribe to efs-help, or to report mailer problems with the list, please mail one of the following addresses:

     efs-help-request@cuckoo.hpl.hp.com

or

     efs-help-request%cuckoo.hpl.hp.com@hplb.hpl.hp.com

Please don't forget the -request part.

For mail to be posted directly to efs-help, send to one of the following addresses:

     efs-help@cuckoo.hpl.hp.com

or

     efs-help%cuckoo.hpl.hp.com@hplb.hpl.hp.com

Alternatively, there is a mailing list that only gets announcements of new EFS releases.  This is called efs-announce, and can be subscribed to by e-mailing to the -request address as above.  Please make it clear in the request which mailing list you wish to join.

Mailing list archives are also accessible from this web page:

     http://www-uk.hpl.hp.com/people/ange/efs

Bugs and Wish List
******************

If you find any bugs or problems with this package, please e-mail the authors. Ideas and constructive comments are especially welcome. So are any enhancements to EFS, preferably debugged and documented. Also welcome are any typo fixes, corrections or additions to this manual.

Here is a list of known bugs:

If you hit a bug in this list, please report it anyway. Most of the bugs here remain unfixed because they are considered too esoteric to be a high priority. If one of them gets reported enough, we will likely change our view on that.

 1. EFS does not check to make sure that when creating a new file, you provide a valid filename for the remote operating system. If you do not, then the remote FTP server will most likely translate your filename in some way. This may cause EFS to get confused about what exactly is the name of the file.

 2. For CMS support, we send too many cd's. Since cd's are cheap, I haven't worried about this too much. Eventually, we should have some caching of the current minidisk. This is complicated by the fact that some CMS servers lie about the current minidisk, so sending redundant cd's helps us recover in this case.

 3. The code to do compression of files over ftp is not as careful as it should be. It deletes the old remote version of the file, before actually checking if the local to remote transfer of the compressed file succeeds. Of course to delete the original version of the file after transferring the compressed version back is also dangerous, because some OS's have severe restrictions on the length of filenames, and when the compressed version is copied back the "-Z" or ".Z" may be truncated. Then, EFS would delete the only remaining version of the file.  Maybe EFS should make backups when it compresses files (of course, the backup "~" could also be truncated off, sigh...). Suggestions?

 4. If a dir listing is attempted for an empty directory on (at least some) VMS hosts, an ftp error is given. This is really an ftp bug, and I don't know how to get EFS work to around it.

 5. EFS gets confused by directories containing file names with embedded newlines. A temporary solution is to add "q" to your dired listing switches. As long as your dired listing switches also contain "l" and either "a" or "A", EFS will use these switches to get listings for its internal cache. The "q" switch should force listings to be exactly one file per line. You still will not be able to access a file with embedded newlines, but at least it won't mess up the parsing of the rest of the files.

 6. EFS cannot parse symlinks which have an embedded " -> " in their name. It's alright to have an embedded " -> " in the name of any other type of file. A fix is possible, but probably not worth the trouble. If you disagree, send us a bug report.

 7. EFS doesn't handle context-dep. files in H-switch listings on HP's. It wouldn't be such a big roaring deal to fix this. I'm waiting until I get an actual bug report though.

 8. If a hard link is added or deleted, EFS will not update its internal cache of the link count for other names of the file. This may cause file-nlinks to return incorrectly. Reverting any dired buffer containing other names for the file will cause the file data to be updated, including the link counts. A fix for this problem is known and will be eventually implemented. How it is implemented will depend on how we decide to handle inodes. See below.

 9. EFS is unable to parse R-switch listings from remote Unix hosts. This is inefficient, because EFS will insist on doing individual listings of the subdirectories to get its file information. This may be fixed if there is enough demand.

 10. In file-attributes, EFS returns a fake inode number. Of course this is necessary, but this inode number is not even necessarily unique.  It is simply the sum of the characters (treated as integers) in the host name, user name, and file name. Possible ways to get a unique inode number are:

      1. Simply keep a count of all remote file in the cache, and return the file's position in this count as a negative number.

      2. For unix systems, we could actually get at the real inode number on the remote host, by adding an "i" to the ls switches.  The inode numbers would then be removed from the listing returned by efs-ls, if the caller hadn't requested the "i" switch. We could then make a unique number out of the host name and the real inode number.

 11. EFS tries to determine if a file is readable or writable by comparing the file modes, file owner, and user name under which it is logged into the remote host. This does not take into account groups. We simply assume that the user belongs to all groups. As a result we may assume that a file is writable, when in fact it is not. Groups are tough to handle correctly over FTP. Suggestions? (For new FTP servers, can do a "QUOTE SITE EXEC groups" to handle this.)