Please reproduce this issue in emacs -Q.

This is a sentence you will often read when you report bugs in Emacs packages; it's often a quick reply from a developer and goes with no further explanation or even the slightest clue on what you’re supposed to do. If you ever found yourself in that situation then this post is for you: I’ll explain what it means to “reproduce a bug in emacs -Q”, how to do that properly, and why developers ask you for this.

How do you reproduce a bug in emacs -Q?

emacs -Q provides a “pure” Emacs session which loads no packages and absolutely no configuration files.

There is also emacs -q which will make Emacs ignore your init (e.g. ~/.emacs or ~/.emacs.d/init.el) and packages that you installed with the built-in packages, but still load code from the global Emacs site which includes Emacs extensions that you installed with the system’s package manager (e.g. apt-get). As such this option is insufficient to reproduce bugs because there may still be packages that infer.

To start one open a terminal window and type emacs -Q. On OS X use the slightly more convoluted command open --new -a Emacs.app --args -Q to start a fresh GUI Emacs.

Now initialise Emacs’ package system with M-x package-initialize. Normally this happens in your init file but since emacs -Q doesn’t load it you need to do that explicitly. If you omit this step you can’t access your installed packages in emacs -Q so it’s a pretty important one.

On OS X you should also run M-x exec-path-from-shell-initialize now, provided that you installed exec-path-from-shell in your Emacs. This ensures that your $PATH in Emacs is correct, i.e. that you have access to the same executables as in your terminal.

Eventually enable the buggy package by following its setup instructions. You should now manually execute the commands that you copied to your init file when you installed the package in your Emacs. For instance, if you’d like to reproduce a bug in Flycheck you should now type M-x global-flycheck-mode to enable Flycheck.

Now you’re ready to reproduce the bug, simply by following the same steps that triggered the bug in your normal Emacs session. For instance, if you’d like to show a broken syntax checker in Flycheck visit the same file that triggered the bug in your normal Emacs session.

While following these steps please take notes about the commands that you run, the keys that you type and the things that you do to reproduce the bug, and include these notes in your bug report.

Why do developers ask for this?

As package developers we ask you to reproduce a bug in this session because it’s an essential first step to pin down the bug.

Emacs provides absolutely no isolation between all the packages that are loaded in your normal Emacs session. This is blessing and curse at the same time: It allows you to customise almost everything in Emacs but a single faulty package can break down your entire Emacs session. We can’t tell if the bug that you reported to us is really a bug in our own package or if there is another buggy or misbehaving package in your Emacs configuration that breaks our package.

If you can reproduce a bug in emacs -Q you give us an important clue: The bug is in our own code and we only need to look at our own code to find it. That is much simpler than looking at your entire Emacs configuration and all the other packages that you have installed. What’s more you also enable us to reproduce the bug ourselves with just our package and a pure Emacs session, and if we can reproduce a bug we are halfway to fixing it.

Emacs isn't just an editor, it’s an entire Emacs Lisp interpreter and environment. We can use Emacs Lisp not only to extend and customize our beloved editor, but also to write entire programs and applications. Nic Ferrier’s elnode server is the most ambitious Emacs Lisp application of this sort, but we can start at a smaller scale and try to write our shell scripts and tools with Emacs Lisp.

However, it turns out that writing programs in Emacs Lisp is more intricate than it looks at a first glance. Emacs decades-long history as interactive application have left deep marks in Emacs and Emacs Lisp, which make independent noninteractive scripts difficult.

Making Emacs Lisp scripts executable

In the early days, we’d muck about with --no-init-file, --batch and --load to enter noninteractive mode an.e.d load a file. Nowadays Emacs has a convenient --script option to load and evaluate a specific file, but how to make a proper shebang out of it? The naive approach won't do:

#!/usr/bin/emacs --script
(message "Hello world")

Emacs is not /bin/sh, and its location varies between different systems. There may even be different Emacs versions at different places. For instance, on OS X /usr/bin/emacs is an outdated Emacs 22, and the “real” Emacs is typically installed via Homebrew at /usr/local/bin/emacs.

Normally, we'd accommodate these differences with /usr/bin/env:

#!/usr/bin/env emacs --script
(message "Hello world")

But this just raises another portability issue: Linux doesn’t split arguments in the shebang, and sends emacs --script as a single argument to /usr/bin/env, which doesn’t really do the trick.

To make our script executable in a portable and reliable way, we need to resort to some dirty trickery (see https://stackoverflow.com/a/6259330/355252):

#!/bin/sh
":"; exec emacs --script "$0" "$@" # -*- mode: emacs-lisp; lexical-binding: t; -*-
(message "Hello world")

This wraps the Emacs Lisp code into a POSIX shell script which calls out to emacs with appropriate arguments. The semicolon in the second line hides the exec statement from Emacs, and the no-op colon statement turns this into a proper sequence statement for the shell. The colon in turn is quoted to make it appear as string literal to Emacs Lisp.

Eventually some file local variables tell Emacs to use Emacs Lisp Mode for the script, regardless of the shebang, and to enable lexical binding.

This particularly evil trick works reliably with any POSIX shell. Even better, we can now pass arbitrary arguments to the emacs executable, which allows us to get rid of a little nuisance of --script.

Inhibiting site-start

The —script option is just a shortcut for --batch -l, i.e. enter batch mode and load the given file. Batch Mode mainly means that Emacs will not create a frame, but instead exit after processing all command line arguments (which includes evaluating our script). Besides, —batch also disables the user initialization file. However, it still processes the global site initialization file:

--batch implies -q (do not load an initialization file), but site-start.el is loaded nonetheless. It also causes Emacs to exit after processing all the command options. In addition, it disables auto-saving except in buffers for which auto-saving is explicitly requested.

The global site initialization is often a kitchen sink which sets up globally installed packages and adds many seconds to Emacs’ startup time in the worst case. Besides, it’s not really a good idea to load arbitrary packages before our script even gets a chance to run.

We can opt out of the global site initialization by adding --quick to the emacs options of our script, which gives us a bare-bones Emacs without any initialization:

#!/bin/sh
":"; exec emacs --quick --script "$0" "$@" # -*- mode: emacs-lisp; lexical-binding: t; -*-
(message "Hello world")

If you need to, you can still load the global site initialization explicitly from site-run-file:

(load site-run-file 'no-error 'no-message)

Processing command line arguments

Emacs exposes the command line arguments in command-line-args-left alias argv (not to be confused with command-line-args which holds all Emacs options, including those that Emacs already interpreted, and is of little use in scripts):

#!/bin/sh
":"; exec emacs --quick --script "$0" "$@" # -*- mode: emacs-lisp; lexical-binding: t; -*-

(message "Hello: %S" argv)

$ ./hello.el 'John Doe'
Hello: ("John Doe")

Passing options doesn’t work that well, though:

$ ./hello.el --greeting 'Good morning %s!' 'John Doe'
Hello: ("--greeting" "Good morning %s!" "John Doe")
Unknown option `--greeting'

Emacs tries to interpret --greeting on its own, and rightfully complains that it has never heard of any such option. How do we keep Emacs away from our options?

The source code of startup.el, more precisely the function command-line-1, reveals the solution: Emacs processes all command line arguments immediately, in order of their appearance. After processing, each argument is removed from argv, hence the name command-line-args-left.

Since command-line-args-left aka argv is a global variable, we can just remove all remaining arguments from argv before our script exits:

#!/bin/sh
":"; exec emacs --quick --script "$0" "$@" # -*- mode: emacs-lisp; lexical-binding: t; -*-

(message "Hello: %S" argv)
(setq argv nil)

$ ./hello.el --greeting 'Good morning %s!' 'John Doe'
Hello: ("--greeting" "Good morning %s!" "John Doe")

We can also just force Emacs to exit early, which is good style anyway:

#!/bin/sh
":"; exec emacs --quick --script "$0" "$@" # -*- mode: emacs-lisp; lexical-binding: t; -*-

(message "Hello: %S" argv)
(kill-emacs 0)

However, as a reader of this blog pointed out that is still not enough. Emacs ignores our custom arguments now, but it will still try to process its own. This means that we can't have a --version argument in our script:

$ ./hello.el --version
GNU Emacs 25.0.50.1
Copyright (C) 2014 Free Software Foundation, Inc.
GNU Emacs comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of Emacs
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.

Emacs printed its own version and exited before our script even saw the --version argument. We need to use the standard double-dash -- argument to separate Emacs options from arguments, so that our script can unaffectedly process what Emacs now considers mere arguments (see https://stackoverflow.com/a/6807133/355252):

#!/bin/sh
":"; exec emacs --quick --script "$0" -- "$@" # -*- mode: emacs-lisp; lexical-binding: t; -*-

(message "Hello: %S" argv)
(kill-emacs 0)

Now we get the --version argument in our script, but also the separator, so we need to remember to drop the first argument:

$ ./hello.el --version
Hello: ("--" "--version")

Typically, you’ll process all arguments in a loop, poping each argument as it is processed. Initially, you need to pop the first argument to remove the argument separator:

#!/bin/sh
":"; exec emacs --quick --script "$0" -- "$@" # -*- mode: emacs-lisp; lexical-binding: t; -*-

(let ((greeting "Hello %s!")
      options-done
      names)
  (pop argv)  ; Remove the -- separator
  (while argv
    (let ((option (pop argv)))
      (cond
       (options-done (push option names))
       ;; Don't process options after "--"
       ((string= option "--") (setq options-done t))
       ((string= option "--greeting")
        (setq greeting (pop argv)))
       ;; --greeting=Foo
       ((string-match "\\`--greeting=\\(\\(?:.\\|\n\\)*\\)\\'" option)
        (setq greeting (match-string 1 option)))
       ((string-prefix-p "--" option)
        (message "Unknown option: %s" option)
        (kill-emacs 1))
       (t (push option names)))

      (unless (> (length greeting) 0)
        (message "Missing argument for --greeting!")
        (kill-emacs 1))))

  (unless names
    (message "Missing names!")
    (kill-emacs 1))

  (dolist (name (nreverse names))
    (message greeting name))

  (kill-emacs 0))

Emacs doesn't interfere with our options and arguments any more:

$ ./hello.el --greeting='Hello %s' 'John Doe' 'Donald Duck'
Hello John Doe
Hello Donald Duck

Standard output and input

In the earlier examples, we used message to print text in our script. There’s a little issue, though. We can't properly redirect the output:

$ ./hello.el 'John Doe' 'Donald Duck' > /dev/null
Hello John Doe!
Hello Donald Duck!

message writes to standard error, but a good script should use standard output. For this output stream, there's another, lesser known family of functions: print, prin1, princ and friends. These functions output “printed representations” of Lisp objects, with varying levels formatting and quoting.

For simple printing, princ is the right candidate, since it prints without any formatting and quoting. And naturally the unquoted “printed representation” of a string is… the string itself, so we can use this function to print a list of names to standard output:

#!/bin/sh
":"; exec emacs --quick --script "$0" "$@" # -*-emacs-lisp-*-

(while argv
  (princ (format "Hello %s!" (pop argv)))
  (terpri))

(kill-emacs 0)

Unlike message, princ doesn't take a format string, so we need to call format ourselves. terpri is a little utility that just prints a newline. The result is as expected, and we can also redirect the output now:

$ ./hello.el 'John Doe' 'Donald Duck'
Hello John Doe!
Hello Donald Duck!
$ ./hello.el 'John Doe' 'Donald Duck' >/dev/null

We have covered standard output now, but what about standard input? There are no obvious input functions in Emacs Lisp, but the minibuffer reads from standard input in batch mode (see https://stackoverflow.com/a/2906967/355252, I’d never have figured this out by myself):

#!/bin/sh
":"; exec emacs --quick --script "$0" "$@" # -*-emacs-lisp-*-

(let (name)
  (while (and (setq name (ignore-errors (read-from-minibuffer "")))
              (> (length name) 0))
    (princ (format "Hello %s!" name))
    (terpri)))

(kill-emacs 0)

We read lines from standard input with read-from-minibuffer, until an empty string is read, or an error occurs. EOF, e.g. C-d signals an error, so we can exit the input with C-d like in other programs.

$ ./hello.el
John Doe
Hello John Doe!
Donald Duck
Hello Donald Duck!

This has limitations, though. We can only read whole lines, and don't have direct access to the underlying TTY. The former doesn't really matter, but the latter limits the graphical capabilities of Emacs scripts and rules out all curses-like stuff or any text UI.

Watch out! This also affects password input in Emacs 24 and older: In these versions read-passwd reads from standard input in batch mode and thus exposes the password input on the terminal. Only as of Emacs 25 read-passwd is safe to use in batch mode.

Debugging

By default, Emacs’ error reporting is pretty terse, in interactive mode as well as in batch mode: It just prints the error message, without any backtraces. Consider this script, which has a little type error inside:

#!/bin/sh
":"; exec emacs --quick --script "$0" "$@" # -*-emacs-lisp-*-

(message "%S" (+ (car argv) (cadr argv)))
(setq argv nil)

The error message isn’t exactly helpful, though:

$ ./hello.el 10 20
Wrong type argument: number-or-marker-p, "10"

In interactive mode, we debug such errors by simply retrying the command after M-x toggle-debug-on-error. Emacs then enters the debugger and creates a backtrace if an error occurs.

In batch mode, we can’t “retry”, though, so we need to enable backtraces right away, by setting debug-on-error:

#!/bin/sh
":"; exec emacs --quick --script "$0" "$@" # -*-emacs-lisp-*-

(setq debug-on-error t)

(message "%S" (+ (car argv) (cadr argv)))

(setq argv nil)

Now we get stracktraces for any error:

$ ./hello.el 10 20
Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p "10")
  +("10" "20")
  (message "%S" (+ (car argv) (cadr argv)))
  eval-buffer(#<buffer  *load*> nil "/Users/swiesner/Developer/Sandbox/hello.el" nil t)  ; Reading at buffer position 140
  load-with-code-conversion("/Users/swiesner/Developer/Sandbox/hello.el" "/Users/swiesner/Developer/Sandbox/hello.el" nil t)
  load("/Users/swiesner/Developer/Sandbox/hello.el" nil t t)
  command-line-1(("-scriptload" "./hello.el" "10" "20"))
  command-line()
  normal-top-level()

Keep your hands clean

As much as we all love Emacs Lisp, it’s not a language that we should use for scripting or independent programs. Emacs Lisp is not an independent language and runtime environment. It’s tied to Emacs, and Emacs is an interactive text editor first and foremost.

I wrote this article partly to help you in the rare cases that you need to write non-interactive Emacs Lisp programs, eg, a runner for your test suite, but even more to show how brittle Emacs Lisp is when used outside Emacs.

Don’t get your hands dirty. Instead, just use any of the plenty of other languages that are available, eg, Python, Ruby or whatever. If you want a Lisp, use Common Lisp. Or try Rust for a change.

Emacs Lisp offers an autoloading mechanism to load libraries on demand. Typically this is used to make interactive commands available to the user without entirely loading the corresponding library upfront. This article explores how autoloads work and how Emacs Lisp packages use autoloads to improve load speed.

Autoloads

The autoload function creates autoloads, for instance for the function magit-status from Magit:

(autoload 'magit-status "magit" "Open a Magit status buffer […]" t nil)

Evaluating this expression tells Emacs to automatically load the library magit.el from load-path, when magit-status is called for the first time—either from Lisp, or interactively with M-x magit-status. You can manually add autoloads to your init.el yourself to autoload 3rd party libraries. In the old days before package.el, this was a common pattern.

Note the emphasis on evaluation. Merely writing this expression somewhere doesn't create any autoloads. Emacs must evaluate it, which comes down to Emacs loading a file with this expression.

It doesn't make any sense to put autoloads into the same file that has the actual definition of the autoloaded function. Emacs would load the rest of the file as well and thus make the definition available right away, leading the purpose of autoloads ad absurdum.

Autoloads should be in a separate file containing only autoloads and nothing else to make it load fast. Emacs calls such files “autoload files”.

Autoload cookies

Maintaining autoload files manually to keep them in sync with the actual definitions in the library file is tiresome and error-prone so Emacs allows to automate this process with update-file-autoloads and update-directory-autoloads.

update-file-autoloads inspects the source files for special comments called “autoload cookies”. These cookies let you declare autoloads right at the corresponding definition. An autoload cookie for magit-status looks like this:

;;;###autoload
(defun magit-status ()
  "Open a Magit status buffer […]"
  (interactive)
  ;; …
)

For each such cookie update-file-autoloads generates a corresponding autoload like the one shown above, and writes it to the autoload file. update-directory-autoloads performs this process for all files in a directory.

These commands only generate autoload files. You still need to load the generated files explicitly to make their autoloads available.

If an autoload cookie occurs on an expression with no special support for autoloading, update-file-autoloads copies the expression verbatim. This is used to register libraries in specific Emacs extension points, like auto-modes-alist.

Package autoloads

Emacs' package manager package.el goes a step further and automatically generates autoloads files during package installation—internally it simply calls update-directory-autoloads. This relieves package maintainers from the tedious work of manually updating autoload files and including them in their packages, and enables autoloads even for single-file packages.

Likewise package-initialize automatically loads autoloads files of all installed packages to make all autoloads available.

What to autoload?

The general rule is to autoload interactive “entry points” of a package. Examples of interactive entry points include:

• definitions of major and minor modes,
• interactive commands by which a user would start to use a specific package (e.g. gnus, erc, magit-status, etc.),
• and interactive commands which offer generic utilities, e.g. occur, find, ace-jump-mode, etc.

If your package just provides a library for use in Emacs Lisp code (e.g. like dash.el or s.el) you should not add any autoloads at all. Libraries are typically requiredd by dependent libraries so autoloads would be redundant.

If your package should automatically register itself in specific Emacs extension points you should add autoloads for these as well to make sure that they are evaluated during package initialization. A typical example is adding a mode to auto-mode-alist:

;;;###autoload
(add-to-list 'auto-mode-alist '("\\.pp\\'" . puppet-mode))

This puts puppet-mode into auto-mode-alist when Emacs starts, so that Puppet Mode is automatically used for all files ending in .pp.

Likewise, colour themes use autoload cookies to add themselves to the color theme search path:

;;;###autoload
(when (and (boundp 'custom-theme-load-path) load-file-name)
  (add-to-list 'custom-theme-load-path
               (file-name-as-directory (file-name-directory load-file-name))))

Emacs Lisp API for autoloads

Emacs Lisp has some functions to work with autoloads. In addition to autoload to create autoloads, there are autoloadp and autoload-do-load. The first lets you check whether an object is an autoload object, and the latter loads the underlying library of an autoload.

Both functions work on autoload objects, and not on symbols with attached autoloads. Hence, (autoloadp 'foo) checks whether the symbol foo is autoloaded, which it isn't. Symbols are not loaded at all, they are either directly created by the reader, or explicitly with intern.

To check whether foo refers to an autoloaded function you need to check the function definition of foo:

(autoloadp (function-definition 'foo))