doc: PAXlike docs

Summary
-------

Implement of a subset of PAX so that we can write PAX:DEFSECTIONlike
forms that supports the same restricted Markdown that we use for
docstrings and generate doc/manual/*.texinfo files from them.

Without PAX
-----------

There is no hard dependency on PAX. When the new SB-MANUAL contrib is
loaded, one can M-. around in documentation (sections are variables),
and function docstrings can now link to sections.

The generated Texinfo files are quite close to the originals, with
some loss of "semantic" markup: e.g. Markdown has `FOO` but no
@var{FOO} and @code{FOO}. We didn't derive much practical benefit from
that distinction.

With PAX
--------

(SB-MANUAL::SWITCH-TO-PAX) ensures that PAX is loaded and patches things
up, as if everything had been defined with PAX to begin with. Now, we
get PAX::@BROWSING-LIVE-DOCUMENTATION for low-latency, interactive
documentation work and PAX::@GENERATING-DOCUMENTATION for auto-linked
documentation.

Notable features:

- auto-generated links within the manual: if SB-EXT:EXIT is mentioned,
  then it's linked to its documentation.

- auto-generated links to the CLHS (these links are red in PDF)

- locatives (e.g. the "[function]" in "- [function] SB-EXT:EXIT" are
  also links and they go to the sources on GitHub (in live browsing,
  they tell Slime to open the definition)

Details
-------

- Factor out the Markdown-to-Texinfo code into a new SB-MANUAL
  contrib.

- Convert doc/manual/*.texinfo files to PAXlike DEFSECTION format and
  add each chapter as a contrib/sb-manual/doc/<chapter>.lisp file.

- Fix a *lot* of small issues during the conversion.

- Make doc/make-doc.sh regenerate the all .texinfo files except
  sbcl.texinfo and backmatter.texinfo.

- Fix docstrings of Lisp definitions used in the manual to conform to
  the supported Markdown syntax.

- Retain the Texinfo function, variable and type indicies but drop the
  concept index. There is no obvious way to support that in Markdown,
  and it was used rather sporadically rather incomplete.

- Even with  the new features, the  amount of Lisp code  didn't change
  significantly.

- See contrib/sb-manual/README for the Todo list

contrib/Makefile

@@ -3,7 +3,7 @@ vpath %.fasl ../obj/sbcl-home/contrib/
 contribs = sb-posix sb-bsd-sockets sb-introspect sb-cltl2 sb-aclrepl \
      sb-sprof sb-capstone sb-md5 sb-capstone sb-executable sb-gmp sb-mpfr \
      sb-queue sb-rotate-byte sb-rt sb-simple-streams sb-concurrency sb-cover \
-     sb-simd sb-grovel sb-perf asdf
+     sb-simd sb-grovel sb-perf sb-manual asdf
 
 active_contribs = $(filter-out $(SBCL_CONTRIB_BLOCKLIST),$(contribs))
 
@@ -20,6 +20,9 @@ sb-mpfr.fasl: sb-gmp.fasl
 sb-queue.fasl: sb-concurrency.fasl
 sb-simple-streams.fasl: sb-posix.fasl sb-bsd-sockets.fasl
 sb-grovel.fasl: asdf.fasl # for building the module, not for our build
+sb-manual.fasl: sb-introspect.fasl sb-bsd-sockets.fasl sb-sprof.fasl \
+	sb-aclrepl.fasl sb-concurrency.fasl sb-cover.fasl sb-grovel.fasl \
+	sb-posix.fasl sb-rotate-byte.fasl sb-cltl2.fasl
 
 # Notes:
 # 1. Invoking $(MAKE) for nested make eliminates a warning about jobserver mode.

contrib/README

@@ -3,7 +3,7 @@ core SBCL functionality, or else they'd be built into the main SBCL
 binary automatically. And they're not portable Common Lisp, or they'd
 be put elsewhere (see http://sbcl.sf.net/libs.php for pointers)
 
-There are two kinds of contrib module in this directory:
+There are two kinds of contrib modules in this directory:
 
   * Newer contrib modules conform to the contrib standard (see 
     STANDARDS) and are automatically built and installed along with

contrib/sb-manual/.gitignore

@@ -0,0 +1 @@
+output/

contrib/sb-manual/Makefile

@@ -0,0 +1,2 @@
+SYSTEM=sb-manual
+include ../asdf-module.mk

contrib/sb-manual/README.md

@@ -0,0 +1,225 @@
+Docstring Style Guide
+---------------------
+
+The Markdown-to-Texinfo converter is documented in `markdown.lisp`.
+Here, we provide a quick howto and style guide. If something is
+unclear (there should be lots), then you can test things with e.g.
+
+    (sb-manual::markdown-to-texinfo "PRINT")
+
+## Writing Names Inline
+
+- Upcase symbols naming definitions (or arguments of the function
+  being documented):
+
+        SB-EXT:EXIT
+        PRINT
+        X
+
+    Unqualified symbols must be accessible in the package where the
+    docstring is parsed. A good practice is to rely on Slime's TAB
+    completion on the name; then it's clear that the symbol exists.
+
+- Upcase and mark strings naming definitions as code:
+
+        `SB-EXT`
+
+    This is necessary because we don't want the creation of package
+    `ANSI` to automatically codify all occurrences of `ANSI` in the
+    documentation.
+
+- Upcase and mark non-existent symbols as code:
+
+        `WIN64::WHATEVER`
+
+    For when the code that defines the package or the symbol cannot be
+    loaded.
+
+- Mark C symbols as code:
+
+        `send`
+        `send(2)`
+
+    The latter is preferable to `send`(2), as it can be parsed by
+    `manual-entry` in Emacs.
+
+    If there is a Lisp symbol `SEND` accessible in the current package
+    and it names a definition, then you need to prevent it from being
+    autolinked (only with PAX, currently):
+
+        `\send(2)`
+
+    > _Note_: Here and in general, the backslashes need to be doubled
+    > when writing docstrings.
+
+- Mark C constants as code and escape downcasing:
+
+       `\\AF_LOCAL`
+
+    All capital code is downcased, which is great in the common case
+    but not here.
+
+## Writing Sexps Inline
+
+- One option is to mark the whole expression as code:
+
+        `(PRINT 'HELLO)`
+
+    Here, everything is rendered in monospace, downcased, no autolinks.
+
+- Alternatively, one could simply write
+
+        (PRINT '`HELLO`)
+
+    and rely on automatic codification for `PRINT`, codify `HELLO`
+    manually, and accept the fact that the parens will be in the
+    proportional font. In return, `PRINT` is autolinked.
+
+- Even more alternatively:
+
+        `(`PRINT` 'HELLO)`
+
+    Here, everything is monospace (except maybe the space), and
+    `PRINT` is autolinked.
+
+## Writing Signatures Inline
+
+Follow the somewhat usual `<TERMINAL-NAME>` convention:
+
+    One option is `(:USE <PACKAGE-NAME>*)`, where `<PACKAGE-NAME>`
+    is a package designator.
+
+If you wrote the mixed-case `(:USE <package-name>*)`, then the
+expression would be downcased, which is bad here.
+
+## Code Blocks
+
+In code blocks, always write as you would write in a source file,
+which means downcase the symbols. Prefer indented code blocks
+
+```
+    this is indented
+```
+
+to fenced code blocks:
+
+    ```
+    this is fenced
+    ```
+
+Use fenced code blocks only if there multiple successive code blocks
+that you definitely want to render as distinct "boxes".
+
+One gotcha to look out for is indenting code blocks is in list items:
+
+```
+- this is a list
+
+    A separate child paragraph indented to be "within" the list item.
+
+        (some code)
+```
+
+Note how the code block is indented 8 spaces from the `-` character.
+The required indentation remains the same if the child paragraph above
+is not present.
+
+## Notes
+
+- In many cases, just write a sentence:
+
+        Note that this may not be always so.
+
+- To add a note, use blockquotes:
+
+        > _Note_: This is not terribly important,
+        > but it can span multiple lines.
+
+    Blockquotes render indented in most formats, sometimes with a
+    vertical line to their left (e.g. Markdown on GitHub).
+
+- To add a warning, use blockquotes:
+
+        > __Warning__: Beware of the dog!
+
+Blockquotes without `_Note_` and `__Warning__` are fine for their
+other purposes: citing stuff, add copyright notices, etc. Markdown is
+not semantic. Use it freely and visually.
+
+Footnotes are not supported.
+
+## Typographic Miscellanea
+
+The Markdown-to-Texinfo converter does not convert e.g. `--` to
+en-dash. This is to prevent messing up the output when something like
+`git <option>* -- <path>` is written without proper markup. The issue
+is similar but maybe more pronounced with literal strings and curly
+quotes: you don't want to copy the rendered version of
+
+    (print "Hello, world")
+
+to the REPL, only to find the double quotes have been replaced by some
+fancy characters.
+
+## Inline Quotes and Emphasis
+
+Use strong emphasis (e.g. `__not__`) very sparingly. It usually gets
+rendered bold, which draws too much attention. However, emphasis
+(usually italic) is fine.
+
+You may use emphasis instead of single or double quotes:
+
+          CLHS `14.1` says that _depending on context, a group of
+          connected conses can be viewed in a variety of different
+          ways_, but that's trivial.
+
+You may also use double quotes, but the straight double quotes are
+kind of ugly in proportional fonts.
+
+## Sections
+
+### Titles
+
+Use title case:
+
+    Big and Bigger Words
+
+### Names
+
+Name the section `@BIG-AND-BIGGER-WORDS` or something shorter like
+`@BIG-WORDS`. In a docstring, you may read `"@BIG-WORDS are
+necessary"`, and `"Big and Bigger Words are necessary"` when it's
+rendered, so make sure that they are similar enough.
+
+Also, do not forget that section names live in a flat namespace: they
+are all exported from SB-MANUAL, so their names should be
+recognizable. Thus, it is better to name the section describing
+`SB-ACLREPL`'s usage `@SB-ACLREPL-USAGE` than `@USAGE`.
+
+## Docstring Formatting
+
+The Emacs style:
+
+    (defun foo (x)
+      "Return X.
+    It signals no errors."
+      x)
+
+You may also indent all but the first line as long as you do it
+consistently within the docstring. Maybe one day we could even agree
+upon a universally accepted style ... just joking.
+
+## Locale Stuff
+
+Currently the manual is a mix of American and British English.
+
+In the vast majority of the cases, the existing docstrings use `e.g.
+X` and `i.e. X` (the British version). For consistency, do not use
+other forms such as `eg. X` or `i.e., X`.
+
+## Paragraph Formatting
+
+Note that docstrings are also inconsistent about whether one or two
+spaces follow a full stop (controlled by `sentence-end-double-space`
+in Emacs) and their `fill-column`. It would be nice to have them all
+`fill-paragraph`ed with the same settings ...

contrib/sb-manual/TODO.md

@@ -0,0 +1,37 @@
+# Declarations are fake and render badly in both PAX and Texinfo
+
+Implement `DOCUMENTATION` and `(SETF DOCUMENTATION)` for type
+`CL:DECLARATION`.
+
+# How/when to load/include docs of contribs?
+
+Currently, `SB-MANUAL` loads *all* contribs to be able to query the
+definition docstrings. Each contrib directory has a `manual.lisp`
+file, which is part of the `SB-MANUAL` contrib (the files are
+symlinked).
+
+On the positive side, this does not load extra stuff until the user
+`REQUIRE`s `SB-MANUAL`. However, then it loads all contribs.
+
+A finer grained approach may be preferable. For example, we could
+make the manual.lisp file part of the contrib itself. Then people
+might complain about the overhead of loading/having the docstrings in
+the image.
+
+Alternatively, we could have `sb-bsd-sockets/manual.lisp` as a new
+`SB-BSD-SOCKETS-MANUAL` module. Eh.
+
+# SWITCH-TO-PAX automatically?
+
+# How to deal with repetitive package names?
+
+For example, `SB-ALIEN` is `:USE`d by `SB-MANUAL` so that the section
+docstrings need not fully qualify with `SB-ALIEN:` a thousand times.
+In the generated Texinfo, this can be a tad confusing. In output
+formats with links (e.g HTML from PAX), this is clearly preferable.
+
+Nicknames, maybe?
+
+# Implement PAX reflinks, e.g. `[function][type]`
+
+See `PAX::@REFLINKS`.

contrib/sb-manual/docstring.lisp

@@ -0,0 +1,122 @@
+(in-package :sb-manual)
+
+;;; Before processing Markdown, the docstring indentation is
+;;; normalized by stripping the longest run of leading spaces common
+;;; to all non-blank lines except the first. This is compatible with
+;;; PAX (see PAX::@MARKDOWN-SUPPORT).
+(defun reindent-docstring (docstring)
+  (let ((indentation (docstring-indentation docstring)))
+    (strip-docstring-indent docstring indentation t)))
+
+
+;;;; Utilities lifted from DRef and MGL-PAX
+
+;;; Return the minimum number of leading spaces in non-blank lines
+;;; after the first.
+(defun docstring-indentation (docstring &key (first-line-special-p t))
+  (let ((n-min-indentation nil))
+    (with-input-from-string (s docstring)
+      (loop for i upfrom 0
+            for line = (read-line s nil nil)
+            while line
+            do (when (and (or (not first-line-special-p) (plusp i))
+                          (not (blankp line)))
+                 (when (or (null n-min-indentation)
+                           (< (n-leading-spaces line) n-min-indentation))
+                   (setq n-min-indentation (n-leading-spaces line))))))
+    (or n-min-indentation 0)))
+
+(defun n-leading-spaces (line)
+  (let ((n 0))
+    (loop for i below (length line)
+          while (char= (aref line i) #\Space)
+          do (incf n))
+    n))
+
+(defun subseq* (seq start)
+  (subseq seq (min (length seq) start)))
+
+(defun strip-docstring-indent (docstring indentation first-line-special-p)
+  (with-output-to-string (out)
+    (with-input-from-string (s docstring)
+      (loop for i upfrom 0
+            do (multiple-value-bind (line missing-newline-p)
+                   (read-line s nil nil)
+                 (unless line
+                   (return))
+                 (write-string (if (and first-line-special-p
+                                        (zerop i))
+                                   line
+                                   (subseq* line indentation))
+                               out)
+                 (unless missing-newline-p
+                   (terpri out)))))))
+
+
+;;;; Determining the package for parsing docstrings
+;;;;
+;;;; The package for parsing is the package that was in effect when
+;;;; the docstring of the definition was read. This is basically the
+;;;; nearest IN-PACKAGE above the definition in the sources.
+;;;;
+;;;; With these semantics, when editing a docstring in Slime, if M-.
+;;;; works on an uppercase symbol name, then you can expect it to be
+;;;; codified by MARKDOWN-TO-TEXINFO. For symbols without a
+;;;; definition, you can use TAB-completion to check, but it's better
+;;;; to actually load PAX and check (see
+;;;; PAX::@BROWSING-LIVE-DOCUMENTATION).
+
+;;; To avoid conflicting with PAX's heuristics, DOCSTRING-PACKAGE
+;;; always returns a non-NIL package. We use a reimplementation of
+;;; DREF-EXT:DEFINITION-PROPERTY for DREF-EXT:DOCSTRING*, which we
+;;; feed to DOCSTRING-PACKAGE-OVERRIDES-TO-PAX in SWITCH-TO-PAX.
+
+;;; These map the SYMBOL-PACKAGE of a definition's XREF-NAME to the
+;;; docstring package.
+(defparameter *package-to-docstring-package*
+  '(("COMMON-LISP" "SB-IMPL")
+    ("SB-ACLREPL" "SB-ACLREPL")
+    ("SB-ALIEN" "SB-ALIEN")
+    ("SB-BSD-SOCKETS" "SB-BSD-SOCKETS")
+    ("SB-CONCURRENCY" "SB-CONCURRENCY")
+    ("SB-COVER" "SB-COVER")
+    ("SB-DEBUG" "SB-DEBUG")
+    ("SB-EXT" "SB-IMPL")
+    ("SB-GRAY" "SB-GRAY")
+    ("SB-GROVEL" "SB-GROVEL")
+    ("SB-INTROSPECT" "SB-INTROSPECT")
+    ("SB-MD5" "SB-MD5")
+    ("SB-POSIX" "SB-POSIX")
+    ("SB-SEQUENCE" "SB-IMPL")
+    ("SB-PROFILE" "SB-PROFILE")
+    ("SB-ROTATE-BYTE" "SB-ROTATE-BYTE")
+    ("SB-UNICODE" "SB-UNICODE")
+    ("SB-SPROF" "SB-SPROF")
+    ("SB-SYS" "SB-IMPL")
+    ("SB-THREAD" "SB-THREAD")))
+
+;;; The package-wide docstring packages are almost correct, but there
+;;; are a couple of definitions in random files.
+(defparameter *definition-to-docstring-package*
+  '(((with-compilation-unit macro) "SB-C")
+    ((sb-ext:restrict-compiler-policy function) "SB-C")
+    ((trace macro) "SB-DEBUG")))
+
+;;; For when this file is recompiled in interactive development after
+;;; SWITCH-TO-PAX
+(eval-when (:load-toplevel :execute)
+  (when *use-pax*
+    (convert-docstring-package-overrides-to-pax)))
+
+(defun docstring-package (xref)
+  (let* ((name (xref-name xref))
+         (key (list name (xref-locative xref))))
+    (or (find-package
+         (or (second (find key *definition-to-docstring-package*
+                           :key #'first :test #'equal))
+             (when (symbolp name)
+               (second (find (package-name (symbol-package name))
+                             *package-to-docstring-package*
+                             :key #'first :test #'equal)))))
+        (assert nil () "Cannot determine package of the docstring of ~S."
+                xref))))

contrib/sb-manual/make-pax-docs.lisp

@@ -0,0 +1,64 @@
+;;;; Generate the SBCL manual in various formats in doc/manual/ with PAX
+
+;;; This file is to be LOADed.
+
+(require :sb-manual)
+(require :mgl-pax/full)
+
+(in-package :sb-manual)
+
+(defvar *git-forge-uri*)
+(defvar *git-root*)
+(defvar *output-dir*)
+
+(defvar *directory* (truename (make-pathname :name nil :type nil
+                                             :defaults *load-truename*)))
+
+(defun sbcl-pages* (format)
+  (let ((source-uri-fn (when (and (not (eq format :plain))
+                                  *git-forge-uri*)
+                         (pax:make-git-source-uri-fn nil *git-forge-uri*
+                                                     :git-root *git-root*)))
+        (output-file (ecase format
+                       ((:plain) "sbcl-manual.txt")
+                       ((:markdown) "sbcl-manual.md")
+                       ((:pdf) "sbcl-manual.pdf")
+                       ((:html) "html/sbcl-manual.html"))))
+    `((:objects (, @sbcl-manual)
+       :output (,(merge-pathnames output-file *output-dir*)
+                :if-does-not-exist :create
+                :if-exists :supersede
+                ,@(when (eq format :pdf)
+                    '(:element-type (unsigned-byte 8)))
+                :ensure-directories-exist t)
+       ,@(when source-uri-fn
+           `(:source-uri-fn ,source-uri-fn))))))
+
+(defun make-pax-docs (&optional git-forge-uri)
+  (switch-to-pax)
+  (let ((*git-forge-uri* (or (and (plusp (length git-forge-uri))
+                                  git-forge-uri)
+                             "https://github.com/sbcl/sbcl"))
+        (*git-root* (truename (merge-pathnames "../../" *directory*)))
+        (*output-dir* (merge-pathnames "output/" *directory*))
+        (pax:*document-downcase-uppercase-code* t)
+        (pax:*document-url-versions* '(1))
+        (pax::*document-pandoc-pdf-options*
+          (remove "--verbose" pax::*document-pandoc-pdf-options*
+                  :test #'equal)))
+    (format t "Git root: ~A~%Git forge URI: ~A~%Output dir: ~A~%"
+            *git-root* *git-forge-uri* *output-dir*)
+    (format t "Generating manual in plain text format~%")
+    (pax:document @sbcl-manual :pages (sbcl-pages* :plain) :format :plain)
+    (format t "Generating manual in Markdown format~%")
+    (pax:document @sbcl-manual :pages (sbcl-pages* :markdown) :format :markdown)
+    (format t "Generating manual in PDF format~%")
+    (pax:document @sbcl-manual :pages (sbcl-pages* :pdf) :format :pdf)
+    (format t "Generating manual in HTML format~%")
+    (pax:update-asdf-system-html-docs @sbcl-manual "sb-manual"
+                                      :pages (sbcl-pages* :html)
+                                      :target-dir (merge-pathnames "html/"
+                                                                   *output-dir*)
+                                      :style :charter)))
+#+nil
+(make-pax-docs)

contrib/sb-manual/make-pax-docs.sh

@@ -0,0 +1,36 @@
+#!/bin/sh
+
+# This software is part of the SBCL system. See the README file for
+# more information.
+#
+# This software is in the public domain and is provided with
+# absolutely no warranty. See the COPYING and CREDITS files for
+# more information.
+
+BASE=`dirname "$0"`
+
+git_forge_uri="$1"
+
+if [ -z "$2" ] ; then
+    SBCL_TOP="$BASE/../.."
+    sbclsystem=$SBCL_TOP/src/runtime/sbcl
+    sbclcore=$SBCL_TOP/output/sbcl.core
+    if [ -f $sbclsystem ] && [ -f $sbclcore ]
+    then
+        SBCLRUNTIME="$sbclsystem --core $sbclcore"
+        SBCL_HOME=$SBCL_TOP/obj/sbcl-home/; export SBCL_HOME
+    else
+        SBCLRUNTIME="`command -v sbcl`"
+    fi
+    . $SBCL_TOP/output/build-config
+else
+    SBCLRUNTIME="$2"
+fi
+
+# We *don't* add --no-sysinit and --no-userinit because we rely on the
+# user to have set things up so that PAX can be loaded.
+${SBCLRUNTIME} \
+    --noinform --noprint --disable-debugger \
+    --load ${BASE}/make-pax-docs.lisp \
+    --eval "(sb-manual::make-pax-docs \"${git_forge_uri}\")" \
+    --quit

contrib/sb-manual/manual.lisp

@@ -0,0 +1,77 @@
+(in-package :sb-manual)
+
+(defsection @sb-manual (:title "sb-manual")
+  "The `SB-MANUAL` module has the sections of the SBCL user manual in
+  Lisp variables. The names of the variables (all start with the
+  character `@`) are exported from the `SB-MANUAL` package. Sections
+  are defined with the `DEFSECTION` macro:
+
+      (defsection @example (:title \"Example\")
+        \"This is an example, but see the real @SB-MANUAL.\"
+        (print function)
+        (@subexample section))
+
+  where `DEFSECTION` is a dummy implementation of
+  `PAX:DEFSECTION` (see <https://github.com/melisgl/mgl-pax/>).
+
+  In Slime, `\\\\M-.` on `\"@SB-MANUAL\"`, `\"print\"`, or on
+  `\"@subexample\"` will take you to the respective definition. This
+  makes it easy to navigate the documentation. Normal Lisp definition
+  docstrings and section docstrings reference sections following the
+  usual convention of uppercasing the name. Docstrings are in a subset
+  of Markdown and use very little markup in general, so they are easy
+  to read directly in the source.
+
+  The official manual in Info, HTML and PDF formats is generated via
+  Texinfo generated from these definitions.
+
+  When \\PAX is loaded, the dummy `DEFSECTION` definitions are made
+  real, so that \\PAX can work with them."
+  (@browsing-live-with-pax section)
+  (@fancy-documentation-with-pax section))
+
+(defsection @browsing-live-with-pax (:title "Browsing Live with PAX")
+  "With \\PAX, you can browse the manual live. The documentation of this
+  feature is available at
+  <https://melisgl.github.io/mgl-pax-world/pax-manual.html#MGL-PAX:@BROWSING-LIVE-DOCUMENTATION%20MGL-PAX:SECTION>.
+
+  If you are browsing this manual live right now, here is the
+  equivalent live link: `PAX::@BROWSING-LIVE-DOCUMENTATION`.
+
+  Notable features:
+
+  - Autolinks within the manual: if SB-EXT:EXIT is mentioned, then
+    it's linked to its documentation. You basically get links to where
+    `\\\\M-.` would go in the sources.
+
+  - Autolinks to the \\CLHS.
+
+  - View the documentation of any Lisp definition or section without
+    generating the entire manual.
+
+  - Locatives (e.g. the `\"[function]\"` in `\"- [function]
+    SB-EXT:EXIT\"`) are also links in live browsing: they tell Slime
+    to visit the definition.
+
+      For this to work, you need to allow Slime to evaluate Elisp sent
+      from SBCL:
+
+          (setq slime-enable-evaluate-in-emacs t)
+
+      and maybe your window manager focus stealing configuration needs
+      tweaking as well.
+
+  Live browsing can greatly reduce the latency of Edit-Compile-View
+  Loop, when working on documentation.")
+
+(defsection @fancy-documentation-with-pax
+    (:title "Fancy Documentation with PAX")
+  "\\PAX can generate dead documentation, too. In the SBCL sources,
+  `contrib/sb-manual/make-pax-docs.sh` generates the manual in plain
+  text, Markdown, PDF, and HTML formats. These differ from those
+  generated via Texinfo in that they are autolinked (like when
+  @BROWSING-LIVE-WITH-PAX).
+
+  Also, you can generate documentation yourself with e.g.
+
+      (pax:document sb-manual:@sbcl-manual :format :markdown)")

contrib/sb-manual/markdown.lisp

@@ -0,0 +1,812 @@
+;;;; A Markdown-to-Texinfo converter for the SBCL manual.
+
+;;;; This software is part of the SBCL software system. SBCL is in the
+;;;; public domain and is provided with absolutely no warranty. See
+;;;; the COPYING file for more information.
+;;;;
+;;;; Written by Rudi Schlatte <rudi@constantly.at>, mangled by
+;;;; Nikodemus Siivola. Brought closer to Markdown by Gabor Melis.
+
+(in-package :sb-manual)
+
+;;; MARKDOWN-TO-TEXINFO converts a strict subset of Markdown to
+;;; Texinfo. It also codifies (marks up as code) and downcases
+;;; uppercase symbols (those that actually exist in the image), and
+;;; autolinks references to sections, attempting to approximate PAX
+;;; semantics.
+;;;
+;;; Note that for writing docstrings, you need to know two more pieces:
+;;;
+;;; - See REINDENT-DOCSTRING for how the docstring relates to the
+;;;   Markdown string passed to MARKDOWN-TO-TEXINFO.
+;;;
+;;; - See DOCSTRING-PACKAGE to understand what *PACKAGE* is when
+;;;   MARKDOWN-TO-TEXINFO is called. This is package in effect when
+;;;   the docstring was READ. If it's wrong, you will see missed
+;;;   opportunities for codification and linking.
+;;;
+;;;
+;;; Markdown Support
+;;; ----------------
+;;;
+;;; The supported Markdown constructs are:
+;;;
+;;; - Emphasis: _italic_ -> @emph{italic}
+;;;
+;;; - Strong emphasis: __bold__ -> @strong{bold}
+;;;
+;;; - Inline code: `monospace` -> @code{monospace}
+;;;
+;;; - <http...> -> @url{http...}
+;;;
+;;; - Itemized lists (like this one). List items can span multiple
+;;;   lines.
+;;;
+;;;     - Nested lists are indented 4 spaces. A blank line is required
+;;;       before the first one.
+;;;
+;;; - Indented code blocks are indented with 4 extra spaces after a
+;;;   blank line:
+;;;
+;;;     Like this:
+;;;
+;;;         void main();
+;;;
+;;; - Fenced code blocks are indented at the normal level after a
+;;;   blank line:
+;;;
+;;;     ```
+;;;     void main();
+;;;     ```
+;;;
+;;;     Use fenced code blocks only when you have consecutive code
+;;;     blocks, which would be collapsed into a single code block when
+;;;     indented.
+;;;
+;;; - Blockquotes:
+;;;
+;;;     > _Note_: They can span multiple lines and anything can be
+;;;     > nested in them. Rendered indented, typically with a vertical
+;;;     > bar on the left.
+;;;
+;;; - Note that ``abc'' is *not* supported and *will* screw up the
+;;;   rendering of the Markdown. This is because it's impossible to
+;;;   reconcile it with backticks: consider the possible semantics of
+;;;
+;;;         ``x'' and ``y''
+;;;
+;;;
+;;; Codification and Downcasing
+;;; ---------------------------
+;;
+;;; Summary: Some text is automatically codified (e.g. FOO -> `FOO`)
+;;; and most code is downcased.
+;;;
+;;; We approximate the semantics of PAX::@CODIFICATION with the
+;;; settings PAX:*DOCUMENT-UPPERCASE-IS-CODE* and
+;;; PAX:*DOCUMENT-DOWNCASE-UPPERCASE-CODE* both true.
+;;;
+;;; - Fully-qualified all-uppercase string representatation of symbols
+;;;   are codified (SB-EXT:CAS, :XYZ).
+;;;
+;;; - All-uppercase SYMBOL-NAMEs accessible in *PACKAGE*.
+;;;
+;;; - When at least 3 uppercase characters are followed by a lowercase
+;;;   character (e.g. SETFable), then the uppercase prefix is codified
+;;;   with the previous rules.
+;;;
+;;; When there is no corresponding symbol, the Markdown backtick
+;;; syntax (`PRINT`) can be used to codify.
+;;;
+;;; When there are no lowercase nor #\" characters in inline code (as
+;;; opposed to code blocks), be it auto-codified or explicitly
+;;; backticked, it's downcased.
+;;;
+;;; When there is a corresponding symbol, but codification or
+;;; downcasing should not happen, use backslash escapes.
+;;;
+;;; Escaping (following PAX::@OVERVIEW-OF-ESCAPING):
+;;;
+;;;   PRINT     -> @code{print}    (Should be autolinked, unimplemented)
+;;;   \PRINT    -> @code{print}    (Prevent autolinking)
+;;;   \\PRINT   -> PRINT           (Prevent autolinking and codification)
+;;;   `PRINT`   -> @code{print}    (Should be autolinked, unimplemented)
+;;;   `\PRINT`  -> @code{print}    (Prevent autolinking)
+;;;   `\\PRINT` -> @code{PRINT}    (Prevent autolinking and downcasing)
+;;;
+;;; Note that in docstrings, the backslashes need to be doubled.
+;;;
+;;;
+;;; Linking
+;;; -------
+;;;
+;;; - Section references: @SECTION-NAME -> @ref{section name}
+;;;
+;;; FIXME:
+;;;
+;;; - Maybe implement glossary-terms (for books, "safe type", etc).
+(defun markdown-to-texinfo (string &optional lambda-list)
+  (let ((*texinfo-local-variables* (flatten lambda-list))
+        (lines (string-lines string))
+        (line-number 0)
+        (current-paragraph nil))
+    (declare (special *texinfo-local-variables*))
+    (flet ((flush-paragraph ()
+             (when current-paragraph
+               (write-string (process-inline-markdown
+                              (format nil "~{~A~^~%~}"
+                                      (nreverse current-paragraph))))
+               (terpri)
+               (setf current-paragraph nil))))
+      (loop while (< line-number (length lines))
+            for line = (svref lines line-number)
+            do (multiple-value-bind (count collected)
+                   (parse-markdown-blocks lines line-number 0)
+                 (cond
+                   (count
+                    (flush-paragraph)
+                    (dolist (c collected)
+                      (write-line c))
+                    (incf line-number count))
+                   ((blankp line)
+                    (flush-paragraph)
+                    (write-line line)
+                    (incf line-number))
+                   (t
+                    (push line current-paragraph)
+                    (incf line-number)))))
+      (flush-paragraph))))
+
+
+;;;; Utilities
+
+(defun flatten (list)
+  (cond ((null list)
+         nil)
+        ((consp (car list))
+         (nconc (flatten (car list)) (flatten (cdr list))))
+        ((null (cdr list))
+         (cons (car list) nil))
+        (t
+         (cons (car list) (flatten (cdr list))))))
+
+(defun whitespacep (char)
+  (find char #(#\tab #\space #\page #\newline #\return)))
+
+;;; Split STRING into a vector of lines.
+(defun string-lines (string)
+  (coerce (with-input-from-string (s string)
+            (loop for line = (read-line s nil nil)
+               while line collect line))
+          'vector))
+
+;;; Position of the first non-SPACE character in LINE.
+(defun indentation (line)
+  (position-if-not (lambda (c) (char= c #\Space)) line))
+
+(defun blankp (line)
+  (null (indentation line)))
+
+(defun flatten-to-string (list)
+  (format nil "~{~A~^-~}" (flatten list)))
+
+(defun internedp (symbol-name package)
+  (nth-value 1 (find-symbol symbol-name package)))
+
+(defun external-symbol-p (symbol &optional (package (symbol-package symbol)))
+  (and package
+       (multiple-value-bind (symbol* status)
+           (find-symbol (symbol-name symbol) package)
+         (and (eq status :external)
+              (eq symbol symbol*)))))
+
+
+;;;; Texinfo escaping
+
+(defparameter *texinfo-special-chars* "@{}")
+
+(defun escape-texinfo (string)
+  (with-output-to-string (s)
+    (loop for char across string
+          do (when (find char *texinfo-special-chars*)
+               (write-char #\@ s))
+             (write-char char s))))
+
+(defun unescape-texinfo (string)
+  (with-output-to-string (s)
+    (let ((prev-escape-p nil))
+      (loop for char across string
+            do (cond (prev-escape-p
+                      (write-char char s)
+                      (setq prev-escape-p nil))
+                     ((char= char #\@)
+                      (setq prev-escape-p t))
+                     (t
+                      (write-char char s)))))))
+
+(progn
+  (assert (equal (escape-texinfo "@code{x}") "@@code@{x@}"))
+  (assert (equal (unescape-texinfo "@@code@{x@}") "@code{x}")))
+
+
+;;;; Codification (following PAX::@CODIFICATION)
+
+(defvar *lower-case-chars* "abcdefghijklmnopqrstuvwxyz")
+
+(defun codifiable-bounds (word)
+  (when (codifiable-word-p word)
+    ;; PAX::@NAMES-IN-RAW-NAMES is involved. We only try two simple
+    ;; cases to get a PAX::@NAME.
+    (flet ((try-name (start end)
+             (let ((name (subseq word start end)))
+               (multiple-value-bind (symbol foundp)
+                   (read-symbol-without-interning name)
+                 (when (and foundp (interesting-name-p word symbol))
+                   (return-from codifiable-bounds (values start end)))))))
+      ;; 1. Trim the lower-case characters
+      (let* ((name (string-left-trim *lower-case-chars* word))
+             (name-start (- (length word) (length name)))
+             (name (string-right-trim *lower-case-chars* name))
+             (name-end (+ name-start (length name))))
+        (try-name name-start name-end))
+      ;; 2. Find the upper-case core
+      (multiple-value-bind (name-start name-end) (uppercase-core-bounds word)
+        (when name-start
+          (try-name name-start name-end))))))
+
+(defun codifiable-word-p (string)
+  (uppercase-core-bounds string))
+
+(defun read-symbol-without-interning (string)
+  (if (and (plusp (length string))
+           (char= (aref string 0) #\:))
+      (find-symbol (subseq string 1) :keyword)
+      (let ((pos (position #\: string)))
+        (if pos
+            (let* ((package-name (subseq string 0 pos))
+                   (symbol-name (subseq string (1+ pos)))
+                   (double-colon-p
+                     (and (plusp (length symbol-name))
+                          (char= (aref symbol-name 0) #\:))))
+              (when double-colon-p
+                (setq symbol-name (subseq symbol-name 1)))
+              (if package-name
+                  (when (find-package package-name)
+                    (multiple-value-bind (symbol status)
+                        (find-symbol symbol-name package-name)
+                      (when (or double-colon-p
+                                (eq status :external))
+                        (values symbol status))))
+                  (find-symbol symbol-name *package*)))
+            (find-symbol string *package*)))))
+
+;;; Approximating PAX::@INTERESTING. This is only called when we
+;;; already found the interned SYMBOL.
+(defun interesting-name-p (word symbol)
+  (or (<= 3 (length word))
+      (external-symbol-p symbol)
+      (has-local-reference-p symbol)))
+
+(defun uppercase-core-bounds (string)
+  (let* ((first-uppercase-pos (position-if #'upper-case-p string))
+         (last-uppercase-pos (position-if #'upper-case-p string
+                                          :from-end t)))
+    (when (and first-uppercase-pos
+               (if (= last-uppercase-pos first-uppercase-pos)
+                   (notany #'lower-case-p string)
+                   (not (find-if #'lower-case-p string
+                                 :start (1+ first-uppercase-pos)
+                                 :end last-uppercase-pos))))
+      (values first-uppercase-pos (1+ last-uppercase-pos)))))
+
+(defvar *texinfo-local-variables* ())
+
+(defun has-local-reference-p (name)
+  (find name *texinfo-local-variables*))
+
+#+nil
+(progn
+  (assert (equal (multiple-value-list (codifiable-bounds "PRINT"))
+                 '(0 5)))
+  (assert (equal (multiple-value-list (codifiable-bounds "T"))
+                 '(0 1)))
+  (if (internedp "A" *package*)
+      (assert (equal (multiple-value-list (codifiable-bounds "A"))
+                     '(0 1)))
+      (assert (null (codifiable-bounds "A"))))
+  (assert (equal (multiple-value-list (codifiable-bounds "*FEATURES*"))
+                 '(0 10))))
+
+;;; We parse words (e.g. nonREADable) and find symbols in them.
+(defparameter *word-characters*
+  (format nil "abcdefghijklmnopqrstuvwxyz~
+               ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789~
+               *@:-+&=<>#'"))
+
+(defparameter *word-delimiters* " ',.!?;()[]{}")
+
+;;; Return a list of index pairs of symbol-like parts of LINE.
+(defun locate-symbols (line)
+  (let (result)
+    (flet ((grab (word-start word-end)
+             (let ((word (subseq line word-start word-end)))
+               (multiple-value-bind (name-start name-end)
+                   (codifiable-bounds word)
+                 (when name-start
+                   (push (list (+ word-start name-start)
+                               (+ word-start name-end))
+                         result)))))
+           (got-symbol-p (start)
+             (let ((end (when (< start (length line))
+                          (position-if (lambda (c)
+                                         (or (whitespacep c)
+                                             (find c *word-delimiters*)))
+                                       line :start start))))
+               (when end
+                 (every (lambda (char) (find char *word-characters*))
+                        (subseq line start end))))))
+      (do ((begin nil)
+           (maybe-begin t)
+           (i 0 (1+ i)))
+          ((>= i (length line))
+           ;; symbol at end of line
+           (when begin
+             (grab begin i))
+           (nreverse result))
+        (cond
+          ((and begin
+                (or (whitespacep (char line i))
+                    (find (char line i) *word-delimiters*)
+                    ;; For e.g. "T:"
+                    (and (char= (char line i) #\:)
+                         (or (= (1+ i) (length line))
+                             (whitespacep (char line (1+ i)))))))
+           ;; symbol end
+           (grab begin i)
+           (setf begin nil
+                 maybe-begin t))
+          ((and begin (not (find (char line i) *word-characters*)))
+           ;; Not a symbol: abort
+           (setf begin nil))
+          ((and maybe-begin (not begin)
+                (find (char line i) *word-characters*))
+           ;; potential symbol begin at this position
+           (setf begin i
+                 maybe-begin nil))
+          ((or (whitespacep (char line i))
+               (find (char line i) *word-delimiters*))
+           ;; potential symbol begin after this position
+           (setf maybe-begin t))
+          ((and (eql #\( (char line i)) (got-symbol-p (1+ i)))
+           ;; a type designator, or a function call as part of the text?
+           (multiple-value-bind (exp end)
+               (let ((*package* (find-package :cl-user)))
+                 (ignore-errors (read-from-string line nil nil :start i)))
+             (when exp
+               (grab i end)
+               (setf begin nil
+                     maybe-begin nil
+                     i (1- end)))))
+          (t
+           ;; Not reading a symbol, not at potential start of symbol
+           (setf maybe-begin nil)))))))
+
+(progn
+  (defsection @test-section ())
+  (defsection @test5 ())
+  (assert (equal (locate-symbols "PRINT") '((0 5))))
+  (assert (equal (locate-symbols "CL:PRINT") '((0 8))))
+  (assert (equal (locate-symbols "*FEATURES*") '((0 10))))
+  (assert (equal (locate-symbols "SETFable") '((0 4))))
+  (assert (equal (locate-symbols "SETF-able") '((0 4))))
+  (assert (equal (locate-symbols "nonREADable") '((3 7))))
+  (assert (equal (locate-symbols "NOSUCHSYMBOL-able") '()))
+  (assert (equal (locate-symbols "ASDF-like") '()))
+  (assert (equal (locate-symbols "@TEST-SECTION") '((0 13))))
+  (assert (equal (locate-symbols "SB-MANUAL:@TEST-SECTION") '((0 23))))
+  (assert (equal (locate-symbols "@NOSUCHSECTION") '()))
+  (assert (equal (locate-symbols "@TEST5") '((0 6))))
+  (assert (equal (locate-symbols ":IR1-CONVERT") '((0 12)))))
+
+
+;;;; Processing Markdown inline elements
+
+;;; Format symbols either as Texinfo code or as variables if the
+;;; symbol in question is contained in symbols *TEXINFO-LOCAL-VARIABLES*.
+(defun codify-and-link (line)
+  (with-output-to-string (result)
+    (let ((last 0))
+      (dolist (symbol/index (locate-symbols line))
+        ;; Flush unwritten text since the end of the previous symbol.
+        (write-string (escape-texinfo (subseq line last (first symbol/index)))
+                      result)
+        (let* ((symbol-name (apply #'subseq line symbol/index))
+               (symbol (read-from-string symbol-name)))
+          (if (section-name-p symbol)
+              (format result "@ref{~A}"
+                      (texinfo-node-id (symbol-value symbol)))
+              ;; We could use for @var{} if (HAS-LOCAL-REFERENCE-P SYMBOL).
+              (format result "@code{~A}" (escape-texinfo
+                                          (maybe-downcase symbol-name)))))
+        (setf last (second symbol/index)))
+      (write-string (escape-texinfo (subseq line last)) result))))
+
+(defvar *downcase-uppercase-code* t)
+
+(defun maybe-downcase (string)
+  (if *downcase-uppercase-code*
+      (string-downcase string)
+      string))
+
+(defun texinfo-node-id (section)
+  (let ((name (symbol-name (section-name section))))
+    (assert (char= (char name 0) #\@))
+    (let ((name (subseq name 1)))
+      (assert (null (find-if (lambda (char)
+                               (find char *texinfo-special-chars*))
+                             name))
+              () "Section name ~S contains special texinfo characters." name)
+      (substitute #\Space #\- (string-downcase name)))))
+
+(defun section-name-p (symbol)
+  (when (boundp symbol)
+    (let ((value (symbol-value symbol)))
+      (and (listp value)
+           (eq (first value) 'defsection)))))
+
+(when (and (not *use-pax*)
+           *downcase-uppercase-code*)
+  (assert (equal (codify-and-link "@TEST-SECTION") "@ref{test section}"))
+  (assert (equal (codify-and-link "@NOSUCHSECTION") "@@NOSUCHSECTION"))
+  (assert (equal (codify-and-link ":START") "@code{:start}"))
+  (assert (equal (codify-and-link "[:START") "[@code{:start}"))
+  (assert (equal (codify-and-link "{:START") "@{@code{:start}")))
+
+;;; Translate backticks, emphasis and codification escapes, while
+;;; delegating normal text to CODIFY-AND-LINK.
+(defun process-inline-markdown (string)
+  (let ((len (length string))
+        (i 0)
+        (codifiable-buffer (make-string-output-stream))
+        (out (make-string-output-stream)))
+    (labels ((out (string)
+               (write-string string out))
+             (out-escaped (string)
+               (out (escape-texinfo (string string))))
+             (buffer-codifiable-char (char)
+               (write-char char codifiable-buffer))
+             (flush-codifiable-buffer ()
+               (let ((codifiable (get-output-stream-string codifiable-buffer)))
+                 (when (plusp (length codifiable))
+                   (out (codify-and-link codifiable))))))
+      (loop while (< i len)
+            for char = (char string i)
+            do (cond
+                 ;; Escapes: \FOO
+                 ((char= char #\\)
+                  (flush-codifiable-buffer)
+                  (incf i)
+                  (when (< i len)
+                    (out-escaped (char string i))
+                    (incf i)
+                    ;; Protect the rest of the contiguous word from
+                    ;; CODIFY-AND-LINK up until
+                    (loop
+                      while (and (< i len)
+                                 (not (or (whitespacep (char string i))
+                                          (find (char string i)
+                                                " ,.!?;()'[]{}\""))))
+                      do (out-escaped (char string i))
+                         (incf i))
+                    (decf i)))
+                 ;; Backticks: `CODE` with PAX downcasing and escape rules
+                 ((char= char #\`)
+                  (flush-codifiable-buffer)
+                  (incf i)
+                  (let ((code-buffer (make-string-output-stream)))
+                    (loop while (and (< i len) (char/= (char string i) #\`))
+                          do (write-char (char string i) code-buffer)
+                             (incf i))
+                    (let* ((code-str (get-output-stream-string code-buffer))
+                           (slash-count (loop for c across code-str
+                                              while (char= c #\\)
+                                              count t))
+                           ;; Consume up to 2 leading backslashes as PAX escapes
+                           (actual-code (subseq code-str (min slash-count 2))))
+                      (out "@code{")
+                      (if (< slash-count 2)
+                          ;; 0 or 1 backslash: Downcase if there are
+                          ;; no lowercase letters (1 backslash turns
+                          ;; off autolinking, which is naturally
+                          ;; handled by bypassing CODIFY-AND-LINK).
+                          (if (and (not (find-if #'lower-case-p actual-code))
+                                   (not (find #\" actual-code)))
+                              (out-escaped (maybe-downcase actual-code))
+                              (out-escaped actual-code))
+                          ;; 2 backslashes turn off autolinking AND downcasing.
+                          (out-escaped actual-code))
+                      (out "}"))))
+                 ;; Strong emphasis: __foo__ -> @strong{foo}
+                 ((and (char= char #\_)
+                       (< (1+ i) len)
+                       (char= (char string (1+ i)) #\_))
+                  (let ((close-pos (search "__" string :start2 (+ i 2))))
+                    (if close-pos
+                        (progn
+                          (flush-codifiable-buffer)
+                          (format out "@strong{~A}"
+                                  (process-inline-markdown
+                                   (subseq string (+ i 2) close-pos)))
+                          (setf i (+ close-pos 1)))
+                        (buffer-codifiable-char char))))
+                 ;; Emphasis: _foo_ -> @emph{foo}
+                 ((char= char #\_)
+                  (let ((close-pos nil)
+                        (j (1+ i)))
+                    (loop while (< j len)
+                          do (if (char= (char string j) #\_)
+                                 (if (and (< (1+ j) len)
+                                          (char= (char string (1+ j)) #\_))
+                                     ;; Skip double underscores so
+                                     ;; they don't falsely close a
+                                     ;; single underscore.
+                                     (incf j 2)
+                                     (progn
+                                       (setf close-pos j)
+                                       (return)))
+                                 (incf j)))
+                    (if close-pos
+                        (progn
+                          (flush-codifiable-buffer)
+                          (format out "@emph{~A}"
+                                  (process-inline-markdown
+                                   (subseq string (1+ i) close-pos)))
+                          (setf i close-pos))
+                        (buffer-codifiable-char char))))
+                 ;; Markdown autolinks: <http...> -> @url{http...}
+                 ((and (char= char #\<)
+                       (<= (+ i 5) len)
+                       (string-equal string "http" :start1 (1+ i)
+                                     :end1 (+ i 5)))
+                  (let ((close-pos (position #\> string :start (1+ i))))
+                    (if close-pos
+                        (progn
+                          (flush-codifiable-buffer)
+                          (out "@url{")
+                          (out-escaped (subseq string (1+ i) close-pos))
+                          (out "}")
+                          (setf i close-pos))
+                        (buffer-codifiable-char char))))
+                 (t
+                  (buffer-codifiable-char char)))
+               (incf i))
+      (flush-codifiable-buffer)
+      (get-output-stream-string out))))
+
+(when *downcase-uppercase-code*
+  (assert (equal (process-inline-markdown "`abc`") "@code{abc}"))
+  (assert (equal (process-inline-markdown "_abc_") "@emph{abc}"))
+  (assert (equal (process-inline-markdown "__abc__") "@strong{abc}"))
+  (assert (equal (process-inline-markdown "_PRINT_") "@emph{@code{print}}"))
+  (assert (equal (process-inline-markdown "<httpabc>") "@url{httpabc}"))
+  (assert (equal (process-inline-markdown "`N`") "@code{n}"))
+  (assert (equal (process-inline-markdown "`N`th") "@code{n}th")))
+
+
+;;;; Processing Markdown block elements
+
+;;; Collect lines enclosed in Markdown ``` fences. Returns the number
+;;; of lines consumed and a list of lines.
+(defun collect-fenced-code (lines starting-line base-indent)
+  (let* ((first-line (svref lines starting-line))
+         (trimmed (string-left-trim " " first-line)))
+    (when (and (>= (length trimmed) 3)
+               (string= (subseq trimmed 0 3) "```"))
+      (let ((lang (string-trim " " (subseq trimmed 3)))
+            (consumed 1)
+            (result nil))
+        (loop for index from (1+ starting-line) below (length lines)
+              for line = (svref lines index)
+              for line-trimmed = (string-left-trim " " line)
+              do (incf consumed)
+              if (and (>= (length line-trimmed) 3)
+                      (string= (subseq line-trimmed 0 3) "```"))
+                do (loop-finish)        ; Closing fence found
+              else
+                ;; Strip up to the base indentation of the environment
+                do (push (if (and (indentation line)
+                                  (>= (indentation line) base-indent))
+                             (subseq line base-indent)
+                             line)
+                         result))
+        (let ((env (if (string-equal lang "lisp") "lisp" "example")))
+          (values consumed
+                  `(,(format nil "@~A" env)
+                    ,@(mapcar #'escape-texinfo (nreverse result))
+                    ,(format nil "@end ~A" env))))))))
+
+;;; Collect lines that start with a Markdown blockquote marker (">").
+;;; A blockquote must be preceded by a blank line or be the first
+;;; line. The marker can be indented up to 3 characters on top of
+;;; BASE-INDENT. By leveraging string streams and passing the stripped
+;;; content recursively back to MARKDOWN-TO-TEXINFO, we maintain full
+;;; support for nested blocks, lists, and inline text wrapping.
+(defun collect-blockquote (lines starting-line base-indent)
+  (unless (and (> starting-line 0)
+               (not (blankp (svref lines (1- starting-line)))))
+    (let* ((first-line (svref lines starting-line))
+           (first-indent (indentation first-line)))
+      (when (and first-indent
+                 (<= first-indent (+ base-indent 3))
+                 (< first-indent (length first-line))
+                 (char= (char first-line first-indent) #\>))
+        (let ((n-lines 0)
+              (stripped-lines nil))
+          (loop for index from starting-line below (length lines)
+                for line = (svref lines index)
+                for indent = (indentation line)
+                do (cond
+                     ((and indent
+                           (<= indent (+ base-indent 3))
+                           (< indent (length line))
+                           (char= (char line indent) #\>))
+                      (let* ((start (1+ indent))
+                             (content-start
+                               (if (and (< start (length line))
+                                        (char= (char line start) #\Space))
+                                   (1+ start)
+                                   start)))
+                        (push (subseq line content-start) stripped-lines)
+                        (incf n-lines)))
+                     (t
+                      (loop-finish))))
+          ;; Trim trailing empty lines
+          (loop while (and stripped-lines (string= (car stripped-lines) ""))
+                do (pop stripped-lines) (decf n-lines))
+          (when stripped-lines
+            (let ((inner-texinfo
+                    ;; Process the stripped sub-document cleanly using
+                    ;; the main loop to handle paragraphs, nesting,
+                    ;; and formatting automatically.
+                    (with-output-to-string (*standard-output*)
+                      (markdown-to-texinfo
+                       (format nil "~{~A~^~%~}" (nreverse stripped-lines))
+                       *texinfo-local-variables*))))
+              (values n-lines
+                      `("@quotation"
+                        ,@(coerce (string-lines inner-texinfo) 'list)
+                        "@end quotation")))))))))
+
+;;; Collect lines indented with an extra 4 character on top of
+;;; BASE-INDENT. An indented code block must be preceeded by a blank
+;;; line or be the first line.
+(defun collect-indented-code (lines starting-line base-indent)
+  (unless (and (> starting-line 0)
+               (not (blankp (svref lines (1- starting-line)))))
+    (let ((indent (indentation (svref lines starting-line))))
+      (when (and indent (>= indent (+ base-indent 4)))
+        (let ((n-lines 0)
+              (result nil))
+          (loop for index from starting-line below (length lines)
+                for line = (svref lines index)
+                for line-indent = (indentation line)
+                do (cond
+                     ((blankp line)
+                      ;; Blank lines are allowed inside indented code blocks.
+                      (push "" result)
+                      (incf n-lines))
+                     ((>= line-indent (+ base-indent 4))
+                      (push (subseq line (+ base-indent 4)) result)
+                      (incf n-lines))
+                     (t
+                      ;; Indentation dropped, so the code block ends.
+                      (loop-finish))))
+          ;; Trim trailing empty lines.
+          (loop while (and result (string= (car result) ""))
+                do (pop result) (decf n-lines))
+          (if result
+              (values n-lines `("@example"
+                                ,@(mapcar #'escape-texinfo (nreverse result))
+                                "@end example"))
+              nil))))))
+
+;;; Return the indent if the line starts with a Markdown list marker
+;;; (#\- or \*) followed by a space.
+(defun maybe-itemize-offset (line)
+  (let ((indent (indentation line)))
+    (when indent
+      (let ((trimmed (string-left-trim " " line)))
+        (when (and (>= (length trimmed) 2)
+                   (member (char trimmed 0) '(#\- #\*))
+                   (char= (char trimmed 1) #\Space))
+          indent)))))
+
+;;; Collect a bulleted list.
+(defun collect-markdown-itemize (lines starting-line base-indent)
+  (let ((this-offset (maybe-itemize-offset (svref lines starting-line))))
+    (when (and this-offset (= this-offset base-indent))
+      (let ((result nil)
+            (lines-consumed 0)
+            (child-base (+ base-indent 4))
+            (current-paragraph nil)
+            (item-pending-p nil))
+        (flet ((flush-paragraph ()
+                 (if current-paragraph
+                     (let ((processed (process-inline-markdown
+                                       (format nil "~{~A~^~%~}"
+                                               (nreverse current-paragraph)))))
+                       (if item-pending-p
+                           (push (format nil "@item ~A" processed) result)
+                           (push processed result))
+                       (setf current-paragraph nil)
+                       (setf item-pending-p nil))
+                     (when item-pending-p
+                       (push "@item" result)
+                       (setf item-pending-p nil)))))
+          (loop for line-number = starting-line then (+ starting-line
+                                                        lines-consumed)
+                while (< line-number (length lines))
+                for line = (svref lines line-number)
+                for indent = (indentation line)
+                for offset = (maybe-itemize-offset line)
+                do (cond
+                     ((blankp line)
+                      ;; Blank lines inside lists are buffered
+                      (flush-paragraph)
+                      (push "" result)
+                      (incf lines-consumed))
+                     ;; New Item in the same list
+                     ((and offset (= offset base-indent))
+                      (flush-paragraph)
+                      (setf item-pending-p t)
+                      (let ((item-text (subseq line (+ offset 2))))
+                        (unless (blankp item-text)
+                          (push item-text current-paragraph)))
+                      (incf lines-consumed))
+                     ;; Indented block/text inside the list item (>= 4 spaces)
+                     ((and indent (>= indent child-base))
+                      (flush-paragraph)
+                      (multiple-value-bind (sub-consumed sub-result)
+                          (parse-markdown-blocks lines line-number child-base)
+                        (if sub-consumed
+                            (progn
+                              (setf result (append (reverse sub-result) result))
+                              (incf lines-consumed sub-consumed))
+                            ;; Fallback: normal text continuing the item body
+                            (progn
+                              (push (subseq line child-base) current-paragraph)
+                              (incf lines-consumed)))))
+                     ;; Normal text continuing the item body
+                     ((and indent (> indent base-indent))
+                      (push line current-paragraph)
+                      (incf lines-consumed))
+                     ;; If we get here, the line is NOT a new bullet,
+                     ;; and it less than 4 spaces of relative
+                     ;; indentation, so the list is over.
+                     (t
+                      (loop-finish))))
+          (flush-paragraph)
+          ;; Trim trailing empty lines, so they return to the outer scope.
+          (loop while (and result (string= (car result) ""))
+                do (pop result) (decf lines-consumed))
+          (values lines-consumed `("@itemize" ,@(reverse result)
+                                   "@end itemize")))))))
+
+;;; Parse the line at INDEX in LINES as a Markdown block. Return the
+;;; number of lines consumed and the parse.
+(defun parse-markdown-blocks (lines index base-indent)
+  (let ((line (svref lines index)))
+    (multiple-value-bind (n-lines-consumed result)
+        (collect-fenced-code lines index base-indent)
+      (when n-lines-consumed
+        (return-from parse-markdown-blocks (values n-lines-consumed result))))
+    (multiple-value-bind (n-lines-consumed result)
+        (collect-blockquote lines index base-indent)
+      (when n-lines-consumed
+        (return-from parse-markdown-blocks (values n-lines-consumed result))))
+    (cond
+      ((maybe-itemize-offset line)
+       (collect-markdown-itemize lines index (maybe-itemize-offset line)))
+      ((and (indentation line) (>= (indentation line) (+ base-indent 4)))
+       (collect-indented-code lines index base-indent))
+      (t nil))))

contrib/sb-manual/package.lisp

@@ -0,0 +1,4 @@
+(locally (declare (sb-ext:muffle-conditions sb-int:package-at-variance))
+  (handler-bind ((sb-int:package-at-variance #'muffle-warning))
+    (defpackage :sb-manual
+      (:use :cl :sb-alien))))

contrib/sb-manual/pax.lisp

@@ -0,0 +1,232 @@
+;;;; PAX stubs
+;;;;
+;;;; Contribs cannot depend on external libraries, so we fake as much
+;;;; of PAX and DRef as necessary. SWITCH-TO-PAX switches to the real
+;;;; implementation.
+;;;;
+;;;; If PAX is not loaded, the dummy DEFSECTION below still gives us
+;;;; the ability to use M-. on section names in docstrings as they are
+;;;; just variables, which makes navigating the documentation faster.
+;;;;
+;;;; When PAX is loaded, we have PAX::@BROWSING-LIVE-DOCUMENTATION for
+;;;; low-latency, interactive documentation work and
+;;;; PAX::@GENERATING-DOCUMENTATION for autolinked documentation.
+
+(in-package :sb-manual)
+
+(eval-when (:compile-toplevel :load-toplevel :execute)
+  (defvar *use-pax* nil)
+  ;; A list of (LOCAL-SYMBOL PACKAGE) elements. Originally,
+  ;; LOCAL-SYMBOL has home package SB-MANUAL. For example, the element
+  ;; (SECTION :PAX) causes PAX:SECTION to be SHADOWING-IMPORTed in
+  ;; SWITCH-TO-PAX.
+  (defvar *dummies* ()))
+
+(defmacro defun-dummy ((name package) lambda-list &body body)
+  (unless *use-pax*
+    `(eval-when (:compile-toplevel :load-toplevel :execute)
+       (pushnew '(,name ,package) *dummies* :test #'equal)
+       (defun ,name ,lambda-list ,@body))))
+
+(defmacro defmacro-dummy ((name package) lambda-list &body body)
+  (unless *use-pax*
+    `(eval-when (:compile-toplevel :load-toplevel :execute)
+       (pushnew '(,name ,package) *dummies* :test #'equal)
+       (defmacro ,name ,lambda-list ,@body))))
+
+(defparameter *extra-dummies*
+  '((argument :pax)
+    (macro :dref)
+    (setf-function :dref)
+    (setf-generic-function :dref)
+    (section :pax)
+    (clhs :pax)))
+
+(defun dummy (symbol)
+  (let ((*package* (find-package :sb-manual)))
+    (read-from-string (symbol-name symbol))))
+
+;;; We might want to populate this with GUESS-PACKAGE-FROM-ARGLIST.
+(defvar *definition-to-docstring-package*)
+(defvar *package-to-docstring-package*)
+
+(defun switch-to-pax ()
+  (unless *use-pax*
+    (require 'mgl-pax)
+    ;; Replace dummies with the real symbols.
+    (let ((dummies (loop for (name package) in (append *dummies*
+                                                       *extra-dummies*)
+                         collect (format nil "~A:~A" package name))))
+      (shadowing-import (mapcar #'read-from-string dummies) :sb-manual))
+    ;; Arrange for that only SECTIONs will be exported by
+    ;; PAX:DEFSECTION.
+    (eval-string
+     "(defmethod pax:exportable-reference-p
+          ((package (eql (find-package 'sb-manual)))
+           symbol locative-type locative-args)
+        (eq locative-type 'section))")
+    ;; Reevaluate DEFSECTION forms with PAX.
+    (do-external-symbols (symbol :sb-manual)
+      (when (and (char= #\@ (aref (symbol-name symbol) 0))
+                 (boundp symbol))
+        (let ((value (symbol-value symbol)))
+          (assert (listp value))
+          (assert (eq (first value) 'defsection))
+          (assert (eq (second value) symbol))
+          (let ((source-location
+                  (sb-int:info :source-location :variable symbol)))
+            (eval `(,(read-from-string "pax:defsection")
+                    ,@(subst-extras (rest value))))
+            (setf (sb-int:info :source-location :variable symbol)
+                  source-location)))))
+    (convert-docstring-package-overrides-to-pax)
+    ;; FIXME: register doc?
+    (setq *use-pax* t)))
+
+;;; Convert *DEFINITION-TO-DOCSTRING-PACKAGE* to
+;;; DREF:DEFINITION-PROPERTIES and *PACKAGE-TO-DOCSTRING-PACKAGE* to
+;;; DREF:DEFINITION-PROPERTIES. See DREF-EXT:DOCSTRING*.
+(defun convert-docstring-package-overrides-to-pax ()
+  (loop for ((name locative) package) in *definition-to-docstring-package*
+        do (eval-format
+            "(setf (dref-ext:definition-property (dref:dref '~S '~S) 'docstring)
+                   (list nil (find-package ~S)))"
+            name (subst-extras locative) package))
+  (loop for (from-package to-package) in *package-to-docstring-package*
+        do (eval-format
+            "(setf (dref-ext:definition-property `(:package ,(find-package ~S))
+                                                 'docstring)
+                   (list nil (find-package ~S)))"
+            from-package to-package)))
+
+(defun eval-string (string)
+  (let ((*package* (find-package :sb-manual)))
+    (eval (read-from-string string))))
+
+(defun eval-format (format-control &rest format-args)
+  (let ((*package* (find-package :sb-manual)))
+    (eval (read-from-string (apply #'format nil format-control format-args)))))
+
+(defun subst-extras (tree)
+  (let ((new-tree tree))
+    (loop
+      for (name package) in *extra-dummies*
+      do (let ((new-name (read-from-string (format nil "~A:~A" package name))))
+           (setq new-tree (subst new-name name new-tree))))
+    new-tree))
+
+
+(defmacro-dummy (defsection pax)
+                (name (&key (package *package*) (export t) title)
+                      &body entries)
+  (let ((defsection-form
+          `(defsection ,name (:package ,package :export ,export :title ,title)
+             ,@entries)))
+    `(progn
+       (defparameter ,name ',defsection-form)
+       ,@(when export
+           `((export ',name :sb-manual))))))
+
+(defun-dummy (section-name :pax) (section)
+  (second section))
+
+(defun-dummy (section-title :pax) (section)
+  (getf (third section) :title))
+
+(defun-dummy (section-package :pax) (section)
+  (find-package (getf (third section) :package)))
+
+;;; This is a list of (NAME LOCATIVE) elements with our dummy DEFSECTION.
+(defun-dummy (section-entries :pax) (section)
+  (nthcdr 3 section))
+
+(defun-dummy (xref-name :dref) (xref)
+  (first xref))
+
+(defun-dummy (xref-locative :dref) (xref)
+  (normalize-locative (second xref)))
+
+(defun normalize-locative (locative)
+  (if (and (listp locative)
+           (null (cdr locative)))
+      (first locative)
+      locative))
+
+(defun-dummy (xref-locative-type :dref) (xref)
+  (first (sb-c::ensure-list (second xref))))
+
+(defun-dummy (resolve :dref) (xref)
+  (cond ((eq (xref-locative-type xref) 'section)
+         (or (ignore-errors (symbol-value (xref-name xref)))
+             (assert nil () "Undefined SECTION ~S." (xref-name xref))))
+        (t
+         (assert nil () "Unexpected locative type in ~S." xref))))
+
+(defun-dummy (arglist :dref) (xref)
+  (let ((name (xref-name xref))
+        (locative-type (xref-locative-type xref)))
+    (lambda-list* name locative-type)))
+
+(defun-dummy (docstring :dref) (xref)
+  (values (let ((name (xref-name xref))
+                (locative-type (xref-locative-type xref)))
+            (case locative-type
+              ((function variable)
+               (documentation name locative-type))
+              ((generic-function)
+               (documentation name 'function))
+              ((type class structure condition)
+               (documentation name 'type))
+              (t
+               (cond ((eq locative-type (dummy 'macro))
+                      (documentation (macro-function name) t))
+                     ((eq locative-type (dummy 'setf-function))
+                      (documentation (fdefinition name) t))
+                     ((eq locative-type (dummy 'setf-generic-function))
+                      (documentation (fdefinition name) t))
+                     (t
+                      (assert nil () "Unexpected locative type in ~S."
+                              xref))))))
+          ;; To be compatible with PAX::@PACKAGE-AND-READTABLE, we
+          ;; always return a non-NIL package.
+          (docstring-package xref)))
+
+
+(defun lambda-list* (name kind)
+  (case kind
+    ((package constant variable type structure class condition method nil)
+     nil)
+    (t
+     ;; KLUDGE: Eugh.
+     ;;
+     ;; believe it or not, the above comment was written before CSR
+     ;; came along and obfuscated this.  (2005-07-04)
+     (when (symbolp name)
+       (labels ((clean (x &key optional key)
+                  (typecase x
+                    (atom x)
+                    ((cons (member &optional))
+                     (cons (car x) (clean (cdr x) :optional t)))
+                    ((cons (member &key))
+                     (cons (car x) (clean (cdr x) :key t)))
+                    ((cons (member &whole &environment))
+                     ;; Skip these
+                     (clean (cdr x) :optional optional :key key))
+                    ((cons cons)
+                     (cons
+                      (cond (key (if (consp (caar x))
+                                     (caaar x)
+                                     (caar x)))
+                            (optional (caar x))
+                            (t (clean (car x))))
+                      (clean (cdr x) :key key :optional optional)))
+                    (cons
+                     (cons
+                      (cond ((or key optional) (car x))
+                            (t (clean (car x))))
+                      (clean (cdr x) :key key :optional optional))))))
+         (multiple-value-bind (ll unknown)
+             (sb-introspect:function-lambda-list name)
+           (if unknown
+               (values nil t)
+               (clean ll))))))))

contrib/sb-manual/sb-manual.asd

@@ -0,0 +1,42 @@
+(error "Can't build contribs with ASDF")
+
+(defsystem "sb-manual"
+  :serial t
+  :components ((:file "package")
+               (:file "pax")
+               (:file "docstring")
+               (:file "markdown")
+               (:file "texinfo")
+               (:file "manual")
+               (:module "doc/"
+                :serial t
+                :components ((:file "sbcl")
+                             (:file "support-and-bugs")
+                             (:file "intro")
+                             (:file "start-stop")
+                             (:file "compiler")
+                             (:file "debugger")
+                             (:file "efficiency")
+                             (:file "beyond-ansi")
+                             (:file "external-formats")
+                             (:file "ffi")
+                             (:file "pathnames")
+                             (:file "streams")
+                             (:file "package-locks")
+                             (:file "threading")
+                             (:file "timers")
+                             (:file "networking")
+                             (:file "profiling")
+                             (:file "contrib-modules")
+                             (:file "sb-aclrepl")
+                             (:file "sb-concurrency")
+                             (:file "sb-cover")
+                             (:file "sb-grovel")
+                             (:file "sb-introspect")
+                             (:file "sb-md5")
+                             (:file "sb-posix")
+                             (:file "sb-queue")
+                             (:file "sb-rotate-byte")
+                             (:file "sb-simd")
+                             (:file "sb-simple-streams")
+                             (:file "deprecation")))))

contrib/sb-manual/sb-manual.texinfo

@@ -0,0 +1,94 @@
+@c Generated by the sb-manual contrib. Do not edit.
+
+@node sb manual
+@section sb-manual
+
+@menu
+* Browsing Live with PAX: browsing live with pax.
+* Fancy Documentation with PAX: fancy documentation with pax.
+@end menu
+
+The @code{sb-manual} module has the sections of the SBCL user manual in
+Lisp variables. The names of the variables (all start with the
+character @code{@@}) are exported from the @code{sb-manual} package. Sections
+are defined with the @code{defsection} macro:
+
+@example
+(defsection @@example (:title "Example")
+  "This is an example, but see the real @@SB-MANUAL."
+  (print function)
+  (@@subexample section))
+@end example
+
+where @code{defsection} is a dummy implementation of
+@code{pax:defsection} (see @url{https://github.com/melisgl/mgl-pax/}).
+
+In Slime, @code{M-.} on @code{"@@SB-MANUAL"}, @code{"print"}, or on
+@code{"@@subexample"} will take you to the respective definition. This
+makes it easy to navigate the documentation. Normal Lisp definition
+docstrings and section docstrings reference sections following the
+usual convention of uppercasing the name. Docstrings are in a subset
+of Markdown and use very little markup in general, so they are easy
+to read directly in the source.
+
+The official manual in Info, HTML and PDF formats is generated via
+Texinfo generated from these definitions.
+
+When PAX is loaded, the dummy @code{defsection} definitions are made
+real, so that PAX can work with them.
+
+@node browsing live with pax
+@subsection Browsing Live with PAX
+
+With PAX, you can browse the manual live. The documentation of this
+feature is available at
+@url{https://melisgl.github.io/mgl-pax-world/pax-manual.html#MGL-PAX:@@BROWSING-LIVE-DOCUMENTATION%20MGL-PAX:SECTION}.
+
+If you are browsing this manual live right now, here is the
+equivalent live link: @code{pax::@@browsing-live-documentation}.
+
+Notable features:
+
+@itemize
+@item Autolinks within the manual: if @code{sb-ext:exit} is mentioned, then
+  it's linked to its documentation. You basically get links to where
+  @code{M-.} would go in the sources.
+
+@item Autolinks to the CLHS.
+
+@item View the documentation of any Lisp definition or section without
+  generating the entire manual.
+
+@item Locatives (e.g. the @code{"[function]"} in @code{"- [function]
+  SB-EXT:EXIT"}) are also links in live browsing: they tell Slime
+  to visit the definition.
+
+For this to work, you need to allow Slime to evaluate Elisp sent
+from SBCL:
+
+@example
+(setq slime-enable-evaluate-in-emacs t)
+@end example
+
+and maybe your window manager focus stealing configuration needs
+tweaking as well.
+@end itemize
+
+Live browsing can greatly reduce the latency of Edit-Compile-View
+Loop, when working on documentation.
+
+@node fancy documentation with pax
+@subsection Fancy Documentation with PAX
+
+PAX can generate dead documentation, too. In the SBCL sources,
+@code{contrib/sb-manual/make-pax-docs.sh} generates the manual in plain
+text, Markdown, PDF, and HTML formats. These differ from those
+generated via Texinfo in that they are autolinked (like when
+@ref{browsing live with pax}).
+
+Also, you can generate documentation yourself with e.g.
+
+@example
+(pax:document sb-manual:@@sbcl-manual :format :markdown)
+@end example
+

contrib/sb-manual/texinfo.lisp

@@ -0,0 +1,206 @@
+(in-package :sb-manual)
+
+(defun locative-type-to-texinfo (locative-type)
+  (case locative-type
+    (function
+     (values "Function" "ffindex"))
+    (generic-function
+     (values "Generic function" "ffindex"))
+    (variable
+     (values "Variable" "vvindex"))
+    (class
+     (values "Class" "ttindex"))
+    (condition
+     (values "Condition" "ttindex"))
+    (structure
+     (values "Structure" "ttindex"))
+    (type
+     (values "Type" "ttindex"))
+    (t
+     (cond
+       ((eq locative-type (dummy 'macro))
+        (values "Macro" "ffindex"))
+       ((eq locative-type (dummy 'setf-function))
+        (values "Setf function" "ffindex"))
+       ((eq locative-type (dummy 'setf-generic-function))
+        (values "Setf generic function" "ffindex"))
+       (t
+        (assert nil () "Unexpected locative type ~S." locative-type))))))
+
+(defmacro with-texinfo-to-file (file &body body)
+  `(call-maybe-with-texinfo-to-file (lambda () ,@body)
+                                   ,file))
+
+(defun call-maybe-with-texinfo-to-file (fn file)
+  (if file
+      (with-open-file (*standard-output* file :direction :output
+                                         :if-does-not-exist :create
+                                         :if-exists :supersede)
+        (format t "@c Generated by the sb-manual contrib. Do not edit.~%~%")
+        (funcall fn))
+      (funcall fn)))
+
+;;; Write the Texinfo for SECTION to *STANDARD-OUTPUT*. When recursing
+;;; into child sections, if a section is in PAGES, then emit an
+;;; @include and open a new a file for output.
+(defun emit-texinfo-for-section (section &key pages (depth 0)
+                                 top-level-menus-to-file
+                                 top-level-contents-to-file)
+  (let ((title (section-title section))
+        (entries (section-entries section)))
+    (format t "@node ~A~%" (texinfo-node-id section))
+    (format t "~A ~A~%~%"
+            (ecase depth
+              (0 "@top")
+              (1 "@chapter")
+              (2 "@section")
+              (3 "@subsection")
+              (4 "@subsubsection"))
+            title)
+    ;; Generate the @menu
+    (let ((child-sections
+            (loop for entry in entries
+                  when (and (not (stringp entry))
+                            (eq (xref-locative-type entry) 'section))
+                    collect (resolve entry))))
+      (when child-sections
+        (unless top-level-menus-to-file
+          (format t "@menu~%"))
+        (with-texinfo-to-file top-level-menus-to-file
+          (dolist (child-section child-sections)
+            (format t "* ~A: ~A.~%" (section-title child-section)
+                    (texinfo-node-id child-section))))
+        (unless top-level-menus-to-file
+          (format t "@end menu~%~%"))))
+    ;; Generate the documentation
+    (let ((*package* (section-package section)))
+      (with-texinfo-to-file top-level-contents-to-file
+        (dolist (entry entries)
+          (cond ((stringp entry)
+                 ;; KLUDGE: @SBCL-MANUAL has an extra docstring that's
+                 ;; pretty much the same as @copying in
+                 ;; doc/manual/sbcl.texinfo. Skip it.
+                 (unless top-level-contents-to-file
+                   (emit-texinfo-for-docstring entry)
+                   (format t "~%")))
+                (t
+                 (if (not (eq (xref-locative-type entry) 'section))
+                     (emit-texinfo-for-definition entry)
+                     (let ((page (find (xref-name entry) pages
+                                       :key #'first)))
+                       (when page
+                         (format t "@include ~A~%" (second page)))
+                       (with-texinfo-to-file (second page)
+                         (emit-texinfo-for-section (resolve entry)
+                                                   :pages pages
+                                                   :depth (1+ depth))))))))))))
+
+(defun emit-texinfo-for-definition (xref)
+  (multiple-value-bind (docstring *package*) (docstring xref)
+    (multiple-value-bind (type index)
+        (locative-type-to-texinfo (xref-locative-type xref))
+      (let* ((name (xref-name xref))
+             (*print-case* :downcase)
+             ;; For e.g. #'print
+             (*print-pretty* t)
+             ;; The arglist must be on the @deffn line.
+             (*print-right-margin* most-positive-fixnum))
+        (format t "@anchor{~A ~A ~A}~%" type
+                (string-downcase (package-name (symbol-package name)))
+                (string-downcase (symbol-name name)))
+        ;; E.g. @vvindex @sortas{save-hooks* sb-ext} *save-hooks* [sb-ext]
+        (let ((symbol-name (string-downcase (symbol-name name)))
+              (symbol-package-name
+                (string-downcase (package-name (symbol-package name)))))
+          (format t "@~A @sortas{~A ~A} ~A [~A]~%"
+                  index
+                  (sort-as-name symbol-name)
+                  (sort-as-name symbol-package-name)
+                  symbol-name
+                  symbol-package-name))
+        ;; Since we took indexing into our own hands, we just use
+        ;; @deffn for all definitions. We could also use @defblock and
+        ;; @defline.
+        (format t "@deffn{~A} ~A~{ ~A~}~%"
+                ;; E.g. "Variable"
+                type
+                (let ((*package* (find-package :cl)))
+                  (prin1-to-string name))
+                (arglist xref))
+        (when docstring
+          (emit-texinfo-for-docstring docstring (arglist xref)))
+        (format t "@end deffn~%")))))
+
+;;; Remove leading non-alphanumeric characters. They are not important
+;;; when sorting names into indices.
+(defun sort-as-name (name)
+  (subseq name (or (position-if #'alphanumericp name) 0)))
+
+(defun emit-texinfo-for-docstring (docstring &optional arglist)
+  (markdown-to-texinfo (reindent-docstring docstring) arglist))
+
+
+(defparameter *pages*
+  '((@support-and-bugs "support-and-bugs.texinfo")
+    (@introduction "intro.texinfo")
+    (@starting-and-stopping "start-stop.texinfo")
+    (@compiler "compiler.texinfo")
+    (@debugger "debugger.texinfo")
+    (@efficiency "efficiency.texinfo")
+    (@beyond-the-ansi-standard "beyond-ansi.texinfo")
+    (@external-formats "external-formats.texinfo")
+    (@foreign-function-interface "ffi.texinfo")
+    (@pathnames "pathnames.texinfo")
+    (@streams "streams.texinfo")
+    (@package-locks "package-locks.texinfo")
+    (@threading "threading.texinfo")
+    (@timers "timers.texinfo")
+    (@networking "../../contrib/sb-bsd-sockets/sb-bsd-sockets.texinfo")
+    (@profiling "profiling.texinfo")
+    (@statistical-profiler "../../contrib/sb-sprof/sb-sprof.texinfo")
+    (@contributed-modules "contrib-modules.texinfo")
+    (@sb-aclrepl "../../contrib/sb-aclrepl/sb-aclrepl.texinfo")
+    (@sb-concurrency "../../contrib/sb-concurrency/sb-concurrency.texinfo")
+    (@sb-cover "../../contrib/sb-cover/sb-cover.texinfo")
+    (@sb-grovel "../../contrib/sb-grovel/sb-grovel.texinfo")
+    (@sb-introspect "../../contrib/sb-introspect/sb-introspect.texinfo")
+    (@sb-manual "../../contrib/sb-manual/sb-manual.texinfo")
+    (@sb-md5 "../../contrib/sb-md5/sb-md5.texinfo")
+    (@sb-posix "../../contrib/sb-posix/sb-posix.texinfo")
+    (@sb-queue "../../contrib/sb-queue/sb-queue.texinfo")
+    (@sb-rotate-byte "../../contrib/sb-rotate-byte/sb-rotate-byte.texinfo")
+    (@sb-sb-simd "../../contrib/sb-simd/sb-simd.texinfo")
+    (@sb-simple-streams
+     "../../contrib/sb-simple-streams/sb-simple-streams.texinfo")
+    (@deprecation "deprecation.texinfo")))
+
+(defun documentation-generation-date-string (&key long)
+  (multiple-value-bind (second minute hour day month year)
+      (decode-universal-time (get-universal-time))
+    (if long
+        (format nil "~D-~2,'0D-~2,'0D ~2,'0D:~2,'0D:~2,'0D"
+                year month day hour minute second)
+        (format nil "~D-~2,'0D" year month))))
+
+(defun generate-texinfo ()
+  (assert (not *use-pax*))
+  (let ((*default-pathname-defaults*
+          (truename (merge-pathnames
+                     "../../doc/manual/"
+                     sb-sys::*sbcl-homedir-pathname*))))
+    (with-texinfo-to-file "variables.texinfo"
+      (format t "@set VERSION ~A~%~
+                 @set UPDATE-MONTH ~A~%"
+              (lisp-implementation-version)
+              (documentation-generation-date-string)))
+    ;; We redirect most lines via *PAGES*, :TOP-LEVEL-MENUS-TO-FILE,
+    ;; :TOP-LEVEL-CONTENTS-TO-FILE. Silence the rest, which are not
+    ;; needed, as sbcl.texinfo only needs the includes.
+    (let ((*standard-output* (make-broadcast-stream)))
+      (emit-texinfo-for-section
+       (symbol-value '@sbcl-manual) :pages *pages*
+       :top-level-menus-to-file "sbcl-menu.texinfo"
+       :top-level-contents-to-file "sbcl-contents.texinfo"))))
+
+#+nil
+(generate-texinfo)

doc/manual/.gitignore

@@ -17,11 +17,16 @@ asdf.pdf
 asdf.ps
 asdf.texinfo
 asdf/
-docstrings/
 html-stamp
+sbcl.ff
+sbcl.ffs
+sbcl.tt
+sbcl.tts
+sbcl.vv
+sbcl.vvs
 sbcl.info*
 sbcl.pdf
 sbcl.ps
 sbcl/
-tempfiles-stamp
 variables.texinfo
+generated-texinfo-stamp

doc/manual/Makefile

@@ -1,6 +1,7 @@
 SBCLTEXI:=sbcl.texinfo
 ASDFTEXI:=asdf.texinfo
-DOCFILES:=*.texinfo $(ASDFTEXI)
+# The rest of the texinfo files are generated.
+DOCFILES:=sbcl.texinfo backmatter.texinfo $(ASDFTEXI)
 TMPTYPES:=aux cp cps fn fns ky log pg toc tp tps vr vrs
 TMPFILES:=$(foreach target,asdf sbcl,$(foreach type,$(TMPTYPES),$(target).$(type)))
 PSFILES=sbcl.ps asdf.ps
@@ -8,12 +9,8 @@ PDFFILES=sbcl.pdf asdf.pdf
 INFOFILES=sbcl.info asdf.info
 HTMLDIRS=$(basename $(SBCLTEXI)) $(basename $(ASDFTEXI))
 HTMLFILES=sbcl.html asdf.html
-# Place where generated documentation ends up. The value of
-# DOCSTRINGDIR has to end with a slash or you lose (it's passed to
-# Lisp's `pathname' function).
-DOCSTRINGDIR="docstrings/"
 CONTRIB_SRC_DIR="../../contrib/"
-I_FLAGS=-I $(DOCSTRINGDIR) -I $(CONTRIB_SRC_DIR)
+I_FLAGS=-I $(CONTRIB_SRC_DIR)
 
 # SBCL_SYSTEM is an optional argument to this make program. If this
 # variable is set, its contents are used as the command line for
@@ -52,7 +49,7 @@ version.texinfo:
 .PHONY: html
 html: html-stamp
 
-html-stamp: $(DOCFILES) docstrings
+html-stamp: $(DOCFILES) generated-texinfo-files
 	@rm -rf $(HTMLDIRS)
 	@rm -f $(HTMLFILES)
 #	$(MAKEINFO) $(I_FLAGS) --html --css-include=style-multi.css $(SBCLTEXI)
@@ -69,39 +66,39 @@ ps: $(PSFILES)
 	dvips -q -o $@ $<
 
 # DVI generation
-%.dvi: %.texinfo $(DOCFILES) docstrings
+%.dvi: %.texinfo $(DOCFILES) generated-texinfo-files
 	texi2dvi -q $(I_FLAGS) $<
 
 # PDF documentation
 .PHONY: pdf
 pdf: $(PDFFILES)
 
-%.pdf: %.texinfo $(DOCFILES) docstrings
+%.pdf: %.texinfo $(DOCFILES) generated-texinfo-files
 	$(TEXI2PDF) -q $(I_FLAGS) $<
 
 # info docfiles
 .PHONY: info
 info: $(INFOFILES)
 
-%.info: %.texinfo $(DOCFILES) docstrings
+%.info: %.texinfo $(DOCFILES) generated-texinfo-files
 	$(MAKEINFO) $(I_FLAGS) $<
 
 # Texinfo docstring snippets
-docstrings variables.texinfo: tempfiles-stamp
-	sh make-tempfiles.sh "$(SBCL_SYSTEM)" "$(DOCSTRINGDIR)" && touch $(DOCSTRINGDIR)
+generated-texinfo-files: generated-texinfo-stamp
+	sh generate-texinfo.sh "$(SBCL_SYSTEM)"
 
-tempfiles-stamp:
-	touch tempfiles-stamp
+generated-texinfo-stamp:
+	touch generated-texinfo-stamp
 
 .PHONY: clean
 clean:
 	rm -f *~ *.bak *.orig \#*\# .\#* texput.log *.fasl
-	rm -rf $(HTMLDIRS) $(DOCSTRINGDIR)
+	rm -rf $(HTMLDIRS)
 	rm -f  $(HTMLFILES)
 	rm -f contrib-docs.texi-temp
 	rm -f package-locks.texi-temp
 	rm -f variables.texinfo
-	rm -f $(PSFILES) $(PDFFILES) html-stamp tempfiles-stamp
+	rm -f $(PSFILES) $(PDFFILES) html-stamp generated-texinfo-stamp
 	rm -f $(TMPFILES) $(INDEXFILES)
 	rm -f sbcl.info sbcl.info-* asdf.info
 

doc/manual/README

@@ -0,0 +1,9 @@
+With the exception of sbcl.texinfo, backmatter.texinfo,
+sbcl-menu.texinf and sbcl-contents.texinfo all other Texinfo files are
+from SB-MANUAL::GENERATE-TEXINFO.
+
+With the exception of variables.texinfo, the generated files are under
+version control, to keep a closer eye on the Markdown-to-Texinfo
+converter, which is young. This is redundant, of course. In the
+fullness of time, in due course, when conditions allow, at the
+appropriate juncture, we may want to revisit this decision.

doc/manual/TEXINFO-HINTS

@@ -1,14 +0,0 @@
--*- text -*-
-
-Some hints for editing the manual files.  Feel free to add anything
-that will save the next person some time.  Thanks!
-
-
-- There's no need for Next, Prev, etc. pointers in @node lines:
-  makeinfo will deduce these automatically when the line after @node
-  contains a sectioning command like @section, @subsection.  Hence,
-  texinfo-multiple-files-update should not be used either.
-
-- Don't create or update Menus by hand; use C-c C-u C-a
-  (texinfo-all-menus-update) instead.  (Doesn't work in sbcl.texinfo,
-  but this file is only changed when an entire chapter is added.)

doc/manual/backmatter.texinfo

@@ -1,37 +1,34 @@
-@node Concept Index
+@node function index
 @comment  node-name,  next,  previous,  up
-@appendix Concept Index
+@appendix Function and Macro Index
 
-@printindex cp
+@printindex ff
 
-@node Function Index
+@node variable index
 @comment  node-name,  next,  previous,  up
-@appendix Function Index
+@appendix Variable and Constant Index
 
-@printindex fn
+@printindex vv
 
-@node Variable Index
-@comment  node-name,  next,  previous,  up
-@appendix Variable Index
-
-@printindex vr
-
-@node Type Index
+@node type index
 @comment  node-name,  next,  previous,  up
 @appendix Type Index
 
-@printindex tp
+@printindex tt
 
-@node Colophon
+@node colophon
 @comment  node-name,  next,  previous,  up
 @unnumbered Colophon
 
-This manual is maintained in Texinfo, and automatically translated
-into other forms (e.g. HTML or pdf). If you're @emph{reading} this
-manual in one of these non-Texinfo translated forms, that's fine, but
-if you want to @emph{modify} this manual, you are strongly advised to
-seek out a Texinfo version and modify that instead of modifying a
-translated version. Even better might be to seek out @emph{the}
-Texinfo version (maintained at the time of this writing as part of the
-SBCL project at @uref{http://sbcl.sourceforge.net/}) and submit a
-patch.
+This manual is maintained as part of the @ref{sb manual} contrib.
+@code{SB-MANUAL} can generate Texinfo, which is in turn converted into
+other formats (e.g. HTML or pdf). If you're @emph{reading} this manual
+in one of these, that's fine, but if you want to @emph{modify} this
+manual, you are strongly advised to seek the sources (most live under
+@file{contrib/sb-manual/doc/}) and submit a patch.
+
+@ifinfo
+When viewing Info in Emacs, the reader inserts superfluous ``see''
+words before internal links. Use @code{(setq Info-hide-note-references
+'hide)} to prevent this.
+@end ifinfo

doc/manual/generate-texinfo.lisp

@@ -1,100 +0,0 @@
-(map nil #'require '("asdf" "uiop"))
-(asdf:initialize-source-registry
- '(:source-registry :ignore-inherited-configuration))
-
-(with-compilation-unit ()
-  (load "docstrings.lisp"))
-
-;;;; Generating documentation strings
-
-(defvar *contrib-directory* #P"../../contrib/")
-
-(defvar *documented-packages*
-  '("COMMON-LISP" "SB-ALIEN" "SB-DEBUG" "SB-EXT" "SB-GRAY" "SB-MOP"
-    "SB-PCL" "SB-SYS" "SB-SEQUENCE" "SB-UNICODE" "SB-PROFILE"
-    "SB-THREAD"))
-
-(defun documented-contribs (&key (exclude '("asdf")))
-  (loop for texinfo-file in (directory (merge-pathnames
-                                        "*/*.texinfo" *contrib-directory*))
-        for name = (car (last (pathname-directory texinfo-file)))
-        for package = (string-upcase name)
-        when (cond
-               ((find name exclude :test #'string=)
-                nil)
-               ((find name result :test #'string= :key #'car)
-                nil)
-               (t
-                t))
-        collect (cons name package) into result
-        finally (return result)))
-
-(defun generate-docstrings-texinfo (runtime
-                                    &key (docstring-directory "docstrings/")
-                                   (blocklist '()))
-  (let* ((contribs (sort (documented-contribs :exclude (append '("asdf") blocklist)) #'string< :key #'car))
-         (packages (sort (append *documented-packages*
-                                 (map 'list #'cdr contribs))
-                         #'string<)))
-    (format t "/creating docstring snippets~@
-               ~2@Tfrom SBCL=\'~A\'~@
-               ~2@Tfor documented contribs~%~4@T~A~@
-               ~2@Tfor packages~%~4@T~A~%"
-            runtime (map 'list #'car contribs) packages)
-    (map nil (lambda (contrib) (require (car contrib))) contribs)
-    (apply #'sb-texinfo:generate-includes docstring-directory packages)))
-
-;;;; Special cases: external formats list, package locks, variables.template
-
-(defun replace-all (new old string)
-  (with-output-to-string (stream)
-    (loop with old-length = (length old)
-       for start = 0 then (+ offset old-length)
-       for offset = (search old string :start2 start)
-       while offset
-       do (write-string (subseq string start offset) stream)
-         (write-string new stream)
-       finally (write-string (subseq string start) stream))))
-
-(defun expand-variables (&key (input-file "variables.template")
-                           (output-file "variables.texinfo"))
-  (format t "/expanding variables in ~A~%" output-file)
-  (let* ((version (lisp-implementation-version))
-         (date (multiple-value-bind (second minute hour day month year)
-                   (decode-universal-time (get-universal-time))
-                 (declare (ignore second minute hour day))
-                 (format nil "~D-~2,'0D" year month)))
-         (template (uiop:read-file-string input-file))
-         (expanded (replace-all version "@VERSION@"
-                                (replace-all date "@MONTH@" template))))
-    (with-open-file (output output-file
-                            :direction :output
-                            :if-exists :supersede
-                            :if-does-not-exist :create)
-      (write-string expanded output))))
-
-(defun generate-external-format-texinfo (&optional (output-file "encodings.texi-temp"))
-  (format t "/creating ~A~%" output-file)
-  (with-open-file (stream output-file :direction :output :if-exists :supersede)
-    (flet ((table (items)
-             (format stream "@table @code~%~%")
-             (loop for (canonical-name . names) in items
-                do (format stream "@item ~S~%~{@code{~S}~^, ~}~%~%"
-                           canonical-name names))
-             (format stream "@end table~%")))
-      (let (result)
-        (loop for ef across sb-impl::*external-formats*
-              when (sb-impl::external-format-p ef)
-              do
-              (pushnew (sb-impl::ef-names ef) result :test #'equal))
-        (table (sort result #'string< :key #'car))))))
-
-;;;; Entry point
-
-(destructuring-bind (program runtime docstring-directory blocklist) *posix-argv*
-  (declare (ignore program))
-  (generate-docstrings-texinfo
-   runtime :docstring-directory docstring-directory :blocklist (uiop:split-string blocklist))
-
-  (expand-variables)
-  (generate-external-format-texinfo))

doc/manual/generate-texinfo.sh

@@ -1,7 +1,5 @@
 #!/bin/sh
 
-# Create Texinfo snippets from the documentation of exported symbols.
-
 # This software is part of the SBCL system. See the README file for
 # more information.
 #
@@ -28,17 +26,9 @@ if [ -z "$1" ] ; then
     . $SBCL_TOP/output/build-config
 else
     SBCLRUNTIME="$1"
-    SBCL_CONTRIB_BLOCKLIST=
 fi
-shift
 
-if [ -z "$1" ] ; then
-    DOCSTRINGDIR="${DOCSTRINGDIR:-docstrings/}"
-else
-    DOCSTRINGDIR="$1"
-fi
-shift
-
-${SBCLRUNTIME}                                                          \
-    --noinform --no-sysinit --no-userinit --noprint --disable-debugger  \
-    --script generate-texinfo.lisp "${SBCLRUNTIME}" "${DOCSTRINGDIR}" "${SBCL_CONTRIB_BLOCKLIST}"
+${SBCLRUNTIME}                                                            \
+    --noinform --no-sysinit --no-userinit --noprint --disable-debugger    \
+    --eval '(require :sb-manual)' --eval '(sb-manual::generate-texinfo)' \
+    --quit

doc/manual/sbcl-contents.texinfo

@@ -0,0 +1,20 @@
+@c Generated by the sb-manual contrib. Do not edit.
+
+@include support-and-bugs.texinfo
+@include intro.texinfo
+@include start-stop.texinfo
+@include compiler.texinfo
+@include debugger.texinfo
+@include efficiency.texinfo
+@include beyond-ansi.texinfo
+@include external-formats.texinfo
+@include ffi.texinfo
+@include pathnames.texinfo
+@include streams.texinfo
+@include package-locks.texinfo
+@include threading.texinfo
+@include timers.texinfo
+@include ../../contrib/sb-bsd-sockets/sb-bsd-sockets.texinfo
+@include profiling.texinfo
+@include contrib-modules.texinfo
+@include deprecation.texinfo

doc/manual/sbcl-menu.texinfo

@@ -0,0 +1,20 @@
+@c Generated by the sb-manual contrib. Do not edit.
+
+* Getting Support and Reporting Bugs: support and bugs.
+* Introduction: introduction.
+* Starting and Stopping: starting and stopping.
+* Compiler: compiler.
+* Debugger: debugger.
+* Efficiency: efficiency.
+* Beyond the ANSI Standard: beyond the ansi standard.
+* External Formats: external formats.
+* Foreign Function Interface: foreign function interface.
+* Pathnames: pathnames.
+* Streams: streams.
+* Package Locks: package locks.
+* Threading: threading.
+* Timers: timers.
+* Networking: networking.
+* Profiling: profiling.
+* Contributed Modules: contributed modules.
+* Deprecation: deprecation.

doc/manual/sbcl.texinfo

@@ -1,15 +1,19 @@
 \input texinfo   @c -*-texinfo-*-
 @c %**start of header
 @setfilename sbcl.info
+@paragraphindent 0
 @documentencoding UTF-8
 @c %**end of header
 @afourwide
 @fonttextsize 10
-@include texinfo-macros.texinfo
 @include variables.texinfo
 @set EDITION 0.1
 @settitle SBCL @value{VERSION} User Manual
 
+@defcodeindex ff
+@defcodeindex vv
+@defcodeindex tt
+
 @c for install-info
 @dircategory Software development
 @direntry
@@ -77,51 +81,16 @@ provided with absolutely no warranty. See the @file{COPYING} and
 @insertcopying
 
 @menu
-* Getting Support and Reporting Bugs::
-* Introduction::
-* Starting and Stopping::
-* Compiler::
-* Debugger::
-* Efficiency::
-* Beyond the ANSI Standard::
-* External Formats::
-* Foreign Function Interface::
-* Pathnames::
-* Streams::
-* Package Locks::
-* Threading::
-* Timers::
-* Networking::
-* Profiling::
-* Contributed Modules::
-* Deprecation::
-* Concept Index::
-* Function Index::
-* Variable Index::
-* Type Index::
-* Colophon::
+@include sbcl-menu.texinfo
+* Function and Macro Index: function index.
+* Variable and Constant Index: variable index.
+* Type Index: type index.
+* Colophon: colophon.
 @end menu
 
 @end ifnottex
 
-@include support-and-bugs.texinfo
-@include intro.texinfo
-@include start-stop.texinfo
-@include compiler.texinfo
-@include debugger.texinfo
-@include efficiency.texinfo
-@include beyond-ansi.texinfo
-@include external-formats.texinfo
-@include ffi.texinfo
-@include pathnames.texinfo
-@include streams.texinfo
-@include package-locks.texinfo
-@include threading.texinfo
-@include timers.texinfo
-@include sb-bsd-sockets/sb-bsd-sockets.texinfo
-@include profiling.texinfo
-@include contrib-modules.texinfo
-@include deprecation.texinfo
+@include sbcl-contents.texinfo
 @include backmatter.texinfo
 
 @bye

doc/manual/texinfo-macros.texinfo

@@ -1,12 +0,0 @@
-@c Some index prettification helper macros, for tricking the texindex
-@c collation "engine"
-@macro earmuffs{name}
-*\name\*
-@end macro
-@macro earstuds{name}
-+\name\+
-@end macro
-@macro setf{name}
-(setf \name\)
-@end macro
-@include docstrings/package-macros.texinfo

doc/manual/variables.template

@@ -1,2 +0,0 @@
-@set VERSION @VERSION@
-@set UPDATE-MONTH @MONTH@