The generated texinfo files (e.g. ffi.texinfo) are under version
control only to be able to conveniently track the Markdown-to-Texinfo
output until it's deemed stable enough.
- Conditionalize it on #+sb-doc.
- Drop Markdown blockquote support (it's not needed by any docstring
currently).
- Make the code more compact.
Without :SB-DOC, the core size impact is thus zero. With :SB-DOC, the
core size is increased by ~0.05% (~10kB).
- Add docstrings for user-facing declaration.
- Fix the hacks for declarations in the manual.
- Add documentation of DOCUMENTATION extensions to the manual.
- Add a "Declaration Index" appendix to the manual.
Summary
-------
Implement of a subset of PAX so that we can write PAX:DEFSECTIONlike
forms that supports the same restricted Markdown that we use for
docstrings and generate doc/manual/*.texinfo files from them.
Without PAX
-----------
There is no hard dependency on PAX. When the new SB-MANUAL contrib is
loaded, one can M-. around in documentation (sections are variables),
and function docstrings can now link to sections.
The generated Texinfo files are quite close to the originals, with
some loss of "semantic" markup: e.g. Markdown has `FOO` but no
@var{FOO} and @code{FOO}. We didn't derive much practical benefit from
that distinction.
With PAX
--------
(SB-MANUAL::SWITCH-TO-PAX) ensures that PAX is loaded and patches things
up, as if everything had been defined with PAX to begin with. Now, we
get PAX::@BROWSING-LIVE-DOCUMENTATION for low-latency, interactive
documentation work and PAX::@GENERATING-DOCUMENTATION for auto-linked
documentation.
Notable features:
- auto-generated links within the manual: if SB-EXT:EXIT is mentioned,
then it's linked to its documentation.
- auto-generated links to the CLHS (these links are red in PDF)
- locatives (e.g. the "[function]" in "- [function] SB-EXT:EXIT" are
also links and they go to the sources on GitHub (in live browsing,
they tell Slime to open the definition)
Details
-------
- Factor out the Markdown-to-Texinfo code into a new SB-MANUAL
contrib.
- Convert doc/manual/*.texinfo files to PAXlike DEFSECTION format and
add each chapter as a contrib/sb-manual/doc/<chapter>.lisp file.
- Fix a *lot* of small issues during the conversion.
- Make doc/make-doc.sh regenerate the all .texinfo files except
sbcl.texinfo and backmatter.texinfo.
- Fix docstrings of Lisp definitions used in the manual to conform to
the supported Markdown syntax.
- Retain the Texinfo function, variable and type indicies but drop the
concept index. There is no obvious way to support that in Markdown,
and it was used rather sporadically rather incomplete.
- Even with the new features, the amount of Lisp code didn't change
significantly.
- See contrib/sb-manual/README for the Todo list
Previously, the format was ad-hoc and implicitly defined by
doc/manual/docstrings.lisp. Now, it is a bit less ad-hoc and still
implicitly defined docstrings.lisp.
The syntax is now a simple, strict subset of Markdown with escaping
extensions. The long comment near the top of docstrings.lisp has the
documentation.
Some notable changes:
- Markdown backticks are supported (e.g. `exit`(3),
`/usr/local/bin/`).
- When the heuristic codification is overly eager, use backslashes to
prevent codification (e.g. \\HTTP, where the backslash is doubled
assuming that this is in a docstring).
- It is now possible to have code that's not downcased
(`\\\\AC_LOCAL`, backslashes doubled again).
- Fenced code blocks are supported (but use them sparingly).
- Texinfo @itemize is used instead of @table.
- Texinfo @var was inconsistently used at best. What used to be @var
is now simply @code.
For the user, the most visible effect of this change is that the
manual has much fewer errors in codification, and a few docstrings
have backticks and escapes in them, which should be acceptably
readable in the sources or via CL:DOCUMENTATION.
1. Previously, all uppercase words were downcased and enclosed in
texinfo @code{}, which didn't handle many common cases correctly. With
this change, only words that name interned symbols or packages are
codified:
POSIX -> POSIX (was @code{posix})
UDP -> UDP (was @code{udp})
The symbol name is looked up in the heuristically determined package,
which tries to match the package in the IN-PACKAGE form in effect
where the docstring is in the sources. This means that if M-. works in
Slime, then the documentation generator should be able to figure out
that it's code.
This change revealed several typos in the docstrings where the all
uppercase word did not correspond to the name of an interned
symbol.
2. Lowercase suffixes are now stripped from the word:
STRINGs -> @code{string}s (was STRINGS)
CLASSes -> @code{class}es (was CLASSes)
SETFable -> @code{setf}able (was SETFable)
3. #\' is now a delimiter, so this works:
ARRAY's -> @code{array}'s (was ARRAY's)
4. Trailing #\: characters followed by whitespace are considered
delimiters:
NIL: -> @code{nil}: (was @code{nil:})
5. Fixed the *NOT-SYMBOLS* opt-out mechanism (renamed to *NOT-CODE*)
and added an opt-in (*CODE*).
Overall, the new heuristics work much better with current docstring
style. However, there is no way to get all cases right, so explicit
markup in docstrings will be needed. Fixing that would require even
more of MGL-PAX, so I'm stopping here.
Also, we should probably have the generated documentation (a single
file, in a format suitable for diffing) under version control so that
we can detect documentation typos more easily.
This fixes the ANSI non-compliance passed down from early
CMUCL (1991-10-31). The bug was that FDEFINITION did not return the
same value as SYMBOL-FUNCTION for symbols (lp#799533). The fix is an
incompatible change but only affects encapsulated (e.g. TRACEd and
SP-PROFILE:PROFILEd) non-generic functions.
However, encapsulations belong to the definition (the name) and not to
the function associated with the name, so (SETF (FDEFINITION NAME) FN)
strips encapsulations from FN and keeps the encapsulations of NAME.
- SB-EXT:UNENCAPSULATED-FUNCTION was added.
- Encapsulated functions are printed as
#<FUNCTION FOO ENCAPSULATED>
or, if they have no proper name, as
#<FUNCTION FOO ENCAPSULATED {B1782332}>
This is the same as for unencapsulated functions except for the
additional " ENCAPSULATED".
- Unused slots were removed from SB-PROFILE:PROFILE-INFO.
- Since generic function identity is stable in face of encapsulation,
CLOS is largely unaffected by this change.
Add the ability to pass and return C structs by value in alien
function calls and callbacks on x86-64 linux and ARM64 darwin. Note
- x86-64: eightbyte classification with INTEGER/SSE register classes
- ARM64: HFA detection for float aggregates, GPR pairs for small structs
Large structs (>16 bytes) use hidden pointer passing. Small structs
are unpacked from registers into heap on return.
Callbacks preserve the hidden return pointer across the Lisp call and
copy results to the caller-provided destination.
With all the machinery in place to insert cleanup code correctly given
multiple derived dynamic extents, we just need to weaken the condition
for stack allocating a closure from having a single reference to
having all references be downward funarg uses.
Update the manual not only to document this new optimization but also
the existing way for users to declare downward funargs for their own
functions.
Also, there are a couple cases in the self-build where this
optimization applies. Remove those explicit dynamic extent
declarations which no longer do (or never did) anything and let the
compiler prove/auto-stack-allocate many of the valid cases. This makes
it easier to find the cases that the compiler cannot prove safe to
stack allocate yet, which led to finding some potential optimizations
that don't work yet, which failing test cases have been added for.
The code for breakpoint commands in the debugger inherited from CMU CL
was deleted in 8a8a892280, presumably
because it was slightly broken on x86oids due to a constant being
wrong. c7e96cb3045bc64179b9e81f266fb611aa67e1ac fixed that, so the
breakpoint commands can be resurrected. The step command based on
breakpoints has been included but under a different name because the
single stepper exists now. All the functionality of the
instrumentation based single stepper could likely be subsumed in the
breakpoint based stepper without the instrumentation overhead, but for
now let's have them coexist.
* Define a way to free alien callbacks so that we can recycle their
trampoline entry points per function type, instead of indefinitely
eating static space.
* Document and define redefinition semantics for alien callables,
allowing incremental redefinition. These semantics are based on how
CMU CL handles redefining callbacks, with the difference that type
incompatibility is stricter, and incompatible redefinitions invalidate
the old callback. I'm not sure whether have strict type equality or
having a weaker notion of compatibility is better. The docstring is
cribbed from CMU CL, and the interface now matches their DEF-CALLBACK
closely otherwise.
* ALIEN-LAMBDA now returns a Lisp function rather than a callback,
since its semantics were always a bit suspicious: it allocated a new
entry trampoline per closure allocation, and users would've needed to
call the new free function on the value explicitly after it was done
being used somehow in order to recycle trampoline. The symbol is
unexported but used by CFFI prior to commit
22762bf8d2.
* Export WITH-ALIEN-CALLABLE instead, which imitates the dynamic
extent behavior of WITH-ALIEN and frees the alien callback at the end
of its scope. This allows a convenient, exported way to create local
callables that close over Lisp values without indefinitely allocating
trampolines, since the types of the local alien functions are constant
and hence the entry point trampoline SAPs will just be reused.
* Define documentation methods for ALIEN-VALUEs and names bound by
SB-ALIEN:DEFINE-ALIEN-CALLABLE.
* The SB-ALIEN:DEFINE-ALIEN-CALLABLE interface does linkage by name,
so we can remove various non-exported functions that tried to deal in
ALIEN-VALUEs directly
* We don't need to bother with caching callback SAPs by function,
since we recycle SAPs per function type for local alien callbacks and
type-incompatibly redefined name-bound alien callbacks.
* Declare ENTER-ALIEN-CALLBACK as unsafe.
* Instead of trying to share the Lisp from alien type parsing code for
all callbacks with the same type signature, just inline the wrapper
code directly into the body of the Lisp function. This simplifies how
callbacks are handled and allows us to remove the Lisp trampoline
layer as well, thereby reducing some call indirection
overhead. Another benefit is that the declared types of the arguments
are available to the compiler now that the type parsing code and the
actual function body are in the same function, allowing the compiler
to use unboxed arithmetic for addition when the arguments are both
declared sb-alien:INT for example.
* We dispose of the ALIEN-CALLBACK macro which associates a callback
entry point directly to an existing Lisp function. This functionality
was not exported and people don't seem to use callbacks in this
way. If the user wants a Lisp function that is associated with
multiple differently typed callbacks, the user can do so manually by
declaring a separate Lisp function and defining multiple alien
callables calling it. It is better to allow the user the option of
using unboxed arithmetic than to default to allowing differently typed
callbacks for the same function.
* The desire to share these Lisp wrappers may have been due to space
concerns, but generally speaking other kinds of argument and result
coercion code is typically inlined and furthermore the type parsing
code resides in GC-managed space, so it's okay that the code now
exists per alien callable instead of per type signature. It's actually
more of a concern that assembler wrappers are allocated per alien
callable in uncollected static space.
* A micro-benchmark with an alien-funcall of a callback that adds two
numbers shows a 5-10% improvement in speed, presumably due to losing a
layer of indirection.
* Update the internals manual entry accordingly, noting in passing how
foreign threads are handled specially.
(which sounds unfortunately similar to #+linkage-table
for alien linkage, but happily that old feature name has been
eradicated, as all platforms implement the alien linkage table
and there are no runtime choices based on presence/absence)
It was previously renamed in rev 851cbd2b but the new name was too wordy.
The motivation here is the same: some pending changes for x86-64 that make
lisp-to-lisp calls go through a linkage table (almost doing away with
all FDEFNs) is nearly ready to be committed for real now.
