sbcl.sbcl/doc/internals/foreign-linkage.texinfo

@node Foreign Linkage
@comment  node-name,  next,  previous,  up
@chapter Foreign Linkage

@menu
* Linkage-table::
* Lazy Alien Resolution::
* Callbacks::
@end menu

@node Linkage-table
@comment  node-name,  next,  previous,  up
@section Linkage-table

Linkage-table allows saving cores with foreign code loaded, and is
also utilized to allow references to as-of-yet unknown aliens.
@xref{Lazy Alien Resolution}.

The SBCL implementation is somewhat simplified from the CMUCL one by
Timothy Moore, but the basic idea and mechanism remain identical:
instead of having addresses from @code{dlsym(3)} in the core, we have
addresses to an mmapped memory area (@code{ALIEN_LINKAGE_SPACE}) that
is initialized at startup to contain jumps & references to the correct
addresses, based on information stored on the lisp side in
@code{*LINKAGE-INFO*}.

@subsection Differences to CMUCL

CMUCL does lazy linkage for code, keeps all foreign addresses in the
linkage-table, and handles the initialization from C. We do eager
linkage for everything, initialize only the runtime-critical symbols
from C, and initialize anything else from lisp.

@subsection Nitty Gritty Details

On system startup @code{FOREIGN-REINIT} iterates through the
@code{*LINKAGE-INFO*}, which is a hash-table mapping dynamic foreign
names to @code{LINKAGE-INFO} structures, and calls
@code{arch_write_linkage_table_entry} to write the
appropriate entries to the linkage-table.

When a foreign symbol is referred to,
@code{ENSURE-FOREIGN-LINKAGE} is called, which looks for the
corresponding entry in @code{*LINKAGE-INFO*}, creating one and writing
the appropriate entry in the linkage table if necessary.

@code{FOREIGN-SYMBOL-ADDRESS} and @code{FOREIGN-SYMBOL-SAP} take an
optional datap argument, used to indicate that the symbol refers to a
variable. In similar fashion there is a new kind of fixup and a new
VOP: @code{:FOREIGN-DATAREF} and @code{FOREIGN-SYMBOL-DATAREF-SAP}.

The @code{DATAP} argument is automagically provided by the alien
interface for normal definitions, but is really needed only for
dynamic foreign variables. For those it indicates the need for the
indirection either within a conditional branch in
@code{FOREIGN-SYMBOL-SAP}, or via @code{:FOREIGN-DATAREF} fixup and
@code{FOREIGN-SYMBOL-DATAREF-SAP} VOP: "this address holds the
address of the foreign variable, not the variable itself".

One thing worth noting is that @code{FOREIGN-SYMBOL-SAP} and friends
now have the potential side-effect of entering information in
@code{*LINKAGE-INFO*} and the linkage-table proper. If the usage case
is about checking if the symbol is available use
@code{FIND-FOREIGN-SYMBOL-ADDRESS}, which is side-effect free. (This
is used by SB-POSIX.)

@subsection Porting

@subsubsection Porting to new operating systems

Find a memory area for the linkage-table, and add it for the OS in
@file{src/compiler/target/parms.lisp} by defining
@code{SB-VM:ALIEN-LINKAGE-SPACE-START}. See existing ports and CMUCL for
examples.

@subsubsection Porting to new architectures

Write @code{arch_write_linkage_table_entry}.

Write @code{FOREIGN-SYMBOL-DATAREF} VOP.

Define correct @code{SB-VM:ALIEN-LINKAGE-TABLE-ENTRY-SIZE} in
@file{src/compiler/target/parms.lisp}.

@page
@node Lazy Alien Resolution
@comment  node-name,  next,  previous,  up
@section Lazy Alien Resolution

SBCL is able to deal with forward-references to
aliens -- which is to say, compile and load code referring to aliens
before the shared object containing the alien in question has been
loaded.

This is handled by @code{ENSURE-FOREIGN-SYMBOL-LINKAGE}, which
first tries to resolve the address in the loaded shared objects, but
failing that records the alien as undefined and returns the address of
a read/write/execute protected guard page for variables, and address
of @code{undefined_alien_function} for routines. These are in turn
responsible for catching attempts to access the undefined alien, and
signalling the appropriate error.

These placeholder addresses get recorded in the linkage-table.

When new shared objects are loaded @code{UPDATE-ALIEN-LINKAGE-TABLE} is
called, which in turn attempts to resolve all currently undefined
aliens, and registers the correct addresses for them in the
linkage-table.

@page
@node Callbacks
@comment  node-name,  next,  previous,  up
@section Callbacks

SBCL is capable of providing C with linkage to Lisp -- the upshot of which is that
C-functions can call Lisp functions thru what look like function pointers to C.

These ``function pointers'' are called Alien Callbacks. An alien
callback sequence has 2 parts / stages / bounces:

@itemize
@item Assembler Wrapper

saves the arguments from the C-call according to the alien-fun-type of
the callback, and calls #'ENTER-ALIEN-CALLBACK with the index
identifying the callback, a pointer to the arguments copied on the
stack and a pointer to return value storage. When control returns to
the wrapper it returns the value to C. There is one assembler wrapper
per callback.[1] The SAP to the wrapper code vector is what is passed
to foreign code as a callback.

The Assembler Wrapper is generated by
@code{ALIEN-CALLBACK-ASSEMBLER-WRAPPER}.

@item #'ENTER-ALIEN-CALLBACK

pulls the Lisp function for the given index, and calls it with the
argument and result pointers.

@end itemize

[1] As assembler wrappers need to be allocated in static addresses and
are (in the current scheme of things) never released it might be worth
it to split it into two parts: per-callback trampoline that pushes the
index of the lisp trampoline on the stack, and jumps to the
appropriate assembler wrapper. The assembler wrapper could then be
shared between all the callbacks with the same alien-fun-type. This
would amortize most of the static allocation costs between multiple
callbacks.

When a foreign thread (i.e. a native thread created by foreign code
which doesn't yet have a Lisp thread structure associated with it) calls
a callback, the Lisp runtime must associate a Lisp thread structure to
it before entering Lisp.