common-lisp/sbcl.sbcl
1
0
Fork 0
mirror of git://git.code.sf.net/p/sbcl/sbcl synced 2026-09-10 07:26:40 -04:00
Code Issues Actions 8 Packages Projects Releases Wiki Activity

Check in some CMU CL documentation verbatim.

Browse source 
Our source code makes direct reference to some internals .tex files
found only in the CMU CL source, and even our own manual has direct
references to the CMU CL manual because the corresponding sections
haven't been written yet (for over 2 decades).

We might as well check-in the original documentation. I doubt anyone
will ever be paid to write or even finish porting such high quality
documentation, and it would be a shame to have 95% correct internals
documentation be completely missing as well.
This commit is contained in:
Charles Zhang 2022-04-13 01:47:59 -07:00
parent cde56f8af3
commit 60f84eda5c
54 changed files with 29549 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

25
doc/cmu-user/.cvsignore Normal file
View file

@ -0,0 +1,25 @@
*.aux
*.cdx
*.cnd
*.dvi
*.fdx
*.fnd
*.html
*.idx
*.ilg
*.log
*.out
*.pdf
*.tdx
*.tnd
*.toc
*.vdx
*.vnd
cmu-user-html.tgz
cmu-user-letter.ps
cmu-user.haux
cmu-user.hcnd
cmu-user.hfnd
cmu-user.htnd
cmu-user.htoc
cmu-user.hvnd

124
doc/cmu-user/Makefile Normal file
View file

@ -0,0 +1,124 @@
# -*- makefile -*-
#
#
# This Makefile contains rules for converting the LaTeX sources of the
# CMUCL User's Manual into various formats: Postscript, PDF, DVI, HTML
# and info. The useful targets are:
#
# make cmu-user.ps (A4 paper)
# make cmu-user-letter.ps (letter paper)
# make cmu-user.dvi
# make cmu-user.pdf
# make cmu-user.html
# make cmu-user.info
# make clean
# A number of addon LaTeX packages are used in the manual, to support
# features such as hyperlinks and multiple indexes. A current
# distribution of teTeX contains everything that is necessary to
# generate the Postscript, PDF and DVI formats. HTML output is
# generated using the Hevea tool, which is available from
# <URL:http://pauillac.inria.fr/~maranget/hevea/>. The HTML files are
# then split into sections using the hacha tool (distributed with
# hevea), and cleaned up using tidy. Note that hevea 1.10 produces
# HTML that looks a bit weird. However, hevea 1.06 works nicely.
#
# The generated DVI file should include clickable hyperlinks. The PDF
# output should include a hyperlinked table of contents, hyperlinked
# cross-references, and an index. The generated Postscript should use
# Postscript fonts that give good quality output at high resolutions.
SHELL = /bin/sh
LATEX ?= latex
BIBTEX ?= bibtex
PDFLATEX ?= pdflatex
HEVEA ?= hevea
HACHA ?= hacha
TIDY ?= tidy
FILES = *.tex
.SUFFIXES:
.SUFFIXES: .tex .dvi .ps .ps1 .pdf .html .info
all: cmu-user.pdf
# Runs LaTeX once, then reruns LaTeX as many times as necessary to get
# rid of the "undefined references" message, generates the indexes,
# the reruns LaTeX. The dependency on the .tex files means that the
# DVI file will be rebuilt only if one of the included LaTeX files has
# been modified.
%.dvi : %.tex $(FILES)
$(LATEX) $<
@while ( grep -q "Rerun to get cross" $*.log > /dev/null ); do \
$(LATEX) $<; \
done
if [ "$<" = "cmu-user.tex" ]; then $(MAKE) index; else $(MAKE) index-letter; fi
$(LATEX) $<
%.pdf : %.tex $(FILES)
$(PDFLATEX) $<
if [ "$<" = "cmu-user.tex" ]; then $(MAKE) index; else $(MAKE) index-letter; fi
@while ( grep 'Rerun to get cross' $*.log > /dev/null ); do \
$(PDFLATEX) $<; \
if [ "$<" = "cmu-user.tex" ]; then $(MAKE) index; else $(MAKE) index-letter; fi; \
done
# the "-fix" option to hevea makes it run as many times as necessary
# to resolve all cross-references and generate an index.
%.html : %.tex $(FILES) cmu-user.hva
$(HEVEA) -fix cmu-user.hva $<
$(HACHA) -tocbis $@
-$(TIDY) -m *.html
%.info : %.tex $(FILES)
$(HEVEA) -fix -info cmu-user.hva $<
%.ps1 : %.dvi
dvips -o $@ $<
# convert the Postscript file to duplex (will print double-sided if
# the printer supports it)
%.ps : %.ps1
if [ -x psset ]; then psset -d -o $@ $<; else cp $< $@; fi
# Tar up the html files. Note: cmucl.css is duplicated here, from the
# cmucl-www repository. Please remember to update cmucl.css here
# whenever cmucl.css changes in cmucl-www.
cmu-user-html.tgz : cmu-user.html
mkdir cmu-user
cp cmucl.css *.html *.gif cmu-user
tar cf - cmu-user | gzip > cmu-user-html.tgz
# generate Postscript for letter format, instead of for A4 paper
.INTERMEDIATE: cmu-user-letter.tex
cmu-user-letter.tex: cmu-user.tex
cp $< $@
perl -pi -e 's/documentclass\[a4paper\]/documentclass\[letter\]/' $@
clean:
rm -f *.log *.bbl *.blg *.ps *.pdf *.aux *.lof *.toc *.out *.ilg
rm -f *.vdx *.cdx *.tdx *.fdx *.idx *.cnd *.fnd *.tnd *.vnd *.haux
rm -f *.html *.hcnd *.htnd *.hvnd *.hfnd *.htoc
rm -f cmu-user.css
index:
makeindex cmu-user.tdx -o cmu-user.tnd
makeindex cmu-user.vdx -o cmu-user.vnd
makeindex cmu-user.fdx -o cmu-user.fnd
makeindex cmu-user.cdx -o cmu-user.cnd
index-letter:
makeindex cmu-user-letter.tdx -o cmu-user-letter.tnd
makeindex cmu-user-letter.vdx -o cmu-user-letter.vnd
makeindex cmu-user-letter.fdx -o cmu-user-letter.fnd
makeindex cmu-user-letter.cdx -o cmu-user-letter.cnd
.PHONY: clean index
# EOF

1163
doc/cmu-user/aliens.tex Normal file
View file

File diff suppressed because it is too large Load diff

460
doc/cmu-user/cmu-user.dict Normal file
View file

@ -0,0 +1,460 @@
'BAR
VARREF
'TEST
UPCASE
ENDLISP
SUBSEQ
ENDDEFUN
FUNARGS
GENSYM
VARS
UNINTERNED
VAR
VSOURCE
CLISP
COND
MYSTUFF
TRADEOFFS
PATHNAME
LLISP
CMUCL
REF
YETMOREKEYS
CLEANUP
ARGS
DEFUN
ZOQ
FOO
'S
CLTL
MACROEXPANDS
MACROEXPANSION
PROXY
ERRORFUL
EQ
ECASE
PYTHON
DEFMACRO
PROMISCUOUS
FLAMAGE
DEBUGGABILITY
FEATUREFULNESS
DEBUGGABLE
ENDDEFVAR
MACROEXPANDED
DEFVAR
ENDDEFMAC
KWD
MGROUP
MSTAR
DEFMAC
OFFS
NOTINLINE
TRADEOFF
FUNCALL
SOMEVAL
SOMEFUN
CM
DEFTYPE
CONSING
FIXNUMS
BIGNUMS
FROB
'FOO
RECOMPILES
FTYPE
TYPECASE
TYPEP
UNTYPED
UNIONED
GLOBALS
MODICUM
MACREF
SLEAZING
ES
STEELE
ETYPECASE
'EQL
'IDENTITY
'FUN
LOCALFUN
ISQRT
ODDP
MYFUN
POS
ZOW
YOW
'YOW
CADR
ZEROP
RES
EXPT
PARED
PUSHING
'ING
RPLACD
IOTA
NTHCDR
NTH
CADDDR
RPLACA
CADDR
FIENDS
SQRT
'SQRT
LISPY
BLANKSPACE
MYCHAPTER
UNENCAPSULATED
ENCAPSULATIONS
UNENCAPSULATE
UNTRACED
UNTRACE
EVALED
SPEC
PUSHES
TRUENAME
MYMAC
UNINFORMATIVE
FOOBAR
BAZ
BACKQUOTE
MALFORMED
MOREKEYS
FUNREF
QUIRKS
UNDILUTED
DISASSEMBLY
NAN
DENORMALIZED
ENDDEFCONST
DEFCONST
HASHTABLES
EFF
OBFUSCATING
SNOC
GRUE
GORP
FLO
NUM
VEC
MULTBY
SOMEOTHERFUN
'CHAR
NOTP
TESTP
FUNVAR
RAZ
ZUG
XFF
IO
GC'ING
EXT
MEGABYTE
SYS
UX
ED
MATCHMAKER
DIRED
PCL
CLOS
CONFORMANCE
ENDDEFCON
DEFCON
DECLAIM
DEFSTRUCT
ENUM
EXTERN
LOWERCASING
DEREFERENCED
MOPT
STRUCT
DEFTP
ENDDEFTP
MALLOC
CSH
PXLREF
ATYPE
CONSTRUCTUED
ANAME
PXREF
ENV
ONECOLUMN
TP
VR
FN
PRINTINDEX
UNNUMBERED
TWOCOLUMN
TLF
UNCOMPILED
DEACTIVATE
CALLABLE
UNREFERENCED
SUPPLIEDP
INTERNING
UNHANDLED
BACKTRACING
TEX
OOB
OBJ
PRIN
OBJS
GP
LINKERS
CC
AR
CFUN
INTS
SIZEOF
PRINTF
CFOO
SUBFORM
SVREF
STASH
FOOS
LC
LD
'N
'X
ERRNO
UPPERCASING
EXPR
ADDR
'STR
STR
DEREF
PTR
SWINDOW
IWINDOW
'SLIDER
DRAWABLE
'KEY
'EXT
TIMEOUTS
'MY
ID
PIXMAPS
'EQ
FUNCALLED
XWINDOW
'IH
SIGSTOP
GETPID
SIGTSTP
SCP
SIGINT
IH
CNT
GENERALRETURN
DEFMACX
'NUKEGARBAGE
GR
HASSLE
PREPENDS
TIMEOUT
FD
MSG
SYSCALL
UNHELPFUL
PREPENDED
VM
PAGEREF
INT
PORTSID
PORTSNAME
SERVPORT
KERN
DATATYPES
TTY
STDERR
STDOUT
STDIN
CMD
AUX
PS
UNACCOUNTED
RUNTIMES
PROFILER
UNPROFILE
REPROFILED
UNPROFILED
CF
ELT
VOPS
MAPCAR
OPTIONALS
CONSES
CONTORTIONS
ALISTS
ALIST
ASSOC
EXP
MYEXP
DEFCONSTANT
INCF
MEMQ
COERCIONS
EQL
LOGAND
AREF
CONSP
TYPEN
LOGIOR
EQUIV
SUPERTYPE
DEFMETHOD
SUBFORMS
CERROR
PSETQ
TAGBODY
DOTIMES
PLOQ
ROQ
SPECS
MPLUS
STEPPER
FDEFINITION
FUNCALLABLE
ST
BR
DB
LB
LL
HFILL
PP
VPRINT
TH
ARGLISTS
SETQ
NAMESPACE
SUBFUNCTION
BACKTRACE
'B
FLET
ARG
'A
CPSUBINDEX
PROGN
CONTRIB
WEEKDAYS
GREENWICH
TIMEZONE
DEST
WEEKDAY
JAN
CINDEX
NAMESTRING
PATHNAMES
FASL
SIGSEGV
PLIST
'ABLE
SETF
PID
EXECVE
DEV
SUBPROCESS
PTY
'TH
UNSUPPLIED
DEFVARX
GCS
CONSED
GC'ED
GC
TRASHING
XLIB
CL
HI
COMMONLOOPS
CTRL
XLREF
DEFUNX
DEFCONSTX
SUBSUBSECTION
VINDEXED
TINDEXED
RESEARCHCREDIT
EM
WHOLEY
SKEF
KAUFMANN
TODD
KOLOJEJCHICK
BUSDIECKER
''
NOINDENT
MOORE
TIM
LOTT
LEINEN
HALLGREN
GLEICHAUF
DUNNING
TED
BADER
MYLISP
NOINIT
FINDEXED
INIT
EVAL
SUBDIRECTORIES
COPYRIGHTED
FTP
LANG
COMP
MEG
MEGABYTES
UNCOMPRESS
CD
OS
USERNAME
SLISP
RT
LIB
SETENV
SAMP
SETPATH
LOGIN
MISC
USR
MODMISC
TXT
DOC
EXECUTABLES
PERQ
UNTAGGED
BENCHMARKING
WINDOWING
INTRO
DOCS
EDU
AFS
VSPACE
IFINFO
DIR
SETFILENAME
TABLEOFCONTENTS
PAGENUMBERING
CLEARPAGE
MAKETITLE
ARPASUPPORT
CITATIONINFO
TRNUMBER
IFTEX
SUNOS
SPARC
DECSTATIONS
THEABSTRACT
DEF
KY
CP
NEWINDEX
ALWAYSREFILL
PAGESTYLE
CMULISP
TITLEPAGE
ELISP
LATEXINFO
DOCUMENTSTYLE

189
doc/cmu-user/cmu-user.hva Normal file
View file

@ -0,0 +1,189 @@
% -*- tex -*-
%
% This file contains macros which are used when translating
% to HTML using Hevea. By Luc Maranget and Eric Marsden.
\newif\ifhmode\hmodefalse
\newcommand{\pdfinfo}[1]{\@print{<!--} #1 \@print{-->}}
\let\hbox\mbox
\def\hfil{}
\def\hfill{}
\newcommand{\parbox}[3][]{\mbox{#3}}
\newcommand{\lengthtest}[1]{true}
\newenvironment{minipage}[2][]{}{}
\newcommand{\layout}[5][]
{\@open{DIV}{align=left}#5%
[#4] \\
\begin{tabbing}
\fcnname{#1}\fcnname{#2} #3 \@print{&nbsp;&nbsp;&nbsp;}
\end{tabbing}
\@close{DIV}}
\newcommand{\keys}[1]{\code{\&key}\@print{&nbsp;}\= #1}
\renewcommand{\textgreater}{\@print{&gt;}}
\renewcommand{\textless}{\@print{&lt;}}
% \newcommand{\layout}[4][]
% {\begin{tabular}
% {llp{10cm}@{\hspace{10cm}}r}\fcnname{#1} &
% \fcnname{#2} &
% \begin{flushleft}#3\end{flushleft} &
% [#4]
% \end{tabular}}
\newcommand{\fcntype}[1]{\@styleattr{TT}{class=function-type}#1\@clearstyle}
\newcommand{\argtype}[1]{\@styleattr{TT}{class=argument-type}#1\@clearstyle}
\newcommand{\fcnname}[1]{\@styleattr{TT}{class=function-name}#1\@clearstyle}
\newcommand{\var}[1]{\@styleattr{TT}{class=variable}#1\@clearstyle}
\newcommand{\code}[1]{\@styleattr{TT}{class=code}#1\@clearstyle}
\newcommand{\file}[1]{\@styleattr{TT}{class=filename}#1\@clearstyle}
%% Define a new type
%%
%% \begin{deftp}{typeclass}{typename}{args}
%% some description
%% \end{deftp}
\newenvironment{deftp}[3]
{\layout{#2}{\var{#3}}{#1}%
{\bigskip\index[types]{#2|textbf}}
\begin{quote}}
{\end{quote}}
%% Define a function
%%
%% \begin{defun}[suffix]{pkg}{name}{params}
%% \defunx[pkg]{name}{params}
%% description of function
%% \end{defun}
\newenvironment{defun}[4][]
{\layout[#2]{#3}{#4}{Function}%
{\defunvspace\fnindexbold{#3}\label{FN:#3#1}}
\begin{quote}}
{\end{quote}}
\newcommand{\defunx}[3][\mbox{}]{%
\layout[#1]{#2}{#3}{Function}%
{\fnindexbold{#2}\label{FN:#2}}}
%% Define a macro
%%
%% \begin{defmac}[suffix]{pkg}{name}{params}
%% \defmacx[pkg]{name}{params}
%% description of macro
%% \end{defmac}
\newenvironment{defmac}[4][]{%
\layout[#2]{#3}{#4}{Macro}%
{\defunvspace\fnindexbold{#3}\label{FN:#3#1}}
\begin{quote}}
{\end{quote}}
\newcommand{\defmacx}[3][\mbox{}]{%
\layout[#1]{#2}{#3}{Function}%
{\fnindexbold{#2}\label{FN:#2}}}
%% Define a variable
%%
%% \begin{defvar}{pkg}{name}
%% \defvarx[pkg]{name}
%% description of defvar
%% \end{defvar}
\newenvironment{defvar}[2]{%
\layout[#1]{*#2*}{}{Variable}%
{\defunvspace\vrindexbold{#2}\label{VR:#2}}
\begin{quote}}
{\end{quote}}
\newcommand{\defvarx}[2][\mbox{}]{%
\layout[#1]{*#2*}{}{Variable}%
{\vrindexbold{#2}\label{VR:#2}}}
%% Define a constant
%%
%% \begin{defconst}{pkg}{name}
%% \defconstx[pkg]{name}
%% description of defconst
%% \end{defconst}
\newenvironment{defconst}[2]{%
\layout[#1]{#2}{}{Constant}%
{\defunvspace}
\begin{quote}}
{\end{quote}}
\newcommand{\defconstx}[2][\mbox{}]{%
\layout[#1]{#2}{}{Constant}{}}
% \@hr[NOSHADE]{.8\linewidth}{}
\newenvironment{example}
{\@open{BLOCKQUOTE}{class=example}\begin{alltt}}
{\end{alltt}\@close{BLOCKQUOTE}}
\newenvironment{lisp}
{\@open{BLOCKQUOTE}{CLASS=lisp}\begin{alltt}}
{\end{alltt}\@close{BLOCKQUOTE}}
\newcommand{\cmucl}{{\scshape cmucl}}
% there are no page numbers in HTML, so we need to override certain
% cross-referencing macros
\newcommand{\pxlref}[1]{see section~\ref{#1}}
\newcommand{\xlref}[1]{See section~\ref{#1}}
\newcommand{\funref}[1]{\findexed{#1}}
\newcommand{\specref}[1]{\findexed{#1}}
\newcommand{\macref}[1]{\findexed{#1}}
\newcommand{\varref}[1]{\vindexed{#1}}
\newcommand{\conref}[1]{\conindexed{#1}}
\newcommand{\ctrl}[1]{\code{Ctrl-#1}}
\newcommand{\result}{==>}
\newcommand{\myequiv}{<==>}
\newcommand{\mopt}[1]{\code{\{#1\}}}
\newcommand{\mstar}[1]{\code{\{#1\}}$^*$}
\newcommand{\mplus}[1]{\code{\{#1\}}$^+$}
\newcommand{\mgroup}[1]{\code{\{#1\}}}
% disable ``this document generated by'' footer
\def\@footer{}
% \htmlhead{\@print{
% <a href="../../">CMUCL</a> -&gt
% <a href="../">Documentation</a> <br>}
% \@hr[NOSHADE]{.8\linewidth}{}\@print{<br>}}
% \htmlfoot{\@hr[NOSHADE]{.99\linewidth}{}
% \copyright{} 1995-2003 CMUCL Project}
\let\orig@meta=\@meta
\renewcommand{\@meta}{%
\orig@meta
\begin{rawhtml}
<link rel="stylesheet" href="cmucl.css" type="text/css">
<meta http-equiv="Content-Language" content="en">
\end{rawhtml}}
% when generating HTML, produce file names based on the chapter name
% instead of machine-generated numbers
\let\orig@input\input
\renewcommand{\input}[1]{\orig@input{#1}\cutname{#1.html}}
\htmlprefix{CMUCL User's Manual: }
% generate HTML with section headers on a blue background
\setcounter{cuttingdepth}{10}
\input{report.hva}
%%\input{fancysection.hva}
\usepackage{color}
\definecolor{chapter}{rgb}{1, 1, 0.74}
\definecolor{part}{rgb}{1, 1, 0.81}
\definecolor{section}{rgb}{1, 1, 0.83}
\definecolor{subsection}{rgb}{1, 1, 0.89}
\definecolor{subsubsection}{rgb}{1, 1, 0.93}
\newstyle{.chapter}{padding:0.5ex;background-color:\@getstylecolor{chapter}}
\newstyle{.section} {padding:.5ex;background-color:\@getstylecolor{section}}%
\newstyle{.subsection} {padding:0.3ex;background-color:\@getstylecolor{subsection}}%
\newstyle{.subsubsection} {padding:0.5ex;background-color:\@getstylecolor{subsubsection}}%
\newstyle{.paragraph} {padding:0.5ex;background-color:\@getstylecolor{paragraph}}%
\renewcommand{\@bodyargs}{}
% EOF

146
doc/cmu-user/cmu-user.tex Normal file
View file

@ -0,0 +1,146 @@
%% cmu-user.tex --- CMUCL User's Manual
%%
%% 2001-04-05 Eric Marsden
%% Modifications to work with hevea and pdflatex.
%%
%% Aug 1997 Raymond Toy
%% This is a modified version of the original CMUCL User's Manual.
%% The key changes are modification of this file to use standard
%% LaTeX2e. This means latexinfo isn't going to work anymore.
%% However, Latex2html support has been added.
%%
%% Jan 1998 Paul Werkowski
%% A few of the packages below are not part of the standard LaTeX2e
%% distribution, and must be obtained from a repository. At this time
%% I was able to fetch from
%% ftp.cdrom.com:pub/tex/ctan/macros/latex/contrib/supported/
%% camel/index.ins
%% camel/index.dtx
%% calc/calc.ins
%% calc/calc.dtx
%% changebar/changebar.ins
%% changebar/changebar.dtx
%% One runs latex on the .ins file to produce .tex and/or .sty
%% files that must be put in a path searched by latex.
%%
%% Note all of the required packages are included in the teTeX distribution,
%% and a current version of latex2html can be obtained from:
%% http://saftsack.fs.uni-bayreuth.de/~latex2ht/
%% Delete "[a4paper]" if you don't want this formatted for A4 paper.
\documentclass[a4paper]{report}
\usepackage{xspace}
\usepackage{alltt}
\usepackage{index}
\usepackage{ifpdf}
\usepackage{ifthen}
\usepackage{calc}
\usepackage{sectsty}
\usepackage{varioref}
\usepackage[hyperindex=false,colorlinks=false,urlcolor=blue]{hyperref}
%% \usepackage{html}
\usepackage{typehtml}
\input{macros}
\title{CMUCL User's Manual}
\author{Robert A. MacLachlan, \textit{Editor}}
\newcommand{\keywords}{lisp, Common Lisp, manual, compiler, programming
language implementation, programming environment}
\date{October 2017 \\ 21c}
\begin{document}
\begin{titlepage}
\makeatletter
\vspace{60pt}
\begin{center}
\rule{\linewidth}{0.7mm}
\vspace{3em}
{\Huge \@title \par}
\vspace{4em}
{\large
\begin{tabular}[t]{c}
\@author
\end{tabular}\par}
\vspace{2em}
{\large \@date \par}
\vspace{2em}
\rule{\linewidth}{0.7mm}
\end{center}
\vfill
\begin{quotation}
\cmucl{} is a free, high-performance implementation of the Common Lisp
programming language, which runs on most major Unix platforms. It
mainly conforms to the ANSI Common Lisp Standard. \cmucl{} features a
sophisticated native-code compiler, a foreign function interface, a
graphical source-level debugger, an interface to the X11 Window
System, and an Emacs-like editor.
\medskip \textbf{Keywords}: \keywords
\end{quotation}
\vspace{5cm}
This manual is based on CMU Technical Report CMU-CS-92-161, edited by
Robert A. MacLachlan, dated July 1992.
\thispagestyle{empty}
\makeatother
\end{titlepage}
\ifpdf
\pdfinfo{
/Author (Robert A. MacLachlan, ed)
/Title (CMUCL User's Manual)
/Keywords (\keywords)
}
% Add section numbers to the bookmarks, and open 2 levels by default.
\hypersetup{bookmarksnumbered=true,
bookmarksopen=true,
bookmarksopenlevel=2}
\fi
% \maketitle
\pagestyle{headings}
\pagenumbering{roman}
\tableofcontents
\clearpage
\pagenumbering{arabic}
\input{introduction}
\input{extensions}
\input{debugger}
\input{compiler}
\input{compiler-hint}
\input{unix}
\input{serve-event}
\input{aliens}
\input{ipc}
\input{internet}
\input{debug-internals}
\input{cross-referencing}
\input{unicode}
\twocolumn
\cindex{Function Index}
\printindex[funs]
\twocolumn
\cindex{Variable Index}
\printindex[vars]
\twocolumn
\cindex{Type Index}
\printindex[types]
\onecolumn
\cindex{Concept Index}
\printindex[concept]
\end{document}

85
doc/cmu-user/cmucl.css Normal file
View file

@ -0,0 +1,85 @@
html {
color: black;
background-color: white;
margin-left: 4%;
margin-right: 2%;
}
h1 {
text-align: center;
font-family: Helvetica, Arial, sans-serif
}
h2 {
color: #515e4b;
font-family: Helvetica, Arial, sans-serif
}
h3 {
font-family: Helvetica, Arial, sans-serif;
}
.lisp {
background: #aaaaaa;
border:solid #aaaaaa 0.1px;
white-space:pre;
font-family: sans-serif;
}
.example {
background: #cccccc;
border:solid #cccccc 0.1px;
white-space:pre;
}
.code {
background: #faf0e6;
border:solid #faf0e6 1px;
white-space:pre-line;
}
.technical {
background: #dddddd;
border: solid black 1px;
padding-left: 1em;
padding-right: 1em;
padding-top: 1ex;
padding-bottom: 1ex;
font-size: smaller;
font-stretch: semi-condensed;
margin-left: 10%;
}
.credits {
background: #eeeeee;
font-stretch: condensed;
font-size: 0.8em;
font-family: Helvetica, Arial, sans-serif;
text-align: right;
}
.function-name {
color: rgb(25%,0%,0%);
}
.variable {
color: rgb(0%,25%,0%);
}
A:link
{
text-decoration: none
}
A:active
{
text-decoration: none
}
A:visited
{
text-decoration: none
}
A:hover
{
text-decoration: underline
}

4138
doc/cmu-user/compiler-hint.tex Normal file
View file

File diff suppressed because it is too large Load diff

1251
doc/cmu-user/compiler.tex Normal file
View file

File diff suppressed because it is too large Load diff

BIN
doc/cmu-user/contents_motif.gif Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 316 B

297
doc/cmu-user/cross-referencing.tex Normal file
View file

@ -0,0 +1,297 @@
\chapter{Cross-Referencing Facility}
\label{xref}
\cindex{cross-referencing}
\credits{by Eric Marsden}
The \cmucl{} cross-referencing facility (abbreviated XREF) assists in
the analysis of static dependency relationships in a program. It
provides introspection capabilities such as the ability to know which
functions may call a given function, and the program contexts in which
a particular global variable is used. The compiler populates a
database of cross-reference information, which can be queried by the
user to know:
\begin{itemize}
\item
the list of program contexts (functions, macros, top-level forms)
where a given function may be called at runtime, either directly or
indirectly (via its function-object);
\item
the list of program contexts where a given global variable may be
read;
\item
the list of program contexts that bind a global variable;
\item
the list of program contexts where a given global variable may be
modified during the execution of the program.
\end{itemize}
A global variable is either a dynamic variable or a constant variable,
for instance declared using \code{defvar} or \code{defparameter} or
\code{defconstant}.
\section{Populating the cross-reference database}
\begin{defvar}{c:}{record-xref-info}
When non-NIL, code that is compiled (either using
\code{compile-file}, or by calling \code{compile} from the
listener), will be analyzed for cross-references. Defaults to
\nil{}.
\end{defvar}
Cross-referencing information is only generated by the compiler; the
interpreter does not populate the cross-reference database. XREF
analysis is independent of whether the compiler is generating native
code or byte code, and of whether it is compiling from a file, from a
stream, or is invoked interactively from the listener.
Alternatively, the \kwd{:xref} option to \code{compile-file} may be
specified to populate the cross-reference database when compiling a
file. In this case, loading the generated fasl file in a fresh lisp
will also populate the cross-reference database.
\begin{defun}{xref:}{init-xref-database}{}
Reinitializes the database of cross-references. This can be used to
reclaim the space occupied by the database contents, or to discard
stale cross-reference information.
\end{defun}
\section{Querying the cross-reference database}
\cmucl{} provides a number of functions in the XREF package that may
be used to query the cross-reference database:
\begin{defun}{xref:}{who-calls}{\args \var{function}}
Returns the list of xref-contexts where \var{function} (either a
symbol that names a function, or a function object) may be called
at runtime. XREF does not record calls to macro-functions (such as
\code{defun}) or to special forms (such as \code{eval-when}).
\end{defun}
\begin{defun}{xref:}{who-references}{\args \var{global-variable}}
Returns the list of program contexts that may reference
\var{global-variable}.
\end{defun}
\begin{defun}{xref:}{who-binds}{\args \var{global-variable}}
Returns a list of program contexts where the specified global
variable may be bound at runtime (for example using \code{LET}).
\end{defun}
\begin{defun}{xref:}{who-sets}{\args \var{global-variable}}
Returns a list of program contexts where the given global variable
may be modified at runtime (for example using \code{SETQ}).
\end{defun}
An \textit{xref-context} is the originating site of a cross-reference.
It identifies a portion of a program, and is defined by an
\code{xref-context} structure, that comprises a name, a source file and a
source-path.
\begin{defun}{xref:}{xref-context-name}{\args \var{context}}
Returns the name slot of an xref-context, which is one of:
\begin{itemize}
\item
a global function, which is named by a symbol or by a list of the form
\code{(setf\ foo)}.
\item
a macro, named by a list \verb|(:macro foo)|.
\item
an inner function (\code{flet}, \code{labels}, or anonymous lambdas) that
is named by a list of the form \code{(:internal outer inner)}.
\item
a method, named by a list of the form
\verb|(:method foo (specializer1 specializer2)|.
\item
a string \verb|"Top-Level Form"| that identifies a reference from a
top-level form. Note that multiple references from top-level forms
will only be listed once.
\item
a compiler-macro, named by a string of the form
\verb|(:compiler-macro foo)|.
\item
a string such as \verb|"DEFSTRUCT FOO"|, identifying a reference from
within a structure accessor or constructor or copier.
\item
a string such as
\begin{verbatim}
"Creation Form for #<KERNEL::CLASS-CELL STRUCT-FOO>"
\end{verbatim}
\item
a string such as \verb|"defun foo"|, or \verb|"defmethod bar (t)"|,
that identifies a reference from within code that has been generated
by the compiler for that form. For example, the compilation of a
\code{defclass} form causes accessor functions to be generated by the
compiler; this code is compiler-generated (it does not appear in the
source file), and so is identified by the XREF facility by a string.
\end{itemize}
\end{defun}
\begin{defun}{xref:}{xref-context-file}{\var{context}}
Return the truename (in the sense of the variable
\vindexed{compile-file-truename}) of the source file from which the
referencing forms were compiled. This slot will be \nil{} if the
code was compiled from a stream, or interactively from the
listener.
\end{defun}
\begin{defun}{xref:}{xref-context-source-path}{\var{context}}
Return a list of positive integers identifying the form that
contains the cross-reference. The first integer in the source-path
is the number of the top-level form containing the cross-reference
(for example, 2 identifies the second top-level form in the source
file). The second integer in the source-path identifies the form
within this top-level form that contains the cross-reference, and so
on. This function will always return \nil{} if the file slot of an
xref-context is \nil{}.
% While walking the top-level form, count one in depth-first order for
% each subform that is a cons.
\end{defun}
\section{Example usage}
In this section, we will illustrate use of the XREF facility on a
number of simple examples.
Consider the following program fragment, that defines a global
variable and a function.
\begin{verbatim}
(defvar *variable-one* 42)
(defun function-one (x)
(princ (* x *variable-one*)))
\end{verbatim}
We save this code in a file named \code{example.lisp}, enable
cross-referencing, clear any previous cross-reference information,
compile the file, and can then query the cross-reference database
(output has been modified for readability).
\begin{verbatim}
USER> (setf c:*record-xref-info* t)
USER> (xref:init-xref-database)
USER> (compile-file "example")
USER> (xref:who-calls 'princ)
(#<xref-context function-one in #p"example.lisp">)
USER> (xref:who-references '*variable-one*)
(#<xref-context function-one in #p"example.lisp">)
\end{verbatim}
From this example, we see that the compiler has noted the call to the
global function \code{princ} in \code{function-one}, and the reference
to the global variable \code{*variable-one*}.
Suppose that we add the following code to the previous file.
\begin{verbatim}
(defconstant +constant-one+ 1)
(defstruct struct-one
slot-one
(slot-two +constant-one+ :type integer)
(slot-three 42 :read-only t))
(defmacro with-different-one (&body body)
`(let ((*variable-one* 666))
,@body))
(defun get-variable-one () *variable-one*)
(defun (setf get-variable-one) (new-value)
(setq *variable-one* new-value))
\end{verbatim}
In the following example, we detect references x and y.
% FIXME add function with LABELS, a binding, a set
The following function illustrates the effect that various forms of
optimization carried out by the \cmucl{} compiler can have on the
cross-references that are reported for a particular program. The
compiler is able to detect that the evaluated condition is always
false, and that the first clause of the \code{if} will never be taken
(this optimization is called dead-code elimination). XREF will
therefore not register a call to the function \code{sin} from the
function \code{foo}. Likewise, no calls to the functions \code{sqrt}
and \code{\textless} are registered, because the compiler has eliminated the
code that evaluates the condition. Finally, no call to the function
\code{expt} is generated, because the compiler was able to evaluate
the result of the expression \code{(expt 3 2)} at compile-time (though
a process called constant-folding).
\begin{verbatim}
;; zero call references are registered for this function!
(defun constantly-nine (x)
(if (< (sqrt x) 0)
(sin x)
(expt 3 2)))
\end{verbatim}
\section{Limitations of the cross-referencing facility}
No cross-reference information is available for interpreted functions.
The cross-referencing database is not persistent: unless you save an
image using \code{save-lisp}, the database will be empty each time
\cmucl{} is restarted. There is no mechanism that saves
cross-reference information in FASL files, so loading a system from
compiled code will not populate the cross-reference database. The XREF
database currently accumulates ``stale'' information: when compiling a
file, it does not delete any cross-references that may have previously
been generated for that file. This latter limitation will be removed
in a future release.
The cross-referencing facility is only able to analyze the static
dependencies in a program; it does not provide any information about
runtime (dynamic) dependencies. For instance, XREF is able to identify
the list of program contexts where a given function may be called, but
is not able to determine which contexts will be activated when the
program is executed with a specific set of input parameters. However,
the static analysis that is performed by the \cmucl{} compiler does
allow XREF to provide more information than would be available from a
mere syntactic analysis of a program. References that occur from
within unreachable code will not be displayed by XREF, because the
\cmucl{} compiler deletes dead code before cross-references are
analyzed. Certain ``trivial'' function calls (where the result of the
function call can be evaluated at compile-time) may be eliminated by
optimizations carried out by the compiler; see the example below.
If you examine the entire database of cross-reference information (by
accessing undocumented internals of the XREF package), you will note
that XREF notes ``bogus'' cross-references to function calls that are
inserted by the compiler. For example, in safe code, the \cmucl{}
compiler inserts a call to an internal function called
\code{c::\%verify-argument-count}, so that the number of arguments
passed to the function is checked each time it is called. The XREF
facility does not distinguish between user code and these forms that
are introduced during compilation. This limitation should not be
visible if you use the documented functions in the XREF package.
As of the 18e release of \cmucl{}, the cross-referencing facility is
experimental; expect details of its implementation to change in future
releases. In particular, the names given to CLOS methods and to inner
functions will change in future releases.

765
doc/cmu-user/debug-internals.tex Normal file
View file

@ -0,0 +1,765 @@
\chapter{Debugger Programmer's Interface}
\label{debug-internals}
The debugger programmers interface is exported from from the
\code{DEBUG-INTERNALS} or \code{DI} package. This is a CMU
extension that allows debugging tools to be written without detailed
knowledge of the compiler or run-time system.
Some of the interface routines take a code-location as an argument. As
described in the section on code-locations, some code-locations are
unknown. When a function calls for a \var{basic-code-location}, it
takes either type, but when it specifically names the argument
\var{code-location}, the routine will signal an error if you give it an
unknown code-location.
\section{DI Exceptional Conditions}
Some of these operations fail depending on the availability debugging
information. In the most severe case, when someone saved a Lisp image
stripping all debugging data structures, no operations are valid. In
this case, even backtracing and finding frames is impossible. Some
interfaces can simply return values indicating the lack of information,
or their return values are naturally meaningful in light missing data.
Other routines, as documented below, will signal
\code{serious-condition}s when they discover awkward situations. This
interface does not provide for programs to detect these situations other
than by calling a routine that detects them and signals a condition.
These are serious-conditions because the program using the interface
must handle them before it can correctly continue execution. These
debugging conditions are not errors since it is no fault of the
programmers that the conditions occur.
\subsection{Debug-conditions}
The debug internals interface signals conditions when it can't adhere
to its contract. These are serious-conditions because the program
using the interface must handle them before it can correctly continue
execution. These debugging conditions are not errors since it is no
fault of the programmers that the conditions occur. The interface
does not provide for programs to detect these situations other than
calling a routine that detects them and signals a condition.
\begin{deftp}{Condition}{debug-condition}{}
This condition inherits from serious-condition, and all debug-conditions
inherit from this. These must be handled, but they are not programmer errors.
\end{deftp}
\begin{deftp}{Condition}{no-debug-info}{}
This condition indicates there is absolutely no debugging information
available.
\end{deftp}
\begin{deftp}{Condition}{no-debug-function-returns}{}
This condition indicates the system cannot return values from a frame since
its debug-function lacks debug information details about returning values.
\end{deftp}
\begin{deftp}{Condition}{no-debug-blocks}{}
This condition indicates that a function was not compiled with debug-block
information, but this information is necessary necessary for some requested
operation.
\end{deftp}
\begin{deftp}{Condition}{no-debug-variables}{}
Similar to \code{no-debug-blocks}, except that variable information was
requested.
\end{deftp}
\begin{deftp}{Condition}{lambda-list-unavailable}{}
Similar to \code{no-debug-blocks}, except that lambda list information was
requested.
\end{deftp}
\begin{deftp}{Condition}{invalid-value}{}
This condition indicates a debug-variable has \kwd{invalid} or \kwd{unknown}
value in a particular frame.
\end{deftp}
\begin{deftp}{Condition}{ambiguous-variable-name}{}
This condition indicates a user supplied debug-variable name identifies more
than one valid variable in a particular frame.
\end{deftp}
\subsection{Debug-errors}
These are programmer errors resulting from misuse of the debugging tools'
programmers' interface. You could have avoided an occurrence of one of these
by using some routine to check the use of the routine generating the error.
\begin{deftp}{Condition}{debug-error}{}
This condition inherits from error, and all user programming errors inherit
from this condition.
\end{deftp}
\begin{deftp}{Condition}{unhandled-condition}{}
This error results from a signalled \code{debug-condition} occurring
without anyone handling it.
\end{deftp}
\begin{deftp}{Condition}{unknown-code-location}{}
This error indicates the invalid use of an unknown-code-location.
\end{deftp}
\begin{deftp}{Condition}{unknown-debug-variable}{}
This error indicates an attempt to use a debug-variable in conjunction with an
inappropriate debug-function; for example, checking the variable's validity
using a code-location in the wrong debug-function will signal this error.
\end{deftp}
\begin{deftp}{Condition}{frame-function-mismatch}{}
This error indicates you called a function returned by
\code{preprocess-for-eval}
on a frame other than the one for which the function had been prepared.
\end{deftp}
\section{Debug-variables}
Debug-variables represent the constant information about where the system
stores argument and local variable values. The system uniquely identifies with
an integer every instance of a variable with a particular name and package. To
access a value, you must supply the frame along with the debug-variable since
these are particular to a function, not every instance of a variable on the
stack.
\begin{defun}{}{debug-variable-name}{\args{\var{debug-variable}}}
This function returns the name of the \var{debug-variable}. The
name is the name of the symbol used as an identifier when writing
the code.
\end{defun}
\begin{defun}{}{debug-variable-package}{\args{\var{debug-variable}}}
This function returns the package name of the \var{debug-variable}.
This is the package name of the symbol used as an identifier when
writing the code.
\end{defun}
\begin{defun}{}{debug-variable-symbol}{\args{\var{debug-variable}}}
This function returns the symbol from interning
\code{debug-variable-name} in the package named by
\code{debug-variable-package}.
\end{defun}
\begin{defun}{}{debug-variable-id}{\args{\var{debug-variable}}}
This function returns the integer that makes \var{debug-variable}'s
name and package name unique with respect to other
\var{debug-variable}'s in the same function.
\end{defun}
\begin{defun}{}{debug-variable-validity}{%
\args{\var{debug-variable} \var{basic-code-location}}}
This function returns three values reflecting the validity of
\var{debug-variable}'s value at \var{basic-code-location}:
\begin{Lentry}
\item[\kwd{valid}] The value is known to be available.
\item[\kwd{invalid}] The value is known to be unavailable.
\item[\kwd{unknown}] The value's availability is unknown.
\end{Lentry}
\end{defun}
\begin{defun}{}{debug-variable-value}{\args{\var{debug-variable}
\var{frame}}}
This function returns the value stored for \var{debug-variable} in
\var{frame}. The value may be invalid. This is \code{SETF}'able.
\end{defun}
\begin{defun}{}{debug-variable-valid-value}{%
\args{\var{debug-variable} \var{frame}}}
This function returns the value stored for \var{debug-variable} in
\var{frame}. If the value is not \kwd{valid}, then this signals an
\code{invalid-value} error.
\end{defun}
\section{Frames}
Frames describe a particular call on the stack for a particular thread. This
is the environment for name resolution, getting arguments and locals, and
returning values. The stack conceptually grows up, so the top of the stack is
the most recently called function.
\code{top-frame}, \code{frame-down}, \code{frame-up}, and
\code{frame-debug-function} can only fail when there is absolutely no
debug information available. This can only happen when someone saved a
Lisp image specifying that the system dump all debugging data.
\begin{defun}{}{top-frame}{}
This function never returns the frame for itself, always the frame
before calling \code{top-frame}.
\end{defun}
\begin{defun}{}{frame-down}{\args{\var{frame}}}
This returns the frame immediately below \var{frame} on the stack.
When \var{frame} is the bottom of the stack, this returns \nil.
\end{defun}
\begin{defun}{}{frame-up}{\args{\var{frame}}}
This returns the frame immediately above \var{frame} on the stack.
When \var{frame} is the top of the stack, this returns \nil.
\end{defun}
\begin{defun}{}{frame-debug-function}{\args{\var{frame}}}
This function returns the debug-function for the function whose call
\var{frame} represents.
\end{defun}
\begin{defun}{}{frame-code-location}{\args{\var{frame}}}
This function returns the code-location where \var{frame}'s
debug-function will continue running when program execution returns
to \var{frame}. If someone interrupted this frame, the result could
be an unknown code-location.
\end{defun}
\begin{defun}{}{frame-catches}{\args{\var{frame}}}
This function returns an a-list for all active catches in
\var{frame} mapping catch tags to the code-locations at which the
catch re-enters.
\end{defun}
\begin{defun}{}{eval-in-frame}{\args{\var{frame} \var{form}}}
This evaluates \var{form} in \var{frame}'s environment. This can
signal several different debug-conditions since its success relies
on a variety of inexact debug information: \code{invalid-value},
\code{ambiguous-variable-name}, \code{frame-function-mismatch}. See
also \funref{preprocess-for-eval}.
\end{defun}
% \begin{defun}{}{return-from-frame}{\args{\var{frame} \var{values}}}
%
% This returns the elements in the list \var{values} as multiple
% values from \var{frame} as if the function \var{frame} represents
% returned these values. This signals a
% \code{no-debug-function-returns} condition when \var{frame}'s
% debug-function lacks information on returning values.
%
% \i{Not Yet Implemented}
% \end{defun}
\section {Debug-functions}
Debug-functions represent the static information about a function determined at
compile time---argument and variable storage, their lifetime information,
etc. The debug-function also contains all the debug-blocks representing
basic-blocks of code, and these contains information about specific
code-locations in a debug-function.
\begin{defmac}{}{do-debug-function-blocks}{%
\args{(\var{block-var} \var{debug-function} \mopt{result-form})
\mstar{form}}}
This executes the forms in a context with \var{block-var} bound to
each debug-block in \var{debug-function} successively.
\var{Result-form} is an optional form to execute for a return value,
and \code{do-debug-function-blocks} returns \nil if there is no
\var{result-form}. This signals a \code{no-debug-blocks} condition
when the \var{debug-function} lacks debug-block information.
\end{defmac}
\begin{defun}{}{debug-function-lambda-list}{\args{\var{debug-function}}}
This function returns a list representing the lambda-list for
\var{debug-function}. The list has the following structure:
\begin{example}
(required-var1 required-var2
...
(:optional var3 suppliedp-var4)
(:optional var5)
...
(:rest var6) (:rest var7)
...
(:keyword keyword-symbol var8 suppliedp-var9)
(:keyword keyword-symbol var10)
...
)
\end{example}
Each \code{var}\var{n} is a debug-variable; however, the symbol
\kwd{deleted} appears instead whenever the argument remains
unreferenced throughout \var{debug-function}.
If there is no lambda-list information, this signals a
\code{lambda-list-unavailable} condition.
\end{defun}
\begin{defmac}{}{do-debug-function-variables}{%
\args{(\var{var} \var{debug-function} \mopt{result})
\mstar{form}}}
This macro executes each \var{form} in a context with \var{var}
bound to each debug-variable in \var{debug-function}. This returns
the value of executing \var{result} (defaults to \nil). This may
iterate over only some of \var{debug-function}'s variables or none
depending on debug policy; for example, possibly the compilation
only preserved argument information.
\end{defmac}
\begin{defun}{}{debug-variable-info-available}{\args{\var{debug-function}}}
This function returns whether there is any variable information for
\var{debug-function}. This is useful for distinguishing whether
there were no locals in a function or whether there was no variable
information. For example, if \code{do-debug-function-variables}
executes its forms zero times, then you can use this function to
determine the reason.
\end{defun}
\begin{defun}{}{debug-function-symbol-variables}{%
\args{\var{debug-function} \var{symbol}}}
This function returns a list of debug-variables in
\var{debug-function} having the same name and package as
\var{symbol}. If \var{symbol} is uninterned, then this returns a
list of debug-variables without package names and with the same name
as \var{symbol}. The result of this function is limited to the
availability of variable information in \var{debug-function}; for
example, possibly \var{debug-function} only knows about its
arguments.
\end{defun}
\begin{defun}{}{ambiguous-debug-variables}{%
\args{\var{debug-function} \var{name-prefix-string}}}
This function returns a list of debug-variables in
\var{debug-function} whose names contain \var{name-prefix-string} as
an initial substring. The result of this function is limited to the
availability of variable information in \var{debug-function}; for
example, possibly \var{debug-function} only knows about its
arguments.
\end{defun}
\begin{defun}{}{preprocess-for-eval}{%
\args{\var{form} \var{basic-code-location}}}
This function returns a function of one argument that evaluates
\var{form} in the lexical context of \var{basic-code-location}.
This allows efficient repeated evaluation of \var{form} at a certain
place in a function which could be useful for conditional breaking.
This signals a \code{no-debug-variables} condition when the
code-location's debug-function has no debug-variable information
available. The returned function takes a frame as an argument. See
also \funref{eval-in-frame}.
\end{defun}
\begin{defun}{}{function-debug-function}{\args{\var{function}}}
This function returns a debug-function that represents debug
information for \var{function}.
\end{defun}
\begin{defun}{}{debug-function-kind}{\args{\var{debug-function}}}
This function returns the kind of function \var{debug-function}
represents. The value is one of the following:
\begin{Lentry}
\item[\kwd{optional}] This kind of function is an entry point to an
ordinary function. It handles optional defaulting, parsing
keywords, etc.
\item[\kwd{external}] This kind of function is an entry point to an
ordinary function. It checks argument values and count and calls
the defined function.
\item[\kwd{top-level}] This kind of function executes one or more
random top-level forms from a file.
\item[\kwd{cleanup}] This kind of function represents the cleanup
forms in an \code{unwind-protect}.
\item[\nil] This kind of function is not one of the above; that is,
it is not specially marked in any way.
\end{Lentry}
\end{defun}
\begin{defun}{}{debug-function-function}{\args{\var{debug-function}}}
This function returns the Common Lisp function associated with the
\var{debug-function}. This returns \nil{} if the function is
unavailable or is non-existent as a user callable function object.
\end{defun}
\begin{defun}{}{debug-function-name}{\args{\var{debug-function}}}
This function returns the name of the function represented by
\var{debug-function}. This may be a string or a cons; do not assume
it is a symbol.
\end{defun}
\section{Debug-blocks}
Debug-blocks contain information pertinent to a specific range of code in a
debug-function.
\begin{defmac}{}{do-debug-block-locations}{%
\args{(\var{code-var} \var{debug-block} \mopt{result})
\mstar{form}}}
This macro executes each \var{form} in a context with \var{code-var}
bound to each code-location in \var{debug-block}. This returns the
value of executing \var{result} (defaults to \nil).
\end{defmac}
\begin{defun}{}{debug-block-successors}{\args{\var{debug-block}}}
This function returns the list of possible code-locations where
execution may continue when the basic-block represented by
\var{debug-block} completes its execution.
\end{defun}
\begin{defun}{}{debug-block-elsewhere-p}{\args{\var{debug-block}}}
This function returns whether \var{debug-block} represents elsewhere
code. This is code the compiler has moved out of a function's code
sequence for optimization reasons. Code-locations in these blocks
are unsuitable for stepping tools, and the first code-location has
nothing to do with a normal starting location for the block.
\end{defun}
\section{Breakpoints}
A breakpoint represents a function the system calls with the current frame when
execution passes a certain code-location. A break point is active or inactive
independent of its existence. They also have an extra slot for users to tag
the breakpoint with information.
\begin{defun}{}{make-breakpoint}{%
\args{\var{hook-function} \var{what} \keys{\kwd{kind} \kwd{info}
\kwd{function-end-cookie}}}}
This function creates and returns a breakpoint. When program
execution encounters the breakpoint, the system calls
\var{hook-function}. \var{hook-function} takes the current frame
for the function in which the program is running and the breakpoint
object.
\var{what} and \var{kind} determine where in a function the system
invokes \var{hook-function}. \var{what} is either a code-location
or a debug-function. \var{kind} is one of \kwd{code-location},
\kwd{function-start}, or \kwd{function-end}. Since the starts and
ends of functions may not have code-locations representing them,
designate these places by supplying \var{what} as a debug-function
and \var{kind} indicating the \kwd{function-start} or
\kwd{function-end}. When \var{what} is a debug-function and
\var{kind} is \kwd{function-end}, then hook-function must take two
additional arguments, a list of values returned by the function and
a function-end-cookie.
\var{info} is information supplied by and used by the user.
\var{function-end-cookie} is a function. To implement function-end
breakpoints, the system uses starter breakpoints to establish the
function-end breakpoint for each invocation of the function. Upon
each entry, the system creates a unique cookie to identify the
invocation, and when the user supplies a function for this argument,
the system invokes it on the cookie. The system later invokes the
function-end breakpoint hook on the same cookie. The user may save
the cookie when passed to the function-end-cookie function for later
comparison in the hook function.
This signals an error if \var{what} is an unknown code-location.
{\em Note: Breakpoints in interpreted code or byte-compiled code are
not implemented. Function-end breakpoints are not implemented for
compiled functions that use the known local return convention
(e.g. for block-compiled or self-recursive functions.)}
\end{defun}
\begin{defun}{}{activate-breakpoint}{\args{\var{breakpoint}}}
This function causes the system to invoke the \var{breakpoint}'s
hook-function until the next call to \code{deactivate-breakpoint} or
\code{delete-breakpoint}. The system invokes breakpoint hook
functions in the opposite order that you activate them.
\end{defun}
\begin{defun}{}{deactivate-breakpoint}{\args{\var{breakpoint}}}
This function stops the system from invoking the \var{breakpoint}'s
hook-function.
\end{defun}
\begin{defun}{}{breakpoint-active-p}{\args{\var{breakpoint}}}
This returns whether \var{breakpoint} is currently active.
\end{defun}
\begin{defun}{}{breakpoint-hook-function}{\args{\var{breakpoint}}}
This function returns the \var{breakpoint}'s function the system
calls when execution encounters \var{breakpoint}, and it is active.
This is \code{SETF}'able.
\end{defun}
\begin{defun}{}{breakpoint-info}{\args{\var{breakpoint}}}
This function returns \var{breakpoint}'s information supplied by the
user. This is \code{SETF}'able.
\end{defun}
\begin{defun}{}{breakpoint-kind}{\args{\var{breakpoint}}}
This function returns the \var{breakpoint}'s kind specification.
\end{defun}
\begin{defun}{}{breakpoint-what}{\args{\var{breakpoint}}}
This function returns the \var{breakpoint}'s what specification.
\end{defun}
\begin{defun}{}{delete-breakpoint}{\args{\var{breakpoint}}}
This function frees system storage and removes computational
overhead associated with \var{breakpoint}. After calling this,
\var{breakpoint} is useless and can never become active again.
\end{defun}
\section{Code-locations}
Code-locations represent places in functions where the system has correct
information about the function's environment and where interesting operations
can occur---asking for a local variable's value, setting breakpoints,
evaluating forms within the function's environment, etc.
Sometimes the interface returns unknown code-locations. These
represent places in functions, but there is no debug information
associated with them. Some operations accept these since they may
succeed even with missing debug data. These operations' argument is
named \var{basic-code-location} indicating they take known and unknown
code-locations. If an operation names its argument
\var{code-location}, and you supply an unknown one, it will signal an
error. For example, \code{frame-code-location} may return an unknown
code-location if someone interrupted Lisp in the given frame. The
system knows where execution will continue, but this place in the code
may not be a place for which the compiler dumped debug information.
\begin{defun}{}{code-location-debug-function}{\args{\var{basic-code-location}}}
This function returns the debug-function representing information
about the function corresponding to the code-location.
\end{defun}
\begin{defun}{}{code-location-debug-block}{\args{\var{basic-code-location}}}
This function returns the debug-block containing code-location if it
is available. Some debug policies inhibit debug-block information,
and if none is available, then this signals a \code{no-debug-blocks}
condition.
\end{defun}
\begin{defun}{}{code-location-top-level-form-offset}{%
\args{\var{code-location}}}
This function returns the number of top-level forms before the one
containing \var{code-location} as seen by the compiler in some
compilation unit. A compilation unit is not necessarily a single
file, see the section on debug-sources.
\end{defun}
\begin{defun}{}{code-location-form-number}{\args{\var{code-location}}}
This function returns the number of the form corresponding to
\var{code-location}. The form number is derived by walking the
subforms of a top-level form in depth-first order. While walking
the top-level form, count one in depth-first order for each subform
that is a cons. See \funref{form-number-translations}.
\end{defun}
\begin{defun}{}{code-location-debug-source}{\args{\var{code-location}}}
This function returns \var{code-location}'s debug-source.
\end{defun}
\begin{defun}{}{code-location-unknown-p}{\args{\var{basic-code-location}}}
This function returns whether \var{basic-code-location} is unknown.
It returns \nil{} when the code-location is known.
\end{defun}
\begin{defun}{}{code-location=}{\args{\var{code-location1}
\var{code-location2}}}
This function returns whether the two code-locations are the same.
\end{defun}
\section{Debug-sources}
Debug-sources represent how to get back the source for some code. The
source is either a file (\code{compile-file} or \code{load}), a
lambda-expression (\code{compile}, \code{defun}, \code{defmacro}), or
a stream (something particular to \cmucl{}, \code{compile-from-stream}).
When compiling a source, the compiler counts each top-level form it
processes, but when the compiler handles multiple files as one block
compilation, the top-level form count continues past file boundaries.
Therefore \code{code-location-top-level-form-offset} returns an offset
that does not always start at zero for the code-location's
debug-source. The offset into a particular source is
\code{code-location-top-level-form-offset} minus
\code{debug-source-root-number}.
Inside a top-level form, a code-location's form number indicates the
subform corresponding to the code-location.
\begin{defun}{}{debug-source-from}{\args{\var{debug-source}}}
This function returns an indication of the type of source. The
following are the possible values:
\begin{Lentry}
\item[\kwd{file}] from a file (obtained by \code{compile-file} if
compiled).
\item[\kwd{lisp}] from Lisp (obtained by \code{compile} if
compiled).
\item[\kwd{stream}] from a non-file stream (\cmucl{} supports
\code{compile-from-stream}).
\end{Lentry}
\end{defun}
\begin{defun}{}{debug-source-name}{\args{\var{debug-source}}}
This function returns the actual source in some sense represented by
debug-source, which is related to \code{debug-source-from}:
\begin{Lentry}
\item[\kwd{file}] the pathname of the file.
\item[\kwd{lisp}] a lambda-expression.
\item[\kwd{stream}] some descriptive string that's otherwise
useless.
\end{Lentry}
\end{defun}
\begin{defun}{}{debug-source-created}{\args{\var{debug-source}}}
This function returns the universal time someone created the source.
This may be \nil{} if it is unavailable.
\end{defun}
\begin{defun}{}{debug-source-compiled}{\args{\var{debug-source}}}
This function returns the time someone compiled the source. This is
\nil{} if the source is uncompiled.
\end{defun}
\begin{defun}{}{debug-source-root-number}{\args{\var{debug-source}}}
This returns the number of top-level forms processed by the compiler
before compiling this source. If this source is uncompiled, this is
zero. This may be zero even if the source is compiled since the
first form in the first file compiled in one compilation, for
example, must have a root number of zero---the compiler saw no other
top-level forms before it.
\end{defun}
\section{Source Translation Utilities}
These two functions provide a mechanism for converting the rather
obscure (but highly compact) representation of source locations into an
actual source form:
\begin{defun}{}{debug-source-start-positions}{\args{\var{debug-source}}}
This function returns the file position of each top-level form as a
vector if \var{debug-source} is from a \kwd{file}. If
\code{debug-source-from} is \kwd{lisp} or \kwd{stream}, or the file
is byte-compiled, then the result is \false{}.
\end{defun}
\begin{defun}{}{form-number-translations}{\args{\var{form}
\var{tlf-number}}}
This function returns a table mapping form numbers (see
\code{code-location-form-number}) to source-paths. A source-path
indicates a descent into the top-level-form \var{form}, going
directly to the subform corresponding to a form number.
\var{tlf-number} is the top-level-form number of \var{form}.
\end{defun}
\begin{defun}{}{source-path-context}{%
\args{\var{form} \var{path} \var{context}}}
This function returns the subform of \var{form} indicated by the
source-path. \var{Form} is a top-level form, and \var{path} is a
source-path into it. \var{Context} is the number of enclosing forms
to return instead of directly returning the source-path form. When
\var{context} is non-zero, the form returned contains a marker,
\code{\#:****HERE****}, immediately before the form indicated by
\var{path}.
\end{defun}

1334
doc/cmu-user/debugger.tex Normal file
View file

File diff suppressed because it is too large Load diff

3077
doc/cmu-user/extensions.tex Normal file
View file

File diff suppressed because it is too large Load diff

213
doc/cmu-user/hierarchical-packages.tex Normal file
View file

@ -0,0 +1,213 @@
\section{Hierarchical Packages}
\cindex{hierarchical packages}
% this section is heavily based on the Franz Inc. documentation for
% the hierarchical packages feature, as per
% <URL:http://www.franz.com/support/tech_corner/hierpackuser.lhtml>
% accessed on 2002-03-18. It is used by permission from Kevin Layer,
% obtained in email to Eric Marsden, in response to spr25795.
%
% Allegro-specific references in the document have been removed.
\subsection{Introduction}
The \clisp{} package system, designed and standardized several years
ago, is not hierarchical. Since \clisp{} was standardized, other
languages, including Java and Perl, have evolved namespaces which are
hierarchical. This document describes a hierarchical package naming
scheme for \clisp{}. The scheme was proposed by Franz Inc and
implemented in their \textit{Allegro Common Lisp} product; a
compatible implementation of the naming scheme is implemented in
\cmucl{}. This documentation is based on the Franz Inc. documentation,
and is included with permission.
The goals of hierarchical packages in \clisp{} are:
\begin{itemize}
\item
Reduce collisions with user-defined packages: it is a well-known
problem that package names used by the Lisp implementation and those
defined by users can easily conflict. The intent of hierarchical
packages is to reduce such conflicts to a minimum.
\item
Improve modularity: the current organization of packages in various
implementations has grown over the years and appears somewhat random.
Organizing future packages into a hierarchy will help make the
intention of the implementation more clear.
\item
Foster growth in \clisp{} programs, or modules, available to the CL
community: the Perl and Java communities are able to contribute code
to repositories, with minimal fear of collision, because of the
hierarchical nature of the name spaces used by the contributed code.
We want the Lisp community to benefit from shared modules in the same
way.
\end{itemize}
In a nutshell, a dot (\verb|.|) is used to separate levels in package
names, and a leading dot signifies a relative package name. The choice
of dot follows Java. Perl, another language with hierarchical
packages, uses a colon (\verb|:|) as a delimiter, but the colon is
already reserved in \clisp{}. Absolute package names require no
modifications to the underlying \clisp{} implementation. Relative
package names require only small and simple modifications.
\subsection{Relative Package Names}
Relative package names are needed for the same reason as relative
pathnames, for brevity and to reduce the brittleness of absolute
names. A relative package name is one that begins with one or more
dots. A single dot means the current package, two dots mean the parent
of the current package, and so on.
Table~\ref{tbl:hierarchical-packages} presents a number of examples,
assuming that the packages named \verb|foo|, \verb|foo.bar|,
\verb|mypack|, \verb|mypack.foo|, \verb|mypack.foo.bar|,
\verb|mypack.foo.baz|, \verb|mypack.bar|, and \verb|mypack.bar.baz|,
have all been created.
\begin{table}[h]
\begin{center}
\begin{tabular}{|l|l|l|}
\hline
relative name & current package & absolute name of referenced package \\
\hline
foo & any & foo \\
foo.bar & any & foo.bar \\
.foo & mypack & mypack.foo \\
.foo.bar & mypack & mypack.foo.bar \\
..foo & mypack.bar & mypack.foo \\
..foo.baz & mypack.bar & mypack.foo.baz \\
...foo & mypack.bar.baz & mypack.foo \\
. & mypack.bar.baz & mypack.bar.baz \\
.. & mypack.bar.baz & mypack.bar \\
... & mypack.bar.baz & mypack \\
\hline
\end{tabular}
\end{center}
\caption{Examples of hierarchical packages}
\label{tbl:hierarchical-packages}
\end{table}
Additional notes:
\begin{enumerate}
\item
All packages in the hierarchy must exist.
\item
\textbf{Warning about nicknames}: Unless you provide nicknames for
your hierarchical packages (and we recommend against doing so because
the number gets quite large), you can only use the names supplied. You
cannot mix in nicknames or alternate names. \code{cl-user}
is nickname of the \code{common-lisp-user} package.
Consider the following:
\begin{verbatim}
(defpackage :cl-user.foo)
\end{verbatim}
When the current package (the value of the variable \code{*package*})
is \code{common-lisp-user}, you might expect \verb|.foo| to refer to
\verb|cl-user.foo|, but it does not. It actually refers to the non-existent
package \verb|common-lisp-user.foo|. Note that the purpose of
nicknames is to provide shorter names in place of the longer names
that are designed to be fully descriptive. The hope is that
hierarchical packages makes longer names unnecessary and thus makes
nicknames unnecessary.
\item
Multiple dots can only appear at the beginning of a package name. For
example, \verb|foo.bar..baz| does not mean \verb|foo.baz| -- it is
invalid. (Of course, it is perfectly legal to name a package
\verb|foo.bar..baz|, but \code{cl:find-package} will not process such
a name to find \verb|foo.baz| in the package hierarchy.)
\end{enumerate}
\subsection{Compatibility with ANSI \clisp{}}
The implementation of hierarchical packages modifies the
\code{cl:find-package} function, and provides certain auxiliary
functions, \code{package-parent}, \code{package-children}, and
\code{relative-package-name-to-package}, as described in this section.
The function \code{defpackage} itself requires no modification.
While the changes to \code{cl:find-package} are small and described
below, it is an important consideration for authors who would like
their programs to run on a variety of implementations that using
hierarchical packages will work in an implementation without the
modifications discussed in this document. We show why after
describing the changes to \code{cl:find-package}.
Absolute hierarchical package names require no changes in the
underlying \clisp{} implementation.
\subsubsection{Changes to \code{cl:find-package}}
Using relative hierarchical package names requires a simple
modification of \code{cl:find-package}.
In ANSI \clisp{}, \code{cl:find-package}, if passed a package object,
returns it; if passed a string, \code{cl:find-package} looks for a
package with that string as its name or nickname, and returns the
package if it finds one, or returns nil if it does not; if passed a
symbol, the symbol name (a string) is extracted and
\code{cl:find-package} proceeds as it does with a string.
For implementing hierarchical packages, the behavior when the argument
is a package object (return it) does not change. But when the argument
is a string starting with one or more dots not directly naming a
package, \code{cl:find-package} will, instead of returning nil, check
whether the string can be resolved as naming a relative package, and
if so, return the associated absolute package object. (If the argument
is a symbol, the symbol name is extracted and \code{cl:find-package}
proceeds as it does with a string argument.)
Note that you should not use leading dots in package names when using
hierarchical packages.
\subsubsection{Using Hierarchical Packages without Modifying cl:find-package}
Even without the modifications to \code{cl:find-package}, authors need
not avoid using relative package names, but the ability to reuse
relative package names is restricted. Consider for example a module
\textit{foo} which is composed of the \verb|my.foo.bar| and
\verb|my.foo.baz| packages. In the code for each of the these packages
there are relative package references, \verb|..bar| and \verb|..baz|.
Implementations that have the new \code{cl:find-package} would have
\verb|:relative-package-names| on their \code{*features*}
list (this is the case of \cmucl{} releases starting from 18d). Then,
in the \textit{foo} module, there would be definitions of the
\verb|my.foo.bar| and \verb|my.foo.baz| packages like so:
\begin{verbatim}
(defpackage :my.foo.bar
#-relative-package-names (:nicknames #:..bar)
...)
(defpackage :my.foo.baz
#-relative-package-names (:nicknames #:..baz)
...)
\end{verbatim}
Then, in a \verb|#-relative-package-names| implementation, the symbol
\verb|my.foo.bar:blam| would be visible from \verb|my.foo.baz| as
\verb|..bar:blam|, just as it would from a
\verb|#+relative-package-names| implementation.
So, even without the implementation of the augmented
\code{cl:find-package}, one can still write \clisp{} code that will
work in both types of implementations, but \verb|..bar| and
\verb|..baz| are now used, so you cannot also have
\verb|otherpack.foo.bar| and \verb|otherpack.foo.baz| and use
\verb|..bar| and \verb|..baz| as relative names. (The point of
hierarchical packages, of course, is to allow reusing relative package
names.)

341
doc/cmu-user/internet.tex Normal file
View file

@ -0,0 +1,341 @@
\chapter{Networking Support}
\label{internet}
\credits{by Mario S. Mommer}
This chapter documents the IPv4 networking and local sockets support
offered by \cmucl{}. It covers most of the basic sockets interface
functionality in a convenient and transparent way.
For reasons of space it would be impossible to include a thorough
introduction to network programming, so we assume some basic knowledge
of the matter.
\section{Byte Order Converters}
These are the functions that convert integers from host byte order to
network byte order (big-endian).
\begin{defun}{extensions:}{htonl}{%
\args{\var{integer}}}
Converts a $32$ bit integer from host byte order to network byte
order.
\end{defun}
\begin{defun}{extensions:}{htons}{%
\args{\var{integer}}}
Converts a $16$ bit integer from host byte order to network byte
order.
\end{defun}
\begin{defun}{extensions:}{ntohs}{%
\args{\var{integer}}}
Converts a $32$ bit integer from network byte order to host
byte order.
\end{defun}
\begin{defun}{extensions:}{ntohl}{%
\args{\var{integer}}}
Converts a $32$ bit integer from network byte order to host byte
order.
\end{defun}
\section{Domain Name Services (DNS)}
The networking support of \cmucl{} includes the possibility of doing
DNS lookups. The function
\begin{defun}{extensions:}{lookup-host-entry}{%
\args{\var{host}}}
returns a structure of type \var{host-entry} (explained below) for
the given \var{host}. If \var{host} is an integer, it will be
assumed to be the IP address in host (byte-)order. If it is a string,
it can contain either the host name or the IP address in dotted
format.
This function works by completing the structure \var{host-entry}.
That is, if the user provides the IP address, then the structure will
contain that information and also the domain names. If the user
provides the domain name, the structure will be complemented with
the IP addresses along with the any aliases the host might have.
\end{defun}
\newpage
\begin{deftp}{structure}{host-entry}\args{\var{name} \var{aliases}
\var{addr-type} \var{addr-list}}
This structure holds all information available at request time on a
given host. The entries are self-explanatory. Aliases is a list of
strings containing alternative names of the host, and addr-list a
list of addresses stored in host byte order. The field
\var{addr-type} contains the number of the address family, as
specified in {\tt socket.h}, to which the addresses belong. Since
only addresses of the IPv4 family are currently supported, this slot
always has the value $2$.
\end{deftp}
\begin{defun}{extensions:}{ip-string}{%
\args{\var{addr}}}
This function takes an IP address in host order and returns a string
containing it in dotted format.
\end{defun}
\section{Binding to Interfaces}
In this section, functions for creating sockets bound to an interface
are documented.
\begin{defun}{extensions:}{create-inet-listener}{%
\args{\var{port} \ampoptional{} \var{kind} %
\keys{\kwd{reuse-address} \kwd{backlog} \kwd{host}}}}
Creates a socket and binds it to a port, prepared to receive
connections of kind \var{kind} (which defaults to \kwd{stream}),
queuing up to \var{backlog} of them. If \kwd{reuse-address} \var{T}
is used, the option SO\_REUSEADDR is used in the call to \var{bind}.
If no value is given for \kwd{host}, it will try to bind to the
default IP address of the machine where the Lisp process is running.
\end{defun}
\begin{defun}{extensions:}{create-unix-listener}{%
\args{\var{path} \ampoptional{} \var{kind} \keys{
\kwd{backlog}}}}
Creates a socket and binds it to the file name given by \var{path},
prepared to receive connections of kind \var{kind} (which defaults
to \kwd{stream}), queuing up to \var{backlog} of them.
% If
% \kwd{reuse-address} \var{T} is used, then the file given by
% \var{path} is unlinked first.
\end{defun}
\section{Accepting Connections}
Once a socket is bound to its interface, we have to explicitly accept
connections. This task is performed by the functions we document here.
\begin{defun}{extensions:}{accept-tcp-connection}{%
\args{\var{unconnected}}}
Waits until a connection arrives on the (internet family) socket
\var{unconnected}. Returns the file descriptor of the connection.
These can be conveniently encapsulated using file descriptor
streams; see \ref{sec:fds}.
\end{defun}
\begin{defun}{extensions:}{accept-unix-connection}{%
\args{\var{unconnected}}}
Waits until a connection arrives on the (unix family) socket
\var{unconnected}. Returns the file descriptor of the connection.
These can be conveniently encapsulated using file descriptor
streams; see \ref{sec:fds}.
\end{defun}
\begin{defun}{extensions:}{accept-network-stream}{%
\args{\var{socket} \keys{\kwd{buffering} \kwd{timeout} \kwd{wait-max}}}}
Accept a connect from the specified \var{socket} and returns a stream
connected to connection.
\end{defun}
\section{Connecting}
The task performed by the functions we present next is connecting to
remote hosts.
\begin{defun}{extensions:}{connect-to-inet-socket}{%
\args{\var{host} \var{port} \ampoptional{} \var{kind}
\keys{\kwd{local-host} \kwd{local-port}}}}
Tries to open a connection to the remote host \var{host} (which may
be an IP address in host order, or a string with either a host name
or an IP address in dotted format) on port \var{port}. Returns the
file descriptor of the connection. The optional parameter
\var{kind} can be either \kwd{stream} (the default) or \kwd{datagram}.
If \var{local-host} and \var{local-port} are specified, the socket
that is created is also bound to the specified \var{local-host} and
\var{port}.
\end{defun}
\begin{defun}{extensions:}{connect-to-unix-socket}{%
\args{\var{path} \ampoptional{} \var{kind}}}
Opens a connection to the unix ``address'' given by \var{path}.
Returns the file descriptor of the connection. The type of
connection is given by \var{kind}, which can be either \kwd{stream}
(the default) or \kwd{datagram}.
\end{defun}
\begin{defun}{extensions:}{open-network-stream}{%
\args{\var{host} \var{port} \keys{\kwd{buffering} \kwd{timeout}}}}
Return a stream connected to the specified \var{port} on the given \var{host}.
\end{defun}
\section{Out-of-Band Data}
\label{internet-oob}
Out-of-band data is data transmitted with a higher priority than
ordinary data. This is usually used by either side of the connection
to signal exceptional conditions. Due to the fact that most TCP/IP
implementations are broken in this respect, only single characters can
reliably be sent this way.
\begin{defun}{extensions:}{add-oob-handler}{%
\args{\var{fd} \var{char} \var{handler}}}
Sets the function passed in \var{handler} as a handler for the
character \var{char} on the connection whose descriptor is \var{fd}.
In case this character arrives, the function in \var{handler} is
called without any argument.
\end{defun}
\begin{defun}{extensions:}{remove-oob-handler}{%
\args{\var{fd} \var{char}}}
Removes the handler for the character \var{char} from the connection
with the file descriptor \var{fd}
\end{defun}
\begin{defun}{extensions:}{remove-all-oob-handlers}{%
\args{\var{fd}}}
After calling this function, the connection whose descriptor is
\var{fd} will ignore any out-of-band character it receives.
\end{defun}
\begin{defun}{extensions:}{send-character-out-of-band}{%
\args{\var{fd} \var{char}}}
Sends the character \var{char} through the connection \var{fd} out
of band.
\end{defun}
\section{Unbound Sockets, Socket Options, and Closing Sockets}
These functions create unbound sockets. This is usually not necessary,
since connectors and listeners create their own.
\begin{defun}{extensions:}{create-unix-socket}{%
\args{\ampoptional{} \var{type}}}
Creates a unix socket for the unix address family, of type
\var{:stream} and (on success) returns its file descriptor.
\end{defun}
\begin{defun}{extensions:}{create-inet-socket}{%
\args{\ampoptional{} \var{kind}}}
Creates a unix socket for the internet address family, of type
\var{:stream} and (on success) returns its file descriptor.
\end{defun}
\bigskip
Once a socket is created, it is sometimes useful to bind the socket to a
local address using \code{bind-inet-socket}:
\begin{defun}{extensions:}{bind-inet-socket}{%
\args{\var{socket} \var{host} \var{port}}}
Bind the \var{socket} to a local interface address specified
by \var{host} and \var{port}.
\end{defun}
\bigskip
Further, it is desirable to be able to change socket options. This is
performed by the following two functions, which are essentially
wrappers for system calls to {\tt getsockopt} and {\tt setsockopt}.
\begin{defun}{extensions:}{get-socket-option}{%
\args{\var{socket} \var{level} \var{optname}}}
Gets the value of option \var{optname} from the socket \var{socket}.
\end{defun}
\begin{defun}{extensions:}{set-socket-option}{%
\args{\var{socket} \var{level} \var{optname} \var{optval}}}
Sets the value of option \var{optname} from the socket \var{socket}
to the value \var{optval}.
\end{defun}
\bigskip
For information on possible options and values we refer to the
manpages of {\tt getsockopt} and {\tt setsockopt}, and to {\tt
socket.h}
Finally, the function
\begin{defun}{extensions:}{close-socket}{%
\args{\var{socket}}}
Closes the socket given by the file descriptor \var{socket}.
\end{defun}
\section{Unix Datagrams}
Datagram network is supported with the following functions.
\begin{defun}{extensions:}{inet-recvfrom}{%
\args{\var{fd} \var{buffer} \var{size}}
\keys{\kwd{flags}}}
A simple interface to the Unix \code{recvfrom} function. Returns
three values: bytecount, source address as integer, and source
port. Bytecount can of course be negative, to indicate faults.
\end{defun}
\begin{defun}{extensions:}{inet-sendto}{%
\args{\var{fd} \var{buffer} \var{size} \var{addr} \var{port}}
\keys{\kwd{flags}}}
A simple interface to the Unix \code{sendto} function.
\end{defun}
\begin{defun}{extensions:}{inet-shutdown}{%
\args{\var{fd} \var{level}}}
A simple interface to the Unix \code{shutdown} function. For
\code{level}, you may use the following symbols to close one or
both ends of a socket: \code{shut-rd}, \code{shut-wr},
\code{shut-rdwr}.
\end{defun}
\section{Errors}
Errors that occur during socket operations signal a
\code{socket-error} condition, a subtype of the \code{error}
condition. Currently this condition includes just the Unix
\code{errno} associated with the error.

357
doc/cmu-user/introduction.tex Normal file
View file

@ -0,0 +1,357 @@
\chapter{Introduction}
\cmucl{} is a free, high-performance implementation of the Common Lisp
programming language which runs on most major Unix platforms. It
mainly conforms to the ANSI Common Lisp standard. Here is a summary of
its main features:
\begin{itemize}
\item a {\em sophisticated native-code compiler} which is capable of
powerful type inferences, and generates code competitive in speed with
C compilers.
\item generational garbage collection and multiprocessing
capability on the x86 ports.
\item a foreign function interface which allows interfacing with C code and
system libraries, including shared libraries on most platforms, and
direct access to Unix system calls.
\item support for interprocess communication and remote procedure
calls.
\item an implementation of CLOS, the Common Lisp Object System, which
includes multimethods and a metaobject protocol.
\item a graphical source-level debugger using a Motif interface, and a
code profiler.
\item an interface to the X11 Window System (CLX), and a sophisticated
graphical widget library (Garnet).
\item programmer-extensible input and output streams.
\item an Emacs-like editor implemented in Common Lisp.
\item public domain: free, with full source code and no
strings attached (and no warranty). Like GNU/Linux and the *BSD
operating systems, \cmucl{} is maintained and improved by a team of
volunteers collaborating over the Internet.
\end{itemize}
This user's manual contains only implementation-specific information
about \cmucl. Users will also need a separate manual describing the
\clisp{} standard, for example, the
\ifpdf
\href{http://www.lispworks.com/documentation/HyperSpec/Front/index.htm}
{Hyperspec}.
\else
\emph{Hyperspec} at \url{http://www.lispworks.com/documentation/HyperSpec/Front/index.htm}
\fi
In addition to the language itself, this document describes a number
of useful library modules that run in \cmucl. \hemlock, an Emacs-like
text editor, is included as an integral part of the \cmucl{}
environment. Two documents describe \hemlock{}: the {\it Hemlock
User's Manual}, and the {\it Hemlock Command Implementor's Manual}.
\section{Distribution and Support}
\cmucl{} is developed and maintained by a group of volunteers who
collaborate over the internet. Sources and binary releases for the
various supported platforms can be obtained from
\href{http://www.cons.org/cmucl/}{www.cons.org/cmucl}. These pages
describe how to download by FTP or CVS.
A number of mailing lists are available for users and developers;
please see the web site for more information.
\section{Command Line Options}
\cindex{command line options}
\label{command-line-options}
The command line syntax and environment is described in the
\verb|lisp(1)| man page in the man/man1 directory of the distribution.
See also \verb|cmucl(1)|. Currently \cmucl{} accepts the following
switches:
\begin{Lentry}
\item[\code{-{}-help}] Same as \code{-help}.
\item[\code{-help}] Print ou the command line options and exit.
\item[\code{-batch}] specifies batch mode, where all input is
directed from standard-input. An error code of 0 is returned upon
encountering an EOF and 1 otherwise.
\item[\code{-quiet}] enters quiet mode. This implies setting the
variables \code{*load-verbose*}, \code{*compile-verbose*},
\code{*compile-print*}, \code{*compile-progress*},
\code{*require-verbose*} and \code{*gc-verbose*} to NIL, and
disables the printing of the startup banner.
\item[\code{-core}] requires an argument that should be the name of a
core file. Rather than using the default core file, which is searched
in a number of places, according to the initial value of the
\code{library:} search-list, the specified core file is loaded. This
switch overrides the value of the \code{CMUCLCORE} environment variable,
if present.
\item[\code{-lib}] requires an argument that should be the path to the
CMUCL library directory, which is going to be used to initialize the
\code{library:} search-list, among other things. This switch overrides
the value of the \code{CMUCLLIB} environment variable, if present.
\item[\code{-dynamic-space-size}] requires an argument that should be
the number of megabytes (1048576 bytes) that should be allocated to
the heap. If not specified, a platform-specific default is used.
The actual maximum allowed heap size is platform-specific.
Currently, this option is only available for the x86 and sparc
platforms.
\item[\code{-edit}] specifies to enter Hemlock. A file to edit may be
specified by placing the name of the file between the program name
(usually \file{lisp}) and the first switch.
\item[\code{-eval}] accepts one argument which should be a Lisp form
to evaluate during the start up sequence. The value of the form
will not be printed unless it is wrapped in a form that does output.
\item[\code{-hinit}] accepts an argument that should be the name of
the hemlock init file to load the first time the function
\findexed{ed} is invoked. The default is to load
\file{hemlock-init.\var{object-type}}, or if that does not exist,
\file{hemlock-init.lisp} from the user's home directory. If the
file is not in the user's home directory, the full path must be
specified.
\item[\code{-init}] accepts an argument that should be the name of an
init file to load during the normal start up sequence. The default
is to load \file{init.\var{object-type}} or, if that does not exist,
\file{init.lisp} from the user's home directory. If neither exists,
\cmucl tries \file{.cmucl-init.\var{object-type}} and then
\file{.cmucl-init.lisp}. If the file is not
in the user's home directory, the full path must be specified. If
the file does not exist, \cmucl silently ignores it.
\item[\code{-noinit}] accepts no arguments and specifies that an init
file should not be loaded during the normal start up sequence.
Also, this switch suppresses the loading of a hemlock init file when
Hemlock is started up with the \code{-edit} switch.
\item[\code{-nositeinit}] accepts no arguments and specifies that the
site init file should not be loaded during the normal start up
sequence.
\item[\code{-load}] accepts an argument which should be the name of a
file to load into Lisp before entering Lisp's read-eval-print loop.
\item[\code{-slave}] specifies that Lisp should start up as a
\i{slave} Lisp and try to connect to an editor Lisp. The name of
the editor to connect to must be specified\dash{}to find the
editor's name, use the \hemlock{} ``\code{Accept Slave
Connections}'' command. The name for the editor Lisp is of the
form:
\begin{example}
\var{machine-name}\code{:}\var{socket}
\end{example}
where \var{machine-name} is the internet host name for the machine
and \var{socket} is the decimal number of the socket to connect to.
\item[\code{-fpu}] specifies what fpu should be used for x87 machines.
The possible values are ``\code{x87}'', ``\code{sse2}'', or
``\code{auto}'', which is the default. By default, \cmucl will
detect if the chip supports the SSE2 instruction set or not. If so
or if \code{-fpu sse2} is specified, the SSE2 core will be loaded
that uses SSE2 for floating-point arithmetic. If SSE2 is not
available or if \code{-fpu x87} is given, the legacy x87 core is
loaded.
\item[\code{--}] indicates that everything after ``\code{--}'' is not
subject to \cmucl's command line parsing. Everything after
``\code{--}'' is placed in the variable
\code{ext:*command-line-application-arguments*}.
\end{Lentry}
For more details on the use of the \code{-edit} and \code{-slave}
switches, see the {\it Hemlock User's Manual}.
Arguments to the above switches can be specified in one of two ways:
\w{\var{switch}\code{=}\var{value}} or
\w{\var{switch}<\var{space}>\var{value}}. For example, to start up
the saved core file mylisp.core use either of the following two
commands:
\begin{example}
lisp -core=mylisp.core
lisp -core mylisp.core
\end{example}
\section{Credits}
\cmucl{} was developed at the Computer Science Department of Carnegie
Mellon University. The work was a small autonomous part within the
Mach microkernel-based operating system project, and started more as a
tool development effort than a research project. The project started
out as Spice Lisp, which provided a modern Lisp implementation for use
in the CMU community. \cmucl{} has been under continual development since
the early 1980's (concurrent with the \clisp{} standardization
effort). Most of the CMU Common Lisp implementors went on to work on
the Gwydion environment for Dylan. The CMU team was lead by Scott E.
Fahlman, the \python{} compiler was written by Robert MacLachlan.
\cmucl{}'s CLOS implementation is derived from the PCL reference
implementation written at Xerox PARC:
\begin{quotation}
\noindent Copyright (c) 1985, 1986, 1987, 1988, 1989, 1990 Xerox
Corporation.\\
All rights reserved.
\vspace{1ex}
\noindent Use and copying of this software and preparation of
derivative works based upon this software are permitted. Any
distribution of this software or derivative works must comply with all
applicable United States export control laws.
\vspace{1ex}
\noindent This software is made available AS IS, and Xerox Corporation
makes no warranty about the software, its performance or its
conformity to any specification.
\end{quotation}
Its implementation of the LOOP macro was derived from code from
Symbolics, which was derived from code written at MIT:
\begin{quotation}
\noindent Portions of LOOP are Copyright (c) 1986 by the Massachusetts
Institute of Technology.\\
All Rights Reserved.
\vspace{1ex}
\noindent Permission to use, copy, modify and distribute this software
and its documentation for any purpose and without fee is hereby granted,
provided that the M.I.T. copyright notice appear in all copies and that
both that copyright notice and this permission notice appear in
supporting documentation. The names "M.I.T." and "Massachusetts
Institute of Technology" may not be used in advertising or publicity
pertaining to distribution of the software without specific, written
prior permission. Notice must be given in supporting documentation that
copying distribution is by permission of M.I.T. M.I.T. makes no
representations about the suitability of this software for any purpose.
It is provided "as is" without express or implied warranty.
\vspace{3ex}
\noindent Portions of LOOP are Copyright (c) 1989, 1990, 1991, 1992 by
Symbolics, Inc.\\
All Rights Reserved.
\vspace{1ex}
\noindent Permission to use, copy, modify and distribute this software
and its documentation for any purpose and without fee is hereby
granted, provided that the Symbolics copyright notice appear in all
copies and that both that copyright notice and this permission notice
appear in supporting documentation. The name "Symbolics" may not be
used in advertising or publicity pertaining to distribution of the
software without specific, written prior permission. Notice must be
given in supporting documentation that copying distribution is by
permission of Symbolics. Symbolics makes no representations about the
suitability of this software for any purpose. It is provided "as is"
without express or implied warranty.
\vspace{1ex}
\noindent Symbolics, CLOE Runtime, and Minima are trademarks, and
CLOE, Genera, and Zetalisp are registered trademarks of Symbolics,
Inc.
\end{quotation}
The CLX code is copyrighted by Texas Instruments Incorporated:
\begin{quotation}
\noindent Copyright (C) 1987 Texas Instruments Incorporated.
\vspace{1ex}
\noindent Permission is granted to any individual or institution to
use, copy, modify, and distribute this software, provided that this
complete copyright and permission notice is maintained, intact, in all
copies and supporting documentation.
\vspace{1ex}
\noindent Texas Instruments Incorporated provides this software "as
is" without express or implied warranty.
\end{quotation}
\cmucl{} was funded by DARPA under CMU's "Research on Parallel Computing"
contract. Rather than doing pure research on programming languages and
environments, the emphasis was on developing practical programming
tools. Sometimes this required new technology, but much of the work
was in creating a \clisp{} environment that incorporates
state-of-the-art features from existing systems (both Lisp and
non-Lisp). Archives of the project are available online.
The project funding stopped in 1994, so support at Carnegie Mellon
University has been discontinued. All code and documentation developed
at CMU was released into the public domain. The project continues as a
group of users and developers collaborating over the Internet. The
current and previous maintainers include:
\begin{itemize}
\item Marco Antoniotti
\item Martin Cracauer
\item Fred Gilham
\item Alex Goncharov
\item Rob MacLachlan
\item Pierre Mai
\item Eric Marsden
\item Gerd Moellman
\item Tim Moore
\item Carl Shapiro
\item Robert Swindells
\item Raymond Toy
\item Peter Van Eynde
\item Paul Werkowski
\end{itemize}
In particular, Paul Werkowski and Douglas Crosher completed the port
for the x86 architecture for FreeBSD. Peter VanEnyde took the FreeBSD
port and created a Linux version. Other people who have contributed to
the development of \cmucl{} since 1981 are
\begin{itemize}
\item David Axmark
\item Miles Bader
\item Rick Busdiecker
\item Bill Chiles
\item Douglas Thomas Crosher
\item Casper Dik
\item Ted Dunning
\item Scott Fahlman
\item Mike Garland
\item Paul Gleichauf
\item Sean Hallgren
\item Richard Harris
\item Joerg-Cyril Hoehl
\item Chris Hoover
\item John Kolojejchick
\item Todd Kaufmann
\item Simon Leinen
\item Sandra Loosemore
\item William Lott
\item Dave McDonald
\item Tim Moore
\item Skef Wholey
\item Paul Foley
\item Helmut Eller
\item Jan Rychter
\end{itemize}
Countless others have contributed to the project by sending in bug
reports, bug fixes, and new features.
This manual is based on CMU Technical Report CMU-CS-92-161, edited by
Robert A. MacLachlan, dated July 1992. Other contributors include
Raymond Toy, Paul Werkowski and Eric Marsden. The Hierarchical
Packages chapter is based on documentation written by Franz. Inc, and
is used with permission. The remainder of the document is in the
public domain.

414
doc/cmu-user/ipc.tex Normal file
View file

@ -0,0 +1,414 @@
\chapter{Interprocess Communication under LISP}
\label{remote}
\credits{by William Lott and Bill Chiles}
\cmucl{} offers a facility for interprocess communication (IPC)
on top of using Unix system calls and the complications of that level
of IPC. There is a simple remote-procedure-call (RPC) package build
on top of TCP/IP sockets.
\section{The REMOTE Package}
The \code{remote} package provides simple RPC facility including
interfaces for creating servers, connecting to already existing
servers, and calling functions in other Lisp processes. The routines
for establishing a connection between two processes,
\code{create-request-server} and \code{connect-to-remote-server},
return \var{wire} structures. A wire maintains the current state of
a connection, and all the RPC forms require a wire to indicate where
to send requests.
\subsection{Connecting Servers and Clients}
Before a client can connect to a server, it must know the network address on
which the server accepts connections. Network addresses consist of a host
address or name, and a port number. Host addresses are either a string of the
form \code{VANCOUVER.SLISP.CS.CMU.EDU} or a 32 bit unsigned integer. Port
numbers are 16 bit unsigned integers. Note: \var{port} in this context has
nothing to do with Mach ports and message passing.
When a process wants to receive connection requests (that is, become a
server), it first picks an integer to use as the port. Only one server
(Lisp or otherwise) can use a given port number on a given machine at
any particular time. This can be an iterative process to find a free
port: picking an integer and calling \code{create-request-server}. This
function signals an error if the chosen port is unusable. You will
probably want to write a loop using \code{handler-case}, catching
conditions of type error, since this function does not signal more
specific conditions.
\begin{defun}{wire:}{create-request-server}{%
\args{\var{port} \ampoptional{} \var{on-connect}}}
\code{create-request-server} sets up the current Lisp to accept
connections on the given port. If port is unavailable for any
reason, this signals an error. When a client connects to this port,
the acceptance mechanism makes a wire structure and invokes the
\var{on-connect} function. Invoking this function has a couple of
purposes, and \var{on-connect} may be \nil{} in which case the
system foregoes invoking any function at connect time.
The \var{on-connect} function is both a hook that allows you access
to the wire created by the acceptance mechanism, and it confirms the
connection. This function takes two arguments, the wire and the
host address of the connecting process. See the section on host
addresses below. When \var{on-connect} is \nil, the request server
allows all connections. When it is non-\nil, the function returns
two values, whether to accept the connection and a function the
system should call when the connection terminates. Either value may
be \nil, but when the first value is \nil, the acceptance mechanism
destroys the wire.
\code{create-request-server} returns an object that
\code{destroy-request-server} uses to terminate a connection.
\end{defun}
\begin{defun}{wire:}{destroy-request-server}{\args{\var{server}}}
\code{destroy-request-server} takes the result of
\code{create-request-server} and terminates that server. Any
existing connections remain intact, but all additional connection
attempts will fail.
\end{defun}
\begin{defun}{wire:}{connect-to-remote-server}{%
\args{\var{host} \var{port} \ampoptional{} \var{on-death}}}
\code{connect-to-remote-server} attempts to connect to a remote
server at the given \var{port} on \var{host} and returns a wire
structure if it is successful. If \var{on-death} is non-\nil, it is
a function the system invokes when this connection terminates.
\end{defun}
\subsection{Remote Evaluations}
After the server and client have connected, they each have a wire
allowing function evaluation in the other process. This RPC mechanism
has three flavors: for side-effect only, for a single value, and for
multiple values.
Only a limited number of data types can be sent across wires as
arguments for remote function calls and as return values: integers
inclusively less than 32 bits in length, symbols, lists, and
\var{remote-objects} (\pxlref{remote-objs}). The system sends symbols
as two strings, the package name and the symbol name, and if the
package doesn't exist remotely, the remote process signals an error.
The system ignores other slots of symbols. Lists may be any tree of
the above valid data types. To send other data types you must
represent them in terms of these supported types. For example, you
could use \code{prin1-to-string} locally, send the string, and use
\code{read-from-string} remotely.
\begin{defmac}{wire:}{remote}{%
\args{\var{wire} \mstar{call-specs}}}
The \code{remote} macro arranges for the process at the other end of
\var{wire} to invoke each of the functions in the \var{call-specs}.
To make sure the system sends the remote evaluation requests over
the wire, you must call \code{wire-force-output}.
Each of \var{call-specs} looks like a function call textually, but
it has some odd constraints and semantics. The function position of
the form must be the symbolic name of a function. \code{remote}
evaluates each of the argument subforms for each of the
\var{call-specs} locally in the current context, sending these
values as the arguments for the functions.
Consider the following example:
\begin{verbatim}
(defun write-remote-string (str)
(declare (simple-string str))
(wire:remote wire
(write-string str)))
\end{verbatim}
The value of \code{str} in the local process is passed over the wire
with a request to invoke \code{write-string} on the value. The
system does not expect to remotely evaluate \code{str} for a value
in the remote process.
\end{defmac}
\begin{defun}{wire:}{wire-force-output}{\args{\var{wire}}}
\code{wire-force-output} flushes all internal buffers associated
with \var{wire}, sending the remote requests. This is necessary
after a call to \code{remote}.
\end{defun}
\begin{defmac}{wire:}{remote-value}{\args{\var{wire} \var{call-spec}}}
The \code{remote-value} macro is similar to the \code{remote} macro.
\code{remote-value} only takes one \var{call-spec}, and it returns
the value returned by the function call in the remote process. The
value must be a valid type the system can send over a wire, and
there is no need to call \code{wire-force-output} in conjunction
with this interface.
If client unwinds past the call to \code{remote-value}, the server
continues running, but the system ignores the value the server sends
back.
If the server unwinds past the remotely requested call, instead of
returning normally, \code{remote-value} returns two values, \nil{}
and \true. Otherwise this returns the result of the remote
evaluation and \nil.
\end{defmac}
\begin{defmac}{wire:}{remote-value-bind}{%
\args{\var{wire} (\mstar{variable}) \var{remote-form}
\mstar{local-forms}}}
\code{remote-value-bind} is similar to \code{multiple-value-bind}
except the values bound come from \var{remote-form}'s evaluation in
the remote process. The \var{local-forms} execute in an implicit
\code{progn}.
If the client unwinds past the call to \code{remote-value-bind}, the
server continues running, but the system ignores the values the
server sends back.
If the server unwinds past the remotely requested call, instead of
returning normally, the \var{local-forms} never execute, and
\code{remote-value-bind} returns \nil.
\end{defmac}
\subsection{Remote Objects}
\label{remote-objs}
The wire mechanism only directly supports a limited number of data
types for transmission as arguments for remote function calls and as
return values: integers inclusively less than 32 bits in length,
symbols, lists. Sometimes it is useful to allow remote processes to
refer to local data structures without allowing the remote process
to operate on the data. We have \var{remote-objects} to support
this without the need to represent the data structure in terms of
the above data types, to send the representation to the remote
process, to decode the representation, to later encode it again, and
to send it back along the wire.
You can convert any Lisp object into a remote-object. When you send
a remote-object along a wire, the system simply sends a unique token
for it. In the remote process, the system looks up the token and
returns a remote-object for the token. When the remote process
needs to refer to the original Lisp object as an argument to a
remote call back or as a return value, it uses the remote-object it
has which the system converts to the unique token, sending that
along the wire to the originating process. Upon receipt in the
first process, the system converts the token back to the same
(\code{eq}) remote-object.
\begin{defun}{wire:}{make-remote-object}{\args{\var{object}}}
\code{make-remote-object} returns a remote-object that has
\var{object} as its value. The remote-object can be passed across
wires just like the directly supported wire data types.
\end{defun}
\begin{defun}{wire:}{remote-object-p}{\args{\var{object}}}
The function \code{remote-object-p} returns \true{} if \var{object}
is a remote object and \nil{} otherwise.
\end{defun}
\begin{defun}{wire:}{remote-object-local-p}{\args{\var{remote}}}
The function \code{remote-object-local-p} returns \true{} if
\var{remote} refers to an object in the local process. This is can
only occur if the local process created \var{remote} with
\code{make-remote-object}.
\end{defun}
\begin{defun}{wire:}{remote-object-eq}{\args{\var{obj1} \var{obj2}}}
The function \code{remote-object-eq} returns \true{} if \var{obj1} and
\var{obj2} refer to the same (\code{eq}) lisp object, regardless of
which process created the remote-objects.
\end{defun}
\begin{defun}{wire:}{remote-object-value}{\args{\var{remote}}}
This function returns the original object used to create the given
remote object. It is an error if some other process originally
created the remote-object.
\end{defun}
\begin{defun}{wire:}{forget-remote-translation}{\args{\var{object}}}
This function removes the information and storage necessary to
translate remote-objects back into \var{object}, so the next
\code{gc} can reclaim the memory. You should use this when you no
longer expect to receive references to \var{object}. If some remote
process does send a reference to \var{object},
\code{remote-object-value} signals an error.
\end{defun}
% This stuff has been moved to internet.tex. *** Remove me someday ***
% \subsection{Host Addresses}
% The operating system maintains a database of all the valid host
% addresses. You can use this database to convert between host names
% and addresses and vice-versa.
% \begin{defun}{ext:}{lookup-host-entry}{\args{\var{host}}}
% \code{lookup-host-entry} searches the database for the given
% \var{host} and returns a host-entry structure for it. If it fails
% to find \var{host} in the database, it returns \nil. \var{Host} is
% either the address (as an integer) or the name (as a string) of the
% desired host.
% \end{defun}
% \begin{defun}{ext:}{host-entry-name}{\args{\var{host-entry}}}
% \defunx[ext:]{host-entry-aliases}{\args{\var{host-entry}}}
% \defunx[ext:]{host-entry-addr-list}{\args{\var{host-entry}}}
% \defunx[ext:]{host-entry-addr}{\args{\var{host-entry}}}
% \code{host-entry-name}, \code{host-entry-aliases}, and
% \code{host-entry-addr-list} each return the indicated slot from the
% host-entry structure. \code{host-entry-addr} returns the primary
% (first) address from the list returned by
% \code{host-entry-addr-list}.
% \end{defun}
\section{The WIRE Package}
The \code{wire} package provides for sending data along wires. The
\code{remote} package sits on top of this package. All data sent
with a given output routine must be read in the remote process with
the complementary fetching routine. For example, if you send so a
string with \code{wire-output-string}, the remote process must know
to use \code{wire-get-string}. To avoid rigid data transfers and
complicated code, the interface supports sending
\var{tagged} data. With tagged data, the system sends a tag
announcing the type of the next data, and the remote system takes
care of fetching the appropriate type.
When using interfaces at the wire level instead of the RPC level,
the remote process must read everything sent by these routines. If
the remote process leaves any input on the wire, it will later
mistake the data for an RPC request causing unknown lossage.
\subsection{Untagged Data}
When using these routines both ends of the wire know exactly what types are
coming and going and in what order. This data is restricted to the following
types:
\begin{itemize}
\item
8 bit unsigned bytes.
\item
32 bit unsigned bytes.
\item
32 bit integers.
\item
simple-strings less than 65535 in length.
\end{itemize}
\begin{defun}{wire:}{wire-output-byte}{\args{\var{wire} \var{byte}}}
\defunx[wire:]{wire-get-byte}{\args{\var{wire}}}
\defunx[wire:]{wire-output-number}{\args{\var{wire} \var{number}}}
\defunx[wire:]{wire-get-number}{\args{\var{wire} \ampoptional{}
\var{signed}}}
\defunx[wire:]{wire-output-string}{\args{\var{wire} \var{string}}}
\defunx[wire:]{wire-get-string}{\args{\var{wire}}}
These functions either output or input an object of the specified
data type. When you use any of these output routines to send data
across the wire, you must use the corresponding input routine
interpret the data.
\end{defun}
\subsection{Tagged Data}
When using these routines, the system automatically transmits and interprets
the tags for you, so both ends can figure out what kind of data transfers
occur. Sending tagged data allows a greater variety of data types: integers
inclusively less than 32 bits in length, symbols, lists, and \var{remote-objects}
(\pxlref{remote-objs}). The system sends symbols as two strings, the
package name and the symbol name, and if the package doesn't exist remotely,
the remote process signals an error. The system ignores other slots of
symbols. Lists may be any tree of the above valid data types. To send other
data types you must represent them in terms of these supported types. For
example, you could use \code{prin1-to-string} locally, send the string, and use
\code{read-from-string} remotely.
\begin{defun}{wire:}{wire-output-object}{%
\args{\var{wire} \var{object} \ampoptional{} \var{cache-it}}}
\defunx[wire:]{wire-get-object}{\args{\var{wire}}}
The function \code{wire-output-object} sends \var{object} over
\var{wire} preceded by a tag indicating its type.
If \var{cache-it} is non-\nil, this function only sends \var{object}
the first time it gets \var{object}. Each end of the wire
associates a token with \var{object}, similar to remote-objects,
allowing you to send the object more efficiently on successive
transmissions. \var{cache-it} defaults to \true{} for symbols and
\nil{} for other types. Since the RPC level requires function
names, a high-level protocol based on a set of function calls saves
time in sending the functions' names repeatedly.
The function \code{wire-get-object} reads the results of
\code{wire-output-object} and returns that object.
\end{defun}
\subsection{Making Your Own Wires}
You can create wires manually in addition to the \code{remote}
package's interface creating them for you. To create a wire, you need
a Unix {\em file descriptor}. If you are unfamiliar with Unix file
descriptors, see section 2 of the Unix manual pages.
\begin{defun}{wire:}{make-wire}{\args{\var{descriptor}}}
The function \code{make-wire} creates a new wire when supplied with
the file descriptor to use for the underlying I/O operations.
\end{defun}
\begin{defun}{wire:}{wire-p}{\args{\var{object}}}
This function returns \true{} if \var{object} is indeed a wire,
\nil{} otherwise.
\end{defun}
\begin{defun}{wire:}{wire-fd}{\args{\var{wire}}}
This function returns the file descriptor used by the \var{wire}.
\end{defun}
\section{Out-Of-Band Data}
The TCP/IP protocol allows users to send data asynchronously, otherwise
known as \var{out-of-band} data. When using this feature, the operating
system interrupts the receiving process if this process has chosen to be
notified about out-of-band data. The receiver can grab this input
without affecting any information currently queued on the socket.
Therefore, you can use this without interfering with any current
activity due to other wire and remote interfaces.
Unfortunately, most implementations of TCP/IP are broken, so use of
out-of-band data is limited for safety reasons. You can only reliably
send one character at a time.
The Wire package is built on top of \cmucl{}s networking support. In
view of this, it is possible to use the routines described in section
\ref{internet-oob} for handling and sending out-of-band data. These
all take a Unix file descriptor instead of a wire, but you can fetch a
wire's file descriptor with \code{wire-fd}.

56
doc/cmu-user/latex2html.tex Normal file
View file

@ -0,0 +1,56 @@
%% Replacement commands when we run latex2html. This should be last
%% so that latex2html uses these commands instead of the LaTeX
%% commands above.
\usepackage{makeidx}
\newcommand{\var}[1]{\textnormal{\textit{#1}}}
\newcommand{\code}[1]{\textnormal{\texttt{#1}}}
%%\newcommand{\printindex}[1][\mbox{}]{}
%% We need the quote environment because the alltt is broken. The
%% quote environment helps us in postprocessing to result to get
%% what we want.
\newenvironment{example}{\begin{quote}\begin{alltt}}{\end{alltt}\end{quote}}
\newenvironment{display}{\begin{quote}\begin{alltt}}{\end{alltt}\end{quote}}
\newcommand{\textnormal}[1]{\rm #1}
\newcommand{\hbox}[1]{\mbox{#1}}
\newcommand{\xspace}{}
\newcommand{newindex}[4]{}
\newcommand{\pxlref}[1]{see section~\ref{#1}}
\newcommand{\xlref}[1]{See section~\ref{#1}}
\newcommand{\tindexed}[1]{\index{#1}\texttt{#1}}
\newcommand{\findexed}[1]{\index{#1}\texttt{#1}}
\newcommand{\vindexed}[1]{\index{#1}\texttt{*#1*}}
\newcommand{\cindex}[1]{\index{#1}}
\newcommand{\cpsubindex}[2]{\index{#1!#2}}
\newcommand{\keys}[1]{\texttt{\&key} #1}
\newcommand{\morekeys}[1]{#1}
\newcommand{\yetmorekeys}[1]{#1}
\newenvironment{defun}[3]{%
\textbf{[Function]}\\
\texttt{#1#2} \emph{#3}\\}{}
\newcommand{\defunx}[3][\mbox{}]{%
\texttt{#1#2} {\em #3}\\}
\newenvironment{defmac}[3]{%
\textbf{[Macro]}\\
\texttt{#1#2} \emph{#3}\\}{}
\newcommand{\defmacx}[3][\mbox{}]{%
\texttt{#1#2} {\em #3}\\}
\newenvironment{defvar}[2]{%
\textbf{[Variable]}\\
\texttt{#1*#2*}\\ \\}{}
\newcommand{\defvarx}[2][\mbox{}]{%
\texttt{#1*#2*}\\}
\newenvironment{defconst}[2]{%
\textbf{[Constant]}\\
\texttt{#1#2}\\}{}
\newcommand{\defconstx}[2][\mbox{}]{\texttt{#1#2}\\}
\newenvironment{deftp}[3]{%
\textbf{[#1]}\\
\texttt{#2} \textit{#3}\\}{}
\newenvironment{Lentry}{\begin{description}}{\end{description}}

284
doc/cmu-user/macros.tex Normal file
View file

@ -0,0 +1,284 @@
% macro.tex
%
% LaTeX macros for CMUCL User's Manual
%
% by Raymond Toy
% use Palatino
\renewcommand{\rmdefault}{ppl}
\ifpdf
\usepackage{palatino}
\fi
%% Define the indices. We need one for Types, Variables, Functions,
%% and a general concept index.
\makeindex
\newindex{types}{tdx}{tnd}{Type Index}
\newindex{vars}{vdx}{vnd}{Variable Index}
\newindex{funs}{fdx}{fnd}{Function Index}
\newindex{concept}{cdx}{cnd}{Concept Index}
\newcommand{\tindexed}[1]{\index[types]{#1}\code{#1}}
\newcommand{\findexed}[1]{\index[funs]{#1}\code{#1}}
\newcommand{\vindexed}[1]{\index[vars]{#1}\code{*#1*}}
\newcommand{\cindex}[1]{\index[concept]{#1}}
\newcommand{\cpsubindex}[2]{\index[concept]{#1!#2}}
%% This code taken from the LaTeX companion. It's meant as a
%% replacement for the description environment. We want one that
%% prints description items in a fixed size box and puts the
%% description itself on the same line or the next depending on the
%% size of the item.
\newcommand{\entrylabel}[1]{\mbox{#1}\hfil}
\newenvironment{entry}{%
\begin{list}{}%
{\renewcommand{\makelabel}{\entrylabel}%
\setlength{\labelwidth}{45pt}%
\setlength{\leftmargin}{\labelwidth+\labelsep}}}%
{\end{list}}
\newlength{\Mylen}
\newcommand{\Lentrylabel}[1]{%
\settowidth{\Mylen}{#1}%
\ifthenelse{\lengthtest{\Mylen > \labelwidth}}%
{\parbox[b]{\labelwidth}% term > labelwidth
{\makebox[0pt][l]{#1}\\}}%
{#1}%
\hfil\relax}
\newenvironment{Lentry}{%
\renewcommand{\entrylabel}{\Lentrylabel}
\begin{entry}}%
{\end{entry}}
\newcommand{\fcntype}[1]{\textit{#1}}
\newcommand{\argtype}[1]{\textit{#1}}
\newcommand{\fcnname}[1]{\textsf{#1}}
\newlength{\formnamelen} % length of a name of a form
\newlength{\pboxargslen} % length of parbox for arguments
\newlength{\typelen} % length of the type label for the form
\newcommand{\args}[1]{#1}
\newcommand{\keys}[1]{\code{\&key} \= #1}
\newcommand{\morekeys}[1]{\\ \> #1}
\newcommand{\yetmorekeys}[1]{\\ \> #1}
\newcommand{\defunvspace}{\ifhmode\unskip \par\fi\addvspace{18pt plus 12pt minus 6pt}}
%% \layout[pkg]{name}{param list}{type}
%%
%% This lays out a entry like so:
%%
%% pkg:name arg1 arg2 [Function]
%%
%% where [Function] is flush right.
%%
\newcommand{\layout}[4][\mbox{}]{%
\par\noindent
\fcnname{#1#2\hspace{1em}}%
\settowidth{\formnamelen}{\fcnname{#1#2\hspace{1em}}}%
\settowidth{\typelen}{[\argtype{#4}]}%
\setlength{\pboxargslen}{\linewidth}%
\addtolength{\pboxargslen}{-1\formnamelen}%
\addtolength{\pboxargslen}{-1\typelen}%
\begin{minipage}[t]{\pboxargslen}
\begin{tabbing}
#3
\end{tabbing}
\end{minipage}
\hfill[\fcntype{#4}]%
\par\addvspace{2pt plus 2pt minus 2pt}}
\newcommand{\vrindexbold}[1]{\index[vars]{#1|textbf}}
\newcommand{\fnindexbold}[1]{\index[funs]{#1|textbf}}
%% Define a new type
%%
%% \begin{deftp}{typeclass}{typename}{args}
%% some description
%% \end{deftp}
\newenvironment{deftp}[3]{%
\par\bigskip\index[types]{#2|textbf}%
\layout{#2}{\var{#3}}{#1}
}{}
%% Define a function with name NAME and given parameters PARAM. The
%% function is in the package PKG. If the optional arg SUFFIX is
%% given, this is used as a suffix for the label. (Useful when you
%% have functions of the same name, such as methods, but want
%% different labels for each version.)
%%
%% The defunx is for additional functions that are related to this one
%% in some way, and we want to group them all together.
%%
%% \begin{defun}[suffix]{pkg}{name}{params}
%% \defunx[pkg]{name}{params}
%% description of function
%% \end{defun}
\newenvironment{defun}[4][]{%
\par\defunvspace\fnindexbold{#3}\label{FN:#3#1}%
\layout[#2]{#3}{#4}{Function}
}{}
\newcommand{\defunx}[3][\mbox{}]{%
\par\fnindexbold{#2}\label{FN:#2}%
\layout[#1]{#2}{#3}{Function}}
%% Define a generic function. Like defun, but for defgeneric.
%%
%% \begin{defgeneric}[suffix]{pkg}{name}{params}
%% \defgenericx[pkg]{name}{params}
%% description of function
%% \end{defgeneric}
\newenvironment{defgeneric}[4][]{%
\par\defunvspace\fnindexbold{#3}\label{FN:#3-generic#1}%
\layout[#2]{#3}{#4}{Generic Function}
}{}
\newcommand{\defgenericx}[3][\mbox{}]{%
\par\fnindexbold{#2}\label{FN:#2}%
\layout[#1]{#2}{#3}{Generic Function}}
%% Define a method. Like defgeneric, but for methods.
%%
%% \begin{defmethod}[suffix]{pkg}{name}{params}
%% \defmethod[pkg]{name}{params}
%% description of function
%% \end{defmethod}
\newenvironment{defmethod}[4][]{%
\par\defunvspace\fnindexbold{#3}\label{FN:#3-method#1}%
\layout[#2]{#3}{#4}{Method}
}{}
\newcommand{\defmethodx}[3][\mbox{}]{%
\par\fnindexbold{#2}\label{FN:#2}%
\layout[#1]{#2}{#3}{Method}}
%% Define a macro
%%
%% \begin{defmac}[suffix]{pkg}{name}{params}
%% \defmacx[pkg]{name}{params}
%% description of macro
%% \end{defmac}
\newenvironment{defmac}[4][]{%
\par\defunvspace\fnindexbold{#3}\label{FN:#3#1}%
\layout[#2]{#3}{#4}{Macro}}{}
\newcommand{\defmacx}[3][\mbox{}]{%
\par\fnindexbold{#2}\label{FN:#2}%
\layout[#1]{#2}{#3}{Macro}}
%% Define a variable
%%
%% \begin{defvar}{pkg}{name}
%% \defvarx[pkg]{name}
%% description of defvar
%% \end{defvar}
\newenvironment{defvar}[2]{%
\par\defunvspace\vrindexbold{#2}\label{VR:#2}
\layout[#1]{*#2*}{}{Variable}}{}
\newcommand{\defvarx}[2][\mbox{}]{%
\par\vrindexbold{#2}\label{VR:#2}
\layout[#1]{*#2*}{}{Variable}}
%% Define a constant
%%
%% \begin{defconst}{pkg}{name}
%% \ddefconstx[pkg]{name}
%% description of defconst
%% \end{defconst}
\newcommand{\defconstx}[2][\mbox{}]{%
\layout[#1]{#2}{}{Constant}}
\newenvironment{defconst}[2]{%
\defunvspace\defconstx[#1]{#2}}{}
\newcommand{\credits}[1]{%
\begin{center}
\textbf{#1}
\end{center}}
\newenvironment{example}{\begin{quote}\begin{alltt}}{\end{alltt}\end{quote}}
\newenvironment{lisp}{\begin{example}}{\end{example}}
\newcommand{\hide}[1]{}
\newcommand{\trnumber}[1]{#1}
\newcommand{\citationinfo}[1]{#1}
\newcommand{\var}[1]{{\textsf{\textsl{#1}}\xspace}}
\newcommand{\code}[1]{\textnormal{{\sffamily #1}}}
\newcommand{\file}[1]{`\texttt{#1}'}
\newcommand{\kwd}[1]{\code{:#1}}
\newcommand{\F}[1]{\code{#1}}
\newcommand{\w}[1]{\hbox{#1}}
\newcommand{\ctrl}[1]{$\uparrow$\textsf{#1}}
\newcommand{\result}{$\Rightarrow$}
\newcommand{\myequiv}{$\equiv$}
\newcommand{\back}[1]{\(\backslash\)#1}
\newcommand{\pxlref}[1]{see section~\ref{#1}, page~\pageref{#1}}
\newcommand{\xlref}[1]{See section~\ref{#1}, page~\pageref{#1}}
\newcommand{\funref}[1]{\findexed{#1} (page~\pageref{FN:#1})}
\newcommand{\specref}[1]{\findexed{#1} (page~\pageref{FN:#1})}
\newcommand{\macref}[1]{\findexed{#1} (page~\pageref{FN:#1})}
\newcommand{\varref}[1]{\vindexed{#1} (page~\pageref{VR:#1})}
\newcommand{\conref}[1]{\conindexed{#1} (page~\pageref{VR:#1})}
\newcommand{\false}{\code{nil}}
\newcommand{\true}{\code{t}}
\newcommand{\nil}{\false{}}
%% Printed lisp character #\foo
\newcommand{\lispchar}[1]{\code{\#\back{#1}}}
\newcommand{\ampoptional}{\code{\&optional}}
\newcommand{\amprest}{\code{\&rest}}
\newcommand{\ampbody}{\code{\&body}}
\newcommand{\mopt}[1]{{$\,\{$}\textnormal{\textsf{\textsl{#1\/}}}{$\}\,$}}
\newcommand{\mstar}[1]{{$\,\{$}\textnormal{\textsf{\textsl{#1\/}}}{$\}^*\,$}}
\newcommand{\mplus}[1]{{$\,\{$}\textnormal{\textsf{\textsl{#1\/}}}{$\}^+\,$}}
\newcommand{\mgroup}[1]{{$\,\{$}\textnormal{\textsf{\textsl{#1\/}}}{$\}\,$}}
\newcommand{\mor}{$|$}
%% Some common abbreviations
\newcommand{\dash}{---}
\newcommand{\alien}{Alien}
\newcommand{\aliens}{Aliens}
\newcommand{\hemlock}{Hemlock}
\newcommand{\python}{Python}
\newcommand{\cmucl}{\textsc{cmucl}}
\newcommand{\clisp}{Common Lisp}
\newcommand{\llisp}{Common Lisp}
\newcommand{\cltl}{\textit{Common Lisp: The Language}}
\newcommand{\cltltwo}{\textit{Common Lisp: The Language II}}
%% Set up margins
\setlength{\oddsidemargin}{-10pt}
\setlength{\evensidemargin}{-10pt}
\setlength{\topmargin}{-40pt}
\setlength{\headheight}{12pt}
\setlength{\headsep}{25pt}
\setlength{\footskip}{30pt}
\setlength{\textheight}{9.25in}
\setlength{\textwidth}{6.75in}
\setlength{\columnsep}{0.375in}
\setlength{\columnseprule}{0pt}
\setcounter{tocdepth}{2}
\setcounter{secnumdepth}{3}
\def\textfraction{.1}
\def\bottomfraction{.9} % was .3
\def\topfraction{.9}
%% Allow TeX some stretching space to avoid overfull and underfull
%% boxes.
\setlength{\emergencystretch}{5pt}
%% requires the sectsty package
\allsectionsfont{\bfseries\sffamily}
\chapterfont{\fontfamily{pag}\selectfont}
%% section numbers in the left margin
\makeatletter
\def\@seccntformat#1{\protect\makebox[0pt][r]{\csname
the#1\endcsname\quad}}
\makeatother

BIN
doc/cmu-user/next_motif.gif Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 317 B

124
doc/cmu-user/package-locks.tex Normal file
View file

@ -0,0 +1,124 @@
\section{Package Locks}
\cindex{package locks}
\cmucl{} provides two types of package locks, as an extension to the
ANSI Common Lisp standard. The package-lock protects a package from
changes in its structure (the set of exported symbols, its use list,
etc). The package-definition-lock protects the symbols in the package
from being redefined due to the execution of a \code{defun},
\code{defmacro}, \code{defstruct}, \code{deftype} or \code{defclass}
form.
\subsection{Rationale}
Package locks are an aid to program development, by helping to detect
inadvertent name collisions and function redefinitions. They are
consistent with the principle that a package ``belongs to'' its
implementor, and that noone other than the package's developer should
be making or modifying definitions on symbols in that package. Package
locks are compatible with the ANSI Common Lisp standard, which states
that the consequences of redefining functions in the
\code{COMMON-LISP} package are undefined.
Violation of a package lock leads to a continuable error of type
\code{lisp::package-locked-error} being signaled. The user may choose
to ignore the lock and proceed, or to abort the computation. Two other
restarts are available, one which disables all locks on all packages,
and one to disable only the package-lock or package-definition-lock
that was tripped.
The following transcript illustrates the behaviour seen when
attempting to redefine a standard macro in the \code{COMMON-LISP}
package, or to redefine a function in one of \cmucl{}'s
implementation-defined packages:
{\small
\begin{verbatim}
CL-USER> (defmacro 1+ (x) (* x 2))
Attempt to modify the locked package COMMON-LISP, by defining macro 1+
[Condition of type LISP::PACKAGE-LOCKED-ERROR]
Restarts:
0: [continue ] Ignore the lock and continue
1: [unlock-package] Disable the package's definition-lock then continue
2: [unlock-all ] Unlock all packages, then continue
3: [abort ] Return to Top-Level.
CL-USER> (defun ext:gc () t)
Attempt to modify the locked package EXTENSIONS, by redefining function GC
[Condition of type LISP::PACKAGE-LOCKED-ERROR]
Restarts:
0: [continue ] Ignore the lock and continue
1: [unlock-package] Disable package's definition-lock, then continue
2: [unlock-all ] Disable all package locks, then continue
3: [abort ] Return to Top-Level.
\end{verbatim}
The following transcript illustrates the behaviour seen when an
attempt to modify the structure of a package is made:
\begin{verbatim}
CL-USER> (unexport 'load-foreign :ext)
Attempt to modify the locked package EXTENSIONS, by unexporting symbols LOAD-FOREIGN
[Condition of type lisp::package-locked-error]
Restarts:
0: [continue ] Ignore the lock and continue
1: [unlock-package] Disable package's lock then continue
2: [unlock-all ] Unlock all packages, then continue
3: [abort ] Return to Top-Level.
\end{verbatim}
}
The \code{COMMON-LISP} package and the \cmucl{}-specific
implementation packages are locked on startup. Users can lock their
own packages by using the \code{ext:package-lock} and
\code{ext:package-definition-lock} accessors.
\subsection{Disabling package locks}
A package's locks can be enabled or disabled by using the
\code{ext:package-lock} and \code{ext:package-definition-lock}
accessors, as follows:
\begin{lisp}
(setf (ext:package-lock (find-package "UNIX")) nil)
(setf (ext:package-definition-lock (find-package "UNIX")) nil)
\end{lisp}
\begin{defun}{ext:}{package-lock}{\var{package}}
This function is an accessor for a package's structural lock, which
protects it against modifications to its list of exported symbols.
\end{defun}
\begin{defun}{ext:}{package-definition-lock}{\var{package}}
This function is an accessor for a package's definition-lock, which
protects symbols in that package from redefinition. As well as
protecting the symbol's fdefinition from change, attempts to change
the symbol's definition using \code{defstruct}, \code{defclass} or
\code{deftype} will be trapped.
\end{defun}
\begin{defmac}{ext:}{without-package-locks}{\args{\amprest{} \var{body}}}
This macro can be used to execute forms with all package locks (both
structure and definition locks) disabled.
\end{defmac}
\begin{defun}{ext:}{unlock-all-packages}{}
This function disables both structure and definition locks on all
currently defined packages. Note that package locks are reset when
\cmucl{} is restarted, so the effect of this function is limited to
the current session.
\end{defun}
% EOF

BIN
doc/cmu-user/previous_motif.gif Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 317 B

575
doc/cmu-user/serve-event.tex Normal file
View file

@ -0,0 +1,575 @@
\chapter{Event Dispatching with SERVE-EVENT}
\label{serve-event}
\credits{by Bill Chiles and Robert MacLachlan}
It is common to have multiple activities simultaneously operating in the same
Lisp process. Furthermore, Lisp programmers tend to expect a flexible
development environment. It must be possible to load and modify application
programs without requiring modifications to other running programs. \cmucl{}
achieves this by having a central scheduling mechanism based on an
event-driven, object-oriented paradigm.
An \var{event} is some interesting happening that should cause the Lisp process
to wake up and do something. These events include X events and activity on
Unix file descriptors. The object-oriented mechanism is only available with
the first two, and it is optional with X events as described later in this
chapter. In an X event, the window ID is the object capability and the X event
type is the operation code. The Unix file descriptor input mechanism simply
consists of an association list of a handler to call when input shows up on a
particular file descriptor.
\section{Object Sets}
\label{object-sets}
\cindex{object sets}
An {\em object set} is a collection of objects that have the same implementation
for each operation. Externally the object is represented by the object
capability and the operation is represented by the operation code. Within
Lisp, the object is represented by an arbitrary Lisp object, and the
implementation for the operation is represented by an arbitrary Lisp function.
The object set mechanism maintains this translation from the external to the
internal representation.
\begin{defun}{system:}{make-object-set}{%
\args{\var{name} \ampoptional{} \var{default-handler}}}
This function makes a new object set. \var{Name} is a string used
only for purposes of identifying the object set when it is printed.
\var{Default-handler} is the function used as a handler when an
undefined operation occurs on an object in the set. You can define
operations with the \code{serve-}\var{operation} functions exported
the \code{extensions} package for X events
(\pxlref{x-serve-mumbles}). Objects are added with
\code{system:add-xwindow-object}. Initially the object set has no
objects and no defined operations.
\end{defun}
\begin{defun}{system:}{object-set-operation}{%
\args{\var{object-set} \var{operation-code}}}
This function returns the handler function that is the
implementation of the operation corresponding to
\var{operation-code} in \var{object-set}. When set with
\code{setf}, the setter function establishes the new handler. The
\code{serve-}\var{operation} functions exported from the
\code{extensions} package for X events (\pxlref{x-serve-mumbles})
call this on behalf of the user when announcing a new operation for
an object set.
\end{defun}
\begin{defun}{system:}{add-xwindow-object}{%
\args{\var{window} \var{object} \var{object-set}}}
These functions add \var{port} or \var{window} to \var{object-set}.
\var{Object} is an arbitrary Lisp object that is associated with the
\var{port} or \var{window} capability. \var{Window} is a CLX
window. When an event occurs, \code{system:serve-event} passes
\var{object} as an argument to the handler function.
\end{defun}
\section{The SERVE-EVENT Function}
The \code{system:serve-event} function is the standard way for an application
to wait for something to happen. For example, the Lisp system calls
\code{system:serve-event} when it wants input from X or a terminal stream.
The idea behind \code{system:serve-event} is that it knows the appropriate
action to take when any interesting event happens. If an application calls
\code{system:serve-event} when it is idle, then any other applications with
pending events can run. This allows several applications to run ``at the
same time'' without interference, even though there is only one thread of
control. Note that if an application is waiting for input of any kind,
then other applications will get events.
\begin{defun}{system:}{serve-event}{\args{\ampoptional{} \var{timeout}}}
This function waits for an event to happen and then dispatches to
the correct handler function. If specified, \var{timeout} is the
number of seconds to wait before timing out. A time out of zero
seconds is legal and causes \code{system:serve-event} to poll for
any events immediately available for processing.
\code{system:serve-event} returns \true{} if it serviced at least
one event, and \nil{} otherwise. Depending on the application, when
\code{system:serve-event} returns \true, you might want to call it
repeatedly with a timeout of zero until it returns \nil.
If input is available on any designated file descriptor, then this
calls the appropriate handler function supplied by
\code{system:add-fd-handler}.
Since events for many different applications may arrive
simultaneously, an application waiting for a specific event must
loop on \code{system:serve-event} until the desired event happens.
Since programs such as \hemlock{} call \code{system:serve-event} for
input, applications usually do not need to call
\code{system:serve-event} at all; \hemlock{} allows other
application's handlers to run when it goes into an input wait.
\end{defun}
\begin{defun}{system:}{serve-all-events}{\args{\ampoptional{} \var{timeout}}}
This function is similar to \code{system:serve-event}, except it
serves all the pending events rather than just one. It returns
\true{} if it serviced at least one event, and \nil{} otherwise.
\end{defun}
\section{Using SERVE-EVENT with Unix File Descriptors}
Object sets are not available for use with file descriptors, as there are
only two operations possible on file descriptors: input and output.
Instead, a handler for either input or output can be registered with
\code{system:serve-event} for a specific file descriptor. Whenever any input
shows up, or output is possible on this file descriptor, the function
associated with the handler for that descriptor is funcalled with the
descriptor as it's single argument.
\begin{defun}{system:}{add-fd-handler}{%
\args{\var{fd} \var{direction} \var{function}}}
This function installs and returns a new handler for the file
descriptor \var{fd}. \var{direction} can be either \kwd{input} if
the system should invoke the handler when input is available or
\kwd{output} if the system should invoke the handler when output is
possible. This returns a unique object representing the handler,
and this is a suitable argument for \code{system:remove-fd-handler}
\var{function} must take one argument, the file descriptor.
\end{defun}
\begin{defun}{system:}{remove-fd-handler}{\args{\var{handler}}}
This function removes \var{handler}, that \code{add-fd-handler} must
have previously returned.
\end{defun}
\begin{defmac}{system:}{with-fd-handler}{%
\args{(\var{fd} \var{direction} \var{function})
\mstar{\var{form}}}}
This macro executes the supplied forms with a handler installed
using \var{fd}, \var{direction}, and \var{function}. See
\code{system:add-fd-handler}. The given forms are wrapped in an
\code{unwind-protect}; the handler is removed (see
\code{system:remove-fd-handler}) when done.
\end{defmac}
\begin{defun}{system:}{wait-until-fd-usable}{%
\args{\var{fd} \var{direction} \ampoptional{} \var{timeout}}}
This function waits for up to \var{timeout} seconds for \var{fd} to
become usable for \var{direction} (either \kwd{input} or
\kwd{output}). If \var{timeout} is \nil{} or unspecified, this
waits forever.
\end{defun}
\begin{defun}{system:}{invalidate-descriptor}{\args{\var{fd}}}
This function removes all handlers associated with \var{fd}. This
should only be used in drastic cases (such as I/O errors, but not
necessarily EOF). Normally, you should use \code{remove-fd-handler}
to remove the specific handler.
\end{defun}
\section{Using SERVE-EVENT with the CLX Interface to X}
\label{x-serve-mumbles}
Remember from section \ref{object-sets}, an object set is a collection of
objects, CLX windows in this case, with some set of operations, event keywords,
with corresponding implementations, the same handler functions. Since X allows
multiple display connections from a given process, you can avoid using object
sets if every window in an application or display connection behaves the same.
If a particular X application on a single display connection has windows that
want to handle certain events differently, then using object sets is a
convenient way to organize this since you need some way to map the window/event
combination to the appropriate functionality.
The following is a discussion of functions exported from the \code{extensions}
package that facilitate handling CLX events through \code{system:serve-event}.
The first two routines are useful regardless of whether you use
\code{system:serve-event}:
\begin{defun}{ext:}{open-clx-display}{%
\args{\ampoptional{} \var{string}}}
This function parses \var{string} for an X display specification
including display and screen numbers. \var{String} defaults to the
following:
\begin{example}
(cdr (assoc :display ext:*environment-list* :test #'eq))
\end{example}
If any field in the display specification is missing, this signals
an error. \code{ext:open-clx-display} returns the CLX display and
screen.
\end{defun}
\begin{defun}{ext:}{flush-display-events}{\args{\var{display}}}
This function flushes all the events in \var{display}'s event queue
including the current event, in case the user calls this from within
an event handler.
\end{defun}
\subsection{Without Object Sets}
Since most applications that use CLX, can avoid the complexity of object sets,
these routines are described in a separate section. The routines described in
the next section that use the object set mechanism are based on these
interfaces.
\begin{defun}{ext:}{enable-clx-event-handling}{%
\args{\var{display} \var{handler}}}
This function causes \code{system:serve-event} to notice when there
is input on \var{display}'s connection to the X11 server. When this
happens, \code{system:serve-event} invokes \var{handler} on
\var{display} in a dynamic context with an error handler bound that
flushes all events from \var{display} and returns. By returning,
the error handler declines to handle the error, but it will have
cleared all events; thus, entering the debugger will not result in
infinite errors due to streams that wait via
\code{system:serve-event} for input. Calling this repeatedly on the
same \var{display} establishes \var{handler} as a new handler,
replacing any previous one for \var{display}.
\end{defun}
\begin{defun}{ext:}{disable-clx-event-handling}{\args{\var{display}}}
This function undoes the effect of
\code{ext:enable-clx-event-handling}.
\end{defun}
\begin{defmac}{ext:}{with-clx-event-handling}{%
\args{(\var{display} \var{handler}) \mstar{form}}}
This macro evaluates each \var{form} in a context where
\code{system:serve-event} invokes \var{handler} on \var{display}
whenever there is input on \var{display}'s connection to the X
server. This destroys any previously established handler for
\var{display}.
\end{defmac}
\subsection{With Object Sets}
This section discusses the use of object sets and
\code{system:serve-event} to handle CLX events. This is necessary
when a single X application has distinct windows that want to handle
the same events in different ways. Basically, you need some way of
asking for a given window which way you want to handle some event
because this event is handled differently depending on the window.
Object sets provide this feature.
For each CLX event-key symbol-name \i{XXX} (for example,
\var{key-press}), there is a function \code{serve-}\i{XXX} of two
arguments, an object set and a function. The \code{serve-}\i{XXX}
function establishes the function as the handler for the \kwd{XXX}
event in the object set. Recall from section \ref{object-sets},
\code{system:add-xwindow-object} associates some Lisp object with a
CLX window in an object set. When \code{system:serve-event} notices
activity on a window, it calls the function given to
\code{ext:enable-clx-event-handling}. If this function is
\code{ext:object-set-event-handler}, it calls the function given to
\code{serve-}\i{XXX}, passing the object given to
\code{system:add-xwindow-object} and the event's slots as well as a
couple other arguments described below.
To use object sets in this way:
\begin{itemize}
\item Create an object set.
\item Define some operations on it using the \code{serve-}\i{XXX}
functions.
\item Add an object for every window on which you receive requests.
This can be the CLX window itself or some structure more meaningful
to your application.
\item Call \code{system:serve-event} to service an X event.
\end{itemize}
\begin{defun}{ext:}{object-set-event-handler}{%
\args{\var{display}}}
This function is a suitable argument to
\code{ext:enable-clx-event-handling}. The actual event handlers
defined for particular events within a given object set must take an
argument for every slot in the appropriate event. In addition to
the event slots, \code{ext:object-set-event-handler} passes the
following arguments:
\begin{itemize}
\item The object, as established by
\code{system:add-xwindow-object}, on which the event occurred.
\item event-key, see \code{xlib:event-case}.
\item send-event-p, see \code{xlib:event-case}.
\end{itemize}
Describing any \code{ext:serve-}\var{event-key-name} function, where
\var{event-key-name} is an event-key symbol-name (for example,
\code{ext:serve-key-press}), indicates exactly what all the
arguments are in their correct order.
%% \begin{comment}
%% \code{ext:object-set-event-handler} ignores \kwd{no-exposure}
%% events on pixmaps, issuing a warning if one occurs. It is only
%% prepared to dispatch events for windows.
%% \end{comment}
When creating an object set for use with
\code{ext:object-set-event-handler}, specify
\code{ext:default-clx-event-handler} as the default handler for
events in that object set. If no default handler is specified, and
the system invokes the default default handler, it will cause an
error since this function takes arguments suitable for handling port
messages.
\end{defun}
\section{A SERVE-EVENT Example}
This section contains two examples using \code{system:serve-event}. The first
one does not use object sets, and the second, slightly more complicated one
does.
\subsection{Without Object Sets Example}
This example defines an input handler for a CLX display connection. It only
recognizes \kwd{key-press} events. The body of the example loops over
\code{system:serve-event} to get input.
\begin{lisp}
(in-package "SERVER-EXAMPLE")
(defun my-input-handler (display)
(xlib:event-case (display :timeout 0)
(:key-press (event-window code state)
(format t "KEY-PRESSED (Window = ~D) = ~S.~%"
(xlib:window-id event-window)
;; See Hemlock Command Implementor's Manual for convenient
;; input mapping function.
(ext:translate-character display code state))
;; Make XLIB:EVENT-CASE discard the event.
t)))
\end{lisp}
\begin{lisp}
(defun server-example ()
"An example of using the SYSTEM:SERVE-EVENT function and object sets to
handle CLX events."
(let* ((display (ext:open-clx-display))
(screen (display-default-screen display))
(black (screen-black-pixel screen))
(white (screen-white-pixel screen))
(window (create-window :parent (screen-root screen)
:x 0 :y 0 :width 200 :height 200
:background white :border black
:border-width 2
:event-mask
(xlib:make-event-mask :key-press))))
;; Wrap code in UNWIND-PROTECT, so we clean up after ourselves.
(unwind-protect
(progn
;; Enable event handling on the display.
(ext:enable-clx-event-handling display #'my-input-handler)
;; Map the windows to the screen.
(map-window window)
;; Make sure we send all our requests.
(display-force-output display)
;; Call serve-event for 100,000 events or immediate timeouts.
(dotimes (i 100000) (system:serve-event)))
;; Disable event handling on this display.
(ext:disable-clx-event-handling display)
;; Get rid of the window.
(destroy-window window)
;; Pick off any events the X server has already queued for our
;; windows, so we don't choke since SYSTEM:SERVE-EVENT is no longer
;; prepared to handle events for us.
(loop
(unless (deleting-window-drop-event *display* window)
(return)))
;; Close the display.
(xlib:close-display display))))
(defun deleting-window-drop-event (display win)
"Check for any events on win. If there is one, remove it from the
event queue and return t; otherwise, return nil."
(xlib:display-finish-output display)
(let ((result nil))
(xlib:process-event
display :timeout 0
:handler #'(lambda (&key event-window &allow-other-keys)
(if (eq event-window win)
(setf result t)
nil)))
result))
\end{lisp}
\subsection{With Object Sets Example}
This example involves more work, but you get a little more for your effort. It
defines two objects, \code{input-box} and \code{slider}, and establishes a
\kwd{key-press} handler for each object, \code{key-pressed} and
\code{slider-pressed}. We have two object sets because we handle events on the
windows manifesting these objects differently, but the events come over the
same display connection.
\begin{lisp}
(in-package "SERVER-EXAMPLE")
(defstruct (input-box (:print-function print-input-box)
(:constructor make-input-box (display window)))
"Our program knows about input-boxes, and it doesn't care how they
are implemented."
display ; The CLX display on which my input-box is displayed.
window) ; The CLX window in which the user types.
;;;
(defun print-input-box (object stream n)
(declare (ignore n))
(format stream "#<Input-Box ~S>" (input-box-display object)))
(defvar *input-box-windows*
(system:make-object-set "Input Box Windows"
#'ext:default-clx-event-handler))
(defun key-pressed (input-box event-key event-window root child
same-screen-p x y root-x root-y modifiers time
key-code send-event-p)
"This is our :key-press event handler."
(declare (ignore event-key root child same-screen-p x y
root-x root-y time send-event-p))
(format t "KEY-PRESSED (Window = ~D) = ~S.~%"
(xlib:window-id event-window)
;; See Hemlock Command Implementor's Manual for convenient
;; input mapping function.
(ext:translate-character (input-box-display input-box)
key-code modifiers)))
;;;
(ext:serve-key-press *input-box-windows* #'key-pressed)
\end{lisp}
\begin{lisp}
(defstruct (slider (:print-function print-slider)
(:include input-box)
(:constructor %make-slider
(display window window-width max)))
"Our program knows about sliders too, and these provide input values
zero to max."
bits-per-value ; bits per discrete value up to max.
max) ; End value for slider.
;;;
(defun print-slider (object stream n)
(declare (ignore n))
(format stream "#<Slider ~S 0..~D>"
(input-box-display object)
(1- (slider-max object))))
;;;
(defun make-slider (display window max)
(%make-slider display window
(truncate (xlib:drawable-width window) max)
max))
(defvar *slider-windows*
(system:make-object-set "Slider Windows"
#'ext:default-clx-event-handler))
(defun slider-pressed (slider event-key event-window root child
same-screen-p x y root-x root-y modifiers time
key-code send-event-p)
"This is our :key-press event handler for sliders. Probably this is
a mouse thing, but for simplicity here we take a character typed."
(declare (ignore event-key root child same-screen-p x y
root-x root-y time send-event-p))
(format t "KEY-PRESSED (Window = ~D) = ~S --> ~D.~%"
(xlib:window-id event-window)
;; See Hemlock Command Implementor's Manual for convenient
;; input mapping function.
(ext:translate-character (input-box-display slider)
key-code modifiers)
(truncate x (slider-bits-per-value slider))))
;;;
(ext:serve-key-press *slider-windows* #'slider-pressed)
\end{lisp}
\begin{lisp}
(defun server-example ()
"An example of using the SYSTEM:SERVE-EVENT function and object sets to
handle CLX events."
(let* ((display (ext:open-clx-display))
(screen (display-default-screen display))
(black (screen-black-pixel screen))
(white (screen-white-pixel screen))
(iwindow (create-window :parent (screen-root screen)
:x 0 :y 0 :width 200 :height 200
:background white :border black
:border-width 2
:event-mask
(xlib:make-event-mask :key-press)))
(swindow (create-window :parent (screen-root screen)
:x 0 :y 300 :width 200 :height 50
:background white :border black
:border-width 2
:event-mask
(xlib:make-event-mask :key-press)))
(input-box (make-input-box display iwindow))
(slider (make-slider display swindow 15)))
;; Wrap code in UNWIND-PROTECT, so we clean up after ourselves.
(unwind-protect
(progn
;; Enable event handling on the display.
(ext:enable-clx-event-handling display
#'ext:object-set-event-handler)
;; Add the windows to the appropriate object sets.
(system:add-xwindow-object iwindow input-box
*input-box-windows*)
(system:add-xwindow-object swindow slider
*slider-windows*)
;; Map the windows to the screen.
(map-window iwindow)
(map-window swindow)
;; Make sure we send all our requests.
(display-force-output display)
;; Call server for 100,000 events or immediate timeouts.
(dotimes (i 100000) (system:serve-event)))
;; Disable event handling on this display.
(ext:disable-clx-event-handling display)
(delete-window iwindow display)
(delete-window swindow display)
;; Close the display.
(xlib:close-display display))))
\end{lisp}
\begin{lisp}
(defun delete-window (window display)
;; Remove the windows from the object sets before destroying them.
(system:remove-xwindow-object window)
;; Destroy the window.
(destroy-window window)
;; Pick off any events the X server has already queued for our
;; windows, so we don't choke since SYSTEM:SERVE-EVENT is no longer
;; prepared to handle events for us.
(loop
(unless (deleting-window-drop-event display window)
(return))))
(defun deleting-window-drop-event (display win)
"Check for any events on win. If there is one, remove it from the
event queue and return t; otherwise, return nil."
(xlib:display-finish-output display)
(let ((result nil))
(xlib:process-event
display :timeout 0
:handler #'(lambda (&key event-window &allow-other-keys)
(if (eq event-window win)
(setf result t)
nil)))
result))
\end{lisp}

21
doc/cmu-user/simple-streams.tex Normal file
View file

@ -0,0 +1,21 @@
\section{Simple Streams}
\cindex{simple-streams}
\label{simple-streams}
\cmucl{} includes a partial implementation of \emph{Simple Streams}, a
protocol that allows user-extensible streams\footnote{This
implementation was donated by Paul Foley}. The protocol was proposed
by Franz, Inc. and is intended to replace the \emph{Gray Streams}
method of extending streams. Simple streams are distributed as a
\cmucl{} subsystem, that can be loaded into the image by saying
\begin{lisp}
(require :simple-streams)
\end{lisp}
Note that CMUCL's implementation of simple streams is incomplete, and
in particular is currently missing support for the functions
\code{read-sequence} and \code{write-sequence}. Please consult the
\textit{Allegro Common Lisp} documentation for more information on
simple streams.

2364
doc/cmu-user/typehtml.dtx Normal file
View file

File diff suppressed because it is too large Load diff

3
doc/cmu-user/typehtml.ins Normal file
View file

@ -0,0 +1,3 @@
\def\batchfile{typehtml.ins}
\input docstrip
\generateFile{typehtml.sty}{f}{\from{typehtml.dtx}{package}}

899
doc/cmu-user/typehtml.sty Normal file
View file

@ -0,0 +1,899 @@
%%
%% This is file `typehtml.sty',
%% generated with the docstrip utility.
%%
%% The original source files were:
%%
%% typehtml.dtx (with options: `package')
%%
%% IMPORTANT NOTICE:
%%
%% For the copyright see the source file.
%%
%% Any modified versions of this file must be renamed
%% with new filenames distinct from typehtml.sty.
%%
%% For distribution of the original source see the terms
%% for copying and modification in the file typehtml.dtx.
%%
%% This generated file may be distributed as long as the
%% original source files, as listed above, are part of the
%% same distribution. (The sources need not necessarily be
%% in the same archive or directory.)
%%
%% Source File `typehtml.dtx'.
%% Copyright (C) 1996 1997 David Carlisle
%% This file may be distributed under the terms of the LPPL.
%% See 00readme.txt for details.
%%
\NeedsTeXFormat{LaTeX2e}[1995/06/01]
\ProvidesPackage{typehtml}
[1997/11/19 v0.12 HTML printer (DPC)]
\DeclareOption{html2}{\let\HTML@two@stop\endinput}
\DeclareOption{netscape}
{\def\HTML@not#1{\SGML@w{<#1> is not valid HTML}}}
\DeclareOption{html3}{\let\HTML@two@stop\relax}
\DeclareOption{nohyperref}{%
\let\HTML@doname\@secondoftwo
\def\HTML@dosrc#1#2{\emph{#2}}}
\DeclareOption{ftnhyperref}{%
\let\HTML@doname\@secondoftwo
\def\HTML@dosrc#1#2{\emph{#2}\footnote{HREF: \texttt{#1}}}}
\DeclareOption{hyperref}{%
\AtBeginDocument{%
\providecommand\href[2]{\special{html:<A href="#1">}%
#2\special{html:</A>}}%
\providecommand\hypertarget[2]{\special{html:<A name="#1">}%
#2\special{html:</A>}}%
\let\HTML@doname\hypertarget
\let\HTML@dosrc\href}}
\DeclareOption{dviwindo}{%
\def\HTML@dosrc#1#2{{%
\leavevmode\sbox\z@{#2}\count@\ht\z@\@tempcnta\wd\z@
\if\string##\@car#1\@nil
\special{button: \the\@tempcnta\space\the\count@\space
"\@gobble#1"}%
\else
\special{button: \the\@tempcnta\space\the\count@\space
launch: typehtml "#1"}
\fi
\special{color push}\special{color rgb 0 1 0}%
\unhbox\z@
\special{color pop}}}%
\def\HTML@doname#1#2{\leavevmode\special{mark: "#1"}#2}}%
\DeclareOption{imgalt}{}
\DeclareOption{imggif}{\SGML@w{img gif support not done yet}}
\DeclareOption{imgps}{\SGML@w{img ps support not done yet}}
\DeclareOption{smartquotedbl}{%
\def\SGMLquotedbla{%
\textquotedblleft\global\let\SGMLquotedbl\SGMLquotedblb}
\def\SGMLquotedblb{%
\textquotedblright\global\let\SGMLquotedbl\SGMLquotedbla}
\let\SGMLquotedbl\SGMLquotedbla
\let\SGML@savedeverypar\everypar
\newtoks\everypar
\SGML@savedeverypar{%
\global\let\SGMLquotedbl\SGMLquotedbla\the\everypar}}
\DeclareOption{straightquotedbl}{%
\DeclareTextCommandDefault{\textquotedbl}{{\ttfamily\char`\"}}%
\let\SGMLquotedbl\textquotedbl}
\DeclareOption{chapter}{%
\def\HTML@headings{%
\chapter\section\subsection%
\subsubsection\paragraph\subparagraph}}
\DeclareOption{chapter*}{%
\def\HTML@headings{%
{\chapter*}{\section*}{\subsection*}%
{\subsubsection*}{\paragraph*}{\subparagraph*}}}
\DeclareOption{section}{%
\def\HTML@headings{%
\section\subsection%
\subsubsection\paragraph\subparagraph\endgraf}}
\DeclareOption{section*}{%
\def\HTML@headings{%
{\section*}{\subsection*}%
{\subsubsection*}{\paragraph*}{\subparagraph*}\endgraf}}
\DeclareOption{subsection}{%
\def\HTML@headings{%
\subsection%
\subsubsection\paragraph\subparagraph\endgraf\endgraf}}
\DeclareOption{subsection*}{%
\def\HTML@headings{%
{\subsection*}%
{\subsubsection*}{\paragraph*}{\subparagraph*}\endgraf\endgraf}}
\DeclareOption{bigint}{%
\let\HTML@int\int
\AtEndOfPackage{\RequirePackage{exscale}}}
\ExecuteOptions{section*,imgalt,html2,nohyperref,straightquotedbl}
\ProcessOptions
\begingroup
\catcode`\<=\active
\catcode`\>=\active
\catcode`\&=\active
\catcode`\$=\active
\catcode`\"=\active
\catcode`\^=\active
\catcode`\_=\active
\catcode`\;=\active
\catcode`\A=\active
\catcode`\B=\active
\catcode`\C=\active
\catcode`\D=\active
\uccode`\A=`\{%
\uccode`\B=`\}%
\uccode`\C=`\|%
\uccode`\D=`\\%
\uppercase{\endgroup
\def\SGMLent@@#1;{\csname SGML@E@#1\endcsname}
\def\SGML@def@active#1>{%
\expandafter\def\csname SGML@#1\endcsname}
\def\dohtml{%
\begingroup
\ifx;\@undefined\expandafter\let\expandafter;\string;\fi
\ifx>\@undefined\expandafter\let\expandafter>\string>\fi
\catcode`\<=\active
\catcode`\>=\active
\catcode`\&=\active
\catcode`\{=\active
\catcode`\}=\active
\catcode`\$=\active
\catcode`\"=\active
\catcode`\^=\active
\catcode`\_=\active
\catcode`\\=\active
\catcode`\|=\active
\catcode\endlinechar=10
\catcode`\%=12
\catcode`\#=12
\catcode`\;=\active
\def\verbatim@nolig@list{\do\`\do\,\do\'\do\-}
\def<{\SGMLopen}%
\def&{\SGMLent}%
\let^\textasciicircum
\let~\textasciitilde
\def_{\_}%
\let$\$%
\def"{\SGMLquotedbl}%
\def A{\{}%
\def B{\}}%
\def C{\texttt{|}}%
\def D{\texttt{\char`\\}}%
\def\addcontentsline##1##2##3{%
{\def<{\string<}\def&{\string&}%
\addtocontents{##1}{\protect\dotochtml<html>}%
\addtocontents{##1}{\protect\contentsline{##2}{##3}{\thepage}}%
\addtocontents{##1}{\protect</html>}}}}
\def\dotochtml{%
\dohtml
\catcode`\\\z@
\catcode`\{\@ne
\catcode`\}\tw@}
\def\SGMLshortend{/}
\def\SGMLgrab@#1<#2>{%
\edef\@tempd{\lowercase{\def\noexpand\@tempd{\gobblespc#2 \relax}}}%
\@tempd
\ifx\@tempd\SGMLshortend\let\@tempd\@tempc\fi
\ifx\@tempb\@tempd
\advance\@tempcnta\@ne
\else
\ifx\@tempc\@tempd
\advance\@tempcnta\m@ne
\fi
\fi
\ifnum\@tempcnta=\z@
\expandafter\@tempa\expandafter{\the\@temptokena#1}%
\else
\addto@hook\@temptokena{#1<#2>}%
\expandafter\SGMLgrab@
\fi}
\def\SGMLopen#1>{%
\SGMLopen@#1 \@nil}
}
\def\htmlinput#1{\dohtml\let\@endhtml\relax\input{#1}\endgroup}
\def\gobblespc#1 #2\relax{#1}
\def\SGMLgrabber#1#2{%
\def\@tempa{#2}%
\@tempcnta\@ne
\@temptokena{}%
\lowercase{\def\@tempb{#1}\def\@tempc{/#1}}%
\SGMLgrab@}
\begingroup
\catcode`\"=\active
\uppercase{\endgroup
\def\SGMLopen@#1 #2\@nil{%
\toks@{}%
\edef\@tempa{\lowercase{\def\noexpand\SGMLelement{#1}}}\@tempa
\if!\@car#1\relax\@nil
\toks@{#1 #2}%
\SGML@w{Declaration ignored\MessageBreak<\the\toks@>\MessageBreak}%
\else
\if$#2$\else
\replacequotes#2"\@nil"%
\SGMLafterfi
\expandafter\toks@\expandafter{\expandafter}%
\expandafter\SGMLgetattrib\the\toks@ \@nil
\fi
\expandafter\ifx\csname SGML@\SGMLelement
\expandafter\endcsname\relax
\SGML@w{<\SGMLelement> undefined}%
\else
\csname SGML@\SGMLelement
\expandafter\expandafter\expandafter\endcsname
\fi
\fi}
\def\replacequotes#1"#2"{%
\def\@tempb{#2}%
\ifx\@tempb\@nnil
\addto@hook\toks@{#1}%
\else
\addto@hook\toks@{#1{#2}}%
\expandafter\replacequotes
\fi}}
\def\SGMLafterfi#1\fi{\fi#1}
\def\SGMLgobbletofi#1\fi{\fi}
\def\SGMLgetattrib#1 #2{%
\ifx\box#1\box\else
\SGMLgetval#1=$=\@nil
\def\@tempa{#2}%
\ifx\@tempa\@nnil
\expandafter\SGMLgobbletofi
\else
\expandafter\SGMLafterfi
\fi
\SGMLgetattrib#2%
\fi}
\def\SGMLgetval#1=#2=#3\@nil{%
\ifcat$#2%
\lowercase{\SGML@addattrib\doimplied{#1}}%
\else
\lowercase{\SGML@addattrib{\do{#1}}}{#2}%
\fi}
\def\SGML@addattrib#1#2{\addto@hook\toks@{#1{#2}}}
\def\SGML@w{\PackageWarning{typehtml}}
\def\SGMLdef#1{%
\ifcat\noexpand#1\noexpand~%
\expandafter\SGML@def@active
\else
\expandafter\SGML@def
\fi}
\edef\@tempa{\def\noexpand\SGML@def##1\string>}\@tempa{%
\expandafter\def\csname SGML@#1\endcsname}
\expandafter\def\expandafter\SGMLent\expandafter{%
\expandafter\protect\csname& \endcsname}
\expandafter\def\csname& \endcsname{%
\futurelet\@let@token\SGMLent@}
\def\SGMLent@{%
\ifx\@let@token\@sptoken
\&%
\else
\expandafter\SGMLent@@
\fi}
\def\SGMLentity#1{%
\expandafter\def\csname SGML@E@#1\endcsname}
\SGMLdef<html>{}
\SGMLdef</html>{\@endhtml}
\let\@endhtml\endgroup
\SGMLdef<title>{\typeout{***TITLE***}\SGMLgrabber{title}\typeout}
\long\def\@tempa#1#2#3#4#5#6{%
\SGMLdef<h1>{\SGMLgrabber{h1}{\HTMLsection{#1}}}%
\SGMLdef<h2>{\SGMLgrabber{h2}{\HTMLsection{#2}}}%
\SGMLdef<h3>{\SGMLgrabber{h3}{\HTMLsection{#3}}}%
\SGMLdef<h4>{\SGMLgrabber{h4}{\HTMLsection{#4}}}%
\SGMLdef<h5>{\SGMLgrabber{h5}{\HTMLsection{#5}}}%
\SGMLdef<h6>{\SGMLgrabber{h6}{\HTMLsection{#6}}}}
\expandafter\@tempa\HTML@headings
\def\HTMLsection#1#2{#1{\ignorespaces#2\unskip}}
\SGMLdef<head>{}
\SGMLdef</head>{}
\SGMLdef<body>{}
\SGMLdef</body>{}
\SGMLdef<bodytext>{}
\SGMLdef</bodytext>{}
\SGMLdef<p>{\par}
\SGMLdef</p>{\par}
\SGMLdef<blockquote>{\begin{quote}}
\SGMLdef</blockquote>{\end{quote}}
\SGMLdef<address>{\begin{quote}}
\SGMLdef</address>{\end{quote}}
\SGMLdef<ul>{\begin{itemize}}
\SGMLdef</ul>{\end{itemize}}
\SGMLdef<ol>{\begin{enumerate}}
\SGMLdef</ol>{\end{enumerate}}
\SGMLdef<li>{\item}
\SGMLdef</li>{}
\SGMLdef<dl>{%
\let\do\dldo
\let\doimplied\dlimplied
\begin{description}\the\toks@}
\SGMLdef</dl>{\end{description}}
\def\dldo#1#2{%
\def\@tempa{compact}\def\@tempb{#1}%
\ifx\@tempa\@tempb
\itemsep\z@
\advance\@totalleftmargin-\leftmargin
\advance\linewidth\leftmargin
\itemindent-\labelsep
\leftmargin\z@
\parshape \@ne \@totalleftmargin \linewidth
\fi}
\def\dlimplied#1{\dldo{#1}\relax}
\def\itx#1{\item[#1]}
\SGMLdef<dt>{\begin{lrbox}\z@\bfseries\let\maybeenddt\enddt}
\SGMLdef</dt>{\maybeenddt}
\SGMLdef<dd>{\maybeenddt}
\SGMLdef</dd>{}
\def\enddt{\end{lrbox}\item[\unhbox\z@]}
\let\maybeenddt\relax
\SGMLdef<a>{\SGMLgrabber{a}\HTML@anchor}
\def\HTML@anchor#1{{%
\let\@tempa\@gobble
\def\_{\string_}%
\let\do\ado
\the\toks@
\@tempa{#1}}}
\def\ado#1#2{%
\def\@tempb{name}\def\@tempc{#1}%
\ifx\@tempb\@tempc
\let\@tempa\@firstofone
\def\@tempa{\HTML@doname{#2}}%
\else
\def\@tempa{\HTML@dosrc{#2}}%
\fi}
\SGMLdef<pre>{%
\par
\begingroup
\parindent\z@
\obeylines\verbatim@font\@noligs
\frenchspacing\@vobeyspaces}
\SGMLdef</pre>{\endgroup}
\SGMLdef<tt>{\SGMLgrabber{tt}\texttt}
\SGMLdef<b>{\SGMLgrabber{b}\textbf}
\SGMLdef<i>{\SGMLgrabber{i}\textit}
\SGMLdef<em>{\SGMLgrabber{em}\emph}
\SGMLdef<strong>{\SGMLgrabber{strong}\textbf}
\SGMLdef<code>{\SGMLgrabber{code}\texttt}
\SGMLdef<samp>{\SGMLgrabber{samp}\textsf}
\SGMLdef<kbd>{\SGMLgrabber{kbd}\texttt}
\SGMLdef<var>{\SGMLgrabber{var}\textit}
\SGMLdef<cite>{\SGMLgrabber{cite}\textit}
\SGMLdef<form>{\par\medskip}
\SGMLdef</form>{\par\medskip}
\SGMLdef<select>{%
\let\do\selectdo
\the\toks@\par
\begin{tabular}{|l|}%
\hline\@tempc\\\hline
\let\tabularnewline\relax
\ignorespaces}
\def\selectdo#1#2{%
\def\@tempa{name}\def\@tempb{#1}%
\ifx\@tempa\@tempb\def\@tempc{#2}\fi}
\SGMLdef</select>{\\\hline\end{tabular}}
\SGMLdef<option>{%
\gdef\optionbul{\phantom{$\bullet$}}%
\let\do\optiondo
\let\doimplied\optionimplied
\the\toks@
\tabularnewline
\let\tabularnewline\\%
\optionbul\space\ignorespaces}
\SGMLdef</option>{}
\def\optiondo#1#2{%
\def\@tempa{selected}\def\@tempb{#1}%
\ifx\@tempa\@tempb\gdef\optionbul{$\bullet$}\fi}
\def\optionimplied#1{%
\def\@tempa{selected}\def\@tempb{#1}%
\ifx\@tempa\@tempb\gdef\optionbul{$\bullet$}\fi}
\SGMLdef<input>{}
\SGMLdef<img>{{%
\let\do\imgdo
\def\@tempa{\doimage}%
\the\toks@
\@tempa}}
\def\doimage{\textsf{[image]}}
\def\imgdo#1{\csname img=#1\endcsname}
\expandafter\def\csname img=align\endcsname#1{%
\SGML@w{align=#1 ignored}}
\expandafter\def\csname img=src\endcsname#1{%
\SGML@w{src=#1 ignored}}
\expandafter\def\csname img=height\endcsname#1{%
\SGML@w{height=#1 ignored}}
\expandafter\def\csname img=alt\endcsname#1{%
\def\doimage{#1}}
\SGMLdef<hr>{\par\smallskip\hrule\smallskip}
\SGMLdef<br>{\leavevmode\\}
\SGMLdef<xmp>{%
\SGML@pre
\def\@tempb{/xmp}%
\let\SGMLopen\HTML@xmptest}
\SGMLdef<listing>{%
\SGML@xmp
\def\@tempb{/listing}}
\SGMLdef<plaintext>{%
\SGML@xmp
\def\@tempb{/plaintext}}%
\def\HTML@xmptest#1>{%
\lowercase{\def\@tempa{#1}}%
\ifx\@tempa\@tempb
\endgroup
\else
\SGMLafterfi
<#1>%
\fi}
\SGMLentity{amp}{\&}
\SGMLentity{lt}{\ensuremath{<}}
\SGMLentity{gt}{\ensuremath{>}}
\SGMLentity{aacute}{\'a}
\SGMLentity{Aacute}\'A{}
\SGMLentity{acirc}{\^a}
\SGMLentity{Acirc}{\^A}
\SGMLentity{agrave}{\`a}
\SGMLentity{Agrave}{\`A}
\SGMLentity{aring}{\r a}
\SGMLentity{Aring}{\r A}
\SGMLentity{atilde}{\~a}
\SGMLentity{Atilde}{\~A}
\SGMLentity{auml}{\"a}
\SGMLentity{Auml}{\"A}
\SGMLentity{aelig}{\ae}
\SGMLentity{AElig}{\AE}
\SGMLentity{ccedil}{\c c}
\SGMLentity{Ccedil}{\c C}
\SGMLentity{eth}{\dh}
\SGMLentity{ETH}{\DH}
\SGMLentity{eacute}{\'e}
\SGMLentity{Eacute}{\`E}
\SGMLentity{ecirc}{\^e}
\SGMLentity{Ecirc}{\^E}
\SGMLentity{egrave}{\`e}
\SGMLentity{Egrave}{\`E}
\SGMLentity{euml}{\"e}
\SGMLentity{Euml}{\"E}
\SGMLentity{iacute}{\'\i}
\SGMLentity{Iacute}{\'I}
\SGMLentity{icirc}{\^\i}
\SGMLentity{Icirc}{\^I}
\SGMLentity{igrave}{\`\i}
\SGMLentity{Igrave}{\`I}
\SGMLentity{iuml}{\"\i}
\SGMLentity{Iuml}{\"I}
\SGMLentity{ntilde}{\~n}
\SGMLentity{Ntilde}{\~N}
\SGMLentity{oacute}{\'o}
\SGMLentity{Oacute}{\'O}
\SGMLentity{ocirc}{\^o}
\SGMLentity{Ocirc}{\^O}
\SGMLentity{ograve}{\`o}
\SGMLentity{Ograve}{\`O}
\SGMLentity{oslash}{\oe}
\SGMLentity{Oslash}{\OE}
\SGMLentity{otilde}{\~o}
\SGMLentity{Otilde}{\~O}
\SGMLentity{ouml}{\"o}
\SGMLentity{Ouml}{\"O}
\SGMLentity{szlig}{\ss}
\SGMLentity{thorn}{\th}
\SGMLentity{THORN}{\TH}
\SGMLentity{uacute}{\'u}
\SGMLentity{Uacute}{\'U}
\SGMLentity{ucirc}{\^u}
\SGMLentity{Ucirc}{\^U}
\SGMLentity{ugrave}{\`u}
\SGMLentity{Ugrave}{\`U}
\SGMLentity{uuml}{\"u}
\SGMLentity{Uuml}{\"U}
\SGMLentity{yacute}{\'y}
\SGMLentity{Yacute}{\'Y}
\SGMLentity{yuml}{\"y}
\ifx\HTML@not\@undefined\else
\SGMLdef<center>{\HTML@not{center}\begin{center}}
\SGMLdef</center>{\end{center}}
\SGMLdef<blink>{\SGMLgrabber{blink}\textbf}
\SGMLdef<font>{\HTML@not{font}\begingroup
\let\do\fontdo\the\toks@}
\SGMLdef</font>{\endgroup}
\def\fontdo#1#2{%
\def\@tempa{size}\def\@tempb{#1}%
\ifx\@tempa\@tempb
\font@switch#2\relax\@nil
\fi}
\toks@\expandafter{\set@fontsize{#1}{#2}{#3}}
\edef\@tempa{%
\def\noexpand\set@fontsize##1##2##3{\the\toks@\noexpand\set@fontnum}}
\@tempa
\def\set@fontnum{\dimen@\f@size\p@
\dimen@\mul@ptsize\dimen@
\count@\dimen@ \divide\count@\p@
\advance\count@ -5\relax
\edef\@fontnum{\number\count@}}
\def\mul@ptsize{}%
\def\@fontnum{5}
\ifcase 0\@ptsize\relax
\global\let\mul@ptsize\@empty% case 0, ptsize = 10
\or\gdef\mul@ptsize{.9091}% case 1, ptsize = 11
\else\gdef\mul@ptsize{.8333}% case 2, ptsize = 12
\fi
\def\font@switch#1#2\@nil{\count@\@fontnum\relax
\ifx +#1\advance\else\ifx -#1\advance\fi\fi
\count@#1#2\relax
\ifcase\count@ \tiny\or \tiny\or \scriptsize
\or\footnotesize \or\small \or\normalsize \or\large
\or\Large \or\LARGE \or\huge \else\Huge \fi}
\fi
\HTML@two@stop
\SGML@w{HTML3 support not finished yet}
\SGMLdef<math>{\SGMLgrabber{math}\domath}
\SGMLdef<sup>{^\bgroup\HTMLscriptmap}
\SGMLdef</sup>{\egroup}
\SGMLdef<sub>{_\bgroup\HTMLscriptmap}
\SGMLdef</sub>{\egroup}
\SGMLdef<box>{\SGMLgrabber{box}\dobox}
\begingroup
\catcode`\<=\active
\catcode`\>=\active
\catcode`\&=\active
\catcode`\_=\active
\catcode`\^=\active
\catcode`\"=\active
\gdef\domath#1{%
{{\def&{\expandafter\expandafter\expandafter\noexpand\SGMLent@@}%
\let<\relax\let>\relax\let_\relax\let^\relax\let"\relax
\def\{{<box>}\def\}{</box>}%
\xdef\@gtempa{#1}}%
\let\do\mathdo
\let\doimplied\mathimplied
\the\toks@
\[%
\m@th\nulldelimiterspace\z@
\def^{<sup>}%
\def_{<sub>}%
\@gtempa\]}}
\gdef\HTMLscriptmap{%
\def^{</sup>}%
\def_{</sub>}}
\gdef\dobox#1{%
{\let\do\boxdo
\let\bigstrut\relax
\the\toks@
\def^{<sup>}%
\def_{<sub>}%
\lookleft@#1<left>\@nil}}
\gdef\lookleft@#1<left>#2\@nil{%
\if$#2$%
{\left.\bgroup#1\mayberight}%
\else
\lookbox@#1<box>\@nil#2\@nil
\fi}
\gdef\lookbox@#1<box>#2\@nil#3<left>\@nil{%
\if$#2$%
{\maybeleft#1\@nil#3\mayberight}
\else
{#1 \boxtofront#2 <left> #3}%
\fi}
\gdef\boxtofront#1<box>{<box>#1}
\endgroup
\def\mathdo#1#2{%
\def\@tempa{class-chem}\def\@tempb{#1-#2}%
\ifx\@tempa\@tempb
\everymath{\fam\z@}\everydisplay{\fam\z@}%
\fi}
\def\mathimplied#1{%
\def\@tempa{box}\def\@tempb{#1}%
\ifx\@tempa\@tempb
\def\[{\center\setbox\z@\hbox\bgroup$\displaystyle}%
\def\]{$\egroup\fbox{\box\z@}\endcenter}%
\fi}
\def\boxdo#1#2{%
\def\@tempa{size}\def\@tempb{#1}%
\ifx\@tempa\@tempb
\def\@tempb{#2}
\def\@tempa{normal}\ifx\@tempa\@tempb\def\@tempc{1}\fi
\def\@tempa{medium}\ifx\@tempa\@tempb\def\@tempc{2}\fi
\def\@tempa{large}\ifx\@tempa\@tempb\def\@tempc{3}\fi
\def\@tempa{huge}\ifx\@tempa\@tempb\def\@tempc{4}\fi
\edef\bigstrut{\vrule\@height\@tempc\ht\strutbox\@width\z@}
\fi}
\SGMLdef<left>{\left.\bgroup}
\def\mayberight{\egroup\bigstrut\right.}
\def\maybeleft#1#2\@nil{%
\in@{#1}{()[]\SGML@E@rbrace\SGML@E@lbrace}%
\ifin@
\left#1\bgroup#2%
\else
\let\SGML@E@int\HTML@bigint
#1#2\left.\bgroup\let\SGML@E@int\int
\fi}
\def\righttest#1{%
\in@{#1}{()[]\SGML@E@rbrace\SGML@E@lbrace}%
\ifin@
\right#1\let\mayberight\relax
\else
\right.\let\mayberight\relax\expandafter#1%
\fi}
\ifx\HTML@int\@undefined
\let\HTML@bigint\int
\else
\def\HTML@int{\int^{\box\tw@}_{\box4}}
\def\HTML@bigint#1\left.\bgroup{%
\def\@tempa{#1}%
\setbox\z@\hbox\bgroup
\aftergroup\HTMLafterbigint$\displaystyle\bgroup
\aftergroup$\aftergroup\egroup}
\def\HTMLafterbigint{%
\dimen@.5\ht\z@
\advance\dimen@.5\dp\z@
{\SGMLdef<sup>{\setbox\tw@\hbox\bgroup\HTMLscriptmap$\scriptstyle}%
\SGMLdef<sub>{\setbox4\hbox\bgroup\HTMLscriptmap$\scriptstyle}%
\SGMLdef</sup>{$\egroup}%
\SGMLdef</sub>{$\egroup}%
\setbox\tw@\box\voidb@x
\setbox4\box\voidb@x
\@tempa
\ifdim\dimen@>\f@size\p@
\mathop{\hbox{\DeclareFixedFont\@tempa{OMX}{cmex}{m}{n}\dimen@
$\displaystyle\textfont\thr@@\@tempa\HTML@int$}}%
\else
\HTML@int
\fi
}\left.\box\z@}
\fi
\SGMLdef<over>{\over}
\SGMLdef<atop>{\atop}
\SGMLdef<choose>{\choose}
\SGMLdef<right>{\egroup\bigstrut\righttest}
\SGMLdef<above>{\SGMLgrabber{above}%
{\let\@tempc\overlineop
\let\do\abovedo
\the\toks@
\@tempc}}
\SGMLdef<below>{\SGMLgrabber{below}%
{\let\@tempc\underlineop
\let\do\abovedo
\the\toks@
\@tempc}}
\def\overlineop#1{\mathop{\overline{#1}}}
\def\underlineop#1{\mathop{\underline{#1}}}
\def\abovedo#1#2{%
\def\@tempa{sym}\def\@tempb{#1}%
\ifx\@tempa\@tempb\def\@tempc{\csname#2\endcsname}\fi}
\SGMLdef<vec>{\SGMLgrabber{vec}\vec}
\SGMLdef<bar>{\SGMLgrabber{bar}\bar}
\SGMLdef<dot>{\SGMLgrabber{dot}\dot}
\SGMLdef<ddot>{\SGMLgrabber{ddot}\ddot}
\SGMLdef<hat>{\SGMLgrabber{hat}\hat}
\SGMLdef<tilde>{\SGMLgrabber{tilde}\tilde}
\SGMLdef<t>{\SGMLgrabber{t}\mathrm}
\SGMLdef<bt>{\SGMLgrabber{bt}\mathbf}
\SGMLdef<text>{\SGMLgrabber{text}\textnormal}%%%%% not in the dtd????
\SGMLdef<root>{\rootfudge}
\def\rootfudge#1{%
\setbox\rootbox\hbox\bgroup$\m@th\scriptscriptstyle\bgroup#1}
\SGMLdef<of>{\egroup$\egroup\SGMLgrabber{root}\offudge}
\SGMLdef</of>{}
\def\offudge#1{\mathpalette\r@@t{#1}}
\SGMLdef<sqrt>{\SGMLgrabber{sqrt}\sqrt}
\newcount\HTMLrow
\newcount\HTMLcol
\newcount\HTMLrowspan
\newcount\HTMLcolspan
\SGMLdef<array>{{\ifnum`}=0\fi
\let\do\arraydo
\let\doimplied\arrayimplied
\let\HTMLal.%
\let\HTMLar.%
\global\HTMLrow\z@
\let\HTMLabox\vcenter
\the\toks@
\setbox\z@\vbox\bgroup\halign\bgroup
\strut\span\HTMLacolspec\cr\nocr}
\def\HTMLacolspec{##&&##}
\def\HTMLamakepream#1{%
\let\HTMLacolspec\@empty
\let\@sharp\relax
\lowercase{\@tfor\@tempc:=#1}\do{%
\if\@tempc l%
\edef\HTMLacolspec{\HTMLacolspec\@sharp\hfill&}%
\else
\if\@tempc c%
\edef\HTMLacolspec{\HTMLacolspec\hfill\@sharp\hfill&}%
\else
\if\@tempc r%
\edef\HTMLacolspec{\HTMLacolspec\hfill\@sharp&}%
\else
\if\@tempc +%
\edef\HTMLacolspec{\HTMLacolspec$+$}%
\else
\if\@tempc -%
\edef\HTMLacolspec{\HTMLacolspec$-$}%
\else
\if\@tempc =%
\edef\HTMLacolspec{\HTMLacolspec$=$}%
\fi
\fi
\fi
\fi
\fi
\fi}%
\def\@sharp{########}%
\edef\HTMLacolspec{\HTMLacolspec&\@sharp}}
\SGMLdef</array>{\HTMLendarray}
\let\HTMLcr\cr
\def\HTMLendarray{%
\endi\crcr\egroup\egroup
\ifx\HTMLabox\vtop
\setbox\z@\vtop{\unvbox\z@}%
\else
\ifx\HTMLabox\vcenter
\dimen@\ht\z@
\advance\dimen@\dp\z@
\divide\dimen@\tw@
\advance\dimen@-\ht\z@
\setbox\z@\hbox{\raise\dimen@\box\z@}%
\fi
\fi
\dimen@=\ht\z@
\setbox\z@
\hbox{$\left\HTMLal\kern-1em\vcenter{\box\z@}\kern-1em\right\HTMLar$}%
\advance\dimen@-\ht\z@
\raise\dimen@\box\z@
\ifnum`{=0\fi}}
\def\HTMLendarraylabels{%
\endi\crcr\strut\cr\egroup\egroup
\setbox2=\vsplit\z@ to \baselineskip
\setbox\z@\vbox{\unvbox\z@\global\setbox\@ne\lastbox}%
\setbox4\hbox{\unhbox\@ne\unskip\global\setbox\@ne\lastbox}%
\vcenter{%
\box2
\hbox{$\kern\wd\@ne
\left\HTMLal\kern-\wd\@ne
\vcenter{\box\z@}%
\right\HTMLar$}}%
\ifnum`{=0\fi}}
\def\nocr{\relax\iffalse{\fi\let\HTMLcr\relax\iffalse}\fi}
\SGMLdef<row>{%
\endi\HTMLcr
\global\advance\HTMLrow\@ne
\global\HTMLcol\z@}
\SGMLdef</row>{}
\SGMLdef<item>{%
\let\do\itemdo
\gdef\@gtempa{\global\advance\HTMLcol\@ne}%
\gdef\@gtempb{}%
\gdef\@gtempc{}%
\global\HTMLcolspan\@ne
\the\toks@
\endi%
\@gtempc
\@gtempa
\spanifneeded
\@gtempb
\setbox\z@\hbox\bgroup$%
\def\endi{\unskip$\egroup%
\quad\HTMLaleft\box\z@\HTMLaright\quad&}%
\ignorespaces}
\def\spanifneeded{%
\edef\@tempa{\noexpand\in@{,\the\HTMLrow/\the\HTMLcol,}{\spanitems}}%
\@tempa
\ifin@
\@firstofone{&}\global\advance\HTMLcol\@ne
\expandafter\spanifneeded
\fi}
\SGMLdef</item>{}
\let\HTMLaleft\hfil
\let\HTMLaright\hfil
\let\endi\relax
\def\arraydo#1#2{%
\def\@tempa{#1}\def\@tempb{#2}%
\def\@tempc{align}%
\ifx\@tempa\@tempc
\def\@tempc{top}%
\ifx\@tempb\@tempc
\let\HTMLabox\vtop
\else
\def\@tempc{bottom}%
\ifx\@tempb\@tempc
\let\HTMLabox\vbox
\fi
\fi
\else
\def\@tempc{ldelim}%
\ifx\@tempa\@tempc
\let\HTMLal\@tempb
\else
\def\@tempc{rdelim}%
\ifx\@tempa\@tempc
\let\HTMLar\@tempb
\else
\def\@tempc{labels}%
\ifx\@tempa\@tempc
\let\HTMLendarray\HTMLendarraylabels
\else
\def\@tempc{colspec}%
\ifx\@tempa\@tempc
\HTMLamakepream{#2}%
\fi
\fi
\fi
\fi
\fi}
\def\arrayimplied#1{%
\def\@tempa{labels}\def\@tempb{#1}%
\ifx\@tempa\@tempb
\let\HTMLendarray\HTMLendarraylabels
\fi}
\def\itemdo#1#2{%
\def\@tempa{#1}\def\@tempb{#2}%
\def\@tempc{colspan}%
\ifx\@tempa\@tempc
\global\HTMLcolspan#2\relax
\gdef\@gtempa{\@multispan#2\relax\global\advance\HTMLcol#2\relax}%
\else
\def\@tempc{align}%
\ifx\@tempa\@tempc
\def\@tempc{left}%
\ifx\@tempb\@tempc
\gdef\@gtempb{\let\HTMLaleft\relax}%
\else
\def\@tempc{right}%
\ifx\@tempb\@tempc
\gdef\@gtempb{\let\HTMLaright\relax}%
\fi
\fi
\else
\def\@tempc{rowspan}%
\ifx\@tempa\@tempc
\global\HTMLrowspan#2\relax
\gdef\@gtempc{%
\@tempcnta=\HTMLrow
\advance\@tempcnta\HTMLrowspan
\loop
\@tempcntb=\HTMLcol
\advance\@tempcntb\HTMLcolspan
\advance\@tempcnta\m@ne
\ifnum\@tempcnta>\HTMLrow
{\loop
\xdef\spanitems{%
\spanitems\the\@tempcnta/\the\@tempcntb,}%
\advance\@tempcntb\m@ne
\ifnum\@tempcntb>\HTMLcol
\repeat}%
\repeat}%
\fi
\fi
\fi}
\def\spanitems{,}
\SGMLentity{thinsp}{\,}
\SGMLentity{emsp}{\quad}
\SGMLentity{alpha}{\alpha}
\SGMLentity{beta}{\beta}
\SGMLentity{gamma}{\gamma}
\SGMLentity{Gamma}{\Gamma}
\SGMLentity{int}{\int}
\SGMLentity{sum}{\sum}
\SGMLentity{lbrace}{\lbrace}
\SGMLentity{rbrace}{\rbrace}
\SGMLentity{times}{\times}
\SGMLentity{cup}{\cup}
\SGMLentity{cap}{\cap}
\SGMLentity{vee}{\vee}
\SGMLentity{wedge}{\wedge}
\SGMLentity{infty}{\infty}
\SGMLentity{oplus}{\oplus}
\SGMLentity{ominus}{\ominus}
\SGMLentity{otimes}{\otimes}
\SGMLentity{sin}{\sin}
\SGMLentity{cos}{\cos}
\SGMLentity{tan}{\tan}
\SGMLdef<table>{\begin{table}[htp]\centering\begin{tabular}{*{10}c}}
\SGMLdef</table>{\end{tabular}\end{table}}
\SGMLdef<tr>{\ifhmode\expandafter\\\fi\relax}
\SGMLdef</tr>{\\\relax}
\SGMLdef<td>{\ifvmode\else\expandafter\hiddenamp\fi}
\def\hiddenamp{&}
\SGMLdef</td>{}
\SGMLdef<caption>{\end{tabular}\begingroup\bfseries}
\SGMLdef</caption>{\endgroup\par\smallskip\begin{tabular}{*{10}{c}}}
\endinput
%%
%% End of file `typehtml.sty'.

733
doc/cmu-user/unicode.tex Normal file
View file

@ -0,0 +1,733 @@
\chapter{Internationalization}
\label{i18n}
\cindex{Internationalization}
\cmucl{} supports internationalization by supporting Unicode
characters internally and by adding support for external formats to
convert from the internal format to an appropriate external character
coding format.
To understand the support for Unicode, we refer the reader to the
\ifpdf
\href{http://www.unicode.org/}{Unicode standard}.
\else
\emph{Unicode standard} at \url{http://www.unicode.org}
\fi
\section{Changes}
To support internationalization, the following changes to Common Lisp
functions have been done.
\subsection{Design Choices}
To support Unicode, there are many approaches. One choice is to
support both 8-bit \code{base-char} and a 21-bit (or larger)
\code{character} since Unicode codepoints use 21 bits. This generally
means strings are much larger, and complicates the compiler by having
to support both \code{base-char} and \code{character} types and the
corresponding string types. This also adds complexity for the user to
understand the difference between the different string and character
types.
Another choice is to have just one character and string type that can
hold the entire Unicode codepoint. While simplifying the compiler and
reducing the burden on the user, this significantly increases memory
usage for strings.
The solution chosen by \cmucl{} is to tradeoff the size and complexity
by having only 16-bit characters. Most of the important languages can
be encoded using only 16-bits. The rest of the codepoints are for
rare languages or ancient scripts. Thus, the memory usage is
significantly reduced while still supporting the the most important
languages. Compiler complexity is also reduced since \code{base-char}
and \code{character} are the same as are the string types.. But we
still want to support the full Unicode character set. This is
achieved by making strings be UTF-16 strings internally. Hence, Lisp
strings are UTF-16 strings, and Lisp characters are UTF-16 code-units.
\subsection{Characters}
\label{sec:i18n:characters}
Characters are now 16 bits long instead of 8 bits, and \code{base-char}
and \code{character} types are the same. This difference is
naturally indicated by changing \code{char-code-limit} from 256 to
65536.
\subsection{Strings}
\label{sec:i18n:strings}
In \cmucl{} there is only one type of string---\code{base-string} and
\code{string} are the same.
Internally, the strings are encoded using UTF-16. This means that in
some rare cases the number of Lisp characters in a string is not the
same as the number of codepoints in the string.
\section{External Formats}
To be able to communicate to the external world, \cmucl{} supports
external formats to convert to and from the external world to
\cmucl{}'s string format. The external format is specified in several
ways. The standard streams \var{*standard-input*},
\var{*standard-output*}, and \var{*standard-error*} take the format
from the value specified by \var{*default-external-format*}. The
default value of \var{*default-external-format*} is \kwd{iso8859-1}.
For files, \code{OPEN} takes the \kwd{external-format}
parameter to specify the format. The default external format is
\kwd{default}.
\subsection{Available External Formats}
The available external formats are listed below in
Table~\ref{table:external-formats}. The first column gives the
external format, and the second column gives a list of aliases that
can be used for this format. The set of aliases can be changed by
changing the \file{aliases} file.
For all of these formats, if an illegal sequence is encountered, no
error or warning is signaled. Instead, the offending sequence is
silently replaced with the Unicode REPLACEMENT CHARACTER (U$+$FFFD).
\begin{table}
\centering
\begin{tabular}{|l|l|p{3in}|}
\hline
\textbf{Format} & \textbf{Aliases} & \textbf{Description} \\
\hline
\hline
\kwd{iso8859-1} & \kwd{latin1} \kwd{latin-1} \kwd{iso-8859-1} & ISO8859-1 \\
\hline
\kwd{iso8859-2} & \kwd{latin2} \kwd{latin-2} \kwd{iso-8859-2} & ISO8859-2 \\
\hline
\kwd{iso8859-3} & \kwd{latin3} \kwd{latin-3} \kwd{iso-8859-3} & ISO8859-3 \\
\hline
\kwd{iso8859-4} & \kwd{latin4} \kwd{latin-4} \kwd{iso-8859-4} & ISO8859-4 \\
\hline
\kwd{iso8859-5} & \kwd{cyrillic} \kwd{iso-8859-5} & ISO8859-5 \\
\hline
\kwd{iso8859-6} & \kwd{arabic} \kwd{iso-8859-6} & ISO8859-6 \\
\hline
\kwd{iso8859-7} & \kwd{greek} \kwd{iso-8859-7} & ISO8859-7 \\
\hline
\kwd{iso8859-8} & \kwd{hebrew} \kwd{iso-8859-8} & ISO8859-8 \\
\hline
\kwd{iso8859-9} & \kwd{latin5} \kwd{latin-5} \kwd{iso-8859-9} & ISO8859-9 \\
\hline
\kwd{iso8859-10} & \kwd{latin6} \kwd{latin-6} \kwd{iso-8859-10} & ISO8859-10 \\
\hline
\kwd{iso8859-13} & \kwd{latin7} \kwd{latin-7} \kwd{iso-8859-13} & ISO8859-13 \\
\hline
\kwd{iso8859-14} & \kwd{latin8} \kwd{latin-8} \kwd{iso-8859-14} & ISO8859-14 \\
\hline
\kwd{iso8859-15} & \kwd{latin9} \kwd{latin-9} \kwd{iso-8859-15} & ISO8859-15 \\
\hline
\kwd{utf-8} & \kwd{utf} \kwd{utf8} & UTF-8 \\
\hline
\kwd{utf-16} & \kwd{utf16} & UTF-16 with optional BOM \\
\hline
\kwd{utf-16-be} & \kwd{utf-16be} \kwd{utf16-be} & UTF-16 big-endian (without BOM) \\
\hline
\kwd{utf-16-le} & \kwd{utf-16le} \kwd{utf16-le} & UTF-16 little-endian (without BOM) \\
\hline
\kwd{utf-32} & \kwd{utf32} & UTF-32 with optional BOM \\
\hline
\kwd{utf-32-be} & \kwd{utf-32be} \kwd{utf32-be} & UTF-32 big-endian (without BOM) \\
\hline
\kwd{utf-32-le} & \kwd{utf-32le} \kwd{utf32-le} & UTF-32 little-endian (without BOM) \\
\hline
\kwd{cp1250} & & \\
\hline
\kwd{cp1251} & & \\
\hline
\kwd{cp1252} & \kwd{windows-1252} \kwd{windows-cp1252} \kwd{windows-latin1} & \\
\hline
\kwd{cp1253} & & \\
\hline
\kwd{cp1254} & & \\
\hline
\kwd{cp1255} & & \\
\hline
\kwd{cp1256} & & \\
\hline
\kwd{cp1257} & & \\
\hline
\kwd{cp1258} & & \\
\hline
\kwd{koi8-r} & & \\
\hline
\kwd{mac-cyrillic} & & \\
\hline
\kwd{mac-greek} & & \\
\hline
\kwd{mac-icelandic} & & \\
\hline
\kwd{mac-latin2} & & \\
\hline
\kwd{mac-roman} & & \\
\hline
\kwd{mac-turkish} & & \\
\hline
\end{tabular}
\caption{External formats}
\label{table:external-formats}
\end{table}
\subsection{Composing External Formats}
A composing external format is an external format that converts between
one codepoint and another, rather than between codepoints and octets.
A composing external format must be used in conjunction with another
(octet-producing) external format. This is specified by
using a list as the external format. For example, we can use
\code{'(\kwd{latin1} \kwd{crlf})} as the external format. In this
particular example, the external format is latin1, but whenever a
carriage-return/linefeed sequence is read, it is converted to the Lisp
\lispchar{Newline} character. Conversely, whenever a string is written,
a Lisp \lispchar{Newline} character is converted to a
carriage-return/linefeed sequence. Without the \kwd{crlf} composing
format, the carriage-return and linefeed will be read in as separate
characters, and on output the Lisp \lispchar{Newline} character is
output as a single linefeed character.
Table~\ref{table:composing-formats} lists the available composing formats.
\begin{table}
\centering
\begin{tabular}{|l|l|p{3in}|}
\hline
\textbf{Format} & \textbf{Aliases} & \textbf{Description} \\
\hline
\hline
\kwd{crlf} & \kwd{dos} & Composing format for converting to/from DOS (CR/LF)
end-of-line sequence to \lispchar{Newline}\\
\kwd{cr} & \kwd{mac} & Composing format for converting to/from DOS (CR/LF)
end-of-line sequence to \lispchar{Newline}\\
\hline
\kwd{beta-gk} & & Composing format that translates (lower-case) Beta
code (an ASCII encoding of ancient Greek) \\
\hline
\kwd{final-sigma} & & Composing format that attempts to detect sigma in
word-final position and change it from U+3C3 to U+3C2\\
\hline
\end{tabular}
\caption{Composing external formats}
\label{table:composing-formats}
\end{table}
\section{Dictionary}
\subsection{Variables}
\begin{defvar}{extensions:}{default-external-format}
This is the default external format to use for all newly opened
files. It is also the default format to use for
\var{*standard-input*}, \var{*standard-output*}, and
\var{*standard-error*}. The default value is \kwd{iso8859-1}.
Setting this will cause the standard streams to start using the new
format immediately. If a stream has been created with external
format \kwd{default}, then setting \var{*default-external-format*}
will cause all subsequent input and output to use the new value of
\var{*default-external-format*}.
\end{defvar}
\subsection{Characters}
Remember that \cmucl{}'s characters are only 16-bits long but Unicode
codepoints are up to 21 bits long. Hence there are codepoints that
cannot be represented via Lisp characters. Operating on individual
characters is not recommended. Operations on strings are better.
(This would be true even if \cmucl{}'s characters could hold a
full Unicode codepoint.)
\begin{defun}{}{char-equal}{\amprest{} \var{characters}}
\defunx{char-not-equal}{\amprest{} \var{characters}}
\defunx{char-lessp}{\amprest{} \var{characters}}
\defunx{char-greaterp}{\amprest{} \var{characters}}
\defunx{char-not-greaterp}{\amprest{} \var{characters}}
\defunx{char-not-lessp}{\amprest{} \var{characters}}
For the comparison, the characters are converted to lowercase and
the corresponding \code{char-code} are compared.
\end{defun}
\begin{defun}{}{alpha-char-p}{\args \var{character}}
Returns non-nil{} if the Unicode category is a letter category.
\end{defun}
\begin{defun}{}{alphanumericp}{\args \var{character}}
Returns non-nil{} if the Unicode category is a letter category or an ASCII
digit.
\end{defun}
\begin{defun}{}{digit-char-p}{\args \var{character} \ampoptional{} \var{radix}}
Only recognizes ASCII digits (and ASCII letters if the radix is larger
than 10).
\end{defun}
\begin{defun}{}{graphic-char-p}{\args \var{character}}
Returns non-nil{} if the Unicode category is a graphic category.
\end{defun}
\begin{defun}{}{upper-case-p}{\args \var{character}}
\defunx{lower-case-p}{\args \var{character}}
Returns non-nil{} if the Unicode category is an uppercase
(lowercase) character.
\end{defun}
\begin{defun}{lisp:}{title-case-p}{\args \var{character}}
Returns non-nil{} if the Unicode category is a titlecase character.
\end{defun}
\begin{defun}{}{both-case-p}{\args \var{character}}
Returns non-nil{} if the Unicode category is an uppercase,
lowercase, or titlecase character.
\end{defun}
\begin{defun}{}{char-upcase}{\args \var{character}}
\defunx{char-downcase}{\args \var{character}}
The Unicode uppercase (lowercase) letter is returned.
\end{defun}
\begin{defun}{lisp:}{char-titlecase}{\args \var{character}}
The Unicode titlecase letter is returned.
\end{defun}
\begin{defun}{}{char-name}{\args \var{char}}
If possible the name of the character \var{char} is returned. If
there is a Unicode name, the Unicode name is returned, except
spaces are converted to underscores and the string is capitalized
via \code{string-capitalize}. If there is no Unicode name, the
form \lispchar{U+xxxx} is returned where ``xxxx'' is the
\code{char-code} of the character, in hexadecimal.
\end{defun}
\begin{defun}{}{name-char}{\args \var{name}}
The inverse to \code{char-name}. If no character has the name
\var{name}, then \nil{} is returned. Unicode names are not
case-sensitive, and spaces and underscores are optional.
\end{defun}
\subsection{Strings}
Strings in \cmucl{} are UTF-16 strings. That is, for Unicode code
points greater than 65535, surrogate pairs are used. We refer the
reader to the Unicode standard for more information about surrogate
pairs. We just want to make a note that because of the UTF-16
encoding of strings, there is a distinction between Lisp characters
and Unicode codepoints. The standard string operations know about
this encoding and handle the surrogate pairs correctly.
\begin{defun}{}{string-upcase}{\args \var{string} \keys{\kwd{start}
\kwd{end} \kwd{casing}}}
\defunx{string-downcase}{\args \var{string} \keys{\kwd{start}
\kwd{end} \kwd{casing}}}
\defunx{string-capitalize}{\args \var{string} \keys{\kwd{start}
\kwd{end} \kwd{casing}}}
The case of the \var{string} is changed appropriately. Surrogate
pairs are handled correctly. The conversion to the appropriate case
is done based on the Unicode conversion. The additional argument
\kwd{casing} controls how case conversion is done. The default
value is \kwd{simple}, which uses simple Unicode case conversion.
If \kwd{casing} is \kwd{full}, then full Unicode case conversion is
done where the string may actually increase in length.
\end{defun}
\begin{defun}{}{nstring-upcase}{\args \var{string} \keys{\kwd{start} \kwd{end}}}
\defunx{nstring-downcase}{\args \var{string} \keys{\kwd{start} \kwd{end}}}
\defunx{nstring-capitalize}{\args \var{string} \keys{\kwd{start}
\kwd{end}}}
The case of the \var{string} is changed appropriately. Surrogate
pairs are handled correctly. The conversion to the appropriate case
is done based on the Unicode conversion. (Full casing is not
available because the string length cannot be increased when needed.)
\end{defun}
\begin{defun}{}{string=}{\args \var{s1} \var{s2} \keys{\kwd{start1}
\kwd{end1} \kwd{start2} \kwd{end2}}}
\defunx{string/=}{\args \var{s1} \var{s2} \keys{\kwd{start1} \kwd{end1} \kwd{start2} \kwd{end2}}}
\defunx{string\textless}{\args \var{s1} \var{s2} \keys{\kwd{start1} \kwd{end1} \kwd{start2} \kwd{end2}}}
\defunx{string\textgreater}{\args \var{s1} \var{s2} \keys{\kwd{start1} \kwd{end1} \kwd{start2} \kwd{end2}}}
\defunx{string\textless=}{\args \var{s1} \var{s2} \keys{\kwd{start1} \kwd{end1} \kwd{start2} \kwd{end2}}}
\defunx{string\textgreater=}{\args \var{s1} \var{s2} \keys{\kwd{start1} \kwd{end1} \kwd{start2} \kwd{end2}}}
The string comparison is done in codepoint order. (This is
different from just comparing the order of the individual characters
due to surrogate pairs.) Unicode collation is not done.
\end{defun}
\begin{defun}{}{string-equal}{\args \var{s1} \var{s2} \keys{\kwd{start1}
\kwd{end1} \kwd{start2} \kwd{end2}}}
\defunx{string-not-equal}{\args \var{s1} \var{s2} \keys{\kwd{start1} \kwd{end1} \kwd{start2} \kwd{end2}}}
\defunx{string-lessp}{\args \var{s1} \var{s2} \keys{\kwd{start1} \kwd{end1} \kwd{start2} \kwd{end2}}}
\defunx{string-greaterp}{\args \var{s1} \var{s2} \keys{\kwd{start1} \kwd{end1} \kwd{start2} \kwd{end2}}}
\defunx{string-not-greaterp}{\args \var{s1} \var{s2} \keys{\kwd{start1} \kwd{end1} \kwd{start2} \kwd{end2}}}
\defunx{string-not-lessp}{\args \var{s1} \var{s2} \keys{\kwd{start1} \kwd{end1} \kwd{start2} \kwd{end2}}}
Each codepoint in each string is converted to lowercase and the
appropriate comparison of the codepoint values is done. Unicode
collation is not done.
\end{defun}
\begin{defun}{}{string-left-trim}{\args \var{bag} \var{string}}
\defunx{string-right-trim}{\args \var{bag} \var{string}}
\defunx{string-trim}{\args \var{bag} \var{string}}
Removes any characters in \code{bag} from the left, right, or both
ends of the string \code{string}, respectively. This has potential
problems if you want to remove a surrogate character from the
string, since a single character cannot represent a surrogate. As
an extension, if \code{bag} is a string, we properly handle
surrogate characters in the \code{bag}.
\end{defun}
\subsection{Sequences}
Since strings are also sequences, the sequence functions can be used
on strings. We note here some issues with these functions. Most
issues are due to the fact that strings are UTF-16 strings and
characters are UTF-16 code units, not Unicode codepoints.
\begin{defun}{}{remove-duplicates}{\args \var{sequence}
\keys{\kwd{from-end} \kwd{test} \kwd{test-not} \kwd{start}
\kwd{end} \kwd{key}}}
\defunx{delete-duplicates}{\args \var{sequence}
\keys{\kwd{from-end} \kwd{test} \kwd{test-not} \kwd{start}
\kwd{end} \kwd{key}}}
Because of surrogate pairs these functions may remove a high or low
surrogate value, leaving the string in an invalid state. Use these
functions carefully with strings.
\end{defun}
\subsection{Reader}
To support Unicode characters, the reader has been extended to
recognize characters written in hexadecimal. Thus \lispchar{U+41} is
the ASCII capital letter ``A'', since 41 is the hexadecimal code for
that letter. The Unicode name of the character is also recognized,
except spaces in the name are replaced by underscores.
Recall, however, that characters in \cmucl{} are only 16 bits long so
many Unicode characters cannot be represented. However, strings can
represent all Unicode characters.
When symbols are read, the symbol name is converted to Unicode NFC
form before interning the symbol into the package. Hence,
\code{symbol-name (intern ``string'')} may produce a string that is
not \code{string=} to ``string''. However, after conversion to NFC
form, the strings will be identical.
\subsection{Printer}
When printing characters, if the character is a graphic character, the
character is printed. Thus \lispchar{U+41} is printed as
\lispchar{A}. If the character is not a graphic character, the Lisp
name (e.g., \lispchar{Tab}) is used if possible;
if there is no Lisp name, the Unicode name is used. If there is no
Unicode name, the hexadecimal char-code is
printed. For example, \lispchar{U+34e}, which is not a graphic
character, is printed as \lispchar{Combining\_Upwards\_Arrow\_Below},
and \lispchar{U+9f} which is not a graphic character and does not have a
Unicode name, is printed as \lispchar{U+009F}.
\subsection{Miscellaneous}
\subsubsection{Files}
\cmucl{} loads external formats using the search-list
\file{ext-formats:}. The \file{aliases} file is also located using
this search-list.
The Unicode data base is stored in compressed form in the file
\file{ext-formats:unidata.bin}. If this file is not found, Unicode
support is severely reduced; you can only use ASCII characters.
\begin{defun}{}{open}{\args \var{filename} \amprest{} \var{options}
\keys{\kwd{direction} \kwd{element-type} \kwd{if-exists}
\kwd{if-does-not-exist} \morekeys \kwd{class} \kwd{mapped}
\kwd{input-handle} \kwd{output-handle}
\yetmorekeys \kwd{external-format} \kwd{decoding-error}
\kwd{encoding-error}}}
The main options are covered elsewhere. Here we describe the
options specific to Unicode. The option \kwd{external-format}
specifies the external format to use for reading and writing the
file. The external format is a keyword.
The options \kwd{decoding-error} and \kwd{encoding-error} are used
to specify how encoding and decoding errors are handled. The
default value on \nil means the external format handles errors
itself and typically replaces invalid sequences with the Unicode
replacement character.
Otherwise, the value for \code{decoding-error} is either a
character, a symbol or a function. If a character is
specified. it is used as the replacement character for any invalid
decoding. If a symbol or a function is given, it must be a
function of three arguments: a message string to be printed, the
offending octet, and the number of octets read. If the function
returns, it should return two values: the code point to use as the
replacement character and the number of octets read. In addition,
\true{} may be specified. This indicates that a continuable error
is signaled, which, if continued, the Unicode replacement
character is used.
For \code{encoding-error}, a character, symbol, or function can be
specified, like \code{decoding-error}, with the same meaning. The
function, however, takes two arguments: a format message string
and the incorrect codepoint. If the function returns, it should
be the replacement codepoint.
\end{defun}
\subsubsection{Utilities}
\begin{defun}{stream:}{set-system-external-format}{\var{terminal} \ampoptional{} \var{filenames}}
This function changes the external format used for
\var{*standard-input*}, \var{*standard-output*}, and
\var{*standard-error*} to the external format specified by
\var{terminal}. Additionally, the Unix file name encoding can be
set to the value specified by \var{filenames} if non-\nil.
\end{defun}
\begin{defun}{extensions:}{list-all-external-formats}{}
list all of the vailable external formats. A list is returned where
each element is a list of the external format name and a list of
aliases for the format. No distinction is made between external
formats and composing external formats.
\end{defun}
\begin{defun}{extensions:}{describe-external-format}{\var{external-format}}
Print a description of the given \var{external-format}. This may
cause the external format to be loaded (silently) if it is not
already loaded.
\end{defun}
Since strings are UTF-16 and hence may contain surrogate pairs, some
utility functions are provided to make access easier.
\begin{defun}{lisp:}{codepoint}{\args \var{string} \var{i}
\ampoptional{} \var{end}}
Return the codepoint value from \var{string} at position \var{i}.
If code unit at that position is a surrogate value, it is combined
with either the previous or following code unit (when possible) to
compute the codepoint. The first return value is the codepoint
itself. The second return value is \nil{} if the position is not a
surrogate pair. Otherwise, $+1$ or $-1$ is returned if the position
is the high (leading) or low (trailing) surrogate value, respectively.
This is useful for iterating through a string in codepoint sequence.
\end{defun}
\begin{defun}{lisp:}{surrogates-to-codepoint}{\args \var{hi} \var{lo}}
Convert the given \var{hi} and \var{lo} surrogate characters to the
corresponding codepoint value
\end{defun}
\begin{defun}{lisp:}{surrogates}{\args \var{codepoint}}
Convert the given \var{codepoint} value to the corresponding high
and low surrogate characters. If the codepoint is less than 65536,
the second value is \nil{} since the codepoint does not need to be
represented as a surrogate pair.
\end{defun}
\begin{defun}{stream:}{string-encode}{\args \var{string}
\var{external-format} \ampoptional{} (\var{start} 0) \var{end}}
\code{string-encode} encodes \var{string} using the format
\var{external-format}, producing an array of octets. Each octet is
converted to a character via \code{code-char} and the resulting
string is returned.
The optional argument \var{start}, defaulting to 0, specifies the
starting index and \var{end}, defaulting to the length of the
string, is the end of the string.
\end{defun}
\begin{defun}{stream:}{string-decode}{\args \var{string}
\var{external-format} \ampoptional{} (\var{start} 0) \var{end}}
\code{string-decode} decodes \var{string} using the format
\var{external-format} and produces a new string. Each character of
\var{string} is converted to octet (by \code{char-code}) and the
resulting array of octets is used by the external format to produce
a string. This is the inverse of \code{string-encode}.
The optional argument \var{start}, defaulting to 0, specifies the
starting index and \var{end}, defaulting to the length of the
string, is the end of the string.
\var{string} must consist of characters whose \code{char-code} is
less than 256.
\end{defun}
\begin{defun}{}{string-to-octets}{\args \var{string} \keys{\kwd{start}
\kwd{end} \kwd{external-format} \kwd{buffer} \kwd{buffer-start}
\kwd{error}}}
\code{string-to-octets} converts \var{string} to a sequence of
octets according to the external format specified by
\var{external-format}. The string to be converted is bounded by
\var{start}, which defaults to 0, and \var{end}, which defaults to
the length of the string. If \var{buffer} is specified, the octets
are placed in \var{buffer}. If \var{buffer} is not specified, a new
array is allocated to hold the octets. \var{buffer-start} specifies
where in the buffer the first octet will be placed.
An error method may also be specified by \var{error}. Any errors
encountered while converting the string to octets will be handled
according to error. If \nil{}, a replacement character is converted
to octets in place of the error. Otherwise, \var{error} should be a
symbol or function that will be called when the error occurs. The
function takes two arguments: an error string and the character
that caused the error. It should return a replacement character.
Three values are returned: The buffer, the number of valid octets
written, and the number of characters converted. Note that the
actual number of octets written may be greater than the returned
value, These represent the partial octets of the next character to
be converted, but there was not enough room to hold the complete set
of octets.
\end{defun}
\begin{defun}{}{octets-to-string}{\args \var{octets} \keys{\kwd{start}
\kwd{end} \kwd{external-format} \kwd{string} \kwd{s-start}
\kwd{s-end} \kwd{state}}}
\code{octets-to-string} converts the sequence of octets in
\var{octets} to a string. \var{octets} must be a
\code{(simple-array (unsigned-byte 8) (*))}. The octets to be
converted are bounded by \var{start} and \var{end}, which default to
0 and the length of the array, respectively. The conversion is
performed according to the external format specified by
\var{external-format}. If \var{string} is specified, the octets are
converted and stored in \var{string}, starting at \var{s-start}
(defaulting to 0) and ending just before \var{s-end} (defaulting to
the end of \var{string}. \var{string} must be \code{simple-string}.
If the bounded string is not large enough to hold all of the
characters, then some octets will not be converted. If \var{string}
is not specified, a new string is created.
The \var{state} is used as the initial state of for the external
format. This is useful when converting buffers of octets where the
buffers are not on character boundaries, and state information is
needed between buffers.
Four values are returned: the string, the number of characters
written to the string, and the number of octets consumed to produce
the characters, and the final state of external format after
converting the octets.
\end{defun}
\section{Writing External Formats}
\subsection{External Formats}
Users may write their own external formats. It is probably easiest to
look at existing external formats to see how do this.
An external format basically needs two functions:
\code{octets-to-code} to convert octets to Unicode codepoints and
\code{code-to-octets} to convert Unicode codepoints to octets. The
external format is defined using the macro
\code{stream::define-external-format}.
% tricky
\begin{defmac}[base]{stream::}{define-external-format}{\args \var{name}
(\keys{\kwd{base} \kwd{min} \kwd{max} \kwd{size} \kwd{documentation}})
(\amprest{} \var{slots})
\\
\= \ampoptional{} \var{octets-to-code} \var{code-to-octets} \var{flush-state} \var{copy-state}}
If \kwd{base} is not given, this defines a new external format of
the name \kwd{name}. \var{min}, \var{max}, and \var{size} are the
minimum and maximum number of octets that make up a character.
(\code{\kwd{size} n} is just a short cut for \code{\kwd{min} n
\kwd{max} n}.) The description of the external format can be
given using \kwd{documentation}. The arguments \var{octets-to-code}
and \var{code-to-octets} are not optional in this case. They
specify how to convert octets to codepoints and vice versa,
respectively. These should be backquoted forms for the body of a
function to do the conversion. See the description below for these
functions. Some good examples are the external format for
\kwd{utf-8} or \kwd{utf-16}. The \kwd{slots} argument is a list of
read-only slots, similar to defstruct. The slot names are available
as local variables inside the \var{code-to-octets} and
\var{octets-to-code} bodies.
If \kwd{base} is given, then an external format is defined with the
name \kwd{name} that is based on a previously defined format
\kwd{base}. The \var{slots} are inherited from the \kwd{base} format
by default, although the definition may alter their values and add
new slots. See, for example, the \kwd{mac-greek} external format.
\end{defmac}
\begin{defmac}{}{octets-to-code}{\args \var{state} \var{input}
\var{unput} \var{error} \amprest{} \var{args}}
This defines a form to be used by an external format to convert
octets to a code point. \var{state} is a form that can be used by
the body to access the state variable of a stream. This can be used
for any reason to hold anything needed by \code{octets-to-code}.
\var{input} is a form that returns one octet from the input stream.
\var{unput} will put back \var{N} octets to the stream. \var{args} is a
list of variables that need to be defined for any symbols in the
body of the macro.
\var{error} controls how errors are handled. If \nil, some suitable
replacement character is used. That is, any errors are silently
ignored and replaced by some replacement character. If non-\nil,
\var{error} is a symbol or function that is called to handle the
error. This function takes three arguments: a message string, the
invalid octet (or \nil), and a count of the number of octets that
have been read so far. If the function returns, it should be the
codepoint of the desired replacement character.
\end{defmac}
\begin{defmac}{}{code-to-octets}{\args \var{code} \var{state}
\var{output} \var{error} \amprest{} \var{args}}
Defines a form to be used by the external format to convert a code
point to octets for output. \var{code} is the code point to be
converted. \var{state} is a form to access the current value of the
stream's state variable. \var{output} is a form that writes one
octet to the output stream.
Similar to \code{octets-to-code}, \var{error} indicates how errors
should be handled. If \nil, some default replacement character is
substituted. If non-\nil, \var{error} should be a symbol or
function. This function takes two arguments: a message string and
the invalid codepoint. If the function returns, it should be the
codepoint that will be substituted for the invalid codepoint.
\end{defmac}
\begin{defmac}{}{flush-state}{\args \var{state}
\var{output} \var{error} \amprest{} \var{args}}
Defines a form to be used by the external format to flush out
any state when an output stream is closed. Similar to
\code{code-to-octets}, but there is no code point to be output. The
\var{error} argument indicates how to handle errors. If \nil, some
default replacement character is used. Otherwise, \var{error} is a
symbol or function that will be called with a message string and
codepoint of the offending state. If the function returns, it
should be the codepoint of a suitable replacement.
If \code{flush-state} is \false, then nothing special is needed to
flush the state to the output.
This is called only when an output character stream is being closed.
\end{defmac}
\begin{defmac}{}{copy-state}{\args \var{state} \amprest{} \var{args}}
Defines a form to copy any state needed by the external format.
This should probably be a deep copy so that if the original
state is modified, the copy is not.
If not given, then nothing special is needed to copy the state
either because there is no state for the external format or that no
special copier is needed.
\end{defmac}
\subsection{Composing External Formats}
\begin{defmac}{stream:}{define-composing-external-format}{\args \var{name}
(\keys{\kwd{min} \kwd{max} \kwd{size} \kwd{documentation}}) \var{input}
\var{output}}
This is the same as \code{define-external-format}, except that a
composing external format is created.
\end{defmac}

508
doc/cmu-user/unix.tex Normal file
View file

@ -0,0 +1,508 @@
\chapter{UNIX Interface}
\label{unix-interface}
\credits{by Robert MacLachlan, Skef Wholey, Bill Chiles and William Lott}
\cmucl{} attempts to make the full power of the underlying
environment available to the Lisp programmer. This is done using
combination of hand-coded interfaces and foreign function calls to C
libraries. Although the techniques differ, the style of interface is
similar. This chapter provides an overview of the facilities available
and general rules for using them, as well as describing specific
features in detail. It is assumed that the reader has a working
familiarity with Unix and X11, as well as access to the standard
system documentation.
\section{Reading the Command Line}
The shell parses the command line with which Lisp is invoked, and
passes a data structure containing the parsed information to Lisp.
This information is then extracted from that data structure and put
into a set of Lisp data structures.
\begin{defvar}{extensions:}{command-line-strings}
\defvarx[extensions:]{command-line-utility-name}
\defvarx[extensions:]{command-line-words}
\defvarx[extensions:]{command-line-switches}
The value of \code{*command-line-words*} is a list of strings that
make up the command line, one word per string. The first word on
the command line, i.e. the name of the program invoked (usually
\code{lisp}) is stored in \code{*command-line-utility-name*}. The
value of \code{*command-line-switches*} is a list of
\code{command-line-switch} structures, with a structure for each
word on the command line starting with a hyphen. All the command
line words between the program name and the first switch are stored
in \code{*command-line-words*}.
\end{defvar}
The following functions may be used to examine \code{command-line-switch}
structures.
\begin{defun}{extensions:}{cmd-switch-name}{\args{\var{switch}}}
Returns the name of the switch, less the preceding hyphen and
trailing equal sign (if any).
\end{defun}
\begin{defun}{extensions:}{cmd-switch-value}{\args{\var{switch}}}
Returns the value designated using an embedded equal sign, if any.
If the switch has no equal sign, then this is null.
\end{defun}
\begin{defun}{extensions:}{cmd-switch-words}{\args{\var{switch}}}
Returns a list of the words between this switch and the next switch
or the end of the command line.
\end{defun}
\begin{defun}{extensions:}{cmd-switch-arg}{\args{\var{switch}}}
Returns the first non-null value from \code{cmd-switch-value}, the
first element in \code{cmd-switch-words}, or the first word in
\var{command-line-words}.
\end{defun}
\begin{defun}{extensions:}{get-command-line-switch}{\args{\var{sname}}}
This function takes the name of a switch as a string and returns the
value of the switch given on the command line. If no value was
specified, then any following words are returned. If there are no
following words, then \true{} is returned. If the switch was not
specified, then \false{} is returned.
\end{defun}
\begin{defmac}{extensions:}{defswitch}{%
\args{\var{name} \ampoptional{} \var{function}}}
This macro causes \var{function} to be called when the switch
\var{name} appears in the command line. Name is a simple-string
that does not begin with a hyphen (unless the switch name really
does begin with one.)
If \var{function} is not supplied, then the switch is parsed into
\var{command-line-switches}, but otherwise ignored. This suppresses
the undefined switch warning which would otherwise take place. The
warning can also be globally suppressed by
\var{complain-about-illegal-switches}.
\end{defmac}
\section{Useful Variables}
\begin{defvar}{system:}{stdin}
\defvarx[system:]{stdout} \defvarx[system:]{stderr}
Streams connected to the standard input, output and error file
descriptors.
\end{defvar}
\begin{defvar}{system:}{tty}
A stream connected to \file{/dev/tty}.
\end{defvar}
\begin{defvar}{extensions:}{environment-list}
The environment variables inherited by the current process, as a
keyword-indexed alist. For example, to access the DISPLAY
environment variable, you could use
\begin{lisp}
(cdr (assoc :display ext:*environment-list*))
\end{lisp}
Note that the case of the variable name is preserved when converting
to a keyword. Therefore, you need to specify the keyword properly for
variable names containing lower-case letters,
\end{defvar}
\section{Lisp Equivalents for C Routines}
The UNIX documentation describes the system interface in terms of C
procedure headers. The corresponding Lisp function will have a somewhat
different interface, since Lisp argument passing conventions and
datatypes are different.
The main difference in the argument passing conventions is that Lisp does not
support passing values by reference. In Lisp, all argument and results are
passed by value. Interface functions take some fixed number of arguments and
return some fixed number of values. A given ``parameter'' in the C
specification will appear as an argument, return value, or both, depending on
whether it is an In parameter, Out parameter, or In/Out parameter. The basic
transformation one makes to come up with the Lisp equivalent of a C routine is
to remove the Out parameters from the call, and treat them as extra return
values. In/Out parameters appear both as arguments and return values. Since
Out and In/Out parameters are only conventions in C, you must determine the
usage from the documentation.
Thus, the C routine declared as
\begin{example}
kern_return_t lookup(servport, portsname, portsid)
port servport;
char *portsname;
int *portsid; /* out */
{
...
*portsid = <expression to compute portsid field>
return(KERN_SUCCESS);
}
\end{example}
has as its Lisp equivalent something like
\begin{lisp}
(defun lookup (ServPort PortsName)
...
(values
success
<expression to compute portsid field>))
\end{lisp}
If there are multiple out or in-out arguments, then there are multiple
additional returns values.
Fortunately, \cmucl{} programmers rarely have to worry about the
nuances of this translation process, since the names of the arguments and
return values are documented in a way so that the \code{describe} function
(and the \hemlock{} \code{Describe Function Call} command, invoked with
\b{C-M-Shift-A}) will list this information. Since the names of arguments
and return values are usually descriptive, the information that
\code{describe} prints is usually all one needs to write a
call. Most programmers use this on-line documentation nearly
all of the time, and thereby avoid the need to handle bulky
manuals and perform the translation from barbarous tongues.
\section{Type Translations}
\cindex{aliens}
\cpsubindex{types}{alien}
\cpsubindex{types}{foreign language}
Lisp data types have very different representations from those used by
conventional languages such as C. Since the system interfaces are
designed for conventional languages, Lisp must translate objects to and
from the Lisp representations. Many simple objects have a direct
translation: integers, characters, strings and floating point numbers
are translated to the corresponding Lisp object. A number of types,
however, are implemented differently in Lisp for reasons of clarity and
efficiency.
Instances of enumerated types are expressed as keywords in Lisp.
Records, arrays, and pointer types are implemented with the \alien{}
facility (\pxlref{aliens}). Access functions are defined
for these types which convert fields of records, elements of arrays,
or data referenced by pointers into Lisp objects (possibly another
object to be referenced with another access function).
One should dispose of \alien{} objects created by constructor
functions or returned from remote procedure calls when they are no
longer of any use, freeing the virtual memory associated with that
object. Since \alien{}s contain pointers to non-Lisp data, the
garbage collector cannot do this itself. If the memory
was obtained from \funref{make-alien} or from a foreign function call
to a routine that used \code{malloc}, then \funref{free-alien} should
be used.
\section{System Area Pointers}
\label{system-area-pointers}
\cindex{pointers}\cpsubindex{malloc}{C function}\cpsubindex{free}{C function}
Note that in some cases an address is represented by a Lisp integer, and in
other cases it is represented by a real pointer. Pointers are usually used
when an object in the current address space is being referred to. The MACH
virtual memory manipulation calls must use integers, since in principle the
address could be in any process, and Lisp cannot abide random pointers.
Because these types are represented differently in Lisp, one must explicitly
coerce between these representations.
System Area Pointers (SAPs) provide a mechanism that bypasses the
\alien{} type system and accesses virtual memory directly. A SAP is a
raw byte pointer into the \code{lisp} process address space. SAPs are
represented with a pointer descriptor, so SAP creation can cause
consing. However, the compiler uses a non-descriptor representation
for SAPs when possible, so the consing overhead is generally minimal.
\xlref{non-descriptor}.
\begin{defun}{system:}{sap-int}{\args{\var{sap}}}
\defunx[system:]{int-sap}{\args{\var{int}}}
The function \code{sap-int} is used to generate an integer
corresponding to the system area pointer, suitable for passing to
the kernel interfaces (which want all addresses specified as
integers). The function \code{int-sap} is used to do the opposite
conversion. The integer representation of a SAP is the byte offset
of the SAP from the start of the address space.
\end{defun}
\begin{defun}{system:}{sap+}{\args{\var{sap} \var{offset}}}
This function adds a byte \var{offset} to \var{sap}, returning a new
SAP.
\end{defun}
\begin{defun}{system:}{sap-ref-8}{\args{\var{sap} \var{offset}}}
\defunx[system:]{sap-ref-16}{\args{\var{sap} \var{offset}}}
\defunx[system:]{sap-ref-32}{\args{\var{sap} \var{offset}}}
These functions return the 8, 16 or 32 bit unsigned integer at
\var{offset} from \var{sap}. The \var{offset} is always a byte
offset, regardless of the number of bits accessed. \code{setf} may
be used with the these functions to deposit values into virtual
memory.
\end{defun}
\begin{defun}{system:}{signed-sap-ref-8}{\args{\var{sap} \var{offset}}}
\defunx[system:]{signed-sap-ref-16}{\args{\var{sap} \var{offset}}}
\defunx[system:]{signed-sap-ref-32}{\args{\var{sap} \var{offset}}}
These functions are the same as the above unsigned operations,
except that they sign-extend, returning a negative number if the
high bit is set.
\end{defun}
\section{Unix System Calls}
You probably won't have much cause to use them, but all the Unix system
calls are available. The Unix system call functions are in the
\code{Unix} package. The name of the interface for a particular system
call is the name of the system call prepended with \code{unix-}. The
system usually defines the associated constants without any prefix name.
To find out how to use a particular system call, try using
\code{describe} on it. If that is unhelpful, look at the source in
\file{unix.lisp} or consult your system maintainer.
The Unix system calls indicate an error by returning \false{} as the
first value and the Unix error number as the second value. If the call
succeeds, then the first value will always be non-\nil, often \code{t}.
For example, to use the \code{chdir} syscall:
\begin{lisp}
(multiple-value-bind (success errno)
(unix:unix-chdir "/tmp")
(unless success
(error "Can't change working directory: ~a"
(unix:get-unix-error-msg errno))))
\end{lisp}
\begin{defun}{Unix:}{get-unix-error-msg}{\args{\var{error}}}
This function returns a string describing the Unix error number
\var{error} (this is similar to the Unix function \code{perror}).
\end{defun}
\section{File Descriptor Streams}
\label{sec:fds}
Many of the UNIX system calls return file descriptors. Instead of using other
UNIX system calls to perform I/O on them, you can create a stream around them.
For this purpose, fd-streams exist. See also \funref{read-n-bytes}.
\begin{defun}{system:}{make-fd-stream}{%
\args{\var{descriptor}} \keys{\kwd{input} \kwd{output}
\kwd{element-type}} \morekeys{\kwd{buffering} \kwd{name}
\kwd{file} \kwd{original}} \yetmorekeys{\kwd{delete-original}
\kwd{auto-close}} \yetmorekeys{\kwd{timeout} \kwd{pathname}}}
This function creates a file descriptor stream using
\var{descriptor}. If \kwd{input} is non-\nil, input operations are
allowed. If \kwd{output} is non-\nil, output operations are
allowed. The default is input only. These keywords are defined:
\begin{Lentry}
\item[\kwd{element-type}] is the type of the unit of transaction for
the stream, which defaults to \code{string-char}. See the \clisp{}
description of \code{open} for valid values.
\item[\kwd{buffering}] is the kind of output buffering desired for
the stream. Legal values are \kwd{none} for no buffering,
\kwd{line} for buffering up to each newline, and \kwd{full} for
full buffering.
\item[\kwd{name}] is a simple-string name to use for descriptive
purposes when the system prints an fd-stream. When printing
fd-streams, the system prepends the streams name with \code{Stream
for }. If \var{name} is unspecified, it defaults to a string
containing \var{file} or \var{descriptor}, in order of preference.
\item[\kwd{file}, \kwd{original}] \var{file} specifies the defaulted
namestring of the associated file when creating a file stream
(must be a \code{simple-string}). \var{original} is the
\code{simple-string} name of a backup file containing the original
contents of \var{file} while writing \var{file}.
When you abort the stream by passing \true{} to \code{close} as
the second argument, if you supplied both \var{file} and
\var{original}, \code{close} will rename the \var{original} name
to the \var{file} name. When you \code{close} the stream
normally, if you supplied \var{original}, and
\var{delete-original} is non-\nil, \code{close} deletes
\var{original}. If \var{auto-close} is true (the default), then
\var{descriptor} will be closed when the stream is garbage
collected.
\item[\kwd{pathname}]: The original pathname passed to open and
returned by \code{pathname}; not defaulted or translated.
\item[\kwd{timeout}] if non-null, then \var{timeout} is an integer
number of seconds after which an input wait should time out. If a
read does time out, then the \code{system:io-timeout} condition is
signalled.
\end{Lentry}
\end{defun}
\begin{defun}{system:}{fd-stream-p}{\args{\var{object}}}
This function returns \true{} if \var{object} is an fd-stream, and
\nil{} if not. Obsolete: use the portable \code{(typep x
'file-stream)}.
\end{defun}
\begin{defun}{system:}{fd-stream-fd}{\args{\var{stream}}}
This returns the file descriptor associated with \var{stream}.
\end{defun}
\section{Unix Signals}
\cindex{unix signals} \cindex{signals}
\cmucl{} allows access to all the Unix signals that can be generated
under Unix. It should be noted that if this capability is abused, it is
possible to completely destroy the running Lisp. The following macros and
functions allow access to the Unix interrupt system. The signal names as
specified in section 2 of the {\em Unix Programmer's Manual} are exported
from the Unix package.
\subsection{Changing Signal Handlers}
\label{signal-handlers}
\begin{defmac}{system:}{with-enabled-interrupts}{
\args{\var{specs} \amprest{} \var{body}}}
This macro should be called with a list of signal specifications,
\var{specs}. Each element of \var{specs} should be a list of
two\hide{ or three} elements: the first should be the Unix signal
for which a handler should be established, the second should be a
function to be called when the signal is received\hide{, and the
third should be an optional character used to generate the signal
from the keyboard. This last item is only useful for the SIGINT,
SIGQUIT, and SIGTSTP signals.} One or more signal handlers can be
established in this way. \code{with-enabled-interrupts} establishes
the correct signal handlers and then executes the forms in
\var{body}. The forms are executed in an unwind-protect so that the
state of the signal handlers will be restored to what it was before
the \code{with-enabled-interrupts} was entered. A signal handler
function specified as NIL will set the Unix signal handler to the
default which is normally either to ignore the signal or to cause a
core dump depending on the particular signal.
\end{defmac}
\begin{defmac}{system:}{without-interrupts}{\args{\amprest{} \var{body}}}
It is sometimes necessary to execute a piece a code that can not be
interrupted. This macro the forms in \var{body} with interrupts
disabled. Note that the Unix interrupts are not actually disabled,
rather they are queued until after \var{body} has finished
executing.
\end{defmac}
\begin{defmac}{system:}{with-interrupts}{\args{\amprest{} \var{body}}}
When executing an interrupt handler, the system disables interrupts,
as if the handler was wrapped in in a \code{without-interrupts}.
The macro \code{with-interrupts} can be used to enable interrupts
while the forms in \var{body} are evaluated. This is useful if
\var{body} is going to enter a break loop or do some long
computation that might need to be interrupted.
\end{defmac}
\begin{defmac}{system:}{without-hemlock}{\args{\amprest{} \var{body}}}
For some interrupts, such as SIGTSTP (suspend the Lisp process and
return to the Unix shell) it is necessary to leave Hemlock and then
return to it. This macro executes the forms in \var{body} after
exiting Hemlock. When \var{body} has been executed, control is
returned to Hemlock.
\end{defmac}
\begin{defun}{system:}{enable-interrupt}{%
\args{\var{signal} \var{function}\hide{ \ampoptional{}
\var{character}}}}
This function establishes \var{function} as the handler for
\var{signal}.
\hide{The optional \var{character} can be specified
for the SIGINT, SIGQUIT, and SIGTSTP signals and causes that
character to generate the appropriate signal from the keyboard.}
Unless you want to establish a global signal handler, you should use
the macro \code{with-enabled-interrupts} to temporarily establish a
signal handler. \hide{Without \var{character},}
\code{enable-interrupt} returns the old function associated with the
signal. \hide{When \var{character} is specified for SIGINT,
SIGQUIT, or SIGTSTP, it returns the old character code.}
\end{defun}
\begin{defun}{system:}{ignore-interrupt}{\args{\var{signal}}}
Ignore-interrupt sets the Unix signal mechanism to ignore
\var{signal} which means that the Lisp process will never see the
signal. Ignore-interrupt returns the old function associated with
the signal or \false{} if none is currently defined.
\end{defun}
\begin{defun}{system:}{default-interrupt}{\args{\var{signal}}}
Default-interrupt can be used to tell the Unix signal mechanism to
perform the default action for \var{signal}. For details on what
the default action for a signal is, see section 2 of the {\em Unix
Programmer's Manual}. In general, it is likely to ignore the
signal or to cause a core dump.
\end{defun}
\subsection{Examples of Signal Handlers}
The following code is the signal handler used by the Lisp system for the
SIGINT signal.
\begin{lisp}
(defun ih-sigint (signal code scp)
(declare (ignore signal code scp))
(without-hemlock
(with-interrupts
(break "Software Interrupt" t))))
\end{lisp}
The \code{without-hemlock} form is used to make sure that Hemlock is exited before
a break loop is entered. The \code{with-interrupts} form is used to enable
interrupts because the user may want to generate an interrupt while in the
break loop. Finally, break is called to enter a break loop, so the user
can look at the current state of the computation. If the user proceeds
from the break loop, the computation will be restarted from where it was
interrupted.
The following function is the Lisp signal handler for the SIGTSTP signal
which suspends a process and returns to the Unix shell.
\begin{lisp}
(defun ih-sigtstp (signal code scp)
(declare (ignore signal code scp))
(without-hemlock
(Unix:unix-kill (Unix:unix-getpid) Unix:sigstop)))
\end{lisp}
Lisp uses this interrupt handler to catch the SIGTSTP signal because it is
necessary to get out of Hemlock in a clean way before returning to the shell.
To set up these interrupt handlers, the following is recommended:
\begin{lisp}
(with-enabled-interrupts ((Unix:SIGINT #'ih-sigint)
(Unix:SIGTSTP #'ih-sigtstp))
<user code to execute with the above signal handlers enabled.>
)
\end{lisp}

11
doc/internals/cmu/.cvsignore Normal file
View file

@ -0,0 +1,11 @@
architecture.aux
compiler.aux
design.aux
design.dvi
design.log
design.out
design.pdf
design.toc
glossary.aux
retargeting.aux
run-time.aux

10
doc/internals/cmu/.gitignore vendored Normal file
View file

@ -0,0 +1,10 @@
/architecture.aux
/compiler.aux
/design.aux
/design.log
/design.out
/design.pdf
/design.toc
/glossary.aux
/retargeting.aux
/run-time.aux

83
doc/internals/cmu/Makefile Normal file
View file

@ -0,0 +1,83 @@
LATEX=latex
PDFLATEX=pdflatex
DVIPS=dvips
DVIPSRES=1200
DVIPSOPTS=-t a4 -D $(DVIPSRES)
RM=rm
CP=cp
MV=mv
HTML_SPLIT_LEVEL=2
MASTERFILE=design
FILES=architecture.tex back.tex compiler-overview.tex compiler.tex debugger.tex \
design.tex environment.tex fasl.tex front.tex glossary.tex interface.tex \
interpreter.tex lowlev.tex middle.tex object.tex retargeting.tex \
run-time.tex vm.tex
FIGURES=
all : $(MASTERFILE).pdf
$(MASTERFILE).dvi : $(FILES) $(FIGURES)
$(LATEX) $(MASTERFILE).tex
# $(MASTERFILE).aux $(MASTERFILE).toc : $(FILES) $(FIGURES)
# $(LATEX) $(MASTERFILE).tex
# $(RM) $(MASTERFILE).dvi
# $(MASTERFILE).dvi : $(FILES) $(MASTERFILE).aux $(MASTERFILE).toc $(FIGURES)
# $(LATEX) $(MASTERFILE).tex
# Three times should be enough for everyone(tm).
.PHONY: firsttime
firsttime: $(MASTERFILE).dvi
$(LATEX) $(MASTERFILE).tex
$(LATEX) $(MASTERFILE).tex
.PRECIOUS: %.aux %.toc %.tex
%.dvi : %.tex
$(LATEX) $<
@while ( grep Rerun $*.log > /dev/null ); do \
$(LATEX) $<; \
done
$(LATEX) $<
%.eps : %.fig
fig2dev -L ps $^ $@
%.pdf : %.tex
$(PDFLATEX) $^
@while ( grep 'Rerun to get cross' $*.log > /dev/null ); do \
$(PDFLATEX) $<; \
done
$(PDFLATEX) $<
%.ps : %.dvi
$(DVIPS) $(DVIPSOPTS) -o $@ $*
.PHONY: html
html : $(MASTERFILE).dvi $(FILES) $(FIGURES)
latex2html -local_icons -antialias_text -antialias -scalable_fonts \
-split $(HTML_SPLIT_LEVEL) -transparent -white $(MASTERFILE).tex
clean distclean cvsclean :
-$(RM) -f *.log *.toc *.aux *.dvi *~ *.bak *.ps *.pdf core *.eps
-$(RM) -rf design
.PHONY : allformats
allformats: $(FILES) $(FIGURES)
$(MAKE) firsttime LATEX=pslatex
$(MAKE) design.ps ; $(MV) design.ps design-psfonts.ps
$(MAKE) firsttime
$(MAKE) design.ps DVIPSRES=1200 ; $(MV) design.ps design-1200.ps
$(MAKE) design.ps DVIPSRES=600 ; $(MV) design.ps design-600.ps
$(MAKE) design.ps DVIPSRES=300 ; $(MV) design.ps design-300.ps
$(MAKE) design.pdf
$(MAKE) html HTML_SPLIT_LEVEL=3 ; $(MV) design design-split3
$(MAKE) html HTML_SPLIT_LEVEL=2 ; $(MV) design design-split2
$(MAKE) html HTML_SPLIT_LEVEL=1 ; $(MV) design design-split1

205
doc/internals/cmu/architecture.tex Normal file
View file

@ -0,0 +1,205 @@
\part{System Architecture}% -*- Dictionary: int:design -*-
\chapter{Package and File Structure}
\section{Source Tree Structure}
The \cmucl{} source tree has subdirectories for each major subsystem:
\begin{description}
\item[{\tt assembly/}] Holds the CMU CL source-file assembler, and has machine
specific subdirectories holding assembly code for that architecture.
\item[{\tt clx/}] The CLX interface to the X11 window system.
\item[{\tt code/}] The Lisp code for the runtime system and standard CL
utilities.
\item[{\tt compiler/}] The Python compiler. Has architecture-specific
subdirectories which hold backends for different machines. The {\tt generic}
subdirectory holds code that is shared across most backends.
\item[{\tt hemlock/}] The Hemlock editor.
\item[{\tt lisp/}] The C runtime system code and low-level Lisp debugger.
\item[{\tt pcl/}] \cmucl{} version of the PCL implementation of CLOS.
\item[{\tt tools/}] System building command files and source management tools.
\end{description}
\section{Package structure}
Goals: with the single exception of LISP, we want to be able to export from the
package that the code lives in.
\begin{description}
\item[Mach, CLX...] --- These Implementation-dependent system-interface
packages provide direct access to specific features available in the operating
system environment, but hide details of how OS communication is done.
\item[system] contains code that must know about the operating system
environment: I/O, etc. Hides the operating system environment. Provides OS
interface extensions such as {\tt print-directory}, etc.
\item[kernel] hides state and types used for system integration: package
system, error system, streams (?), reader, printer. Also, hides the VM, in
that we don't export anything that reveals the VM interface. Contains code
that needs to use the VM and SYSTEM interface, but is independent of OS and VM
details. This code shouldn't need to be changed in any port of CMU CL, but
won't work when plopped into an arbitrary CL. Uses SYSTEM, VM, EXTENSIONS. We
export "hidden" symbols related to implementation of CL: setf-inverses,
possibly some global variables.
The boundary between KERNEL and VM is fuzzy, but this fuzziness reflects the
fuzziness in the definition of the VM. We can make the VM large, and bring
everything inside, or we can make it small. Obviously, we want the VM to be
as small as possible, subject to efficiency constraints. Pretty much all of
the code in KERNEL could be put in VM. The issue is more what VM hides from
KERNEL: VM knows about everything.
\item[lisp] Originally, this package had all the system code in it. The
current ideal is that this package should have {\it no} code in it, and only
exist to export the standard interface. Note that the name has been changed by
x3j13 to common-lisp.
\item[extensions] contains code that any random user could have written: list
operations, syntactic sugar macros. Uses only LISP, so code in EXTENSIONS is
pure CL. Exports everything defined within that is useful elsewhere. This
package doesn't hide much, so it is relatively safe for users to use
EXTENSIONS, since they aren't getting anything they couldn't have written
themselves. Contrast this to KERNEL, which exports additional operations on
CL's primitive data structures: PACKAGE-INTERNAL-SYMBOL-COUNT, etc. Although
some of the functionality exported from KERNEL could have been defined in CL,
the kernel implementation is much more efficient because it knows about
implementation internals. Currently this package contains only extensions to
CL, but in the ideal scheme of things, it should contain the implementations of
all CL functions that are in KERNEL (the library.)
\item[VM] hides information about the hardware and data structure
representations. Contains all code that knows about this sort of thing: parts
of the compiler, GC, etc. The bulk of the code is the compiler back-end.
Exports useful things that are meaningful across all implementations, such as
operations for examining compiled functions, system constants. Uses COMPILER
and whatever else it wants. Actually, there are different {\it machine}{\tt
-VM} packages for each target implementation. VM is a nickname for whatever
implementation we are currently targeting for.
\item[compiler] hides the algorithms used to map Lisp semantics onto the
operations supplied by the VM. Exports the mechanisms used for defining the
VM. All the VM-independent code in the compiler, partially hiding the compiler
intermediate representations. Uses KERNEL.
\item[eval] holds code that does direct execution of the compiler's ICR. Uses
KERNEL, COMPILER. Exports debugger interface to interpreted code.
\item[debug-internals] presents a reasonable, unified interface to
manipulation of the state of both compiled and interpreted code. (could be in
KERNEL) Uses VM, INTERPRETER, EVAL, KERNEL.
\item[debug] holds the standard debugger, and exports the debugger
\end{description}
\chapter{System Building}
It's actually rather easy to build a CMU CL core with exactly what you want in
it. But to do this you need two things: the source and a working CMU CL.
Basically, you use the working copy of CMU CL to compile the sources,
then run a process call ``genesis'' which builds a ``kernel'' core.
You then load whatever you want into this kernel core, and save it.
In the \verb|tools/| directory in the sources there are several files that
compile everything, and build cores, etc. The first step is to compile the C
startup code.
{\bf Note:} {\it the various scripts mentioned below have hard-wired paths in
them set up for our directory layout here at CMU. Anyone anywhere else will
have to edit them before they will work.}
\section{Compiling the C Startup Code}
There is a circular dependancy between lisp/internals.h and lisp/lisp.map that
causes bootstrapping problems. The easiest way to get around this problem
is to make a fake lisp.nm file that has nothing in it but a version number:
\begin{verbatim}
% echo "Map file for lisp version 0" > lisp.nm
\end{verbatim}
and then run genesis with NIL for the list of files:
\begin{verbatim}
* (load ".../compiler/generic/new-genesis") ; compile before loading
* (lisp::genesis nil ".../lisp/lisp.nm" "/dev/null"
".../lisp/lisp.map" ".../lisp/lisp.h")
\end{verbatim}
It will generate
a whole bunch of warnings about things being undefined, but ignore
that, because it will also generate a correct lisp.h. You can then
compile lisp producing a correct lisp.map:
\begin{verbatim}
% make
\end{verbatim}
and then use \verb|tools/do-worldbuild| and \verb|tools/mk-lisp| to build
\verb|kernel.core| and \verb|lisp.core| (see section \ref{building-cores}.)
\section{Compiling the Lisp Code}
The \verb|tools| directory contains various lisp and C-shell utilities for
building CMU CL:
\begin{description}
\item[compile-all*] Will compile lisp files and build a kernel core. It has
numerous command-line options to control what to compile and how. Try -help to
see a description. It runs a separate Lisp process to compile each
subsystem. Error output is generated in files with ``{\tt .log}'' extension in
the root of the build area.
\item[setup.lisp] Some lisp utilities used for compiling changed files in batch
mode and collecting the error output. Sort of a crude defsystem. Loads into the
``user'' package. See {\tt with-compiler-log-file} and {\tt comf}.
\item[{\it foo}com.lisp] Each system has a ``\verb|.lisp|'' file in
\verb|tools/| which compiles that system.
\end{description}
\section{Building Core Images}
\label{building-cores}
Both the kernel and final core build are normally done using shell script
drivers:
\begin{description}
\item[do-worldbuild*] Builds a kernel core for the current machine. The
version to build is indicated by an optional argument, which defaults to
``alpha''. The \verb|kernel.core| file is written either in the \verb|lisp/|
directory in the build area, or in \verb|/usr/tmp/|. The directory which
already contains \verb|kernel.core| is chosen. You can create a dummy version
with e.g. ``touch'' to select the initial build location.
\item[mk-lisp*] Builds a full core, with conditional loading of subsystems.
The version is the first argument, which defaults to ``alpha''. Any additional
arguments are added to the \verb|*features*| list, which controls system
loading (among other things.) The \verb|lisp.core| file is written in the
current working directory.
\end{description}
These scripts load Lisp command files. When \verb|tools/worldbuild.lisp| is
loaded, it calls genesis with the correct arguments to build a kernel core.
Similarly, \verb|worldload.lisp|
builds a full core. Adding certain symbols to \verb|*features*| before
loading worldload.lisp suppresses loading of different parts of the
system. These symbols are:
\begin{description}
\item[:no-compiler] don't load the compiler.
\item[:no-clx] don't load CLX.
\item[:no-clm] don't load CLM.
\item[:no-hemlock] don't load Hemlock.
\item[:no-pcl] don't load PCL.
\item[:runtime] build a runtime code, implies all of the above, and then some.
\end{description}
Note: if you don't load the compiler, you can't (successfully) load the
pretty-printer or pcl. And if you compiled hemlock with CLX loaded, you can't
load it without CLX also being loaded.
These features are only used during the worldload process; they are
not propagated to the generated \verb|lisp.core| file.

728
doc/internals/cmu/back.tex Normal file
View file

@ -0,0 +1,728 @@
% -*- Dictionary: design -*-
\chapter{Copy propagation}
File: {\tt copyprop}
This phase is optional, but should be done whenever speed or space is more
important than compile speed. We use global flow analysis to find the reaching
definitions for each TN. This information is used here to eliminate
unnecessary TNs, and is also used later on by loop invariant optimization.
In some cases, VMR conversion will unnecessarily copy the value of a TN into
another TN, since it may not be able to tell that the initial TN has the same
value at the time the second TN is referenced. This can happen when ICR
optimize is unable to eliminate a trivial variable binding, or when the user
does a setq, or may also result from creation of expression evaluation
temporaries during VMR conversion. Whatever the cause, we would like to avoid
the unnecessary creation and assignment of these TNs.
What we do is replace TN references whose only reaching definition is a Move
VOP with a reference to the TN moved from, and then delete the Move VOP if the
copy TN has no remaining references. There are several restrictions on copy
propagation:
\begin{itemize}
\item The TNs must be ``ordinary'' TNs, not restricted or otherwise
unusual. Extending the life of restricted (or wired) TNs can make register
allocation impossible. Some other TN kinds have hidden references.
\item We don't want to defeat source-level debugging by replacing named
variables with anonymous temporaries.
\item We can't delete moves that representation selected might want to change
into a representation conversion, since we need the primitive types of both TNs
to select a conversion.
\end{itemize}
Some cleverness reduces the cost of flow analysis. As for lifetime analysis,
we only need to do flow analysis on global packed TNs. We can't do the real
local TN assignment pass before this, since we allocate TNs afterward, so we do
a pre-pass that marks the TNs that are local for our purposes. We don't care
if block splitting eventually causes some of them to be considered global.
Note also that we are really only interested in knowing if there is a
unique reaching definition, which we can mash into our flow analysis rules by
doing an intersection. Then a definition only appears in the set when it is
unique. We then propagate only definitions of TNs with only one write, which
allows the TN to stand for the definition.
\chapter{Representation selection}
File: {\tt represent}
Some types of object (such as {\tt single-float}) have multiple possible
representations. Multiple representations are useful mainly when there is a
particularly efficient non-descriptor representation. In this case, there is
the normal descriptor representation, and an alternate non-descriptor
representation.
This possibility brings up two major issues:
\begin{itemize}
\item The compiler must decide which representation will be most efficient for
any given value, and
\item Representation conversion code must be inserted where the representation
of a value is changed.
\end{itemize}
First, the representations for TNs are selected by examining all the TN
references and attempting to minimize reference costs. Then representation
conversion code is introduced.
This phase is in effect a pre-pass to register allocation. The main reason for
its existence is that representation conversions may be farily complex (e.g.
involving memory allocation), and thus must be discovered before register
allocation.
VMR conversion leaves stubs for representation specific move operations.
Representation selection recognizes {\tt move} by name. Argument and return
value passing for call VOPs is controlled by the {\tt :move-arguments} option
to {\tt define-vop}.
Representation selection is also responsible for determining what functions use
the number stack. If any representation is chosen which could involve packing
into the {\tt non-descriptor-stack} SB, then we allocate the NFP register
throughout the component. As an optimization, permit the decision of whether a
number stack frame needs to be allocated to be made on a per-function basis.
If a function doesn't use the number stack, and isn't in the same tail-set as
any function that uses the number stack, then it doesn't need a number stack
frame, even if other functions in the component do.
\chapter{Lifetime analysis}
File: {\tt life}
This phase is a preliminary to Pack. It involves three passes:
-- A pre-pass that computes the DEF and USE sets for live TN analysis, while
also assigning local TN numbers, splitting blocks if necessary. \#\#\# But
not really...
-- A flow analysis pass that does backward flow analysis on the
component to find the live TNs at each block boundary.
-- A post-pass that finds the conflict set for each TN.
\#|
Exploit the fact that a single VOP can only exhaust LTN numbers when there are
large more operands. Since more operand reference cannot be interleaved with
temporary reference, the references all effectively occur at the same time.
This means that we can assign all the more args and all the more results the
same LTN number and the same lifetime info.
|\#
\section{Flow analysis}
It seems we could use the global-conflicts structures during compute the
inter-block lifetime information. The pre-pass creates all the
global-conflicts for blocks that global TNs are referenced in. The flow
analysis pass just adds always-live global-conflicts for the other blocks the
TNs are live in. In addition to possibly being more efficient than SSets, this
would directly result in the desired global-conflicts information, rather than
having to create it from another representation.
The DFO sorted per-TN global-conflicts thread suggests some kind of algorithm
based on the manipulation of the sets of blocks each TN is live in (which is
what we really want), rather than the set of TNs live in each block.
If we sorted the per-TN global-conflicts in reverse DFO (which is just as good
for determining conflicts between TNs), then it seems we could scan though the
conflicts simultaneously with our flow-analysis scan through the blocks.
The flow analysis step is the following:
If a TN is always-live or read-before-written in a successor block, then we
make it always-live in the current block unless there are already
global-conflicts recorded for that TN in this block.
The iteration terminates when we don't add any new global-conflicts during a
pass.
We may also want to promote TNs only read within a block to always-live when
the TN is live in a successor. This should be easy enough as long as the
global-conflicts structure contains this kind of info.
The critical operation here is determining whether a given global TN has global
conflicts in a given block. Note that since we scan the blocks in DFO, and the
global-conflicts are sorted in DFO, if we give each global TN a pointer to the
global-conflicts for the last block we checked the TN was in, then we can
guarantee that the global-conflicts we are looking for are always at or after
that pointer. If we need to insert a new structure, then the pointer will help
us rapidly find the place to do the insertion.]
\section{Conflict detection}
[\#\#\# Environment, :more TNs.]
This phase makes use of the results of lifetime analysis to find the set of TNs
that have lifetimes overlapping with those of each TN. We also annotate call
VOPs with information about the live TNs so that code generation knows which
registers need to be saved.
The basic action is a backward scan of each block, looking at each TN-Ref and
maintaining a set of the currently live TNs. When we see a read, we check if
the TN is in the live set. If not, we:
-- Add the TN to the conflict set for every currently live TN,
-- Union the set of currently live TNs with the conflict set for the TN, and
-- Add the TN to the set of live TNs.
When we see a write for a live TN, we just remove it from the live set. If we
see a write to a dead TN, then we update the conflicts sets as for a read, but
don't add the TN to the live set. We have to do this so that the bogus write
doesn't clobber anything.
[We don't consider always-live TNs at all in this process, since the conflict
of always-live TNs with other TNs in the block is implicit in the
global-conflicts structures.
Before we do the scan on a block, we go through the global-conflicts structures
of TNs that change liveness in the block, assigning the recorded LTN number to
the TN's LTN number for the duration of processing of that block.]
Efficiently computing and representing this information calls for some
cleverness. It would be prohibitively expensive to represent the full conflict
set for every TN with sparse sets, as is done at the block-level. Although it
wouldn't cause non-linear behavior, it would require a complex linked structure
containing tens of elements to be created for every TN. Fortunately we can
improve on this if we take into account the fact that most TNs are ``local'' TNs:
TNs which have all their uses in one block.
First, many global TNs will be either live or dead for the entire duration of a
given block. We can represent the conflict between global TNs live throughout
the block and TNs local to the block by storing the set of always-live global
TNs in the block. This reduces the number of global TNs that must be
represented in the conflicts for local TNs.
Second, we can represent conflicts within a block using bit-vectors. Each TN
that changes liveness within a block is assigned a local TN number. Local
conflicts are represented using a fixed-size bit-vector of 64 elements or so
which has a 1 for the local TN number of every TN live at that time. The block
has a simple-vector which maps from local TN numbers to TNs. Fixed-size
vectors reduce the hassle of doing allocations and allow operations to be
open-coded in a maximally tense fashion.
We can represent the conflicts for a local TN by a single bit-vector indexed by
the local TN numbers for that block, but in the global TN case, we need to be
able to represent conflicts with arbitrary TNs. We could use a list-like
sparse set representation, but then we would have to either special-case global
TNs by using the sparse representation within the block, or convert the local
conflicts bit-vector to the sparse representation at the block end. Instead,
we give each global TN a list of the local conflicts bit-vectors for each block
that the TN is live in. If the TN is always-live in a block, then we record
that fact instead. This gives us a major reduction in the amount of work we
have to do in lifetime analysis at the cost of some increase in the time to
iterate over the set during Pack.
Since we build the lists of local conflict vectors a block at a time, the
blocks in the lists for each TN will be sorted by the block number. The
structure also contains the local TN number for the TN in that block. These
features allow pack to efficiently determine whether two arbitrary TNs
conflict. You just scan the lists in order, skipping blocks that are in only
one list by using the block numbers. When we find a block that both TNs are
live in, we just check the local TN number of one TN in the local conflicts
vector of the other.
In order to do these optimizations, we must do a pre-pass that finds the
always-live TNs and breaks blocks up into small enough pieces so that we don't
run out of local TN numbers. If we can make a block arbitrarily small, then we
can guarantee that an arbitrarily small number of TNs change liveness within
the block. We must be prepared to make the arguments to unbounded arg count
VOPs (such as function call) always-live even when they really aren't. This is
enabled by a panic mode in the block splitter: if we discover that the block
only contains one VOP and there are still too many TNs that aren't always-live,
then we promote the arguments (which we'd better be able to do...).
This is done during the pre-scan in lifetime analysis. We can do this because
all TNs that change liveness within a block can be found by examining that
block: the flow analysis only adds always-live TNs.
When we are doing the conflict detection pass, we set the LTN number of global
TNs. We can easily detect global TNs that have not been locally mapped because
this slot is initially null for global TNs and we null it out after processing
each block. We assign all Always-Live TNs to the same local number so that we
don't need to treat references to them specially when making the scan.
We also annotate call VOPs that do register saving with the TNs that are live
during the call, and thus would need to be saved if they are packed in
registers.
We adjust the costs for TNs that need to be saved so that TNs costing more to
save and restore than to reference get packed on the stack. We would also like
more often saved TNs to get higher costs so that they are packed in more
savable locations.
\chapter{Packing}
File: {\tt pack}
\#|
Add lifetime/pack support for pre-packed save TNs.
Fix GTN/VMR conversion to use pre-packed save TNs for old-cont and return-PC.
(Will prevent preference from passing location to save location from ever being
honored?)
We will need to make packing of passing locations smarter before we will be
able to target the passing location on the stack in a tail call (when that is
where the callee wants it.) Currently, we will almost always pack the passing
location in a register without considering whether that is really a good idea.
Maybe we should consider schemes that explicitly understand the parallel
assignment semantics, and try to do the assignment with a minimum number of
temporaries. We only need assignment temps for TNs that appear both as an
actual argument value and as a formal parameter of the called function. This
only happens in self-recursive functions.
Could be a problem with lifetime analysis, though. The write by a move-arg VOP
would look like a write in the current env, when it really isn't. If this is a
problem, then we might want to make the result TN be an info arg rather than a
real operand. But this would only be a problem in recursive calls, anyway.
[This would prevent targeting, but targeting across passing locations rarely
seems to work anyway.] [\#\#\# But the :ENVIRONMENT TN mechanism would get
confused. Maybe put env explicitly in TN, and have it only always-live in that
env, and normal in other envs (or blocks it is written in.) This would allow
targeting into environment TNs.
I guess we would also want the env/PC save TNs normal in the return block so
that we can target them. We could do this by considering env TNs normal in
read blocks with no successors.
ENV TNs would be treated totally normally in non-env blocks, so we don't have
to worry about lifetime analysis getting confused by variable initializations.
Do some kind of TN costing to determine when it is more trouble than it is
worth to allocate TNs in registers.
Change pack ordering to be less pessimal. Pack TNs as they are seen in the LTN
map in DFO, which at least in non-block compilations has an effect something
like packing main trace TNs first, since control analysis tries to put the good
code first. This could also reduce spilling, since it makes it less likely we
will clog all registers with global TNs.
If we pack a TN with a specified save location on the stack, pack in the
specified location.
Allow old-cont and return-pc to be kept in registers by adding a new ``keep
around'' kind of TN. These are kind of like environment live, but are only
always-live in blocks that they weren't referenced in. Lifetime analysis does
a post-pass adding always-live conflicts for each ``keep around'' TN to those
blocks with no conflict for that TN. The distinction between always-live and
keep-around allows us to successfully target old-cont and return-pc to passing
locations. MAKE-KEEP-AROUND-TN (ptype), PRE-PACK-SAVE-TN (tn scn offset).
Environment needs a KEEP-AROUND-TNS slot so that conflict analysis can find
them (no special casing is needed after then, they can be made with :NORMAL
kind). VMR-component needs PRE-PACKED-SAVE-TNS so that conflict analysis or
somebody can copy conflict info from the saved TN.
Note that having block granularity in the conflict information doesn't mean
that a localized packing scheme would have to do all moves at block boundaries
(which would clash with the desire to have saving done as part of this
mechanism.) All that it means is that if we want to do a move within the
block, we would need to allocate both locations throughout that block (or
something).
Load TN pack:
A location is out for load TN packing if:
The location has TN live in it after the VOP for a result, or before the VOP
for an argument, or
The location is used earlier in the TN-ref list (after) the saved results ref
or later in the TN-Ref list (before) the loaded argument's ref.
To pack load TNs, we advance the live-tns to the interesting VOP, then
repeatedly scan the vop-refs to find vop-local conflicts for each needed load
TN. We insert move VOPs and change over the TN-Ref-TNs as we go so the TN-Refs
will reflect conflicts with already packed load-TNs.
If we fail to pack a load-TN in the desired SC, then we scan the Live-TNs for
the SB, looking for a TN that can be packed in an unbounded SB. This TN must
then be repacked in the unbounded SB. It is important the load-TNs are never
packed in unbounded SBs, since that would invalidate the conflicts info,
preventing us from repacking TNs in unbounded SBs. We can't repack in a finite
SB, since there might have been load TNs packed in that SB which aren't
represented in the original conflict structures.
Is it permissible to ``restrict'' an operand to an unbounded SC? Not impossible
to satisfy as long as a finite SC is also allowed. But in practice, no
restriction would probably be as good.
We assume all locations can be used when an sc is based on an unbounded sb.
]
TN-Refs are convenient structures to build the target graph out of. If we
allocated space in every TN-Ref, then there would certainly be enough to
represent arbitrary target graphs. Would it be enough to allocate a single
Target slot? If there is a target path through a given VOP, then the Target of
the write ref would be the read, and vice-versa. To find all the TNs that
target us, we look at the TN for the target of all our write refs.
We separately chain together the read refs and the write refs for a TN,
allowing easy determination of things such as whether a TN has only a single
definition or has no reads. It would also allow easier traversal of the target
graph.
Represent per-location conflicts as vectors indexed by block number of
per-block conflict info. To test whether a TN conflicts on a location, we
would then have to iterate over the TNs global-conflicts, using the block
number and LTN number to check for a conflict in that block. But since most
TNs are local, this test actually isn't much more expensive than indexing into
a bit-vector by GTN numbers.
The big win of this scheme is that it is much cheaper to add conflicts into the
conflict set for a location, since we never need to actually compute the
conflict set in a list-like representation (which requires iterating over the
LTN conflicts vectors and unioning in the always-live TNs). Instead, we just
iterate over the global-conflicts for the TN, using BIT-IOR to combine the
conflict set with the bit-vector for that block in that location, or marking
that block/location combination as being always-live if the conflict is
always-live.
Generating the conflict set is inherently more costly, since although we
believe the conflict set size to be roughly constant, it can easily contain
tens of elements. We would have to generate these moderately large lists for
all TNs, including local TNs. In contrast, the proposed scheme does work
proportional to the number of blocks the TN is live in, which is small on
average (1 for local TNs). This win exists independently from the win of not
having to iterate over LTN conflict vectors.
[\#\#\# Note that since we never do bitwise iteration over the LTN conflict
vectors, part of the motivation for keeping these a small fixed size has been
removed. But it would still be useful to keep the size fixed so that we can
easily recycle the bit-vectors, and so that we could potentially have maximally
tense special primitives for doing clear and bit-ior on these vectors.]
This scheme is somewhat more space-intensive than having a per-location
bit-vector. Each vector entry would be something like 150 bits rather than one
bit, but this is mitigated by the number of blocks being 5-10x smaller than the
number of TNs. This seems like an acceptable overhead, a small fraction of the
total VMR representation.
The space overhead could also be reduced by using something equivalent to a
two-dimensional bit array, indexed first by LTN numbers, and then block numbers
(instead of using a simple-vector of separate bit-vectors.) This would
eliminate space wastage due to bit-vector overheads, which might be 50% or
more, and would also make efficient zeroing of the vectors more
straightforward. We would then want efficient operations for OR'ing LTN
conflict vectors with rows in the array.
This representation also opens a whole new range of allocation algorithms: ones
that store allocate TNs in different locations within different portions of the
program. This is because we can now represent a location being used to hold a
certain TN within an arbitrary subset of the blocks the TN is referenced in.
Pack goals:
Pack should:
Subject to resource constraints:
-- Minimize use costs
-- ``Register allocation''
Allocate as many values as possible in scarce ``good'' locations,
attempting to minimize the aggregate use cost for the entire program.
-- ``Save optimization''
Don't allocate values in registers when the save/restore costs exceed
the expected gain for keeping the value in a register. (Similar to
``opening costs'' in RAOC.) [Really just a case of representation
selection.]
-- Minimize preference costs
Eliminate as many moves as possible.
``Register allocation'' is basically an attempt to eliminate moves between
registers and memory. ``Save optimization'' counterbalances ``register
allocation'' to prevent it from becoming a pessimization, since saves can
introduce register/memory moves.
Preference optimization reduces the number of moves within an SC. Doing a good
job of honoring preferences is important to the success of the compiler, since
we have assumed in many places that moves will usually be optimized away.
The scarcity-oriented aspect of ``register allocation'' is handled by a greedy
algorithm in pack. We try to pack the ``most important'' TNs first, under the
theory that earlier packing is more likely to succeed due to fewer constraints.
The drawback of greedy algorithms is their inability to look ahead. Packing a
TN may mess up later ``register allocation'' by precluding packing of TNs that
are individually ``less important,'' but more important in aggregate. Packing a
TN may also prevent preferences from being honored.
Initial packing:
Pack all TNs restricted to a finite SC first, before packing any other TNs.
One might suppose that Pack would have to treat TNs in different environments
differently, but this is not the case. Pack simply assigns TNs to locations so
that no two conflicting TNs are in the same location. In the process of
implementing call semantics in conflict analysis, we cause TNs in different
environments not to conflict. In the case of passing TNs, cross environment
conflicts do exist, but this reflects reality, since the passing TNs are
live in both the caller and the callee. Environment semantics has already been
implemented at this point.
This means that Pack can pack all TNs simultaneously, using one data structure
to represent the conflicts for each location. So we have only one conflict set
per SB location, rather than separating this information by environment.
Load TN packing:
We create load TNs as needed in a post-pass to the initial packing. After TNs
are packed, it may be that some references to a TN will require it to be in a
SC other than the one it was packed in. We create load-TNs and pack them on
the fly during this post-pass.
What we do is have an optional SC restriction associated with TN-refs. If we
pack the TN in an SC which is different from the required SC for the reference,
then we create a TN for each such reference, and pack it into the required SC.
In many cases we will be able to pack the load TN with no hassle, but in
general we may need to spill a TN that has already been packed. We choose a
TN that isn't in use by the offending VOP, and then spill that TN onto the
stack for the duration of that VOP. If the VOP is a conditional, then we must
insert a new block interposed before the branch target so that the TN
value is restored regardless of which branch is taken.
Instead of remembering lifetime information from conflict analysis, we rederive
it. We scan each block backward while keeping track of which locations have
live TNs in them. When we find a reference that needs a load TN packed, we try
to pack it in an unused location. If we can't, we unpack the currently live TN
with the lowest cost and force it into an unbounded SC.
The per-location and per-TN conflict information used by pack doesn't
need to be updated when we pack a load TN, since we are done using those data
structures.
We also don't need to create any TN-Refs for load TNs. [??? How do we keep
track of load-tn lifetimes? It isn't really that hard, I guess. We just
remember which load TNs we created at each VOP, killing them when we pass the
loading (or saving) step. This suggests we could flush the Refs thread if we
were willing to sacrifice some flexibility in explicit temporary lifetimes.
Flushing the Refs would make creating the VMR representation easier.]
The lifetime analysis done during load-TN packing doubles as a consistency
check. If we see a read of a TN packed in a location which has a different TN
currently live, then there is a packing bug. If any of the TNs recorded as
being live at the block beginning are packed in a scarce SB, but aren't current
in that location, then we also have a problem.
The conflict structure for load TNs is fairly simple, the load TNs for
arguments and results all conflict with each other, and don't conflict with
much else. We just try packing in targeted locations before trying at random.
\chapter{Code generation}
This is fairly straightforward. We translate VOPs into instruction sequences
on a per-block basis.
After code generation, the VMR representation is gone. Everything is
represented by the assembler data structures.
\chapter{Assembly}
In effect, we do much of the work of assembly when the compiler is compiled.
The assembler makes one pass fixing up branch offsets, then squeezes out the
space left by branch shortening and dumps out the code along with the load-time
fixup information. The assembler also deals with dumping unboxed non-immediate
constants and symbols. Boxed constants are created by explicit constructor
code in the top-level form, while immediate constants are generated using
inline code.
[\#\#\# The basic output of the assembler is:
A code vector
A representation of the fixups along with indices into the code vector for
the fixup locations
A PC map translating PCs into source paths
This information can then be used to build an output file or an in-core
function object.
]
The assembler is table-driven and supports arbitrary instruction formats. As
far as the assembler is concerned, an instruction is a bit sequence that is
broken down into subsequences. Some of the subsequences are constant in value,
while others can be determined at assemble or load time.
\begin{verbatim}
Assemble Node Form*
Allow instructions to be emitted during the evaluation of the Forms by
defining Inst as a local macro. This macro caches various global
information in local variables. Node tells the assembler what node
ultimately caused this code to be generated. This is used to create the
pc=>source map for the debugger.
Assemble-Elsewhere Node Form*
Similar to Assemble, but the current assembler location is changed to
somewhere else. This is useful for generating error code and similar
things. Assemble-Elsewhere may not be nested.
Inst Name Arg*
Emit the instruction Name with the specified arguments.
Gen-Label
Emit-Label (Label)
Gen-Label returns a Label object, which describes a place in the code.
Emit-Label marks the current position as being the location of Label.
\end{verbatim}
\chapter{Dumping}
So far as input to the dumper/loader, how about having a list of Entry-Info
structures in the VMR-Component? These structures contain all information
needed to dump the associated function objects, and are only implicitly
associated with the functional/XEP data structures. Load-time constants that
reference these function objects should specify the Entry-Info, rather than the
functional (or something). We would then need to maintain some sort of
association so VMR conversion can find the appropriate Entry-Info.
Alternatively, we could initially reference the functional, and then later
clobber the reference to the Entry-Info.
We have some kind of post-pass that runs after assembly, going through the
functions and constants, annotating the VMR-Component for the benefit of the
dumper:
Resolve :Label load-time constants.
Make the debug info.
Make the entry-info structures.
Fasl dumper and in-core loader are implementation (but not instruction set)
dependent, so we want to give them a clear interface.
\begin{verbatim}
open-fasl-file name => fasl-file
Returns a ``fasl-file'' object representing all state needed by the dumper.
We objectify the state, since the fasdumper should be reentrant. (but
could fail to be at first.)
close-fasl-file fasl-file abort-p
Close the specified fasl-file.
fasl-dump-component component code-vector length fixups fasl-file
Dump the code, constants, etc. for component. Code-Vector is a vector
holding the assembled code. Length is the number of elements of Vector
that are actually in use. Fixups is a list of conses (offset . fixup)
describing the locations and things that need to be fixed up at load time.
If the component is a top-level component, then the top-level lambda will
be called after the component is loaded.
load-component component code-vector length fixups
Like Fasl-Dump-Component, but directly installs the code in core, running
any top-level code immediately. (???) but we need some way to glue
together the componenents, since we don't have a fasl table.
\end{verbatim}
Dumping:
Dump code for each component after compiling that component, but defer dumping
of other stuff. We do the fixups on the code vectors, and accumulate them in
the table.
We have to grovel the constants for each component after compiling that
component so that we can fix up load-time constants. Load-time constants are
values needed by the code that are computed after code generation/assembly
time. Since the code is fixed at this point, load-time constants are always
represented as non-immediate constants in the constant pool. A load-time
constant is distinguished by being a cons (Kind . What), instead of a Constant
leaf. Kind is a keyword indicating how the constant is computed, and What is
some context.
Some interesting load-time constants:
\begin{verbatim}
(:label . <label>)
Is replaced with the byte offset of the label within the code-vector.
(:code-vector . <component>)
Is replaced by the component's code-vector.
(:entry . <function>)
(:closure-entry . <function>)
Is replaced by the function-entry structure for the specified function.
:Entry is how the top-level component gets a handle on the function
definitions so that it can set them up.
\end{verbatim}
We also need to remember the starting offset for each entry, although these
don't in general appear as explicit constants.
We then dump out all the :Entry and :Closure-Entry objects, leaving any
constant-pool pointers uninitialized. After dumping each :Entry, we dump some
stuff to let genesis know that this is a function definition. Then we dump all
the constant pools, fixing up any constant-pool pointers in the already-dumped
function entry structures.
The debug-info *is* a constant: the first constant in every constant pool. But
the creation of this constant must be deferred until after the component is
compiled, so we leave a (:debug-info) placeholder. [Or maybe this is
implicitly added in by the dumper, being supplied in a VMR-component slot.]
Work out details of the interface between the back-end and the
assembler/dumper.
Support for multiple assemblers concurrently loaded? (for byte code)
We need various mechanisms for getting information out of the assembler.
We can get entry PCs and similar things into function objects by making a
Constant leaf, specifying that it goes in the closure, and then
setting the value after assembly.
We have an operation Label-Value which can be used to get the value of a
label after assembly and before the assembler data structures are
deallocated.
The function map can be constructed without any special help from the
assembler. Codegen just has to note the current label when the function
changes from one block to the next, and then use the final value of these
labels to make the function map.
Probably we want to do the source map this way too. Although this will
make zillions of spurious labels, we would have to effectively do that
anyway.
With both the function map and the source map, getting the locations right
for uses of Elsewhere will be a bit tricky. Users of Elsewhere will need
to know about how these maps are being built, since they must record the
labels and corresponding information for the elsewhere range. It would be
nice to have some cooperation from Elsewhere so that this isn't necessary,
otherwise some VOP writer will break the rules, resulting in code that is
nowhere.
The Debug-Info and related structures are dumped by consing up the
structure and making it be the value of a constant.
Getting the code vector and fixups dumped may be a bit more interesting. I
guess we want a Dump-Code-Vector function which dumps the code and fixups
accumulated by the current assembly, returning a magic object that will
become the code vector when it is dumped as a constant.
]

550
doc/internals/cmu/compiler-overview.tex Normal file
View file

@ -0,0 +1,550 @@
\chapter{Compiler Overview} % -*- Dictionary: design -*-
The structure of the compiler may be broadly characterized by describing the
compilation phases and the data structures that they manipulate. The steps in
the compilation are called phases rather than passes since they don't
necessarily involve a full pass over the code. The data structure used to
represent the code at some point is called an {\it intermediate
representation.}
Two major intermediate representations are used in the compiler:
\begin{itemize}
\item The Implicit Continuation Representation (ICR) represents the lisp-level
semantics of the source code during the initial phases. Partial evaluation and
semantic analysis are done on this representation. ICR is roughly equivalent
to a subset of Common Lisp, but is represented as a flow-graph rather than a
syntax tree. Phases which only manipulate ICR comprise the ``front end''. It
would be possible to use a different back end such as one that directly
generated code for a stack machine.
\item The Virtual Machine Representation (VMR) represents the implementation of
the source code on a virtual machine. The virtual machine may vary depending
on the the target hardware, but VMR is sufficiently stylized that most of the
phases which manipulate it are portable.
\end{itemize}
Each phase is briefly described here. The phases from ``local call analysis''
to ``constraint propagation'' all interact; for maximum optimization, they
are generally repeated until nothing new is discovered. The source files which
primarily contain each phase are listed after ``Files: ''.
\begin{description}
\item[ICR conversion]
Convert the source into ICR, doing macroexpansion and simple source-to-source
transformation. All names are resolved at this time, so we don't have to worry
about name conflicts later on. Files: {\tt ir1tran, srctran, typetran}
\item[Local call analysis] Find calls to local functions and convert them to
local calls to the correct entry point, doing keyword parsing, etc. Recognize
once-called functions as lets. Create {\it external entry points} for
entry-point functions. Files: {\tt locall}
\item[Find components]
Find flow graph components and compute depth-first ordering. Separate
top-level code from run-time code, and determine which components are top-level
components. Files: {\tt dfo}
\item[ICR optimize] A grab-bag of all the non-flow ICR optimizations. Fold
constant functions, propagate types and eliminate code that computes unused
values. Special-case calls to some known global functions by replacing them
with a computed function. Merge blocks and eliminate IF-IFs. Substitute let
variables. Files: {\tt ir1opt, ir1tran, typetran, seqtran, vm/vm-tran}
\item[Type constraint propagation]
Use global flow analysis to propagate information about lexical variable
types. Eliminate unnecessary type checks and tests. Files: {\tt constraint}
\item[Type check generation]
Emit explicit ICR code for any necessary type checks that are too complex to be
easily generated on the fly by the back end. Files: {\tt checkgen}
\item[Event driven operations]
Various parts of ICR are incrementally recomputed, either eagerly on
modification of the ICR, or lazily, when the relevant information is needed.
\begin{itemize}
\item Check that type assertions are satisfied, marking places where type
checks need to be done.
\item Locate let calls.
\item Delete functions and variables with no references
\end{itemize}
Files: {\tt ir1util}, {\tt ir1opt}
\item[ICR finalize]
This phase is run after all components have been compiled. It scans the
global variable references, looking for references to undefined variables
and incompatible function redefinitions. Files: {\tt ir1final}, {\tt main}.
\item[Environment analysis]
Determine which distinct environments need to be allocated, and what
context needed to be closed over by each environment. We detect non-local
exits and set closure variables. We also emit cleanup code as funny
function calls. This is the last pure ICR pass. Files: {\tt envanal}
\item[Global TN allocation (GTN)]
Iterate over all defined functions, determining calling conventions
and assigning TNs to local variables. Files: {\tt gtn}
\item[Local TN allocation (LTN)]
Use type and policy information to determine which VMR translation to use
for known functions, and then create TNs for expression evaluation
temporaries. We also accumulate some random information needed by VMR
conversion. Files: {\tt ltn}
\item[Control analysis]
Linearize the flow graph in a way that minimizes the number of branches. The
block-level structure of the flow graph is basically frozen at this point.
Files: {\tt control}
\item[Stack analysis]
Maintain stack discipline for unknown-values continuation in the presence
of local exits. Files: {\tt stack}
\item[Entry analysis]
Collect some back-end information for each externally callable function.
\item[VMR conversion] Convert ICR into VMR by translating nodes into VOPs.
Emit type checks. Files: {\tt ir2tran, vmdef}
\item[Copy propagation] Use flow analysis to eliminate unnecessary copying of
TN values. Files: {\tt copyprop}
\item[Representation selection]
Look at all references to each TN to determine which representation has the
lowest cost. Emit appropriate move and coerce VOPS for that representation.
\item[Lifetime analysis]
Do flow analysis to find the set of TNs whose lifetimes
overlap with the lifetimes of each TN being packed. Annotate call VOPs with
the TNs that need to be saved. Files: {\tt life}
\item[Pack]
Find a legal register allocation, attempting to minimize unnecessary moves.
Files: {\tt pack}
\item[Code generation]
Call the VOP generators to emit assembly code. Files: {\tt codegen}
\item[Pipeline reorganization] On some machines, move memory references
backward in the code so that they can overlap with computation. On machines
with delayed branch instructions, locate instructions that can be moved into
delay slots. Files: {\tt assem-opt}
\item[Assembly]
Resolve branches and convert into object code and fixup information.
Files: {\tt assembler}
\item[Dumping] Convert the compiled code into an object file or in-core
function. Files: {\tt debug-dump}, {\tt dump}, {\tt vm/core}
\end{description}
\chapter{The Implicit Continuation Representation}
The set of special forms recognized is exactly that specified in the Common
Lisp manual. Everything that is described as a macro in CLTL is a macro.
Large amounts of syntactic information are thrown away by the conversion to an
anonymous flow graph representation. The elimination of names eliminates the
need to represent most environment manipulation special forms. The explicit
representation of control eliminates the need to represent BLOCK and GO, and
makes flow analysis easy. The full Common Lisp LAMBDA is implemented with a
simple fixed-arg lambda, which greatly simplifies later code.
The elimination of syntactic information eliminates the need for most of the
``beta transformation'' optimizations in Rabbit. There are no progns, no
tagbodys and no returns. There are no ``close parens'' which get in the way of
determining which node receives a given value.
In ICR, computation is represented by Nodes. These are the node types:
\begin{description}
\item[if] Represents all conditionals.
\item[set] Represents a {\tt setq}.
\item[ref] Represents a constant or variable reference.
\item[combination] Represents a normal function call.
\item[MV-combination] Represents a {\tt multiple-value-call}. This is used to
implement all multiple value receiving forms except for {\tt
multiple-value-prog1}, which is implicit.
\item[bind]
This represents the allocation and initialization of the variables in
a lambda.
\item[return]
This collects the return value from a lambda and represents the
control transfer on return.
\item[entry] Marks the start of a dynamic extent that can have non-local exits
to it. Dynamic state can be saved at this point for restoration on re-entry.
\item[exit] Marks a potentially non-local exit. This node is interposed
between the non-local uses of a continuation and the {\tt dest} so that code to
do a non-local exit can be inserted if necessary.
\end{description}
Some slots are shared between all node types (via defstruct inheritance.) This
information held in common between all nodes often makes it possible to avoid
special-casing nodes on the basis of type. This shared information is
primarily concerned with the order of evaluation and destinations and
properties of results. This control and value flow is indicated in the node
primarily by pointing to continuations.
The {\tt continuation} structure represents information sufficiently related
to the normal notion of a continuation that naming it so seems sensible.
Basically, a continuation represents a place in the code, or alternatively the
destination of an expression result and a transfer of control. These two
notions are bound together for the same reasons that they are related in the
standard functional continuation interpretation.
A continuation may be deprived of either or both of its value or control
significance. If the value of a continuation is unused due to evaluation for
effect, then the continuation will have a null {\tt dest}. If the {\tt next}
node for a continuation is deleted by some optimization, then {\tt next} will
be {\tt :none}.
[\#\#\# Continuation kinds...]
The {\tt block} structure represents a basic block, in the the normal sense.
Control transfers other than simple sequencing are represented by information
in the block structure. The continuation for the last node in a block
represents only the destination for the result.
It is very difficult to reconstruct anything resembling the original source
from ICR, so we record the original source form in each node. The location of
the source form within the input is also recorded, allowing for interfaces such
as ``Edit Compiler Warnings''. See section \ref{source-paths}.
Forms such as special-bind and catch need to have cleanup code executed at all
exit points from the form. We represent this constraint in ICR by annotating
the code syntactically within the form with a Cleanup structure describing what
needs to be cleaned up. Environment analysis determines the cleanup locations
by watching for a change in the cleanup between two continuations. We can't
emit cleanup code during ICR conversion, since we don't know which exits will
be local until after ICR optimizations are done.
Special binding is represented by a call to the funny function \%Special-Bind.
The first argument is the Global-Var structure for the variable bound and the
second argument is the value to bind it to.
Some subprimitives are implemented using a macro-like mechanism for translating
\%PRIMITIVE forms into arbitrary lisp code. Subprimitives special-cased by VMR
conversion are represented by a call to the funny function \%\%Primitive. The
corresponding Template structure is passed as the first argument.
We check global function calls for syntactic legality with respect to any
defined function type function. If the call is illegal or we are unable to
tell if it is legal due to non-constant keywords, then we give a warning and
mark the function reference as :notinline to force a full call and cause
subsequent phases to ignore the call. If the call is legal and is to a known
function, then we annotate the Combination node with the Function-Info
structure that contains the compiler information for the function.
\section{Tail sets}
\#|
Probably want to have a GTN-like function result equivalence class mechanism
for ICR type inference. This would be like the return value propagation being
done by Propagate-From-Calls, but more powerful, less hackish, and known to
terminate. The ICR equivalence classes could probably be used by GTN, as well.
What we do is have local call analysis eagerly maintain the equivalence classes
of functions that return the same way by annotating functions with a Tail-Info
structure shared between all functions whose value could be the value of this
function. We don't require that the calls actually be tail-recursive, only
that the call deliver its value to the result continuation. [\#\#\# Actually
now done by ICR-OPTIMIZE-RETURN, which is currently making ICR optimize
mandatory.]
We can then use the Tail-Set during ICR type inference. It would have a type
that is the union across all equivalent functions of the types of all the uses
other than in local calls. This type would be recomputed during optimization
of return nodes. When the type changes, we would propagate it to all calls to
any of the equivalent functions. How do we know when and how to recompute the
type for a tail-set? Recomputation is driven by type propagation on the result
continuation.
This is really special-casing of RETURN nodes. The return node has the type
which is the union of all the non-call uses of the result. The tail-set is
found though the lambda. We can then recompute the overall union by taking the
union of the type per return node, rather than per-use.
How do result type assertions work? We can't intersect the assertions across
all functions in the equivalence class, since some of the call combinations may
not happen (or even be possible). We can intersect the assertion of the result
with the derived types for non-call uses.
When we do a tail call, we obviously can't check that the returned value
matches our assertion. Although in principle, we would like to be able to
check all assertions, to preserve system integrity, we only need to check
assertions that we depend on. We can afford to lose some assertion information
as long as we entirely lose it, ignoring it for type inference as well as for
type checking.
Things will work out, since the caller will see the tail-info type as the
derived type for the call, and will emit a type check if it needs a stronger
result.
A remaining question is whether we should intersect the assertion with
per-RETURN derived types from the very beginning (i.e. before the type check
pass). I think the answer is yes. We delay the type check pass so that we can
get our best guess for the derived type before we decide whether a check is
necessary. But with the function return type, we aren't committing to doing
any type check when we intersect with the type assertion; the need to type
check is still determined in the type check pass by examination of the result
continuation.
What is the relationship between the per-RETURN types and the types in the
result continuation? The assertion is exactly the Continuation-Asserted-Type
(note that the asserted type of result continuations will never change after
ICR conversion). The per-RETURN derived type is different than the
Continuation-Derived-Type, since it is intersected with the asserted type even
before Type Check runs. Ignoring the Continuation-Derived-Type probably makes
life simpler anyway, since this breaks the potential circularity of the
Tail-Info-Type will affecting the Continuation-Derived-Type, which affects...
When a given return has no non-call uses, we represent this by using
*empty-type*. This is consistent with the interpretation that a return type of
NIL means the function can't return.
\section{Hairy function representation}
Non-fixed-arg functions are represented using Optional-Dispatch. An
Optional-Dispatch has an entry-point function for each legal number of
optionals, and one for when extra args are present. Each entry point function
is a simple lambda. The entry point function for an optional is passed the
arguments which were actually supplied; the entry point function is expected to
default any remaining parameters and evaluate the actual function body.
If no supplied-p arg is present, then we can do this fairly easily by having
each entry point supply its default and call the next entry point, with the
last entry point containing the body. If there are supplied-p args, then entry
point function is replaced with a function that calls the original entry
function with T's inserted at the position of all the supplied args with
supplied-p parameters.
We want to be a bit clever about how we handle arguments declared special when
doing optional defaulting, or we will emit really gross code for special
optionals. If we bound the arg specially over the entire entry-point function,
then the entry point function would be caused to be non-tail-recursive. What
we can do is only bind the variable specially around the evaluation of the
default, and then read the special and store the final value of the special
into a lexical variable which we then pass as the argument. In the common case
where the default is a constant, we don't have to special-bind at all, since
the computation of the default is not affected by and cannot affect any special
bindings.
Keyword and rest args are both implemented using a LEXPR-like ``more
args'' convention. The More-Entry takes two arguments in addition to
the fixed and optional arguments: the argument context and count.
\verb+(ARG <context> <n>)+ accesses the N'th additional argument. Keyword
args are implemented directly using this mechanism. Rest args are
created by calling \%Listify-Rest-Args with the context and count.
The More-Entry parses the keyword arguments and passes the values to the main
function as positional arguments. If a keyword default is not constant, then
we pass a supplied-p parameter into the main entry and let it worry about
defaulting the argument. Since the main entry accepts keywords in parsed form,
we can parse keywords at compile time for calls to known functions. We keep
around the original parsed lambda-list and related information so that people
can figure out how to call the main entry.
\section{ICR representation of non-local exits}
All exits are initially represented by EXIT nodes:
How about an Exit node:
\begin{verbatim}
(defstruct (exit (:include node))
value)
\end{verbatim}
The Exit node uses the continuation that is to receive the thrown Value.
During optimization, if we discover that the Cont's home-lambda is the same as
the exit node's, then we can delete the Exit node, substituting the Cont for
all of the Value's uses.
The successor block of an EXIT is the entry block in the entered environment.
So we use the Exit node to mark the place where exit code is inserted. During
environment analysis, we need only insert a single block containing the entry
point stub.
We ensure that all Exits that aren't for a NLX don't have any Value, so that
local exits never require any value massaging.
The Entry node marks the beginning of a block or tagbody:
\begin{verbatim}
(defstruct (entry (:include node))
(continuations nil :type list))
\end{verbatim}
It contains a list of all the continuations that the body could exit to. The
Entry node is used as a marker for the place to snapshot state, including
the control stack pointer. Each lambda has a list of its Entries so
that environment analysis can figure out which continuations are really being
closed over. There is no reason for optimization to delete Entry nodes,
since they are harmless in the degenerate case: we just emit no code (like a
no-var let).
We represent CATCH using the lexical exit mechanism. We do a transformation
like this:
\begin{verbatim}
(catch 'foo xxx) ==>
(block #:foo
(%catch #'(lambda () (return-from #:foo (%unknown-values))) 'foo)
(%within-cleanup :catch
xxx))
\end{verbatim}
\%CATCH just sets up the catch frame which points to the exit function. \%Catch
is an ordinary function as far as ICR is concerned. The fact that the catcher
needs to be cleaned up is expressed by the Cleanup slots in the continuations
in the body. \%UNKNOWN-VALUES is a dummy function call which represents the
fact that we don't know what values will be thrown.
\%WITHIN-CLEANUP is a special special form that instantiates its first argument
as the current cleanup when converting the body. In reality, the lambda is
also created by the special special form \%ESCAPE-FUNCTION, which gives the
lambda a special :ESCAPE kind so that the back end knows not to generate any
code for it.
We use a similar hack in Unwind-Protect to represent the fact that the cleanup
forms can be invoked at arbitrarily random times.
\begin{verbatim}
(unwind-protect p c) ==>
(flet ((#:cleanup () c))
(block #:return
(multiple-value-bind
(#:next #:start #:count)
(block #:unwind
(%unwind-protect #'(lambda (x) (return-from #:unwind x)))
(%within-cleanup :unwind-protect
(return-from #:return p)))
(#:cleanup)
(%continue-unwind #:next #:start #:count))))
\end{verbatim}
We use the block \#:unwind to represent the entry to cleanup code in the case
where we are non-locally unwound. Calling of the cleanup function in the
drop-through case (or any local exit) is handled by cleanup generation. We
make the cleanup a function so that cleanup generation can add calls at local
exits from the protected form. \#:next, \#:start and \#:count are state used in
the case where we are unwound. They indicate where to go after doing the
cleanup and what values are being thrown. The cleanup encloses only the
protected form. As in CATCH, the escape function is specially tagged as
:ESCAPE. The cleanup function is tagged as :CLEANUP to inhibit let conversion
(since references are added in environment analysis.)
Notice that implementing these forms using closures over continuations
eliminates any need to special-case ICR flow analysis. Obviously we don't
really want to make heap-closures here. In reality these functions are
special-cased by the back-end according to their KIND.
\section{Block compilation}
One of the properties of ICR is that it supports ``block compilation'' by allowing
arbitrarily large amounts of code to be converted at once, with actual
compilation of the code being done at will.
In order to preserve the normal semantics we must recognize that proclamations
(possibly implicit) are scoped. A proclamation is in effect only from the time
of appearance of the proclamation to the time it is contradicted. The current
global environment at the end of a block is not necessarily the correct global
environment for compilation of all the code within the block. We solve this
problem by closing over the relevant information in the ICR at the time it is
converted. For example, each functional variable reference is marked as
inline, notinline or don't care. Similarly, each node contains a structure
known as a Cookie which contains the appropriate settings of the compiler
policy switches.
We actually convert each form in the file separately, creating a separate
``initial component'' for each one. Later on, these components are merged as
needed. The main reason for doing this is to cause EVAL-WHEN processing to be
interleaved with reading.
\section{Entry points}
\#|
Since we need to evaluate potentially arbitrary code in the XEP argument forms
(for type checking), we can't leave the arguments in the wired passing
locations. Instead, it seems better to give the XEP max-args fixed arguments,
with the passing locations being the true passing locations. Instead of using
\%XEP-ARG, we reference the appropriate variable.
Also, it might be a good idea to do argument count checking and dispatching
with explicit conditional code in the XEP. This would simplify both the code
that creates the XEP and the VMR conversion of XEPs. Also, argument count
dispatching would automatically benefit from any cleverness in compilation of
case-like forms (jump tables, etc). On the downside, this would push some
assumptions about how arg dispatching is done into ICR. But then we are
currently violating abstraction at least as badly in VMR conversion, which is
also supposed to be implementation independent.
|\#
As a side-effect of finding which references to known functions can be
converted to local calls, we find any references that cannot be converted.
References that cannot be converted to a local call must evaluate to a
``function object'' (or function-entry) that can be called using the full call
convention. A function that can be called from outside the component is called
an ``entry-point''.
Lots of stuff that happens at compile-time with local function calls must be
done at run-time when an entry-point is called.
It is desirable for optimization and other purposes if all the calls to every
function were directly present in ICR as local calls. We cannot directly do
this with entry-point functions, since we don't know where and how the
entry-point will be called until run-time.
What we do is represent all the calls possible from outside the component by
local calls within the component. For each entry-point function, we create a
corresponding lambda called the external entry point or XEP. This is a
function which takes the number of arguments passed as the first argument,
followed by arguments corresponding to each required or optional argument.
If an optional argument is unsupplied, the value passed into the XEP is
undefined. The XEP is responsible for doing argument count checking and
dispatching.
In the case of a fixed-arg lambda, we emit a call to the \%VERIFY-ARGUMENT-COUNT
funny function (conditional on policy), then call the real function on the
passed arguments. Even in this simple case, we benefit several ways from
having a separate XEP:
\begin{itemize}
\item The argument count checking is factored out, and only needs to
be done in full calls.
\item Argument type checking happens automatically as a consequence of
passing the XEP arguments in a local call to the real function.
This type checking is also only done in full calls.
\item The real function may use a non-standard calling convention for
the benefit of recursive or block-compiled calls. The XEP converts
arguments/return values to/from the standard convention. This also
requires little special-casing of XEPs.
\end{itemize}
If the function has variable argument count (represented by an
OPTIONAL-DISPATCH), then the XEP contains a COND which dispatches off of the
argument count, calling the appropriate entry-point function (which then does
defaulting). If there is a more entry (for keyword or rest args), then the XEP
obtains the more arg context and count by calling the \%MORE-ARG-CONTEXT funny
function.
All non-local-call references to functions are replaced with references to the
corresponding XEP. ICR optimization may discover a local call that was
previously a non-local reference. When we delete the reference to the XEP, we
may find that it has no references. In this case, we can delete the XEP,
causing the function to no longer be an entry-point.

6
doc/internals/cmu/compiler.tex Normal file
View file

@ -0,0 +1,6 @@
\part{Compiler Organization}
\input{compiler-overview}
\input{front}
\input{middle}
\input{back}
\input{interface}

657
doc/internals/cmu/debugger.tex Normal file
View file

@ -0,0 +1,657 @@
% -*- Dictionary: design; Package: C -*-
\chapter{Debugger}
Two classes of errors are handled by the Lisp debugger. These are
synchronous errors caused by something erring in program code and
asynchronous errors caused by some external context of execution
(clock interrupts, control-c interrupts). Asynchronous errors can
often be postponed if they are delivered at an inconvenient time.
Synchronous errors are frequently handled by directly invoking the
debugger. However, there are several places where the strategy of
jumping into the debugger is not used. In those situations the
compiler emits a stylized breakpoint; a breakpoint instruction
(usually an INT3) followed by several bytes of argument data. This
will cause a trip through the operating system and ultimately the
invocation of the C-level SIGTRAP handler which, in turn, interprets
the argument bytes following the breakpoint and dispatches to the
correct handler. There is a switch statement in ``sigtrap\_handler''
which gives the whole story on what types of errors rely on this
mechanism. The most commonly invoked handler is probably
``interrupt\_internal\_error'' as it fields such common exceptions as
the use of unbound symbols. To familiarize with the context these
traps are created in, one can disassemble just about any function and
look at the bottom of the disassembly for blocks of error handling code.
There will often be ``BREAK 10'' opcodes followed by several ``BYTE''
opcodes with the meaning of the arguments in neatly decoded form off
in the right-hand column.
The other types of synchronous errors are those errors delivered by
the operating system such as FPU traps and SIGSEGVs. The invocation
of those signals should be funneled through a C-level trampoline which
makes a callback into Lisp passing all of the signal handler
arguments. That code is pretty straight forward and the
``interrupt\_handle\_now'' function is pretty much where all of the
runtime logic is localized.
Handling asynchronous errors and deferred asynchronous errors is a bit more
involved\ldots
\section{Tracing and Breakpoints}
\label{sec:trace-and-breakpoints}
Here are a few notes on how tracing of compiled code works.
When a function is traced, a breakpoint instruction is placed at the
start of the function, replacing the instruction that was there.
(This is a \verb+:function-start+ breakpoint.) (This appears to be
one instruction after the no-arg parsing entry point.) The breakpoint
instruction is, of course, architecture-specific, but it must signal a
\verb+trap_Breakpoint+ trap.
When the code is run, the breakpoint instruction is executed causing a
trap. The trap handler runs \verb+HANDLE-BREAKPOINT+ to process it.
After doing the appropriate processing, we now need to continue. Of
course, since the real instruction has been replaced, we to run the
original instruction. This is done by now inserting a \emph{new}
breakpoint after the original breakpoint. This breakpoint must be of
the type \verb+trap_AfterBreakpoint+. The original instruction is
restored and execution continues from there. Then the
\verb+trap_AfterBreakpoint+ instruction gets executed. The handler
for this puts back the original breakpoint, thereby preserving the
breakpoint. Then we replace the AfterBreakpoint with the original
instruction and continue from there.
That's all pretty straightforward in concept.
When tracing, additional information is needed. Breakpoints have the
ability to run arbitrary lisp code to process the breakpoint. Tracing
uses this feature.
When this breakpoint is reached, \verb+HANDLE-BREAKPOINT+ runs the
breakpoint hook function. This function figures out where this
function would return to and creates a new return area and replaces
the original return address with this new address. Thus, when the
function returns, it returns to this new location instead of the
original.
This new return address is a specially created bogus LRA object. It
is a code-component whose body consists of a code template copied from
an assembly routine into the body. The assembly routine is the code
in \verb+function_end_breakpoint_guts+. This bogus LRA object stores
the real LRA for the function, and also an indication if the
known-return convention is used for this function.
The bogus LRA object contains a function-end breakpoint
(\verb+trap_FunctionEndBreakpoint+). When it's executed the trap
handler handles this breakpoint. It figures out where this trap come
from and calls \verb+HANDLE-BREAKPOINT+ to handle it.
\verb+HANDLE-BREAKPOINT+ returns and the trap handler arranges it so
that this bogus LRA returns to the real LRA.
Thus, we can do something when a Lisp function returns, like printing
out the return value for the function for tracing.
There are lots of internal details left out here, but gives a short
overview of how this works. For more info, look at
\verb+code/debug-int.lisp+ and \verb+lisp/breakpoint.c+, and, of
course, the various \verb+<foo>-arch.c+ files.
\chapter{Debugger Information}
\index{debugger information}
\label{debug-info}
Although the compiler's great freedom in choice of function call conventions
and variable representations has major efficiency advantages, it also has
unfortunate consequences for the debugger. The debug information that we need
is even more elaborate than for conventional ``compiled'' languages, since we
cannot even do a simple backtrace without some debug information. However,
once having gone this far, it is not that difficult to go the extra distance,
and provide full source level debugging of compiled code.
Full debug information has a substantial space penalty, so we allow different
levels of debug information to be specified. In the extreme case, we can
totally omit debug information.
\section{The Debug-Info Structure}
\index{debug-info structure}
The Debug-Info structure directly represents information about the
source code, and points to other structures that describe the layout of
run-time data structures.
Make some sort of minimal debug-info format that would support at least the
common cases of level 1 (since that is what we would release), and perhaps
level 0. Actually, it seems it wouldn't be hard to crunch nearly all of the
debug-function structure and debug-info function map into a single byte-vector.
We could have an uncrunch function that restored the current format. This
would be used by the debugger, and also could be used by purify to delete parts
of the debug-info even when the compiler dumps it in crunched form.
[Note that this isn't terribly important if purify is smart about
debug-info...]
Compiled source map representation:
[\#\#\# store in debug-function PC at which env is properly initialized, i.e.
args (and return-pc, etc.) in internal locations. This is where a
:function-start breakpoint would break.]
[\#\#\# Note that that we can easily cache the form-number => source-path or
form-number => form translation using a vector indexed by form numbers that we
build during a walk.]
Instead of using source paths in the debug-info, use ``form numbers''. The form
number of a form is the number of forms that we walk to reach that form when
doing a pre-order walk of the source form. [Might want to use a post-order
walk, as that would more closely approximate evaluation order.]
We probably want to continue using source-paths in the compiler, since they are
quick to compute and to get you to a particular form. [\#\#\# But actually, I
guess we don't have to precompute the source paths and annotate nodes with
them: instead we could annotate the nodes with the actual original source form.
Then if we wanted to find the location of that form, we could walk the root
source form, looking that original form. But we might still need to enter all
the forms in a hashtable so that we can tell during IR1 conversion that a given
form appeared in the original source.]
Note that form numbers have an interesting property: it is quite efficient to
determine whether an arbitrary form is a subform of some other form, since the
form number of B will be \verb+>+ than A's number and \verb+<+ A's next sibling's number iff
B is a subform of A.
This should be quite useful for doing the \verb|source=>pc| mapping in the debugger,
since that problem reduces to finding the subset of the known locations that
are for subforms of the specified form.
Assume a byte vector with a standard variable-length integer format, something
like this:
\begin{verbatim}
0..253 => the integer
254 => read next two bytes for integer
255 => read next four bytes for integer
\end{verbatim}
Then a compiled debug block is just a sequence of variable-length integers in a
particular order, something like this:
\begin{verbatim}
number of successors
...offsets of each successor in the function's blocks vector...
first PC
[offset of first top-level form (in forms) (only if not component default)]
form number of first source form
first live mask (length in bytes determined by number of VARIABLES)
...more <PC, top-level form offset, form-number, live-set> tuples...
\end{verbatim}
We determine the number of locations recorded in a block by finding the
start of the next compiled debug block in the blocks vector.
[\#\#\# Actually, only need 2 bits for number of successors {0,1,2}. We might
want to use other bits in the first byte to indicate the kind of location.]
[\#\#\# We could support local packing by having a general concept of ``alternate
locations'' instead of just regular and save locations. The location would have
a bit indicating that there are alternate locations, in which case we read the
number of alternate locations and then that many more SC-OFFSETs. In the
debug-block, we would have a second bit mask with bits set for TNs that are in
an alternate location. We then read a number for each such TN, with the value
being interpreted as an index into the Location's alternate locations.]
It looks like using structures for the compiled-location-info is too bulky.
Instead we need some packed binary representation.
First, let's represent an SC/offset pair with an ``SC-Offset'', which is an
integer with the SC in the low 5 bits and the offset in the remaining bits:
\begin{verbatim}
----------------------------------------------------
| Offset (as many bits as necessary) | SC (5 bits) |
----------------------------------------------------
\end{verbatim}
Probably the result should be constrained to fit in a fixnum, since it will be
more efficient and gives more than enough possible offsets.
We can then represent a compiled location like this:
\begin{verbatim}
single byte of boolean flags:
uninterned name
packaged name
environment-live
has distinct save location
has ID (name not unique in this fun)
name length in bytes (as var-length integer)
...name bytes...
[if packaged, var-length integer that is package name length]
...package name bytes...]
[If has ID, ID as var-length integer]
SC-Offset of primary location (as var-length integer)
[If has save SC, SC-Offset of save location (as var-length integer)]
\end{verbatim}
But for a whizzy breakpoint facility, we would need a good \verb+source=>code+ map.
Dumping a complete \verb+code=>source map+ might be as good a way as any to represent
this, due to the one-to-many relationship between source and code locations.
We might be able to get away with just storing the source locations for the
beginnings of blocks and maintaining a mapping from code ranges to blocks.
This would be fine both for the profiler and for the ``where am I running now''
indication. Users might also be convinced that it was most interesting to
break at block starts, but I don't really know how easily people could develop
an understanding of basic blocks.
It could also be a bit tricky to map an arbitrary user-designated source
location to some ``closest'' source location actually in the debug info.
This problem probably exists to some degree even with a full source map, since
some forms will never appear as the source of any node. It seems you might
have to negotiate with the user. He would mouse something, and then you would
highlight some source form that has a common prefix (i.e. is a prefix of the
user path, or vice-versa.) If they aren't happy with the result, they could
try something else. In some cases, the designated path might be a prefix of
several paths. This ambiguity might be resolved by picking the shortest path
or letting the user choose.
At the primitive level, I guess what this means is that the structure of source
locations (i.e. source paths) must be known, and the \verb+source=>code+ operation
should return a list of \verb+<source,code>+ pairs, rather than just a list of code
locations. This allows the debugger to resolve the ambiguity however it wants.
I guess the formal definition of which source paths we would return
is:
\begin{quote}
All source paths in the debug info that have a maximal common prefix with
the specified path. i.e. if several paths have the complete specified path
as a prefix, we return them all. Otherwise, all paths with an equally
large common prefix are returned: if the path with the most in common
matches only the first three elements, then we return all paths that match
in the first three elements. As a degenerate case (which probably
shouldn't happen), if there is no path with anything in common, then we
return *all* of the paths.
\end{quote}
In the DEBUG-SOURCE structure we may ultimately want a vector of the start
positions of each source form, since that would make it easier for the debugger
to locate the source. It could just open the file, FILE-POSITION to the form,
do a READ, then loop down the source path. Of course, it could read each form
starting from the beginning, but that might be too slow.
Do XEPs really need Debug-Functions? The only time that we will commonly end
up in the debugger on an XEP is when an argument type check fails. But I
suppose it would be nice to be able to print the arguments passed...
Note that assembler-level code motion such as pipeline reorganization can cause
problems with our PC maps. The assembler needs to know that debug info markers
are different from real labels anyway, so I suppose it could inhibit motion
across debug markers conditional on policy. It seems unworthwhile to remember
the node for each individual instruction.
For tracing block-compiled calls:
\begin{verbatim}
Info about return value passing locations?
Info about where all the returns are?
\end{verbatim}
We definitely need the return-value passing locations for debug-return. The
question is what the interface should be. We don't really want to have a
visible debug-function-return-locations operation, since there are various
value passing conventions, and we want to paper over the differences.
Probably should be a compiler option to initialize stack frame to a special
uninitialized object (some random immediate type). This would aid debugging,
and would also help GC problems. For the latter reason especially, this should
be locally-turn-onable (off of policy? the new debug-info quality?).
What about the interface between the evaluator and the debugger? (i.e. what
happens on an error, etc.) Compiler error handling should be integrated with
run-time error handling. Ideally the error messages should look the same.
Practically, in some cases the run-time errors will have less information. But
the error should look the same to the debugger (or at least similar).
\subsection{Debugger Interface}
How does the debugger interface to the ``evaluator'' (where the evaluator means
all of native code, byte-code and interpreted IR1)? It seems that it would be
much more straightforward to have a consistent user interface to debugging
all code representations if there was a uniform debugger interface to the
underlying stuff, and vice-versa.
Of course, some operations might not be supported by some representations, etc.
For example, fine-control stepping might not be available in native code.
In other cases, we might reduce an operation to the lowest common denominator,
for example fetching lexical variables by string and admitting the possibility
of ambiguous matches. [Actually, it would probably be a good idea to store the
package if we are going to allow variables to be closed over.]
Some objects we would need:
\begin{verbatim}
Location:
The constant information about the place where a value is stored,
everything but which particular frame it is in. Operations:
location name, type, etc.
location-value frame location (setf'able)
monitor-location location function
Function is called whenever location is set with the location,
frame and old value. If active values aren't supported, then we
dummy the effect using breakpoints, in which case the change won't
be noticed until the end of the block (and intermediate changes
will be lost.)
debug info:
All the debug information for a component.
Frame:
frame-changed-locations frame => location*
Return a list of the locations in frame that were changed since the
last time this function was called. Or something. This is for
displaying interesting state changes at breakpoints.
save-frame-state frame => frame-state
restore-frame-state frame frame-state
These operations allow the debugger to back up evaluation, modulo
side-effects and non-local control transfers. This copies and
restores all variables, temporaries, etc, local to the frame, and
also the current PC and dynamic environment (current catch, etc.)
At the time of the save, the frame must be for the running function
(not waiting for a call to return.) When we restore, the frame
becomes current again, effectively exiting from any frames on top.
(Of course, frame must not already be exited.)
Thread:
Representation of which stack to use, etc.
Block:
What successors the block has, what calls there are in the block.
(Don't need to know where calls are as long as we know called function,
since can breakpoint at the function.) Whether code in this block is
wildly out of order due to being the result of loop-invariant
optimization, etc. Operations:
block-successors block => code-location*
block-forms block => (source-location code-location)*
Return the corresponding source locations and code locations for
all forms (and form fragments) in the block.
\end{verbatim}
\subsection{Variable maps}
There are about five things that the debugger might want to know about a
variable:
\begin{itemize}
\item[Name]
Although a lexical variable's name is ``really'' a symbol (package and
all), in practice it doesn't seem worthwhile to require all the symbols
for local variable names to be retained. There is much less VM and GC
overhead for a constant string than for a symbol. (Also it is useful
to be able to access gensyms in the debugger, even though they are
theoretically ineffable).
\item[ID]
Which variable with the specified name is this? It is possible to have
multiple variables with the same name in a given function. The ID is
something that makes Name unique, probably a small integer. When
variables aren't unique, we could make this be part of the name, e.g.
``FOO\#1'', ``FOO\#2''. But there are advantages to keeping this separate,
since in many cases lifetime information can be used to disambiguate,
making qualification unnecessary.
\item[SC]
When unboxed representations are in use, we must have type information
to properly read and write a location. We only need to know the
SC for this, which would be amenable to a space-saving
numeric encoding.
\item[Location]
Simple: the offset in SC. [Actually, we need the save location too.]
\item[Lifetime]
In what parts of the program does this variable hold a meaningful
value? It seems prohibitive to record precise lifetime information,
both in space and compiler effort, so we will have to settle for some
sort of approximation.
The finest granularity at which it is easy to determine liveness is
the block: we can regard the variable lifetime as the set of blocks
that the variable is live in. Of course, the variable may be dead (and
thus contain meaningless garbage) during arbitrarily large portions of
the block.
Note that this subsumes the notion of which function a variable belongs
to. A given block is only in one function, so the function is
implicit.
\end{itemize}
The variable map should represent this information space-efficiently and with
adequate computational efficiency.
The SC and ID can be represented as small integers. Although the ID can in
principle be arbitrarily large, it should be $<$100 in practice. The location
can be represented by just the offset (a moderately small integer), since the
SB is implicit in the SC.
The lifetime info can be represented either as a bit-vector indexed by block
numbers, or by a list of block numbers. Which is more compact depends both on
the size of the component and on the number of blocks the variable is live in.
In the limit of large component size, the sparse representation will be more
compact, but it isn't clear where this crossover occurs. Of course, it would
be possible to use both representations, choosing the more compact one on a
per-variable basis. Another interesting special case is when the variable is
live in only one block: this may be common enough to be worth picking off,
although it is probably rarer for named variables than for TNs in general.
If we dump the type, then a normal list-style type descriptor is fine: the
space overhead is small, since the shareability is high.
We could probably save some space by cleverly representing the var-info as
parallel vectors of different types, but this would be more painful in use.
It seems better to just use a structure, encoding the unboxed fields in a
fixnum. This way, we can pass around the structure in the debugger, perhaps
even exporting it from the low-level debugger interface.
[\#\#\# We need the save location too. This probably means that we need two slots
of bits, since we need the save offset and save SC. Actually, we could let the
save SC be implied by the normal SC, since at least currently, we always choose
the same save SC for a given SC. But even so, we probably can't fit all that
stuff in one fixnum without squeezing a lot, so we might as well split and
record both SCs.
In a localized packing scheme, we would have to dump a different var-info
whenever either the main location or the save location changes. As a practical
matter, the save location is less likely to change than the main location, and
should never change without the main location changing.
One can conceive of localized packing schemes that do saving as a special case
of localized packing. If we did this, then the concept of a save location
might be eliminated, but this would require major changes in the IR2
representation for call and/or lifetime info. Probably we will want saving to
continue to be somewhat magical.]
How about:
\begin{verbatim}
(defstruct var-info
;;
;; This variable's name. (symbol-name of the symbol)
(name nil :type simple-string)
;;
;; The SC, ID and offset, encoded as bit-fields.
(bits nil :type fixnum)
;;
;; The set of blocks this variable is live in. If a bit-vector, then it has
;; a 1 when indexed by the number of a block that it is live in. If an
;; I-vector, then it lists the live block numbers. If a fixnum, then that is
;; the number of the sole live block.
(lifetime nil :type (or vector fixnum))
;;
;; The variable's type, represented as list-style type descriptor.
type)
\end{verbatim}
Then the debug-info holds a simple-vector of all the var-info structures for
that component. We might as well make it sorted alphabetically by name, so
that we can binary-search to find the variable corresponding to a particular
name.
We need to be able to translate PCs to block numbers. This can be done by an
I-Vector in the component that contains the start location of each block. The
block number is the index at which we find the correct PC range. This requires
that we use an emit-order block numbering distinct from the IR2-Block-Number,
but that isn't any big deal. This seems space-expensive, but it isn't too bad,
since it would only be a fraction of the code size if the average block length
is a few words or more.
An advantage of our per-block lifetime representation is that it directly
supports keeping a variable in different locations when in different blocks,
i.e. multi-location packing. We use a different var-info for each different
packing, since the SC and offset are potentially different. The Name and ID
are the same, representing the fact that it is the same variable. It is here
that the ID is most significant, since the debugger could otherwise make
same-name variables unique all by itself.
\subsection{Stack parsing}
[\#\#\# Probably not worth trying to make the stack parseable from the bottom up.
There are too many complications when we start having variable sized stuff on
the stack. It seems more profitable to work on making top-down parsing robust.
Since we are now planning to wire the bottom-up linkage info, scanning from the
bottom to find the top frame shouldn't be too inefficient, even when there was
a runaway recursion. If we somehow jump into hyperspace, then the debugger may
get confused, but we can debug this sort of low-level system lossage using
ADB.]
There are currently three relevant context pointers:
\begin{itemize}
\item The PC. The current PC is wired (implicit in the machine). A saved
PC (RETURN-PC) may be anywhere in the current frame.
\item The current stack context (CONT). The current CONT is wired.
A saved CONT (OLD-CONT) may be anywhere in the current frame.
\item The current code object (ENV). The current ENV is wired.
When saved, this is extra-difficult to locate, since it is saved
by the caller, and is thus at an unknown offset in OLD-CONT,
rather than anywhere in the current frame.
\end{itemize}
We must have all of these to parse the stack.
With the proposed Debug-Function, we parse the stack (starting at the top) like
this:
\begin{enumerate}
\item Use ENV to locate the current Debug-Info
\item Use the Debug-Info and PC to determine the current Debug-Function.
\item Use the Debug-Function to find the OLD-CONT and RETURN-PC.
\item Find the old ENV by searching up the stack for a saved code object
containing the RETURN-PC.
\item Assign old ENV to ENV, OLD-CONT to CONT, RETURN-PC to PC and goto 1.
\end{enumerate}
If we changed the function representation so that the code and environment were
a single object, then the location of the old ENV would be simplified. But we
still need to represent ENV as separate from PC, since interrupts and errors
can happen when the current PC isn't positioned at a valid return PC.
It seems like it might be a good idea to save OLD-CONT, RETURN-PC and ENV at
the beginning of the frame (before any stack arguments). Then we wouldn't have
to search to locate ENV, and we also have a hope of parsing the stack even if
it is damaged. As long as we can locate the start of some frame, we can trace
the stack above that frame. We can recognize a probable frame start by
scanning the stack for a code object (presumably a saved ENV).
Probably we want some fairly general
mechanism for specifying that a TN should be considered to be live for the
duration of a specified environment. It would be somewhat easier to specify
that the TN is live for all time, but this would become very space-inefficient
in large block compilations.
This mechanism could be quite useful for other debugger-related things. For
example, when debuggability is important, we could make the TNs holding
arguments live for the entire environment. This would guarantee that a
backtrace would always get the right value (modulo setqs).
Note that in this context, ``environment'' means the Environment structure (one
per non-let function). At least according to current plans, even when we do
inter-routine register allocation, the different functions will have different
environments: we just ``equate'' the environments. So the number of live
per-environment TNs is bounded by the size of a ``function'', and doesn't blow up
in block compilation.
The implementation is simple: per-environment TNs are flagged by the
:Environment kind. :Environment TNs are treated the same as :Normal TNs by
everyone except for lifetime/conflict analysis. An environment's TNs are also
stashed in a list in the IR2-Environment structure. During the conflict
analysis post-pass, we look at each block's environment, and make all the
environment's TNs always-live in that block.
We can implement the ``fixed save location'' concept needed for lazy frame
creation by allocating the save TNs as wired TNs at IR2 conversion time. We
would use the new ``environment lifetime'' concept to specify the lifetimes of
the save locations. There isn't any run-time overhead if we never get around
to using the save TNs. [Pack would also have to notice TNs with pre-allocated
save TNs, packing the original TN in the stack location if its FSC is the
stack.]
We want a standard (recognizable) format for an ``escape'' frame. We must make
an escape frame whenever we start running another function without the current
function getting a chance to save its registers. This may be due either to a
truly asynchronous event such as a software interrupt, or due to an ``escape''
from a miscop. An escape frame marks a brief conversion to a callee-saves
convention.
Whenever a miscop saves registers, it should make an escape frame. This
ensures that the ``current'' register contents can always be located by the
debugger. In this case, it may be desirable to be able to indicate that only
partial saving has been done. For example, we don't want to have to save all
the FP registers just so that we can use a couple extra general registers.
When the debugger see an escape frame, it knows that register values are
located in the escape frame's ``register save'' area, rather than in the normal
save locations.
It would be nice if there was a better solution to this internal error concept.
One problem is that it seems there is a substantial space penalty for emitting
all that error code, especially now that we don't share error code between
errors because we want to preserve the source context in the PC. But this
probably isn't really all that bad when considered as a fraction of the code.
For example, the check part of a type check is 12 bytes, whereas the error part
is usually only 6. In this case, we could never reduce the space overhead for
type checks by more than 1/3, thus the total code size reduction would be
small. This will be made even less important when we do type check
optimizations to reduce the number of type checks.
Probably we should stick to the same general internal error mechanism, but make
it interact with the debugger better by allocating linkage registers and
allowing proceedable errors. We could support shared error calls and
non-proceedable errors when space is more important than debuggability, but
this is probably more complexity than is worthwhile.
We jump or trap to a routine that saves the context (allocating at most the
return PC register). We then encode the error and context in the code
immediately following the jump/trap. (On the MIPS, the error code can be
encoded in the trap itself.) The error arguments would be encoded as
SC-offsets relative to the saved context. This could solve both the
arg-trashing problem and save space, since we could encode the SC-offsets more
tersely than the corresponding move instructions.

92
doc/internals/cmu/design.tex Normal file
View file

@ -0,0 +1,92 @@
%%\documentstyle[cmu-titlepage]{report} % -*- Dictionary: design -*-
%\documentstyle{report} % -*- Dictionary: design -*-
\documentclass{report}
\usepackage{ifpdf}
\usepackage{ifthen}
\usepackage{calc}
\usepackage{palatino}
\usepackage[hyperindex=false,colorlinks=false,urlcolor=blue]{hyperref}
\title{Design of CMU Common Lisp}
\date{January 15, 2003}
\author{Robert A. MacLachlan (ed)}
\ifpdf
\pdfinfo{
/Author (Robert A. MacLachlan, ed)
/Title (Design of CMU Common Lisp)
}
% Add section numbers to the bookmarks, and open 2 levels by default.
\hypersetup{bookmarksnumbered=true,
bookmarksopen=true,
bookmarksopenlevel=2}
\fi
%%\trnumber{CMU-CS-91-???}
%% This code taken from the LaTeX companion. It's meant as a
%% replacement for the description environment. We want one that
%% prints description items in a fixed size box and puts the
%% description itself on the same line or the next depending on the
%% size of the item.
\newcommand{\entrylabel}[1]{\mbox{#1}\hfil}
\newenvironment{entry}{%
\begin{list}{}%
{\renewcommand{\makelabel}{\entrylabel}%
\setlength{\labelwidth}{45pt}%
\setlength{\leftmargin}{\labelwidth+\labelsep}}}%
{\end{list}}
\newlength{\Mylen}
\newcommand{\Lentrylabel}[1]{%
\settowidth{\Mylen}{#1}%
\ifthenelse{\lengthtest{\Mylen > \labelwidth}}%
{\parbox[b]{\labelwidth}% term > labelwidth
{\makebox[0pt][l]{#1}\\}}%
{#1}%
\hfil\relax}
\newenvironment{Lentry}{%
\renewcommand{\entrylabel}{\Lentrylabel}
\begin{entry}}%
{\end{entry}}
\setcounter{tocdepth}{2}
\setcounter{secnumdepth}{3}
\def\textfraction{.1}
\def\bottomfraction{.9} % was .3
\def\topfraction{.9}
\newcommand{\code}[1]{\textnormal{{\sffamily #1}}}
%% Some common abbreviations
\newcommand{\cmucl}{\textsc{cmucl}}
%% Set up margins
\setlength{\oddsidemargin}{-10pt}
\setlength{\evensidemargin}{-10pt}
\setlength{\topmargin}{-40pt}
\setlength{\headheight}{12pt}
\setlength{\headsep}{25pt}
\setlength{\footskip}{30pt}
\setlength{\textheight}{9.25in}
\setlength{\textwidth}{6.75in}
\setlength{\columnsep}{0.375in}
\setlength{\columnseprule}{0pt}
\begin{document}
\maketitle
\abstract{This report documents internal details of the CMU Common Lisp
compiler and run-time system. CMU Common Lisp is a public domain
implementation of Common Lisp that runs on various Unix workstations.
This document is a work in progress: neither the contents nor the
presentation are completed. Nevertheless, it provides some useful
background information, in particular regarding the \cmucl{} compiler.}
\tableofcontents
\include{architecture}
\include{compiler}
\include{retargeting}
\include{run-time}
\appendix
\include{glossary}
\end{document}

23
doc/internals/cmu/environment.tex Normal file
View file

@ -0,0 +1,23 @@
\chapter{The Type System}
\chapter{The Info Database}
The info database provides a functional interface to global
information about named things in \cmucl{}. Information is considered to
be global if it must persist between invocations of the compiler. The
use of a functional interface eliminates the need for the compiler to
worry about the details of the representation. The info database also
handles the need to multiple ``global'' environments, which makes it
possible to change something in the compiler without trashing the
running Lisp environment.
The info database contains arbitrary lisp values, addressed by a
combination of name, class and type. The Name is an EQUAL-thing which
is the name of the thing that we are recording information about.
Class is the kind of object involved: typical classes are Function,
Variable, Type. A type names a particular piece of information within
a given class. Class and Type are symbols, but are compared with
STRING=.

100
doc/internals/cmu/executable.tex Normal file
View file

@ -0,0 +1,100 @@
\chapter{Saving Executable Lisp Images}
The :executable flag to SAVE-LISP allows saving the Lisp core sections into an ELF binary that
is executable. To do this, the ELF format is adapted to emulate the old core format.
\section{Saving}
The first part of the save process is almost the same as saving a core-file image except that
instead of a single core image, three ELF object files are created. These are temporary files
called CORRO.o, CORSTA.o and CORDYN.o containing the read-only, static and dynamic spaces.
After that, a script called linker.sh is invoked to use the system linker to create the
executable file. The linker links a file called lisp.a with the three object files. When
SAVE-LISP calls linker.sh, it passes the address of the initial function to be run when lisp is
restarted.
The linker.sh script does several things.
\begin{enumerate}
\item It locates the directory where the C runtime object files are kept.
\item It passes the name of the linker directive script to the linker.
\item It uses the linker to define two symbols: builtin_image_flag and initial_function_addr.
The first symbol has to be the address of some location in memory with non-zero contents
because it is used to tell the system that it is an executable rather than the traditional
runtime loader with core file. This is definitely a kludge but something like this is needed
because it gets around a bootstrapping issue. The second symbol is the startup function;
ordinarily this would be saved in the core file header. It would be possible to save this
information in the ELF file but since we have to use the linker to define builtin_image_flag
anyway, it is simpler to just define initial_function_addr the same way.
\item Finally, the linker script links the executable with the correct set of libraries and
object files. One of these files is lisp.a, containing the object code of the runtime
loader program, which is no longer a separate file.
\end{enumerate}
\section{Running}
The startup procedure for an executable image attempts to emulate the startup procedure for a
core image file by using information saved in the ELF section headers.
When an executable image starts up, it sees the builtin_image_flag that tells it that it is an
executable image. It reads the ELF header of the executable file. It finds the string section
of the file. It then reads ELF section headers, looking for sections with the names CORRO,
CORSTA or CORDYN (it uses the string section to look up the names). Once it finds them, it
mmaps them into the memory space indicated by the parameters in the ELF section header.
Finally it sets variables indicating the starts of the three spaces. It also sets a variable
to indicate how big the dynamic space is.
\section{Porting the Linker Directive Script}
The following describes how to create a linker directive script. It assumes that the system is
based on GCC and the GCC compiler toolchain (in particular, GNU ld).
The linker directive script is used to set up the memory layout for the system. It is a
modified version of the normal linker directive script used by the system.
To obtain a copy of this script (for porting to another operating system), the command ``ld
--verbose'' must be run. This will dump the linker script to standard output. Save this to a
file and edit it as follows.
First, add a ``PHDRS'' part after the SEARCH_DIR part and before the SECTIONS part. The PHDRS
part should look like this:
PHDRS
{
headers PT_PHDR PHDRS ;
interp PT_INTERP ;
text PT_LOAD FILEHDR PHDRS ;
data PT_LOAD ;
dynamic PT_DYNAMIC ;
note PT_NOTE ;
/* Add core sections for Lisp. */
CORRO PT_NOTE ;
CORSTA PT_NOTE ;
CORDYN PT_NOTE ;
}
Note the three CORXXX sections. They are given type PT_NOTE so the system will not try to
process them. An attempt was made to use the PT_LOAD type (and thereby taking advantage of the
ELF loader automatically mmapping the sections into place) but this caused systems to get
confused about the memory layout so that any use of malloc would fail.
Right after the place where the file has something like this:
_end = .;
PROVIDE (end = .);
. = DATA_SEGMENT_END (.);
add the following lines (modified for your system):
/* Lisp core sections. */
CORRO 0x10000000 : { CORRO.o (CORRO) } :CORRO
CORSTA 0x28f00000 : { CORSTA.o (CORSTA) } :CORSTA
CORDYN 0x48000000 : { CORDYN.o (CORDYN) } :CORDYN
Note that the addresses (the second field) must correspond to the addresses of the read-only,
static and dynamic sections given in src/lisp/xxx-validate.h.
This linker directive script will reside in the library: directory along with the linker.sh
script and the lisp.a file created by rebuild-lisp.sh. It should be given a descriptive name
that indicates the system it is used for.

640
doc/internals/cmu/fasl.tex Normal file
View file

@ -0,0 +1,640 @@
\chapter{Fasload File Format}% -*- Dictionary: design -*-
\section{General}
The purpose of Fasload files is to allow concise storage and rapid
loading of Lisp data, particularly function definitions. The intent
is that loading a Fasload file has the same effect as loading the
source file from which the Fasload file was compiled, but accomplishes
the tasks more efficiently. One noticeable difference, of course, is
that function definitions may be in compiled form rather than
S-expression form. Another is that Fasload files may specify in what
parts of memory the Lisp data should be allocated. For example,
constant lists used by compiled code may be regarded as read-only.
In some Lisp implementations, Fasload file formats are designed to
allow sharing of code parts of the file, possibly by direct mapping
of pages of the file into the address space of a process. This
technique produces great performance improvements in a paged
time-sharing system. Since the Mach project is to produce a
distributed personal-computer network system rather than a
time-sharing system, efficiencies of this type are explicitly {\it not}
a goal for the CMU Common Lisp Fasload file format.
On the other hand, CMU Common Lisp is intended to be portable, as it will
eventually run on a variety of machines. Therefore an explicit goal
is that Fasload files shall be transportable among various
implementations, to permit efficient distribution of programs in
compiled form. The representations of data objects in Fasload files
shall be relatively independent of such considerations as word
length, number of type bits, and so on. If two implementations
interpret the same macrocode (compiled code format), then Fasload
files should be completely compatible. If they do not, then files
not containing compiled code (so-called ``Fasdump'' data files) should
still be compatible. While this may lead to a format which is not
maximally efficient for a particular implementation, the sacrifice of
a small amount of performance is deemed a worthwhile price to pay to
achieve portability.
The primary assumption about data format compatibility is that all
implementations can support I/O on finite streams of eight-bit bytes.
By ``finite'' we mean that a definite end-of-file point can be detected
irrespective of the content of the data stream. A Fasload file will
be regarded as such a byte stream.
\section{Strategy}
A Fasload file may be regarded as a human-readable prefix followed by
code in a funny little language. When interpreted, this code will
cause the construction of the encoded data structures. The virtual
machine which interprets this code has a {\it stack} and a {\it table},
both initially empty. The table may be thought of as an expandable
register file; it is used to remember quantities which are needed
more than once. The elements of both the stack and the table are
Lisp data objects. Operators of the funny language may take as
operands following bytes of the data stream, or items popped from the
stack. Results may be pushed back onto the stack or pushed onto the
table. The table is an indexable stack that is never popped; it is
indexed relative to the base, not the top, so that an item once
pushed always has the same index.
More precisely, a Fasload file has the following macroscopic
organization. It is a sequence of zero or more groups concatenated
together. End-of-file must occur at the end of the last group. Each
group begins with a series of seven-bit ASCII characters terminated
by one or more bytes of all ones \verb|#xFF|; this is called the
{\it header}. Following the bytes which terminate the header is the
{\it body}, a stream of bytes in the funny binary language. The body
of necessity begins with a byte other than \verb|#xFF|. The body is
terminated by the operation {\tt FOP-END-GROUP}.
The first nine characters of the header must be \verb|FASL FILE| in
upper-case letters. The rest may be any ASCII text, but by
convention it is formatted in a certain way. The header is divided
into lines, which are grouped into paragraphs. A paragraph begins
with a line which does {\it not} begin with a space or tab character,
and contains all lines up to, but not including, the next such line.
The first word of a paragraph, defined to be all characters up to but
not including the first space, tab, or end-of-line character, is the
{\it name} of the paragraph. A Fasload file header might look something like
this:
\begin{verbatim}
FASL FILE >SteelesPerq>User>Guy>IoHacks>Pretty-Print.Slisp
Package Pretty-Print
Compiled 31-Mar-1988 09:01:32 by some random luser
Compiler Version 1.6, Lisp Version 3.0.
Functions: INITIALIZE DRIVER HACK HACK1 MUNGE MUNGE1 GAZORCH
MINGLE MUDDLE PERTURB OVERDRIVE GOBBLE-KEYBOARD
FRY-USER DROP-DEAD HELP CLEAR-MICROCODE
%AOS-TRIANGLE %HARASS-READTABLE-MAYBE
Macros: PUSH POP FROB TWIDDLE
\end{verbatim}
{\it one or more bytes of \verb|#xFF|}
The particular paragraph names and contents shown here are only intended as
suggestions.
\section{Fasload Language}
Each operation in the binary Fasload language is an eight-bit
(one-byte) opcode. Each has a name beginning with ``{\tt FOP-}''. In
the following descriptions, the name is followed by operand
descriptors. Each descriptor denotes operands that follow the opcode
in the input stream. A quantity in parentheses indicates the number
of bytes of data from the stream making up the operand. Operands
which implicitly come from the stack are noted in the text. The
notation ``$\Rightarrow$ stack'' means that the result is pushed onto the
stack; ``$\Rightarrow$ table'' similarly means that the result is added to the
table. A construction like ``{\it n}(1) {\it value}({\it n})'' means that
first a single byte {\it n} is read from the input stream, and this
byte specifies how many bytes to read as the operand named {\it value}.
All numeric values are unsigned binary integers unless otherwise
specified. Values described as ``signed'' are in two's-complement form
unless otherwise specified. When an integer read from the stream
occupies more than one byte, the first byte read is the least
significant byte, and the last byte read is the most significant (and
contains the sign bit as its high-order bit if the entire integer is
signed).
Some of the operations are not necessary, but are rather special
cases of or combinations of others. These are included to reduce the
size of the file or to speed up important cases. As an example,
nearly all strings are less than 256 bytes long, and so a special
form of string operation might take a one-byte length rather than a
four-byte length. As another example, some implementations may
choose to store bits in an array in a left-to-right format within
each word, rather than right-to-left. The Fasload file format may
support both formats, with one being significantly more efficient
than the other for a given implementation. The compiler for any
implementation may generate the more efficient form for that
implementation, and yet compatibility can be maintained by requiring
all implementations to support both formats in Fasload files.
Measurements are to be made to determine which operation codes are
worthwhile; little-used operations may be discarded and new ones
added. After a point the definition will be ``frozen'', meaning that
existing operations may not be deleted (though new ones may be added;
some operations codes will be reserved for that purpose).
\begin{description}
\item[0:] \hspace{2em} {\tt FOP-NOP} \\
No operation. (This is included because it is recognized
that some implementations may benefit from alignment of operands to some
operations, for example to 32-bit boundaries. This operation can be used
to pad the instruction stream to a desired boundary.)
\item[1:] \hspace{2em} {\tt FOP-POP} \hspace{2em} $\Rightarrow$ \hspace{2em} table \\
One item is popped from the stack and added to the table.
\item[2:] \hspace{2em} {\tt FOP-PUSH} \hspace{2em} {\it index}(4) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
Item number {\it index} of the table is pushed onto the stack.
The first element of the table is item number zero.
\item[3:] \hspace{2em} {\tt FOP-BYTE-PUSH} \hspace{2em} {\it index}(1) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
Item number {\it index} of the table is pushed onto the stack.
The first element of the table is item number zero.
\item[4:] \hspace{2em} {\tt FOP-EMPTY-LIST} \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
The empty list ({\tt ()}) is pushed onto the stack.
\item[5:] \hspace{2em} {\tt FOP-TRUTH} \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
The standard truth value ({\tt T}) is pushed onto the stack.
\item[6:] \hspace{2em} {\tt FOP-SYMBOL-SAVE} \hspace{2em} {\it n}(4) \hspace{2em} {\it name}({\it n})
\hspace{2em} $\Rightarrow$ \hspace{2em} stack \& table\\
The four-byte operand {\it n} specifies the length of the print name
of a symbol. The name follows, one character per byte,
with the first byte of the print name being the first read.
The name is interned in the default package,
and the resulting symbol is both pushed onto the stack and added to the table.
\item[7:] \hspace{2em} {\tt FOP-SMALL-SYMBOL-SAVE} \hspace{2em} {\it n}(1) \hspace{2em} {\it name}({\it n}) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \& table\\
The one-byte operand {\it n} specifies the length of the print name
of a symbol. The name follows, one character per byte,
with the first byte of the print name being the first read.
The name is interned in the default package,
and the resulting symbol is both pushed onto the stack and added to the table.
\item[8:] \hspace{2em} {\tt FOP-SYMBOL-IN-PACKAGE-SAVE} \hspace{2em} {\it index}(4)
\hspace{2em} {\it n}(4) \hspace{2em} {\it name}({\it n})
\hspace{2em} $\Rightarrow$ \hspace{2em} stack \& table\\
The four-byte {\it index} specifies a package stored in the table.
The four-byte operand {\it n} specifies the length of the print name
of a symbol. The name follows, one character per byte,
with the first byte of the print name being the first read.
The name is interned in the specified package,
and the resulting symbol is both pushed onto the stack and added to the table.
\item[9:] \hspace{2em} {\tt FOP-SMALL-SYMBOL-IN-PACKAGE-SAVE} \hspace{2em} {\it index}(4)
\hspace{2em} {\it n}(1) \hspace{2em} {\it name}({\it n}) \hspace{2em}
$\Rightarrow$ \hspace{2em} stack \& table\\
The four-byte {\it index} specifies a package stored in the table.
The one-byte operand {\it n} specifies the length of the print name
of a symbol. The name follows, one character per byte,
with the first byte of the print name being the first read.
The name is interned in the specified package,
and the resulting symbol is both pushed onto the stack and added to the table.
\item[10:] \hspace{2em} {\tt FOP-SYMBOL-IN-BYTE-PACKAGE-SAVE} \hspace{2em} {\it index}(1)
\hspace{2em} {\it n}(4) \hspace{2em} {\it name}({\it n})
\hspace{2em} $\Rightarrow$ \hspace{2em} stack \& table\\
The one-byte {\it index} specifies a package stored in the table.
The four-byte operand {\it n} specifies the length of the print name
of a symbol. The name follows, one character per byte,
with the first byte of the print name being the first read.
The name is interned in the specified package,
and the resulting symbol is both pushed onto the stack and added to the table.
\item[11:]\hspace{2em} {\tt FOP-SMALL-SYMBOL-IN-BYTE-PACKAGE-SAVE} \hspace{2em} {\it index}(1)
\hspace{2em} {\it n}(1) \hspace{2em} {\it name}({\it n}) \hspace{2em}
$\Rightarrow$ \hspace{2em} stack \& table\\
The one-byte {\it index} specifies a package stored in the table.
The one-byte operand {\it n} specifies the length of the print name
of a symbol. The name follows, one character per byte,
with the first byte of the print name being the first read.
The name is interned in the specified package,
and the resulting symbol is both pushed onto the stack and added to the table.
\item[12:] \hspace{2em} {\tt FOP-UNINTERNED-SYMBOL-SAVE} \hspace{2em} {\it n}(4) \hspace{2em} {\it name}({\it n})
\hspace{2em} $\Rightarrow$ \hspace{2em} stack \& table\\
Like {\tt FOP-SYMBOL-SAVE}, except that it creates an uninterned symbol.
\item[13:] \hspace{2em} {\tt FOP-UNINTERNED-SMALL-SYMBOL-SAVE} \hspace{2em} {\it n}(1)
\hspace{2em} {\it name}({\it n}) \hspace{2em} $\Rightarrow$ \hspace{2em} stack
\& table\\
Like {\tt FOP-SMALL-SYMBOL-SAVE}, except that it creates an uninterned symbol.
\item[14:] \hspace{2em} {\tt FOP-PACKAGE} \hspace{2em} $\Rightarrow$ \hspace{2em} table \\
An item is popped from the stack; it must be a symbol. The package of
that name is located and pushed onto the table.
\item[15:] \hspace{2em} {\tt FOP-LIST} \hspace{2em} {\it length}(1) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
The unsigned operand {\it length} specifies a number of
operands to be popped from the stack. These are made into a list
of that length, and the list is pushed onto the stack.
The first item popped from the stack becomes the last element of
the list, and so on. Hence an iterative loop can start with
the empty list and perform ``pop an item and cons it onto the list''
{\it length} times.
(Lists of length greater than 255 can be made by using {\tt FOP-LIST*}
repeatedly.)
\item[16:] \hspace{2em} {\tt FOP-LIST*} \hspace{2em} {\it length}(1) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
This is like {\tt FOP-LIST} except that the constructed list is terminated
not by {\tt ()} (the empty list), but by an item popped from the stack
before any others are. Therefore {\it length}+1 items are popped in all.
Hence an iterative loop can start with
a popped item and perform ``pop an item and cons it onto the list''
{\it length}+1 times.
\item[17-24:] \hspace{2em} {\tt FOP-LIST-1}, {\tt FOP-LIST-2}, ..., {\tt FOP-LIST-8} \\
{\tt FOP-LIST-{\it k}} is like {\tt FOP-LIST} with a byte containing {\it k}
following it. These exist purely to reduce the size of Fasload files.
Measurements need to be made to determine the useful values of {\it k}.
\item[25-32:] \hspace{2em} {\tt FOP-LIST*-1}, {\tt FOP-LIST*-2}, ..., {\tt FOP-LIST*-8} \\
{\tt FOP-LIST*-{\it k}} is like {\tt FOP-LIST*} with a byte containing {\it k}
following it. These exist purely to reduce the size of Fasload files.
Measurements need to be made to determine the useful values of {\it k}.
\item[33:] \hspace{2em} {\tt FOP-INTEGER} \hspace{2em} {\it n}(4) \hspace{2em} {\it value}({\it n}) \hspace{2em}
$\Rightarrow$ \hspace{2em} stack \\
A four-byte unsigned operand specifies the number of following
bytes. These bytes define the value of a signed integer in two's-complement
form. The first byte of the value is the least significant byte.
\item[34:] \hspace{2em} {\tt FOP-SMALL-INTEGER} \hspace{2em} {\it n}(1) \hspace{2em} {\it value}({\it n})
\hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
A one-byte unsigned operand specifies the number of following
bytes. These bytes define the value of a signed integer in two's-complement
form. The first byte of the value is the least significant byte.
\item[35:] \hspace{2em} {\tt FOP-WORD-INTEGER} \hspace{2em} {\it value}(4) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
A four-byte signed integer (in the range $-2^{31}$ to $2^{31}-1$) follows the
operation code. A LISP integer (fixnum or bignum) with that value
is constructed and pushed onto the stack.
\item[36:] \hspace{2em} {\tt FOP-BYTE-INTEGER} \hspace{2em} {\it value}(1) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
A one-byte signed integer (in the range -128 to 127) follows the
operation code. A LISP integer (fixnum or bignum) with that value
is constructed and pushed onto the stack.
\item[37:] \hspace{2em} {\tt FOP-STRING} \hspace{2em} {\it n}(4) \hspace{2em} {\it name}({\it n})
\hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
The four-byte operand {\it n} specifies the length of a string to
construct. The characters of the string follow, one per byte.
The constructed string is pushed onto the stack.
\item[38:] \hspace{2em} {\tt FOP-SMALL-STRING} \hspace{2em} {\it n}(1) \hspace{2em} {\it name}({\it n}) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
The one-byte operand {\it n} specifies the length of a string to
construct. The characters of the string follow, one per byte.
The constructed string is pushed onto the stack.
\item[39:] \hspace{2em} {\tt FOP-VECTOR} \hspace{2em} {\it n}(4) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
The four-byte operand {\it n} specifies the length of a vector of LISP objects
to construct. The elements of the vector are popped off the stack;
the first one popped becomes the last element of the vector.
The constructed vector is pushed onto the stack.
\item[40:] \hspace{2em} {\tt FOP-SMALL-VECTOR} \hspace{2em} {\it n}(1) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
The one-byte operand {\it n} specifies the length of a vector of LISP objects
to construct. The elements of the vector are popped off the stack;
the first one popped becomes the last element of the vector.
The constructed vector is pushed onto the stack.
\item[41:] \hspace{2em} {\tt FOP-UNIFORM-VECTOR} \hspace{2em} {\it n}(4) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
The four-byte operand {\it n} specifies the length of a vector of LISP objects
to construct. A single item is popped from the stack and used to initialize
all elements of the vector. The constructed vector is pushed onto the stack.
\item[42:] \hspace{2em} {\tt FOP-SMALL-UNIFORM-VECTOR} \hspace{2em} {\it n}(1) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
The one-byte operand {\it n} specifies the length of a vector of LISP objects
to construct. A single item is popped from the stack and used to initialize
all elements of the vector. The constructed vector is pushed onto the stack.
\item[43:] \hspace{2em} {\tt FOP-INT-VECTOR} \hspace{2em} {\it len}(4) \hspace{2em}
{\it size}(1) \hspace{2em} {\it data}($\left\lceil len*count/8\right\rceil$)
\hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
The four-byte operand {\it n} specifies the length of a vector of
unsigned integers to be constructed. Each integer is {\it size}
bits long, and is packed according to the machine's native byte ordering.
{\it size} must be a directly supported i-vector element size. Currently
supported values are 1,2,4,8,16 and 32.
\item[44:] \hspace{2em} {\tt FOP-UNIFORM-INT-VECTOR} \hspace{2em} {\it n}(4) \hspace{2em} {\it size}(1) \hspace{2em}
{\it value}(@ceiling$<${\it size}/8$>$) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
The four-byte operand {\it n} specifies the length of a vector of unsigned
integers to construct.
Each integer is {\it size} bits big, and is initialized to the value
of the operand {\it value}.
The constructed vector is pushed onto the stack.
\item[45:] \hspace{2em} {\tt FOP-LAYOUT} \hspace{2em} \\
Pops the stack four times to get the name, length, inheritance and depth for a layout object.
\item[46:] \hspace{2em} {\tt FOP-SINGLE-FLOAT} \hspace{2em} {\it data}(4) \hspace{2em}
$\Rightarrow$ \hspace{2em} stack \\
The {\it data} bytes are read as an integer, then turned into an IEEE single
float (as though by {\tt make-single-float}).
\item[47:] \hspace{2em} {\tt FOP-DOUBLE-FLOAT} \hspace{2em} {\it data}(8) \hspace{2em}
$\Rightarrow$ \hspace{2em} stack \\
The {\it data} bytes are read as an integer, then turned into an IEEE double
float (as though by {\tt make-double-float}).
\item[48:] \hspace{2em} {\tt FOP-STRUCT} \hspace{2em} {\it n}(4) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
The four-byte operand {\it n} specifies the length structure to construct. The
elements of the vector are popped off the stack; the first one popped becomes
the last element of the structure. The constructed vector is pushed onto the
stack.
\item[49:] \hspace{2em} {\tt FOP-SMALL-STRUCT} \hspace{2em} {\it n}(1) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
The one-byte operand {\it n} specifies the length structure to construct. The
elements of the vector are popped off the stack; the first one popped becomes
the last element of the structure. The constructed vector is pushed onto the
stack.
\item[50-52:] Unused
\item[53:] \hspace{2em} {\tt FOP-EVAL} \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
Pop an item from the stack and evaluate it (give it to {\tt EVAL}).
Push the result back onto the stack.
\item[54:] \hspace{2em} {\tt FOP-EVAL-FOR-EFFECT} \\
Pop an item from the stack and evaluate it (give it to {\tt EVAL}).
The result is ignored.
\item[55:] \hspace{2em} {\tt FOP-FUNCALL} \hspace{2em} {\it nargs}(1) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
Pop {\it nargs}+1 items from the stack and apply the last one popped
as a function to
all the rest as arguments (the first one popped being the last argument).
Push the result back onto the stack.
\item[56:] \hspace{2em} {\tt FOP-FUNCALL-FOR-EFFECT} \hspace{2em} {\it nargs}(1) \\
Pop {\it nargs}+1 items from the stack and apply the last one popped
as a function to
all the rest as arguments (the first one popped being the last argument).
The result is ignored.
\item[57:] \hspace{2em} {\tt FOP-CODE-FORMAT} \hspace{2em} {\it implementation}(1)
\hspace{2em} {\it version}(1) \\
This FOP specifiers the code format for following code objects. The operations
{\tt FOP-CODE} and its relatives may not occur in a group until after {\tt
FOP-CODE-FORMAT} has appeared; there is no default format. The {\it
implementation} is an integer indicating the target hardware and environment.
See {\tt compiler/generic/vm-macs.lisp} for the currently defined
implementations. {\it version} for an implementation is increased whenever
there is a change that renders old fasl files unusable.
\item[58:] \hspace{2em} {\tt FOP-CODE} \hspace{2em} {\it nitems}(4) \hspace{2em} {\it size}(4) \hspace{2em}
{\it code}({\it size}) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
A compiled function is constructed and pushed onto the stack.
This object is in the format specified by the most recent
occurrence of {\tt FOP-CODE-FORMAT}.
The operand {\it nitems} specifies a number of items to pop off
the stack to use in the ``boxed storage'' section. The operand {\it code}
is a string of bytes constituting the compiled executable code.
\item[59:] \hspace{2em} {\tt FOP-SMALL-CODE} \hspace{2em} {\it nitems}(1) \hspace{2em} {\it size}(2) \hspace{2em}
{\it code}({\it size}) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
A compiled function is constructed and pushed onto the stack.
This object is in the format specified by the most recent
occurrence of {\tt FOP-CODE-FORMAT}.
The operand {\it nitems} specifies a number of items to pop off
the stack to use in the ``boxed storage'' section. The operand {\it code}
is a string of bytes constituting the compiled executable code.
\item[60] \hspace{2em} {\tt FOP-FDEFINITION} \hspace{2em} \\
Pops the stack to get an fdefinition.
\item[61] \hspace{2em} {\tt FOP-SANCTIFY-FOR-EXECUTION} \hspace{2em} \\
A code component is popped from the stack, and the necessary magic is applied
to the code so that it can be executed.
\item[62:] \hspace{2em} {\tt FOP-VERIFY-TABLE-SIZE} \hspace{2em} {\it size}(4) \\
If the current size of the table is not equal to {\it size},
then an inconsistency has been detected. This operation
is inserted into a Fasload file purely for error-checking purposes.
It is good practice for a compiler to output this at least at the
end of every group, if not more often.
\item[63:] \hspace{2em} {\tt FOP-VERIFY-EMPTY-STACK} \\
If the stack is not currently empty,
then an inconsistency has been detected. This operation
is inserted into a Fasload file purely for error-checking purposes.
It is good practice for a compiler to output this at least at the
end of every group, if not more often.
\item[64:] \hspace{2em} {\tt FOP-END-GROUP} \\
This is the last operation of a group. If this is not the
last byte of the file, then a new group follows; the next
nine bytes must be ``{\tt FASL FILE}''.
\item[65:] \hspace{2em} {\tt FOP-POP-FOR-EFFECT} \hspace{2em} stack \hspace{2em} $\Rightarrow$ \hspace{2em} \\
One item is popped from the stack.
\item[66:] \hspace{2em} {\tt FOP-MISC-TRAP} \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
A trap object is pushed onto the stack.
\item[67:] \hspace{2em} {\tt FOP-DOUBLE-DOUBLE-FLOAT} \hspace{2em} {\it double-double-float}(8) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
The next 8 bytes are read, and a double-double-float number is constructed.
\item[68:] \hspace{2em} {\tt FOP-CHARACTER} \hspace{2em} {\it character}(3) \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
The three bytes are read as an integer then converted to a character. This FOP
is currently rather useless, as extended characters are not supported.
\item[69:] \hspace{2em} {\tt FOP-SHORT-CHARACTER} \hspace{2em} {\it character}(1) \hspace{2em}
$\Rightarrow$ \hspace{2em} stack \\
The one byte specifies the code of a Common Lisp character object. A character
is constructed and pushed onto the stack.
\item[70:] \hspace{2em} {\tt FOP-RATIO} \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
Creates a ratio from two integers popped from the stack.
The denominator is popped first, the numerator second.
\item[71:] \hspace{2em} {\tt FOP-COMPLEX} \hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
Creates a complex number from two numbers popped from the stack.
The imaginary part is popped first, the real part second.
\item[72] \hspace{2em} {\tt FOP-COMPLEX-SINGLE-FLOAT} {\it real(4)} {\it imag(4)}\hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
Creates a complex single-float number from the following 8 bytes.
\item[73] \hspace{2em} {\tt FOP-COMPLEX-DOUBLE-FLOAT} {\it real(8)} {\it imag(8)}\hspace{2em} $\Rightarrow$ \hspace{2em} stack \\
Creates a complex double-float number from the following 16 bytes.
\item[74:] \hspace{2em} {\tt FOP-FSET} \hspace{2em} \\
Except in the cold loader (Genesis), this is a no-op with two stack arguments.
In the initial core this is used to make DEFUN functions defined at cold-load
time so that global functions can be called before top-level forms are run
(which normally installs definitions.) Genesis pops the top two things off of
the stack and effectively does (SETF SYMBOL-FUNCTION).
\item[75:] \hspace{2em} {\tt FOP-LISP-SYMBOL-SAVE} \hspace{2em} {\it n}(4) \hspace{2em} {\it name}({\it n})
\hspace{2em} $\Rightarrow$ \hspace{2em} stack \& table\\
Like {\tt FOP-SYMBOL-SAVE}, except that it creates a symbol in the LISP
package.
\item[76:] \hspace{2em} {\tt FOP-LISP-SMALL-SYMBOL-SAVE} \hspace{2em} {\it n}(1)
\hspace{2em} {\it name}({\it n}) \hspace{2em} $\Rightarrow$ \hspace{2em} stack
\& table\\
Like {\tt FOP-SMALL-SYMBOL-SAVE}, except that it creates a symbol in the LISP
package.
\item[77:] \hspace{2em} {\tt FOP-KEYWORD-SYMBOL-SAVE} \hspace{2em} {\it n}(4) \hspace{2em} {\it name}({\it n})
\hspace{2em} $\Rightarrow$ \hspace{2em} stack \& table\\
Like {\tt FOP-SYMBOL-SAVE}, except that it creates a symbol in the
KEYWORD package.
\item[78:] \hspace{2em} {\tt FOP-KEYWORD-SMALL-SYMBOL-SAVE} \hspace{2em} {\it n}(1)
\hspace{2em} {\it name}({\it n}) \hspace{2em} $\Rightarrow$ \hspace{2em} stack
\& table\\
Like {\tt FOP-SMALL-SYMBOL-SAVE}, except that it creates a symbol in the
KEYWORD package.
\item[79-80:] Unused
\item[81:] \hspace{2em} {\tt FOP-NORMAL-LOAD}\\
This FOP is used in conjunction with the cold loader (Genesis) to read
top-level package manipulation forms. These forms are to be read as though by
the normal loaded, so that they can be evaluated at cold load time, instead of
being dumped into the initial core image. A no-op in normal loading.
\item[82:] \hspace{2em} {\tt FOP-MAYBE-COLD-LOAD}\\
Undoes the effect of {\tt FOP-NORMAL-LOAD}.
\item[83:] \hspace{2em} {\tt FOP-ARRAY} \hspace{2em} {\it rank}(4)
\hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
This operation creates a simple array header (used for simple-arrays with rank
/= 1). The data vector is popped off of the stack, and then {\it rank}
dimensions are popped off of the stack (the highest dimensions is on top.)
\item[84:] \hspace{2em} {\tt FOP-SINGLE-FLOAT-VECTOR} \hspace{2em} {\it length}(4) {\it data}(n)
\hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
Creates a {\it (simple-array single-float (*))} object. The number of single-floats is {\it length}.
\item[85:] \hspace{2em} {\tt FOP-DOUBLE-FLOAT-VECTOR} \hspace{2em} {\it length}(4) {\it data}(n)
\hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
Creates a {\it (simple-array double-float (*))} object. The number of double-floats is {\it length}.
\item[86:] \hspace{2em} {\tt FOP-COMPLEX-SINGLE-FLOAT-VECTOR} \hspace{2em} {\it length}(4) {\it data}(n)
\hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
Creates a {\it (simple-array (complex single-float) (*))} object. The number of complex single-floats is {\it length}.
\item[87:] \hspace{2em} {\tt FOP-COMPLEX-DOUBLE-FLOAT-VECTOR} \hspace{2em} {\it length}(4) {\it data}(n)
\hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
Creates a {\it (simple-array (complex double-float) (*))} object. The number of complex double-floats is {\it length}.
\item[88:] \hspace{2em} {\tt FOP-DOUBLE-DOUBLE-FLOAT-VECTOR} \hspace{2em} {\it length}(4) {\it data}(n)
\hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
Creates a {\it (simple-array double-double-float (*))} object. The number of double-double-floats is {\it length}.
\item[89:] \hspace{2em} {\tt FOP-COMPLEX-DOUBLE-DOUBLE-FLOAT} \hspace{2em} {\it data}(32)
\hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
Creates a {\it (complex double-double-float)} object from the following 32 bytes of data.
\item[90:] \hspace{2em} {\tt FOP-COMPLEX-DOUBLE-DOUBLE-FLOAT-VECTOR} \hspace{2em} {\it length}(4) {\it data}(n)
\hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
Creates a {\it (simple-arra (complex double-double-float) (*))} object. The number of complex double-double-floats is {\it length}.
\item[91-139:] Unused
\item[140:] \hspace{2em} {\tt FOP-ALTER-CODE} \hspace{2em} {\it index}(4)\\
This operation modifies the constants part of a code object (necessary for
creating certain circular function references.) It pops the new value and code
object are off of the stack, storing the new value at the specified index.
\item[141:] \hspace{2em} {\tt FOP-BYTE-ALTER-CODE} \hspace{2em} {\it index}(1)\\
Like {\tt FOP-ALTER-CODE}, but has only a one byte offset.
\item[142:] \hspace{2em} {\tt FOP-FUNCTION-ENTRY} \hspace{2em} {\it index}(4)
\hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
Initializes a function-entry header inside of a pre-existing code object, and
returns the corresponding function descriptor. {\it index} is the byte offset
inside of the code object where the header should be plunked down. The stack
arguments to this operation are the code object, function name, function debug
arglist and function type.
\item[143:] \hspace{2em} {\tt FOP-MAKE-BYTE-COMPILED-FUNCTION} \hspace{2em} {\it size}(1) \hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
Create a byte-compiled function. {\it FIXME:} describe what's on the stack.
\item[144:] \hspace{2em} {\tt FOP-ASSEMBLER-CODE} \hspace{2em} {\it length}(4)
\hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
This operation creates a code object holding assembly routines. {\it length}
bytes of code are read and placed in the code object, and the code object
descriptor is pushed on the stack. This FOP is only recognized by the cold
loader (Genesis.)
\item[145:] \hspace{2em} {\tt FOP-ASSEMBLER-ROUTINE} \hspace{2em} {\it offset}(4)
\hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
This operation records an entry point into an assembler code object (for use
with {\tt FOP-ASSEMBLER-FIXUP}). The routine name (a symbol) is on stack top.
The code object is underneath. The entry point is defined at {\it offset}
bytes inside the code area of the code object, and the code object is left on
stack top (allowing multiple uses of this FOP to be chained.) This FOP is only
recognized by the cold loader (Genesis.)
\item[146:] Unused
\item[147:] \hspace{2em} {\tt FOP-FOREIGN-FIXUP} \hspace{2em} {\it len}(1)
\hspace{2em} {\it name}({\it len})
\hspace{2em} {\it offset}(4) \hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
This operation resolves a reference to a foreign (C) symbol. {\it len} bytes
are read and interpreted as the symbol {\it name}. First the {\it kind} and the
code-object to patch are popped from the stack. The kind is a target-dependent
symbol indicating the instruction format of the patch target (at {\it offset}
bytes from the start of the code area.) The code object is left on
stack top (allowing multiple uses of this FOP to be chained.)
\item[148:] \hspace{2em} {\tt FOP-ASSEMBLER-FIXUP} \hspace{2em} {\it offset}(4)
\hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
This operation resolves a reference to an assembler routine. The stack args
are ({\it routine-name}, {\it kind} and {\it code-object}). The kind is a
target-dependent symbol indicating the instruction format of the patch target
(at {\it offset} bytes from the start of the code area.) The code object is
left on stack top (allowing multiple uses of this FOP to be chained.)
\item[149:] \hspace{2em} {\tt FOP-CODE-OBJECT-FIXUP}
\hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
{\it FIXME:} Describe what this does!
\item[150:] \hspace{2em} {\tt FOP-FOREIGN-DATA-FIXUP}
\hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
{\it FIXME:} Describe what this does!
\item[151-156:] Unused
\item[157:] \hspace{2em} {\tt FOP-LONG-CODE-FORMAT} \hspace{2em} {\it implementation}(1)
\hspace{2em} {\it version}(4) \\
Like FOP-CODE-FORMAT, except that the version is 32 bits long.
\item[158-199:] Unused
\item[200:] \hspace{2em} {\tt FOP-RPLACA} \hspace{2em} {\it table-idx}(4)
\hspace{2em} {\it cdr-offset}(4)\\
\item[201:] \hspace{2em} {\tt FOP-RPLACD} \hspace{2em} {\it table-idx}(4)
\hspace{2em} {\it cdr-offset}(4)\\
These operations destructively modify a list entered in the table. {\it
table-idx} is the table entry holding the list, and {\it cdr-offset} designates
the cons in the list to modify (like the argument to {\tt nthcdr}.) The new
value is popped off of the stack, and stored in the {\tt car} or {\tt cdr},
respectively.
\item[202:] \hspace{2em} {\tt FOP-SVSET} \hspace{2em} {\it table-idx}(4)
\hspace{2em} {\it vector-idx}(4)\\
Destructively modifies a {\tt simple-vector} entered in the table. Pops the
new value off of the stack, and stores it in the {\it vector-idx} element of
the contents of the table entry {\it table-idx.}
\item[203:] \hspace{2em} {\tt FOP-NTHCDR} \hspace{2em} {\it cdr-offset}(4)
\hspace{2em} $\Rightarrow$ \hspace{2em} stack\\
Does {\tt nthcdr} on the top-of stack, leaving the result there.
\item[204:] \hspace{2em} {\tt FOP-STRUCTSET} \hspace{2em} {\it table-idx}(4)
\hspace{2em} {\it vector-idx}(4)\\
Like {\tt FOP-SVSET}, except it alters structure slots.
\item[205-254:] Unused
\item[255:] \hspace{2em} {\tt FOP-END-HEADER} \\ Indicates the end of a group header,
as described above.
\end{description}

968
doc/internals/cmu/front.tex Normal file
View file

@ -0,0 +1,968 @@
\chapter{ICR conversion} % -*- Dictionary: design -*-
\section{Canonical forms}
\#|
Would be useful to have a Freeze-Type proclamation. Its primary use would be
to say that the indicated type won't acquire any new subtypes in the future.
This allows better open-coding of structure type predicates, since the possible
types that would satisfy the predicate will be constant at compile time, and
thus can be compiled as a skip-chain of EQ tests.
Of course, this is only a big win when the subtypes are few: the most important
case is when there are none. If the closure of the subtypes is much larger
than the average number of supertypes of an inferior, then it is better to grab
the list of superiors out of the object's type, and test for membership in that
list.
Should type-specific numeric equality be done by EQL rather than =? i.e.
should = on two fixnums become EQL and then convert to EQL/FIXNUM?
Currently we transform EQL into =, which is complicated, since we have to prove
the operands are the class of numeric type before we do it. Also, when EQL
sees one operand is a FIXNUM, it transforms to EQ, but the generator for EQ
isn't expecting numbers, so it doesn't use an immediate compare.
\subsection{Array hackery}
Array type tests are transformed to \verb|%array-typep|, separation of the
implementation-dependent array-type handling. This way we can transform
STRINGP to:
\begin{verbatim}
(or (simple-string-p x)
(and (complex-array-p x)
(= (array-rank x) 1)
(simple-string-p (%array-data x))))
\end{verbatim}
In addition to the similar bit-vector-p, we also handle vectorp and any type
tests on which the a dimension isn't wild.
[Note that we will want to expand into frobs compatible with those that
array references expand into so that the same optimizations will work on both.]
These changes combine to convert hairy type checks into hairy typep's, and then
convert hairyp typeps into simple typeps.
Do we really need non-VOP templates? It seems that we could get the
desired effect through implementation-dependent ICR transforms. The
main risk would be of obscuring the type semantics of the code. We
could fairly easily retain all the type information present at the
time the tranform is run, but if we discover new type information,
then it won't be propagated unless the VM also supplies type inference
methods for its internal frobs (precluding the use of
\verb|%PRIMITIVE|, since primitives don't have derive-type methods.)
I guess one possibility would be to have the call still considered ``known'' even
though it has been transformed. But this doesn't work, since we start doing
LET optimizations that trash the arglist once the call has been transformed
(and indeed we want to.)
Actually, I guess the overhead for providing type inference methods for the
internal frobs isn't that great, since we can usually borrow the inference
method for a Common Lisp function. For example, in our AREF case:
\begin{verbatim}
(aref x y)
==>
(let ((#:len (array-dimension x 0)))
(%unchecked-aref x (%check-in-bounds y #:len)))
\end{verbatim}
Now in this case, if we made \verb|%UNCHECKED-AREF| have the same
derive-type method as AREF, then if we discovered something new about
X's element type, we could derive a new type for the entire
expression.
Actually, it seems that baring this detail at the ICR level is
beneficial, since it admits the possibility of optimizing away the bounds
check using type information. If we discover X's dimensions, then
\verb|#:LEN| becomes a constant that can be substituted. Then
\verb|%CHECK-IN-BOUNDS| can notice that the bound is constant and
check it against the type for Y. If Y is known to be in range, then we
can optimize away the bounds check.
Actually in this particular case, the best thing to do would be if we
discovered the bound is constant, then replace the bounds check with an
implicit type check. This way all the type check optimization mechanisms would
be brought into the act.
So we actually want to do the bounds-check expansion as soon as possible,
rather than later than possible: it should be a source-transform, enabled by
the fast-safe policy.
With multi-dimensional arrays we probably want to explicitly do the index
computation: this way portions of the index computation can become loop
invariants. In a scan in row-major order, the inner loop wouldn't have to do
any multiplication: it would only do an addition. We would use normal
fixnum arithmetic, counting on * to cleverly handle multiplication by a
constant, and appropriate inline expansion.
Note that in a source transform, we can't make any assumptions the type of the
array. If it turns out to be a complex array without declared dimensions, then
the calls to ARRAY-DIMENSION will have to turn into a VOP that can be affected.
But if it is simple, then the VOP is unaffected, and if we know the bounds, it
is constant. Similarly, we would have %ARRAY-DATA and %ARRAY-DISPLACEMENT
operations. %ARRAY-DISPLACEMENT would optimize to 0 if we discover the array
is simple. [This is somewhat inefficient when the array isn't eventually
discovered to be simple, since finding the data and finding the displacement
duplicate each other. We could make %ARRAY-DATA return both as MVs, and then
optimize to (VALUES (%SIMPLE-ARRAY-DATA x) 0), but this would require
optimization of trivial VALUES uses.]
Also need (THE (ARRAY * * * ...) x) to assert correct rank.
|\#
A bunch of functions have source transforms that convert them into the
canonical form that later parts of the compiler want to see. It is not legal
to rely on the canonical form since source transforms can be inhibited by a
Notinline declaration. This shouldn't be a problem, since everyone should keep
their hands off of Notinline calls.
Some transformations:
\begin{verbatim}
Endp ==> (NULL (THE LIST ...))
(NOT xxx) or (NULL xxx) => (IF xxx NIL T)
(typep x '<simple type>) => (<simple predicate> x)
(typep x '<complex type>) => ...composition of simpler operations...
\end{verbatim}
TYPEP of AND, OR and NOT types turned into conditionals over multiple TYPEP
calls. This makes hairy TYPEP calls more digestible to type constraint
propagation, and also means that the TYPEP code generators don't have to deal
with these cases. [\#\#\# In the case of union types we may want to do something
to preserve information for type constraint propagation.]
\begin{verbatim}
(apply \#'foo a b c)
==>
(multiple-value-call \#'foo (values a) (values b) (values-list c))
\end{verbatim}
This way only MV-CALL needs to know how to do calls with unknown numbers of
arguments. It should be nearly as efficient as a special-case VMR-Convert
method could be.
\begin{verbatim}
Make-String => Make-Array
N-arg predicates associated into two-arg versions.
Associate N-arg arithmetic ops.
Expand CxxxR and FIRST...nTH
Zerop, Plusp, Minusp, 1+, 1-, Min, Max, Rem, Mod
(Values x), (Identity x) => (Prog1 x)
All specialized aref functions => (aref (the xxx) ...)
\end{verbatim}
Convert (ldb (byte ...) ...) into internal frob that takes size and position as
separate args. Other byte functions also...
Change for-value primitive predicates into \verb+(if <pred> t nil)+. This isn't
particularly useful during ICR phases, but makes life easy for VMR conversion.
This last can't be a source transformation, since a source transform can't tell
where the form appears. Instead, ICR conversion special-cases calls to known
functions with the Predicate attribute by doing the conversion when the
destination of the result isn't an IF. It isn't critical that this never be
done for predicates that we ultimately discover to deliver their value to an
IF, since IF optimizations will flush unnecessary IFs in a predicate.
\section{Inline functions}
[\#\#\# Inline expansion is especially powerful in the presence of good lisp-level
optimization (``partial evaluation''). Many ``optimizations'' usually done in Lisp
compilers by special-case source-to-source transforms can be had simply by
making the source of the general case function available for inline expansion.
This is especially helpful in Common Lisp, which has many commonly used
functions with simple special cases but bad general cases (list and sequence
functions, for example.)
Inline expansion of recursive functions is allowed, and is not as silly as it
sounds. When expanded in a specific context, much of the overhead of the
recursive calls may be eliminated (especially if there are many keyword
arguments, etc.)
[Also have MAYBE-INLINE]
]
We only record a function's inline expansion in the global environment when the
function is in the null lexical environment, since the expansion must be
represented as source.
We do inline expansion of functions locally defined by FLET or LABELS even when
the environment is not null. Since the appearances of the local function must
be nested within the desired environment, it is possible to expand local
functions inline even when they use the environment. We just stash the source
form and environments in the Functional for the local function. When we
convert a call to it, we just reconvert the source in the saved environment.
An interesting alternative to the inline/full-call dichotomy is ``semi-inline''
coding. Whenever we have an inline expansion for a function, we can expand it
only once per block compilation, and then use local call to call this copied
version. This should get most of the speed advantage of real inline coding
with much less code bloat. This is especially attractive for simple system
functions such as Read-Char.
The main place where true inline expansion would still be worth doing is where
large amounts of the function could be optimized away by constant folding or
other optimizations that depend on the exact arguments to the call.
\section{Compilation policy}
We want more sophisticated control of compilation safety than is offered in CL,
so that we can emit only those type checks that are likely to discover
something (i.e. external interfaces.)
\section{Notes}
Generalized back-end notion provides dynamic retargeting? (for byte code)
The current node type annotations seem to be somewhat unsatisfactory, since we
lose information when we do a THE on a continuation that already has uses, or
when we convert a let where the actual result continuation has other uses.
But the case with THE isn't really all that bad, since the test of whether
there are any uses happens before conversion of the argument, thus THE loses
information only when there are uses outside of the declared form. The LET
case may not be a big deal either.
Note also that losing user assertions isn't really all that bad, since it won't
damage system integrity. At worst, it will cause a bug to go undetected. More
likely, it will just cause the error to be signaled in a different place (and
possibly in a less informative way). Of course, there is an efficiency hit for
losing type information, but if it only happens in strange cases, then this
isn't a big deal.
\chapter{Local call analysis}
All calls to local functions (known named functions and LETs) are resolved to
the exact LAMBDA node which is to be called. If the call is syntactically
illegal, then we emit a warning and mark the reference as :notinline, forcing
the call to be a full call. We don't even think about converting APPLY calls;
APPLY is not special-cased at all in ICR. We also take care not to convert
calls in the top-level component, which would join it to normal code. Calls to
functions with rest args and calls with non-constant keywords are also not
converted.
We also convert MV-Calls that look like MULTIPLE-VALUE-BIND to local calls,
since we know that they can be open-coded. We replace the optional dispatch
with a call to the last optional entry point, letting MV-Call magically default
the unsupplied values to NIL.
When ICR optimizations discover a possible new local call, they explicitly
invoke local call analysis on the code that needs to be reanalyzed.
[\#\#\# Let conversion. What it means to be a let. Argument type checking done
by caller. Significance of local call is that all callers are known, so
special call conventions may be used.]
A lambda called in only one place is called a ``let'' call, since a Let would
turn into one.
In addition to enabling various ICR optimizations, the let/non-let distinction
has important environment significance. We treat the code in function and all
of the lets called by that function as being in the same environment. This
allows exits from lets to be treated as local exits, and makes life easy for
environment analysis.
Since we will let-convert any function with only one call, we must be careful
about cleanups. It is possible that a lexical exit from the let function may
have to clean up dynamic bindings not lexically apparent at the exit point. We
handle this by annotating lets with any cleanup in effect at the call site.
The cleanup for continuations with no immediately enclosing cleanup is the
lambda that the continuation is in. In this case, we look at the lambda to see
if any cleanups need to be done.
Let conversion is disabled for entry-point functions, since otherwise we might
convert the call from the XEP to the entry point into a let. Then later on, we
might want to convert a non-local reference into a local call, and not be able
to, since once a function has been converted to a let, we can't convert it
back.
A function's return node may also be deleted if it is unreachable, which can
happen if the function never returns normally. Such functions are not lets.
\chapter{Find components}
This is a post-pass to ICR conversion that massages the flow graph into the
shape subsequent phases expect. Things done:
Compute the depth-first ordering for the flow graph.
Find the components (disconnected parts) of the flow graph.
This pass need only be redone when newly converted code has been added to the
flow graph. The reanalyze flag in the component structure should be set by
people who mess things up.
We create the initial DFO using a variant of the basic algorithm. The initial
DFO computation breaks the ICR up into components, which are parts that can be
compiled independently. This is done to increase the efficiency of large block
compilations. In addition to improving locality of reference and reducing the
size of flow analysis problems, this allows back-end data structures to be
reclaimed after the compilation of each component.
ICR optimization can change the connectivity of the flow graph by discovering
new calls or eliminating dead code. Initial DFO determination splits up the
flow graph into separate components, but does so conservatively, ensuring that
parts that might become joined (due to local call conversion) are joined from
the start. Initial DFO computation also guarantees that all code which shares
a lexical environment is in the same component so that environment analysis
needs to operate only on a single component at a time.
[This can get a bit hairy, since code seemingly reachable from the
environment entry may be reachable from a NLX into that environment. Also,
function references must be considered as links joining components even though
the flow graph doesn't represent these.]
After initial DFO determination, components are neither split nor joined. The
standard DFO computation doesn't attempt to split components that have been
disconnected.
\chapter{ICR optimize}
{\bf Somewhere describe basic ICR utilities: continuation-type,
constant-continuation-p, etc. Perhaps group by type in ICR description?}
We are conservative about doing variable-for-variable substitution in ICR
optimization, since if we substitute a variable with a less restrictive type,
then we may prevent use of a ``good'' representation within the scope of the
inner binding.
Note that variable-variable substitutions aren't really crucial in ICR, since
they don't create opportunities for new optimizations (unlike substitution of
constants and functions). A spurious variable-variable binding will show up as
a Move operation in VMR. This can be optimized away by reaching-definitions
and also by targeting. [\#\#\# But actually, some optimizers do see if operands
are the same variable.]
\#|
The IF-IF optimization can be modeled as a value driven optimization, since
adding a use definitely is cause for marking the continuation for
reoptimization. [When do we add uses? Let conversion is the only obvious
time.] I guess IF-IF conversion could also be triggered by a non-immediate use
of the test continuation becoming immediate, but to allow this to happen would
require Delete-Block (or somebody) to mark block-starts as needing to be
reoptimized when a predecessor changes. It's not clear how important it is
that IF-IF conversion happen under all possible circumstances, as long as it
happens to the obvious cases.
[\#\#\# It isn't totally true that code flushing never enables other worthwhile
optimizations. Deleting a functional reference can cause a function to cease
being an XEP, or even trigger let conversion. It seems we still want to flush
code during ICR optimize, but maybe we want to interleave it more intimately
with the optimization pass.
Ref-flushing works just as well forward as backward, so it could be done in the
forward pass. Call flushing doesn't work so well, but we could scan the block
backward looking for any new flushable stuff if we flushed a call on the
forward pass.
When we delete a variable due to lack of references, we leave the variable
in the lambda-list so that positional references still work. The initial value
continuation is flushed, though (replaced with NIL) allowing the initial value
for to be deleted (modulo side-effects.)
Note that we can delete vars with no refs even when they have sets. I guess
when there are no refs, we should also flush all sets, allowing the value
expressions to be flushed as well.
Squeeze out single-reference unset let variables by changing the dest of the
initial value continuation to be the node that receives the ref. This can be
done regardless of what the initial value form is, since we aren't actually
moving the evaluation. Instead, we are in effect using the continuation's
locations in place of the temporary variable.
Doing this is of course, a wild violation of stack discipline, since the ref
might be inside a loop, etc. But with the VMR back-end, we only need to
preserve stack discipline for unknown-value continuations; this ICR
transformation must be already inhibited when the DEST of the REF is a
multiple-values receiver (EXIT, RETURN or MV-COMBINATION), since we must
preserve the single-value semantics of the let-binding in this case.
The REF and variable must be deleted as part of this operation, since the ICR
would otherwise be left in an inconsistent state; we can't wait for the REF to
be deleted due to being unused, since we have grabbed the arg continuation and
substituted it into the old DEST.
The big reason for doing this transformation is that in macros such as INCF and
PSETQ, temporaries are squeezed out, and the new value expression is evaluated
directly to the setter, allowing any result type assertion to be applied to the
expression evaluation. Unlike in the case of substitution, there is no point
in inhibiting this transformation when the initial value type is weaker than
the variable type. Instead, we intersect the asserted type for the old REF's
CONT with the type assertion on the initial value continuation. Note that the
variable's type has already been asserted on the initial-value continuation.
Of course, this transformation also simplifies the ICR even when it doesn't
discover interesting type assertions, so it makes sense to do it whenever
possible. This reduces the demands placed on register allocation, etc.
There are three dead-code flushing rules:
\begin{enumerate}
\item Refs with no DEST may be flushed.
\item Known calls with no dest that are flushable may be flushed. We null the
DEST in all the args.
\item If a lambda-var has no refs, then it may be deleted. The flushed
argument continuations have their DEST nulled.
\end{enumerate}
These optimizations all enable one another. We scan blocks backward, looking
for nodes whose CONT has no DEST, then type-dispatching off of the node. If we
delete a ref, then we check to see if it is a lambda-var with no refs. When we
flush an argument, we mark the blocks for all uses of the CONT as needing to be
reoptimized.
\section{Goals for ICR optimizations}
\#|
When an optimization is disabled, code should still be correct and not
ridiculously inefficient. Phases shouldn't be made mandatory when they have
lots of non-required stuff jammed into them.
|\#
This pass is optional, but is desirable if anything is more important than
compilation speed.
This phase is a grab-bag of optimizations that concern themselves with the flow
of values through the code representation. The main things done are type
inference, constant folding and dead expression elimination. This phase can be
understood as a walk of the expression tree that propagates assertions down the
tree and propagates derived information up the tree. The main complication is
that there isn't any expression tree, since ICR is flow-graph based.
We repeat this pass until we don't discover anything new. This is a bit of
feat, since we dispatch to arbitrary functions which may do arbitrary things,
making it hard to tell if anything really happened. Even if we solve this
problem by requiring people to flag when they changed or by checking to see if
they changed something, there are serious efficiency problems due to massive
redundant computation, since in many cases the only way to tell if anything
changed is to recompute the value and see if it is different from the old one.
We solve this problem by requiring that optimizations for a node only depend on
the properties of the CONT and the continuations that have the node as their
DEST. If the continuations haven't changed since the last pass, then we don't
attempt to re-optimize the node, since we know nothing interesting will happen.
We keep track of which continuations have changed by a REOPTIMIZE flag that is
set whenever something about the continuation's value changes.
When doing the bottom up pass, we dispatch to type specific code that knows how
to tell when a node needs to be reoptimized and does the optimization. These
node types are special-cased: COMBINATION, IF, RETURN, EXIT, SET.
The REOPTIMIZE flag in the COMBINATION-FUN is used to detect when the function
information might have changed, so that we know when there are new assertions
that could be propagated from the function type to the arguments.
When we discover something about a leaf, or substitute for leaf, we reoptimize
the CONT for all the REF and SET nodes.
We have flags in each block that indicate when any nodes or continuations in
the block need to be re-optimized, so we don't have to scan blocks where there
is no chance of anything happening.
It is important for efficiency purposes that optimizers never say that they did
something when they didn't, but this by itself doesn't guarantee timely
termination. I believe that with the type system implemented, type inference
will converge in finite time, but as a practical matter, it can take far too
long to discover not much. For this reason, ICR optimization is terminated
after three consecutive passes that don't add or delete code. This premature
termination only happens 2\% of the time.
\section{Flow graph simplification}
Things done:
\begin{itemize}
\item Delete blocks with no predecessors.
\item Merge blocks that can be merged.
\item Convert local calls to Let calls.
\item Eliminate degenerate IFs.
\end{itemize}
We take care not to merge blocks that are in different functions or have
different cleanups. This guarantees that non-local exits are always at block
ends and that cleanup code never needs to be inserted within a block.
We eliminate IFs with identical consequent and alternative. This would most
likely happen if both the consequent and alternative were optimized away.
[Could also be done if the consequent and alternative were different blocks,
but computed the same value. This could be done by a sort of cross-jumping
optimization that looked at the predecessors for a block and merged code shared
between predecessors. IFs with identical branches would eventually be left
with nothing in their branches.]
We eliminate IF-IF constructs:
\begin{verbatim}
(IF (IF A B C) D E) ==>
(IF A (IF B D E) (IF C D E))
\end{verbatim}
In reality, what we do is replicate blocks containing only an IF node where the
predicate continuation is the block start. We make one copy of the IF node for
each use, leaving the consequent and alternative the same. If you look at the
flow graph representation, you will see that this is really the same thing as
the above source to source transformation.
\section{Forward ICR optimizations}
In the forward pass, we scan the code in forward depth-first order. We
examine each call to a known function, and:
\begin{itemize}
\item Eliminate any bindings for unused variables.
\item Do top-down type assertion propagation. In local calls, we propagate
asserted and derived types between the call and the called lambda.
\item
Replace calls of foldable functions with constant arguments with the
result. We don't have to actually delete the call node, since Top-Down
optimize will delete it now that its value is unused.
\item
Run any Optimizer for the current function. The optimizer does arbitrary
transformations by hacking directly on the IR. This is useful primarily
for arithmetic simplification and similar things that may need to examine
and modify calls other than the current call. The optimizer is responsible
for recording any changes that it makes. An optimizer can inhibit further
optimization of the node during the current pass by returning true. This
is useful when deleting the node.
\item
Do ICR transformations, replacing a global function call with equivalent
inline lisp code.
\item
Do bottom-up type propagation/inferencing. For some functions such as
Coerce we will dispatch to a function to find the result type. The
Derive-Type function just returns a type structure, and we check if it is
different from the old type in order to see if there was a change.
\item
Eliminate IFs with predicates known to be true or false.
\item
Substitute the value for unset let variables that are bound to constants,
unset lambda variables or functionals.
\item
Propagate types from local call args to var refs.
\end{itemize}
We use type info from the function continuation to find result types for
functions that don't have a derive-type method.
\subsection{ICR transformation}
ICR transformation does ``source to source'' transformations on known global
functions, taking advantage of semantic information such as argument types and
constant arguments. Transformation is optional, but should be done if speed or
space is more important than compilation speed. Transformations which increase
space should pass when space is more important than speed.
A transform is actually an inline function call where the function is computed
at compile time. The transform gets to peek at the continuations for the
arguments, and computes a function using the information gained. Transforms
should be cautious about directly using the values of constant continuations,
since the compiler must preserve eqlness of named constants, and it will have a
hard time if transforms go around randomly copying constants.
The lambda that the transform computes replaces the original function variable
reference as the function for the call. This lets the compiler worry about
evaluating each argument once in the right order. We want to be careful to
preserve type information when we do a transform, since it may be less than
obvious what the transformed code does.
There can be any number of transforms for a function. Each transform is
associated with a function type that the call must be compatible with. A
transform is only invoked if the call has the right type. This provides a way
to deal with the common case of a transform that only applies when the
arguments are of certain types and some arguments are not specified. We always
use the derived type when determining whether a transform is applicable. Type
check is responsible for setting the derived type to the intersection of the
asserted and derived types.
If the code in the expansion has insufficient explicit or implicit argument
type checking, then it should cause checks to be generated by making
declarations.
A transformation may decide to pass if it doesn't like what it sees when it
looks at the args. The Give-Up function unwinds out of the transform and deals
with complaining about inefficiency if speed is more important than brevity.
The format args for the message are arguments to Give-Up. If a transform can't
be done, we just record the message where ICR finalize can find it. note. We
can't complain immediately, since it might get transformed later on.
\section{Backward ICR optimizations}
In the backward pass, we scan each block in reverse order, and
eliminate any effectless nodes with unused values. In ICR this is the
only way that code is deleted other than the elimination of unreachable blocks.
\chapter{Type checking}
% Somehow split this section up into three parts:
% -- Conceptual: how we know a check is necessary, and who is responsible for
% doing checks.
% -- Incremental: intersection of derived and asserted types, checking for
% non-subtype relationship.
% -- Check generation phase.
We need to do a pretty good job of guessing when a type check will ultimately
need to be done. Generic arithmetic, for example: In the absence of
declarations, we will use the safe variant, but if we don't know this, we
will generate a check for NUMBER anyway. We need to look at the fast-safe
templates and guess if any of them could apply.
We compute a function type from the VOP arguments
and assertions on those arguments. This can be used with Valid-Function-Use
to see which templates do or might apply to a particular call. If we guess
that a safe implementation will be used, then we mark the continuation so as to
force a safe implementation to be chosen. [This will happen if ICR optimize
doesn't run to completion, so the ICR optimization after type check generation
can discover new type information. Since we won't redo type check at that
point, there could be a call that has applicable unsafe templates, but isn't
type checkable.]
[\#\#\# A better and more general optimization of structure type checks: in type
check conversion, we look at the *original derived* type of the continuation:
if the difference between the proven type and the asserted type is a simple
type check, then check for the negation of the difference. e.g. if we want a
FOO and we know we've got (OR FOO NULL), then test for (NOT NULL). This is a
very important optimization for linked lists of structures, but can also apply
in other situations.]
If after ICR phases, we have a continuation with check-type set in a context
where it seems likely a check will be emitted, and the type is too
hairy to be easily checked (i.e. no CHECK-xxx VOP), then we do a transformation
on the ICR equivalent to:
\begin{verbatim}
(... (the hair <foo>) ...)
==>
(... (funcall \#'(lambda (\#:val)
(if (typep \#:val 'hair)
\#:val
(%type-check-error \#:val 'hair)))
<foo>)
...)
\end{verbatim}
This way, we guarantee that VMR conversion never has to emit type checks for
hairy types.
[Actually, we need to do a MV-bind and several type checks when there is a MV
continuation. And some values types are just too hairy to check. We really
can't check any assertion for a non-fixed number of values, since there isn't
any efficient way to bind arbitrary numbers of values. (could be done with
MV-call of a more-arg function, I guess...)
]
[Perhaps only use CHECK-xxx VOPs for types equivalent to a ptype? Exceptions
for CONS and SYMBOL? Anyway, no point in going to trouble to implement and
emit rarely used CHECK-xxx vops.]
One potential lose in converting a type check to explicit conditionals rather
than to a CHECK-xxx VOP is that VMR code motion optimizations won't be able to
do anything. This shouldn't be much of an issue, though, since type constraint
propagation has already done global optimization of type checks.
This phase is optional, but should be done if anything is more important than
compile speed.
Type check is responsible for reconciling the continuation asserted and derived
types, emitting type checks if appropriate. If the derived type is a subtype
of the asserted type, then we don't need to do anything.
If there is no intersection between the asserted and derived types, then there
is a manifest type error. We print a warning message, indicating that
something is almost surely wrong. This will inhibit any transforms or
generators that care about their argument types, yet also inhibits further
error messages, since NIL is a subtype of every type.
If the intersection is not null, then we set the derived type to the
intersection of the asserted and derived types and set the Type-Check flag in
the continuation. We always set the flag when we can't prove that the type
assertion is satisfied, regardless of whether we will ultimately actually emit
a type check or not. This is so other phases such as type constraint
propagation can use the Type-Check flag to detect an interesting type
assertion, instead of having to duplicate much of the work in this phase.
[\#\#\# 7 extremely random values for CONTINUATION-TYPE-CHECK.]
Type checks are generated on the fly during VMR conversion. When VMR
conversion generates the check, it prints an efficiency note if speed is
important. We don't flame now since type constraint progpagation may decide
that the check is unnecessary. [\#\#\# Not done now, maybe never.]
In local function call, it is the caller that is in effect responsible for
checking argument types. This happens in the same way as any other type check,
since ICR optimize propagates the declared argument types to the type
assertions for the argument continuations in all the calls.
Since the types of arguments to entry points are unknown at compile time, we
want to do runtime checks to ensure that the incoming arguments are of the
correct type. This happens without any special effort on the part of type
check, since the XEP is represented as a local call with unknown type
arguments. These arguments will be marked as needing to be checked.
\chapter{Constraint propagation}
New lambda-var-slot:
constraints: a list of all the constraints on this var for either X or Y.
How to maintain consistency? Does it really matter if there are constraints
with deleted vars lying around? Note that whatever mechanism we use for
getting the constraints in the first place should tend to keep them up to date.
Probably we would define optimizers for the interesting relations that look at
their CONT's dest and annotate it if it is an IF.
But maybe it is more trouble then it is worth trying to build up the set of
constraints during ICR optimize (maintaining consistency in the process).
Since ICR optimize iterates a bunch of times before it converges, we would be
wasting time recomputing the constraints, when nobody uses them till constraint
propagation runs.
It seems that the only possible win is if we re-ran constraint propagation
(which we might want to do.) In that case, we wouldn't have to recompute all
the constraints from scratch. But it seems that we could do this just as well
by having ICR optimize invalidate the affected parts of the constraint
annotation, rather than trying to keep them up to date. This also fits better
with the optional nature of constraint propagation, since we don't want ICR
optimize to commit to doing a lot of the work of constraint propagation.
For example, we might have a per-block flag indicating that something happened
in that block since the last time constraint propagation ran. We might have
different flags to represent the distinction between discovering a new type
assertion inside the block and discovering something new about an if
predicate, since the latter would be cheaper to update and probably is more
common.
It's fairly easy to see how we can build these sets of restrictions and
propagate them using flow analysis, but actually using this information seems
a bit more ad-hoc.
Probably the biggest thing we do is look at all the refs. If we have proven that
the value is EQ (EQL for a number) to some other leaf (constant or lambda-var),
then we can substitute for that reference. In some cases, we will want to do
special stuff depending on the DEST. If the dest is an IF and we proved (not
null), then we can substitute T. And if the dest is some relation on the same
two lambda-vars, then we want to see if we can show that relation is definitely
true or false.
Otherwise, we can do our best to invert the set of restrictions into a type.
Since types hold only constant info, we have to ignore any constraints between
two vars. We can make some use of negated type restrictions by using
TYPE-DIFFERENCE to remove the type from the ref types. If our inferred type is
as good as the type assertion, then the continuation's type-check flag will be
cleared.
It really isn't much of a problem that we don't infer union types on joins,
since union types are relatively easy to derive without using flow information.
The normal bottom-up type inference done by ICR optimize does this for us: it
annotates everything with the union of all of the things it might possibly be.
Then constraint propagation subtracts out those types that can't be in effect
because of predicates or checks.
This phase is optional, but is desirable if anything is more important than
compilation speed. We use an algorithm similar to available expressions to
propagate variable type information that has been discovered by implicit or
explicit type tests, or by type inference.
We must do a pre-pass which locates set closure variables, since we cannot do
flow analysis on such variables. We set a flag in each set closure variable so
that we can quickly tell that it is losing when we see it again. Although this
may seem to be wastefully redundant with environment analysis, the overlap
isn't really that great, and the cost should be small compared to that of the
flow analysis that we are preparing to do. [Or we could punt on set
variables...]
A type constraint is a structure that includes sset-element and has
the type and variable. [Also a not-p flag indicating whether the sense
is negated.]
Each variable has a list of its type constraints. We create a type
constraint when we see a type test or check. If there is already a
constraint for the same variable and type, then we just re-use it. If
there is already a weaker constraint, then we generate both the weak
constraints and the strong constraint so that the weak constraints
won't be lost even if the strong one is unavailable.
We find all the distinct type constraints for each variable during the pre-pass
over the lambda nesting. Each constraint has a list of the weaker constraints
so that we can easily generate them.
Every block generates all the type constraints in it, but a constraint is
available in a successor only if it is available in all predecessors. We
determine the actual type constraint for a variable at a block by intersecting
all the available type constraints for that variable.
This isn't maximally tense when there are constraints that are not
hierarchically related, e.g. (or a b) (or b c). If these constraints were
available from two predecessors, then we could infer that we have an (or a b c)
constraint, but the above algorithm would come up with none. This probably
isn't a big problem.
[\#\#\# Do we want to deal with \verb+(if (eq <var> '<foo>) ...)+ indicating singleton
member type?]
We detect explicit type tests by looking at type test annotation in the IF
node. If there is a type check, the OUT sets are stored in the node, with
different sets for the consequent and alternative. Implicit type checks are
located by finding Ref nodes whose Cont has the Type-Check flag set. We don't
actually represent the GEN sets, we just initialize OUT to it, and then form
the union in place.
When we do the post-pass, we clear the Type-Check flags in the continuations
for Refs when we discover that the available constraints satisfy the asserted
type. Any explicit uses of typep should be cleaned up by the ICR optimizer for
typep. We can also set the derived type for Refs to the intersection of the
available type assertions. If we discover anything, we should consider redoing
ICR optimization, since better type information might enable more
optimizations.
\chapter{ICR finalize} % -*- Dictionary: design -*-
This pass looks for interesting things in the ICR so that we can forget about
them. Used and not defined things are flamed about.
We postpone these checks until now because the ICR optimizations may discover
errors that are not initially obvious. We also emit efficiency notes about
optimizations that we were unable to do. We can't emit the notes immediately,
since we don't know for sure whether a repeated attempt at optimization will
succeed.
We examine all references to unknown global function variables and update the
approximate type accordingly. We also record the names of the unknown
functions so that they can be flamed about if they are never defined. Unknown
normal variables are flamed about on the fly during ICR conversion, so we
ignore them here.
We check each newly defined global function for compatibility with previously
recorded type information. If there is no :defined or :declared type, then we
check for compatibility with any approximate function type inferred from
previous uses.
\chapter{Environment analysis}
A related change would be to annotate ICR with information about tail-recursion
relations. What we would do is add a slot to the node structure that points to
the corresponding Tail-Info when a node is in a TR position. This annotation
would be made in a final ICR pass that runs after cleanup code is generated
(part of environment analysis). When true, the node is in a true TR position
(modulo return-convention incompatibility). When we determine return
conventions, we null out the tail-p slots in XEP calls or known calls where we
decided not to preserve tail-recursion.
In this phase, we also check for changes in the dynamic binding environment
that require cleanup code to be generated. We just check for changes in the
Continuation-Cleanup on local control transfers. If it changes from
an inner dynamic context to an outer one that is in the same environment, then
we emit code to clean up the dynamic bindings between the old and new
continuation. We represent the result of cleanup detection to the back end by
interposing a new block containing a call to a funny function. Local exits
from CATCH or UNWIND-PROTECT are detected in the same way.
|\#
The primary activity in environment analysis is the annotation of ICR with
environment structures describing where variables are allocated and what values
the environment closes over.
Each lambda points to the environment where its variables are allocated, and
the environments point back. We always allocate the environment at the Bind
node for the sole non-let lambda in the environment, so there is a close
relationship between environments and functions. Each ``real function'' (i.e.
not a LET) has a corresponding environment.
We attempt to share the same environment among as many lambdas as possible so
that unnecessary environment manipulation is not done. During environment
analysis the only optimization of this sort is realizing that a Let (a lambda
with no Return node) cannot need its own environment, since there is no way
that it can return and discover that its old values have been clobbered.
When the function is called, values from other environments may need to be made
available in the function's environment. These values are said to be ``closed
over''.
Even if a value is not referenced in a given environment, it may need to be
closed over in that environment so that it can be passed to a called function
that does reference the value. When we discover that a value must be closed
over by a function, we must close over the value in all the environments where
that function is referenced. This applies to all references, not just local
calls, since at other references we must have the values on hand so that we can
build a closure. This propagation must be applied recursively, since the value
must also be available in *those* functions' callers.
If a closure reference is known to be ``safe'' (not an upward funarg), then the
closure structure may be allocated on the stack.
Closure analysis deals only with closures over values, while Common Lisp
requires closures over variables. The difference only becomes significant when
variables are set. If a variable is not set, then we can freely make copies of
it without keeping track of where they are. When a variable is set, we must
maintain a single value cell, or at least the illusion thereof. We achieve
this by creating a heap-allocated ``value cell'' structure for each set variable
that is closed over. The pointer to this value cell is passed around as the
``value'' corresponding to that variable. References to the variable must
explicitly indirect through the value cell.
When we are scanning over the lambdas in the component, we also check for bound
but not referenced variables.
Environment analysis emits cleanup code for local exits and markers for
non-local exits.
A non-local exit is a control transfer from one environment to another. In a
non-local exit, we must close over the continuation that we transfer to so that
the exiting function can find its way back. We indicate the need to close a
continuation by placing the continuation structure in the closure and also
pushing it on a list in the environment structure for the target of the exit.
[\#\#\# To be safe, we would treat the continuation as a set closure variable so
that we could invalidate it when we leave the dynamic extent of the exit point.
Transferring control to a meaningless stack pointer would be apt to cause
horrible death.]
Each local control transfer may require dynamic state such as special bindings
to be undone. We represent cleanup actions by funny function calls in a new
block linked in as an implicit MV-PROG1.

411
doc/internals/cmu/glossary.tex Normal file
View file

@ -0,0 +1,411 @@
\chapter{Glossary}% -*- Dictionary: int:design -*-
% Note: in an entry, any word that is also defined should be \it
% should entries have page references as well?
\begin{description}
\item[assert (a type)]
In Python, all type checking is done via a general type assertion
mechanism. Explicit declarations and implicit assertions (e.g. the arg to
+ is a number) are recorded in the front-end (implicit continuation)
representation. Type assertions (and thus type-checking) are ``unbundled''
from the operations that are affected by the assertion. This has two major
advantages:
\begin{itemize}
\item Code that implements operations need not concern itself with checking
operand types.
\item Run-time type checks can be eliminated when the compiler can prove that
the assertion will always be satisfied.
\end{itemize}
See also {\it restrict}.
\item[back end] The back end is the part of the compiler that operates on the
{\it virtual machine} intermediate representation. Also included are the
compiler phases involved in the conversion from the {\it front end}
representation (or {\it ICR}).
\item[bind node] This is a node type the that marks the start of a {\it lambda}
body in {\it ICR}. This serves as a placeholder for environment manipulation
code.
\item[IR1] The first intermediate representation, also known as {\it ICR}, or
the Implicit Continuation Represenation.
\item[IR2] The second intermediate representation, also known as {\it VMR}, or
the Virtual Machine Representation.
\item[basic block] A basic block (or simply ``block'') has the pretty much the
usual meaning of representing a straight-line sequence of code. However, the
code sequence ultimately generated for a block might contain internal branches
that were hidden inside the implementation of a particular operation. The type
of a block is actually {\tt cblock}. The {\tt block-info} slot holds an
{\tt VMR-block} containing backend information.
\item[block compilation] Block compilation is a term commonly used to describe
the compile-time resolution of function names. This enables many
optimizations.
\item[call graph]
Each node in the call graph is a function (represented by a {\it flow graph}.)
The arcs in the call graph represent a possible call from one function to
another. See also {\it tail set}.
\item[cleanup]
A cleanup is the part of the implicit continuation representation that
retains information scoping relationships. For indefinite extent bindings
(variables and functions), we can abandon scoping information after ICR
conversion, recovering the lifetime information using flow analysis. But
dynamic bindings (special values, catch, unwind protect, etc.) must be
removed at a precise time (whenever the scope is exited.) Cleanup
structures form a hierarchy that represents the static nesting of dynamic
binding structures. When the compiler does a control transfer, it can use
the cleanup information to determine what cleanup code needs to be emitted.
\item[closure variable]
A closure variable is any lexical variable that has references outside of
its {\it home environment}. See also {\it indirect value cell}.
\item[closed continuation] A closed continuation represents a {\tt tagbody} tag
or {\tt block} name that is closed over. These two cases are mostly
indistinguishable in {\it ICR}.
\item[home] Home is a term used to describe various back-pointers. A lambda
variable's ``home'' is the lambda that the variable belongs to. A lambda's ``home
environment'' is the environment in which that lambda's variables are allocated.
\item[indirect value cell]
Any closure variable that has assignments ({\tt setq}s) will be allocated in an
indirect value cell. This is necessary to ensure that all references to
the variable will see assigned values, since the compiler normally freely
copies values when creating a closure.
\item[set variable] Any variable that is assigned to is called a ``set
variable''. Several optimizations must special-case set variables, and set
closure variables must have an {\it indirect value cell}.
\item[code generator] The code generator for a {\it VOP} is a potentially
arbitrary list code fragment which is responsible for emitting assembly code to
implement that VOP.
\item[constant pool] The part of a compiled code object that holds pointers to
non-immediate constants.
\item[constant TN]
A constant TN is the {\it VMR} of a compile-time constant value. A
constant may be immediate, or may be allocated in the {\it constant pool}.
\item[constant leaf]
A constant {\it leaf} is the {\it ICR} of a compile-time constant value.
\item[combination]
A combination {\it node} is the {\it ICR} of any fixed-argument function
call (not {\tt apply} or {\tt multiple-value-call}.)
\item[top-level component]
A top-level component is any component whose only entry points are top-level
lambdas.
\item[top-level lambda]
A top-level lambda represents the execution of the outermost form on which
the compiler was invoked. In the case of {\tt compile-file}, this is often a
truly top-level form in the source file, but the compiler can recursively
descend into some forms ({\tt eval-when}, etc.) breaking them into separate
compilations.
\item[component] A component is basically a sequence of blocks. Each component
is compiled into a separate code object. With {\it block compilation} or {\it
local functions}, a component will contain the code for more than one function.
This is called a component because it represents a connected portion of the
call graph. Normally the blocks are in depth-first order ({\it DFO}).
\item[component, initial] During ICR conversion, blocks are temporarily
assigned to initial components. The ``flow graph canonicalization'' phase
determines the true component structure.
\item[component, head and tail]
The head and tail of a component are dummy blocks that mark the start and
end of the {\it DFO} sequence. The component head and tail double as the root
and finish node of the component's flow graph.
\item[local function (call)]
A local function call is a call to a function known at compile time to be
in the same {\it component}. Local call allows compile time resolution of the
target address and calling conventions. See {\it block compilation}.
\item[conflict (of TNs, set)]
Register allocation terminology. Two TNs conflict if they could ever be
live simultaneously. The conflict set of a TN is all TNs that it conflicts
with.
\item[continuation]
The ICR data structure which represents both:
\begin{itemize}
\item The receiving of a value (or multiple values), and
\item A control location in the flow graph.
\end{itemize}
In the Implicit Continuation Representation, the environment is implicit in the
continuation's BLOCK (hence the name.) The ICR continuation is very similar to
a CPS continuation in its use, but its representation doesn't much resemble (is
not interchangeable with) a lambda.
\item[cont] A slot in the {\it node} holding the {\it continuation} which
receives the node's value(s). Unless the node ends a {\it block}, this also
implicitly indicates which node should be evaluated next.
\item[cost] Approximations of the run-time costs of operations are widely used
in the back end. By convention, the unit is generally machine cycles, but the
values are only used for comparison between alternatives. For example, the
VOP cost is used to determine the preferred order in which to try possible
implementations.
\item[CSP, CFP] See {\it control stack pointer} and {\it control frame
pointer}.
\item[Control stack] The main call stack, which holds function stack frames.
All words on the control stack are tagged {\it descriptors}. In all ports done
so far, the control stack grows from low memory to high memory. The most
recent call frames are considered to be ``on top'' of earlier call frames.
\item[Control stack pointer] The allocation pointer for the {\it control
stack}. Generally this points to the first free word at the top of the stack.
\item[Control frame pointer] The pointer to the base of the {\it control stack}
frame for a particular function invocation. The CFP for the running function
must be in a register.
\item[Number stack] The auxiliary stack used to hold any {\it non-descriptor}
(untagged) objects. This is generally the same as the C call stack, and thus
typically grows down.
\item[Number stack pointer] The allocation pointer for the {\it number stack}.
This is typically the C stack pointer, and is thus kept in a register.
\item[NSP, NFP] See {\it number stack pointer}, {\it number frame pointer}.
\item[Number frame pointer] The pointer to the base of the {\it number stack}
frame for a particular function invocation. Functions that don't use the
number stack won't have an NFP, but if an NFP is allocated, it is always
allocated in a particular register. If there is no variable-size data on the
number stack, then the NFP will generally be identical to the NSP.
\item[Lisp return address] The name of the {\it descriptor} encoding the
``return pc'' for a function call.
\item[LRA] See {\it lisp return address}. Also, the name of the register where
the LRA is passed.
\item[Code pointer] A pointer to the header of a code object. The code pointer
for the currently running function is stored in the {\tt code} register.
\item[Interior pointer] A pointer into the inside of some heap-allocated
object. Interior pointers confuse the garbage collector, so their use is
highly constrained. Typically there is a single register dedicated to holding
interior pointers.
\item[dest]
A slot in the {\it continuation} which points the the node that receives this
value. Null if this value is not received by anyone.
\item[DFN, DFO] See {\it Depth First Number}, {\it Depth First Order}.
\item[Depth first number] Blocks are numbered according to their appearance in
the depth-first ordering (the {\tt block-number} slot.) The numbering actually
increases from the component tail, so earlier blocks have larger numbers.
\item[Depth first order] This is a linearization of the flow graph, obtained by
a depth-first walk. Iterative flow analysis algorithms work better when blocks
are processed in DFO (or reverse DFO.)
\item[Object] In low-level design discussions, an object is one of the
following:
\begin{itemize}
\item a single word containing immediate data (characters, fixnums, etc)
\item a single word pointing to an object (structures, conses, etc.)
\end{itemize}
These are tagged with three low-tag bits as described in the section
\ref{sec:tagging} This is synonymous with {\it descriptor}.
In other parts of the documentation, may be used more loosely to refer to a
{\it lisp object}.
\item[Lisp object]
A Lisp object is a high-level object discussed as a data type in the Common
Lisp definition.
\item[Data-block]
A data-block is a dual-word aligned block of memory that either manifests a
Lisp object (vectors, code, symbols, etc.) or helps manage a Lisp object on
the heap (array header, function header, etc.).
\item[Descriptor]
A descriptor is a tagged, single-word object. It either contains immediate
data or a pointer to data. This is synonymous with {\it object}. Storage
locations that must contain descriptors are referred to as descriptor
locations.
\item[Pointer descriptor]
A descriptor that points to a {\it data block} in memory (i.e. not an immediate
object.)
\item[Immediate descriptor]
A descriptor that encodes the object value in the descriptor itself; used for
characters, fixnums, etc.
\item[Word]
A word is a 32-bit quantity.
\item[Non-descriptor]
Any chunk of bits that isn't a valid tagged descriptor. For example, a
double-float on the number stack. Storage locations that are not scanned by
the garbage collector (and thus cannot contain {\it pointer descriptors}) are
called non-descriptor locations. {\it Immediate descriptors} can be stored in
non-descriptor locations.
\item[Entry point] An entry point is a function that may be subject to
``unpredictable'' control transfers. All entry points are linked to the root
of the flow graph (the component head.) The only functions that aren't entry
points are {\it let} functions. When complex lambda-list syntax is used,
multiple entry points may be created for a single lisp-level function.
See {\it external entry point}.
\item[External entry point] A function that serves as a ``trampoline'' to
intercept function calls coming in from outside of the component. The XEP does
argument syntax and type checking, and may also translate the arguments and
return values for a locally specialized calling calling convention.
\item[XEP] An {\it external entry point}.
\item[lexical environment] A lexical environment is a structure that is used
during VMR conversion to represent all lexically scoped bindings (variables,
functions, declarations, etc.) Each {\tt node} is annotated with its lexical
environment, primarily for use by the debugger and other user interfaces. This
structure is also the environment object passed to {\tt macroexpand}.
\item[environment] The environment is part of the ICR, created during
environment analysis. Environment analysis apportions code to disjoint
environments, with all code in the same environment sharing the same stack
frame. Each environment has a ``{\it real}'' function that allocates it, and
some collection {\tt let} functions. Although environment analysis is the
last ICR phase, in earlier phases, code is sometimes said to be ``in the
same/different environment(s)''. This means that the code will definitely be
in the same environment (because it is in the same real function), or that is
might not be in the same environment, because it is not in the same function.
\item[fixup] Some sort of back-patching annotation. The main sort encountered
are load-time {\it assembler fixups}, which are a linkage annotation mechanism.
\item[flow graph] A flow graph is a directed graph of basic blocks, where each
arc represents a possible control transfer. The flow graph is the basic data
structure used to represent code, and provides direct support for data flow
analysis. See component and ICR.
\item[foldable] An attribute of {\it known functions}. A function is foldable
if calls may be constant folded whenever the arguments are compile-time
constant. Generally this means that it is a pure function with no side
effects.
\item[FSC]
\item[full call]
\item[function attribute]
function
``real'' (allocates environment)
meaning function-entry
more vague (any lambda?)
funny function
GEN (kill and...)
global TN, conflicts, preference
GTN (number)
IR ICR VMR ICR conversion, VMR conversion (translation)
inline expansion, call
kill (to make dead)
known function
LAMBDA
leaf
let call
lifetime analysis, live (tn, variable)
load tn
LOCS (passing, return locations)
local call
local TN, conflicts, (or just used in one block)
location (selection)
LTN (number)
main entry
mess-up (for cleanup)
more arg (entry)
MV
non-local exit
non-packed SC, TN
non-set variable
operand (to vop)
optimizer (in icr optimize)
optional-dispatch
pack, packing, packed
pass (in a transform)
passing
locations (value)
conventions (known, unknown)
policy (safe, fast, small, ...)
predecessor block
primitive-type
reaching definition
REF
representation
selection
for value
result continuation (for function)
result type assertion (for template) (or is it restriction)
restrict
a TN to finite SBs
a template operand to a primitive type (boxed...)
a tn-ref to particular SCs
return (node, vops)
safe, safety
saving (of registers, costs)
SB
SC (restriction)
semi-inline
side-effect
in ICR
in VMR
sparse set
splitting (of VMR blocks)
SSET
SUBPRIMITIVE
successor block
tail recursion
tail recursive
tail recursive loop
user tail recursion
template
TN
TNBIND
TN-REF
transform (source, ICR)
type
assertion
inference
top-down, bottom-up
assertion propagation
derived, asserted
descriptor, specifier, intersection, union, member type
check
type-check (in continuation)
UNBOXED (boxed) descriptor
unknown values continuation
unset variable
unwind-block, unwinding
used value (dest)
value passing
VAR
VM
VOP
\item[XEP]
\end{description}

6
doc/internals/cmu/interface.tex Normal file
View file

@ -0,0 +1,6 @@
\chapter{User Interface of the Compiler}
\section{Error Message Utilities}
\section{Source Paths}
\label{source-paths}

694
doc/internals/cmu/internal-design.txt Normal file
View file

@ -0,0 +1,694 @@
;;;; Terminology.
OBJECT
An object is one of the following:
a single word containing immediate data (characters, fixnums, etc)
a single word pointing to an object (structures, conses, etc.)
These are tagged with three low-tag bits as described in the section
"Tagging". This is synonymous with DESCRIPTOR.
LISP OBJECT
A Lisp object is a high-level object discussed as a data type in Common
Lisp: The Language.
DATA-BLOCK
A data-block is a dual-word aligned block of memory that either manifests a
Lisp object (vectors, code, symbols, etc.) or helps manage a Lisp object on
the heap (array header, function header, etc.).
DESCRIPTOR
A descriptor is a tagged, single-word object. It either contains immediate
data or a pointer to data. This is synonymous with OBJECT.
WORD
A word is a 32-bit quantity.
;;;; Tagging.
The following is a key of the three bit low-tagging scheme:
000 even fixnum
001 function pointer
010 other-immediate (header-words, characters, symbol-value trap value, etc.)
011 list pointer
100 odd fixnum
101 structure pointer
110 unused
111 other-pointer to data-blocks (other than conses, structures,
and functions)
This taging scheme forces a dual-word alignment of data-blocks on the heap, but
this can be pretty negligible:
RATIOS and COMPLEX must have a header-word anyway since they are not a
major type. This wastes one word for these infrequent data-blocks since
they require two words for the data.
BIGNUMS must have a header-word and probably contain only one other word
anyway, so we probably don't waste any words here. Most bignums just
barely overflow fixnums, that is by a bit or two.
Single and double FLOATS?
no waste
one word wasted
SYMBOLS are dual-word aligned with the header-word.
Everything else is vector-like including code, so these probably take up
so many words that one extra one doesn't matter.
;;;; GC Comments.
Data-Blocks comprise only descriptors, or they contain immediate data and raw
bits interpreted by the system. GC must skip the latter when scanning the
heap, so it does not look at a word of raw bits and interpret it as a pointer
descriptor. These data-blocks require headers for GC as well as for operations
that need to know how to interpret the raw bits. When GC is scanning, and it
sees a header-word, then it can determine how to skip that data-block if
necessary. Header-Words are tagged as other-immediates. See the sections
"Other-Immediates" and "Data-Blocks and Header-Words" for comments on
distinguishing header-words from other-immediate data. This distinction is
necessary since we scan through data-blocks containing only descriptors just as
we scan through the heap looking for header-words introducing data-blocks.
Data-Blocks containing only descriptors do not require header-words for GC
since the entire data-block can be scanned by GC a word at a time, taking
whatever action is necessary or appropriate for the data in that slot. For
example, a cons is referenced by a descriptor with a specific tag, and the
system always knows the size of this data-block. When GC encounters a pointer
to a cons, it can transport it into the new space, and when scanning, it can
simply scan the two words manifesting the cons interpreting each word as a
descriptor. Actually there is no cons tag, but a list tag, so we make sure the
cons is not nil when appropriate. A header may still be desired if the pointer
to the data-block does not contain enough information to adequately maintain
the data-block. An example of this is a simple-vector containing only
descriptor slots, and we attach a header-word because the descriptor pointing
to the vector lacks necessary information -- the type of the vector's elements,
its length, etc.
There is no need for a major tag for GC forwarding pointers. Since the tag
bits are in the low end of the word, a range check on the start and end of old
space tells you if you need to move the thing. This is all GC overhead.
;;;; Structures.
Structures comprise a word for each slot in the definition in addition to one
word, a type slot which is a pointer descriptor. This points to a structure
describing the data-block as a structure, a defstruct-descriptor object. When
operating on a structure, doing a structure test can be done by simply checking
the tag bits on the pointer descriptor referencing it. As described in section
"GC Comments", data-blocks such as those representing structures may avoid
having a header-word since they are GC-scanable without any problem. This
saves two words for every structure instance.
;;;; Fixnums.
A fixnum has one of the following formats in 32 bits:
-------------------------------------------------------
| 30 bit 2's complement even integer | 0 0 0 |
-------------------------------------------------------
or
-------------------------------------------------------
| 30 bit 2's complement odd integer | 1 0 0 |
-------------------------------------------------------
Effectively, there is one tag for immediate integers, two zeros. This buys one
more bit for fixnums, and now when these numbers index into simple-vectors or
offset into memory, they point to word boundaries on 32-bit, byte-addressable
machines. That is, no shifting need occur to use the number directly as an
offset.
This format has another advantage on byte-addressable machines when fixnums are
offsets into vector-like data-blocks, including structures. Even though we
previously mentioned data-blocks are dual-word aligned, most indexing and slot
accessing is word aligned, and so are fixnums with effectively two tag bits.
Two tags also allow better usage of special instructions on some machines that
can deal with two low-tag bits but not three.
Since the two bits are zeros, we avoid having to mask them off before using the
words for arithmetic, but division and multiplication require special shifting.
;;;; Other-immediates.
An other-immediate has the following format:
----------------------------------------------------------------
| Data (24 bits) | Type (8 bits with low-tag) | 0 1 0 |
----------------------------------------------------------------
The system uses eight bits of type when checking types and defining system
constants. This allows allows for 32 distinct other-immediate objects given
the three low-tag bits tied down.
The system uses this format for characters, SYMBOL-VALUE unbound trap value,
and header-words for data-blocks on the heap. The type codes are laid out to
facilitate range checks for common subtypes; for example, all numbers will have
contiguous type codes which are distinct from the contiguous array type codes.
See section "Data-Blocks and Other-immediates Typing" for details.
;;;; Data-Blocks and Header-Word Format.
Pointers to data-blocks have the following format:
----------------------------------------------------------------
| Dual-word address of data-block (29 bits) | 1 1 1 |
----------------------------------------------------------------
The word pointed to by the above descriptor is a header-word, and it has the
same format as an other-immediate:
----------------------------------------------------------------
| Data (24 bits) | Type (8 bits with low-tag) | 0 1 0 |
----------------------------------------------------------------
This is convenient for scanning the heap when GC'ing, but it does mean that
whenever GC encounters an other-immediate word, it has to do a range check on
the low byte to see if it is a header-word or just a character (for example).
This is easily acceptable performance hit for scanning.
The system interprets the data portion of the header-word for non-vector
data-blocks as the word length excluding the header-word. For example, the
data field of the header for ratio and complex numbers is two, one word each
for the numerator and denominator or for the real and imaginary parts.
For vectors and data-blocks representing Lisp objects stored like vectors, the
system ignores the data portion of the header-word:
----------------------------------------------------------------
| Unused Data (24 bits) | Type (8 bits with low-tag) | 0 1 0 |
----------------------------------------------------------------
| Element Length of Vector (30 bits) | 0 0 |
----------------------------------------------------------------
Using a separate word allows for much larger vectors, and it allows LENGTH to
simply access a single word without masking or shifting. Similarly, the header
for complex arrays and vectors has a second word, following the header-word,
the system uses for the fill pointer, so computing the length of any array is
the same code sequence.
;;;; Data-Blocks and Other-immediates Typing.
These are the other-immediate types. We specify them including all low eight
bits, including the other-immediate tag, so we can think of the type bits as
one type -- not an other-immediate major type and a subtype. Also, fetching a
byte and comparing it against a constant is more efficient than wasting even a
small amount of time shifting out the other-immediate tag to compare against a
five bit constant.
Number (< 30)
00000 010 bignum 10
00000 010 ratio 14
00000 010 single-float 18
00000 010 double-float 22
00000 010 complex 26
Array (>= 30 code 86)
Simple-Array (>= 20 code 70)
00000 010 simple-array 30
Vector (>= 34 code 82)
00000 010 simple-string 34
00000 010 simple-bit-vector 38
00000 010 simple-vector 42
00000 010 (simple-array (unsigned-byte 2) (*)) 46
00000 010 (simple-array (unsigned-byte 4) (*)) 50
00000 010 (simple-array (unsigned-byte 8) (*)) 54
00000 010 (simple-array (unsigned-byte 16) (*)) 58
00000 010 (simple-array (unsigned-byte 32) (*)) 62
00000 010 (simple-array single-float (*)) 66
00000 010 (simple-array double-float (*)) 70
00000 010 complex-string 74
00000 010 complex-bit-vector 78
00000 010 (array * (*)) -- general complex vector. 82
00000 010 complex-array 86
00000 010 code-header-type 90
00000 010 function-header-type 94
00000 010 closure-header-type 98
00000 010 funcallable-instance-header-type 102
00000 010 unused-function-header-1-type 106
00000 010 unused-function-header-2-type 110
00000 010 unused-function-header-3-type 114
00000 010 closure-function-header-type 118
00000 010 return-pc-header-type 122
00000 010 value-cell-header-type 126
00000 010 symbol-header-type 130
00000 010 base-character-type 134
00000 010 system-area-pointer-type (header type) 138
00000 010 unbound-marker 142
00000 010 weak-pointer-type 146
;;;; Strings.
All strings in the system are C-null terminated. This saves copying the bytes
when calling out to C. The only time this wastes memory is when the string
contains a multiple of eight characters, and then the system allocates two more
words (since Lisp objects are dual-word aligned) to hold the C-null byte.
Since the system will make heavy use of C routines for systems calls and
libraries that save reimplementation of higher level operating system
functionality (such as pathname resolution or current directory computation),
saving on copying strings for C should make C call out more efficient.
The length word in a string header, see section "Data-Blocks and Header-Word
Format", counts only the characters truly in the Common Lisp string.
Allocation and GC will have to know to handle the extra C-null byte, and GC
already has to deal with rounding up various objects to dual-word alignment.
;;;; Symbols and NIL.
Symbol data-block has the following format:
-------------------------------------------------------
| 5 (data-block words) | Symbol Type (8 bits) |
-------------------------------------------------------
| Value Descriptor |
-------------------------------------------------------
| Function Pointer |
-------------------------------------------------------
| Raw Function Address |
-------------------------------------------------------
| Setf Function |
-------------------------------------------------------
| Property List |
-------------------------------------------------------
| Print Name |
-------------------------------------------------------
| Package |
-------------------------------------------------------
Most of these slots are self-explanatory given what symbols must do in Common
Lisp, but a couple require comments. We added the Raw Function Address slot to
speed up named call which is the most common calling convention. This is a
non-descriptor slot, but since objects are dual word aligned, the value
inherently has fixnum low-tag bits. The GC method for symbols must know to
update this slot. The Setf Function slot is currently unused, but we had an
extra slot due to adding Raw Function Address since objects must be dual-word
aligned.
The issues with nil are that we want it to act like a symbol, and we need list
operations such as CAR and CDR to be fast on it. CMU Common Lisp solves this
by putting nil as the first object in static space, where other global values
reside, so it has a known address in the system:
------------------------------------------------------- <-- start static
| 0 | space
-------------------------------------------------------
| 5 (data-block words) | Symbol Type (8 bits) |
------------------------------------------------------- <-- nil
| Value/CAR |
-------------------------------------------------------
| Definition/CDR |
-------------------------------------------------------
| Raw Function Address |
-------------------------------------------------------
| Setf Function |
-------------------------------------------------------
| Property List |
-------------------------------------------------------
| Print Name |
-------------------------------------------------------
| Package |
-------------------------------------------------------
| ... |
-------------------------------------------------------
In addition, we make the list typed pointer to nil actually point past the
header word of the nil symbol data-block. This has usefulness explained below.
The value and definition of nil are nil. Therefore, any reference to nil used
as a list has quick list type checking, and CAR and CDR can go right through
the first and second words as if nil were a cons object.
When there is a reference to nil used as a symbol, the system adds offsets to
the address the same as it does for any symbol. This works due to a
combination of nil pointing past the symbol header-word and the chosen list and
other-pointer type tags. The list type tag is four less than the other-pointer
type tag, but nil points four additional bytes into its symbol data-block.
;;;; Array Headers.
The array-header data-block has the following format:
----------------------------------------------------------------
| Header Len (24 bits) = Array Rank +5 | Array Type (8 bits) |
----------------------------------------------------------------
| Fill Pointer (30 bits) | 0 0 |
----------------------------------------------------------------
| Available Elements (30 bits) | 0 0 |
----------------------------------------------------------------
| Data Vector (29 bits) | 1 1 1 |
----------------------------------------------------------------
| Displacement (30 bits) | 0 0 |
----------------------------------------------------------------
| Displacedp (29 bits) -- t or nil | 1 1 1 |
----------------------------------------------------------------
| Range of First Index (30 bits) | 0 0 |
----------------------------------------------------------------
.
.
.
The array type in the header-word is one of the eight-bit patterns from section
"Data-Blocks and Other-immediates Typing", indicating that this is a complex
string, complex vector, complex bit-vector, or a multi-dimensional array. The
data portion of the other-immediate word is the length of the array header
data-block. Due to its format, its length is always five greater than the
array's number of dimensions. The following words have the following
interpretations and types:
Fill Pointer
This is a fixnum indicating the number of elements in the data vector
actually in use. This is the logical length of the array, and it is
typically the same value as the next slot. This is the second word, so
LENGTH of any array, with or without an array header, is just four bytes
off the pointer to it.
Available Elements
This is a fixnum indicating the number of elements for which there is
space in the data vector. This is greater than or equal to the logical
length of the array when it is a vector having a fill pointer.
Data Vector
This is a pointer descriptor referencing the actual data of the array.
This a data-block whose first word is a header-word with an array type as
described in sections "Data-Blocks and Header-Word Format" and
"Data-Blocks and Other-immediates Typing"
Displacement
This is a fixnum added to the computed row-major index for any array.
This is typically zero.
Displacedp
This is either t or nil. This is separate from the displacement slot, so
most array accesses can simply add in the displacement slot. The rare
need to know if an array is displaced costs one extra word in array
headers which probably aren't very frequent anyway.
Range of First Index
This is a fixnum indicating the number of elements in the first dimension
of the array. Legal index values are zero to one less than this number
inclusively. IF the array is zero-dimensional, this slot is
non-existent.
... (remaining slots)
There is an additional slot in the header for each dimension of the
array. These are the same as the Range of First Index slot.
;;;; Bignums.
Bignum data-blocks have the following format:
-------------------------------------------------------
| Length (24 bits) | Bignum Type (8 bits) |
-------------------------------------------------------
| least significant bits |
-------------------------------------------------------
.
.
.
The elements contain the two's complement representation of the integer with
the least significant bits in the first element or closer to the header. The
sign information is in the high end of the last element.
;;;; Code Data-Blocks.
A code data-block is the run-time representation of a "component". A component
is a connected portion of a program's flow graph that is compiled as a single
unit, and it contains code for many functions. Some of these functions are
callable from outside of the component, and these are termed "entry points".
Each entry point has an associated user-visible function data-block (of type
FUNCTION). The full call convention provides for calling an entry point
specified by a function object.
Although all of the function data-blocks for a component's entry points appear
to the user as distinct objects, the system keeps all of the code in a single
code data-block. The user-visible function object is actually a pointer into
the middle of a code data-block. This allows any control transfer within a
component to be done using a relative branch.
Besides a function object, there are other kinds of references into the middle
of a code data-block. Control transfer into a function also occurs at the
return-PC for a call. The system represents a return-PC somewhat similarly to
a function, so GC can also recognize a return-PC as a reference to a code
data-block.
It is incorrect to think of a code data-block as a concatenation of "function
data-blocks". Code for a function is not emitted in any particular order with
respect to that function's function-header (if any). The code following a
function-header may only be a branch to some other location where the
function's "real" definition is.
The following are the three kinds of pointers to code data-blocks:
Code pointer (labeled A below):
A code pointer is a descriptor, with other-pointer low-tag bits, pointing
to the beginning of the code data-block. The code pointer for the
currently running function is always kept in a register (CODE). In
addition to allowing loading of non-immediate constants, this also serves
to represent the currently running function to the debugger.
Return-PC (labeled B below):
The return-PC is a descriptor, with other-pointer low-tag bits, pointing
to a location for a function call. Note that this location contains no
descriptors other than the one word of immediate data, so GC can treat
return-PC locations the same as instructions.
Function (labeled C below):
A function is a descriptor, with function low-tag bits, that is user
callable. When a function header is referenced from a closure or from
the function header's self-pointer, the pointer has other-pointer low-tag
bits, instead of function low-tag bits. This ensures that the internal
function data-block associated with a closure appears to be uncallable
(although users should never see such an object anyway).
Information about functions that is only useful for entry points is kept
in some descriptors following the function's self-pointer descriptor.
All of these together with the function's header-word are known as the
"function header". GC must be able to locate the function header. We
provide for this by chaining together the function headers in a NIL
terminated list kept in a known slot in the code data-block.
A code data-block has the following format:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ <-- A
| Header-Word count (24 bits) | %Code-Type (8 bits) |
----------------------------------------------------------------
| Number of code words (fixnum tag) |
----------------------------------------------------------------
| Pointer to first function header (other-pointer tag) |
----------------------------------------------------------------
| Debug information (structure tag) |
----------------------------------------------------------------
| First constant (a descriptor) |
----------------------------------------------------------------
| ... |
----------------------------------------------------------------
| Last constant (and last word of code header) |
----------------------------------------------------------------
| Some instructions (non-descriptor) |
----------------------------------------------------------------
| (pad to dual-word boundary if necessary) |
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ <-- B
| Word offset from code header (24) | %Return-PC-Type (8) |
----------------------------------------------------------------
| First instruction after return |
----------------------------------------------------------------
| ... more code and return-PC header-words |
----------------------------------------------------------------
| (pad to dual-word boundary if necessary) |
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ <-- C
| Offset from code header (24) | %Function-Header-Type (8) |
----------------------------------------------------------------
| Self-pointer back to previous word (with other-pointer tag) |
----------------------------------------------------------------
| Pointer to next function (other-pointer low-tag) or NIL |
----------------------------------------------------------------
| Function name (a string or a symbol) |
----------------------------------------------------------------
| Function debug arglist (a string) |
----------------------------------------------------------------
| Function type (a list-style function type specifier) |
----------------------------------------------------------------
| Start of instructions for function (non-descriptor) |
----------------------------------------------------------------
| More function headers and instructions and return PCs, |
| until we reach the total size of header-words + code |
| words. |
----------------------------------------------------------------
The following are detailed slot descriptions:
Code data-block header-word:
The immediate data in the code data-block's header-word is the number of
leading descriptors in the code data-block, the fixed overhead words plus
the number of constants. The first non-descriptor word, some code,
appears at this word offset from the header.
Number of code words:
The total number of non-header-words in the code data-block. The total
word size of the code data-block is the sum of this slot and the
immediate header-word data of the previous slot. The system accesses
this slot with the system constant, %Code-Code-Size-Slot, offset from the
header-word.
Pointer to first function header:
A NIL-terminated list of the function headers for all entry points to
this component. The system accesses this slot with the system constant,
%Code-Entry-Points-Slot, offset from the header-word.
Debug information:
The DEBUG-INFO structure describing this component. All information that
the debugger wants to get from a running function is kept in this
structure. Since there are many functions, the current PC is used to
locate the appropriate debug information. The system keeps the debug
information separate from the function data-block, since the currently
running function may not be an entry point. There is no way to recover
the function object for the currently running function, since this
data-block may not exist. The system accesses this slot with the system
constant, %Code-Debug-Info-Slot, offset from the header-word.
First constant ... last constant:
These are the constants referenced by the component, if there are any.
The system accesses the first constant slot with the system constant,
%Code-Constants-Offset, offset from the header-word.
Return-PC header word:
The immediate header-word data is the word offset from the enclosing code
data-block's header-word to this word. This allows GC and the debugger
to easily recover the code data-block from a return-PC. The code at the
return point restores the current code pointer using a subtract immediate
of the offset, which is known at compile time.
Function entry point header-word:
The immediate header-word data is the word offset from the enclosing code
data-block's header-word to this word. This is the same as for the
retrun-PC header-word.
Self-pointer back to header-word:
In a non-closure function, this self-pointer to the previous header-word
allows the call sequence to always indirect through the second word in a
user callable function. See section "Closure Format". With a closure,
indirecting through the second word gets you a function header-word. The
system ignores this slot in the function header for a closure, since it
has already indirected once, and this slot could be some random thing
that causes an error if you jump to it. This pointer has an
other-pointer tag instead of a function pointer tag, indicating it is not
a user callable Lisp object. The system accesses this slot with the
system constant, %Function-Code-Slot, offset from the function
header-word.
Pointer to next function:
This is the next link in the thread of entry point functions found in
this component. This value is NIL when the current header is the last
entry point in the component. The system accesses this slot with the
system constant, %Function-Header-Next-Slot, offset from the function
header-word.
Function name:
This function's name (for printing). If the user defined this function
with DEFUN, then this is the defined symbol, otherwise it is a
descriptive string. The system accesses this slot with the system
constant, %Function-Header-Name-Slot, offset from the function
header-word.
Function debug arglist:
A printed string representing the function's argument list, for human
readability. If it is a macroexpansion function, then this is the
original DEFMACRO arglist, not the actual expander function arglist. The
system accesses this slot with the system constant,
%Function-Header-Debug-Arglist-Slot, offset from the function
header-word.
Function type:
A list-style function type specifier representing the argument signature
and return types for this function. For example,
(FUNCTION (FIXNUM FIXNUM FIXNUM) FIXNUM)
or
(FUNCTION (STRING &KEY (:START UNSIGNED-BYTE)) STRING)
This information is intended for machine readablilty, such as by the
compiler. The system accesses this slot with the system constant,
%Function-Header-Type-Slot, offset from the function header-word.
;;;; Closure Format.
A closure data-block has the following format:
----------------------------------------------------------------
| Word size (24 bits) | %Closure-Type (8 bits) |
----------------------------------------------------------------
| Pointer to function header (other-pointer low-tag) |
----------------------------------------------------------------
| . |
| Environment information |
| . |
----------------------------------------------------------------
A closure descriptor has function low-tag bits. This means that a descriptor
with function low-tag bits may point to either a function header or to a
closure. The idea is that any callable Lisp object has function low-tag bits.
Insofar as call is concerned, we make the format of closures and non-closure
functions compatible. This is the reason for the self-pointer in a function
header. Whenever you have a callable object, you just jump through the second
word, offset some bytes, and go.
;;;; Function call.
Due to alignment requirements and low-tag codes, it is not possible to use a
hardware call instruction to compute the return-PC. Instead the return-PC
for a call is computed by doing an add-immediate to the start of the code
data-block.
An advantage of using a single data-block to represent both the descriptor and
non-descriptor parts of a function is that both can be represented by a
single pointer. This reduces the number of memory accesses that have to be
done in a full call. For example, since the constant pool is implicit in a
return-PC, a call need only save the return-PC, rather than saving both the
return PC and the constant pool.
;;;; Memory Layout.
CMU Common Lisp has four spaces, read-only, static, dynamic-0, and dynamic-1.
Read-only contains objects that the system never modifies, moves, or reclaims.
Static space contains some global objects necessary for the system's runtime or
performance (since they are located at a known offset at a know address), and
the system never moves or reclaims these. However, GC does need to scan static
space for references to moved objects. Dynamic-0 and dynamic-1 are the two
heap areas for stop-and-copy GC algorithms.
What global objects are at the head of static space???
NIL
eval::*top-of-stack*
lisp::*current-catch-block*
lisp::*current-unwind-protect*
FLAGS (RT only)
BSP (RT only)
HEAP (RT only)
In addition to the above spaces, the system has a control stack, binding stack,
and a number stack. The binding stack contains pairs of descriptors, a symbol
and its previous value. The number stack is the same as the C stack, and the
system uses it for non-Lisp objects such as raw system pointers, saving
non-Lisp registers, parts of bignum computations, etc.
;;;; System Pointers.
The system pointers reference raw allocated memory, data returned by foreign
function calls, etc. The system uses these when you need a pointer to a
non-Lisp block of memory, using an other-pointer. This provides the greatest
flexibility by relieving contraints placed by having more direct references
that require descriptor type tags.
A system area pointer data-block has the following format:
-------------------------------------------------------
| 1 (data-block words) | SAP Type (8 bits) |
-------------------------------------------------------
| system area pointer |
-------------------------------------------------------
"SAP" means "system area pointer", and much of our code contains this naming
scheme. We don't currently restrict system pointers to one area of memory, but
if they do point onto the heap, it is up to the user to prevent being screwed
by GC or whatever.

196
doc/internals/cmu/interpreter.tex Normal file
View file

@ -0,0 +1,196 @@
% -*- Dictionary: design; Package: C -*-
\chapter{The IR1 Interpreter}
May be worth having a byte-code representation for interpreted code. This way,
an entire system could be compiled into byte-code for debugging (the
``check-out'' compiler?).
Given our current inclination for using a stack machine to interpret IR1, it
would be straightforward to layer a byte-code interpreter on top of this.
Instead of having no interpreter, or a more-or-less conventional interpreter,
or byte-code interpreter, how about directly executing IR1?
We run through the IR1 passes, possibly skipping optional ones, until we get
through environment analysis. Then we run a post-pass that annotates IR1 with
information about where values are kept, i.e. the stack slot.
We can lazily convert functions by having FUNCTION make an interpreted function
object that holds the code (really a closure over the interpreter). The first
time that we try to call the function, we do the conversion and processing.
Also, we can easily keep track of which interpreted functions we have expanded
macros in, so that macro redefinition automatically invalidates the old
expansion, causing lazy reconversion.
Probably the interpreter will want to represent MVs by a recognizable structure
that is always heap-allocated. This way, we can punt the stack issues involved
in trying to spread MVs. So a continuation value can always be kept in a
single cell.
The compiler can have some special frobs for making the interpreter efficient,
such as a call operation that extracts arguments from the stack
slots designated by a continuation list. Perhaps
\begin{verbatim}
(values-mapcar fun . lists)
<==>
(values-list (mapcar fun . lists))
\end{verbatim}
This would be used with MV-CALL.
This scheme seems to provide nearly all of the advantages of both the compiler
and conventional interpretation. The only significant disadvantage with
respect to a conventional interpreter is that there is the one-time overhead of
conversion, but doing this lazily should make this quite acceptable.
With respect to a conventional interpreter, we have major advantages:
+ Full syntax checking: safety comparable to compiled code.
+ Semantics similar to compiled code due to code sharing. Similar diagnostic
messages, etc. Reduction of error-prone code duplication.
+ Potential for full type checking according to declarations (would require
running IR1 optimize?)
+ Simplifies debugger interface, since interpreted code can look more like
compiled code: source paths, edit definition, etc.
For all non-run-time symbol annotations (anything other than SYMBOL-FUNCTION
and SYMBOL-VALUE), we use the compiler's global database. MACRO-FUNCTION will
use INFO, rather than vice-versa.
When doing the IR1 phases for the interpreter, we probably want to suppress
optimizations that change user-visible function calls:
-- Don't do local call conversion of any named functions (even lexical ones).
This is so that a call will appear on the stack that looks like the call in
the original source. The keyword and optional argument transformations
done by local call mangle things quite a bit. Also, note local-call
converting prevents unreferenced arguments from being deleted, which is
another non-obvious transformation.
-- Don't run source-transforms, IR1 transforms and IR1 optimizers. This way,
TRACE and BACKTRACE will show calls with the original arguments, rather
than the ``optimized'' form, etc. Also, for the interpreter it will
actually be faster to call the original function (which is compiled) than
to ``inline expand'' it. Also, this allows implementation-dependent
transforms to expand into %PRIMITIVE uses.
There are some problems with stepping, due to our non-syntactic IR1
representation. The source path information is the key that makes this
conceivable. We can skip over the stepping of a subform by quietly evaluating
nodes whose source path lies within the form being skipped.
One problem with determining what value has been returned by a form. With a
function call, it is theoretically possible to precisely determine this, since
if we complete evaluation of the arguments, then we arrive at the Combination
node whose value is synonymous with the value of the form. We can even detect
this case, since the Node-Source will be EQ to the form. And we can also
detect when we unwind out of the evaluation, since we will leave the form
without having ever reached this node.
But with macros and special-forms, there is no node whose value is the value of
the form, and no node whose source is the macro call or special form. We can
still detect when we leave the form, but we can't be sure whether this was a
normal evaluation result or an explicit RETURN-FROM.
But does this really matter? It seems that we can print the value returned (if
any), then just print the next form to step. In the rare case where we did
unwind, the user should be able to figure it out.
[We can look at this as a side-effect of CPS: there isn't any difference
between a ``normal'' return and a non-local one.]
[Note that in any control transfer (normal or otherwise), the stepper may need
to unwind out of an arbitrary number of levels of stepping. This is because a
form in a TR position may yield its to a node arbitrarily far out.]
Another problem is with deciding what form is being stepped. When we start
evaluating a node, we dive into code that is nested somewhere down inside that
form. So we actually have to do a loop of asking questions before we do any
evaluation. But what do we ask about?
If we ask about the outermost enclosing form that is a subform of the last
form that the user said to execute, then we might offer a form that isn't
really evaluated, such as a LET binding list.
But once again, is this really a problem? It is certainly different from a
conventional stepper, but a pretty good argument could be made that it is
superior. Haven't you ever wanted to skip the evaluation of all the
LET bindings, but not the body? Wouldn't it be useful to be able to skip the
DO step forms?
All of this assumes that nobody ever wants to step through the guts of a
macroexpansion. This seems reasonable, since steppers are for weenies, and
weenies don't define macros (hence don't debug them). But there are probably
some weenies who don't know that they shouldn't be writing macros.
We could handle this by finding the ``source paths'' in the expansion of each
macro by sticking some special frob in the source path marking the place where
the expansion happened. When we hit code again that is in the source, then we
revert to the normal source path. Something along these lines might be a good
idea anyway (for compiler error messages, for example).
The source path hack isn't guaranteed to work quite so well in generated code,
though, since macros return stuff that isn't freshly consed. But we could
probably arrange to win as long as any given expansion doesn't return two EQ
forms.
It might be nice to have a command that skipped stepping of the form, but
printed the results of each outermost enclosed evaluated subform, i.e. if you
used this on the DO step-list, it would print the result of each new-value
form. I think this is implementable. I guess what you would do is print each
value delivered to a DEST whose source form is the current or an enclosing
form. Along with the value, you would print the source form for the node that
is computing the value.
The stepper can also have a ``back'' command that ``unskips'' or ``unsteps''. This
would allow the evaluation of forms that are pure (modulo lexical variable
setting) to be undone. This is useful, since in stepping it is common that you
skip a form that you shouldn't have, or get confused and want to restart at
some earlier point.
What we would do is remember the current node and the values of all local
variables. heap before doing each step or skip action. We can then back up
the state of all lexical variables and the ``program counter''. To make this
work right with set closure variables, we would copy the cell's value, rather
than the value cell itself.
[To be fair, note that this could easily be done with our current interpreter:
the stepper could copy the environment alists.]
We can't back up the ``program counter'' when a control transfer leaves the
current function, since this state is implicitly represented in the
interpreter's state, and is discarded when we exit. We probably want to ask
for confirmation before leaving the function to give users a chance to ``unskip''
the forms in a TR position.
Another question is whether the conventional stepper is really a good thing to
imitate... How about an editor-based mouse-driven interface? Instead of
``skipping'' and ``stepping'', you would just designate the next form that you
wanted to stop at. Instead of displaying return values, you replace the source
text with the printed representation of the value.
It would show the ``program counter'' by highlighting the *innermost* form that
we are about to evaluate, i.e. the source form for the node that we are stopped
at. It would probably also be useful to display the start of the form that was
used to designate the next stopping point, although I guess this could be
implied by the mouse position.
Such an interface would be a little harder to implement than a dumb stepper,
but it would be much easier to use. [It would be impossible for an evalhook
stepper to do this.]
\section{Use of \%PRIMITIVE}
Note: \verb|%PRIMITIVE| can only be used in compiled code. It is a
trapdoor into the compiler, not a general syntax for accessing
``sub-primitives''. It's main use is in implementation-dependent
compiler transforms. It saves us the effort of defining a ``phony
function'' (that is not really defined), and also allows direct
communication with the code generator through codegen-info arguments.
Some primitives may be exported from the VM so that \verb|%PRIMITIVE|
can be used to make it explicit that an escape routine or interpreter
stub is assuming an operation is implemented by the compiler.

85
doc/internals/cmu/lowlev.tex Normal file
View file

@ -0,0 +1,85 @@
\chapter{Memory Management}
\section{Stacks and Globals}
\section{Heap Layout}
\section{Garbage Collection}
\chapter{Interface to C and Assembler}
\section{Linkage Table}
The linkage table feature is based on how dynamic libraries dispatch.
A table of functions is used which is filled in with the appropriate
code to jump to the correct address.
For \cmucl{}, this table is stored at
\code{target-foreign-linkage-space-start}. Each entry is
\code{target-foreign-linkage-entry-size} bytes long.
At startup, the table is initialized with default values in
\code{os\_foreign\_linkage\_init}. On x86 platforms, the first entry is
code to call the routine \code{resolve\_linkage\_tramp}. All other
entries jump to the first entry. The function
\code{resolve\_linkage\_tramp} looks at where it was called from to
figure out which entry in the table was used. It calls
\code{lazy\_resolve\_linkage} with the address of the linkage entry.
This routine then fills in the appropriate linkage entry with code to
jump to where the real routine is located, and returns the address of
the entry. On return, \code{resolve\_linkage\_tramp} then just jumps to
the returned address to call the desired function. On all subsequent
calls, the entry no longer points to \code{resolve\_linkage\_tramp} but
to the real function.
This describes how function calls are made. For foreign data,
\code{lazy\_resolve\_linkage} stuffs the address of the actual foreign
data into the linkage table. The lisp code then just loads the address
from there to get the actual address of the foreign data.
For sparc, the linkage table is slightly different. The first entry is
the entry for \code{call\_into\_c} so we never have to look this up. All
other entries are for \code{resolve\_linkage\_tramp}. This has the
advantage that \code{resolve\_linkage\_tramp} can be much simpler since
all calls to foreign code go through \code{call\_into\_c} anyway, and
that means all live Lisp registers have already been saved. Also, to
make life simpler, we lie about \code{closure\_tramp} and
\code{undefined\_tramp} in the Lisp code. These are really functions,
but we treat them as foreign data since these two routines are only
used as addresses in the Lisp code to stuff into a lisp function
header.
On the Lisp side, there are two supporting data structures for the
linkage table: \code{*linkage-table-data*} and
\code{*foreign-linkage-symbols*}. The latter is a hash table whose key
is the foreign symbol (a string) and whose value is an index into
\code{*linkage-table-data*}.
\code{*linkage-table-data*} is a vector with an unlispy layout. Each
entry has 3 parts:
\begin{itemize}
\item symbol name
\item type, a fixnum, 1 = code, 2 = data
\item library list - the library list at the time the symbol is registered.
\end{itemize}
Whenever a new foreign symbol is defined, a new
\code{*linkage-table-data*} entry is created.
\code{*foreign-linkage-symbols*} is updated with the symbol and the
entry number into \code{*linkage-table-data*}.
The \code{*linkage-table-data*} is accessed from C (hence the unlispy
layout), to figure out the symbol name and the type so that the
address of the symbol can be determined. The type tells the C code
how to fill in the entry in the linkage-table itself.
% (Should say something about genesis too, but I don't know how that
% works other than the initial table is setup with the appropriate first
% entry.)
\chapter{Low-level debugging}
\chapter{Core File Format}

667
doc/internals/cmu/middle.tex Normal file
View file

@ -0,0 +1,667 @@
% -*- Dictionary: design -*-
\chapter{Virtual Machine Representation Introduction}
\chapter{Global TN assignment}
% Rename this phase so as not to be confused with the local/global TN
% representation.
The basic mechanism for closing over values is to pass the values as additional
implicit arguments in the function call. This technique is only applicable
when:
\begin{itemize}
\item the calling function knows which values the called function wants to close
over, and
\item the values to be closed over are available in the calling
environment.
\end{itemize}
The first condition is always true of local function calls. Environment
analysis can guarantee that the second condition holds by closing over any
needed values in the calling environment.
If the function that closes over values may be called in an environment where
the closed over values are not available, then we must store the values in a
``closure'' so that they are always accessible. Closures are called using the
``full call'' convention. When a closure is called, control is transferred to
the ``external entry point'', which fetches the values out of the closure and
then does a local call to the real function, passing the closure values as
implicit arguments.
In this scheme there is no such thing as a ``heap closure variable'' in code,
since the closure values are moved into TNs by the external entry point. There
is some potential for pessimization here, since we may end up moving the values
from the closure into a stack memory location, but the advantages are also
substantial. Simplicity is gained by always representing closure values the
same way, and functions with closure references may still be called locally
without allocating a closure. All the TN based VMR optimizations will apply
to closure variables, since closure variables are represented in the same way
as all other variables in VMR. Closure values will be allocated in registers
where appropriate.
Closures are created at the point where the function is referenced, eliminating
the need to be able to close over closures. This lazy creation of closures has
the additional advantage that when a closure reference is conditionally not
done, then the closure consing will never be done at all. The corresponding
disadvantage is that a closure over the same values may be created multiple
times if there are multiple references. Note however, that VMR loop and common
subexpression optimizations can eliminate redundant closure consing. In any
case, multiple closures over the same variables doesn't seem to be that common.
\#|
Having the Tail-Info would also make return convention determination trivial.
We could just look at the type, checking to see if it represents a fixed number
of values. To determine if the standard return convention is necessary to
preserve tail-recursion, we just iterate over the equivalent functions, looking
for XEPs and uses in full calls.
|\#
The Global TN Assignment pass (GTN) can be considered a post-pass to
environment analysis. This phase assigns the TNs used to hold local lexical
variables and pass arguments and return values and determines the value-passing
strategy used in local calls.
To assign return locations, we look at the function's tail-set.
If the result continuation for an entry point is used as the continuation for a
full call, then we may need to constrain the continuation's values passing
convention to the standard one. This is not necessary when the call is known
not to be part of a tail-recursive loop (due to being a known function).
Once we have figured out where we must use the standard value passing strategy,
we can use a more flexible strategy to determine the return locations for local
functions. We determine the possible numbers of return values from each
function by examining the uses of all the result continuations in the
equivalence class of the result continuation.
If the tail-set type is for a fixed number of
values, then we return that fixed number of values from all the functions whose
result continuations are equated. If the number of values is not fixed, then
we must use the unknown-values convention, although we are not forced to use
the standard locations. We assign the result TNs at this time.
We also use the tail-sets to see what convention we want to use. What we do is
use the full convention for any function that has a XEP its tail-set, even if
we aren't required to do so by a tail-recursive full call, as long as there are
no non-tail-recursive local calls in the set. This prevents us from
gratuitously using a non-standard convention when there is no reason to.
\chapter{Local TN assignment}
[Want a different name for this so as not to be confused with the different
local/global TN representations. The really interesting stuff in this phase is
operation selection, values representation selection, return strategy, etc.
Maybe this phase should be conceptually lumped with GTN as ``implementation
selection'', since GTN determines call strategies and locations.]
\#|
[\#\#\# I guess I believe that it is OK for VMR conversion to dick the ICR flow
graph. An alternative would be to give VMR its very own flow graph, but that
seems like overkill.
In particular, it would be very nice if a TR local call looked exactly like a
jump in VMR. This would allow loop optimizations to be done on loops written
as recursions. In addition to making the call block transfer to the head of
the function rather than to the return, we would also have to do something
about skipping the part of the function prolog that moves arguments from the
passing locations, since in a TR call they are already in the right frame.
In addition to directly indicating whether a call should be coded with a TR
variant, the Tail-P annotation flags non-call nodes that can directly return
the value (an ``advanced return''), rather than moving the value to the result
continuation and jumping to the return code. Then (according to policy), we
can decide to advance all possible returns. If all uses of the result are
Tail-P, then LTN can annotate the result continuation as :Unused, inhibiting
emission of the default return code.
[\#\#\# But not really. Now there is a single list of templates, and a given
template has only one policy.]
In LTN, we use the :Safe template as a last resort even when the policy is
unsafe. Note that we don't try :Fast-Safe; if this is also a good unsafe
template, then it should have the unsafe policies explicitly specified.
With a :Fast-Safe template, the result type must be proven to satisfy the
output type assertion. This means that a fast-safe template with a fixnum
output type doesn't need to do fixnum overflow checking. [\#\#\# Not right to
just check against the Node-Derived-Type, since type-check intersects with
this.]
It seems that it would be useful to have a kind of template where the args must
be checked to be fixnum, but the template checks for overflow and signals an
error. In the case where an output assertion is present, this would generate
better code than conditionally branching off to make a bignum, and then doing a
type check on the result.
How do we deal with deciding whether to do a fixnum overflow check? This
is perhaps a more general problem with the interpretation of result type
restrictions in templates. It would be useful to be able to discriminate
between the case where the result has been proven to be a fixnum and where
it has simply been asserted to be so.
The semantics of result type restriction is that the result must be proven
to be of that type *except* for safe generators, which are assumed to
verify the assertion. That way ``is-fixnum'' case can be a fast-safe
generator and the ``should-be-fixnum'' case is a safe generator. We could
choose not to have a safe ``should-be-fixnum'' generator, and let the
unrestricted safe generator handle it. We would then have to do an
explicit type check on the result.
In other words, for all template except Safe, a type restriction on either
an argument or result means ``this must be true; if it is not the system may
break.'' In contrast, in a Safe template, the restriction means ``If this is
not true, I will signal an error.''
Since the node-derived-type only takes into consideration stuff that can be
proved from the arguments, we can use the node-derived-type to select
fast-safe templates. With unsafe policies, we don't care, since the code
is supposed to be unsafe.
|\#
Local TN assignment (LTN) assigns all the TNs needed to represent the values of
continuations. This pass scans over the code for the component, examining each
continuation and its destination. A number of somewhat unrelated things are
also done at the same time so that multiple passes aren't necessary.
-- Determine the Primitive-Type for each continuation value and assigns TNs
to hold the values.
-- Use policy information to determine the implementation strategy for each
call to a known function.
-- Clear the type-check flags in continuations whose destinations have safe
implementations.
-- Determine the value-passing strategy for each continuation: known or
unknown.
-- Note usage of unknown-values continuations so that stack analysis can tell
when stack values must be discarded.
If safety is more important than speed and space, then we consider generating
type checks on the values of nodes whose CONT has the Type-Check flag set. If
the destination for the continuation value is safe, then we don't need to do
a check. We assume that all full calls are safe, and use the template
information to determine whether inline operations are safe.
This phase is where compiler policy switches have most of their effect. The
speed/space/safety tradeoff can determine which of a number of coding
strategies are used. It is important to make the policy choice in VMR
conversion rather than in code generation because the cost and storage
requirement information which drives TNBIND will depend strongly on what actual
VOP is chosen. In the case of +/FIXNUM, there might be three or more
implementations, some optimized for speed, some for space, etc. Some of these
VOPS might be open-coded and some not.
We represent the implementation strategy for a call by either marking it as a
full call or annotating it with a ``template'' representing the open-coding
strategy. Templates are selected using a two-way dispatch off of operand
primitive-types and policy. The general case of LTN is handled by the
LTN-Annotate function in the function-info, but most functions are handled by a
table-driven mechanism. There are four different translation policies that a
template may have:
\begin{description}
\item[Safe]
The safest implementation; must do argument type checking.
\item[Small]
The (unsafe) smallest implementation.
\item[Fast]
The (unsafe) fastest implementation.
\item[Fast-Safe]
An implementation optimized for speed, but which does any necessary
checks exclusive of argument type checking. Examples are array bounds
checks and fixnum overflow checks.
\end{description}
Usually a function will have only one or two distinct templates. Either or
both of the safe and fast-safe templates may be omitted; if both are specified,
then they should be distinct. If there is no safe template and our policy is
safe, then we do a full call.
We use four different coding strategies, depending on the policy:
\begin{description}
\item[Safe:] safety $>$ space $>$ speed, or
we want to use the fast-safe template, but there isn't one.
\item[Small:] space $>$ (max speed safety)
\item[Fast:] speed $>$ (max space safety)
\item[Fast-Safe (and type check):] safety $>$ speed $>$ space, or we want to use
the safe template, but there isn't one.
\end{description}
``Space'' above is actually the maximum of space and cspeed, under the theory
that less code will take less time to generate and assemble. [\#\#\# This could
lose if the smallest case is out-of-line, and must allocate many linkage
registers.]
\chapter{Control optimization}
In this phase we annotate blocks with drop-throughs. This controls how code
generation linearizes code so that drop-throughs are used most effectively. We
totally linearize the code here, allowing code generation to scan the blocks
in the emit order.
There are basically two aspects to this optimization:
\begin{enumerate}
\item
Dynamically reducing the number of branches taken v.s. branches not
taken under the assumption that branches not taken are cheaper.
\item
Statically minimizing the number of unconditional branches, saving
space and presumably time.
\end{enumerate}
These two goals can conflict, but if they do it seems pretty clear that the
dynamic optimization should get preference. The main dynamic optimization is
changing the sense of a conditional test so that the more commonly taken branch
is the fall-through case. The problem is determining which branch is more
commonly taken.
The most clear-cut case is where one branch leads out of a loop and the other
is within. In this case, clearly the branch within the loop should be
preferred. The only added complication is that at some point in the loop there
has to be a backward branch, and it is preferable for this branch to be
conditional, since an unconditional branch is just a waste of time.
In the absence of such good information, we can attempt to guess which branch
is more popular on the basis of difference in the cost between the two cases.
Min-max strategy suggests that we should choose the cheaper alternative, since
the percentagewise improvement is greater when the branch overhead is
significant with respect to the cost of the code branched to. A tractable
approximation of this is to compare only the costs of the two blocks
immediately branched to, since this would avoid having to do any hairy graph
walking to find all the code for the consequent and the alternative. It might
be worthwhile discriminating against ultra-expensive functions such as ERROR.
For this to work, we have to detect when one of the options is empty. In this
case, the next for one branch is a successor of the other branch, making the
comparison meaningless. We use dominator information to detect this situation.
When a branch is empty, one of the predecessors of the first block in the empty
branch will be dominated by the first block in the other branch. In such a
case we favor the empty branch, since that's about as cheap as you can get.
Statically minimizing branches is really a much more tractable problem, but
what literature there is makes it look hard. Clearly the thing to do is to use
a non-optimal heuristic algorithm.
A good possibility is to use an algorithm based on the depth first ordering.
We can modify the basic DFO algorithm so that it chooses an ordering which
favors any drop-thrus that we may choose for dynamic reasons. When we are
walking the graph, we walk the desired drop-thru arc last, which will place it
immediately after us in the DFO unless the arc is a retreating arc.
We scan through the DFO and whenever we find a block that hasn't been done yet,
we build a straight-line segment by setting the drop-thru to the unreached
successor block which has the lowest DFN greater than that for the block. We
move to the drop-thru block and repeat the process until there is no such
block. We then go back to our original scan through the DFO, looking for the
head of another straight-line segment.
This process will automagically implement all of the dynamic optimizations
described above as long as we favor the appropriate IF branch when creating the
DFO. Using the DFO will prevent us from making the back branch in a loop the
drop-thru, but we need to be clever about favoring IF branches within loops
while computing the DFO. The IF join will be favored without any special
effort, since we follow through the most favored path until we reach the end.
This needs some knowledge about the target machine, since on most machines
non-tail-recursive calls will use some sort of call instruction. In this case,
the call actually wants to drop through to the return point, rather than
dropping through to the beginning of the called function.
\chapter{VMR conversion}
\#|
Single-use let var continuation substitution not really correct, since it can
cause a spurious type error. Maybe we do want stuff to prove that an NLX can't
happen after all. Or go back to the idea of moving a combination arg to the
ref location, and having that use the ref cont (with its output assertion.)
This lossage doesn't seem very likely to actually happen, though.
[\#\#\# must-reach stuff wouldn't work quite as well as combination substitute in
psetq, etc., since it would fail when one of the new values is random code
(might unwind.)]
Is this really a general problem with eager type checking? It seems you could
argue that there was no type error in this code:
\begin{verbatim}
(+ :foo (throw 'up nil))
\end{verbatim}
But we would signal an error.
Emit explicit you-lose operation when we do a move between two non-T ptypes,
even when type checking isn't on. Can this really happen? Seems we should
treat continuations like this as though type-check was true. Maybe LTN should
leave type-check true in this case, even when the policy is unsafe. (Do a type
check against NIL?)
At continuation use time, we may in general have to do both a coerce-to-t and a
type check, allocating two temporary TNs to hold the intermediate results.
\section{VMR Control representation}
We represent all control transfer explicitly. In particular, :Conditional VOPs
take a single Target continuation and a Not-P flag indicating whether the sense
of the test is negated. Then an unconditional Branch VOP will be emitted
afterward if the other path isn't a drop-through.
So we linearize the code before VMR-conversion. This isn't a problem,
since there isn't much change in control flow after VMR conversion (none until
loop optimization requires introduction of header blocks.) It does make
cost-based branch prediction a bit ucky, though, since we don't have any cost
information in ICR. Actually, I guess we do have pretty good cost information
after LTN even before VMR conversion, since the most important thing to know is
which functions are open-coded.
|\#
VMR preserves the block structure of ICR, but replaces the nodes with a target
dependent virtual machine (VM) representation. Different implementations may
use different VMs without making major changes in the back end. The two main
components of VMR are Temporary Names (TNs) and Virtual OPerations (VOPs). TNs
represent the locations that hold values, and VOPs represent the operations
performed on the values.
A ``primitive type'' is a type meaningful at the VM level. Examples are Fixnum,
String-Char, Short-Float. During VMR conversion we use the primitive type of
an expression to determine both where we can store the result of the expression
and which type-specific implementations of an operation can be applied to the
value. [Ptype is a set of SCs == representation choices and representation
specific operations]
The VM specific definitions provide functions that do stuff like find the
primitive type corresponding to a type and test for primitive type subtypep.
Usually primitive types will be disjoint except for T, which represents all
types.
The primitive type T is special-cased. Not only does it overlap with all the
other types, but it implies a descriptor (``boxed'' or ``pointer'') representation.
For efficiency reasons, we sometimes want to use
alternate representations for some objects such as numbers. The majority of
operations cannot exploit alternate representations, and would only be
complicated if they had to be able to convert alternate representations into
descriptors. A template can require an operand to be a descriptor by
constraining the operand to be of type T.
A TN can only represent a single value, so we bare the implementation of MVs at
this point. When we know the number of multiple values being handled, we use
multiple TNs to hold them. When the number of values is actually unknown, we
use a convention that is compatible with full function call.
Everything that is done is done by a VOP in VMR. Calls to simple primitive
functions such as + and CAR are translated to VOP equivalents by a table-driven
mechanism. This translation is specified by the particular VM definition; VMR
conversion makes no assumptions about which operations are primitive or what
operand types are worth special-casing. The default calling mechanisms and
other miscellaneous builtin features are implemented using standard VOPs that
must be implemented by each VM.
Type information can be forgotten after VMR conversion, since all type-specific
operation selections have been made.
Simple type checking is explicitly done using CHECK-xxx VOPs. They act like
innocuous effectless/unaffected VOPs which return the checked thing as a
result. This allows loop-invariant optimization and common subexpression
elimination to remove redundant checks. All type checking is done at the time
the continuation is used.
Note that we need only check asserted types, since if type inference works, the
derived types will also be satisfied. We can check whichever is more
convenient, since both should be true.
Constants are turned into special Constant TNs, which are wired down in a SC
that is determined by their type. The VM definition provides a function that
returns a constant TN to represent a Constant Leaf.
Each component has a constant pool. There is a register dedicated to holding
the constant pool for the current component. The back end allocates
non-immediate constants in the constant pool when it discovers them during
translation from ICR.
[\#\#\# Check that we are describing what is actually implemented. But this
really isn't very good in the presence of interesting unboxed
representations...]
Since LTN only deals with values from the viewpoint of the receiver, we must be
prepared during the translation pass to do stuff to the continuation at the
time it is used.
-- If a VOP yields more values than are desired, then we must create TNs to
hold the discarded results. An important special-case is continuations
whose value is discarded. These continuations won't be annotated at all.
In the case of a Ref, we can simply skip evaluation of the reference when
the continuation hasn't been annotated. Although this will eliminate
bogus references that for some reason weren't optimized away, the real
purpose is to handle deferred references.
-- If a VOP yields fewer values than desired, then we must default the extra
values to NIL.
-- If a continuation has its type-check flag set, then we must check the type
of the value before moving it into the result location. In general, this
requires computing the result in a temporary, and having the type-check
operation deliver it in the actual result location.
-- If the template's result type is T, then we must generate a boxed
temporary to compute the result in when the continuation's type isn't T.
We may also need to do stuff to the arguments when we generate code for a
template. If an argument continuation isn't annotated, then it must be a
deferred reference. We use the leaf's TN instead. We may have to do any of
the above use-time actions also. Alternatively, we could avoid hair by not
deferring references that must be type-checked or may need to be boxed.
\section{Stack analysis}
Think of this as a lifetime problem: a values generator is a write and a values
receiver is a read. We want to annotate each VMR-Block with the unknown-values
continuations that are live at that point. If we do a control transfer to a
place where fewer continuations are live, then we must deallocate the newly
dead continuations.
We want to convince ourselves that values deallocation based on lifetime
analysis actually works. In particular, we need to be sure that it doesn't
violate the required stack discipline. It is clear that it is impossible to
deallocate the values before they become dead, since later code may decide to
use them. So the only thing we need to ensure is that the ``right'' time isn't
later than the time that the continuation becomes dead.
The only reason why we couldn't deallocate continuation A as soon as it becomes
dead would be that there is another continuation B on top of it that isn't dead
(since we can only deallocate the topmost continuation).
The key to understanding why this can't happen is that each continuation has
only one read (receiver). If B is on top of A, then it must be the case that A
is live at the receiver for B. This means that it is impossible for B to be
live without A being live.
The reason that we don't solve this problem using a normal iterative flow
analysis is that we also need to know the ordering of the continuations on the
stack so that we can do deallocation. When it comes time to discard values, we
want to know which discarded continuation is on the bottom so that we can reset
SP to its start.
[I suppose we could also decrement SP by the aggregate size of the discarded
continuations.] Another advantage of knowing the order in which we expect
continuations to be on the stack is that it allows us to do some consistency
checking. Also doing a localized graph walk around the values-receiver is
likely to be much more efficient than doing an iterative flow analysis problem
over all the code in the component (not that big a consideration.)
\#|
Actually, what we do is a backward graph walk from each unknown-values
receiver. As we go, we mark each walked block with the ordered list of
continuations we believe are on the stack. Starting with an empty stack, we:
-- When we encounter another unknown-values receiver, we push that
continuation on our simulated stack.
-- When we encounter a receiver (which had better be for the topmost
continuation), we pop that continuation.
-- When we pop all continuations, we terminate our walk.
[\#\#\# not quite right... It seems we may run into ``dead values'' during the
graph walk too. It seems that we have to check if the pushed continuation is
on stack top, and if not, add it to the ending stack so that the post-pass will
discard it.]
[\#\#\# Also, we can't terminate our walk just because we hit a block previously
walked. We have to compare the End-Stack with the values received along
the current path: if we have more values on our current walk than on the walk
that last touched the block, then we need to re-walk the subgraph reachable
from that block, using our larger set of continuations. It seems that our
actual termination condition is reaching a block whose End-Stack is already EQ
to our current stack.]
If at the start, the block containing the values receiver has already been
walked, we skip the walk for that continuation, since it has already been
handled by an enclosing values receiver. Once a walk has started, we
ignore any signs of a previous walk, clobbering the old result with our own,
since we enclose that continuation, and the previous walk doesn't take into
consideration the fact that our values block underlies its own.
When we are done, we have annotated each block with the stack current both at
the beginning and at the end of that block. Blocks that aren't walked don't
have anything on the stack either place (although they may hack MVs
internally).
We then scan all the blocks in the component, looking for blocks that have
predecessors with a different ending stack than that block's starting stack.
(The starting stack had better be a tail of the predecessor's ending stack.)
We insert a block intervening between all of these predecessors that sets SP to
the end of the values for the continuation that should be on stack top. Of
course, this pass needn't be done if there aren't any global unknown MVs.
Also, if we find any block that wasn't reached during the walk, but that USEs
an outside unknown-values continuation, then we know that the DEST can't be
reached from this point, so the values are unused. We either insert code to
pop the values, or somehow mark the code to prevent the values from ever being
pushed. (We could cause the popping to be done by the normal pass if we
iterated over the pushes beforehand, assigning a correct END-STACK.)
[\#\#\# But I think that we have to be a bit clever within blocks, given the
possibility of blocks being joined. We could collect some unknown MVs in a
block, then do a control transfer out of the receiver, and this control
transfer could be squeezed out by merging blocks. How about:
\begin{verbatim}
(tagbody
(return
(multiple-value-prog1 (foo)
(when bar
(go UNWIND))))
UNWIND
(return
(multiple-value-prog1 (baz)
bletch)))
\end{verbatim}
But the problem doesn't happen here (can't happen in general?) since a node
buried within a block can't use a continuation outside of the block. In fact,
no block can have more then one PUSH continuation, and this must always be the
last continuation. So it is trivially (structurally) true that all pops come
before any push.
[\#\#\# But not really: the DEST of an embedded continuation may be outside the
block. There can be multiple pushes, and we must find them by iterating over
the uses of MV receivers in LTN. But it would be hard to get the order right
this way. We could easily get the order right if we added the generators as we
saw the uses, except that we can't guarantee that the continuations will be
annotated at that point. (Actually, I think we only need the order for
consistency checks, but that is probably worthwhile). I guess the thing to do
is when we process the receiver, add the generator blocks to the
Values-Generators, then do a post-pass that re-scans the blocks adding the
pushes.]
I believe that above concern with a dead use getting mashed inside a block
can't happen, since the use inside the block must be the only use, and if the
use isn't reachable from the push, then the use is totally unreachable, and
should have been deleted, which would prevent it from ever being
annotated.
]
]
|\#
We find the partial ordering of the values globs for unknown values
continuations in each environment. We don't have to scan the code looking for
unknown values continuations since LTN annotates each block with the
continuations that were popped and not pushed or pushed and not popped. This
is all we need to do the inter-block analysis.
After we have found out what stuff is on the stack at each block boundary, we
look for blocks with predecessors that have junk on the stack. For each such
block, we introduce a new block containing code to restore the stack pointer.
Since unknown-values continuations are represented as \verb+<start, count>+, we can
easily pop a continuation using the Start TN.
Note that there is only doubt about how much stuff is on the control stack,
since only it is used for unknown values. Any special stacks such as number
stacks will always have a fixed allocation.
\section{Non-local exit}
If the starting and ending continuations are not in the same environment, then
the control transfer is a non-local exit. In this case just call Unwind with
the appropriate stack pointer, and let the code at the re-entry point worry
about fixing things up.
It seems like maybe a good way to organize VMR conversion of NLX would be to
have environment analysis insert funny functions in new interposed cleanup
blocks. The thing is that we need some way for VMR conversion to:
1] Get its hands on the returned values.
2] Do weird control shit.
3] Deliver the values to the original continuation destination.
I.e. we need some way to interpose arbitrary code in the path of value
delivery.
What we do is replace the NLX uses of the continuation with another
continuation that is received by a MV-Call to \%NLX-VALUES in a cleanup block
that is interposed between the NLX uses and the old continuation's block. The
MV-Call uses the original continuation to deliver its values to.
[Actually, it's not really important that this be an MV-Call, since it has to
be special-cased by LTN anyway. Or maybe we would want it to be an MV call.
If we did normal LTN analysis of an MV call, it would force the returned values
into the unknown values convention, which is probably pretty convenient for use
in NLX.
Then the entry code would have to use some special VOPs to receive the unknown
values. But we probably need special VOPs for NLX entry anyway, and the code
can share with the call VOPs. Also we probably need the technology anyway,
since THROW will use truly unknown values.]
On entry to a dynamic extent that has non-local-exists into it (always at an
ENTRY node), we take a complete snapshot of the dynamic state:
\begin{itemize}
\item the top pointers for all stacks
\item current Catch and Unwind-Protect
\item current special binding (binding stack pointer in shallow binding)
\end{itemize}
We insert code at the re-entry point which restores the saved dynamic state.
All TNs live at an NLX EP are forced onto the stack, so we don't have to restore
them, and we don't have to worry about getting them saved.

770
doc/internals/cmu/object.tex Normal file
View file

@ -0,0 +1,770 @@
\chapter{Object Format}
\label{sec:tagging}
\section{Tagging}
The following is a key of the three bit low-tagging scheme:
\begin{description}
\item[000] even fixnum
\item[001] function pointer
\item[010] even other-immediate (header-words, characters, symbol-value trap value, etc.)
\item[011] list pointer
\item[100] odd fixnum
\item[101] structure pointer
\item[110] odd other immediate
\item[111] other-pointer to data-blocks (other than conses, structures,
and functions)
\end{description}
This tagging scheme forces a dual-word alignment of data-blocks on the heap,
but this can be pretty negligible:
\begin{itemize}
\item RATIOS and COMPLEX must have a header-word anyway since they are not a
major type. This wastes one word for these infrequent data-blocks since
they require two words for the data.
\item BIGNUMS must have a header-word and probably contain only one other word
anyway, so we probably don't waste any words here. Most bignums just
barely overflow fixnums, that is by a bit or two.
\item Single and double FLOATS?
no waste, or
one word wasted
\item SYMBOLS have a pad slot (current called the setf function, but unused.)
\end{itemize}
Everything else is vector-like including code, so these probably take up
so many words that one extra one doesn't matter.
\section{GC Comments}
Data-Blocks comprise only descriptors, or they contain immediate data and raw
bits interpreted by the system. GC must skip the latter when scanning the
heap, so it does not look at a word of raw bits and interpret it as a pointer
descriptor. These data-blocks require headers for GC as well as for operations
that need to know how to interpret the raw bits. When GC is scanning, and it
sees a header-word, then it can determine how to skip that data-block if
necessary. Header-Words are tagged as other-immediates. See
``Other-Immediates'', section~\ref{sec:other-immediates} and
``Data-Blocks and Header-Words'', section~\ref{sec:data-blocks-and-header} for comments on
distinguishing header-words from other-immediate data. This distinction is
necessary since we scan through data-blocks containing only descriptors just as
we scan through the heap looking for header-words introducing data-blocks.
Data-Blocks containing only descriptors do not require header-words for GC
since the entire data-block can be scanned by GC a word at a time, taking
whatever action is necessary or appropriate for the data in that slot. For
example, a cons is referenced by a descriptor with a specific tag, and the
system always knows the size of this data-block. When GC encounters a pointer
to a cons, it can transport it into the new space, and when scanning, it can
simply scan the two words manifesting the cons interpreting each word as a
descriptor. Actually there is no cons tag, but a list tag, so we make sure the
cons is not nil when appropriate. A header may still be desired if the pointer
to the data-block does not contain enough information to adequately maintain
the data-block. An example of this is a simple-vector containing only
descriptor slots, and we attach a header-word because the descriptor pointing
to the vector lacks necessary information -- the type of the vector's elements,
its length, etc.
There is no need for a major tag for GC forwarding pointers. Since the tag
bits are in the low end of the word, a range check on the start and end of old
space tells you if you need to move the thing. This is all GC overhead.
\section{Structures}
A structure descriptor has the structure lowtag type code, making
{\tt structurep} a fast operation. A structure
data-block has the following format:
\begin{verbatim}
-------------------------------------------------------
| length (24 bits) | Structure header type (8 bits) |
-------------------------------------------------------
| structure type name (a symbol) |
-------------------------------------------------------
| structure slot 0 |
-------------------------------------------------------
| ... structure slot length - 2 |
-------------------------------------------------------
\end{verbatim}
The header word contains the structure length, which is the number of words
(other than the header word.) The length is always at least one, since the
first word of the structure data is the structure type name.
\section{Fixnums}
A fixnum has one of the following formats in 32 bits:
\begin{verbatim}
-------------------------------------------------------
| 30 bit 2's complement even integer | 0 0 0 |
-------------------------------------------------------
\end{verbatim}
or
\begin{verbatim}
-------------------------------------------------------
| 30 bit 2's complement odd integer | 1 0 0 |
-------------------------------------------------------
\end{verbatim}
Effectively, there is one tag for immediate integers, two zeros. This buys one
more bit for fixnums, and now when these numbers index into simple-vectors or
offset into memory, they point to word boundaries on 32-bit, byte-addressable
machines. That is, no shifting need occur to use the number directly as an
offset.
This format has another advantage on byte-addressable machines when fixnums are
offsets into vector-like data-blocks, including structures. Even though we
previously mentioned data-blocks are dual-word aligned, most indexing and slot
accessing is word aligned, and so are fixnums with effectively two tag bits.
Two tags also allow better usage of special instructions on some machines that
can deal with two low-tag bits but not three.
Since the two bits are zeros, we avoid having to mask them off before using the
words for arithmetic, but division and multiplication require special shifting.
\section{Other-immediates}
\label{sec:other-immediates}
As for fixnums, there are two different three-bit lowtag codes for
other-immediate, allowing 64 other-immediate types:
\begin{verbatim}
----------------------------------------------------------------
| Data (24 bits) | Type (8 bits with low-tag) | 1 0 |
----------------------------------------------------------------
\end{verbatim}
The type-code for an other-immediate type is considered to include the two
lowtag bits. This supports the concept of a single ``type code'' namespace for
all descriptors, since the normal lowtag codes are disjoint from the
other-immediate codes.
For other-pointer objects, the full eight bits of the header type code are used
as the type code for that kind of object. This is why we use two lowtag codes
for other-immediate types: each other-pointer object needs a distinct
other-immediate type to mark its header.
The system uses the other-immediate format for characters,
the {\tt symbol-value} unbound trap value, and header-words for data-blocks on
the heap. The type codes are laid out to facilitate range checks for common
subtypes; for example, all numbers will have contiguous type codes which are
distinct from the contiguous array type codes. See
section~\ref{sec:data-blocks-and-o-i}
for details.
\section{Data-Blocks and Header-Word Format}
\label{sec:data-blocks-and-header}
Pointers to data-blocks have the following format:
\begin{verbatim}
----------------------------------------------------------------
| Dual-word address of data-block (29 bits) | 1 1 1 |
----------------------------------------------------------------
\end{verbatim}
The word pointed to by the above descriptor is a header-word, and it has the
same format as an other-immediate:
\begin{verbatim}
----------------------------------------------------------------
| Data (24 bits) | Type (8 bits with low-tag) | 0 1 0 |
----------------------------------------------------------------
\end{verbatim}
This is convenient for scanning the heap when GC'ing, but it does mean that
whenever GC encounters an other-immediate word, it has to do a range check on
the low byte to see if it is a header-word or just a character (for example).
This is easily acceptable performance hit for scanning.
The system interprets the data portion of the header-word for non-vector
data-blocks as the word length excluding the header-word. For example, the
data field of the header for ratio and complex numbers is two, one word each
for the numerator and denominator or for the real and imaginary parts.
For vectors and data-blocks representing Lisp objects stored like vectors, the
system (usually) ignores the data portion of the header-word:
\begin{verbatim}
----------------------------------------------------------------
| Unused Data (24 bits) | Type (8 bits with low-tag) | 0 1 0 |
----------------------------------------------------------------
| Element Length of Vector (30 bits) | 0 0 |
----------------------------------------------------------------
\end{verbatim}
Using a separate word allows for much larger vectors, and it allows {\tt
length} to simply access a single word without masking or shifting. Similarly,
the header for complex arrays and vectors has a second word, following the
header-word, the system uses for the fill pointer, so computing the length of
any array is the same code sequence.
For normal Lisp vectors, the data portion MUST be zero. For hash
tables, a vector is used to store information about the hash key and
value, and the data portion is non-zero to indicate to GC that this is
the key/value vector for the hash table. GENCGC uses this to
determine scavenge the key/value pairs correctly. Cheney GC also uses
this to determine if rehashing (for EQ hash tables) is needed.
\section{Data-Blocks and Other-immediates Typing}
\label{sec:data-blocks-and-o-i}
These are the other-immediate types. We specify them including all low eight
bits, including the other-immediate tag, so we can think of the type bits as
one type -- not an other-immediate major type and a subtype. Also, fetching a
byte and comparing it against a constant is more efficient than wasting even a
small amount of time shifting out the other-immediate tag to compare against a
five bit constant. (The current values can be obtained from the
generated \code{internals.h} file.)
\begin{verbatim}
HEX
Number (< 36)
bignum 10 0A
ratio 14 0E
single-float 18 12
double-float 22 16
double-double-float 26 1A
complex 30 1E
(complex single-float) 34 22
(complex double-float) 38 26
(complex double-double-float) 42 2A
Array (<= 46 code 118)
Simple-Array (<= 46 code 118)
simple-array 46 2E
Vector (<= 50 code 118)
simple-string 50 32
simple-bit-vector 54 36
simple-vector 58 3A
(simple-array (unsigned-byte 2) (*)) 62 3E
(simple-array (unsigned-byte 4) (*)) 66 42
(simple-array (unsigned-byte 8) (*)) 70 46
(simple-array (unsigned-byte 16) (*)) 74 4A
(simple-array (unsigned-byte 32) (*)) 78 4E
(simple-array (signed-byte 8) (*)) 82 52
(simple-array (signed-byte 16) (*)) 86 56
(simple-array (signed-byte 30) (*)) 90 5A
(simple-array (signed-byte 32) (*)) 94 5E
(simple-array single-float (*)) 98 62
(simple-array double-float (*)) 102 66
(simple-array double-double-float (*)) 106 6A
(simple-array (complex single-float) (*) 110 6E
(simple-array (complex double-float) (*) 114 72
(simple-array (complex double-double) (*) 118 76
complex-string 122 7A
complex-bit-vector 126 7E
(array * (*)) -- general complex vector. 130 82
complex-array 134 86
code-header-type 138 8A
function-header-type 142 8E
closure-header-type 146 92
funcallable-instance-header-type 150 96
byte-code-function-header-type 154 9A
byte-code-closure-header-type 158 9E
closure-function-header-type 162 A2
return-pc-header-type (a.k.a LRA) 166 A6
value-cell-header-type 170 AA
symbol-header-type 174 AE
base-character-type 178 B2
system-area-pointer-type (header type) 182 B6
unbound-marker 186 BA
weak-pointer-type 190 BE
instance-header-type 194 C2
fdefn-type 198 C6
scavenger-hook-type 202 CA
\end{verbatim}
\section{Strings}
All strings in the system are C-null terminated. This saves copying the bytes
when calling out to C. The only time this wastes memory is when the string
contains a multiple of eight characters, and then the system allocates two more
words (since Lisp objects are dual-word aligned) to hold the C-null byte.
Since the system will make heavy use of C routines for systems calls and
libraries that save reimplementation of higher level operating system
functionality (such as pathname resolution or current directory computation),
saving on copying strings for C should make C call out more efficient.
The length word in a string header, see ``Data-Blocks and Header-Word
Format'', section~\ref{sec:data-blocks-and-header}, counts only the characters truly in the Common Lisp string.
Allocation and GC will have to know to handle the extra C-null byte, and GC
already has to deal with rounding up various objects to dual-word alignment.
\section{Symbols and NIL}
Symbol data-block has the following format:
\begin{verbatim}
-------------------------------------------------------
| 5 (data-block words) | Symbol Type (8 bits) |
-------------------------------------------------------
| Value Descriptor |
-------------------------------------------------------
| Hash Value (x86/amd64/sparc) Unused (other arch.) |
-------------------------------------------------------
| Property List |
-------------------------------------------------------
| Print Name |
-------------------------------------------------------
| Package |
-------------------------------------------------------
\end{verbatim}
All of these slots are self-explanatory given what symbols must do in Common
Lisp.
The issues with nil are that we want it to act like a symbol, and we need list
operations such as CAR and CDR to be fast on it. CMU Common Lisp solves this
by putting nil as the first object in static space, where other global values
reside, so it has a known address in the system:
\begin{verbatim}
------------------------------------------------------- <-- space
| 6 (data-block words) | 0 | start
-------------------------------------------------------
| 0 (data-block words) | Symbol Type (8 bits) |
------------------------------------------------------- <-- nil
| Value/CAR |
-------------------------------------------------------
| Hash Value/CDR |
-------------------------------------------------------
| Property List |
-------------------------------------------------------
| Print Name |
-------------------------------------------------------
| Package |
-------------------------------------------------------
| ... |
-------------------------------------------------------
\end{verbatim}
In addition, we make the list typed pointer to nil actually point past the
header word of the nil symbol data-block. This has usefulness explained below.
The value and hash-value of nil are nil. Therefore, any reference to nil used
as a list has quick list type checking, and CAR and CDR can go right through
the first and second words as if nil were a cons object.
When there is a reference to nil used as a symbol, the system adds offsets to
the address the same as it does for any symbol. This works due to a
combination of nil pointing past the symbol header-word and the chosen list and
other-pointer type tags. The list type tag is four less than the other-pointer
type tag, but nil points four additional bytes into its symbol data-block.
\section{Array Headers}
The array-header data-block has the following format:
\begin{verbatim}
----------------------------------------------------------------
| Header Len (24 bits) = Array Rank +6 | Array Type (8 bits) |
----------------------------------------------------------------
| Fill Pointer (30 bits) | 0 0 |
----------------------------------------------------------------
| Fill Pointer p (29 bits) -- t or nil | 1 1 1 |
----------------------------------------------------------------
| Available Elements (30 bits) | 0 0 |
----------------------------------------------------------------
| Data Vector (29 bits) | 1 1 1 |
----------------------------------------------------------------
| Displacement (30 bits) | 0 0 |
----------------------------------------------------------------
| Displacedp (29 bits) -- t or nil | 1 1 1 |
----------------------------------------------------------------
| Range of First Index (30 bits) | 0 0 |
----------------------------------------------------------------
.
.
.
\end{verbatim}
The array type in the header-word is one of the eight-bit patterns from
``Data-Blocks and Other-immediates Typing'', section~\ref{sec:data-blocks-and-header}, indicating that this is a complex
string, complex vector, complex bit-vector, or a multi-dimensional array. The
data portion of the other-immediate word is the length of the array header
data-block. Due to its format, its length is always six greater than the
array's number of dimensions. The following words have the following
interpretations and types:
\begin{description}
\item[Fill Pointer:]
This is a fixnum indicating the number of elements in the data vector
actually in use. This is the logical length of the array, and it is
typically the same value as the next slot. This is the second word, so
LENGTH of any array, with or without an array header, is just four bytes
off the pointer to it.
\item[Fill Pointer P:]
This is either T or NIL and indicates whether the array uses the
fill-pointer or not.
\item[Available Elements:]
This is a fixnum indicating the number of elements for which there is
space in the data vector. This is greater than or equal to the logical
length of the array when it is a vector having a fill pointer.
\item[Data Vector:]
This is a pointer descriptor referencing the actual data of the array.
This a data-block whose first word is a header-word with an array type as
described in ``Data-Blocks and Header-Word Format'', section~\ref{sec:data-blocks-and-header} and
``Data-Blocks and Other-immediates Typing'', section~\ref{sec:data-blocks-and-o-i}
\item[Displacement:]
This is a fixnum added to the computed row-major index for any array.
This is typically zero.
\item[Displacedp:]
This is either t or nil. This is separate from the displacement slot, so
most array accesses can simply add in the displacement slot. The rare
need to know if an array is displaced costs one extra word in array
headers which probably aren't very frequent anyway.
\item[Range of First Index:]
This is a fixnum indicating the number of elements in the first dimension
of the array. Legal index values are zero to one less than this number
inclusively. IF the array is zero-dimensional, this slot is
non-existent.
\item[... (remaining slots):]
There is an additional slot in the header for each dimension of the
array. These are the same as the Range of First Index slot.
\end{description}
\section{Bignums}
Bignum data-blocks have the following format:
\begin{verbatim}
-------------------------------------------------------
| Length (24 bits) | Bignum Type (8 bits) |
-------------------------------------------------------
| least significant bits |
-------------------------------------------------------
.
.
.
\end{verbatim}
The elements contain the two's complement representation of the integer with
the least significant bits in the first element or closer to the header. The
sign information is in the high end of the last element.
\section{Code Data-Blocks}
A code data-block is the run-time representation of a ``component''. A component
is a connected portion of a program's flow graph that is compiled as a single
unit, and it contains code for many functions. Some of these functions are
callable from outside of the component, and these are termed ``entry points''.
Each entry point has an associated user-visible function data-block (of type
{\tt function}). The full call convention provides for calling an entry point
specified by a function object.
Although all of the function data-blocks for a component's entry points appear
to the user as distinct objects, the system keeps all of the code in a single
code data-block. The user-visible function object is actually a pointer into
the middle of a code data-block. This allows any control transfer within a
component to be done using a relative branch.
Besides a function object, there are other kinds of references into the middle
of a code data-block. Control transfer into a function also occurs at the
return-PC for a call. The system represents a return-PC somewhat similarly to
a function, so GC can also recognize a return-PC as a reference to a code
data-block. This representation is known as a Lisp Return Address (LRA).
It is incorrect to think of a code data-block as a concatenation of ``function
data-blocks''. Code for a function is not emitted in any particular order with
respect to that function's function-header (if any). The code following a
function-header may only be a branch to some other location where the
function's ``real'' definition is.
The following are the three kinds of pointers to code data-blocks:
\begin{description}
\item[Code pointer (labeled A below):]
A code pointer is a descriptor, with other-pointer low-tag bits, pointing
to the beginning of the code data-block. The code pointer for the
currently running function is always kept in a register (CODE). In
addition to allowing loading of non-immediate constants, this also serves
to represent the currently running function to the debugger.
\item[LRA (labeled B below):]
The LRA is a descriptor, with other-pointer low-tag bits, pointing
to a location for a function call. Note that this location contains no
descriptors other than the one word of immediate data, so GC can treat
LRA locations the same as instructions.
\item[Function (labeled C below):]
A function is a descriptor, with function low-tag bits, that is user
callable. When a function header is referenced from a closure or from
the function header's self-pointer, the pointer has other-pointer low-tag
bits, instead of function low-tag bits. This ensures that the internal
function data-block associated with a closure appears to be uncallable
(although users should never see such an object anyway).
Information about functions that is only useful for entry points is kept
in some descriptors following the function's self-pointer descriptor.
All of these together with the function's header-word are known as the
``function header''. GC must be able to locate the function header. We
provide for this by chaining together the function headers in a NIL
terminated list kept in a known slot in the code data-block.
\end{description}
A code data-block has the following format:
\begin{verbatim}
A -->
****************************************************************
| Header-Word count (24 bits) | Code-Type (8 bits) |
----------------------------------------------------------------
| Number of code words (fixnum tag) |
----------------------------------------------------------------
| Pointer to first function header (other-pointer tag) |
----------------------------------------------------------------
| Debug information (structure tag) |
----------------------------------------------------------------
| First constant (a descriptor) |
----------------------------------------------------------------
| ... |
----------------------------------------------------------------
| Last constant (and last word of code header) |
----------------------------------------------------------------
| Some instructions (non-descriptor) |
----------------------------------------------------------------
| (pad to dual-word boundary if necessary) |
B -->
****************************************************************
| Word offset from code header (24) | Return-PC-Type (8) |
----------------------------------------------------------------
| First instruction after return |
----------------------------------------------------------------
| ... more code and LRA header-words |
----------------------------------------------------------------
| (pad to dual-word boundary if necessary) |
C -->
****************************************************************
| Offset from code header (24) | Function-Header-Type (8) |
----------------------------------------------------------------
| x86/amd64/sparc: Address of start of instructions for |
| function (non-descriptor) |
| other architectures: |
| Self-pointer back to previous word (with other-pointer tag) |
----------------------------------------------------------------
| Pointer to next function (other-pointer low-tag) or NIL |
----------------------------------------------------------------
| Function name (a string or a symbol) |
----------------------------------------------------------------
| Function debug arglist (a string) |
----------------------------------------------------------------
| Function type (a list-style function type specifier) |
----------------------------------------------------------------
| Start of instructions for function (non-descriptor) |
----------------------------------------------------------------
| More function headers and instructions and return PCs, |
| until we reach the total size of header-words + code |
| words. |
----------------------------------------------------------------
\end{verbatim}
The following are detailed slot descriptions:
\begin{description}
\item[Code data-block header-word:]
The immediate data in the code data-block's header-word is the number of
leading descriptors in the code data-block, the fixed overhead words plus
the number of constants. The first non-descriptor word, some code,
appears at this word offset from the header.
\item[Number of code words:]
The total number of non-header-words in the code data-block. The total
word size of the code data-block is the sum of this slot and the
immediate header-word data of the previous slot.
header-word.
\item[Pointer to first function header:]
A NIL-terminated list of the function headers for all entry points to
this component.
\item[Debug information:]
The DEBUG-INFO structure describing this component. All information that
the debugger wants to get from a running function is kept in this
structure. Since there are many functions, the current PC is used to
locate the appropriate debug information. The system keeps the debug
information separate from the function data-block, since the currently
running function may not be an entry point. There is no way to recover
the function object for the currently running function, since this
data-block may not exist.
\item[First constant ... last constant:]
These are the constants referenced by the component, if there are any.
\vspace{1ex}
\item[LRA header word:]
The immediate header-word data is the word offset from the enclosing code
data-block's header-word to this word. This allows GC and the debugger
to easily recover the code data-block from an LRA. The code at the
return point restores the current code pointer using a subtract immediate
of the offset, which is known at compile time.
\vspace{1ex}
\item[Function entry point header-word:]
The immediate header-word data is the word offset from the enclosing code
data-block's header-word to this word. This is the same as for the
return-PC header-word.
\item[Address of start of instructions for function:] This is
implemented on x86, amd64, and sparc only. In a non-closure
function, this address allows the call sequence to always
indirect through the second word in a user callable function.
See section ``Closure Format''. With a closure, indirecting
through the second word also gets you the start of instructions
of a function. This pointer is a raw address, not a descriptor.
\item[Self-pointer back to header-word:]
In a non-closure function, this self-pointer to the previous header-word
allows the call sequence to always indirect through the second word in a
user callable function. See section ``Closure Format''. With a closure,
indirecting through the second word gets you a function header-word. The
system ignores this slot in the function header for a closure, since it
has already indirected once, and this slot could be some random thing
that causes an error if you jump to it. This pointer has an
other-pointer tag instead of a function pointer tag, indicating it is not
a user callable Lisp object.
\item[Pointer to next function:]
This is the next link in the thread of entry point functions found in
this component. This value is NIL when the current header is the last
entry point in the component.
\item[Function name:]
This function's name (for printing). If the user defined this function
with DEFUN, then this is the defined symbol, otherwise it is a
descriptive string.
\item[Function debug arglist:]
A printed string representing the function's argument list, for human
readability. If it is a macroexpansion function, then this is the
original DEFMACRO arglist, not the actual expander function arglist.
\item[Function type:]
A list-style function type specifier representing the argument signature
and return types for this function. For example,
\begin{verbatim}
(function (fixnum fixnum fixnum) fixnum)
\end{verbatim}
or
\begin{verbatim}
(function (string &key (:start unsigned-byte)) string)
\end{verbatim}
This information is intended for machine readablilty, such as by the
compiler.
\end{description}
\section{Closure Format}
A closure data-block has the following format:
\begin{verbatim}
----------------------------------------------------------------
| Word size (24 bits) | Closure-Type (8 bits) |
----------------------------------------------------------------
| Pointer to function header (other-pointer low-tag) |
----------------------------------------------------------------
| . |
| Environment information |
| . |
----------------------------------------------------------------
\end{verbatim}
A closure descriptor has function low-tag bits. This means that a descriptor
with function low-tag bits may point to either a function header or to a
closure. The idea is that any callable Lisp object has function low-tag bits.
Insofar as call is concerned, we make the format of closures and non-closure
functions compatible. This is the reason for the self-pointer in a function
header. Whenever you have a callable object, you just jump through the second
word, offset some bytes, and go.
\section{Function call}
Due to alignment requirements and low-tag codes, it is not possible to use a
hardware call instruction to compute the LRA. Instead the LRA
for a call is computed by doing an add-immediate to the start of the code
data-block.
An advantage of using a single data-block to represent both the descriptor and
non-descriptor parts of a function is that both can be represented by a
single pointer. This reduces the number of memory accesses that have to be
done in a full call. For example, since the constant pool is implicit in an
LRA, a call need only save the LRA, rather than saving both the
return PC and the constant pool.
\section{Memory Layout}
\cmucl{} has four spaces, read-only, static, dynamic-0, and dynamic-1.
Read-only contains objects that the system never modifies, moves, or reclaims.
Static space contains some global objects necessary for the system's runtime or
performance (since they are located at a known offset at a known address), and
the system never moves or reclaims these. However, GC does need to scan static
space for references to moved objects. Dynamic-0 and dynamic-1 are the two
heap areas for stop-and-copy GC algorithms.
What global objects are at the head of static space???
\begin{verbatim}
NIL
eval::*top-of-stack*
lisp::*current-catch-block*
lisp::*current-unwind-protect*
FLAGS (RT only)
BSP (RT only)
HEAP (RT only)
\end{verbatim}
In addition to the above spaces, the system has a control stack, binding stack,
and a number stack. The binding stack contains pairs of descriptors, a symbol
and its previous value. The number stack is the same as the C stack, and the
system uses it for non-Lisp objects such as raw system pointers, saving
non-Lisp registers, parts of bignum computations, etc.
\section{System Pointers}
The system pointers reference raw allocated memory, data returned by foreign
function calls, etc. The system uses these when you need a pointer to a
non-Lisp block of memory, using an other-pointer. This provides the greatest
flexibility by relieving contraints placed by having more direct references
that require descriptor type tags.
A system area pointer data-block has the following format:
\begin{verbatim}
-------------------------------------------------------
| 1 (data-block words) | SAP Type (8 bits) |
-------------------------------------------------------
| system area pointer |
-------------------------------------------------------
\end{verbatim}
``SAP'' means ``system area pointer'', and much of our code contains this naming
scheme. We don't currently restrict system pointers to one area of memory, but
if they do point onto the heap, it is up to the user to prevent being screwed
by GC or whatever.
\section{Weak Pointers}
\label{sec:weak-pointers}
A weak-pointer data-block has the following format:
\begin{verbatim}
-------------------------------------------------------
| 4 (data-block words) | Weak pointer Type (8 bits) |
-------------------------------------------------------
| weak-pointer-value |
-------------------------------------------------------
| weak-pointer-broken |
-------------------------------------------------------
| mark-bit (T or NIL) |
-------------------------------------------------------
| next |
-------------------------------------------------------
\end{verbatim}
The mark-bit is used when gencgc is available. It's used to note if
this weak pointer has been visited before so that scavenging
weak-pointers isn't an $O(n^2)$ process.
The last slot is an internal slot used by the C runtime to chain all
the weak pointers together for GC.

120
doc/internals/cmu/outline.txt Normal file
View file

@ -0,0 +1,120 @@
Todo:
fasl.tex
In good shape.
object.tex
Fairly good, but should probably be integrated with description of primitives
in vm.tex.
front.tex
Needs updating cleanup scan. Not too bad.
middle.tex
Need VMR overview. New names for GTN/LTN? Needs general cleanup, but not too
bad. NLX and stack are the worst.
back.tex
Pack and assembler need more info. General cleanup.
compiler-overview.tex
Adapt introductory material from /../fred/usr/ram/comp.mss, pap:talk.mss
Division between ICR overview and ICR convert needs work.
debugger.tex
Needs much work. Merge much info from debug-info and debug-int. Duplicating a
fair amount of stuff in the source may make sense where, since this is a part
of the system that is generally interesting. And also, a part that people
building on CMU CL might want to understand.
glossary.tex
Finish, integrate w/ main text?
interpreter.tex
Very sketchy and tentative. Needs to be fleshed out from the code.
retargeting.tex
Very rough. Needs to be merged with parts of vm.tex (call vops). Needs some
additional text. Documentation of assembler, and all other exported
interfaces. (Generate defined VOP descriptions from the core, keyed to files?)
vm.tex
This file should probably cease to exist, going into object, retargeting and
introductory material. [Also other scrap in stuff/]
[VMR and ICR overview also needed...]
architecture.tex
Missing sections on startup code, compiling, building.
environment.tex
Needs to be written: type system and info database interfaces.
interface.tex
Needs to be written: source paths and error message utilities.
lowlev.tex
Needs to be written. All manner of low-level stuff: memory layout and
management, core file format, C interface, low-level debugging (and ldb.)
Several different audiences:
-- Curious compiler implementors (not a big priority. Downplay academic
aspects, i.e. comparisons to other techniques, analysis of limitations,
future work...) Compiler part can be more academic, and include some
justifications of other design decisions.
-- System maintainers.
-- People retargeting the compiler.
-- People bringing up the system in a new environment.
Sys arch part:
Package + file structure [system.txt]
system building [compiling.txt]
bootstrapping & cross compiling
Compiler design:
Overview (mirror structure of rest of the part)
ICR data structure
Front end [front.tex]
Basic VMR data structures (no back-end stuff)
Middle end [middle.tex]
Back end + data structures [back.tex]
Error system interface
Source tracking
Compiler retargeting:
VM definition concepts [porting.txt, mail.txt, retargeting.tex]
SCs, SBs, primitive-types
Defining VOPS
time specification
defining
and using the assembler
Required VOPs [internal.txt, lowlev.txt, vm.mss]
Standard primitives [vm.mss] (broken down by type, parallels object format
section structure.)
Customizing VMR conversion
multiple hardware
constant operands
VM specific transforms
special-case IR2 convert methods
Run-time system:
type system
info database
Data format [object.tex]
Debugger:
Info format [debug.txt]
Stack parsing [debug.txt]
Breakpoints
Internal errors
Signals
Memory management: [William]
heap Layout
stacks
GC
misc implementation stuff: foreign call, assembly routines [lowlev.txt]
LDB and low-level debugging
core file format [William]
fasl format [fasl.tex]

1113
doc/internals/cmu/retargeting.tex Normal file
View file

File diff suppressed because it is too large Load diff

8
doc/internals/cmu/run-time.tex Normal file
View file

@ -0,0 +1,8 @@
\part{Run-Time System}
\input{environment}
\input{interpreter}
\input{debugger}
\input{object}
\input{lowlev}
\input{fasl}

1460
doc/internals/cmu/vm.tex Normal file
View file

File diff suppressed because it is too large Load diff