This is work-in-progress, however it does usually complete self-build
on Linux (less so on macOS) and can run some of the regression suite.
This commit is mainly for other developers to view the state.
Using the "buildit" script which is for now the suggested way to build,
make-target-2 will show (in lines prefixed with "Stack scan") pause times
as low as 5 microseconds, or up to maybe 100 microseconds on the high end.
Anything having to do with thread start/exit is potentially broken,
and there are some obviously missing pieces which are denoted by purposely-
inserted lose() calls.
* Simplify scan_finalizers
- Don't bother with weak pointers on keys in need of rehash. They'll soon
get popped and re-inserted into the weak split-ordered list anyway.
- Exploit two-phase deletion by performing a bit flip in C,
and allow Lisp to repair the list on next table operation.
* Acquire the mutex in FINALIZERS-REHASH.
* Object that die in final GC can have their finalizer run on restart.
There were/are limitations with the mechanism, e.g. compiling "file_a"
which could have benefited from a compiler-macro in "file_b", then
quitting and retarting lisp before compiling "file_b" should somehow warn.
This is particularly a problem for sandboxed builds where each
build executor compiles 1 file.
To do it, info about missed optimizations has to be part of the fasl.
I was able to get either the existing regression tests to pass, or the
warnings from genesis to be right, but not both. I left it where
the tests pass, but I gave up for the moment on doing it right.
If nothing else, this rectifies the poor decision of using globaldb for
the compiler's dynamic data. It meant that COMPILE on the simplest of
functions could touch a ton of GC cards for no particularly good reason,
as each mentioned symbol might accrete a new PACKED-INFO instance.
This replaces the usual hash-table with a new storage representation
that is almost always lockfree, and removes all complexity around
assigning a unique ID to each finalized object.
The ugly hack in cull_weak_hash_table_bucket() is no more.
As the benchmark shows, with 4 threads we can create finalizers
about 3x faster at the cost of about 1.5x more memory.
Fixes lp#1998064 where finalizers are concerned, but if the bug
exists more generally, this doesn't do anything for it.
Internal symbols of SB-IMPL are supposed to be potential garbage.
But XC-SANITY-CHECKS survived final GC with no path to it.
So are the tree-shaker and search-roots BOTH wrong?
* *wp* -> #<weak pointer: #:XC-SANITY-CHECKS>
* (sb-ext:search-roots *wp* :criterion :static) -> nothing
This change resulted in a huge time saving in INTERN-PATHNAME. Tested on
the same operations, the former hash algorithm required as many as 120 probes
worst case, where this hashset needed at most 8 probes, with the mean
being between 2 and 3 probes.
Also, it's threadsafe for reading (but not truly concurrent- just not
unsafe), so it does not use a mutex for lookups except if not found,
in which case it retries with a lock and possibly insertion.
The memory use is typically under half that of a hash-table having the
same keys, and it also measurably outperforms a weak hash-table.
where N=3 or 4, I'm not certain.
This at long last makes MAKE-PATHNAME non-consy for EQUAL pathnames
and makes pathname-directory EQ-comparable by avoiding evicting them ever.
And yet we do quite well at not retainining excess garbage now.
With no brute-force clobbering in save-lisp-and-die, the restarted pristine
core has only about a dozen cache entries, and the cache vectors are
the original size. The remark about misuse of MAKE-LOAD-FORM-SAVING-SLOTS
was obsolete- apparently it was fixed a while back to use INTERN-PATHAME.
Also fix a problem that there could be more than one logical host
named by the empty string, one coming from src/code/class's prototype-form
which was really ugly considering that logical-hosts are unique atoms.
Put a vector of entrypoints in static space, use "call [ea]" form
just like for asm calls, allowing text space to be relocated anywhere
without needing to reapply those fixups if the space undergoes
relocation on startup.
Make a vector of entrypoints in static space so that dynamic-space code can
use "CALL [ea]". We don't need to retain the fixup for reapplication if the
text space moves on core start. Removes some complexity from rev 7d84b420e0
("create fewer absolute fixups") but keeps the level of indirection for
allocation profiler fixups, because that logic is finicky and needs the CALL
to encode in 5 bytes. This gets a little nearer to being able to relocate
text space to any address, not just sub-2GB.
* Remove the optional args. Always enforce that the thing returned
is not a macro or special-operator guard trampoline.
* And now that symbols directly store a FDEFN there's nothing complicated
about SYMBOL-FUNCTION, so generally pick that off as a special case.
And with a few related improvements:
* a finalizer gives memory back to the allocator
* code never acts as filler
* easier way of tracking text page scan start offset
Never JIT-compile a writer for a readonly slot. Instead when macroexpanding
DEFSTRUCT, always create anonymized incarnations of each slot accessor
and hand those to the CLOS class installer. Thusly all rigmarole about whether
there is a pair of globally BOUNDP function names per slot is irrelevant.
* Use a dedicated FOP to dump named constant references. We used to
dump a FOP-FUNCALL, but we need to do something more exotic to support
forward references in both cold load and the target. The backpatching
is crucially needed to make cold load work, but it also allows us to
actually install constants into code components at load time via
%DEFCONSTANT, which solves the issues with DEFCONSTANT and block
compilation mixing. See the test case, which has now been fixed.
* Loading a dedicated FOP is faster than loading a dumped load time
lambda, not to mention smaller, but we are actually mostly using it to
help with the backpatching info. As a side effect, we can get rid of
all the bookkeeping for named constants in the dumper, since we now do
it by name. We can't even cache the result in the dumper because we
need to record backpatches into different code components.
* We get rid of the extremely ugly *HAIRY-DEFCONSTANTS* hack that was
only used to minimize the cost of block compiling DEFCONSTANTs and
reduce the amount of non-EQness of DEFCONSTANT in that context.
* Define a new info type to keep track of the forward references. For
the most common scenario, the info database isn't hit at all, since we
do things normally when the symbol is bound at the point of fasl load.
* A sort-of weird but justifiable side effect of this change: If you
compile
(defun f () foo)
where foo has been declared a constant, and load the fasl without
defining foo, (f) returns <unbound>, rather than throwing a load time
error. This matches DEFGLOBAL or ALWAYS-BOUND behavior, which is
totally acceptable and conforming for DEFCONSTANT, since DEFCONSTANT
is a promise that the constant will actually be defined and bound to
its real value in a reconstructed image via loading.
* Change LVAR-VALUE to return the LEAF as well, so that there's a
better chance that we dump a named constant as a named constant.
* Enable using DEFCONSTANT-EQX in target only files with arbitrary
init-forms (i.e. they don't need to be literals), by no longer
reflecting host values into the core with specialized genesis
machinery.
* Anonymize constants that are needed really early. Because usually
said constants are defined in the same file, there is no additional
space usage as the file compiler coalesces EQ literal constants. This
is a good tradeoff for being able to bootstrap. After all, we
anonymized the value forms of pretty much all complex init forms for
DEFCONSTANTs. Now we can deanonymize the initforms of DEFCONSTANTs, as
many of them had to be anonymized even when there was no bootstrapping
problem present.
We no longer need to intercept top level form processing specially in
order to cross-compile, so the normal compiler can always do its thing
instead.
There is no logical association between the execution order of top
level forms and the order in which the system should run cold init
functions before that.
Now that we print the filename after loading each file, there's no
need to print mod 1000 where we are in the load. Also,
making *!cold-toplevels* unbound here was useless, since it just gets
set to NIL right after.
Type system bootstrapping has been sufficiently worked out since 2004
so that now *QUEUED-PROCLAIMS* is always NIL on the host. Therefore,
enforce the constraint that we never proclaim any types before the
type system is initialized, thereby allowing us to simplify deferred
proclamations so we don't have to do any of this on the host.
Specifying when the checks happen explicitly in cold init makes it
more effective and also more robust.
Also, doing (eval-when (:compile-toplevel) ...) instead of making the
action happen explicitly on the host is a bit weird. Cross compiled
files should have no conception of the cold loader anyway.
There are a few interesting things you could do with this integer:
1. implement a novel "flat" lookup mechanism where all symbols in the entire
image are in a single (multivalued) hash-table keyed by string, each symbol
being associated with a bitmask over packages IDs that can access that symbol.
Generally speaking, a successful hit on a symbol would affirm that you have
found the right one by checking if its mask contains the current package ID.
This would optimize for packages that have USE-PACKAGEd a dozen or more other
packages. I haven't though it through with nicknames, just spitballing it.
2. pass around packages as immediate values (for whatever that's worth)
3. store the small integer instead of a pointer in symbols.
I plan to implement the 3rd idea in a subsequent patch. It represents
a nice space savings for 64-bit word size.
This commit is composed of two main almost-independent-but-not-quite
changes.
1. Do not reflect host CTYPEs into cold load for bootstrap.
03606c9205 wanted to remove the OAOO
violating idiom of
(defglobal xxx -1)
(!cold-init-forms
(setq xxx ...))
in the type system and also remove a META-INFO hack. The way it
accomplished this was by teaching genesis how to reflect host CTYPEs
into the target, which requires genesis to know both about how CTYPEs
are internally represented and also causes complications with making
sure the interned types at cold load time are sane with respect to
cold init time type initializations.
However, recent build order wrangling (merging most of the type system
back into one file, for example) allows us to solve this in a much
simpler way: We introduce a macro !COLD-INIT-TIME-VALUE to avoid
having to write a bunch of once-use globals and the fact that most of
the type system initialization happens in one file makes ordering the
initialization of specifier types and ctypes pretty simple. As shown
by the line diff count, this simplifies cold load quite a bit, and
removes hacks for specifier-type to directly evaluate into a constant
just so that the dumper can hand off the value to genesis and removes
the whole LITERAL-CTYPE-... stuff.
Also, deduplicate some logic in array.lisp with respect to fetching
element types from widetags in the process, avoiding some dumping of
host CTYPEs at the same time.
2. Do not use coordination between the fopcompiler (essentially a
top-level form special caser/interceptor) and Genesis to set
hairy DEFCONSTANT values during the cold load.
This also means that we need to go back to dumping named constant
references as essentially (load-time-value (symbol-global-value
+constant+)).
The rationale behind this is that it is becoming clear that special
casing toplevel forms hinders using the normal compiler, such as for
block compilation or a future CMU CL style auto top level form
smasher. It also has to be done to enable the above CTYPE change,
since having DEFCONSTANT symbol values be set at cold load time means
that the value forms need to be cold loadable. It also has the same
issue of relying on host value reflection, which is not super clean
considering we loses compile time side effects, so constants defined
in target-only code can't use this technique anyway. Therefore, it is
simpler to just rely on normal compiler machinery to do DEFCONSTANT
bootstrapping. I am also hoping to solve EQness of constants not being
preserved by block compilation, which this is going to help, since
fasl ops can't get ordered with smashed up top level forms at all,
while it may be possible to do something with normal load-time-value
lambdas.
Of course, there are downsides, as now certain constant values are no
longer available until top-level forms get run. Usually, it's not much
of a problem: We add extra #. to anonymize those constants that need
it, which is compensated by the removal of many #. which were forced
by the previous DEFCONSTANT-EQX. We change BLT-COPIER-FOR-WIDETAG to
not even use any global constant at all by localizing the value into a
closure.
Instead of creating a function pointer table simulating a jump table
which then gets funcalled, do the more pedestrian conditional
dispatch. This allows block compilation, getting the advantages of
let-conversion (inlining essentially) and local call. The savings from
saving a few (predicatable) conditional branches in favor of array
dispatch is easily dwarfed by the removal of full calls where the
function argument isn't even constant in favor of local calls which
are essentially free, having no argument parsing or indirection to
speak of. Most of these functions actually get let-converted away
entirely, as they aren't used anywhere else, so that's as if the body
of the function were written inline. Only two `output-foo-symbol
function` actually gets genuinely local called as its called in more
than one place.
This speeds up printing symbols measurably. Printing 'Z 20000 times
takes 10% less cycles and 10% less wall time.
This also allows us to get rid of the C-like enum in favor of the more
traditional Lisp keyword symbols, as well as simplifying cold-init a
bit.
Commit 303853a624 introduced a cold init
function and an obfuscating idiom to work around the limitation that
non-top-level DEFUNs were not handled properly by genesis. This
limitation has been removed completely with the reintroduction of CMU
CL style dumping of FOP-FSETs which replaced fopcompiler DEFUN linkage
machinery. Therefore, we can remove the obfuscation and replace the
cold-init time linkage with the more rebust cold load linkage while
still getting local unboxed calling convention goodness.
This reverses the change that made these initialized by cold load. For
reasons explained previously (see commit removing
!DEFINE-LOAD-TIME-GLOBAL), it is necessary for some features that we
not rely on genesis to set the values of certain variables. However,
I've tried to organize the cold initializations a bit better. Also,
I'm going to try to mark cold initializations which are technically
not necessary for bootstrap but good for debugging with #+sb-devel. Hopefully, that will make cold init seem less cluttered.
Unfortunately there was no nice way to do this without creating a new
cold-init function. However, now the depedency is clear and we can get
FORMAT working all at once.
* Don't modify a string literal.
* Package up the random SETQs into a cohesive cold init function.
* Remove !DEFINE-LOAD-TIME-GLOBAL while we're at it.
