Because GC scavenging is more finicky now that it does not necessarily
proceed strictly object-by-object, non-zeroing could have caused
inadvertent floating garbage at best, or a crash at worst.
When I tried implementing non-prezeroed unboxed pages, the first things
to break were hash-tables because I totally failed to remove the assumption
of 0-fill in the array allocator. The hash-table code is correct as-is,
but might as well keep the checker as an amulet to ward off evil.
* Anticipate the division of generation 0 into two kinds of pages-
those allocated by threads and those allocated by the collector.
The latter are ok to scan from Lisp because the collector updates
page_bytes_used() whenever it runs, whereas TLAB pages have
some concurrency problems that are not easily resolved.
(Hence the occasional crash in sb-sprof trying to find a code blob)
Also, we can more easily implement different policies on whether to
pre-zero generation 0 for the mutators versus collector.
* Allow for writing to a stdio stream or a file descriptor or both, so
this can be called from Lisp. Avoiding stdio streams in the collector
is preferable because we've seen many reports that enabling GC logging
may cause lockup. I've found snprintf() and write() to be more reliable
in that regard even though the entire printf() family is absent
from https://man7.org/linux/man-pages/man7/signal-safety.7.html
In general any cons with most-positive-word in the car.
I do like the idea of using -1 because it is unambiguously a filler
where (0 . 0) is possibly a real cons.
This test had a tendency to prevent the non-GC worker threads from running,
and/or cause heap exhaustion. Sometimes it would report thousands of GCs instead
of the expected couple of dozen or so. And ironically, tests that hammer on
GC rather than let it auto-trigger can cause garbage to promote from gen0
to gen1 on each cycle filling up gen1 by accident. The auto-trigger isn't
able to compensate for this unusual use-case by triggering in time.
This was a very misleading change. This does not actually allow one to
block compile multiple files, and this approach can't work to actually
do so. The only thing it does is essentially the equivalent of
concatenating fasl files, which third party tools like ASDF can
already do. None of the state variables for compilation like ir1
namespacing get shared, which is crucial for block compilation.
We probably want to do multi-file block compilation through
WITH-COMPILATION-UNIT, or if we are going to do the "different
signature" route, we should go back to having source infos handle
multiple files, like CMU CL, as that is more efficient and more
general, because it actually treats multiple files as one thing to
read forms out of.
By pushing around the scoping of some special variables. Bonus: We
cons less by not making a new hash table for every top level form,
instead reusing the *SOURCE-PATHS* hash table. (I understand the
intent of wanting to cons fresh objects instead of reusing them, but a
new hash table for every top level form is not nothing in terms of
cost.)
The parent commit added functionality to allow the normal compiler to
produce smaller fasls than the fopcompiler in many cases, without a
real compile-time or load-time tradeoff. Therefore, remove the
fopcompiler, as it adds a lot of complexity and needs to be turned off
on high debug or instrumentation. As a bonus, the fopcode space is
freed up a bit.
It also suffers from multiple macroexpansion. Some things this allows
us to simplify:
* No need to have LOAD-FORM-IS-DEFAULT-MLFSS-P be used in the
compiler. The way we used it was sketchy anyway.
* The compiler and loader no longer need to keep track of skipping on
the FOP stack, or whether LOAD-TIME-VALUE needs it to be done or not.
For top level forms we now smash top level lambdas into one component,
as if we were block compiling. We do it automatically 20 tlls at a
time, which significantly reduces the number of components that need
to get loaded and called at load time, reducing fasl size. Unlike CMU
CL, we've solved the problems of package environment interaction, so
that the optimization is semantically sound.
I also found a solution for recursive IR1 namespacing with respect to
forcing top level lambda smashing before emitting load time value and
constant junk, as it was causing problems with constant infos not
getting cleaned up properly. CMU CL also ran into this issue, but took
one step forward and one step back by removing a force call and adding
another one. Fix this by doing both force calls.
As a representative of compiling a large file, compiling
"src/compiler/node.lisp"
- with no tlf smashing: produces a fasl of 940039 bytes.
- with fop compilation: produces a fasl of 830003 bytes.
- with tlf smashing 20 forms at a time: produces a fasl of 752591 bytes.
Compiling ironclad shows no difference in speed. Loading the resulting
fasls also shows no measurable difference in speed when TIMEd.
Since we use the normal compiler, we don't sacrifice any debuggability
or coverage instrumentation, as well as avoiding issues with multiple
macroexpansion. This also opens up the possibility of using a real
byte-compiler in conjunction with tlf smashing to squeeze out even
more bytes out of fasls for loading top level code. The ir1-based cmu
cl byte compiler, for example, gets 6x space savings over (risc)
machine code on average (cf. cmu documentation). Note from the above
that fop-based byte compilation does not even produce more compact
code than machine code once top level lambdas are merged.
The fopcompiler will get removed in a subsequent change.
When block compiling this file, using unquoting rather than read eval
caused some unintuitive list* fops to get dumped on sbcl host, whereas
ccl host wouldn't have used a list* fop. With the reader eval, sbcl
dumps the list structure similarly to CCL. It's so weird that we have
a host sensitivity here, since the dump logic should be host agnostic
at this point, but I suspect it has something to do with how this list
is literally spliced in into the next DEFTRANSFORM, causing
coalescing, and maybe because it was spliced into a host only function
in the same file. ?????????????
When delaying toplevel form execution until after component loading in
the loader, shadowing would not effect the package environment
correctly during fasloading. Fix this by recording whether inheritance
for the symbol is allowed at the point of dumping. We are allowed to
do this because the specification says that in conforming code, the
package environment during compilation should be congruent to the
package environment during fasload.
This is the most expedient fix, but this could probably be done better
in a way that unifies the accessibility logic in cold load.
We share the logic for deprecation warnings between compile file and
fasload, so that the same exact conditions are signalled when
fasloading as during compilation. This also significantly simplifies
the loader, as well as making it less flaky with respect to trying to
literally grovel the fop stack for a hardcoded pattern which doesn't
manifest at all on higher debug or coverage
instrumentation (specifically relying on fopcompilation to happen).
By reusing %REDEFINE-DEFSTRUCT. This makes loading incompatible
defstructs less brittle when it comes to what order top-level forms in
compiled files get run with respect to literal layout loads.
Check consistency doesn't work here on high debug. In addition, the
original test case didn't call for it anyway. The reason it doesn't
work is that signalling a compiler error while compiling breaks the
consistency of the IR. That's just how it is.
The usual never-ending battle to figure out how to receive the fewest
warnings on the most machines. So today on the machine that I tried,
os_context_register_t has to be unsigned int, not unsigned long,
which is just as likely wrong for some other libc release.
This essentially restores the state of things prior to rev d2ca90b6
but with the advantage of ahead-of-time compiling the anonymous
(and oxymoronic) writer functions for readonly slots.
