Scripts using COMPILE or LOAD, either directly or through third party systems,
pollute the standard output with messages unrelated to the script itself.
This is undesirable (to quote a comment in toplevel.lisp, "scripts don't need
to be stylish or fast, but silence is usually a desirable quality") and can
cause bugs when the output of a script is processed through a UNIX pipe by a
program expecting to get output and not verbose diagnostic messages.
Problem originally signaled by Hraban Luyat <hraban@0brg.net>.
On x86, the stack guard page could get missed when allocating large
objects on the stack (over 1 page). Instead of giving up or only
allowing this on low safety, just check explicitly whether an overflow
could happen and signal an appropriate condition.
Other platforms don't need this overflow check as urgently, since they
do zero-filling which ensures that they will in fact hit the guard
page eventually before the stack is manipulated again. (Although there
may not be any space left for the handler to do its thing. In any
case, a similar check could be added.)
We can remove the TRULY-DYNAMIC-EXTENT declaration now, since its only
purpose was to facilitate the old way of just avoiding allocations
that could silently overflow the stack. If we really want to skip the
overflow check during self-build always, that could just be achieved
via a special variable. No need to have an extra "internal-only"
declaration to do that. This simplifies frontend processing of dynamic
extent.
Add a test exhibiting large stack allocation interacting with the
limited stack size.
The spec says that *all* values that a dynamic-extent declared
variable takes on are dynamic extent and hence can be stack allocated,
not just the initial variable binding. That is, SETQ value forms
should also participate in stack allocation when possible if the
SETQ'd var is declared dynamic extent.
Special care needs to be taken when the dynamic extent declaration is
in a local function which is called by multiple local calls, since
then it's not obvious where to put the cleanup code. In fact no
cleanup code should be inserted at all, as function return will clean
the stack. So teach the dynamic extent representation how to represent
that. The work for overhauling dynamic extent representation in the IR
in general and work on stack analysis is paying off now: it was
basically impossible to do this in the older IR way of handling
dynamic extent, but now it's fairly easy with a dedicated
DYNAMIC-EXTENT node. And since stack analysis has been cleaned up with
many restrictions lifted, it doesn't care about the exact dataflow
related to lvars with dynamic extent and SETQs in the scope of a
dynamic extent cleanup don't bother the stack-pointer-initializing
walk through the graph.
Also reorganize/edit/add to the manual section for dynamic extent, not
only to advertise this new feature and its applications but also to
make it more coherent in general (bringing it up to date at the same
time).
The DerivedBidiClass.txt format (among others) has changed, with
multiple @missing lines for a single property now being used. This
means that the text file itself has many fewer explicit entries, and
consequently the :bidi-class test (if left alone) would be
substantially less rigorous. Adapt the test to keep track of which
characters have been checked already; parse the @missing lines; and
use a separate test to check the un-tested characters against the
specifications in the parsed @missing lines.
Other than new defined characters, the most significant change is
revisions to the grapheme, word and line-breaking algorithms,
particularly as relates to emoji.
Grapheme-breaking needs a new mini-state-machine in order to track the
extended pictographic state.
Word-breaking changes just enough to invalidate our previous
strategy of (attempting to) implement Replacing Ignore Rules (section
6.2 of UAX 29); instead, we now look back when we are in an extended
context to find the correct class. This is not maximally efficient,
and the strategy in line-breaking is probably better, but it is not
terrible either. We also need to handle the new requirement to
consider non-Glue Zs characters as their own WSegSpace class.
The line-breaking needs a very minor adjustment: nothing breaks after
a zero-width joiner.
Uses NDK as C crosscompiler and ADB to run code on the target device.
Uses both :linux and :android features (android is almost a linux).
It also reserves a folder android-libs/ for putting precompiled
libraries to link agains (for example zstd for core compression).
See doc/internals-notes/Android-build.txt for more details
For system code, this decreases consing from closures principally in
condition, restart, and debugger handlers, but also will allow us to
directly stack allocate closures without a manual rewrite/transform to
DX-FLET in the future.
179 additional closures were stack allocated in self-build due to this
change.
Also update the manual to reflect this, removing an outdated and
somewhat irrelevant statement pertaining to blocks and tags inside
dynamic extent closures. (They are now never heap allocated if akk
non-local transfers to them are inside functions declared dynamic
extent.)
This allows the programmer to explicitly annotate that the order of
the methods in the method group does not matter (for example, because
they are ordered explicitly by the method combination body), and
therefore that the matching of methods to groups should not throw an
error if there are multiple methods with the same specializers.
This is (strictly speaking) a violation of the specification, which
requires that the only acceptable arguments to :ORDER should be
:MOST-SPECIFIC-FIRST and :MOST-SPECIFIC-LAST. Document this violation
in the manual.
This allows removal of the coupling between dispatch functions of
SLOT-MAKUNBOUND-USING-CLASS and SLOT-BOUNDP-USING-CLASS.
Condition slots can be unbound; make the boundp and makunbound logic
explicit, and integrate it into the CLOS mechanisms.
Structure non-raw slots can also (rarely) be unbound, when a
by-order-of-arguments constructor is defined with the initialization
coming from an &AUX variable with no default. Integrate this
knowledge into the CLOS mechanisms, avoiding handling boundness of
slots in the code fragments that CLOS uses (while preserving existing
behaviour of the accessor functions defined by DEFSTRUCT.)
Add tests of various expected behaviours of SLOT-BOUNDP and friends.
These tests are not complete, given the number of different ways CLOS
can walk and optimize code and that instances of structures,
conditions and classes can be constructed, but they hopefully exercise
some previously-untested pathways.
and a design doc explaining why.
Unless compiled in with #+system-tlabs, there is no change to the
generated code. So average users won't care either way.
The implementation of this feature is much further along than implied
by the design. Which is to say, it works, and will be committed
in a series of changes, pending some touch-ups.
The big problem is that it explains how to define an external-format
which SBCL simply doesn't allow. Apart from that, the entire section
on "Design Choices" (internally UTF-16) does not pertain to SBCL.
sb-simple-streams is both more complete and less complete than CMUCL's.
It certainly seems that we have have read- and write- sequence afaict,
so there's no reason to say that we don't, but ultimately referring the
user to Allegro's docs. And the (REQUIRE ...) example is wrong, so ....
A few problems:
1. our ENCAPSULATE isn't in SB-EXT, it's in SB-INT
so it's clearly not for user consumption
2. ours is not the same - there's no ARGUMENT-LIST or BASIC-DEFINITION
3. nothing internal uses ENCAPSULATED-P, therefore nothing can
use it, because see 1.
* binary-distribution makes no use of cache_dir now
* generate-texinfo hasn't needed asdf-object-cache since git rev 2e266c9674
* 'test' recipes are never used
Anticipating that FDEFNs can be subsumed by something other than 1st-class
objects, we'll need a unified space of lisp function pointers. I tried to pick
a non-conflicting name for such space, but "linkage table" really fits.
Therefore this change renames the existing table. It was FOREIGN-LINKAGE-TABLE
in CMUCL, but "ALIEN" requires fewer keystrokes and matches DEFINE-ALIEN.
It was explicitly added in 60f84eda5c, which caused me some skepticism
as I saw no merit to keeping deceptive information in-tree. i.e. some of the
internals are not right for SBCL (and possibly not CMUCL any more) such as:
- SBCL executables don't produce three ELF files, there is no linker.sh etc.
- object.tex is more wrong than right. The widetags are completely wrong;
as are nearly all the descriptions of various headered object types.
(functions don't contain a pointer to "next function", and they don't embed
their FTYPE in the raw data portion, etc etc)
- fasl.tex is sufficiently wrong as to be of little utility - the entire point
of our LEB128-encoding is to remove bifurcation of all fops as it alludes to.
Not to mention there's no fop-uniform-vector, fop-code-format,
fop-pop-for-effect, and others.
- foreign-linkage-symbols and linkage-table-data aren't the same at all.
- our INFO database uses <category,kind> for the taxonomy, but CMUCL
named them <class,type> and said that they are compared via STRING=.
Neither is more obviously right, so this is just ridiculously confusing.
- The interpreter documentation is flat-out 100% wrong as pertains to SBCL.
All files files removed in this change are obsolete relative to SBCL, but to
revise them in-place would be worse - anyone would wonder whether they document
the current state of SBCL or CMUCL. The right fix would have been to take out
dangling links from our docs - someone can still do that! - and not to add
explictly incorrect text that increases the amount of stuff you have to
mentally weed out when doing "git grep".
Some "internals" docs are more prosasic and theoretical than detail-oriented,
so it's hard to say whether they're right or wrong. (Is it wrong to postulate
counterfactuals like "it would be nice if x,y,z held ..."?)
I didn't know what to do with Makefile, so I removed it, as well as skeletal
table-of-contents files.
If some contrib module is blocked for building also block it in
documentation generation since this will fail otherwise.
Additionally this fixes a shift operation that is needed
unconditionally when parsing the script parameters since otherwise
the first parameter is used for the second parameter as well if it
turns out to be empty.
This commit contains the sb-simd contrib in its most recent form. The
original sb-simd repository will be archived, and development of sb-simd
will now be a part of the regular SBCL development. Thanks for everyone
who made this work possible!
So that TLS can be initialized using memset(ptr,0xff,size).
In making this change I discovered that scavenge_control_stack() was slightly
inefficient - if (is_lisp_pointer() && !from_space_p()) there is no need
to check the scavtab[] entry.
Also the text in 'objects-in-memory' was wrong- no-tls-value marker
never appears "in" a symbol's TLS field, only on the binding stack.
And all remarks about enabling those. Worst advice ever.
The comments even say they cause more problems.
("Yo dawg, I herd u like locks so I added extra locks
so you can deadlock while u deadlock")
I don't know what QSHOW_SAFE was either.
:REPORT option of NIL and TRACE maintain the same semantics, and
additionally any function designator may be provided which is then
funcalled for each trace event and may be used to implement custom
report methods, e.g., an integration with SLIME.
New special variable *TRACE-REPORT-DEFAULT* defaults to 'TRACE and
controls the default value of the :REPORT option.
I started to update it but then came to the conclusion that it is
naught but maintenance burden. It's not as its contents will lead
us to discovering elusive thread-related bugs at this point.
Our source code makes direct reference to some internals .tex files
found only in the CMU CL source, and even our own manual has direct
references to the CMU CL manual because the corresponding sections
haven't been written yet (for over 2 decades).
We might as well check-in the original documentation. I doubt anyone
will ever be paid to write or even finish porting such high quality
documentation, and it would be a shame to have 95% correct internals
documentation be completely missing as well.
This change adds many things:
* There is now a proper interface to defining and using foreign
callbacks in Lisp.
* There is now a usable script to build
libsbcl.so ("make-shared-library.sh"), which also slightly adjusts the
runtime to make it more amenable for external use with respect to
signalling.
* There is now a way to use libsbcl.so in conjunction with other
object code to actually initialize Lisp and call Lisp foreign
callables from C directly. (this works, at least on Linux, but support
is not quite complete yet, because we don't have a way to
de-initialize Lisp)
* `save-lisp-and-die' has gained a new argument to support the above.
I guess technically DEFINE-ALIEN-CALLBACK can be removed, since the
DEFINE-ALIEN-CALLABLE interface now supersedes it, but internal stuff
like tests still use it, so that change can wait for now.
* Simple-vector without any specified initialization gets initialized
with a value that signals an error if read.
* Stores to an unboxed vectors update the shadow ("written") bits.
This essentially shows that there are no bugs relating to the garbage
collector failing to relocate shadow bits along with the vector
(which was among the failure modes). The bits are still not read though.
See doc/internals-notes/array-ubsan for further information.
The rationale and mechanics of this change have been adequately hashed
out on sbcl-devel, so nothing further to add here other than that a few
renamings of local variables from LAYOUT to WRAPPER may be worthwhile
depending on how other maintainers feel about the current state.
There are no user-observable differences in the normal configuration.
If built with :METASPACE, then classoids point to wrappers, not layouts.
If built without that, there should be zero impact, but the slot name
is still WRAPPER, which means LAYOUT.
All functions that actually end up in the manual have known lambda
lists, but docstrings are also generated for some functions in the
sb-md5 contrib which do not have known lambda lists.
Compare layouts by a stable ID instead of the address.
As can be seen from the "perf stat" results accompanying the design notes
in the doc directory, the time for self-build on x86-64 decreases somewhere
between 3% to 8% depending on the build phase.
Since this change is not specific to the compiler, but just type-checking
in general, the results suggest that users may similarly see anything in
that range of speedup for code making heavy use of defstructs.
Code compiled in 0 safety will see less speedup, because there are no type
assertions, while there may be type-based dispatch that is improved.
Additionally, core files tend to be slightly smaller due to having
fewer code header constants that reference layouts, and correspondingly
there are fewer boxed words to examine in GC.
This undoes parts of 902e9373 in the interest of following the spec by its
intent and de-facto correct interpretation versus a theoretical view
that is of little practical value.
As the test file shows, among all other implementations examined, the consensus
is 8-to-2 against treating (VECTOR NIL) as a string even where support for
(ARRAY NIL) as a type exists. And the minority stance is weak- both outliers
upgrade NIL to their widest character type in MAKE-STRING, proving beyond a
doubt that they don't really think that NIL means empty type in that usage.
So now SBCL goes along with that for maximum compatibility with everyone.
I considered a hypothetical build-time #+vector-nil-is-string feature,
but it's not worth cluttering the code for. Nobody would/should use it.
In searching for strings to remove along with the Alpha backend,
I found obsolete references to what is and isn't supported:
* The man page implied that SBCL runs on PPC-based Macs, but lacking
any recent positive evidence, it's not worth making that claim.
* base-target-features said that gencgc doesn't work on Sparc
and PPC, but it does; similarly in make-config.sh.
* efficiency.texinfo failed to mention 3 of the 64-bit platforms
in the description of modular arithmetic functions.
* SIGTRAP is used on CPUs other than Alpha, so generalize the comment.
* Omit mention of Alpha in a few places so I don't have to make
trivial edits again. (The text is fine with or without it)
* Remove the create_thread_lock and thread setup semaphore. The start function
is handed off in a GC-safe way and the creating thread is allowed to continue
as soon as pthread_create() returns.
* Retain the memory from up to 2 exited threads for reuse by new threads.
This is not shared with the FOREIGN-THREAD memory pool, but perhaps could be.
While this adds more conditionalization to both C and Lisp, it should be
adaptable to OS_THREAD_STACK so that ultimately we can have only one way
of performing post-mortem freeing instead of three.
Benchmarks show about 6x to 7x faster lisp thread creation.
Incidentally this fixes lp#1595699.
And respect ANSI with the proclamation processing at that. (The CMU CL
code did some magic proclaim handling instead which was the initial
reason the support got gutted in the first place, as far as I can
tell.)
Yes, it is 2020, time to end the pain of changing LANG.
LANG no longer has any effect, sb-ext:*default-external-format* and
sb-ext:*default-c-string-external-format* replace it. (Still
accessible through their old packages).
Saved cores won't clear these variables on start up anymore.
As reported by Shinmera on #lisp, in CHANGE-CLASS, slots with
:INSTANCE allocation were being read correctly in the presence of
applicable non-standard SLOT-VALUE-USING-CLASS methods, but were being
written with a direct access to the slot vector even if the new class
had (SETF SLOT-VALUE-USING-CLASS) methods applicable.
The fix is to short-circuit the slot-value-using-class methods, rather
than execute them. The reason is not directly related to
CHANGE-CLASS, but rather the protocol for updating obsolete instances:
in order to fulfil that case, we must read and write an object's slots
at a lower-level than going through SLOT-VALUE-USING-CLASS, because
SLOT-VALUE-USING-CLASS will cause another obsolete instance trap. The
protocol is specified to support only :INSTANCE and :CLASS slots, so
if we impose a rule that the same methods must be applicable to the
old and new slots of the same name, our direct reads and writes will
have the same effect on the slot vectors as full protocol reads and
writes, preserving the illusion that an object is mutated in-place to
the new or updated class.
Unfortunately moving with-ir1-namespace did cause enough leakage to be
a problem for a project in the Quicklisp world. Punt for release to
rethink namespace issues.
makeinfo complains if (some?) @ref node targets have colons in them.
Adjust docstrings.lisp to produce @anchor nodes without colons, and
rewrite explicit references accordingly.
The info reader cannot parse the generated index for cross-references
if anchors have newlines in them. Be more conservative about printing
anchor names, and also shorten the anchor name for methods by not
including the whole lambda list.
The @lw macro which we were using to try to break class precedence
lists appropriately broke, again. Remove it completely and use
@raggedright instead.
Include a version identifier in the @copying text for info format, as
otherwise the version number doesn't appear anywhere in the output.
Use a smaller font size, for fewer horrible linebreaks (since there
are fewer linebreaks in total)
Add a couple of index entries for deprecation conditions.
