sbcl.sbcl/doc/manual/external-formats.texinfo

@c Generated by the sb-manual contrib. Do not edit.

@node external formats
@cindex external format
@cindex format, external
@chapter External Formats

@menu
* The Default External Format: default external format.
* External Format Designators: external format designators.
* Character Coding Conditions: character coding conditions.
* Converting between Strings and Octet Vectors: converting between strings and octet vectors.
* Supported External Formats: supported external formats.
@end menu

External formats determine the coding of characters from/to sequences
of octets when exchanging data with the outside world. Examples of
such exchanges are:

@itemize
@item Character streams associated with files, sockets and process
  input/output (see @ref{stream external formats} and
  @ref{running external programs})

@item Names of files

@item Foreign strings (see @ref{foreign types and lisp types})

@item Posix interface (see @ref{sb posix})

@item Hostname- and protocol-related functions of the BSD-socket interface
  (see @ref{networking})
@end itemize

Technically, external formats in SBCL are named objects describing
coding of characters as well as policies in case de- or encoding is
not possible. Each external format has a canonical name and zero or
more aliases. User code mostly interacts with external formats by
supplying external format designators to functions that use external
formats internally.

@node default external format
@section The Default External Format

@anchor{Variable sb-ext *default-external-format*}
@vvindex @sortas{default-external-format* sb-ext} *default-external-format* [sb-ext]
@deffn{Variable} sb-ext:*default-external-format*
Most functions interacting with external formats (@code{open}, notably)
use this default.
@end deffn
@anchor{Variable sb-ext *default-source-external-format*}
@vvindex @sortas{default-source-external-format* sb-ext} *default-source-external-format* [sb-ext]
@deffn{Variable} sb-ext:*default-source-external-format*
@end deffn
@anchor{Variable sb-ext *default-c-string-external-format*}
@vvindex @sortas{default-c-string-external-format* sb-ext} *default-c-string-external-format* [sb-ext]
@deffn{Variable} sb-ext:*default-c-string-external-format*
@end deffn
@node external format designators
@section External Format Designators

In situations where an external format designator is required, such as
the @code{:external-format} argument in calls to @code{open} or @code{with-open-file},
users may supply the name of an encoding to denote the external
format which is applying that encoding to Lisp characters.

In addition to the basic encoding for an external format, options
controlling various special cases may be passed, by using a list
(whose first element must be an encoding name and whose rest is a
plist) as an external file format designator.

More specifically, external format designators can take the
following forms:

@itemize
@item @code{:default}: Designates the current default external format (see
  @ref{default external format}).

@item @code{<keyword>}: Designates the supported external format that has
  @code{<keyword>} as one of its names (see @ref{supported external formats}).

@item @code{(<keyword> . <options-plist>)}: Designates an external format
  that is like the one designated by @code{<keyword>} with options as
  specified in @code{<options-plist>}.
@end itemize

Valid options for @code{<options-plist>} are:

@itemize
@item @code{:NEWLINE <newline>}

An external format with an explicit @code{:newline} option is like its
@code{<keyword>} parent but recognizes certain characters or
character sequences as newlines. For @code{:lf} (the default), the
@code{#\Linefeed} character is treated as @code{#\Newline} for both
input and output. For @code{:cr}, @code{#\Return} is treated as
@code{#\Newline}, while for @code{:crlf} the two-character sequence
@code{#\Return #\Linefeed} is translated to and from
@code{#\Newline}.

@item @code{:REPLACEMENT <replacement>}

An external format with an explicit @code{:replacement} option is like
its @code{<keyword>} parent but does not signal an error in case a
character or octet sequence cannot be en- or decoded. Instead,
it inserts @code{<replacement>} at the position in question.
@code{<replacement>} must be a string designator; that is, a
character or a string.
@end itemize

For example:

@example
(with-open-file (stream pathname :external-format '(:utf-8 :replacement #\?))
  (read-line stream))
@end example

will read the first line of @code{pathname}, replacing any octet
sequence that is not valid in the UTF-8 external format with a
question mark character.

@node character coding conditions
@section Character Coding Conditions

De- or encoding characters using a given external format is not always
possible:

@itemize
@item Decoding an octet vector using a given external format can fail if
  it contains an octet or sequence of octets that does not have an
  interpretation as a character according to the external format.

@item Conversely, a string may contain characters that a given external
  format cannot encode. For example, the ASCII external format
  cannot encode the character @code{#\ö}.
@end itemize

Unless the external format governing the coding uses the
@code{:replacement} option, SBCL will signal (continuable) errors under the
above circumstances. The types of the condition signaled are not
currently exported or documented but will be in future SBCL
versions.

@node converting between strings and octet vectors
@section Converting between Strings and Octet Vectors

To encode Lisp strings as octet vectors and decode octet vectors as
Lisp strings, the following SBCL-specific functions can be used:

@anchor{Function sb-ext string-to-octets}
@ffindex @sortas{string-to-octets sb-ext} string-to-octets [sb-ext]
@deffn{Function} sb-ext:string-to-octets string &key external-format start end null-terminate
Return an octet vector that is @code{string} encoded according to @code{external-format}.

If @code{external-format} is given, it must designate an external format.

If given, @code{start} and @code{end} must be bounding index designators and
designate a subsequence of @code{string} that should be encoded.

If @code{null-terminate} is true, the returned octet vector ends with an
additional 0 element that does not correspond to any part of @code{string}.

If some of the characters of @code{string} (or the subsequence bounded by
@code{start} and @code{end}) cannot be encoded by @code{external-format} an error of a
subtype of @code{sb-int:character-encoding-error} is signaled.

Note that for some values of @code{external-format} and @code{null-terminate} the
length of the returned vector may be different from the length of
@code{string} (or the subsequence bounded by @code{start} and @code{end}).
@end deffn
@anchor{Function sb-ext octets-to-string}
@ffindex @sortas{octets-to-string sb-ext} octets-to-string [sb-ext]
@deffn{Function} sb-ext:octets-to-string vector &key external-format start end
Return a string obtained by decoding @code{vector} according to @code{external-format}.

If @code{external-format} is given, it must designate an external format.

If given, @code{start} and @code{end} must be bounding index designators and
designate a subsequence of @code{vector} that should be decoded.

If some of the octets of @code{vector} (or the subsequence bounded by @code{start}
and @code{end}) cannot be decoded by @code{external-format} an error of a subtype of
@code{sb-int:character-decoding-error} is signaled.

Note that for some values of @code{external-format} the length of the
returned string may be different from the length of @code{vector} (or the
subsequence bounded by @code{start} and @code{end}).
@end deffn
@node supported external formats
@section Supported External Formats

The following lists the external formats supported by SBCL in
the form of the respective canonical name followed by the list of aliases:

@itemize
@item @code{:euc-jp}

@code{:eucjp}, @code{:|eucJP|}

@item @code{:gbk}

@code{:cp936}

@item @code{:shift_jis}

@code{:sjis}, @code{:|Shift_JIS|}, @code{:cp932}

@item @code{:ucs-2be}

@code{:ucs2be}

@item @code{:ucs-2le}

@code{:ucs2le}

@item @code{:ucs-4be}

@code{:ucs4be}

@item @code{:ucs-4le}

@code{:ucs4le}

@item @code{:utf-16be}

@code{:utf16be}

@item @code{:utf-16le}

@code{:utf16le}

@item @code{:utf-32be}

@code{:utf32be}

@item @code{:utf-32le}

@code{:utf32le}
@end itemize