# String Representation Deft strings are **codepoint-indexed over a byte backing store** — the Emacs model. `len` counts characters, `$s(a..b)` slices characters, and slicing can never land mid-character; internally the string is an immutable UTF-8 byte array that also safely holds arbitrary binary. This entry explains the coordinate systems, why the byte layer stays visible through `bytes/*`, and how display width fits in — so questions like "why are there three length-ish functions" read as designed behaviour. ## One type, three coordinate systems | Coordinate | Unit | Operations | |---|---|---| | **Character index** | codepoint | `len`, `$s(a..b)` / `slice`, `nth`, `char-at`, `index-of`, `split ""`, `reverse`, `chars/count`, `chars/slice`, `re/find` start/end | | **Display column** | terminal cell (East Asian Wide = 2) | `width`, `cells`, `truncate-width`, `pad-width-left/right`, `wrap-width` | | **Byte offset** | raw byte | `bytes/len`, `bytes/slice`, `byte-at` | Character indexing is the *editing* coordinate — the editor's cursor, selection, and find offsets are character indices, and all string verbs agree with each other on it. Display columns exist because the cell grid is monospace: a wide glyph (中文) occupies two cells, so "column on screen" is a different number than "character index", and layout verbs compute it. Byte offsets exist because strings double as the binary type: `fs/read` of a PNG, crypto blobs, protocol bodies. ## Why indexing is codepoint-based Before this design, string indexing was byte-based — inherited from the byte array — which meant slicing a `─` (three UTF-8 bytes) produced its lead byte `0xE2` (rendered as `â`): one character could come through as three fragments of mojibake. Every consumer had to bridge by hand. The migration moved the semantics into the language: - `s[i]`, `$s(a..b)`, `index-of` results, and `len` all speak characters — the bridge disappeared, and the editor's cursor model became a plain integer with no boundary-snapping. - Mojibake is structurally impossible: a slice boundary that would land inside a character resolves to the character's edge. The trade-offs accepted: indexing a character position requires resolving it to a byte offset (a scan — O(n); fine at line/label sizes, and `len` itself is O(1) via a count memoized on the string object, exactly like the hash cache). And "character" is a codepoint, not a grapheme cluster — `é` written as `e` + combining accent is two characters, and emoji ZWJ sequences are several. Grapheme segmentation is a possible future display-layer addition, not an indexing change. ## The binary layer Strings hold arbitrary bytes; that will not change (there is no separate bytes type). The rule is that **binary data uses the `bytes/*` verbs for offsets and sizing**:

 set bin [fs/read "image.png"]
set chunk [bytes/slice $bin 0 4096]   # exact bytes — never re-aligned
if ([bytes/len $bin] < 16) { ... } 

The reason is sharp: binary data can contain accidental valid UTF-8 runs (three bytes `e4 b8 80` inside a PNG decode as 中). Under character slicing those three bytes are ONE character — chunk boundaries computed as character counts would shift and corrupt the data. Byte math on binary must be byte math. The healthy pattern already used throughout the packages: pass binary strings WHOLE to native functions (crypto, codecs, media) and never index into them from Deft. Text, conversely, should never touch `bytes/*` — it reads text through characters and lays out through widths. ## Immutability and caches Strings are immutable after creation (single allocation: header + inline bytes). Two values are memoized lazily on the object, following the same pattern: `hash` (for map keys) and the character count (for `len`). Both are computed at most once per string, on the heap that owns it, written through a const-cast — safe because the value is a pure function of the immutable bytes. ## Rendering: where characters become columns The TUI hosts draw text as a grid of monospace cells. The renderer walks a line's `[cells]` — per character: its index, its string, its display width — writing wide glyphs as a two-cell pair. The editor maps its cursor (a character index) to a screen column through the same cells; clicks map back. See [Font Rendering](font-rendering) for the atlas side of that pipeline.