Skip to content
CraftDocs
GitHub
Home
Home
Changelog
What's New
Guide
Guide
Getting Started
Principles
Styling Components
Theme System
Foundations
All Tokens
Color
Elevation
Icons
Motion
Shape
Spacing
Typography
Libraries
Libraries
@xds/cli
@xds/core
Themes
Themes
Theme: daily
Default Theme
Theme: matcha
Neutral Theme
Components
Components
AppShell
AspectRatio
Avatar
Avatar
AvatarStatusDot
Badge
Banner
Breadcrumbs
BreadcrumbItem
Breadcrumbs
Button
Button
IconButton
ToggleButton
ToggleButtonGroup
Calendar
Card
Carousel
Chat
ChatComposer
ChatComposerDrawer
ChatComposerInput
ChatComposerTokenElement
ChatDictationButton
ChatLayout
ChatLayoutScrollButton
ChatMessage
ChatMessageBubble
ChatMessageList
ChatMessageMetadata
ChatSendButton
ChatSystemMessage
ChatTokenizedText
ChatToolCalls
Checkbox
CheckboxInput
CheckboxList
CheckboxListItem
ClickableCard
Code
CodeBlock
Collapsible
Collapsible
CollapsibleGroup
useXDSCollapsible
CommandPalette
CommandPalette
CommandPaletteEmpty
CommandPaletteFooter
CommandPaletteGroup
CommandPaletteInput
CommandPaletteItem
CommandPaletteList
DateInput
Dialog
AlertDialog
Dialog
DialogHeader
useXDSImperativeAlertDialog
useXDSImperativeDialog
Divider
DropdownMenu
DropdownMenu
DropdownMenuDivider
DropdownMenuItem
DropdownMenuItemData
DropdownMenuSection
EmptyState
Field
Field
FieldLabel
FieldStatus
Heading
HoverCard
Icon
Kbd
Layout
Center
FormLayout
Grid
GridSpan
HStack
Layout
LayoutContainer
LayoutContent
LayoutFooter
LayoutHeader
LayoutPanel
Section
StackItem
VStack
Link
List
List
ListItem
Markdown
MetadataList
MetadataList
MetadataListItem
MobileNav
MoreMenu
NavIcon
NavMenuItem
NumberInput
OverflowList
Pagination
Popover
PowerSearch
ProgressBar
Radio
RadioList
RadioListItem
Resizable
ResizeHandle
useXDSResizable
SegmentedControl
SegmentedControl
SegmentedControlItem
SelectableCard
Selector
MultiSelector
Selector
SelectorOption
SideNav
SideNav
SideNavCollapseButton
SideNavHeading
SideNavItem
SideNavSection
Skeleton
Slider
Spinner
StatusDot
Switch
Table
BaseTable
Table
TableCell
TableHeaderCell
TableRow
useXDSTableColumnSettings
useXDSTablePagination
useXDSTableSelection
useXDSTableSelectionState
useXDSTableSortable
Tabs
Tab
TabList
TabMenu
Text
TextArea
TextInput
Thumbnail
TimeInput
Timestamp
Toast
Toast
useXDSToast
Token
Tokenizer
Toolbar
Tooltip
TopNav
TopNav
TopNavHeading
TopNavItem
TopNavMegaMenu
TopNavMegaMenuFeaturedCard
TopNavMegaMenuItem
TopNavMenu
TreeList
Typeahead
BaseTypeahead
Typeahead
TypeaheadItem
useXDSHoverCard
useXDSPopover
useXDSTooltip
Utilities
Utilities
LinkProvider
MediaTheme
SyntaxTheme
Theme
useClickableContainer
useEntryAnimation
useFocusTrap
useGridFocus
useImageMode
useInputContainer
useListFocus
useMediaQuery
useOverflow
useScrollLock
useScrollOverflow
useXDSLayer
useXDSStreamingText
Terms of UsePrivacy Policy
Type to search
↑↓Navigate↵SelectEscClose
Tokenizer@xds/core · XDSTokenizer v0.0.13

Usage

Tokenizer is a multi-select input that lets users search, select, and manage multiple items displayed as removable chips. Use it when users need to build a set of selections from a searchable data source, like adding team members, applying tags, or choosing filters.
ts
import {XDSTokenizer} from '@xds/core/Tokenizer'

Best practices

GuidancePractices
DoWrite a placeholder that tells users what they can search for — "Search people…" or "Add tags…" — so the input is not a blank mystery.
DoSet maxEntries when the number of selections should be bounded, like limiting a review to 5 approvers.
DoUse hasCreate for free-form tagging where users need to enter values that do not exist in the search source.
DoShow validation status with the status prop so users know immediately when a selection is missing or invalid.
Don'tDon’t use Tokenizer for single-item selection — use Typeahead instead. Tokenizer is for building sets of two or more items.
Don'tAvoid applying custom colors to individual tokens inside a Tokenizer — use the default token style for visual consistency across the set.
Don'tDon’t hide the label — every Tokenizer needs a visible label so users understand what they are selecting. Use isLabelHidden only when surrounding context makes the purpose obvious.

Examples

Common configurations, variations, and states.
Tokenizer \u2014 ClearTokenizer with a built-in clear-all button for bulk removal of all selected tokens.
tsx
'use client';
​
import {useState} from 'react';
import * as stylex from '@stylexjs/stylex';
import {XDSTokenizer} from '@xds/core/Tokenizer';
import {XDSStack} from '@xds/core/Layout';
import {XDSText} from '@xds/core/Text';
import type {XDSSearchableItem, XDSSearchSource} from '@xds/core/Typeahead';
​
const styles = stylex.create({
fixed: {width: 400},
});
​
const users: XDSSearchableItem[] = [
{id: '1', label: 'Alice Johnson'},
{id: '2', label: 'Bob Smith'},
{id: '3', label: 'Charlie Brown'},
{id: '4', label: 'Diana Prince'},
{id: '5', label: 'Eve Williams'},
];
​
const userSource: XDSSearchSource = {
search: (query: string) =>
users.filter(u => u.label.toLowerCase().includes(query.toLowerCase())),
bootstrap: () => users,
};
​
export default function TokenizerClear() {
const [value, setValue] = useState<XDSSearchableItem[]>([users[0], users[1]]);
​
return (
<XDSStack direction="vertical" gap={2}>
<XDSText type="supporting" color="secondary">
Clear-all button appears when tokens are selected
</XDSText>
<XDSTokenizer
label="Team Members"
placeholder="Search people..."
searchSource={userSource}
value={value}
onChange={items => setValue(items)}
hasClear
xstyle={styles.fixed}
/>
</XDSStack>
);
}
Tokenizer \u2014 CreatableFree-text tokenizer for creating custom tags and a combined create-or-search pattern. Use when users need to enter values that may not exist in a predefined list.
tsx
'use client';
​
import {useState} from 'react';
import * as stylex from '@stylexjs/stylex';
import {XDSTokenizer} from '@xds/core/Tokenizer';
import {XDSStack} from '@xds/core/Layout';
import {XDSText} from '@xds/core/Text';
import type {XDSSearchableItem, XDSSearchSource} from '@xds/core/Typeahead';
​
const styles = stylex.create({
fixed: {width: 400},
});
​
const emptySource: XDSSearchSource = {
search: () => [],
bootstrap: () => [],
};
​
const users: XDSSearchableItem[] = [
{id: '1', label: 'Alice Johnson'},
{id: '2', label: 'Bob Smith'},
{id: '3', label: 'Charlie Brown'},
{id: '4', label: 'Diana Prince'},
{id: '5', label: 'Eve Williams'},
];
​
const userSource: XDSSearchSource = {
search: (query: string) =>
users.filter(u => u.label.toLowerCase().includes(query.toLowerCase())),
bootstrap: () => users,
};
​
export default function TokenizerCreatable() {
const [tags, setTags] = useState<XDSSearchableItem[]>([]);
const [members, setMembers] = useState<XDSSearchableItem[]>([]);
​
return (
<XDSStack direction="vertical" gap={4}>
<XDSStack direction="vertical" gap={1}>
<XDSText type="supporting" color="secondary">
Free-text only
</XDSText>
<XDSTokenizer
label="Tags"
searchSource={emptySource}
value={tags}
onChange={items => setTags(items)}
hasCreate
placeholder="Type a tag and press Enter..."
xstyle={styles.fixed}
/>
</XDSStack>
<XDSStack direction="vertical" gap={1}>
<XDSText type="supporting" color="secondary">
Create or search
</XDSText>
<XDSTokenizer
label="Team Members"
searchSource={userSource}
value={members}
onChange={items => setMembers(items)}
hasCreate
hasEntriesOnFocus
placeholder="Search or type a new name..."
xstyle={styles.fixed}
/>
</XDSStack>
</XDSStack>
);
}
Tokenizer \u2014 End ContentTokenizer with an action button in the end slot. Use for inline actions like applying selections alongside the input.
tsx
'use client';
​
import {useState} from 'react';
import * as stylex from '@stylexjs/stylex';
import {XDSTokenizer} from '@xds/core/Tokenizer';
import {XDSButton} from '@xds/core/Button';
import {XDSStack} from '@xds/core/Layout';
import {XDSText} from '@xds/core/Text';
import type {XDSSearchableItem, XDSSearchSource} from '@xds/core/Typeahead';
​
const styles = stylex.create({
fixed: {width: 400},
});
​
const users: XDSSearchableItem[] = [
{id: '1', label: 'Alice Johnson'},
{id: '2', label: 'Bob Smith'},
{id: '3', label: 'Charlie Brown'},
{id: '4', label: 'Diana Prince'},
{id: '5', label: 'Eve Williams'},
{id: '6', label: 'Frank Miller'},
];
​
const userSource: XDSSearchSource = {
search: (query: string) =>
users.filter(u => u.label.toLowerCase().includes(query.toLowerCase())),
bootstrap: () => users,
};
​
export default function TokenizerEndContent() {
const [value, setValue] = useState<XDSSearchableItem[]>([users[0], users[2]]);
​
return (
<XDSStack direction="vertical" gap={2}>
<XDSText type="supporting" color="secondary">
Action button in the end slot
</XDSText>
<XDSTokenizer
label="Team Members"
placeholder="Search people..."
searchSource={userSource}
value={value}
onChange={items => setValue(items)}
endContent={<XDSButton label="Apply" variant="primary" size="sm" />}
xstyle={styles.fixed}
/>
</XDSStack>
);
}
Tokenizer \u2014 IconTokenizer with a leading search icon to visually reinforce the search behavior.
tsx
'use client';
​
import {useState} from 'react';
import * as stylex from '@stylexjs/stylex';
import {XDSTokenizer} from '@xds/core/Tokenizer';
import {XDSStack} from '@xds/core/Layout';
import {XDSText} from '@xds/core/Text';
import {MagnifyingGlassIcon} from '@heroicons/react/24/outline';
import type {XDSSearchableItem, XDSSearchSource} from '@xds/core/Typeahead';
​
const styles = stylex.create({
fixed: {width: 400},
});
​
const users: XDSSearchableItem[] = [
{id: '1', label: 'Alice Johnson'},
{id: '2', label: 'Bob Smith'},
{id: '3', label: 'Charlie Brown'},
{id: '4', label: 'Diana Prince'},
{id: '5', label: 'Eve Williams'},
];
​
const userSource: XDSSearchSource = {
search: (query: string) =>
users.filter(u => u.label.toLowerCase().includes(query.toLowerCase())),
bootstrap: () => users,
};
​
export default function TokenizerIcon() {
const [value, setValue] = useState<XDSSearchableItem[]>([users[0], users[2]]);
​
return (
<XDSStack direction="vertical" gap={2}>
<XDSText type="supporting" color="secondary">
Leading icon reinforces the search affordance
</XDSText>
<XDSTokenizer
label="Team Members"
placeholder="Search people..."
searchSource={userSource}
value={value}
onChange={items => setValue(items)}
startIcon={MagnifyingGlassIcon}
xstyle={styles.fixed}
/>
</XDSStack>
);
}
Tokenizer \u2014 Max EntriesTokenizer with a maximum selection limit. The input hides automatically when the limit is reached, preventing further additions.
tsx
'use client';
​
import {useState} from 'react';
import * as stylex from '@stylexjs/stylex';
import {XDSTokenizer} from '@xds/core/Tokenizer';
import {XDSStack} from '@xds/core/Layout';
import {XDSText} from '@xds/core/Text';
import type {XDSSearchableItem, XDSSearchSource} from '@xds/core/Typeahead';
​
const styles = stylex.create({
fixed: {width: 400},
});
​
const skills: XDSSearchableItem[] = [
{id: '1', label: 'React'},
{id: '2', label: 'TypeScript'},
{id: '3', label: 'GraphQL'},
{id: '4', label: 'Node.js'},
{id: '5', label: 'Python'},
{id: '6', label: 'Rust'},
{id: '7', label: 'Go'},
{id: '8', label: 'Swift'},
];
​
const skillSource: XDSSearchSource = {
search: (query: string) =>
skills.filter(s => s.label.toLowerCase().includes(query.toLowerCase())),
bootstrap: () => skills,
};
​
const MAX_SKILLS = 3;
​
export default function TokenizerMaxEntries() {
const [value, setValue] = useState<XDSSearchableItem[]>([
skills[0],
skills[1],
]);
​
return (
<XDSStack direction="vertical" gap={2}>
<XDSText type="supporting" color="secondary">
Limited to {MAX_SKILLS} selections — {MAX_SKILLS - value.length}{' '}
remaining
</XDSText>
<XDSTokenizer
label="Top Skills"
placeholder="Search skills..."
description={`Choose up to ${MAX_SKILLS} skills`}
searchSource={skillSource}
value={value}
onChange={items => setValue(items)}
maxEntries={MAX_SKILLS}
xstyle={styles.fixed}
/>
</XDSStack>
);
}
Tokenizer \u2014 OverflowTokenizer with overflow truncation when unfocused. Inline mode pushes content down on expand; layer mode overlays without shifting layout.
tsx
'use client';
​
import {useState} from 'react';
import * as stylex from '@stylexjs/stylex';
import {XDSTokenizer} from '@xds/core/Tokenizer';
import {XDSStack} from '@xds/core/Layout';
import {XDSText} from '@xds/core/Text';
import type {XDSSearchableItem, XDSSearchSource} from '@xds/core/Typeahead';
​
const styles = stylex.create({
fixed: {width: 400, maxWidth: 400},
});
​
const users: XDSSearchableItem[] = [
{id: '1', label: 'Alice Johnson'},
{id: '2', label: 'Bob Smith'},
{id: '3', label: 'Charlie Brown'},
{id: '4', label: 'Diana Prince'},
{id: '5', label: 'Eve Williams'},
{id: '6', label: 'Frank Miller'},
];
​
const userSource: XDSSearchSource = {
search: (query: string) =>
users.filter(u => u.label.toLowerCase().includes(query.toLowerCase())),
bootstrap: () => users,
};
​
export default function TokenizerOverflow() {
const [inlineValue, setInlineValue] = useState<XDSSearchableItem[]>(users);
const [layerValue, setLayerValue] = useState<XDSSearchableItem[]>(users);
​
return (
<XDSStack direction="vertical" gap={4}>
<XDSStack direction="vertical" gap={1}>
<XDSText type="supporting" color="secondary">
Inline overflow — content shifts down on expand
</XDSText>
<XDSTokenizer
label="Inline Overflow"
placeholder="Add more..."
searchSource={userSource}
value={inlineValue}
onChange={items => setInlineValue(items)}
tokenOverflowBehavior="unfocusedInline"
xstyle={styles.fixed}
/>
</XDSStack>
<XDSStack direction="vertical" gap={1}>
<XDSText type="supporting" color="secondary">
Layer overflow — expands as overlay, no layout shift
</XDSText>
<XDSTokenizer
label="Layer Overflow"
placeholder="Add more..."
searchSource={userSource}
value={layerValue}
onChange={items => setLayerValue(items)}
tokenOverflowBehavior="unfocusedLayer"
xstyle={styles.fixed}
/>
</XDSStack>
</XDSStack>
);
}
Tokenizer \u2014 StatesTokenizer in disabled, error, warning, and success states. Use to communicate validation feedback or lock a selection from editing.
tsx
'use client';
​
import {useState} from 'react';
import * as stylex from '@stylexjs/stylex';
import {XDSTokenizer} from '@xds/core/Tokenizer';
import {XDSStack} from '@xds/core/Layout';
import type {XDSSearchableItem, XDSSearchSource} from '@xds/core/Typeahead';
​
const styles = stylex.create({
fixed: {width: 400},
});
​
const users: XDSSearchableItem[] = [
{id: '1', label: 'Alice Johnson'},
{id: '2', label: 'Bob Smith'},
{id: '3', label: 'Charlie Brown'},
{id: '4', label: 'Diana Prince'},
{id: '5', label: 'Eve Williams'},
];
​
const userSource: XDSSearchSource = {
search: (query: string) =>
users.filter(u => u.label.toLowerCase().includes(query.toLowerCase())),
bootstrap: () => users,
};
​
export default function TokenizerStates() {
const [errorValue, setErrorValue] = useState<XDSSearchableItem[]>([]);
const [warningValue, setWarningValue] = useState<XDSSearchableItem[]>([
users[0],
]);
const [successValue, setSuccessValue] = useState<XDSSearchableItem[]>([
users[1],
users[3],
]);
​
return (
<XDSStack direction="vertical" gap={4}>
<XDSTokenizer
label="Disabled field"
searchSource={userSource}
value={[users[0], users[2]]}
onChange={() => {}}
isDisabled
xstyle={styles.fixed}
/>
<XDSTokenizer
label="Error message"
placeholder="Search people..."
searchSource={userSource}
value={errorValue}
onChange={items => setErrorValue(items)}
isRequired
status={{type: 'error', message: 'At least one reviewer is required'}}
xstyle={styles.fixed}
/>
<XDSTokenizer
label="Warning message"
placeholder="Search people..."
searchSource={userSource}
value={warningValue}
onChange={items => setWarningValue(items)}
status={{
type: 'warning',
message: 'Consider adding at least 2 approvers',
}}
xstyle={styles.fixed}
/>
<XDSTokenizer
label="Success message"
placeholder="Search people..."
searchSource={userSource}
value={successValue}
onChange={items => setSuccessValue(items)}
status={{type: 'success', message: 'All required reviewers added'}}
xstyle={styles.fixed}
/>
</XDSStack>
);
}

Showcase source

tsx
'use client';
​
import * as stylex from '@stylexjs/stylex';
import {XDSTokenizer} from '@xds/core/Tokenizer';
import type {XDSSearchSource} from '@xds/core/Typeahead';
​
const styles = stylex.create({
fixed: {width: 400},
});
​
const source: XDSSearchSource = {
search: () => [],
bootstrap: () => [],
};
​
export default function TokenizerShowcase() {
return (
<XDSTokenizer
label="Tags"
placeholder="Search..."
searchSource={source}
value={[
{id: '1', label: 'Design'},
{id: '2', label: 'Engineering'},
]}
onChange={() => {}}
xstyle={styles.fixed}
/>
);
}