---
name: shepherd-sources
description: Read, search and query the user's own connected Shepherd sources (iMessage, Gmail, Slack, Google Docs/Calendar/Contacts, Notion, Linear, Jira, Granola, coding sessions, ...) as a read-only filesystem plus read-only SQL, through the `shepherd` CLI or its MCP tools. Use when a task needs the user's synced messages, email, documents, meetings or tickets, or mentions Shepherd, $SOURCES or $SCRATCH.
---

# Shepherd sources

Shepherd keeps an ingested, versioned copy of the sources the user connected. You reach it through tools that run as
the signed-in user and see only that user's data:

- `$SOURCES/<provider>/...` — read-only files (Markdown projections, raw provider JSON under `_raw/`, attachment bytes).
- `$SCRATCH/...` — the user's private, persistent, writable scratch (notes, exports, intermediate results).
- SQL — read-only SQLite over the `$SOURCES` files you can read: `files`, `records` (every JSON document) and curated
  views such as `imessage_messages`, `gmail_messages`, `slack_messages`, `calendar_events`, `linear_issues`.

It is a captured copy, not the live provider: check `sources_status` before claiming something is complete or current,
and cite the `$SOURCES` path (and `capturedAt`) of what you used.

## Which interface

- MCP tools named `ls`, `read`, `find`, `grep`, `sql_schema`, `sql_query`, `sources_status`, `sources_history`,
  `write`, `edit`, `scratch_*` from a server called `shepherd`: use them directly.
- Otherwise use the CLI in a shell: `shepherd ls|read|cat|find|grep|history|status|schema|sql|put|write|mkdir|mv|rm`.
  Quote paths so the shell does not expand them: `shepherd ls '$SOURCES/imessage'`. Add `--json` for exact receipts.
- Check access with `shepherd whoami`. "not signed in" means the user must run `shepherd login` themselves (it opens
  their browser). Never ask for passwords, tokens or codes, and never copy credentials into files or agent configs.
- Setup on a new machine and client configs: [references/setup.md](references/setup.md).

## Files or SQL

- **Start with `ls '$SOURCES'`** (one directory per connected provider, then account/workspace/mailbox directories) or
  `sources_status` (what is connected, coverage, freshness).
- **SQL** for counting, filtering, sorting, joining and time ranges over structured records (messages, emails,
  events, issues, contacts). Run `sql_schema` first, then `sql_schema` with `table` for columns and examples. Every row
  has `path`, the exact `$SOURCES` file it came from: `read` that file for full fields or context.
- **semantic_search** (`shepherd search`) when you know what something is about but not its wording ("the doc where
  we decided…", "messages about the trip"); then `read` the hit's `retrieve` range. Say so when `scope.excluded` or
  `progress` shows parts not searched; images/audio/binaries are retrievable but not text-searchable.
- **grep/read** for exact text inside documents and transcripts, and for anything the views do not model. `find` for
  file names. Raw JSON keeps every provider field even when no view exposes it (`json_extract(data, '$.field')` on
  `records`).
- Messages (iMessage/SMS): `imessage_messages` (one row per message), `imessage_conversations`,
  `imessage_participants`, `imessage_attachments` (byte state and `file_path`), `imessage_contacts`,
  `imessage_message_edits`. Readable monthly transcripts: `imessage_conversation_months.transcript_path`. Recipes and
  caveats: [references/sql.md](references/sql.md).

## Bounded answers — always continue explicitly

Every answer is bounded; nothing is silently dropped, so read the receipt:

| Field | Meaning | Next step |
| --- | --- | --- |
| `truncated` + `nextAfter` (ls/find/grep) | more entries/matches | same call with `after = nextAfter` |
| `truncatedReason` | `limit` (your limit), `walk_limit`/`scan_limit` (server bound per call) | continue with `after`, or narrow the path/glob |
| `matchesMayContinueIn` (grep) | that file may hold more matches | grep that single file |
| `skipped.binary/tooLarge` (grep) | not line-searchable | read by byte range, or query SQL |
| `nextOffset` / `nextByteOffset` (read) | file continues | read again from there |
| `lineCut` (read) | one line longer than an answer | read its rest with `byteOffset = lineCut.restByteOffset` |
| `nextOffset` (sql_query) | more rows | same sql/params with `offset = nextOffset`, `seq = snapshot.seq` (use ORDER BY); `snapshotMoved` = data changed between pages |
| `complete: false`, `pending`, `errors` (SQL) | some files not indexed yet / cannot be rows | say so; retry later or read the named files |
| `cellsCut` (SQL) | values longer than `maxCellChars` (default 16384) | each is listed with its full length: ask again with a larger `maxCellChars` (≤ 4194304) or page it with `substr()` |
| `lineChars` (grep) | matching line shown clipped at 500 characters | `read` the file at `offset = line` |

Use `limit`/`maxRows` sized to the question; page deliberately instead of fetching everything. `grep` takes
`context` (lines around each match) like Pi's grep.

## Attachments and images

Attachment bytes are stored files in `$SOURCES` (e.g. `imessage_attachments.file_path`; `bytes_verified = 1` means the
file exists with the recorded sha256 and size). `read` of a PNG/JPEG/GIF/WebP without offsets returns the image itself
to the model (up to 3.75 MB); other formats (HEIC, PDF, audio) are read by byte range or downloaded exactly with
`shepherd get PATH FILE`. An attachment in state `pending`/`unavailable` has no bytes yet: say so, never describe it.

## Permissions

`$SOURCES` refuses every write; write only below `$SCRATCH` (`write`, `edit`, `put`, `mkdir`, `mv`, `rm`). SQL accepts
one SELECT (or WITH ... SELECT) with `?` parameters; writes, PRAGMA, ATTACH and extensions are refused. Put values in
`params` rather than splicing them into SQL. Access errors are final: they are not bugs to work around.

## Running code

The CLI and MCP tools need no sandbox. For heavy processing (scripts over many files) the user may have Modal
workspaces: `shepherd modal status`; only if they are enabled, `shepherd modal start`, then
`shepherd modal exec ID -- <command>` (MCP: `sandbox_exec`) runs in a network-blocked sandbox with `/sources`
read-only, `/scratch` writable and the CLI inside in local mode (in mount mode both are symlinks: `find -H /sources`,
`ls /sources/`). Enabling workspaces copies the user's sources to the service's Modal storage: leave
`shepherd modal enable` to the user. Stop sessions you started (`shepherd modal stop ID`).

Full tool and command reference with every parameter and error code: [references/tools.md](references/tools.md).