discrawl mirrors Discord guild data into local SQLite so you can search, inspect, and query server history without depending on Discord search.
It is a bot-token crawler. No user-token hacks. Data stays local.
What It Does
discovers every guild the configured bot can access
syncs channels, threads, members, and message history into SQLite
maintains FTS5 search indexes for fast local text search
builds an offline member directory from archived profile payloads
extracts small text-like attachments into the local search index
records structured user and role mentions for direct querying
tails Gateway events for live updates, with periodic repair syncs
exposes read-only SQL for ad hoc analysis
keeps schema multi-guild ready while preserving a simple single-guild default UX
What's new in 0.17.0
0.17.0 - 2026-10-23 Highlights: Keep live Discord archives complete, protect public snapshots from excluded channels, and accept protected credential files for unattended services. - Reconcile bounded exact-message and metadata failures during ordinary repair, expose failures and resumable text-repair progress in status, and preserve vectors when text is unchanged. - Capture supported bulk message deletions, channel/thread tombstones, role changes and thread-list metadata; include voice/stage text in scoped REST repair. - Open every archive and metrics writer connection with FULL synchronization and macOS full-fsync checkpoints, including pool replacements, while keeping read-only connections unchanged. - Retain channel scope metadata independently of content, fail closed on unknown ancestry, and protect public snapshots from excluded ancestors; resolve allowed new channels without a restart. - Honor channel and ancestor tombstones before page, exact-message and attachment requests even without configured exclusions, resolve every scope rejection as a policy outcome, and count only actionable attachment failures in status. Desktop cache imports no longer clear channel tombstones, and text repair keeps author and channel names in multilingual lexical indexes. - Preserve attributed embed fields and whitespace boundaries in derived text; add durable attachment extraction status and retain successful text through transient failures. - Settle channel message failures answered Unknown Channel only after an exact channel lookup corroborates them, as a provider outcome that keeps history and the retry marker; ordinary repair also settles earlier failures superseded by a recorded deletion or exclusion. Public-only snapshots withhold channels with an unavailable marker until a successful crawl re-admits them, and routine syncs no longer panic when only marked channels remain. - Apply Gateway GUILD_CREATE and THREAD_LIST_SYNC channel catalogs in one transaction, so large guilds no longer exceed the tail handler deadline at startup under FULL synchronization or leave a partially applied catalog. - Reconcile full guild members inside the running tail owner with separately bounded pagination waits, without treating cached authors as complete enumeration. - Keep native Components V2 text as attributed parts and fence recovery pages so a strictly newer edit or deletion is never replaced by a delayed page. …
