Knowledge Library sync pipeline

This package moves the Rails Knowledge Library synchronization pipeline from GoodJob to API2/BullMQ. The rolling 12-hour production window ending September 9, 2026 at ~10:13 UTC showed SyncKnowledgeBaseDocumentJob with 2,629 executions for 158 unique jobs and 2,488 US GoodJob throttle failures, plus 113 SyncKnowledgeBaseDocumentChunksJob executions and residual SyncKnowledgeBaseSourcesJob, SyncKnowledgeBaseSourceJob, StartKnowledgeBaseCrawlJob, and CrawlKnowledgeBaseUrlJob rows. The storm was GoodJob perform_throttle retrying each throttled job as a failure, not provider load.

Ownership boundary

The pipeline writes Rails Knowledge Library records only: knowledge_base_sources, knowledge_base_documents, knowledge_base_document_sections, knowledge_base_sections, knowledge_base_crawl_tasks, document_chunks (source_type = 'KnowledgeBaseDocument'), and pg_search_documents. API2-native agent/workflow knowledge tables under the api2 schema are a different product surface and are not touched by this package.

Rails producers

Every normal-path Rails producer now calls KnowledgeBaseSyncDispatch instead of perform_later:

• KnowledgeBaseSource#sync_later → sync_source
• KnowledgeBaseDocument#sync_later / sync_chunks_later (after-commit callbacks on source-identifier and content changes) → sync_document / sync_document_chunks
• KnowledgeBaseCrawlTask#start_crawl_later / child URL scheduling → start_crawl / crawl_url

KnowledgeBaseSyncDispatch posts each request to POST /internal/knowledge_base_sync/enqueue on API2 and treats only a 202 as durable acceptance. On any non-2xx, timeout, or transport error it logs, reports to Sentry as a warning, and enqueues the retained GoodJob job for that unit, so no work is lost while API2 is unreachable. Provider execution no longer passes through Rails: the native runners write the Rails tables directly and return follow-up requests to the worker, so a BullMQ retry of a parent unit re-derives the same deterministic child ids instead of fanning out duplicates. The Rails KnowledgeBaseSource#sync / KnowledgeBaseDocument#sync / crawl-task provider code is retained only as the GoodJob rollback path.

Durability and rate limits

• Deterministic job ids: kb-source-sync-<id>, kb-document-sync-<id>, kb-document-chunks-<id>, kb-crawl-start-<task>, kb-crawl-url-<task>-<sha256(url)[0:24]>. A request whose job is already waiting/active is a no-op; a completed/failed job of the same id is replaced so the work runs again (addReplacingFinishedJob).
• Retries: three attempts, exponential backoff from 60s. A missing or cross-tenant record → job completes as record-missing; provider 4xx other than 429, unknown source types, and invalid credentials → KnowledgeBaseSyncUnrecoverableError → BullMQ UnrecoverableError (no retry); 429/5xx/transport → retry. A provider 404 for a single document discards that document (as Rails did) rather than failing.
• Provider windows (Redis fixed-window counters, shared across workers, keyed exactly like the retired GoodJob perform_throttle keys): source sync Notion 4/minute, others 20/second; document sync Notion 4/minute, Scribe 50/minute, others 3,000/minute per organization data source. A full window delays the job (moveToDelayed + DelayedError) rather than failing it, so throttling never consumes a retry attempt — the exact failure mode behind the 2,488 throttle errors.
• Chunk worker: BullMQ queue limiter of 60 jobs/minute (the old SyncKnowledgeBaseDocumentChunksJob 1/second throttle), concurrency 4.
• Tenant isolation: every request carries organization_id; every native SQL statement (source/document/section/crawl-task/data-source lookups and writes) filters on organization_id. A cross-tenant id completes as record-missing rather than acting.
• Failure isolation: one unit per job; a failing source, document, or URL fails only its own job. The daily tick enqueues each request under Promise.allSettled and logs per-request failures.
• Durable follow-ups: a unit’s Rails writes commit before its children are enqueued, so a failed enqueue fails the job and the retry re-derives the same requests from a ledger instead of trusting the provider’s “unchanged” answer. Source runs do not advance last_synced_at for listed documents — the document sync does once content is stored — and runSourceSync appends every kept document with last_synced_at IS NULL (Notion/Confluence/Scribe) or chunks_synced_at IS NULL to its follow-ups; runDocumentSync re-reads its own row after every run and re-emits document_chunks while chunks_synced_at IS NULL with content, so a document with no source still recovers from a failed chunk enqueue on its own retry. Content changes clear chunks_synced_at, which is also what the daily recovery sweep scans. Crawl tasks keep a PostgreSQL ledger, api2.knowledge_base_crawl_task_urls (unique per (organization, task, url), with parent_url, depth, finished_at): the claim row and the urls_pending increment commit in one transaction, and finished_at plus the decrement/completion commit in one transaction (success, 4xx, or the final BullMQ attempt — never on a retryable failure), so no failure window exists between “marked finished” and “counted”. A retried parent re-emits the children it claimed, and a re-run finished URL replays its follow-ups without fetching or decrementing again, so urls_pending cannot strand, go negative, or complete early.
• Snapshot-safe removal: Zendesk discards documents/sections only against a complete listing. A failed article page aborts the run (job retries, nothing removed); a failed translation fetch keeps that article’s known locales and skips the removal pass for the run.

Pause and rollback

• Kill switch: Flipper pause_api2_knowledge_base_sync (read by both Rails and API2). Owner: engineering. Purpose: rollback for the Knowledge Library pipeline migration. Created 2026-09-09; review/remove by 2026-12-08 after bake-in. Safe state is off (API2 owns the pipeline).
  • Rails: with the flag on, KnowledgeBaseSyncDispatch enqueues the retained GoodJob jobs directly and never calls API2.
  • API2: the daily tick and both workers re-check the flag at execution time; queued jobs complete as skipped/paused rather than racing the Rails path.
• Cron: sync_knowledge_base_sources stays in good_job.rb with enabled_by_default: false. Rollback = enable the flag, then enable the cron from the GoodJob dashboard (or revert the line).
• Retained jobs: all six *KnowledgeBase*Job classes remain deployed with their original throttles, so already-enqueued rows drain and the pause fallback works.

Re-driving work after a pause

Skipped BullMQ jobs do not resume on their own. Once the flag is off again:

• The next 01:00 UTC tick re-scans every source and sweeps documents with content but no chunks, which covers the common case.
• To catch up immediately, run the scan by hand from an API2 shell: await runKnowledgeBaseSyncTick() from dist/lib/knowledge-base-sync/tick.js, or enqueue specific units via POST /internal/knowledge_base_sync/enqueue with the internal bearer token and a requests array.
• From Rails, KnowledgeBaseSyncDispatch.sync_source(source) (or sync_document, sync_document_chunks, start_crawl) re-dispatches a single unit through the normal path.

Production observation

Expected after cutover in each region: zero new normal-path rows for the six classes (rows inside the 12-hour retention window created before the deploy are historical), a bounded knowledge-base-sync queue that drains after the 01:00 UTC tick, no DelayedError-induced failures, and existing GoodJob rows completing or being recoverable.

-- New GoodJob rows since the deploy (should be 0 unless the pause flag is on)
SELECT job_class, count(*) AS jobs,
       min(created_at) AS first_seen, max(created_at) AS last_seen
FROM good_jobs
WHERE job_class IN (
  'SyncKnowledgeBaseSourcesJob', 'SyncKnowledgeBaseSourceJob',
  'SyncKnowledgeBaseDocumentJob', 'SyncKnowledgeBaseDocumentChunksJob',
  'StartKnowledgeBaseCrawlJob', 'CrawlKnowledgeBaseUrlJob')
  AND created_at > '<deploy timestamp UTC>'
GROUP BY job_class ORDER BY jobs DESC;
 
-- Throttle-storm check: executions vs unique jobs in the retained window
SELECT j.job_class, count(e.*) AS executions, count(DISTINCT j.active_job_id) AS unique_jobs,
       count(*) FILTER (WHERE e.error ILIKE '%throttle%') AS throttle_failures
FROM good_job_executions e JOIN good_jobs j ON j.active_job_id = e.active_job_id
WHERE j.job_class LIKE '%KnowledgeBase%' AND e.created_at > now() - interval '12 hours'
GROUP BY j.job_class;
 
-- Recovery-sweep backlog: kept documents with content but no chunks
SELECT count(*) FROM knowledge_base_documents
WHERE discarded_at IS NULL AND chunks_synced_at IS NULL
  AND content IS NOT NULL AND content <> '' AND source_type <> 'intercom';
 
-- Sources still failing after the daily scan
SELECT id, organization_id, type, sync_status, last_synced_at
FROM knowledge_base_sources
WHERE type <> 'intercom' AND (sync_status = 'failed' OR last_synced_at < now() - interval '2 days')
ORDER BY last_synced_at NULLS FIRST LIMIT 50;

Flipper state: Flipper.enabled?(:pause_api2_knowledge_base_sync) in a Rails console (expected false). BullMQ state: knowledge-base-sync and knowledge-base-chunks queue depths and delayed counts in the API2 queue dashboard; a growing delayed count on a Notion-heavy tenant is expected throttling, a growing failed count is not.

Bake and rollback gates

1. Deploy both services; confirm the daily tick ran once per region (knowledge-base-sync worker scan done logs, sources’ last_synced_at advancing) and that the 01:00 UTC GoodJob cron did not run.
2. 24 clean hours: zero new normal-path GoodJob rows for the six classes, zero Api2KnowledgeBaseSyncEnqueuer::DeliveryError Sentry warnings that indicate fallback traffic, no duplicate document_chunks sets, and no cross-tenant Sentry events.
3. Roll back immediately (flag on, cron enabled) if the fallback warning fires continuously, chunks stop being written, or provider limits are exceeded upstream (Notion/Scribe 429s).
4. After a clean bake in both regions, remove the flag branch, the cron entry, and the six GoodJob job classes in a follow-up.

Verification

API2: unit tests for enqueue mapping/dedupe/deterministic ids, job execution (pause re-check, provider windows, delayed-not-failed throttling, 404/4xx/5xx handling, follow-up enqueue), tick fan-out and failure isolation, and the internal enqueue route (auth, validation, no 202 on Redis failure); PGlite tests for the candidate/recovery SQL, tenant-checked rate-limit scopes, section-path loading, and the chunk replacement transaction (idempotent, tenant-scoped, search index refreshed). Rails: dispatcher specs (API2 handoff with organization scope, no GoodJob rows on success, pause and delivery fallbacks) and image-endpoint specs. Native providers are covered by providers/providers.db.test.ts (PGlite Rails tables): Notion cursor pagination/auth/reclaim/discard/rendering/4xx-vs-429, Zendesk brands, translations, recursive sections, locale fallback, markdown; API-key Scribe offset pagination and session-token no-op; URL crawl sanitising, same-host links, depth/URL caps, urls_pending completion, replayed parents and finished URLs, once-only decrement on 4xx/final attempt, Zendesk failed first/later page and translation fetches leaving old content in place; and run.ts status transitions plus the enqueue-failed-then-retry replay of document/chunk follow-ups. pnpm typecheck, pnpm test, and scoped RuboCop are required before push.