Skip to Content
Internal docs are powered by Nextra Docs Theme.
ProjectsQA knowledge-base chunking

QA KB Chunking and Retrieval: Phased Implementation Plan

Date: 2026-04-07 Status: Draft plan for implementation Scope: Replace OpenAI file_search-backed QA KB/feedback retrieval with app-managed chunk retrieval, ending with integration in the single-conversation QA evaluation pipeline.

Conversation review: key decisions to carry forward

  1. Latency is the primary problem to solve now. Current file_search behavior can produce very high time-to-first-token and unpredictable runtime.
  2. Do the simple architecture first. Keep retrieval in Rails/Postgres, and use Modal (rulebase-ml) only for heavy ingestion work (HTML -> markdown -> chunk -> embed).
  3. Use OpenAI embeddings for now. Standardize on text-embedding-3-large from the start (no text-embedding-3-small) and avoid adding another vendor at this stage.
  4. Use structure-aware chunking. Convert HTML to markdown, preserve heading hierarchy, chunk by section/element boundaries instead of arbitrary token windows.
  5. Contextual retrieval/prefix generation is explicitly deferred. Ship baseline retrieval first; evaluate advanced retrieval only after we have quality/latency measurements.

Current state in Ruby (baseline)

  • KnowledgeBase currently queries OpenAI vector stores via responses.create with file_search.
  • QA tools currently depend on this path:
    • ToolCalls::QAEvaluation::AskKnowledgeBase
    • ToolCalls::QAEvaluation::AskFeedbackVectorStore
    • ToolCalls::QAEvaluation::AskFeedbackStore
  • QA evaluation runs in the single-conversation flow (Evaluations::RunConversationQAEvaluationJob), with per-agent evaluation jobs as the fallback.
  • Expanded criterion feedback comments are generated and then uploaded to feedback vector store (EvaluationScorecardCriterionResultComment).

This plan keeps the runner model (single conversation evaluated end-to-end) and changes retrieval internals beneath it.

Phase 0: Design and instrumentation (short prep)

Goals

  • Freeze API contracts and data model before implementation.
  • Add baseline latency and quality instrumentation for before/after comparison.

Deliverables

  • Contract doc for chunking/embedding Modal function I/O (single-article request/response).
  • Retrieval metrics schema (query latency, embedding latency, retrieved chunk count, fallback path used).
  • Embedding spec captured in contract (model: text-embedding-3-large, expected dimensions, and re-embed strategy for future model changes).
  • Success metrics agreed:
    • p50/p95 retrieval latency per tool call
    • QA pipeline runtime impact
    • citation quality checks (manual spot checks to start)

Phase 1: Modal service for KB chunking + embeddings (using rulebase-ml)

Goals

  • Introduce a stateless Modal function in rulebase-ml that processes one KB document payload at a time.
  • Return chunk payloads (with embeddings) back to Rails; Rails remains the only writer to Postgres.

Implementation

  • Build a function that accepts:
    • source document identity (knowledge_base_document_id or external reference),
    • locale/source metadata,
    • raw HTML content,
    • source timestamps/version markers.
  • Function pipeline:
    1. HTML -> markdown conversion
    2. Cleanup/normalization
    3. structure-aware chunking (heading/section aware, table/list safe)
    4. batch embedding call using text-embedding-3-large
    5. return chunk array with metadata + vector
  • Include deterministic chunk ordering (chunk_index) and token counts.
  • Add idempotency hash/version field in output so Rails can skip no-op rewrites.

Out of scope (for now)

  • Hybrid BM25 ranking
  • contextual chunk prefix generation
  • reranking models

Phase 2: Rails KB chunk persistence and retrieval integration

Goals

  • Persist KB chunks in Postgres and retrieve them directly from Rails.
  • Keep knowledge_base_documents as parent records and attach chunks via association.

Data model and migrations

  • Create kb_chunks table with (at minimum):
    • organization_id (tenant required)
    • knowledge_base_document_id (required FK)
    • locale
    • heading_path
    • content
    • chunk_index
    • token_count
    • embedding (pgvector, 3072 dims for text-embedding-3-large, e.g. t.vector :embedding, limit: 3072)
    • timestamps
  • Add retrieval-oriented indexes:
    • tenant/document indexes for fast upsert+lookup
    • vector index (hnsw, cosine ops)
  • Add Rails model:
    • KbChunk with acts_as_tenant :organization
    • belongs_to :knowledge_base_document

Sync and lifecycle

  • Add callback/job path from KnowledgeBaseDocument updates to enqueue chunk refresh.
  • Replace vector-store file upload path for KB docs with:
    • call Modal chunker
    • transactionally replace chunks for that document
  • Add maintenance task to backfill all existing KB documents into kb_chunks.
    • idempotent
    • batched
    • resumable and safe for long-running execution

Tool integration

  • Update ToolCalls::QAEvaluation::AskKnowledgeBase to:
    • embed the query
    • retrieve top-k chunks from kb_chunks with existing filters (document_ids, locale/brand equivalents as applicable)
    • return chunk-first payload quickly (not a long synthesized narrative from external file_search)

Phase 3: Criterion feedback expanded comments -> feedback chunks

Goals

  • Apply the same chunk/index approach to feedback memory (expanded_comment) currently uploaded to feedback vector store.

Implementation

  • Add feedback_chunks table (or shared chunk table with source type; choose one and keep query simplicity high).
  • Replace EvaluationScorecardCriterionResultComment#upload_to_feedback_vector_store flow with:
    • generate/refresh chunks + embeddings for expanded_comment
    • store tenant-scoped chunks with scorecard/criterion metadata fields used by retrieval filters.
  • Keep existing metadata filters (scorecard_id, criterion_id) as first-class indexed fields.
  • Backfill existing expanded comments with a maintenance task.

Tool integration

  • Update AskFeedbackVectorStore / AskFeedbackStore to query local chunk tables instead of OpenAI vector store.
  • Return top chunks + metadata/citation anchors, not only synthesized narrative text.

Phase 4: QA conversation runner adoption (single-conversation pipeline)

Goals

  • Make chunk retrieval the default path for the current single-conversation QA evaluation pipeline.
  • Remove dependency on OpenAI vector stores in the hot evaluation path.

Integration points

  • Keep Evaluations::RunConversationQAEvaluationJob as the orchestration boundary.
  • Ensure evaluator tool calls resolve through Rails chunk retrieval for both:
    • KB document context
    • feedback memory context
  • Preserve current scorecard/criterion-level behavior, but retrieval now executes quickly against local chunks.

Rollout strategy

  • Feature flag per organization:
    • local_kb_chunk_retrieval_enabled
    • local_feedback_chunk_retrieval_enabled
  • Shadow mode:
    • run old and new retrieval in parallel for selected traffic
    • compare citations/relevance and latency
  • Progressive rollout:
    • internal orgs -> low-risk orgs -> high-volume orgs

Phase N: Decommission and simplification

Goals

  • Retire old vector-store-only paths once confidence is established.

Tasks

  • Remove obsolete OpenAI vector store dependencies for QA retrieval paths.
  • Keep temporary fallback switch for incident response window.
  • Update runbooks, alerts, and dashboards to new retrieval architecture.

Risks and mitigations

  • Retrieval quality regression: mitigate with shadow runs, manual citation audits, and conservative rollout flags.
  • Chunking quality on messy HTML/tables: add fixture-based tests from real docs and table-specific handling rules.
  • Operational backfill load: use maintenance tasks with batching/throttling and progress visibility.
  • Model mismatch over time: store embedding model metadata and support controlled re-embed migrations.

Suggested sequencing and ownership

  1. Phase 0 + Phase 1 (platform/ML + API pairing)
  2. Phase 2 (API/data infra)
  3. Phase 3 (feedback pipeline owners)
  4. Phase 4 (QA evaluation pipeline owners + staged rollout)
  5. Phase N cleanup after stable production confidence

Definition of done for this initiative (initial target)

  • QA evaluations for a single conversation complete using local chunk retrieval for KB + feedback tools.
  • p95 retrieval latency is materially lower than current file_search path.
  • Citation quality is acceptable in rollout cohorts.
  • Backfill complete for existing KB docs and expanded feedback comments.
  • Legacy QA retrieval path can be disabled by default.
Last updated on
Legacy data views removalPlan