Weekly agent performance reports

Moves the weekly agent performance report pipeline from Rails/GoodJob to native API2/BullMQ ownership: the top-level schedule, per-organization fan-out, report lifecycle, content generation, citation persistence, and delivery side effects all execute in api2 against Postgres. Unlike the routine-workload migration, no Rails /internal/* domain endpoints execute this work — report preparation, OpenAI content generation, citation writes, and Resend email delivery are implemented natively in src/lib/agent-performance-reports/.

Contract

One BullMQ queue, agent-performance-reports, carries six job kinds: generate-tick, generate-organization, content, preview-tick, preview-organization, send-organization. Two schedulers preserve the retired crons exactly: generation daily at 08:00 UTC (0 8 * * *) and preview Monday at 09:00 UTC (0 9 * * 1).
The generate tick resolves every org’s current and reportable reporting windows natively — org timezone, agent_performance_reporting_week_start_day (integer enum, Sunday default), and the 48-hour default agent_performance_reporting_resolution_hours — through resolveWindowsForOrganizations, and persists the resolved windows in the tick job’s resolvedWindows data before the first fan-out so a BullMQ retry cannot re-resolve a drifting window.
Deterministic job IDs coalesce duplicate schedules and retries: apr-generate-<org>-<start>-<end>-<type> for per-org generation, apr-content-<report> for per-report content, apr-preview-org-<org>-<scope> / apr-send-org-<org>-<scope> for delivery. Every completed step is written back via job.updateData, so a retry resumes after the last durable marker instead of re-creating reports, re-enqueueing content, or re-sending email.
Per-organization failures are isolated in each tick: a broken org is logged and reported to Sentry without blocking the remaining fan-out, per the api2 background-job isolation rule.
Report rows, report-conversation links, and citations are written natively. Existing reports for the period are deduped by findReportForPeriod; failed reports retry content on the same row, pending/in_progress/completed reports are skipped. Employees with no eligible conversations produce no report and get the weekly latest_agent_performance_report_end_date cache clear.
Content generation calls the OpenAI Responses API directly with inline report instructions and variables, model (gpt-4.1), strict agent_performance_result JSON schema, metadata, and 240-second timeout as the Rails generator. Bot employees are skipped; failures mark the report failed and rethrow so BullMQ retries.
Preview and send emails are claim-guarded per report through public.outbound_delivery_claims (apr-weekly-preview:<scope>:<report> / apr-weekly-email:<scope>:<report>). A proven non-delivery releases the claim so a retry re-sends; an ambiguous outcome stays uncertain, is surfaced to Sentry, and is never resent automatically. Sent reports get emails_sent_at, and the period-level emails_already_sent_for_period gate keeps a re-fired tick from scheduling a second auto-send.
AgentPerformanceReport#generate_later, the admin generate/send/preview actions, and the agent_performance_reports rake tasks enqueue through Api2AgentPerformanceReportEnqueuer onto the API2 queue — Rails produces no GoodJob rows for these paths. The legacy job classes stay deployed for already-enqueued rows and rollback.
The two retired cron entries remain in config/initializers/good_job.rb with enabled_by_default: false and a pause_api2_* kill switch for a non-racing code revert.

Rollback and rollout assertion

The Flipper kill switch pause_api2_agent_performance_reports (owner: platform-eng; created 2026-09-09; review by 2026-12-09) stops the API2 ticks and per-org workers immediately — queued BullMQ jobs complete as skipped/paused rather than racing a reverted Rails path. Content jobs intentionally do not check the flag: generation executes natively in api2 either way, so a paused flag cannot strand pending reports.

Rollback is a two-step, non-racing revert:

Enable pause_api2_agent_performance_reports.
Re-enable generate_weekly_agent_performance_reports and send_weekly_agent_performance_emails_preview in the GoodJob dashboard (or revert the enabled_by_default: false lines).

Report rows created under API2 carry identical status/generate_later semantics, so the Rails path resumes cleanly; a re-delivered period is deduped by the emails_already_sent_for_period gate and per-report emails_sent_at.

Re-driving work skipped while paused

Generation — enqueue one generate-organization job per reportable org via POST /internal/agent_performance_reports/enqueue with the window dates, or revert to GenerateWeeklyAgentPerformanceReportsJob.perform_later once the cron is re-enabled. Rails-side re-drives still enqueue content on API2 through generate_later, so a mixed run cannot produce a GoodJob content row.
Preview/auto-send — POST /internal/agent_performance_reports/enqueue_preview re-runs the tick natively; enqueue_send with auto_scheduled: false performs a manual org send under a fresh manual: delivery scope. The claim keys on the window scope make a duplicate auto-send impossible.
Single report content — report.generate_later or POST /internal/agent_performance_reports/enqueue_content with report_id.

Production observation

Production evidence before cutover (EU+US, rolling 12h ending ~2026-09-09 10:13 UTC): GenerateWeeklyAgentPerformanceReportsForOrganizationJob had 53 executions and GenerateAgentPerformanceReportContentJob about 52; including the top-level scheduler the theme was ~107 executions. After deployment, require for a clean bake:

Zero new normal-path GoodJob rows for GenerateWeeklyAgentPerformanceReportsJob, GenerateWeeklyAgentPerformanceReportsForOrganizationJob, GenerateAgentPerformanceReportContentJob, SendAgentPerformanceEmailsPreviewJob, and SendAgentPerformanceReportEmailsForOrganizationJob:


SELECT job_class, count(*)
FROM good_jobs
WHERE job_class IN (
  'GenerateWeeklyAgentPerformanceReportsJob',
  'GenerateWeeklyAgentPerformanceReportsForOrganizationJob',
  'GenerateAgentPerformanceReportContentJob',
  'SendAgentPerformanceEmailsPreviewJob',
  'SendAgentPerformanceReportEmailsForOrganizationJob'
)
  AND created_at >= now() - interval '12 hours'
GROUP BY job_class;

Rows inside the 12-hour cleanup window created before the deploy are historical, not new traffic.

BullMQ acceptance and drain — schedulers agent-performance-reports-generate and agent-performance-reports-preview produce ticks, org jobs complete, and the queue depth returns to zero:
```
SELECT state, count(*)
FROM api2.bullmq_jobs -- BullMQ keyspace inspection via the queue dashboard
WHERE queue = 'agent-performance-reports'
GROUP BY state;
```
(Equivalently, inspect agent-performance-reports:* keys in the queue Redis or the Bull Board UI.)
Report lifecycle parity — new pending → completed transitions on public.agent_performance_reports for the current windows, with agent_performance_report_conversations and agent_performance_report_citations rows populated:
```
SELECT processing_status, count(*)
FROM public.agent_performance_reports
WHERE created_at >= now() - interval '24 hours'
GROUP BY processing_status;
```
No duplicate deliveries — every outbound claim row is either delivered or a single uncertain surfaced to Sentry; no report shows emails_sent_at twice for the same period:
```
SELECT report_id, count(*)
FROM public.outbound_delivery_claims
WHERE delivery_key LIKE 'apr-weekly-%'
GROUP BY report_id
HAVING count(*) > 1;
```

Hold 24 clean hours across both regions — at least one completed daily generate tick and, on Mondays, the preview tick plus the 30-minute auto-send — before claiming ownership. Roll back at any sign of duplicate reports, duplicate emails, or unclaimed manual sends.