WorkOS event polling

ENG-2858  transfers the ten-minute WorkOS Events poll and retry budget from Rails’ recurring SyncWorkOSEventsJob to API2’s maintenance BullMQ queue. API2 also owns the durable event ledger and all identity, organization, membership, invitation, permission and role effects; the normal path no longer calls Rails.

Contract

• Request the same 17 supported event types as Rails, using one repeated events query parameter per type and a page limit of 100.
• Parse WorkOS’ raw JSON instead of SDK-normalized objects so snake-case payload keys reach the native handlers unchanged.
• Persist each event before applying its effect. A failed effect records its attempt and error durably, so the shared cursor can advance without one poison event blocking later tenants. Allow five total processing attempts from the ledger; retain the event after its fifth failed attempt for inspection and manual resolution.
• Make replay safe with a transaction-scoped advisory lock, a ten-minute processing lease, and the unique WorkOS event ID. The worker renews the lease during external work and fences local writes with the claimed attempt. A matching processed event is a no-op; a matching retryable event resumes processing; the same ID with different content fails without advancing the cursor.
• Apply all 17 effects in API2: initialize organizations and Grant customers; create/update accounts; create, update, discard and restore memberships and linked employees; accept invitations; synchronize and delete assignable permissions and roles; and preserve system audit history.
• Perform WorkOS role lookups and Grant customer reconciliation outside database transactions. Organization initialization commits before Grant provisioning, and retries reuse the committed organization slug as Grant’s stable external ID.
• Update the cursor only when its previous value still matches, so overlapping poll attempts fail retryably instead of overwriting newer progress.
• Retry the BullMQ job five total times with exponential backoff. Retain failed jobs and fifth-attempt event rows for inspection, alert on retained failures, and do not log complete event payloads.

The Rails cron entry is removed in the same candidate release. Keep the Rails job class and payload classes for already-enqueued work and code rollback only.

Verification

Focused API2 tests pin the full event filter, raw WorkOS request shape, processing-before-cursor ordering, empty pages and partial failures. PGlite tests exercise event-ledger idempotency and native organization, account, membership, invitation, permission, role and audit effects. API2 typecheck and scoped Rails lint are required before push.

For an inspected and corrected retained failure, clear its bounded-processing state without changing its immutable envelope; the next scheduler run will retry it:

UPDATE public.workos_events
SET processing_attempts = 0,
    processing_started_at = NULL,
    last_processing_error = NULL,
    last_processing_error_at = NULL,
    failed_at = NULL,
    updated_at = now()
WHERE workos_event_id = '<verified-event-id>'
  AND processed_at IS NULL
  AND failed_at IS NOT NULL
RETURNING workos_event_id, type;

Treat this as a production mutation requiring the usual separate authorization.

Read-only preflight must find exactly one non-empty public.workos_event_cursors row in each production region. The September 7 preflight found one in US and one in EU; it made no database changes.

Rollout and rollback

Deploy API2 and Rails from the same revision. Before starting the API2 scheduler, confirm Rails has stopped enqueueing SyncWorkOSEventsJob, allow active runs to finish, and drain or cancel every queued instance. Confirm no Rails poller can start while exactly one sync-workos-events-ten-minute scheduler runs per regional Redis instance. Observe real identity, membership and RBAC events in each region, their native event rows and effects, cursor movement after success, bounded retry/retained-failure counts, and no Rails callback, payload conflict or duplicate effect. Require clean checks at 30, 60 and 120 minutes and through 24 hours before recording production ownership.

Block rollback while any retained public.workos_events rows exist: inspect and release them through the guarded procedure above, then let compatible API2 workers replay them to completion. Remove the persisted BullMQ scheduler, allow active API2 jobs to drain, confirm no unprocessed or retained event rows remain, and only then revert the revision so the Rails cron returns. Do not manually run both pollers, edit the shared cursor or event rows, or purge BullMQ or GoodJob work.