Skip to Content
Internal docs are powered by Nextra Docs Theme.
Incidents2026UI access outage from API2 organization route collision

UI access outage from API2 organization route collision

Incident date: August 10, 2026

Duration: 40 minutes, approximately 7:18-7:58 AM Eastern Time

Environment: US and EU production

Status: Resolved

Severity: High

Summary

A new API-key-authenticated GET /organization endpoint collided with the UI’s existing session-authenticated endpoint at the same method and path. API2 registered the public router before the internal organization router, so authenticated UI requests reached the API-key middleware first. The middleware interpreted the UI bearer token as an organization API key and returned 401 Unauthorized.

The organization request is part of the application’s bootstrap flow. Users attempting to open the Rulebase UI in US and EU could therefore not access the application for approximately 40 minutes. API2 remained otherwise healthy, and we found no evidence of data loss, unauthorized access, or corruption.

We restored access by reverting the public organization endpoint in PR #9647  and promoting that revert to production in PR #9648 . The effective fix deployed through Flightcontrol because the conflicting route was in API2. The concurrent Cloudflare UI deployment  completed successfully but did not contain the server-side route fix.

Customer impact

  • Users attempting to access the Rulebase UI in US and EU received 401 responses during application bootstrap.
  • Existing sessions did not avoid the failure because API-key middleware rejected their bearer tokens after session authentication.
  • API2 health checks continued returning 200, masking the customer-facing failure from basic availability checks.
  • We found no evidence that background processing, customer data, or unrelated authenticated API2 routes were corrupted.

Detection

The incident surfaced as widespread 401 Unauthorized responses in the UI. Initial investigation focused on recent signup, region-selection, and WorkOS authorization changes because the visible symptom was loss of application access.

Those changes were reverted and deployed through Cloudflare, but access did not recover. Tracing the failing bootstrap request identified GET /organization as the common failure. Comparing the composed API2 route table with the recent API-key work then revealed that 2 routers registered the same method and path with different authentication requirements.

Timeline

All times are Eastern Time.

  • Approximately 7:18 AM: UI access begins failing with 401 Unauthorized responses.
  • 7:20 AM: Investigation begins with recent login and region-selection changes.
  • 7:21-7:30 AM: UI and WorkOS-related changes are reverted.
  • 7:30 AM: The UI rollback is promoted to release-prod, starting the Cloudflare deployments.
  • 7:38 AM: The Cloudflare EU deployment completes successfully.
  • 7:40 AM: The API2 route collision is identified as the actual cause.
  • 7:40 AM: PR #9647  reverts the API-key organization endpoint.
  • 7:40 AM: PR #9648  promotes the revert to release-prod.
  • 7:44 AM: The Cloudflare US deployment completes. Access remains affected because the faulty route is in API2, not the UI bundle.
  • 7:52 AM: Corrected API2 images are available in both regions.
  • 7:57 AM: Traffic begins moving to the corrected API2 tasks.
  • 7:58 AM: Repeated US and EU checks show the public /organization route is gone, and UI access is restored.
  • 7:59 AM: Both regional API2 rollouts are complete.

Root cause

API2 already exposed GET /organization for the UI. The route uses the authenticated application session to resolve the current organization and return the data required during application bootstrap.

PR #9638  introduced another GET /organization route for CLI and external API clients. The new route used externalApiAuthMiddleware to authenticate organization API keys.

Both routers were mounted at /, and the public router appeared first in the composed application:

publicApiV2Router
    GET /organization — organization API key

organizationReadsRouter
    GET /organization — UI session

API2’s global authentication middleware successfully authenticated the UI session. Route dispatch then matched the public endpoint first, whose second authentication layer treated the UI bearer token as an organization API key. That validation failed and returned 401 before the session-authenticated handler could run.

The new endpoint worked in isolation. The failure appeared only when it was composed with the existing router at the same method and path.

CodeRabbit review was disabled for the breaking change, and no CodeRabbit review ran before it merged. This did not create the route collision, but it removed a review control that could have identified the duplicate path and authentication mismatch before deployment. CodeRabbit has now been assigned to the post-mortem PR.

Contributing factors

  • Public API endpoints and internal UI endpoints share an unversioned root namespace.
  • API2 did not detect duplicate method-and-path registrations across routers.
  • Session tokens and organization API keys both use bearer authentication, but their middleware applies different validation rules.
  • Tests covered the new API-key endpoint without exercising the existing session-authenticated endpoint against the fully composed application.
  • The critical UI bootstrap flow did not have an authenticated synthetic check in both regions.
  • /health remained green throughout the incident.
  • The visible authentication symptom initially directed investigation toward recent UI and WorkOS changes.
  • UI and API2 deployments use separate pipelines, which made the Cloudflare rollback appear relevant even though the effective fix required Flightcontrol.
  • CodeRabbit review was disabled, so the change merged without its automated review of the composed routing and authentication behavior.

What went well

  • Reverting the endpoint was low-risk and removed the collision without changing the established UI authentication path.
  • The revert and production promotion were merged within 1 minute of confirming the root cause.
  • US and EU rollout state, task definitions, image tags, and live API behavior were independently verified.
  • Cache-busted production checks exposed the brief mixed-task period during rollout and prevented us from declaring recovery before traffic had moved.

What did not go well

  • The change reused a path that already belonged to a critical UI bootstrap flow.
  • Route-level tests did not detect behavior introduced by router ordering.
  • Basic health checks could not distinguish a healthy process from an unusable authenticated application.
  • Initial mitigations targeted plausible UI authentication changes but did not test the failing organization request directly.
  • The distinction between Cloudflare UI deployment and Flightcontrol API2 deployment was not immediately clear during response.
  • No CodeRabbit review ran before the breaking change merged.

Resolution

PR #9647  removed the public organization router, its API-key middleware registration, and its OpenAPI entry. PR #9648  promoted the revert to production.

Flightcontrol built region-specific API2 images from production commit 052c65460e419126e80f63d2c782bd498c98a3fa and deployed them to US and EU. Traffic briefly included old and new tasks, so the public specification alternated between containing and omitting /organization. We waited for the old task sets to drain before marking the incident resolved.

Production verification

After traffic cut over:

  • https://api2.rulebase.co/health returned 200.
  • https://eu.api2.rulebase.co/health returned 200.
  • 10 cache-busted public OpenAPI requests in US returned no /organization path.
  • 10 cache-busted public OpenAPI requests in EU returned no /organization path.
  • The US API2 service ran 2 corrected tasks with no pending tasks; its old task set had 0 running tasks.
  • The EU API2 service ran its corrected task and reported a completed rollout.
  • UI access was confirmed restored.

Corrective actions

Immediate

  • Revert the conflicting public GET /organization endpoint.
  • Deploy the rollback to US and EU.
  • Confirm the public route is absent from both production OpenAPI specifications.
  • Confirm both regional health endpoints return 200.
  • Confirm UI access is restored.
  • Assign CodeRabbit to review the post-mortem PR.

Route and authentication safety

  • Put API-key endpoints under a dedicated, versioned namespace such as /api/v1/identity.
  • Add a CI check that rejects duplicate method-and-path registrations across composed API2 routers.
  • Add an integration test proving session-authenticated GET /organization still succeeds with all public routers registered.
  • Test API-key and session-token behavior against the fully composed application, not only individual routers.
  • Treat changes to shared authentication middleware and bootstrap routes as high-risk in review and deployment checklists.

Detection and response

  • Add authenticated synthetic monitoring for the complete UI bootstrap flow in US and EU.
  • Alert on elevated 401 rates for bootstrap endpoints, especially GET /organization and session reads.
  • Expose the deployed commit SHA for each production service and region.
  • Document which repository paths deploy through Cloudflare, Rails, and Flightcontrol.
  • Add a post-deploy smoke check that loads organization data with a normal UI session.

Lessons

Authentication boundaries need URL boundaries. A public endpoint using a different credential type should not share an unversioned method and path with an internal application endpoint.

Route-level tests are not sufficient when behavior depends on router order and layered middleware. Authentication changes must be tested against the fully composed application with every supported credential type.

A green process-health endpoint does not prove that customers can use the application. Critical bootstrap flows need authenticated regional monitoring and post-deploy verification.

Last updated on
Conversations index cleanupDemos