Skip to Content
Internal docs are powered by Nextra Docs Theme.
SystemsPlatformAPI2Rails to API2 migrationPackagesDaily provider field-definition sync

Daily provider field-definition sync

ENG-3010  moves the 02:00 UTC SyncProviderFieldsJob sweep and its per-source SyncProviderFieldsForOrganizationDataSourceJob fan-out from Rails/GoodJob to the API2 provider-fields-sync BullMQ queue. The sync keeps the Rails conversation_fields, customer_fields, employee_fields, and company_fields rows current with each provider’s field definitions so imported conversations, customers, employees, and companies can attach field values.

Why

The production CloudWatch census for the 24 hours ending 2026-09-10 22:48 UTC recorded 12 US and 6 EU SyncProviderFieldsForOrganizationDataSourceJob attempts plus one SyncProviderFieldsJob sweep per region. The workload is small but it is a recurring normal-path GoodJob producer, and its provider clients, credential loaders, and Rails-mirrored field schemas already exist in API2.

Contract

  • One scheduler, provider-fields-sync-daily, fires a scan job at 0 2 * * * UTC (the existing cadence). The scan lists non-discarded Rails organization data sources whose data_sources.key is zendesk, freshdesk, or intercom — without a tenant, as ActsAsTenant.without_tenant did — and enqueues one source job per row carrying organizationId, organizationDataSourceId, and provider. An enqueue failure for one source is logged, the remaining sources are still enqueued, and the scan fails afterwards so the failure is visible.
  • A source job re-checks the kill switch, reloads the source scoped to its organization (a discarded or re-typed source completes as skipped), and then syncs each provider category independently:
    • Zendesk ticket_fields → conversation fields (name = description, else title); user_fields → customer and employee fields (name = key); organization_fields → company fields (name = key). Enum options keep custom_field_options[].value as values and .name as labels.
    • Freshdesk /api/v2/admin/ticket_fields → conversation fields; /api/v2/contact_fields → customer fields; /api/v2/company_fields → company fields. Enum values and labels are both choices[].value.
    • Intercom data_attributes?model=contact|company|conversation → customer, company, and conversation fields; only custom and non-archived attributes are kept, and any options force the enum type with identical values and labels.
    • Value-type tables match the Rails class helpers exactly; unsupported provider types are skipped, not persisted.
  • Field identity is (organization_data_source_id, external_id). A re-run updates label, name, value_type, options, and last_synced_at in place; a new row takes its id from the Rails sequence and conversation fields receive the Rails-style conversation_field_… public id.
  • A category failure (credentials, provider error) is logged with the organization, source, and category ids and does not stop sibling categories; a field that cannot be mapped or persisted is logged and does not stop its siblings. The job only throws — and BullMQ retries it — when every category failed, which is the credential/provider-outage case.
  • Deduplication id provider_fields_source_<org>_<ods> keeps one live source job per data source. Source jobs have 3 attempts with exponential backoff from 60s; the scan has 1 attempt and is retried by the next daily tick.
  • The Rails sync_provider_fields cron stays in good_job.rb with enabled_by_default: false. SyncProviderFieldsJob and SyncProviderFieldsForOrganizationDataSourceJob are a documented recovery-only shim.

Kill switch and rollback

Flipper pause_api2_provider_fields_sync is checked at scan and source execution time; paused jobs complete as paused without calling a provider or touching the field tables. Rollback order:

  1. Enable pause_api2_provider_fields_sync in the affected region.
  2. Only then re-enable the sync_provider_fields cron in the GoodJob dashboard (or revert enabled_by_default: false). Never run both owners: both upsert the same field rows.
  3. Leave the API2 worker and queued BullMQ jobs alone; they drain as paused.

Work skipped while paused is covered by the next sweep because the sync is a full re-read of every provider’s current definitions.

Bake criteria

Deploy the API2 worker/scheduler and the Rails revision together (overlap is safe: the upsert is idempotent per field identity). Then observe two 02:00 UTC ticks in US and EU and require:

  • every kept Zendesk/Freshdesk/Intercom source logged provider fields sync completed with error: null on each category, or an explained failure;
  • last_synced_at advanced on the field rows of every such source;
  • zero new SyncProviderFieldsJob / SyncProviderFieldsForOrganizationDataSourceJob GoodJob rows after the deploy;
  • no duplicate (organization_data_source_id, external_id) rows and an empty or fully explained provider-fields-sync failed set.

Observation queries

Rails (each region):

SELECT 'conversation' AS kind, organization_data_source_id, max(last_synced_at)
FROM conversation_fields WHERE organization_data_source_id IS NOT NULL
GROUP BY 1, 2
UNION ALL
SELECT 'customer', organization_data_source_id, max(last_synced_at)
FROM customer_fields WHERE organization_data_source_id IS NOT NULL GROUP BY 1, 2
UNION ALL
SELECT 'employee', organization_data_source_id, max(last_synced_at)
FROM employee_fields WHERE organization_data_source_id IS NOT NULL GROUP BY 1, 2
UNION ALL
SELECT 'company', organization_data_source_id, max(last_synced_at)
FROM company_fields WHERE organization_data_source_id IS NOT NULL GROUP BY 1, 2
ORDER BY 3 NULLS FIRST;
 
SELECT job_class, count(*) AS jobs, sum(executions_count) AS executions
FROM good_jobs
WHERE job_class IN ('SyncProviderFieldsJob', 'SyncProviderFieldsForOrganizationDataSourceJob')
  AND created_at > now() - interval '24 hours'
GROUP BY job_class;

The second query must return no rows after the deploy. API2: open /admin/queues, select provider-fields-sync, and read waiting, failed counts and the scheduler’s next time. Worker logs are tagged provider-fields-sync with provider, organizationId, organizationDataSourceId, and per-category fetched, synced, skipped, fieldErrors, and error.

Recover a failed source

A source job that exhausted its attempts keeps organizationId, organizationDataSourceId, and provider in the failed set. With production authorization: match it to its final-attempt log/Sentry event, fix the credential or provider cause, and retry the individual job from /admin/queues. The next daily tick also re-syncs it. Run the Rails shim manually for one source only while pause_api2_provider_fields_sync is enabled, then unpause.

Verification

src/lib/provider-fields-sync/mappings.test.ts covers every Zendesk, Freshdesk, and Intercom value-type mapping, enum option shapes, the Intercom custom/non-archived filter, and Rails presence validations. sync.test.ts covers scan fan-out, pause at scan and source, per-category and per-field error isolation, the all-categories-failed retry path, deduplication and retry options, the 02:00 UTC scheduler registration, worker lifecycle, and the paused Rails cron. sync.db.test.ts runs the scan and source sync against PGlite copies of the Rails tables, asserting tenant/source filtering, the Flipper kill switch, and idempotent upserts on all four field tables. Rails GoodJob configuration specs assert the disabled cron entry.

Last updated on
Organization directory Rails-client retirementPublic route ownership contract