Skip to content

Platform ingestion and rendering

Platform behavior has two owners: Core acquires and normalizes content; Web renders canonical resources. They share the Resource.type vocabulary but not a runtime registry or contracts package.

source cron or saved URL
-> ingest/acquisition.ts
-> ResourceProcessingWorkflow
-> ingest/platforms/*
-> canonical resources/resource_translations
-> AI Search sync

Relevant files:

  • workers/core-worker/src/index.ts: cron dispatch;
  • workers/core-worker/src/ingest/acquisition.ts: saved-URL detection and acquisition routing;
  • workers/core-worker/src/ingest/platforms/: platform-specific durable processing;
  • workers/core-worker/src/ingest/domain/resource-store.ts: canonical durable resource reads/writes;
  • workers/core-worker/src/ingest/resource-persistence.ts: enrichment and entity persistence.

Monitored source platforms are RSS, Twitter, and YouTube. Other canonical kinds such as web, Hacker News, PDF, image, and file enter through their owning acquisition path. paper is academic enrichment that can apply to an existing resource type; it is not another type.

To add a platform:

  1. implement durable processing in ingest/platforms/;
  2. add an acquisition branch when saved URLs should resolve to it;
  3. add only the required cron line when it has a monitored source;
  4. persist provider-only payload in platform_metadata;
  5. keep origin policy in library.origin_type and monitored provenance in resources.source_id.

There is no Core platform registry to update.

Web receives canonical resource cards from the shared read layer, then chooses specialized presentation in:

  • web-tanstack/src/components/feed/platformRegistry.tsx for card bodies and masonry height estimates;
  • web-tanstack/src/components/feed/platformMetadata.ts for safe provider payload readers;
  • web-tanstack/src/components/article/ArticleContent.tsx for detail content.

The ordered PLATFORMS table is the single UI dispatch table. Twitter and YouTube match on canonical resource.type; academic paper rendering uses an explicit enrichment predicate. Default resources use the default card.

Do not dispatch from platform_metadata.type. resources.type is canonical, while metadata contains source-specific data only.

  • Run Core lint/typecheck for every ingest change.
  • Use resyncResource to exercise acquisition → enrichment for an existing resource whose source changed.
  • Use wrangler dev --test-scheduled to exercise cron discovery and batch error isolation; resync does not cover cron.
  • Tail logs and compare enrichment_status='failed' counts before and after production smoke tests.