← Index
Wireframe In Progress

Pipeline Manager

Round 1 Created 2026-05-23 Section Operate · Pipelines
Desktop
Operate›Pipelines refresh-benchmarks-monthly Healthy
Pipelines
12
All · 12 Ingest · 5 Transform · 4 Load · 3
refresh-benchmarks-monthly transform · cron 1d 0 1 * * 41m ago
ingest-quanta-feedback ingest · webhook 2h
embed-account-summaries transform · cron 0 */6 * * * 4h
ingest-public-rfps ingest · cron 0 9 * * * 8h
ontology-version-sync transform · on-change 1d
load-warehouse-snapshot load · cron 0 2 * * 0 3d
deprecated-cf-sync disabled · ADR-006 —
Pipeline · pln-refresh-benchmarks-monthly

refresh-benchmarks-monthly

Type · transform Owner · G. Felice Created 2026-02-14 v7 Schedule · 1st of month 01:00 UTC
Config Runs Monitor Permissions Version

Sourcerequired

Where data is read from. Pipeline must read from at least one registered source.

egan-graph · prod (postgres + AGE) ▾
Registered in Sources. Currently 4 sources available.
# Cypher — read accounts updated in the last 30d MATCH (a:Account) WHERE a.updated_at > datetime() - duration('P30D') RETURN a
Cypher or SQL accepted. Parameters bind from vars.
cohort_id: "bm-122" window_days: 30 include_inactive: false

Transform2 steps

Ordered steps run sequentially. Output of step N is input to step N+1.

1
aggregate-by-cohort function · piggie.aggregate · group_by=industry, sub_industry
2
compute-percentile-bands function · egan.benchmarks.percentiles · bands=[25, 50, 75, 90]
+ Add step (form-based; visual DAG in v3)

Destinationrequired

Where transformed output lands.

egan-graph · benchmark_cohorts (write) ▾
replace append upsert merge
cohort_id + window_start
Reruns with the same key replace prior output. Required for replace/upsert modes.

Schedule

When the pipeline runs. Combine cron + event triggers.

manual hourly daily monthly custom cron
0 1 1 * *
1st of every month at 01:00 UTC · next run 2026-06-01 01:00 UTC
on source update on upstream completion on webhook
No event triggers active. Pipeline runs strictly on cron.

Reliability

3 attempts, exponential backoff (1m / 5m / 25m)
30 minutes
alert · ops-high page on-call auto-disable rollback
Status
statehealthy last run2026-05-01 01:00 UTC duration14 min next run2026-06-01 01:00 UTC success rate98.4% · 90d avg duration12 min · 90d last failure2026-02-12 (config error)
Recent runs
2026-05-01 Completed 3 rows updated · cohort bm-122 14m
2026-04-01 Completed 2 rows updated 11m
2026-03-01 Completed 3 rows updated 13m
2026-02-12 Failed · config error missing cohort_id var 12s
2026-02-01 Completed 10m
Dependencies
Reads from
· egan-graph · prod
· source · public-rfp-feed
Writes to
· egan-graph · benchmark_cohorts
Triggers
· embed-account-summaries (downstream)
Permissions
readdata-scientist, ops editops, owner runops