Skip to content
ArchLang

26. An Event-Driven Pipeline

The previous chapter modeled a request/response SaaS backend with sync calls as the dominant style. This chapter takes a system shaped the other way: an event-driven data pipeline where almost every interaction is async, where the architectural questions are different — and where the temptation to draw the broker as a node in the middle is strongest.

Specifically: an analytics pipeline. Raw events come in from product surfaces, get enriched with user attributes, get aggregated, land in a warehouse, and refresh a real-time dashboard. The shape that matters: the flow is a chain of producer-owned events, and each consumer couples to the producer it reads from. The broker that physically carries every hop is real — but it lives on a separate plane.

What we’re modeling

Section titled “What we’re modeling”
ProductApp.rawEvents  ─┐
StoreFront.rawEvents  ─┴─► Enricher        (consumes both)
                           Enricher.enrichedEvents ─┬─► Aggregator ─► Metrics.push
                                                    └─► Warehouse
                                                        Warehouse.landedEvents ─► Dashboard


   Every event interface carries  aspect broker: EventStream
   EventStream (kafka_cluster) is on the infra plane — reached by aspect, not by an edge.

Seven modules: two emitters (ProductApp, StoreFront), one enricher, one aggregator, a metrics surface, a warehouse, and a dashboard. Four event interfaces, each owned by its producer (rawEvents ×2, enrichedEvents, landedEvents). Every cross-module link is async, and the process below carries the whole flow.

Package layout

Section titled “Package layout”
analytics/
├── package.archspace
├── sources.arch         # ProductApp, StoreFront (event emitters)
├── pipeline.arch        # Enricher, Aggregator, Metrics (transformers)
├── sinks.arch           # Warehouse, Dashboard
├── infra.arch           # EventStream (shared broker, infra plane)
├── processes.arch       # the ingest flow + admin touchpoints
└── views.arch

Manifest:

package: acme.analytics
version: "0.1.0"


// Explicit working set — never `use *` the stdlib.
use frontend, service, database, message_broker, kafka_cluster,
    kafka, grpc_unary, rest_read, db_read
    from arch.backend
use user from arch.extras

The modules

Section titled “The modules”

sources.arch:

frontend #pa31 ProductApp {
    aspect team: "Product"
    repo.url: "https://github.com/acme/product-app"
    "Customer-facing product. Emits clickstream and lifecycle events."


    aspect {
        domain: "Analytics"
        security.zone: "Internal"
    }


    kafka rawEvents {
        "page_view, click, purchase, signup, churn"
        aspect {
            broker: EventStream
            topic: "analytics.product.raw.v1"
        }
    }
}


frontend #sf72 StoreFront {
    aspect team: "Storefront"
    repo.url: "https://github.com/acme/storefront"
    "E-commerce frontend. Emits its own clickstream into the same pipeline."


    aspect {
        domain: "Analytics"
        security.zone: "Internal"
    }


    kafka rawEvents {
        "cart, checkout, fulfillment events"
        aspect {
            broker: EventStream
            topic: "analytics.storefront.raw.v1"
        }
    }
}

Two emitters, each owning its own rawEvents. The event lives on the producer because the producer owns the schema — the contract. The two events are distinct interfaces (ProductApp.rawEvents, StoreFront.rawEvents); a consumer reads each one by name. There is no shared “RawEvents” node and no per-source handler to declare on the consumer side — the consumer simply edges to each producer in a process.

pipeline.arch:

service #en44 Enricher {
    aspect team: "Data"
    repo.url: "https://github.com/acme/enricher"
    "Joins raw events with user-profile attributes, emits enriched events."


    aspect {
        domain: "Analytics"
        security.zone: "Internal"
    }


    kafka enrichedEvents {
        "Raw event + resolved user_id, session_id, plan_tier."
        aspect {
            broker: EventStream
            topic: "analytics.enriched.v1"
        }
    }
}


service #ag17 Aggregator {
    aspect team: "Data"
    repo.url: "https://github.com/acme/aggregator"
    "Rolls enriched events into minute / hour / day buckets, pushes to Metrics."


    aspect {
        domain: "Analytics"
        security.zone: "Internal"
    }
}


service #me08 Metrics {
    aspect team: "Data"
    repo.url: "https://github.com/acme/metrics"
    "Tier-1 metrics surface — counters and rollups."


    aspect {
        domain: "Analytics"
        security.zone: "Internal"
    }


    grpc_unary push { "Receive rolled-up metric increments from Aggregator." }
}

Enricher and Aggregator are pure transformers: each reads upstream and (for Enricher) emits downstream. They have no inbound handler interface — being triggered by an event is a process edge to the producer, not a declared subscription. Metrics is the one sync sink here: Aggregator calls push directly.

sinks.arch:

database #wh90 Warehouse {
    aspect team: "Data"
    "Long-term analytical store. Append-only, partitioned by event date."


    aspect {
        domain: "Analytics"
        data.classification: "pii"
        engine: "clickhouse"
        host: "analytics-cluster"
    }
    catalog.url: "https://datahub.acme.internal/dataset/warehouse"
    console.url: "https://clickhouse.acme.internal"


    kafka landedEvents {
        "Emitted after a successful land. Downstream consumers read here."
        aspect {
            broker: EventStream
            topic: "analytics.landed.v1"
        }
    }


    grpc_unary replay { "Re-emit archived events for a date range. Admin only." }
    db_read query { "Ad-hoc analytical reads." }
}


frontend #db55 Dashboard {
    aspect team: "Data"
    repo.url: "https://github.com/acme/dashboard"
    "Real-time analytics dashboard. Refreshes as new data lands."


    aspect {
        domain: "Analytics"
        security.zone: "Internal"
    }


    rest_read loadDashboard { "Initial page load — render current rollups." }
}

Warehouse both consumes (the landing path is a process edge to Enricher.enrichedEvents) and produces (landedEvents, which Dashboard reads). The two-role node — receive upstream, broadcast downstream — is common at a pipeline’s joints, and the producer-owned event makes it natural: the inbound side is just an edge in the process, the outbound side is an event it owns.

Note the hosting chain on Warehouse, expressed entirely with aspects: engine: "clickhouse" names the DBMS plane, host: "analytics-cluster" names the server/cluster plane. Each is a different plane of existence, so each is an aspect, not nesting — exactly like broker. Nesting would wrongly say “part of the domain”; an aspect correctly says “deployed on.” (Use a bare value — host: AnalyticsCluster — when you want that plane materialized as a real, drawable module.)

infra.arch:

// The broker every event rides. It is a real module — but it lives on
// the infra plane and is reached through the `broker` aspect, never by
// routing a call through it. The EventBackbone view toggles its aspect.
kafka_cluster #ev21 EventStream {
    aspect team: "Data Platform"
    aspect engine: "kafka"
    "Shared streaming backbone for the analytics pipeline."
    console.url: "https://kafka.acme.internal/clusters/analytics"
}

Mistake to avoid. Don’t route the flow through the broker — Producer → EventStream → Consumer is the anti-pattern. It buries the architecture that matters (who depends on whom) under transport mechanics, and every module ends up pointing at the same broker node, so the diagram says nothing. The broker is not absent from the model — it’s a real kafka_cluster module — but it sits on the infra plane and is wired in by the broker aspect on each event interface. Reveal it on demand with an overlay view; never put it in the call path.

The pipeline as a process

Section titled “The pipeline as a process”

The async dataflow isn’t latent wiring — it’s a process. Each consumer edges to the producer’s event interface; the step order is the pipeline:

process #pi63 AnalyticsIngest {
    "Producer-owned events. A consumer edges to the producer it reads from;
    the broker carries each hop on a separate plane (aspects, not steps)."


    Enricher > ProductApp.rawEvents
    Enricher > StoreFront.rawEvents


    Aggregator > Enricher.enrichedEvents
    Aggregator > Metrics.push


    Warehouse > Enricher.enrichedEvents
    Dashboard > Warehouse.landedEvents
}

Six steps, every module on the board, the data flow legible top to bottom. The dependency edge points consumer→producer (Enricher > ProductApp.rawEvents = “Enricher depends on ProductApp’s event”); the renderer flips the arrowhead so data reads the natural way. That reversal is a render concern, not a modeling one.

Why a process and not subscribes: wiring? Because edges in ArchLang derive from use — what actually happens — and a process is how use is stated. A standing subscription with no flow is the mirror of an unused outbound client: latent capability the model deliberately doesn’t draw. Model the consume as the process step it is; the connection falls out of it, and every module proves it isn’t dead code by appearing here.

Even an event-driven pipeline has sync moments — usually administration:

user #u0e1 DataEngineer
user #u0u2 Viewer


process #bf02 BackfillReplay {
    DataEngineer > Warehouse.replay
}


process #dl04 DashboardLoad {
    Viewer > Dashboard.loadDashboard
}

Two processes for the two sync touchpoints — a manual backfill and the dashboard’s initial load.

Views

Section titled “Views”
view #pd01 PipelineDataflow {
    "Full pipeline — sources, transformers, sinks. Async edges follow the ingest process."
    show @@domain:"Analytics"
}


view #pi02 PIIScope {
    "Every module touching PII data. Used for data-classification audits."
    show @@data.classification:"pii"
    group by @@team
}


view #eb03 EventBackbone {
    "The infra overlay: every event interface that rides the shared broker."
    show @@broker:EventStream
}

PIIScope works because of the data.classification: "pii" aspect on Warehouse; cascade pulls it through to anything nested inside, and the show filter does the rest. EventBackbone is the payoff for keeping transport on an aspect — show @@broker:EventStream reveals the broker plane without that plane ever cluttering the dataflow view.

What an event-driven model looks like

Section titled “What an event-driven model looks like”

After validation:

  • Seven modules, four producer-owned event interfaces, one shared broker on the infra plane.
  • A diagram with directed edges that follow the ingest process — sources at the top, sinks at the bottom, dashed edges for async.
  • Three views: dataflow for engineers, PII scope for governance, and the broker overlay for infra.

Where the SaaS backend (Chapter 25) was dominated by sync request/response, this pipeline is dominated by async events — but the modeling is the same in both: events live on their producer, consumers edge to them in processes, and the broker is an aspect, not a waypoint.

Decisions you’ll face

Section titled “Decisions you’ll face”

Where the broker lives. On an aspect (aspect broker: EventStream) — never as a node in the call path. It stays a real module so you can own it, link it, and overlay it; it just doesn’t pollute the dataflow.

Who owns the event. The producer. The event interface lives on the module that emits it, because that module owns the schema. Adding a consumer is then a pure addition on the consumer side — the producer file is untouched, exactly like real pub/sub.

When a stream deserves its own node. The rule bans the transport waypoint, not a domain stream. If a channel carries identity the architecture cares about — a cross-team event backbone with its own schema and SLA, owned by a platform team — that stream is a first-class node publishers and subscribers both edge to. The discriminator is “does the channel have its own identity,” the same judgment you make for a database: dumb storage nests or takes an aspect; a data product gets a node.

The hosting chain. A nested or standalone store isn’t the bottom of the stack. Warehouse → engine (DBMS) → host (cluster) chains by aspect, each hop a different plane. Bare values materialize the plane as a node; string values keep it as an aspect overlay.

Backpressure and retries. Not in the model. Those are operational concerns. The model captures who consumes what; how reliably it does so is a different artifact (an SLO doc, an incident runbook).

Summary

Section titled “Summary”
  • Events are owned by their producer; a consumer edges to the producer’s event interface in a process. No subscribes: indirection.
  • The async flow is a process — the step order is the pipeline, and every module proves it isn’t dead code by appearing in it.
  • The broker is a real module on the infra plane, attached by aspect broker. Never route the call path through it.
  • The hosting stack (DB → DBMS → server) chains by aspect, each hop a separate plane.
  • Views for dataflow, governance (PII), and the broker overlay all fall out of cascading aspects.

What’s next

Section titled “What’s next”

Chapter 27: An External Integration → — your system plus a third-party API plus their webhooks, modeling the boundary precisely.