arch.data

Package arch.data · Version 0.1.0

The data platform vocabulary — the middle of a modern analytics stack that arch.backend deliberately leaves out. Provides the orchestration and processing tier (data_pipeline, stream_processor, batch_engine, query_engine, ingestion, transform_tool), the analytical stores (warehouse, lakehouse, data_lake — subtypes of arch.backend’s data categories, so a database filter still catches them), and the consumption layer (bi_tool, data_catalog, notebook).

Model an ELT flow the same way as any other: producer-owned events and Caller > Callee.interface process steps. A pipeline is a module; the tables it reads and writes are database stores; the stream it consumes is a producer-owned event/kafka interface.

Import the kinds you use, e.g. use stream_processor, warehouse, bi_tool from arch.data.

Types

Type	Extends	Description
`bi_tool`	`module`	A business-intelligence / dashboard tool — queries and charts data from the analytical stores
`tableau`	`bi_tool`	Visual analytics platform with a broad connector set; a widely-used default
`looker`	`bi_tool`	Governed BI built on a modeled semantic layer (LookML); choose when a single metrics definition matters
`superset`	`bi_tool`	Open-source exploration and dashboarding over SQL sources
`metabase`	`bi_tool`	Lightweight, self-service BI aimed at non-analysts; choose for quick question-and-dashboard workflows
`powerbi`	`bi_tool`	Microsoft’s BI platform; choose in Microsoft-centric stacks
`redash`	`bi_tool`	Query-and-visualize tool oriented toward SQL users and shared queries
`data_catalog`	`module`	A data catalog — metadata, lineage and discovery across the data estate
`datahub`	`data_catalog`	Metadata platform with lineage, discovery and governance; a widely-used open default
`amundsen`	`data_catalog`	Data discovery and metadata engine focused on table search
`openmetadata`	`data_catalog`	Unified metadata platform spanning catalog, lineage, quality and governance
`collibra`	`data_catalog`	Enterprise data governance and catalog suite; choose for formal stewardship programs
`atlas`	`data_catalog`	Open metadata and governance framework rooted in the Hadoop ecosystem
`notebook`	`module`	An interactive analysis notebook — exploratory code and charts against the platform
`jupyter`	`notebook`	Interactive notebooks for exploratory data work; a common default
`zeppelin`	`notebook`	Web notebook with built-in support for Spark and multiple interpreters
`data_pipeline`	`module`	A batch/ELT orchestrator — schedules DAGs of data tasks with retries and backfills
`airflow`	`data_pipeline`	Python-defined DAG scheduler for batch data pipelines; a common default for orchestration
`prefect`	`data_pipeline`	Python workflow orchestrator with dynamic, code-defined flows
`dagster`	`data_pipeline`	Orchestrator built around software-defined assets rather than plain task DAGs; choose when data assets are the unit of work
`kestra`	`data_pipeline`	Declarative orchestrator with YAML-defined flows and a large plugin set
`mage`	`data_pipeline`	Notebook-style pipeline tool blending code and scheduling for smaller teams
`transform_tool`	`module`	An in-warehouse transformation framework — the T of ELT, compiling models to SQL run inside the warehouse
`dbt`	`transform_tool`	SQL-first transformation framework compiling modular models into warehouse queries; a common default
`sqlmesh`	`transform_tool`	Transformation framework with column-level lineage and virtual environments; choose for stronger change management
`stream_processor`	`module`	A continuous stream-processing engine — windowed, stateful computation over unbounded event streams
`flink`	`stream_processor`	Stateful stream processor with exactly-once semantics and event-time windowing; choose for demanding low-latency pipelines
`spark_streaming`	`stream_processor`	Micro-batch streaming on Spark; choose to share a Spark stack across batch and streaming
`kafka_streams`	`stream_processor`	Library embedding stream processing directly in a JVM service against Kafka topics; choose to avoid a separate cluster
`ksqldb`	`stream_processor`	SQL layer over Kafka streams for continuous queries; choose for SQL-defined stream transforms
`materialize`	`stream_processor`	Streaming database maintaining incrementally-updated materialized views over changing inputs
`risingwave`	`stream_processor`	Postgres-compatible streaming database for materialized views over event streams
`arroyo`	`stream_processor`	SQL stream-processing engine focused on operational simplicity
`batch_engine`	`module`	A distributed batch-compute engine for large-scale jobs — the compute a pipeline task runs on
`spark`	`batch_engine`	General-purpose distributed compute for batch ETL and analytics; a common default
`hadoop`	`batch_engine`	MapReduce-and-HDFS batch platform; choose for legacy on-prem data estates
`dask`	`batch_engine`	Parallel computing for Python that scales pandas/NumPy workloads across a cluster
`ray`	`batch_engine`	Distributed execution framework for Python batch and mixed compute; its ML libraries pair with arch.ai training
`query_engine`	`module`	A federated SQL query engine — interactive queries over lakes and multiple sources, storing no data of its own
`trino`	`query_engine`	Distributed SQL engine querying data across many sources in place; a common default for lake querying
`presto`	`query_engine`	Distributed SQL query engine; choose where an existing Presto deployment is in use
`dremio`	`query_engine`	Lakehouse query engine with a semantic layer and acceleration; choose for self-service on the lake
`starrocks`	`query_engine`	High-performance analytical query engine for real-time and lake analytics
`ingestion`	`module`	An ingestion / change-data-capture connector — moves data from sources into the platform
`kafka_connect`	`ingestion`	Connector framework streaming data between Kafka and external systems; choose in a Kafka-centric stack
`airbyte`	`ingestion`	Open-source EL platform with a broad connector catalog; choose for many long-tail sources
`fivetran`	`ingestion`	Managed EL service with maintained connectors; choose to offload connector upkeep
`debezium`	`ingestion`	Change-data-capture platform streaming database row changes as events; choose for log-based CDC
`nifi`	`ingestion`	Flow-based data routing and transformation with a visual designer; choose for complex routing across systems
`meltano`	`ingestion`	Singer-based, config-driven EL toolkit; choose for version-controlled pipeline definitions
`warehouse`	`columnar`	A managed columnar data warehouse for SQL analytics, typically with separated storage and compute
`snowflake`	`warehouse`	Managed multi-cloud warehouse with per-workload virtual compute; a widely-used SaaS default
`firebolt`	`warehouse`	Managed analytics warehouse focused on low-latency interactive queries
`motherduck`	`warehouse`	Managed, DuckDB-based warehouse blending local and cloud analytics
`lakehouse`	`columnar`	A lakehouse — warehouse semantics (ACID, schema, time travel) over open table formats on a lake
`databricks`	`lakehouse`	Managed lakehouse platform over Delta Lake, unifying engineering, analytics and ML; a widely-used SaaS default
`delta_lake`	`lakehouse`	Open table format adding ACID transactions and time travel to files on object storage
`iceberg`	`lakehouse`	Open table format with hidden partitioning and snapshot isolation; choose for engine-neutral lakehouse tables
`hudi`	`lakehouse`	Open table format optimized for incremental upserts and CDC on the lake
`data_lake`	`object_storage`	A data lake — raw, schema-on-read analytical storage over blob buckets; the landing zone beneath a lakehouse