arch.data

Package arch.data · Version 0.1.0

The data platform vocabulary — the middle of a modern analytics stack that arch.backend deliberately leaves out. Provides the orchestration and processing tier (data_pipeline, stream_processor, batch_engine, query_engine, ingestion, transform_tool), the analytical stores (warehouse, lakehouse, data_lake — subtypes of arch.backend’s data categories, so a database filter still catches them), and the consumption layer (bi_tool, data_catalog, notebook).

Model an ELT flow the same way as any other: producer-owned events and Caller > Callee.interface process steps. A pipeline is a module; the tables it reads and writes are database stores; the stream it consumes is a producer-owned event/kafka interface.

Import the kinds you use, e.g. use stream_processor, warehouse, bi_tool from arch.data.

Types

Type	Extends	Description
bi_tool	module	A business-intelligence / dashboard tool — queries and charts data from the analytical stores
tableau	bi_tool	Visual analytics platform with a broad connector set; a widely-used default
looker	bi_tool	Governed BI built on a modeled semantic layer (LookML); choose when a single metrics definition matters
superset	bi_tool	Open-source exploration and dashboarding over SQL sources
metabase	bi_tool	Lightweight, self-service BI aimed at non-analysts; choose for quick question-and-dashboard workflows
powerbi	bi_tool	Microsoft’s BI platform; choose in Microsoft-centric stacks
redash	bi_tool	Query-and-visualize tool oriented toward SQL users and shared queries
data_catalog	module	A data catalog — metadata, lineage and discovery across the data estate
datahub	data_catalog	Metadata platform with lineage, discovery and governance; a widely-used open default
amundsen	data_catalog	Data discovery and metadata engine focused on table search
openmetadata	data_catalog	Unified metadata platform spanning catalog, lineage, quality and governance
collibra	data_catalog	Enterprise data governance and catalog suite; choose for formal stewardship programs
atlas	data_catalog	Open metadata and governance framework rooted in the Hadoop ecosystem
notebook	module	An interactive analysis notebook — exploratory code and charts against the platform
jupyter	notebook	Interactive notebooks for exploratory data work; a common default
zeppelin	notebook	Web notebook with built-in support for Spark and multiple interpreters
data_pipeline	module	A batch/ELT orchestrator — schedules DAGs of data tasks with retries and backfills
airflow	data_pipeline	Python-defined DAG scheduler for batch data pipelines; a common default for orchestration
prefect	data_pipeline	Python workflow orchestrator with dynamic, code-defined flows
dagster	data_pipeline	Orchestrator built around software-defined assets rather than plain task DAGs; choose when data assets are the unit of work
kestra	data_pipeline	Declarative orchestrator with YAML-defined flows and a large plugin set
mage	data_pipeline	Notebook-style pipeline tool blending code and scheduling for smaller teams
transform_tool	module	An in-warehouse transformation framework — the T of ELT, compiling models to SQL run inside the warehouse
dbt	transform_tool	SQL-first transformation framework compiling modular models into warehouse queries; a common default
sqlmesh	transform_tool	Transformation framework with column-level lineage and virtual environments; choose for stronger change management
stream_processor	module	A continuous stream-processing engine — windowed, stateful computation over unbounded event streams
flink	stream_processor	Stateful stream processor with exactly-once semantics and event-time windowing; choose for demanding low-latency pipelines
spark_streaming	stream_processor	Micro-batch streaming on Spark; choose to share a Spark stack across batch and streaming
kafka_streams	stream_processor	Library embedding stream processing directly in a JVM service against Kafka topics; choose to avoid a separate cluster
ksqldb	stream_processor	SQL layer over Kafka streams for continuous queries; choose for SQL-defined stream transforms
materialize	stream_processor	Streaming database maintaining incrementally-updated materialized views over changing inputs
risingwave	stream_processor	Postgres-compatible streaming database for materialized views over event streams
arroyo	stream_processor	SQL stream-processing engine focused on operational simplicity
batch_engine	module	A distributed batch-compute engine for large-scale jobs — the compute a pipeline task runs on
spark	batch_engine	General-purpose distributed compute for batch ETL and analytics; a common default
hadoop	batch_engine	MapReduce-and-HDFS batch platform; choose for legacy on-prem data estates
dask	batch_engine	Parallel computing for Python that scales pandas/NumPy workloads across a cluster
ray	batch_engine	Distributed execution framework for Python batch and mixed compute; its ML libraries pair with arch.ai training
query_engine	module	A federated SQL query engine — interactive queries over lakes and multiple sources, storing no data of its own
trino	query_engine	Distributed SQL engine querying data across many sources in place; a common default for lake querying
presto	query_engine	Distributed SQL query engine; choose where an existing Presto deployment is in use
dremio	query_engine	Lakehouse query engine with a semantic layer and acceleration; choose for self-service on the lake
starrocks	query_engine	High-performance analytical query engine for real-time and lake analytics
ingestion	module	An ingestion / change-data-capture connector — moves data from sources into the platform
kafka_connect	ingestion	Connector framework streaming data between Kafka and external systems; choose in a Kafka-centric stack
airbyte	ingestion	Open-source EL platform with a broad connector catalog; choose for many long-tail sources
fivetran	ingestion	Managed EL service with maintained connectors; choose to offload connector upkeep
debezium	ingestion	Change-data-capture platform streaming database row changes as events; choose for log-based CDC
nifi	ingestion	Flow-based data routing and transformation with a visual designer; choose for complex routing across systems
meltano	ingestion	Singer-based, config-driven EL toolkit; choose for version-controlled pipeline definitions
warehouse	columnar	A managed columnar data warehouse for SQL analytics, typically with separated storage and compute
snowflake	warehouse	Managed multi-cloud warehouse with per-workload virtual compute; a widely-used SaaS default
firebolt	warehouse	Managed analytics warehouse focused on low-latency interactive queries
motherduck	warehouse	Managed, DuckDB-based warehouse blending local and cloud analytics
lakehouse	columnar	A lakehouse — warehouse semantics (ACID, schema, time travel) over open table formats on a lake
databricks	lakehouse	Managed lakehouse platform over Delta Lake, unifying engineering, analytics and ML; a widely-used SaaS default
delta_lake	lakehouse	Open table format adding ACID transactions and time travel to files on object storage
iceberg	lakehouse	Open table format with hidden partitioning and snapshot isolation; choose for engine-neutral lakehouse tables
hudi	lakehouse	Open table format optimized for incremental upserts and CDC on the lake
data_lake	object_storage	A data lake — raw, schema-on-read analytical storage over blob buckets; the landing zone beneath a lakehouse

---

See also: package.archspace reference · The standard library.