Crunch: the data lake optimization engine

Crunch commits all results via the native Delta transaction log or Iceberg commit protocol, there is no intermediate state visible to readers. File-level rollback tracks exactly which files were modified; if a job is interrupted, only those files are cleaned up. Cross-catalog shared-file detection protects files referenced by any connected catalog from accidental deletion during vacuum or orphan cleanup. Crunch has managed 100 PB of production data with no data correctness issues.

No lock-in. Crunch uses only open table formats (Delta Lake, Iceberg, Hive/Parquet) via their native commit protocols. No proprietary file formats or catalog extensions are introduced. If Crunch is turned off, all tables remain fully readable and writable by any engine: Spark, Trino, Flink, DuckDB, Snowflake, Athena. Files Crunch wrote retain their compression; new files use the engine's default encoding. There is no required migration or "un-Crunching" step.

In-house systems work well up to a few hundred tables. They break predictably at scale: Airflow has no SLA awareness or resource contention visibility, so failures retry at full size and cascade. There is no file-level state tracking, in-house tools know a table ran last Tuesday; Crunch knows the state of every file. General-purpose engines also apply the same executor configuration to a 10-file job and a 10-million-file job. Crunch is the tool you reach for when your in-house pipeline becomes a source of incidents rather than a solution.

Yes. Crunch connects to Hive Metastore, AWS Glue, Databricks Unity Catalog, and Apache Polaris simultaneously. It exposes a full REST API, so existing Airflow DAGs or dbt workflows can trigger Crunch jobs on-demand or check optimization status alongside Crunch's own scheduler. It also emits OpenTelemetry spans for every job phase so your observability stack gets full visibility.

It depends on your deployment model. In the Hybrid model (recommended), only table metadata crosses the Granica boundary, no actual data files leave your cloud. In the On-Prem model, nothing leaves your environment at all. In Full SaaS, data is retrieved into Granica's environment for processing under standard data processing agreements. See the deployment section above for details.

Crunch supports partition-level deletion on a schedule or on-demand for partitioned fact tables, the standard path for GDPR right-to-be-forgotten requests. For non-isolated PII, it performs row-level deletes using native format mechanisms (Delta delete vectors, Iceberg position deletes), then compacts the affected files to physically remove data from storage. Every deletion operation is logged with commit ID, timestamp, the triggering policy, and affected files, retained independently for compliance reporting.

Yes: Crunch manages Delta Lake, Apache Iceberg, Hive, and Parquet natively. It is the only platform that integrates all catalogs simultaneously, which lets it detect shared files across table formats during migration. This prevents a class of accidental data loss, for example, a vacuum that deletes Parquet files still referenced by a Hive table, that format-specific tools cannot see because they only connect to one catalog at a time.

In our internal TPC-DS benchmark on m7g.xlarge hardware, the projected cost is approximately $900/PiB for Crunch, compared to $1,212/PiB for the best Databricks Photon configuration (including DBU charges). Crunch's advantage compounds over time because its superior data reduction (36% vs 14%) lowers every downstream scan cost permanently. Actual cost will vary by cluster shape, data characteristics, and contract pricing.

Crunch runs on customer-managed compute, EMR, Dataproc, Spark on Kubernetes, or Crunch-managed ephemeral clusters. It does not require a dedicated long-running cluster for most workloads. Contact us for specific runtime version requirements and EMR or Dataproc support timelines.

Crunch runs as a background process and commits via atomic Delta or Iceberg transactions, so readers always see a consistent snapshot, there is no partial state visible to queries during a run. Once optimization completes, query scan I/O falls proportionally to the data reduction achieved. At ShareChat, storage and scan costs dropped over 30% across the table estate with no pipeline disruptions.