Data lakes such as Parquet, Delta Lake, and Iceberg complement warehouses by providing low‑cost, schema‑on‑read storage for raw and semi‑structured data while the warehouse handles curated, schema‑on‑write analytics.
Integration workflow
1. Ingest raw events into a cloud bucket (e.g., s3://raw/events/) as Apache Parquet files using Kafka Connect or Spark Structured Streaming.
2. Apply ACID & versioning with Delta Lake or Iceberg tables:
df.write.format("delta").mode("append").option("mergeSchema", "true").save("s3://lake/delta/events")3. Expose the lake tables to the warehouse:
- Snowflake: CREATE EXTERNAL TABLE lake_events ... LOCATION='s3://lake/delta/events' FILE_FORMAT = (TYPE = PARQUET);
- BigQuery: CREATE EXTERNAL TABLE project.dataset.lake_events OPTIONS (format='PARQUET', uris=['gs://lake/delta/events/*']);
4. Curate a star schema in the warehouse (e.g., FACT_SALES, DIM_DATE, DIM_PRODUCT) using INSERT ... SELECT from the external table, applying the warehouse’s optimizer.
5. Refresh materialized aggregates nightly with a threshold of > 10 M rows or > 5 TB processed to trigger REFRESH MATERIALIZED VIEW.
Quick comparison
| Feature | Parquet (file) | Delta Lake | Iceberg |
|--------------------|----------------|---------------------------|-----------------------------|
| ACID support | No | Yes (transaction log) | Yes (metadata tables) |
| Time travel | No | Up to 30 days via VERSION AS OF | Up to 7 days via AS OF TIMESTAMP |
| Spark integration | ✅ | ✅ (native) | ✅ (catalog API) |
| Warehouse external table | ✅ | ✅ (via delta format) | ✅ (via iceberg format) |
Decision checklist
- Do you need schema evolution without ETL? → Choose Delta or Iceberg.
- Is sub‑second query latency on raw data required? → Keep data in the warehouse, not the lake.
- Is cost the primary driver for petabyte‑scale storage? → Store in Parquet/Delta on object storage and only materialize aggregates in the warehouse.