Hadoop to Lakehouse
Loading the numbers…
From HDFS to Gold
- Step 1 Hadoop HDFS Legacy source, HDFS 3.2.1 in Docker — rows in — CSV files under /legacy/erp/orders
- Step 2 GCS landing bucket Copied by Airflow over WebHDFS — files identical to HDFS, — in all
- Step 3 Bronze Databricks Auto Loader — raw rows; new source columns are added automatically
- Step 4 Silver Data-quality rules, then MERGE — unique orders; — rows quarantined
- Step 5 Gold Delta tables BigQuery can read — day × SKU rows; — customers
Each batch day
Where each day's Bronze rows went
Rows neither kept nor quarantined were older copies of an order, removed when Silver deduplicates.
Run time per batch day, in minutes
The HDFS check and copy take seconds. Nearly all the time is the three Databricks steps on serverless compute.
Files copied from HDFS to GCS
Each file's size in HDFS and in the landing bucket after Airflow's copy step.
| File | Batch day | Rows | HDFS bytes | GCS bytes | Identical |
|---|
Data quality
Why rows were quarantined
The mock source data is dirty on purpose. A row can fail more than one rule, so reasons add up to more than the quarantined rows.
Sales in Gold
Net revenue—excludes cancelled orders
Orders—across all order dates
Cancellation rate—cancelled orders ÷ orders
Customers—with at least one order
Net revenue by order date
Small days hold only orders whose timestamps fall outside a batch day.