Big thanks to:

🚀 Madhukar & Satya from S&P Global for sharing #DeltaLake learnings!

👏 Maintainer R. Tyler Croy for championing #deltars + #CDC in the #rustlang community

🙌 Sonia Mankin (SA at Databricks) for bringing everyone together!

🔥 & Denny Lee for bringing deep #DeltaLake context & energy!

Using AWS Lambdas + #deltars, S&P Global cut costs from $52 → $7.30/month 💸

⚡ Spin-up time: 6 min → 1.4 sec
⚡ Job run time: 10 min → 4 min

✅ In addition to using #ApacheSpark for their large loads, they also use #pandas, #polars, #datafusion, #daft and #duckdb - all together with #DeltaLake.

Just wrapped an amazing session: Rusty Waters: Elevating Lakehouses Beyond Spark 🌊

S&P Global Commodity Insights runs 3K+ TBs of processed data & 75K+ jobs/day! ⚙️

By the numbers, for example, for a 100MB dataset hourly processing workflow, by using AWS Lambdas and #deltars.

#opensource #deltalake