Final Peer Reviews for #DEZoomcamp complete! ✅
Just evaluated a complex SkyPulse Streaming Pipeline. Impressed by the real-time flight data visualization and the use of Apache Flink + Redpanda. Reviewing expert architectures like this is a game-changer for my engineering criteria.
Final Project for the Data Engineering Zoomcamp complete! 🚀
Built an end-to-end pipeline to audit Colombian public procurement (SECOP II) bridging Law and Tech. Used AWS (S3, Glue, Athena), dbt for transformations, and Looker Studio.
Check my repo & dashboard: github.com/CodingJhames...
The Data Warehouse is officially online! 🌩️
Mapped 100k raw SECOP II records in S3 using AWS Glue and queried them with Athena. The best part? Connecting my local James-T-850 via #dbt to run the first staging models directly in the cloud. 🚀
github.com/CodingJhames...
#DataEngineering #AWS #dbt
Data Ingestion milestone reached! 🏗️
Successfully deployed #AWS S3 with #Terraform and ingested 100k records from the SECOP II API using Python on my James-T-850. Everything is now stored as Parquet.
Repo: github.com/CodingJhames...
#DataEngineering #Python #Cloud #DEZoomcamp
Week 7 #DataEngineering Zoomcamp 🏎️
Streamed 4.4M records via #Redpanda & #PySpark on my James-T-850. Speed is nothing without logic!
Results: 📍
📏 Dist: 9506
🏙️ Zone: 74
⏳ Session: 31m
💰 Peak Tip: 10-16 18:00
Progress: github.com/CodingJhames...
#Streaming #Python #BigData #DataTalksClub
Week 6 Spark Module of #DataEngineeringZoomcamp completed! 🛠️
Bridging legal logic with large-scale batch processing.
🔹 PySpark & DataFrames
🔹 Parquet optimization
🔹 AWS EC2 deployment
🔹 Spark UI monitoring (4040)
Solution: github.com/CodingJhames...
Course: github.com/DataTalksClu...
Totally agree! Schema drift was a great lesson today. dlt handles schema versioning out of the box, storing it as YAML in the destination. It definitely gives me peace of mind knowing the evolution is tracked even if the API changes. Thanks for the insight!
#DEZoomcamp W6: dlt & DuckDB! 🏁
Schema drift: API changed tip_amount to tip_amt. ⚠️
DuckDB's error logs pointed out the fix instantly.
Running #dlt on an AWS t3.micro (1GB RAM).
0.2666 Credit prop | 6063.41 Tips.
Repo: github.com/CodingJhames/de-zoomcamp-james
#DataEngineering #AWS #BuildInPublic
Thanks, Jeremy! Performance took a hit due to disk I/O, but the goal was to prove that logic beats a tight budget. I used the zone data for a join, and you're right—it’s the perfect case for broadcast joins to avoid exploding that 1GB RAM limit!.
Data Engineering Week 5: Done! 🏁
Pivoted to #AWS from GCP. ☁️
Ran #PySpark on a t3.micro (1GB RAM) using a 4GB Swapfile. Processed NYC Taxi data smoothly without crashes. 🧠
Adaptability > Tools. 🦾
Code: github.com/CodingJhames...
#DataEngineering #Spark #AWS #OpenSource
Week 4 Analytics Engineering: Complete! 👨💻
Mastered dbt with NYC Taxi data in AWS Athena. Overcoming those Parquet type mismatches was the final boss! 🚀
Project link:
github.com/CodingJhames...
#DataEngineering #dbt #BuildInPublic
Week 3 #DataZoomcamp done! 🚀
Migrated the DW logic to AWS Athena & S3 ☁️
🔹 20M+ records with #Kestra 🔹 Optimized queries: 310MB → 26MB scan using Partitioning & Clustering 🔹 Mastered cloud-agnostic DW concepts
Check my repo: 🔗 github.com/CodingJhames...
#DataEngineering #AWS #Athena