New chapter out about Data Engineering Workspace Packaging.

Random shoutout - I just started to read Data Engineering Design Patterns by @ssp.sh and it is full of tremendous inspiration, not only from a technical perspective, but also on how to effectively communicate data challenges. Getting pumped abt returning to work after paternity leave

dedp.online

What should be next?

Latest pattern published.

Data-Asset Reusability Pattern Accelerate data engineering development through code and data reuse patterns, from templating to materialization and abstraction.

I am happy to share that I released my second pattern: «Data-Asset Reusability Pattern», focusing on #EfficientDataReuse.

This chapter explores reusability in data engineering with dedicated sub-patterns. But first, what is data-asset reusability?

The evolution of data engineering.

The history and state of data engineering have changed drastically over the years. Let's explore what and how it changed.

Much has been written about data engineering, but how long has it existed, and where does it come from?

🌐 dedp.online/part-1/1-int... (direct link)

The book on the front page on Hacker News.

The Evolution of Schema Change and database change management.

📖 New chapter: Evolution of Schema Change and database change management.

> #SchemaEvolution: Managing database changes while preserving integrity
> #NoSQL: Flexible schemas; speed over consistency
> #DataContracts: Producer-consumer agreements with automated validation

dedp.online/part-2/4-ce/...

Challenges in Data Engineering This chapter explores the multifaceted challenges within data engineering, providing insights into the data lifecycle, from collection to actionable insights. It delves into the complexities of data s...

After years in #dataengineering, I've noticed we focus too much on tools & tech, missing the fundamental challenges. It's not about picking the right stack—it's about mastering the complete data lifecycle.

The reality: We can't control source systems and their upstream data yet.

If you enjoy the content, be sure to also check the connected «Data Engineering Vault», which is referenced in the book.

A recent review I've got 🥰.

> what a great content! "awesome, fresh view. I shown it to my fiancee & said "that's what wisdom is, like I would meet an 60 year old druid living in a forest and doing DE pipelines work".

> I'm patiently waiting to read more, It's a great piece &work that inspires me.

Book: Data Engineering Design Patterns (DEDP): Timeless Practices for Data Engineers A living Book about Data Engineering Design Patterns, exploring the evolution and its significance. Different from usual books. It does not come finished.

Data Engineering Design Patterns.

This book aims to provide a practical guide based on convergent evolution, resulting in design patterns applicable for navigating the data engineering lifecycle—timeless practices for data engineers.