Data ingestion with dlt and Dagster: An end-to-end pipeline tutorial:
Curious like us to see what people are sharing with #dataBS and #datasky? Check out this post to learn how to do it using dlt!"
@matthausk.bsky.social
@datateam.bsky.social
@hgeren.bsky.social
@hopefanhe.bsky.social
#dlt
We are starting a 32-week Data Engineering Interview Guide program, covering everything from fundamentals to advanced topics, with sessions every Saturday.
Do you think we're missing any critical topics? We're curious about your opinions😊
#dataBS
#datasky
As a Data Engineer, understanding the data storage lifecycle and data retention policies is critical for designing efficient, cost-effective, and compliant data systems.
@joereis.bsky.social
#dataBS #datasky
substack.com/@pipeline2in...
In our new post, we've covered 10 of the most popular data pipeline design patterns.
We’d love to hear your thoughts. For more details, please check out the full post created by (@hgeren.bsky.social and @hopefanhe.bsky.social ): open.substack.com/pub/pipeline...
#dataBS #datasky
As a Data Engineer and Monster Hunter fan, love this metaphor!
Discover how dlt simplifies data ingestion.
Learn its origins and real-world use cases. Follow a step-by-step guide to build your first pipeline and join the growing dlt community!
@matthausk.bsky.social
@datateam.bsky.social
@hgeren.bsky.social
@hopefanhe.bsky.social
#dataBS #datasky
Hi, wishing everyone a great Thanksgiving!
Recently we wrote about how SQL queries are executed behind the scenes.
If you are interested, check out our post: open.substack.com/pub/pipeline...
#dataBS #datasky
Storage is at the heart of Data Engineering.
In this post, we explore the hierarchy of data storage from the ground up, drawing inspiration from Fundamentals of Data Engineering by
@joereis.bsky.social
and Matt Housley, as well as insights from the DE Professionals on Coursera.
#dataBS #datasky
Thank you so much! I am also planning to study cost estimation step in detail soon, so I will definitely write about it when I deepen my knowledge 🙏🏻
Hey #dataBS and #datasky folks,
Our new post about "how understanding Big O Notation & Execution Plans can optimize SQL queries" has just been posted.
Check it out if you're interested, and we'd love to hear your thoughts! @hopefanhe.bsky.social
open.substack.com/pub/pipeline...
yeah you are right, it was posted about 10 days ago 😊
Yeah, maybe Data Science can also be the navigation system with its predictions capabilities and Data Analytics can be driving assistants. While Data Engineering ensuring the whole coordination.
Hey #dataBS, I've been thinking of an analogy for Data Teams' roles.
Imagine a company as a vehicle. How would you map Data Engineering, Analytics, and Science to vehicle parts? Teams could have multiple parts or overlap with other Teams.
Curious about your thoughts!
Looking for a distraction? Try this great interview between @hannes.muehleisen.org and @medriscoll.bsky.social covering all things @duckdb.org. I especially enjoyed the philosophy around improving SQL usability. www.youtube.com/watch?v=a-Rm... #databs
#dstaBS can you repost?
Filled up the first 150 and so am creating a second starter pack! Let’s all keep finding each other and make this place the best for all things data
Week 1 of "100 Days of SQL Optimisation" covered key techniques like column selection, multicolumn indexes, filtering, window functions, Rank, CTE and composite indexes with IMDb data.
Check out the full post for more!
@hgeren.bsky.social
#dataBS #datasky
I made an infra engineer starter pack. Folks posting about databases, stream processing, durable execution, orchestrators, service meshes, and more.
go.bsky.app/SCZe42X
Hello everyone! I’m Hasan.
I transitioned from Industrial Engineering to Data Science, then found my passion in Data Engineering. Currently, doing a PhD in distributed stream processing while working as a Data Engineer.
Looking forward to connecting with fellow data enthusiasts to learn and share.
I’d say SQL
