🚀 Built Asgarde a while ago to simplify error handling in #ApacheBeam

Shared a short video:
• Native Beam
• Asgarde #java
• Asgarde #kotlin

Cleaner, more expressive pipelines

⭐️ on GitHub appreciated, I share the repo in comment!

#ApacheBeam #GoogleCloud

Build scalable data pipelines with Google Cloud Dataflow, Apache Beam & Apache Hop!

Learn how to set up, create & deploy in our blog post! 🔗 www.know.bi/blog/our-blo...

Author: Bart Maertens @bart.know.bi

#databs #datasky #apachebeam #googlecloud #apachehop

NEW VIDEO: Transcribe Newscasts in Parallel with #ApacheBeam and @huggingface.bsky.social
youtu.be/_bKyFREDvZc

Let's try: Apache Beam part 8 - Tags & Side inputs We sometimes have to apply some complex conditions in our Beam pipeline. This blog we will get along together to see how can we design those complex ideas into a simple-readable yet powerful workflow.

#bluebirzblog
Let's try: Apache Beam part 8 - Tags & Side inputs
We can create our own version of #ApacheBeam io packages Let's see how to make it for #Firestore.
[EN] www.bluebirz.net/en/lets-try-...
[TH] www.bluebirz.net/th/lets-try-...
[Medium] medium.com/@bluebirz/le...

Let's try: Apache Beam part 7 - custom IO in real world, there would be some cases we need to connect to some sources that Apache Beam doesn't have the IO packages for. Let's see how can we implement IO package in our own styles.

#bluebirzblog
Let's try: Apache Beam part 7 - custom IO
We can create our own version of #ApacheBeam io packages Let's see how to make it for #Firestore.
[EN] www.bluebirz.net/en/lets-try-...
[TH] www.bluebirz.net/th/lets-try-...
[Medium] medium.com/@bluebirz/le...

Let's try: Apache Beam part 6 - instant IO Apache Beam provides inputs and outputs for PCollection in many packages. We just import and call them properly and get the job done.

#bluebirzblog
Let's try: Apache Beam part 6 - instant IO
#ApacheBeam provides inputs and outputs for PCollection in many packages.We just import and call them properly and get the job done
[Medium] medium.com/@bluebirz/le...
[TH] www.bluebirz.net/th/lets-try-...
[EN] www.bluebirz.net/en/lets-try-...

❓❓ Do you use #ApacheBeam and deal with errors in data processing pipelines?

If it's the case, you know how important this topic is

💡 ➡️ In this case, the best practice and recommended approach is to apply the #DeadLetterQueue principle in your pipelines 🧵

#GoogleCloud

Quiet before the storm – the "backstage view". 5 minutes later the room was packed! 🤩

Just done presenting on data pipelines, #apachebeam, #OSS, #Java & #Cloud @ Devfest Bucharest, on the Cloud stage. 💾

Thanks to the awesome GDGBucharest team for the organization. 👏

Build a scalable, self-managed streaming infrastructure with Beam and Flink Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterpris...

Discover how Apache Beam & Flink are revolutionizing streaming infrastructure for scalability & cost-efficiency. Dive into our journey of creating a self-managed, robust platform on Kubernetes! #ApacheBeam #ApacheFlink #Kubernetes #DataEngineering 🚀 Read More: beam.apache.org/blog/apache-...

ETL Batch pipeline with Cloud Storage, Dataflow and BigQuery orchestrated by Airflow/Composer 1. Explanation of the use case presented in this article

🚀 I share my article with a complete #ETL Batch pipeline on #GCP with #CloudStorage #Dataflow #BigQuery orchestrated by #Airflow #Composer

➡️ ➡️ ➡️ #GoogleCloud
➡️ ➡️ #Airflow
✅ Extract: GCS
✅ Transform: Dataflow #ApacheBeam #Python
✅ Load: BigQuery

medium.com/google-cloud...

🚀 Si vous utilisez #ApacheBeam pour faire du processing de la donnée en mode #streaming et #batch ou vous vous intéressez au sujet, j’ai créé une librairie open source appelé #Asgarde

Cette lib permet de simplifier la gestion d’erreurs avec le principe de #DeadLetterQueue

👇🏻

The Dataflow Model: A Practical Approach to Balancing Cor...

Parallel processing science in motion: http://research.google.com/pubs/pub43864.html http://research.google.com/pubs/pub35650.html http://research.google.com/pubs/pub41378.html #Dataflow #ApacheBeam

"No one at Google uses MapReduce anymore" is my session at #NABDConf today. #Dataflow #ApacheBeam.