Deep Dive into Apache Iceberg with Flink CDC - tech.kakao.com 이 글은 를 영어로 작성한 문서입니다. 다른 번역본 보기: 🇰🇷 한국어: h...

Some excellent deep-dive blogs from Kakao on their use of Flink CDC and Iceberg:

tech.kakao.com/posts...
tech.kakao.com/posts...
tech.kakao.com/posts...

I'm excited to be speaking at SREday London 2025!

Join me on March 27-28 to hear how we built a foundational, cloud-native, Streaming Platform using the operator pattern (and some magic ✨)

Register here:
sreday.com/2025-london-...
Don't forget to use the code 𝗟𝗗𝗡𝟭𝟬 for 10% off!

[FLINK-34440][formats][protobuf-confluent] Protobuf confluent dynamic format by dmariassy · Pull Request #25114 · apache/flink What is the purpose of the change Add support for deserializing protobuf messages using the Confluent wire format and whose schemas can be fetched from Confluent Schema Registry Add support for se...

Can’t believe this will be left out:
github.com/apache/flink...
:(

Curious to know what connectors are mostly in use, and how many of them are in-house/forks?

Robert Metzger - Tiny Flink — Minimizing the memory footprint of Apache Flink YouTube video by Plain Schwarz

Other frameworks: Apache Flink is too heavy and it requires a cluster

Apache Flink: let's process 100MB/s of data in Kafka with only 100 MB of memory heap and only one single JVM instance 😎

Great (and useful) talk! www.youtube.com/watch?v=04pw...

Effective Communication in Slack for Engineering teams Transform Slack from a distraction into a productivity powerhouse for engineering teams with proven strategies for channel organization, automation, and data-driven insights

Slightly different from my usual posts, but I wrote a thing!
This time, on how to transform Slack from a distraction into a productivity powerhouse for engineering teams with proven strategies for channel organisation, automation, and data-driven insights -
leevs.dev/effective-co...

A 5-10% performance improvement with every new Java stable release is a dopamine hit... for CFOs. If you work with the JVM ecosystem, you should upgrade often, it's easy money

What’s your take on this implementation? I have love-hate relationship with it. Totally get why it was built that way (consumer offsets, Schema Registry topic, kStreams state, etc etc), but does it stand the test of time?

flink.apache.org/2024/12/05/i...

The History and Evolution of Open Table Formats From Hive to High Performance: A Journey Through the Evolution of Data Management on Data Lakes

Fantastic post on the history and evolution of OTFs

alirezasadeghi1.medium.com/the-history-...

jepsen.io/analyses/buf...
A new Jepsen report, and as always, super interesting. 🔥

This is really useful: Awesome Data Catalogs and Observability Platforms: github.com/opendatadisc...

(h/t @adipolak.bsky.social for sharing this with me)

#dataBS

[FLINK-34440][formats][protobuf-confluent] Protobuf confluent dynamic format by dmariassy · Pull Request #25114 · apache/flink What is the purpose of the change Add support for deserializing protobuf messages using the Confluent wire format and whose schemas can be fetched from Confluent Schema Registry Add support for se...

Unfortunately, not really. We have a very (very) raw version of it open sourced, but it’s missing 7 months of advancement.
I do know there is an implementation which about to get merged:

github.com/apache/flink...

Worth giving it a try!

Also:

go.bsky.app/U1hZhum

BigTech is moving to object storage too, but not cuz it's cheaper. The idea is you no longer need every system to be distributed and durable. Everything can be stateless and simple, cuz your storage is distributed and durable.

definitely the coolest feature so far on 🦋

For those who are interested in keeping track of Apache Flink's disaggregated state storage work, this is the umbrella FLIP:

cwiki.apache.org/con...

Having said that, everything is a trade off, and although it put more load (in terms of eng time) on the Data Platform teams, our clients are extremely happy with it. Proto tooling (code gen, grpc, protovalidte) are just superior to Avro

We’ve made the decision to go full Proto this time (always used Avro as main format in past companies). The amount of forks, bugs, and code changes we had to make to support it is insane. Most notably, we had to build our own Flink format to pull from registry, as Flink currently doesn’t support it

I made an infra engineer starter pack. Folks posting about databases, stream processing, durable execution, orchestrators, service meshes, and more.

go.bsky.app/SCZe42X