This will be the foundation for being able to branch @kafkastreams.bsky.social apps to support seamless blue/green deploys. It also allows you to branch a previous version of the state to debug an issue from the past.
Powerful stuff coming to the world of stream processing!
For any Kafka Streams fans out there who want to know what it's like scaling an EOS application up to multiple TB of state at ~10k+ records/s with minimal downtime -- make sure to catch the Q&A we're hosting tomorrow with Metronome.
Join us Dec 12th at 9:30am PST!
www.linkedin.com/events/72647...
This was a fun conversation with @thegeeknarrator.bsky.social and @yingjunwu.bsky.social about the world of SQL stream processing on one side and event driven apps on the other. Thanks for having me on the show, Kaivalya!
It's said that Silicon Valley is special because the density of smart and motivated people leads to chance encounters that don't happen elsewhere.
I can attest to that.
Here's how a coffee resulted in @responsive.dev building a database optimized for stream processing in 8 months. (1/n)
From one of @responsive.dev 's customers:
"will report back .. when we start to change ttl lengths, which we're currently set up to do through launchdarkly based on the client and environment...pretty slick."
They are changing the TTL of rows in their Kafka Streams state stores dynamically 🤩
Welcome @ableegoldman.bsky.social to BlueSky! Kafka Streams extraordinaire!
It makes a huge difference because it lets you avoid doing remote lookups for the first key in each window. Without the filters we were seeing a 50% drop-off in throughput at the start of each window.
Some problems are impossible to solve without stream processing: for instance, did you know that metronome.com leverages Kafka and Kafka Streams to deliver real time billing features like spend limits at scale? (1/3)
Coming from Kafka land, I’m always surprised at how “time” is an afterthought in many other databases.
Hoping SlateDB gets it right: 1) have only one clock, 2) let users specify the clock, 3) enforce monotonic clocks, 4) use seq numbers (not time) for txns.
Are we missing anything?
Looking forward to this! We just started using SlateDB in @tensorlake.bsky.social’s compute engine, looking forward to not needing dynamodb for CAS. One less moving part!
@chris.blue when is the new release coming? :)
And it's done. Thank you so much @benesch.bsky.social for doing this. Now SlateDB will be able to have CAS for all major object stores. 🔥 It just works.
👋 @vigneshc.bsky.social! All, Vinesh is a maintainer and committer for slatedb.io. He implemented the manifest and has been working on fizzbee.io proofs with JP. In his spare time, he’s a manager for Azure’s streaming team. 😁
I am soon interviewing @apurvamehta.com from @responsive.dev and Yingjun Wu from @risingwave.bsky.social on all things stream processing.
This is going to be an amazing episode. Feel free to ask any questions related to Stream processing and I will add the most interesting ones. Shoot!
Yep flamegraph-rs is just for CPU profiles. I guess I should have figured given you mentioned bytehound/valgrind 😀. Bytehound has worked great for me in the past (on linux).
For a cpu profile, I found flamegraph-rs (github.com/flamegraph-r...) to be a useful starting point. I've only tried on linux but their README says it works on osx using dtrace. The README also suggests trying github.com/mstange/samply. It says it has better osx support. Not sure what that means.
Is it end of the road for RocksDB in stream processing?
Disaggregated state is the clearly superior architecture, with @responsive.dev investing heavily in SlateDB.io while Flink 2.0 has forked RocksDB.
Here's why we've bet on SlateDB for Kafka Streams: www.responsive.dev/blog/why-sla...
