Latest Tech News

Stay updated with the latest in technology, AI, cybersecurity, and more

Filtered by: clickhouse Clear Filter

Load Test GlassFlow for ClickHouse: Real-Time Dedup at Scale

news.ycombinator.com Unknown 2025-08-31 16:56:10

Load Test GlassFlow for ClickHouse: Real-Time Deduplication at Scale By Ashish Bagri, Co-founder & CTO of GlassFlow TL;DR We tested GlassFlow on a real-world deduplication pipeline with Kafka and ClickHouse. It handled 55,00 records/sec published by Kafka and processed 9,000+ records/sec on a MacBook Pro, with sub-0.12ms latency. No crashes, no message loss, no disordering. Even with 20M records and 12 concurrent publishers, it remained robust. Want to try it yourself? The full test setup

Topics: clickhouse glassflow kafka test time
Read More
Shop Amazon

Scaling our observability platform by embracing wide events and replacing OTel

news.ycombinator.com Unknown 2025-09-03 08:23:21

TLDR # Observability at scale: Our internal system grew from 19 PiB to 100 PB of uncompressed logs and from ~40 trillion to 500 trillion rows. Efficiency breakthrough: We absorbed a 20× surge in event volume using under 10% of the CPU previously needed. OTel pitfalls: The required parsing and marshalling of events in OpenTelemetry proved a bottleneck and didn’t scale - our custom pipeline addressed this. Introducing HyperDX: ClickHouse-native observability UI for seamless exploration, correlatio

Topics: clickhouse data loghouse query time
Read More
Shop Amazon

ClickHouse scales beyond 100 petabytes of logs

news.ycombinator.com Unknown 2025-09-03 13:23:21

TLDR # Observability at scale: Our internal system grew from 19 PiB to 100 PB of uncompressed logs and from ~40 trillion to 500 trillion rows. Efficiency breakthrough: We absorbed a 20× surge in event volume using under 10% of the CPU previously needed. OTel pitfalls: The required parsing and marshalling of events in OpenTelemetry proved a bottleneck and didn’t scale - our custom pipeline addressed this. Introducing HyperDX: ClickHouse-native observability UI for seamless exploration, correlatio

Topics: clickhouse data loghouse query time
Read More
Shop Amazon
Trending Topics
Hot Now
ai 1495 apple 1095 new 903 content 844 google 676
Popular
like 654 zdnet 647 amazon 646 does 527 company 527 reviews 520 editorial 506 day 492 app 480 said 450
Emerging
data 447 prime 446 samsung 398 phone 394 game 385 pro 384 just 373 android 352 deals 338 galaxy 323
About GoKawiil

GoKawiil is a project by nerdhub.co that curates technology news from trusted sources. We built this site to provide a cleaner, more mobile-friendly experience without intrusive ads or heavy JavaScript.

Privacy

Your privacy matters. GoKawiil doesn't use Google Analytics or Facebook Pixel. The only tracking occurs through affiliate links to Amazon.

Advertising

Interested in advertising? Contact us at [email protected]