“Streaming vs. Batch” Is a Wrong Dichotomy, and I Think It's Confusing

"Streaming vs. Batch" Is a Wrong Dichotomy, and I Think It's Confusing Often times, "Stream vs. Batch" is discussed as if it’s one or the other, but to me this does not make that much sense really. Many streaming systems will apply batching too, i.e. processing or transferring multiple records (a "batch") at once, thus offsetting connection overhead, amortizing the cost of fanning out work to multiple threads, opening the door for highly efficient SIMD processing, etc., all to ensure high performance. The prevailing trend towards storage/compute separation in data streaming and processing architectures (for instance, thinking of platforms such as WarpStream, and Diskless Kafka at large) further accelerates this development. Typically, this is happening transparently to users, done in an opportunistic way: handling all of those records (up to some limit) which have arrived in a buffer since the last batch. This makes for a very nice self-regulating system. High arrival rate of record ... Read full article.