Pushing computation down to the data
A theme that keeps coming up in data systems work: the cheapest byte to move is the one you never move. When a query engine and a storage system live on opposite sides of a network, where you do the work matters as much as what work you do.
The default is wasteful
Say you join two tables, \(A\) and \(B\), that live in remote storage. The naive plan reads both tables across the network and joins them on the client:
\[\text{bytes moved} \;\approx\; |A| + |B|\]But the result you actually wanted is usually much smaller than its inputs. If the engine can push the join down to run next to the data, you only ship the answer back:
\[\text{bytes moved} \;\approx\; |A \bowtie B|\]When the join is selective — say it keeps a fraction \(s \ll 1\) of the rows — the win is roughly \(\tfrac{|A| + |B|}{s \cdot |A| \cdot |B|}\) less data crossing the wire. On a TPC-DS 10 TB workload, pushing down filters, aggregations, and joins this way got some queries to around 100× faster. Most of that wasn’t clever CPU work; it was simply refusing to move data that the query was only going to throw away.
What “pushdown” really buys you
Three things, roughly in order of impact:
- Less network I/O. The big one, as above.
- Less client-side memory pressure. You’re not materializing giant intermediate tables just to discard most of them.
- Better parallelism at the source. Storage-side execution can fan the work out across partitions before anything comes back.
The catch is that the engine’s optimizer has to know what the source can do. That’s the real work in a connector: faithfully advertising which filters, projections, and operators you can handle, and gracefully falling back when you can’t.
More on that fallback logic — and how to keep the optimizer honest — in a later post.