Batch and Streaming Ingestion Patterns

Triggered execution: the batch pattern

Every ingestion design answers two questions that are easy to confuse. The first is when does processing happen? Start there, because it is the cleanest distinction available and it does not depend on any product choice.

A triggered ingestion pipeline processes new data on a schedule or when it is started manually.

This execution mode fits a batch ingestion pattern because available records are handled at distinct run boundaries. Work accumulates between runs; a run begins, drains what is waiting, and ends. That boundary is the whole idea of a batch.

Choose triggered mode when new data should be processed periodically or on demand. Those two words cover both ways a triggered run can begin:

One clarification prevents a common misreading. A triggered run can still process only newly available data, but its defining characteristic here is when processing begins. Triggered does not mean "reload everything" — it means the run waits for a clock or a person rather than for the data itself.

Continuous execution: the streaming pattern

The other cadence removes the boundary altogether.

A continuous ingestion pipeline processes new data as it arrives in the source.

This execution mode fits a streaming ingestion pattern because the pipeline remains active instead of waiting for a scheduled or manual run. Nothing accumulates while it waits, because it is not waiting.

Choose continuous mode when new source data should be processed as it arrives. A continuous run is defined by ongoing arrival-driven processing — the source, not a schedule, sets the pace.

Side by side, the two cadences answer the same question differently:

Neither is a quality judgement. A nightly intake is not a worse pipeline than a streaming one; it is a pipeline whose requirement is expressed in hours rather than seconds.

Structured Streaming: the customizable engine

The second question is how much of the pipeline do you want to run yourself? The documented stack answers it with three layers, and the most hands-on of them is the engine underneath everything else.

Apache Spark Structured Streaming is a streaming engine with end-to-end fault tolerance and exactly-once processing guarantees through Spark APIs.

Two promises are packed into that sentence. End-to-end fault tolerance means a failure does not leave the pipeline in an unknown state. Exactly-once processing means a record is neither lost nor applied twice when a run recovers.

It is the most customizable ingestion layer in the documented stack. You write against Spark APIs, which is exactly what you want when the workload has requirements no declarative framework expresses — and exactly what you do not want when it does not, because everything you write is yours to maintain.

What this layer does not do is take over source-specific work. Authentication schemes, change data capture, and long-term API maintenance for a particular source are not part of it; those belong to the most automated layer, further up.

Lakeflow pipelines: declarative management on top

The middle layer keeps that engine and takes the operational work off your hands.

Lakeflow pipelines extend Structured Streaming with a declarative framework that manages orchestration, monitoring, data quality, errors, and other pipeline concerns.

Declarative means you state what the pipeline should produce rather than coding every mechanism that gets it there. They provide more automation and less operational overhead than using Structured Streaming directly.

Note the boundary carefully, because two plausible-sounding claims are wrong.

Lakeflow pipelines do not add source-specific authentication or long-term API maintenance — that is the layer above. And they are not restricted to batch pipelines: the declarative framework is built over a streaming engine, and it serves both patterns.

If a streaming workload benefits from declarative orchestration, monitoring, data quality, and error management, Lakeflow pipelines supply the more managed layer. If the same workload needs direct control through Spark APIs instead, Structured Streaming supplies the customizable one.

Fully-managed connectors: the most automated layer

The top of the stack builds on the layer below it rather than replacing it.

Fully-managed connectors build on Lakeflow pipelines and add source-specific authentication, change data capture, edge-case handling, long-term API maintenance, automated retries, and automated schema evolution for supported sources.

They are the most automated layer in this stack. Every item on that list is work that would otherwise be written and then maintained forever, and each of them is specific to the source rather than generic pipeline plumbing.

Two words limit the top layer: supported sources. Automation exists only where a connector exists.

Which layer to start from

With three layers on the table, the selection rule is a direction rather than a preference.

Start with the most managed layer and move to the next less-managed layer when the current layer does not meet the requirement, such as when it does not support the source.

Two things follow. First, an unmet requirement is the only reason to move down — an unsupported source, or behaviour the layer cannot express. Second, the movement is one-directional: you do not begin at the engine and climb toward automation, because the point of the rule is to avoid writing what you could have been given.

Keep the rule away from cadence, too. Moving between layers has nothing to do with switching between triggered and continuous execution; those are answers to a different question entirely.

Two decisions, taken separately

The batch-versus-streaming decision and the implementation-layer decision answer different questions. The first decides when ingestion runs. The second decides how much pipeline behavior the platform manages.

The clearest proof that they are independent is this: Lakeflow pipelines can be used to build both batch and streaming data pipelines in SQL and Python. Therefore, selecting Lakeflow pipelines does not by itself determine whether execution is triggered or continuous.

Laid out as a grid, all four corners are legitimate designs:

So read a requirement in two passes. "Records must be handled as they appear" fixes the column. "The team wants declarative orchestration, monitoring, and data-quality management" fixes the row. Neither phrase answers the other question, and a design that treats them as one will get one of the two wrong.

Key takeaways

• Triggered mode processes new data on a schedule or after a manual trigger; continuous mode processes new data as it arrives.
• Structured Streaming provides Spark API control with end-to-end fault tolerance and exactly-once processing guarantees.
• Lakeflow pipelines add declarative management of orchestration, monitoring, data quality, and errors.
• Fully-managed connectors build on those pipelines and add source-specific authentication, change data capture, edge-case handling, long-term API maintenance, automated retries, and automated schema evolution for supported sources.
• Begin with the most managed suitable layer and move toward more customization only when requirements demand it.

Two requirements, resolved with both axes:

Arrival-driven processing, with direct control through Spark APIs. The cadence is continuous. The layer is Structured Streaming, because "direct control through Spark APIs" is a requirement only the most customizable layer meets.

A supported streaming source, arrival-driven, with declarative transformations, monitoring, and data-quality management. The cadence is still continuous — the arrival requirement did not change. But the layer moves up: declarative orchestration, monitoring, and data quality are exactly what Lakeflow pipelines manage, so writing it against Spark APIs would mean building all of that by hand.

One phrase changed the layer and left the cadence alone. That is the whole point of keeping the two decisions apart.