Data Engineering Fundamentals final test: 25 free questions

Q1 / 25

How does `reschedule` mode reduce a Sensor's capacity use?

1. AIt converts the Sensor into an asset-producing task
2. BIt runs every check inside the Dag processor process
3. CIt creates a global Connection
4. DIt releases the worker slot between checks

Q2 / 25

Which statement is true of both Parquet and ORC in Spark?

1. AThey can be read only after conversion to CSV
2. BNeither format supports typed analytical data
3. CEach file alone supplies atomic commits across all files in a changing table
4. DSpark can read and write them as efficient compact file formats

Q3 / 25

Which pair correctly classifies an RDD transformation and an action?

1. A`collect` is a transformation; `filter` is an action
2. B`map` is a transformation; `reduce` is an action
3. C`reduce` is a transformation; `map` is an action
4. D`count` is a transformation; `first` is another transformation

Q4 / 25

Why can Spark SQL apply additional optimizations compared with the basic RDD API?

1. AIt stores every structured record in a Hive table
2. BIt receives structural information about data and computation
3. CIt runs each language through a separate execution engine
4. DIt receives column names but no computation structure

Q5 / 25

Which two subjects does the Spark tuning guide treat as its main concerns?

1. APrecedence rules and unit suffixes
2. BAuthentication filters and channel encryption
3. CData serialization and memory tuning
4. DScheduler pools and executor idle timeouts

Q6 / 25

A wait condition must be checked roughly once per minute, may take hours to occur, and worker capacity is scarce. Which mode best balances the stated latency and capacity constraints?

1. AUse `reschedule` so the slot is held only during each check
2. BUse a local executor and keep the check running in the scheduler process
3. CUse `poke` for the whole wait
4. DStore the condition in XCom and remove the Sensor from the workflow

Q7 / 25

Which constraint most directly favors ETL?

1. AAnalysts need to repeatedly transform retained raw inputs
2. BThe target can scale transformations in parallel
3. CMany input formats should land before their use is known
4. DSensitive raw fields must be masked before entering the target

Q8 / 25

An architecture must let Java and Python teams describe work through supported interfaces while retaining one optimized execution layer for nontrivial graphs. Which design matches Spark's model?

1. AUse the Java and Python high-level APIs over Spark's optimized engine
2. BGive each language a separate Spark engine that supports only linear execution
3. CUse the cluster manager as the API and bypass Spark's execution engine
4. DConvert both teams to R because the deprecated API unifies execution

Q9 / 25

A table holds several rows per customer identifier that differ in their event timestamp and in other columns. The output must keep exactly one row per customer, choose the most recent event, and preserve the remaining columns of that row. Which documented construction fits?

1. AAggregate the customers with collect_set so that every identifier appears once in the output of the query
2. BSelect the rows with the DISTINCT qualifier so that the repeated customer identifiers are removed from the result
3. CNumber the rows with rank in a partition by customer ordered by descending timestamp, and keep the rows numbered one
4. DNumber the rows with row_number in a partition by customer ordered by descending timestamp, and keep number one

Q10 / 25

Which Airflow surface lists state, try number, duration, operator type, and Dag version for every task in one run?

1. AThe Dag Run Task Instances tab
2. BThe Dag Details Runs tab
3. CThe Home Page recent-asset panel
4. DThe Task Instance XCom tab

Q11 / 25

How should one task make a large generated file available to a downstream task?

1. APlace the file contents inside the Dag's Connection password field
2. BPut the complete file in XCom
3. CStore the file remotely and pass its location through XCom
4. DWrite the file only to a worker's local disk and assume worker reuse

Q12 / 25

A query is moved from the default execution mode to Continuous Processing. What happens to its guarantees and its latency floor?

1. AGuarantees stay exactly-once and latency stays near one hundred milliseconds
2. BGuarantees stay exactly-once and latency can fall to one millisecond
3. CGuarantees drop to at-least-once and latency stays near one hundred milliseconds
4. DGuarantees drop to at-least-once and latency can fall to one millisecond

Q13 / 25

Queries usually filter by event date, rarely by event ID, and daily volume is high enough for parallel scans. Which partition strategy is strongest?

1. AUse date boundaries and size files within each date for useful parallel work
2. BUse event ID boundaries because high cardinality guarantees less listing overhead
3. CCreate one partition per event ID and keep every resulting file as small as possible
4. DKeep one file for all dates because common date filters cannot benefit from layout

Q14 / 25

What is an RDD?

1. AA fault-tolerant collection partitioned across nodes for parallel operations
2. BA driver-only collection that cannot be divided across cluster nodes
3. CA cluster manager that allocates executors to multiple Spark applications
4. DA mutable variable whose executor updates automatically reach the driver

Q15 / 25

How do execution memory and storage memory relate in the documented memory model?

1. AThey are separate fixed regions, and neither one may ever borrow space from the other
2. BThey share a unified region, and caching always receives the whole of it before any execution memory is used
3. CThey share one region; execution may evict storage down to a reserved threshold, but not the reverse
4. DThey share a unified region, and storage may evict execution whenever a cached block needs room

Q16 / 25

A historical reprocessing must never put more than three runs in flight at once. Which backfill setting bounds that?

1. AThe ordering in which the runs are created
2. BThe start and end dates of the range
3. CThe reprocess behavior chosen for the range
4. DThe maximum number of active runs allowed

Q17 / 25

Beyond running workflows, what does the Airflow web interface contribute?

1. AA metrics store that replaces the processing engine's own monitoring
2. BA surface for visualizing, managing, and debugging the workflows
3. CA catalog that documents the ownership and freshness of served tables
4. DAn authoring surface that removes the need to write Python

Q18 / 25

How do append and update modes differ for a watermarked aggregation?

1. AAppend waits for finalization; update can revise rows before state expires
2. BAppend revises rows immediately; update waits until the threshold passes
3. CAppend retains all state; update rewrites the whole target each trigger
4. DAppend supports outer joins; update writes unmatched rows before expiration

Q19 / 25

A source contains 500 million rows, about 0.2 percent change daily, updates to old keys matter, and the pipeline can store a reliable last-modified watermark. Which load design minimizes repeated work without losing revisions?

1. AReplace the entire target from a complete source extract daily
2. BAppend every extracted row as a new target record
3. CSelect and apply rows beyond the saved change watermark
4. DLoad only newly created keys and ignore older modified keys

Q20 / 25

What does a Spark Connect client send to the server?

1. AExecutor JVM objects serialized through Py4J into the client process
2. BRDD partitions encoded as cluster-manager resource requests over YARN
3. CDriver memory pages streamed as Apache Arrow batches to Kubernetes
4. DUnresolved logical plans encoded with protocol buffers over gRPC

Q21 / 25

Why does the Python documentation encourage indexing a DataFrame column rather than reading it as an attribute?

1. AOnly the indexing form works inside a projection, and the attribute form is limited to filters
2. BThe attribute form returns a plain Python value while indexing returns a Column instance
3. CIt is future proof and does not break with column names that are also class attributes
4. DThe attribute form is evaluated eagerly while the indexing form keeps the plan lazy

Q22 / 25

Where should an operator look first after an Airflow task fails?

1. AThe Task Instance Logs tab
2. BThe Dag Details Code tab
3. CThe Asset Graph View
4. DThe Home Page health indicators

Q23 / 25

What is the basic unit of execution in Airflow?

1. AA Dag schedule
2. BA task arranged inside a Dag
3. CA data interval assigned to a worker
4. DA callback stored outside the workflow

Q24 / 25

Which configuration parameter gates Structured Streaming metrics, and what is its default value?

1. Aspark.metrics.executorMetricsSource.enabled, whose default value is false
2. Bspark.sql.streaming.metricsEnabled, whose default value is false
3. Cspark.metrics.executorMetricsSource.enabled, whose default value is true
4. Dspark.sql.streaming.metricsEnabled, whose default value is true

Q25 / 25

Which evidence shows ORC is handled as a columnar format by Spark?

1. ASpark requires ORC to be converted into JSON rows before any job can scan it
2. BSpark disables compression whenever ORC is written
3. CSpark treats ORC only as an orchestration asset
4. DSpark exposes vectorized ORC reader and writer batch settings