Reviewed by Aditya Kumar · Last reviewed 2026-03-24
Why this design: Separation of concerns—Operators define what to do; Hooks abstract how to connect; Scheduler coordinates when. Operators: single unit of work (BashOperator, PythonOperator); encapsulate logic; should be idempotent. Hooks: interface to external systems; manage...
Red Flag: Heavy logic in DAG files or operators that aren't idempotent. Pro-Move: 'We use K8s executor—autoscale 0-50 workers; DAG parse time under 30s with minimal imports.'
Why this design: Separation of concerns—Operators define what to do; Hooks abstract how to connect; Scheduler coordinates when. Operators: single unit of work (BashOperator, PythonOperator); encapsulate logic; should be idempotent. Hooks: interface to external systems; manage connections, connection pooling; reused across operators. Scheduler: reads DAGs, evaluates DAG/task state, triggers ready tasks; uses executor (LocalExecutor, Celery, K8s) to distribute work. Scalability: CeleryExecutor/K8s—horizontal scaling; LocalExecutor = single point. Trade-offs: K8s scales to zero but has cold-start; Celery needs Redis + workers. Cost: Scheduler is single point; HA requires careful setup. DAG parsing—keep DAG files light; heavy imports slow parsing.
Red Flag: Heavy logic in DAG files or operators that aren't idempotent. Pro-Move: 'We use K8s executor—autoscale 0-50 workers; DAG parse time under 30s with minimal imports.'
According to DataEngPrep.tech, this is one of the most frequently asked Cloud/Tools interview questions, reported at 2 companies. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.