Reviewed by Aditya Kumar · Last reviewed 2026-03-24
**Why secrets management**: Credentials in DAG code get committed to Git—instant breach. Airflow Variables and Connections stored in metadata DB are better but still risk exposure. **Architecture**: Google Secret Manager as source of truth. Configure Airflow Secrets Backend...
This hard-level Cloud/Tools question appears frequently in data engineering interviews at companies like Aarete. While less common, it tests deeper understanding that distinguishes strong candidates. Mastering the underlying concepts (airflow) will help you answer variations of this question confidently.
This is a senior-level question that tests architectural thinking. Lead with the high-level design, then drill into specifics. Discuss trade-offs explicitly - there is rarely one correct answer. Show awareness of scale, fault tolerance, and operational complexity.
Why secrets management: Credentials in DAG code get committed to Git—instant breach. Airflow Variables and Connections stored in metadata DB are better but still risk exposure. Architecture: Google Secret Manager as source of truth. Configure Airflow Secrets Backend (CloudSecretManagerBackend) so Variable.get() and Connection lookups resolve from Secret Manager. Create secrets in Secret Manager; reference by path. Use workload identity—Composer's GSA gets minimal IAM roles (e.g., secretAccessor); no key files. Scalability: Secret Manager has quotas; for high-frequency lookups, cache in task (not in DAG parsing). Cost: Secret Manager charges per secret version and access; at 10K accesses/month, cost is negligible. Blast radius: Use VPC-SC for data exfiltration protection. Rotate secrets regularly; use Workload Identity Federation for cross-project access without keys. Never log secret values; use get_conn_id for connections. In production, we moved all 200+ connections to Secret Manager and eliminated 15+ key files.
Pro-Move: 'We use Workload Identity Federation so Composer tasks access cross-project BigQuery without a single JSON key—zero secrets to rotate.' Red Flag: Storing credentials in Variables or hardcoding—immediate disqualifier for security-sensitive roles.
Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
According to DataEngPrep.tech, this is one of the most frequently asked Cloud/Tools interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.