Reviewed by Aditya Kumar · Last reviewed 2026-03-24
**Why this matters**: APIs are external dependencies—unreliable, rate-limited, schema-evolving. Production ingestion must be resilient, auditable, and idempotent. **Steps**: (1) **Contract & auth**: Define schema (Pydantic, JSON Schema); store credentials in secrets manager...
Red Flag: Saying you'd 'parse and transform in memory' without mentioning idempotency, retries, or landing raw data. Pro-Move: 'We always land raw JSON to S3 with run_id; our dbt models are idempotent and can re-process any partition. We use a typed contract (Pydantic) so schema drift fails the job before corrupting the warehouse.'
This medium-level General/Other question appears frequently in data engineering interviews at companies like Altimetrik, Infosys. While less common, it tests deeper understanding that distinguishes strong candidates. Mastering the underlying concepts (etl, partition) will help you answer variations of this question confidently.
Break this problem into components. Identify the core trade-offs involved, then walk the interviewer through your reasoning step by step. Demonstrate awareness of edge cases and production considerations - this is what separates good answers from great ones.
Why this matters: APIs are external dependencies—unreliable, rate-limited, schema-evolving. Production ingestion must be resilient, auditable, and idempotent. Steps: (1) Contract & auth: Define schema (Pydantic, JSON Schema); store credentials in secrets manager (Vault, AWS Secrets). OAuth tokens need refresh logic. (2) Pagination & rate limiting: Offset/cursor-based; loop until empty. Respect Retry-After; implement exponential backoff. (3) Land raw first: Write to object storage (S3, GCS) or landing table with partition by date/run_id. Never transform in-flight—preserves audit trail and enables replay. (4) Validate: Schema checks, row counts, null ratios. Fail fast or quarantine. (5) Process: ETL into staging, then warehouse. Idempotent by run_id or watermark. (6) Monitor: Log success/failure, alert on anomalies (volume drops, schema drift). Scalability: Parallelize by partition (e.g., date range) or use message queues for high-volume APIs. Cost: API calls may be metered; batching and incremental fetches reduce cost.
Red Flag: Saying you'd 'parse and transform in memory' without mentioning idempotency, retries, or landing raw data. Pro-Move: 'We always land raw JSON to S3 with run_id; our dbt models are idempotent and can re-process any partition. We use a typed contract (Pydantic) so schema drift fails the job before corrupting the warehouse.'
Practice the 40 most asked data engineering questions at Altimetrik. Covers Behavioral, Spark/Big Data, Python/Coding and more.
8 min read →Practice the 39 most asked data engineering questions at Infosys. Covers Spark/Big Data, Python/Coding, Cloud/Tools and more.
8 min read →Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
According to DataEngPrep.tech, this is one of the most frequently asked General/Other interview questions, reported at 2 companies. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.