Question 1

Explain the Medallion Architecture (Bronze, Silver, Gold layers).

Accepted Answer

**Section 1 — The Context (The 'Why')**
Medallion addresses tension between raw preservation (audit) and curated analytics. Bronze retains everything; Silver/Gold add structure.

**Section 2 — The Diagram**
```
[Sources] --> [Bronze] --> [Silver] --> [Gold]
  Append    Dedup/Merge   Marts
```

**Section 3 — Component Logic**
**Bronze** raw append-only, immutable. **Silver** schema, dedup, Delta MERGE. **Gold** star schemas. **TTL policies** move Bronze to Glacier....

Question 2

Explain the difference between Service Principal and Managed Identity in Azure.

Accepted Answer

Architectural difference: Service Principal = app identity; client ID, tenant, secret/certificate; for automation, CI/CD, third-party. Managed Identity = Azure-managed; no secrets; tied to resource (VM, ADF, Function). System-assigned vs user-assigned. When: Managed Identity for Azure resources—simpler, no secret rotation. Service Principal for external, non-Azure, Terraform....

Question 3

Secret Scope usage for managing credentials securely.

Accepted Answer

**Why Secret Scopes**: Credentials in notebooks or config get committed or logged. Secret Scopes provide a namespace for secrets, accessed via dbutils.secrets.get(scope="my-scope", key="api-key"). Never log the return value. **Types**: Databricks-backed (stored in Databricks) and Key Vault-backed (Azure) or AWS Secrets Manager-backed—enterprise secrets stay in your vault. **Scalability**: Many secrets across many clusters—use ACLs to restrict scope access per group....

Question 4

When to use each for secure resource access.

Accepted Answer

**IAM roles:** Service-to-service (Lambda→S3). Prefer over keys. **Service accounts:** Apps; rotate creds. **OAuth2/OpenID:** User-facing; PKCE for public. **API keys:** External partners; narrow scope; rotate. **Secrets managers:** Store keys; never commit. **Why each:** Least privilege; workload identity when available....

Question 5

Importance of each layer in Databricks.

Accepted Answer

**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.

Databricks layers: (1) Bronze—raw ingestion, schema on read. (2) Silver—cleaned, deduplicated, validated. (3) Gold—aggregated, business-ready. Medallion architecture enables incremental processing, auditability, and clear ownership....

Question 6

Steps to mount storage in Databricks.

Accepted Answer

**Why Mount vs. URI**: Mounts provide a stable path (`/mnt/data`) and handle auth once; URIs (`s3://bucket/path`) require per-access credentials. For multi-workspace, Unity Catalog external locations replace mounts.

**Steps (Azure ADLS)**: (1) Create service principal; grant Storage Blob Data Contributor. (2) Mount: `dbutils.fs.mount(source="abfss://container@account.dfs.core.windows.net/", mount_point="/mnt/mydata", extra_configs={...})`....

Chubb Data Engineer Interview Questions

+6 More Questions with Expert Answers

Other Companies

Chubb Data Engineer Interview Questions

+6 More Questions with Expert Answers

Other Companies