Reviewed by Aditya Kumar · Last reviewed 2026-03-24
**Why This Pattern:** Multi-file concat + transform + filter is the core ETL pattern. At JP Morgan, regulatory data often comes as daily files—concat, validate, filter, load. **Scalability:** pd.concat([df1,df2,df3]) loads all into memory. For 3 large files: read each with...
Why This Pattern: Multi-file concat + transform + filter is the core ETL pattern. At JP Morgan, regulatory data often comes as daily files—concat, validate, filter, load.
Scalability: pd.concat([df1,df2,df3]) loads all into memory. For 3 large files: read each with chunksize, concat chunks, process incrementally. Or use Dask for out-of-core. For string search: df['col'].str.contains('pat', na=False)—vectorized; avoid apply with regex for big data.
Best Practice: Specify dtypes, parse_dates. Log row counts per step. Use query() for complex filters. Write Parquet for columnar storage.
dfs = [pd.read_csv(f) for f in files]
df = pd.concat(dfs).drop(columns=['drop'])
df = df[df['amount']>0 & df['name'].str.contains('X', na=False)]
Red Flag: pd.concat without ignore_index=True when indices don't matter. Pro-Move: 'We validate schema with pd.io.json.build_table_schema before concat—catch schema drift early.'
According to DataEngPrep.tech, this is one of the most frequently asked Python/Coding interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.