Read data from three files into a Pandas DataFrame, perform transformations, remove columns, filter rows, search for strings

Why This Pattern: Multi-file concat + transform + filter is the core ETL pattern. At JP Morgan, regulatory data often comes as daily files—concat, validate, filter, load.

Scalability: pd.concat([df1,df2,df3]) loads all into memory. For 3 large files: read each with chunksize, concat chunks, process incrementally. Or use Dask for out-of-core. For string search: df['col'].str.contains('pat', na=False)—vectorized; avoid apply with regex for big data.

Best Practice: Specify dtypes, parse_dates. Log row counts per step. Use query() for complex filters. Write Parquet for columnar storage.

dfs = [pd.read_csv(f) for f in files]
df = pd.concat(dfs).drop(columns=['drop'])
df = df[df['amount']>0 & df['name'].str.contains('X', na=False)]