Reviewed by Aditya Kumar · Last reviewed 2026-03-24
**Why Unix in Data Eng:** File staging, cleanup, lightweight transforms before Spark. find, awk, sed, xargs, cronâubiquitous, no runtime deps. **Use Cases:** find /data -name '*.csv' -mtime +7 | xargs gzip. awk -F',' '{print $1,$3}' for column extraction. sed for bulk replace....
Why Unix in Data Eng: File staging, cleanup, lightweight transforms before Spark. find, awk, sed, xargs, cronâubiquitous, no runtime deps.
Use Cases: find /data -name '.csv' -mtime +7 | xargs gzip. awk -F',' '{print $1,$3}' for column extraction. sed for bulk replace. Cron for scheduling: 0 2 /scripts/ingest.sh.
Scalability: Shell for single-node, pre-processing. For distributed: Spark. Best practice: set -e (exit on error), validate paths, log to file. Combine: shell orchestrates, Python/Spark does heavy lifting.
find /data -name '*.csv' -mtime +7 | xargs -I {} gzip {}
Red Flag: Complex logic in shell (unreadable). Pro-Move: 'We use shell for find|gzip|xargs; Python for parsingâright tool per layer.'
According to DataEngPrep.tech, this is one of the most frequently asked Python/Coding interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.