Count occurrences of a specific word in a file

Counting word occurrences efficiently depends heavily on file size and the execution environment. For small files, command-line tools or simple Python scripts suffice. For large, distributed files, a distributed processing framework like Apache Spark is essential.

Mechanics and Trade-offs

Command Line (Linux/Unix): For single, moderately sized files, grep -o 'word' file.txt | wc -l is effective for counting occurrences. Note that grep -c counts lines* containing the word, not individual occurrences. Use grep -i for case-insensitivity and grep -w for whole word matching. This approach is single-threaded and can be slow or memory-intensive for multi-gigabyte files.
* Python: For larger files that fit into memory, collections.Counter is efficient. For files exceeding available memory, iterate line by line, processing chunks to avoid loading the entire file. This can be combined with regular expressions (re module) for robust case-insensitivity (re.IGNORECASE) and word boundary (\b) handling.

    import re
    from collections import Counter
def count_word_occurrences_python(filepath: str, word: str) -> int:
        target_pattern = re.compile(r'\b' + re.escape(word) + r'\b', re.IGNORECASE)
        total_count = 0
        with open(filepath, 'r', encoding='utf-8') as f:
            for line in f:
                total_count += len(target_pattern.findall(line))
        return total_count
    

* Apache Spark (PySpark): For petabyte-scale data distributed across a cluster, Spark is ideal. It leverages parallel processing across partitions. The workflow involves reading the text file into an RDD or DataFrame, using flatMap to tokenize lines into individual words, filter to select the target word (handling case-insensitivity and word boundaries), and then count() to get the total. Spark handles fault tolerance and distributes computation, potentially involving shuffles for aggregations across worker nodes.

Best Practices

Always consider case-insensitivity (e.g., lower() in Python/Spark, grep -i) and word boundaries (e.g., \b in regex, grep -w) to ensure accurate counts. For large files, prioritize streaming or distributed processing over loading the entire file into memory to prevent OutOfMemory errors.

In the interview, also mention the importance of handling various character encodings (e.g., UTF-8) and the performance implications of complex regular expressions versus simpler string matching.