Implement a Python function to count unique words from a file and write them to another file.

To count unique words from a file and write them to another, use a Python set for efficient deduplication and standard file I/O.

Mechanics and Why

The core idea is to read the input file line by line, split each line into words, and add these words to a set. A set automatically handles uniqueness, as it can only contain distinct elements, offering average O(1) time complexity for additions. After processing all lines, the set will contain every unique word. These can then be sorted and written to the output file, each on a new line. While collections.Counter can count word frequencies, set is more direct for simply identifying unique words.

Production Considerations and Trade-offs

For production-grade solutions, several aspects need attention: * Encoding: Always specify encoding='utf-8' (or the correct encoding) when opening files to prevent UnicodeDecodeError with non-ASCII characters. * Tokenization: The default str.split() handles whitespace. For more robust word extraction, consider: * Case-insensitivity: Convert words to lowercase (word.lower()) before adding to the set. * Punctuation: Remove leading/trailing punctuation using word.strip(string.punctuation) or regular expressions (re.sub(r'[^\w\s]', '', word)). * Large Files: For files exceeding available memory, the current approach of loading all unique words into a single set might fail. * Memory Efficiency: Ensure you're processing line-by-line using file iterators. * Distributed Processing: In big data environments, this task would be distributed. Frameworks like Apache Spark could read the file in partitions, map each word to a key, and then use a distinct or groupByKey followed by count operation to get unique words across the cluster, handling shuffling and memory management. * External Sorting: If the final sorted list of unique words is too large to fit in memory, an external sort algorithm would be required.

import string
def count_unique_words(input_filepath: str, output_filepath: str):
    unique_words = set()
    with open(input_filepath, 'r', encoding='utf-8') as infile:
        for line in infile:
            for word in line.lower().split(): # Case-insensitive tokenization
                cleaned_word = word.strip(string.punctuation)
                if cleaned_word: # Avoid adding empty strings after cleaning
                    unique_words.add(cleaned_word)
    
    with open(output_filepath, 'w', encoding='utf-8') as outfile:
        outfile.write('\n'.join(sorted(unique_words)))

In the interview, also mention asking clarifying questions about case sensitivity, punctuation handling, and expected file size.