How do you copy all files from one source path to target in ADF?

In Azure Data Factory (ADF), you can copy all files from a source path to a target using primarily two methods: a single Copy Data activity with wildcards, or a combination of GetMetadata, ForEach, and Copy Data activities.

Method 1: Copy Data with Wildcards

This is the most straightforward and performant method for bulk file transfers. You configure a single Copy Data activity where the source dataset's file path or folder path includes a wildcard character (e.g., *). ADF efficiently handles the enumeration and copying of all matching files within that path.

# Example Source Dataset Path
data/raw/source_folder/*

This approach is ideal for "lift and shift" scenarios, archiving, or replicating entire directories where no per-file specific logic is required. ADF optimizes the transfer, often leveraging parallel processing internally for better throughput.

Method 2: GetMetadata, ForEach, and Copy Data

This method provides granular control over individual files. First, a GetMetadata activity is used to retrieve a list of child items (files and subfolders) from the source folder. Then, a ForEach activity iterates through this list. Inside the ForEach loop, a Copy Data activity copies each file, using dynamic content like @item().name or @item().fullName to specify the current file's path.

This pattern is necessary when you need to apply specific logic to each file, such as filtering files based on their name or size, conditionally processing them, or performing transformations before copying. While offering flexibility, it introduces overhead due to the iterative nature, potentially making it slower for a very large number of files compared to the wildcard approach.

In the interview, also mention: Discuss how error handling (e.g., on failure paths) and logging would be implemented for each method, especially for large-scale operations.