Essential cookies keep authentication working. With your permission, we also use analytics cookies to understand and improve the product. Read our Privacy Policy

DataEngPrep.tech
QuestionsPracticeAI CoachDashboardPricingBlog
ProLogin
Home/Questions/SQL/How would you identify duplicate records based on a composite key in SQL?

How would you identify duplicate records based on a composite key in SQL?

SQLmedium0.8 min read

Reviewed by Aditya Kumar · Last reviewed 2026-03-24

To identify duplicates by composite key in SQL, use GROUP BY with HAVING COUNT(*) > 1, or window functions. Example: SELECT order_id, product_id, COUNT(*) cnt FROM orders GROUP BY order_id, product_id HAVING COUNT(*) > 1. Window approach: SELECT * FROM (SELECT *, ROW_NUMBER()...

🤖 Analyze Your Answer
Frequency
Low
Asked at 1 company
Category
487
questions in SQL
Difficulty Split
130E|271M|86H
in this category
Total Bank
1,863
across 7 categories
Asked at these companies
Amazon
Key Concepts Tested
partitionsqlwindow

Why This Question Matters

This medium-level SQL question appears frequently in data engineering interviews at companies like Amazon. While less common, it tests deeper understanding that distinguishes strong candidates. Mastering the underlying concepts (partition, sql, window) will help you answer variations of this question confidently.

How to Approach This

Break this problem into components. Identify the core trade-offs involved, then walk the interviewer through your reasoning step by step. Demonstrate awareness of edge cases and production considerations - this is what separates good answers from great ones.

Expert Answer
151 words

To identify duplicates by composite key in SQL, use GROUP BY with HAVING COUNT() > 1, or window functions. Example: SELECT order_id, product_id, COUNT() cnt FROM orders GROUP BY order_id, product_id HAVING COUNT() > 1. Window approach: SELECT FROM (SELECT *, ROW_NUMBER() OVER (PARTITION BY order_id, product_id ORDER BY created_at) rn FROM orders) t WHERE rn > 1. The GROUP BY method returns only the composite key and count; the window method returns full rows of duplicates. For removal, use DELETE with a CTE: WITH dups AS (SELECT order_id, product_id, ROW_NUMBER() OVER (PARTITION BY order_id, product_id ORDER BY created_at DESC) rn FROM orders) DELETE FROM orders WHERE (order_id, product_id) IN (SELECT order_id, product_id FROM dups WHERE rn > 1); Why it matters: Design choices compound at scale—wrong approach can cause 100× overhead. Scalability trade-offs: Profile before optimizing; validate on sample then full. Cost implications: Suboptimal choices multiply at billion-row scale.

⚡
Pro Tip

Red Flag: Assuming pipeline success means data correctness. Pro-Move: 'We added row checksums and reconciliation—caught 0.02% drift that success status missed.'

Want all answers as a PDF for offline study?
Seven focused volumes with 750+ in-depth answers — Answer Vault →

Related SQL Questions

mediumWrite an SQL query to find the second-highest salary from an employee table.FreemediumDemonstrate the difference between DENSE_RANK() and RANK()FreemediumDiscuss differences between ROW_NUMBER(), RANK(), and DENSE_RANK(), and provide examples from your projects.FreemediumExplain the differences between Data Warehouse, Data Lake, and Delta LakeFreemediumExplain the differences between Repartition and Coalesce. When would you use each?Free

Level up your prep

Recommended
Educative
Educative Unlimited

800+ hands-on courses — Grokking System Design, Coding Patterns, and AI mock interviews for your DE loop.

Start learning →

Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.

According to DataEngPrep.tech, this is one of the most frequently asked SQL interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.

← Back to all questionsMore SQL questions →
Categories
All QuestionsSQLSpark / Big DataPython / CodingSystem DesignCloud / ToolsBehavioral
By Company
AmazonGoogleDatabricksSnowflakeAWSAzureMicrosoftNetflixUberTCS
Interview Guides
All GuidesTop SQL QuestionsTop Spark QuestionsPySpark QuestionsTop Python QuestionsTop System DesignKafka QuestionsAirflow QuestionsSQL Window FunctionsETL QuestionsData Modeling
Products
AI Interview CoachAnswer AnalyzerSQL PlaygroundResume AnalyzerAnswer Vault PDFsPricing
Company
About & Editorial PolicyContact UsAI DisclosureDisclaimerTerms of ServicePrivacy Policy
© 2026 DataEngPrep.tech. All rights reserved.
AboutBlogContactDisclaimer