Handling Duplicate Records During Migration: Strategies That Work
- Ohana Focus Team
- Jan 24
- 9 min read
By Ohana Focus Team | January 17, 2025 | 19 min read
Every organization migrating from Raiser's Edge to Salesforce confronts the same uncomfortable truth: your source database contains duplicate records. Maybe it's 5%, maybe it's 20%, but they're there. Donors with multiple constituent records, organizations listed twice with slight name variations, and contacts appear in both individual and organizational contexts. These duplicates may be tolerated or even unnoticed in Raiser's Edge, but they become glaring problems in Salesforce, where data integrity standards are higher and duplicate detection is more sophisticated.
Let's examine some practical strategies for handling duplicates throughout the migration process: understanding why duplicates exist and what types you'll encounter, identifying duplicates in your source data before migration, preventing duplicate creation during the import process, using Salesforce tools to detect duplicates post-migration, and establishing ongoing duplicate prevention practices.
Understanding Duplicate Records: Types and Origins
Not all duplicates are created equal. Understanding different duplicate types helps you develop targeted detection and resolution strategies.
The Three Types of Duplicates
Exact Duplicates
These are true duplicates where nearly all fields match: John Smith at 123 Main St appears twice with identical names, addresses, and contact information. Often created through accidental re-import, data entry errors, or system glitches.
• Detection difficulty: Easy—automated matching catches these reliably
• Resolution confidence: High—clearly the same person
• Common in: Organizations with poor data entry protocols
Fuzzy Duplicates
These are likely duplicates with slight variations: 'John Smith' and 'Jon Smith,' 'ABC Corporation' and 'ABC Corp,' '123 Main Street' and '123 Main St.' Same person or organization, but data entry inconsistencies make exact matching fail.
• Detection difficulty: Moderate—requires fuzzy matching algorithms
• Resolution confidence: Moderate—usually the same entity, but edge cases exist
• Common in: Every organization—human data entry naturally creates variation
Questionable Duplicates
These might be duplicates or might be legitimate separate records: 'John Smith Sr.' and 'John Smith Jr.,' spouses with the same last name and address, employees at the same company. Only human judgment can determine if they're actually duplicates.
• Detection difficulty: Hard—automated tools flag but can't definitively determine
• Resolution confidence: Low—requires human review of context
• Common in: Family foundations, corporate giving programs
Why Duplicates Exist in Source Data
Understanding root causes helps prevent recurrence in Salesforce:
• No search before adding: Staff add records without checking if person/organization already exists
• Poor search effectiveness: Raiser's Edge search doesn't find existing records due to spelling variations
• Multiple data entry points: Event registration creates one record, gift processing creates another
• Legacy system migrations: Previous migrations brought duplicates that were never cleaned
• Decentralized data management: Multiple departments entering data independently
• Lack of data governance: No clear rules about when to create new records vs. update existing ones
Cleaning Duplicate Records in Source Data During Pre-Migration
The best time to eliminate duplicates is before they migrate. Cleaning in Raiser's Edge reduces what you're moving and simplifies Salesforce setup.
Step 1: Identify Duplicate Candidates
Raiser's Edge has built-in duplicate checking, but it's not comprehensive. Use multiple methods:
RE Duplicate Check: Tools > Globally Change Records > Duplicate Check. Finds exact and fuzzy matches.
Query-Based Detection: Build queries to find records with the same email, phone, or address. Manual review required.
Third-Party Tools: Tools like Omatic's DupeCatcher provide more sophisticated matching than native RE.
Step 2: Prioritize Which Duplicates to Clean
You likely can't clean all duplicates before migration. Prioritize strategically:
High Priority: Major Donors - Clean all duplicates for donors with lifetime giving over your major gift threshold.
High Priority: Active Constituents - Donors who gave in past 2 years, current volunteers, board members, active prospects.
Medium Priority: Organizations - Corporate donors, foundation funders, vendors. Clean obvious organizational duplicates.
Low Priority: Inactive/Lapsed - Constituents who haven't given in 5+ years. Don't invest heavy time hunting these.
Step 3: Merge Duplicates in Raiser's Edge
Raiser's Edge merge process consolidates duplicate records. Key considerations:
Step 1: Choose Primary Record Carefully
Select the most complete, most recently updated record as primary. The primary record survives; secondary records are absorbed.
Step 2: Review Gift Attribution
Ensure gifts from both records transfer to the merged record. Check that gift history totals make sense post-merge.
Step 3: Preserve Relationships
If duplicates have different relationships, verify that they merge correctly.
Step 4: Consolidate Actions/Notes
Raiser's Edge should transfer actions and notes from secondary records, but verify that important interaction history is preserved.
Step 5: Document Merges
Keep a log of which records were merged (constituent IDs). Useful if migration reveals issues requiring investigation. Warning: Raiser's Edge merges cannot be undone. Once records are merged, they're permanently combined. Test your merge process on a backup copy first.
During Migration: Preventing Duplicate Creation
Even with clean source data, the migration process itself can create duplicates if not handled properly.
The External ID Strategy
External IDs are your primary defense against migration-created duplicates.
Setup: Before migration, create a custom field on Contact and Account objects in Salesforce called 'Raiser's Edge ID.' Mark this field as:
• External ID (allows it to be used as a matching key)
• Unique (prevents two records with the same RE ID)
• Text or Number depending on your RE constituent ID format
Usage: During migration, populate this field with the original Raiser's Edge constituent ID. When importing data, use this field as the matching key—if a record with that RE ID already exists in Salesforce, update it rather than creating a duplicate.
Benefit: Allows you to run a migration multiple times without creating duplicates. Also providea s permanent link back to the original RE record for troubleshooting.
Upsert Operations vs. Insert-Only
Understanding the difference between insert and upsert is critical:
Insert: Creates new records always. If you run the same data twice, you get duplicates. Use only for truly new data.
Upsert: Updates if a matching record exists (based on External ID), creates a new one if no match. Use for migration to prevent duplicates.
Example Scenario: Batch Migration
You're migrating 50,000 constituents in batches of 10,000. First batch imports successfully. The second batch encounters an error halfway through, so you need to re-run it.
Problem with Insert: Re-running the second batch creates duplicates for the 5,000 records that succeeded before the error.
Solution with Upsert: Re-running the second batch updates the 5,000 existing records and creates the 5,000 that failed—no duplicates.
Data Loader Configuration
If using Salesforce Data Loader for migration, configure it properly:
• Use 'Upsert' operation (not 'Insert')
• Select your External ID field (Raiser's Edge ID) as the matching key
• Map your RE constituent ID to the External ID field
• Enable 'Assignment Rule' if you want assignment rules to run
• Consider disabling 'Duplicate Rules' temporarily during migration
Post-Migration: Detecting and Resolving Duplicates
Despite best efforts, some duplicates will slip through or be created post-migration. Salesforce has powerful tools to find and fix them.
Salesforce Duplicate Management Overview
Salesforce has two complementary systems for handling duplicates:
Matching Rules: Define what constitutes a potential duplicate. 'Two Contacts are potential duplicates if they have the same email address,' or 'similar names and the same account.'
Duplicate Rules: Define what happens when a match is found. Options: Block creation, Allow with Alert, or Report.
Standard Matching Rule Examples
Field Name | Match Type |
Email (Exact) | Exact Match |
First Name + Last Name | Fuzzy Match |
Company Name | Fuzzy Match (90% similar) |
Phone | Exact Match (ignore formatting) |
Mailing Address | Exact Match |
Configuring Duplicate Detection Post-Migration
Step 1: Enable Standard Matching Rules
Navigate to Setup > Duplicate Management > Matching Rules. Activate standard rules or customize them for your needs.
Step 2: Create Custom Matching Rules if Needed
For nonprofit-specific matching (matching on alternate email, matching on household address), create custom rules.
Step 3: Configure Duplicate Rules
Navigate to Setup > Duplicate Management > Duplicate Rules. Activate rules. For the initial post-migration period, set to 'Allow' with alerts rather than 'Block.'
Step 4: Run Duplicate Reports
Create reports showing potential duplicates: Contacts > Potential Duplicates, Accounts > Potential Duplicates. Review regularly and schedule cleanup sessions.
Merging Duplicates in Salesforce
When you've identified true duplicates, Salesforce's merge function consolidates them.
Salesforce Merge Capabilities
Salesforce lets you merge up to 3 records at a time. The merge wizard shows a field-by-field comparison and lets you choose which values to keep.
What Gets Merged:
• Related records (Opportunities, Cases) transfer to the master record
• Activities (Tasks, Events) consolidate
• Notes and Attachments combine
• Campaign Members merge
• Field values you select from any record
What Gets Deleted:
• Non-master records deleted (moved to Recycle Bin for 15 days)
• Field values not selected are lost
• Record ownership of deleted records changes to the master
• Duplicate records no longer appear in searches/reports
Bulk Merge Strategies
Merging hundreds of duplicates one by one is tedious. Consider these approaches:
Native Salesforce: Good for small quantities (under 50 merges). Manual, but gives complete control.
Data Loader + VLOOKUP: For high-confidence duplicates, update duplicate records to point to the master ID, then delete. Technical but fast.
Third-party tools: Tools like DemandTools and Cloudingo provide bulk merge capabilities. Costs money but saves significant time.
Preventing Future Duplicates
Cleaning existing duplicates is only half the battle. Preventing new ones requires process changes and tool configuration.
Training Staff on Duplicate Prevention
Most duplicates are created by well-meaning staff who don't search properly. Training is essential:
• How to search effectively (partial names, alternate spellings, email search)
• When to create a new record vs. update an existing one
• Understanding duplicate warnings and how to respond
• How to report suspected duplicates for cleanup
• Importance of complete data entry to improve matching
• Special cases: family members, corporate contacts
Configuring Duplicate Rules for Prevention
After initial cleanup, tighten duplicate rules to prevent creation:
Phase 1: Allow with Alert - First 90 days post-migration. Show warnings when potential duplicates are detected, but allow creation.
Phase 2: Block for High-Confidence - After 90 days. Block creation of exact email matches. Continue alerting for fuzzy matches.
Phase 3: Expanded Blocking - After 6 months. Block fuzzy name matches with the same account. Requires override for legitimate edge cases.
Ongoing: Monitoring - Run monthly reports on duplicate sets. Identify patterns. Adjust rules and training accordingly.
Quarterly Duplicate Audits
Even with prevention in place, schedule quarterly audits:
• Run duplicate detection jobs using matching rules
• Review and merge any duplicates found
• Analyze patterns: which rules caught most duplicates?
• Adjust rules, training, or processes based on findings
• Track duplicate rate over time (goal: declining trend)
Special Cases: Complex Duplicate Scenarios
Some duplicate situations require special handling beyond standard merge processes.
Household Duplicates
Nonprofit NPC uses Household Accounts to group related individuals. Duplicate issues arise when:
• Same household created twice with slightly different names
• Individuals from the same household assigned to different household accounts
• Household and organizational relationships mixed
Resolution: Merge duplicate household accounts using Account merge. Reassign Contacts to correct the household. Verify Primary Contact settings and household naming after merge.
Organization vs. Individual Duplicates
Corporate contacts often appear as both individual Contacts and organizational Accounts.
Scenario:
'Microsoft Corporation' exists as an Account (organization). 'Microsoft' also exists as a Contact (someone entered it as an individual donor name). Gifts are split between the two records.
Resolution:
Convert the Contact to proper structure—create or locate the correct Contact (the actual human donor at Microsoft), link them to the Microsoft Account, then transfer their giving history. Delete the organizational name entered as Contact.
Measuring Success: Duplicate Metrics
How do you know if your duplicate management is working? Track these metrics:
• Duplicate Rate: Goal: < 3% of records
• Duplicates Created Per Month: Goal: Declining trend
• Duplicates Cleaned Per Month: Goal: > Duplicates created
• Duplicate Rule Override Rate: Goal: < 10% of warnings
• Time to Merge Duplicates: Goal: < 7 days from detection
Partner with Ohana Focus
Expert duplicate management services. Schedule your free consultation today.
Ohana Focus has cleaned millions of duplicate records for nonprofits during and after Salesforce migration. We combine automated detection with expert human review to ensure high-confidence merges that preserve critical data relationships. Our duplicate management services include:
• Pre-migration duplicate detection in source data
• Strategic prioritization of which duplicates to clean when
• Expert merge execution preserving gift history and relationships
• External ID field setup to prevent migration-created duplicates
• Salesforce matching and duplicate rule configuration
• Post-migration duplicate detection and cleanup
• Staff training on duplicate prevention
• Quarterly audit and cleanup services
• Ongoing monitoring and optimization
We don't just clean duplicates—we establish processes and systems that prevent them from recurring, giving you confidence in your data quality for years to come.
About Ohana Focus
Ohana Focus is a certified Salesforce consulting partner specializing in nonprofit data quality and migration. We understand that duplicates are one of the most common—and most frustrating—challenges nonprofits face during Salesforce migration. Our duplicate management methodology combines automated detection tools with expert human judgment to identify and resolve duplicates efficiently while preserving critical data relationships.
Our team has developed proven processes for handling every type of duplicate scenario: exact matches, fuzzy matches, household duplicates, organization vs. individual confusion, and complex relationship entanglements. We know when to use automated bulk merges and when human review is essential. We understand NPC's household model and how it impacts duplicate handling.
When you work with Ohana Focus on duplicate management, you get systematic detection using multiple methods to find all duplicates, strategic prioritization so effort focuses where it matters most, expert merge execution that preserves gift history and relationships, prevention strategies including matching rules and staff training, documentation of merge decisions for future reference, and confidence that your Salesforce data is clean and trustworthy.
Topics: Duplicate Records, Data Quality, Salesforce Migration, Deduplication, Matching Rules, Data Cleanup, NPC Migration
Comments