Abhishek.DataMigrationTool 1.2026.10.1

DataMigrationToolbyXTB

An XrmToolBox tool that migrates records from one Dataverse environment to another while preserving the record GUID. A record that does not exist on the target is created with the same Id; a record whose Id already exists on the target is updated.

This is ideal for moving configuration / reference data (and any GUID-sensitive records) between DEV → TEST → PROD without breaking the lookups and references that depend on those Ids.

The tool's display name in XrmToolBox is DataMigrationToolbyXTB; the compiled assembly is DataMigrationToolbyXTB.dll.

Features

• Two-environment workflow – select a Source and a Target connection independently, using the standard XrmToolBox connection picker.
• Entity picker from the Default solution – loads the entities contained in the source environment's Default solution into a dropdown.
• FetchXML-driven selection – you supply the FetchXML that decides exactly which records are migrated. A starter template is generated automatically when you pick an entity.
• GUID preserved – records keep their primary key across environments (create-with-Id / update-by-Id).
• Schema-aware writes – read-only columns (calculated/rollup, audit, and other non-writable system fields) and link-entity (AliasedValue) columns are skipped automatically so writes don't fail.
• Optional ownership – migrate ownerid (off by default).
• Optional status – migrate statecode / statuscode using SetStateRequest (a proper, supported state transition), applied after the record is written (off by default).
• Compare / preview before migrating – a Compare button reads the records selected by the FetchXML and opens a dashboard that shows, per record, a New / Update / Identical status and a field-by-field (line-by-line) source-vs-target diff (the target column is blank for records that don't exist on the target yet). Tick the records you want and click Migrate selected to migrate only those. The Migrate Data button still migrates everything directly, with no comparison needed.
• Related-record migration (N:N, N:1, 1:N) – a relationship picker lets you also migrate related records. N:N replicates many-to-many links via Associate. N:1 migrates the referenced parent records first (so the record's lookups, e.g. marketing area, resolve). 1:N migrates the child records after. You pick which relationships to include per entity (custom tables only).
• N:N name/email matching (cross-environment users/teams) – optionally, when an N:N related record's GUID is missing on the target, the tool matches it to a target record by a unique field: the user's email (internalemailaddress) for systemuser, otherwise the entity's primary name (e.g. an Approver user/team whose Id differs across environments). Records that already exist on the target by GUID keep using that GUID; ambiguous or unmatched values are reported, never guessed.
• Bulk performance – records are written in ExecuteMultiple batches (default 100, configurable 10–1000) with ReturnResponses=false and bulk existence checks.
• Parallel writes – with Continue on error on, batches run concurrently on cloned target connections (auto = server-recommended degree of parallelism, capped at 4; manual up to 8). Throttling (service-protection 429s) is retried honouring Retry-After. Continue on error off always runs sequentially.
• Skip unchanged (fast re-runs) – the target snapshot is read once; identical fields are not re-sent, fully identical records are not written at all (reported as Unchanged), and status is only set when it differs.
• Metadata cache – entity metadata (writable columns, option sets, default status) is retrieved once per entity per run.
• Batched N:N – existing target links are pre-read and only missing links are sent, as batched Associate requests.
• Self-referencing lookups – a lookup to a record in the same set that isn't created yet is deferred and set in a second pass (no more cleared parent lookups when order is unlucky).
• Suppress flow triggers (optional) – writes can skip queuing Dataverse-triggered Power Automate flows.
• Robust paging – source queries are paged automatically (handles > 5000 records).
• Resilient writes (transient-fault retry) – throttling / timeout / transport failures are retried automatically with exponential backoff + jitter (honouring Retry-After); a create whose record has already committed is treated as success, so a temporary throttle never fails a whole batch. (PR #17145)
• Throughput & ETA – progress shows live records/second and an estimated time remaining, and each entity summary reports its throughput and any transient retries performed. (PR #17145)
• Migration profiles (save / load) – save the current entity, FetchXML, options and checked relationships to a portable JSON profile and reload it later for a repeatable migration (connection names only — never secrets). (PR #17145)
• Progress, cancel, and a full log – created / updated / failed counts plus per-record errors.

Requirements

• XrmToolBox (current release – runs on .NET Framework 4.8).
• Permissions on both environments: read on the source, create/update (and, for status migration, the relevant state-transition privileges) on the target.

Build

Open MigrateData.sln in Visual Studio 2022 and build, or from a developer prompt:

& "C:\Program Files\Microsoft Visual Studio\2022\Professional\MSBuild\Current\Bin\MSBuild.exe" `
    MigrateData.sln /t:Build /p:Configuration=Debug /p:Platform="Any CPU"

A Debug build copies DataMigrationToolbyXTB.dll (+ .pdb) into DataTransPorter\bin\Debug\Plugins\ via the post-build step.

The XrmToolBox host assemblies (XrmToolBox.Extensibility, XrmToolBox.ToolLibrary, McTools.Xrm.Connection, McTools.Xrm.Connection.WinForms) are referenced with Private=False – they are provided by the host at runtime and are not copied to the output, which avoids version conflicts.

Install into XrmToolBox (manual / local)

Copy DataMigrationToolbyXTB.dll into your XrmToolBox Plugins folder, then restart XrmToolBox:

%APPDATA%\MscrmTools\XrmToolBox\Plugins

The tool appears in the tool list as DataMigrationToolbyXTB.

Publish to the XrmToolBox Tool Library (public)

XrmToolBox's in-app Tool Library (Configuration ▸ Tool Library) is fed from NuGet.org. A package is auto-discovered and installable for everyone when it:

1. is tagged XrmToolBox,
2. ships the tool assembly under lib\<tfm>\Plugins\ (third-party dependency DLLs, if any, go in a sub-folder named exactly like the tool DLL), and
3. declares a dependency on the XrmToolBox package (the host-version marker).

This repo already contains everything for that: DataMigrationToolbyXTB.nuspec and pack-and-publish.ps1.

One-time setup

1. Create a free NuGet.org account and an API key: https://www.nuget.org/account/apikeys (scope it to Push new packages and package versions).
2. Edit DataMigrationToolbyXTB.nuspec and set:
   • <id> – must be globally unique on NuGet.org (currently DataMigrationToolbyXTB). It must not collide with another package, and the tool DLL name (DataMigrationToolbyXTB.dll) must not collide with another installed tool.
   • <authors>, <projectUrl>, <iconUrl>, <license> – your details. (<iconUrl> must point at a publicly reachable copy of images\icon.png, e.g. the raw URL in your GitHub repo.)
   • The icon (images\icon.png) and tool tile images are already wired in.

Build, pack and publish

# build + pack only (inspect artifacts\*.nupkg first)
.\pack-and-publish.ps1

# build + pack + publish to NuGet.org
.\pack-and-publish.ps1 -Push -ApiKey oy2xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

After NuGet finishes indexing (a few minutes) the tool shows up in the Tool Library; users click Install. There is no separate approval step, but the package must pass NuGet's validation and follow the three rules above.

Publishing requirements checklist

• ✅ Tagged XrmToolBox (first tag).
• ✅ Tool DLL in lib\net48\Plugins\ (the pack script + nuspec do this).
• ✅ Dependency on XrmToolBox (host-version marker).
• ✅ Assembly is not strong-named (this project isn't).
• ✅ Unique NuGet id and unique tool DLL name.
• ✅ Host assemblies (XrmToolBox.Extensibility, McTools.Xrm.Connection, Xrm SDK) are not bundled — they're referenced with Private=False and provided by the host at runtime.

Releasing updates

NuGet package versions are immutable — you cannot overwrite a published version. To update, bump <version> in the nuspec and AssemblyVersion/AssemblyFileVersion in Properties\AssemblyInfo.cs (keep them equal; date-based 1.YYYY.M.rev is the XrmToolBox convention), re-run pack-and-publish.ps1 -Push -ApiKey …, and XrmToolBox offers the update automatically.

XrmToolBox Tool Library validation checklist

How this package/tool satisfies the official Tool Library checklist:

The images are stored as source PNGs under images\ (icon.png, tool-large.png, tool-small.png); the 80×80 and 32×32 are base64-embedded in MyPlugin.cs. Regenerate by replacing the PNGs and updating the two …ImageBase64 strings.

Usage

1. 1. Select Source – choose the environment to read records from. (The connection you are already on when you open the tool is used as the source by default.)
2. 2. Select Target – choose the environment to write records to.
3. 3. Load Entities – populates the dropdown with entities from the source Default solution.
4. Pick an entity. A default FetchXML template is placed in the editor, and the entity's relationships load into the Relationships to also migrate list (N:N, N:1, 1:N).
5. 4. FetchXML – edit the FetchXML to select exactly the records you want to migrate. Use Count source records to preview how many records match.
6. (Optional) In the Relationships to also migrate list, check any relationships to also bring across: [N:N] replicates links, [N:1] migrates the referenced parent records first, [1:N] migrates the child records after.
7. 5. Migrate Data – confirm the prompt. Referenced parents are migrated first, then the selected records (same GUID), then child records, then N:N links. The log shows a per-entity summary.

Compare / preview (optional)

Instead of migrating straight away you can preview exactly what will change:

1. Select Source and Target, Load Entities, pick an entity and set the FetchXML (steps 1–4 above).
2. Click 4. Compare. The tool reads the records matched by the FetchXML and the matching target records (by GUID) and opens the comparison dashboard.
3. The top grid lists one row per record with a Migrate tick-box and a status:
   • New – the record does not exist on the target (the target side is blank).
   • Update – the record exists and one or more migratable fields differ.
   • Identical – the record exists and every migratable field already matches.
4. Select a row to see the field-by-field comparison in the bottom grid: Field, Source value, Target value, and a per-field New / Changed / Same status (changed/new rows are highlighted).
5. Tick the records to migrate (New and Update are ticked by default; Select all / none / new & changed help-buttons are provided), then click Migrate selected to migrate only those records. The same migration pipeline runs (GUID preserved, options and relationships honoured) against the chosen subset.

Copy GUID

• Copy GUID button (top bar) copies the GUID of the highlighted record.
• Right-click the records grid: Copy GUID, Copy name + GUID, Copy GUIDs of ticked records, Copy GUIDs of all listed records (one per line — handy for a FetchXML in filter).
• Right-click a node in the related-records tree: Copy GUID / Copy name + GUID.
• Right-click the field grid: Copy value / Copy GUID from value (e.g. the GUID of a lookup).

Extra records on the target (clean-up)

When Compare runs, the same FetchXML is also executed on the target. Any target record it returns that the source FetchXML did not return is an extra record. The Extra on target (N)... button (red when N > 0) opens a list of them:

• Select a row to see all its target field values in the details grid; search and Hide records already inactive help filter; Copy GUID / right-click copy and Export CSV are available.
• Tick records (Select all not in source helps) and click Deactivate selected (N). After a confirmation (default No), the tool sets them to the table's Inactive state via SetStateRequest in batches. Records are not deleted. The Bypass custom plug-ins / Suppress flows options are honoured and every deactivated or failed GUID is written to the log.
• Deactivation is offered only for tables with a plain Inactive state; tables such as incident (Resolved/Cancelled) show the reason instead. Extras are not checked for aggregate FetchXML, FetchXML with top, or when the FetchXML entity differs from the selected entity (a note explains why).

The comparison reflects the current option check-boxes: lookups are compared by referenced GUID; the Migrate owner option adds ownerid; the Migrate status option adds statecode/statuscode rows. Audit/system and non-writable columns are excluded (they are never migrated).

Options

Advanced menu

In the compare dashboard, Extra on target now also offers Delete (typed confirmation). The ticked records are re-read right before the action, and each full record is journaled first so Run history → Roll back can recreate it (cascade-deleted children are not journaled).

Command line (CI / scheduled runs)

DataTransPorter.Cli builds dmt.exe, which runs a job (or a single profile) without XrmToolBox:

# Connection strings come ONLY from environment variables (never pass secrets as arguments).
$env:DMT_SOURCE = "AuthType=Certificate;Url=https://dev.crm.dynamics.com;ClientId=...;Thumbprint=..."
$env:DMT_TEST   = "AuthType=Certificate;Url=https://test.crm.dynamics.com;ClientId=...;Thumbprint=..."
dmt --job .\seed.job.json --source-env DMT_SOURCE --target-env DMT_TEST [--target-env DMT_UAT ...]
dmt --job .\seed.job.json --check   # validate the file and that the tool loads; no connection

Exit codes: 0 success, 1 failures (see the log, journal and audit trail), 2 usage error. Prefer certificate or managed-identity auth; ConfirmBetweenTargets is ignored headless.

Build (after building MigrateData.sln in the same configuration): dotnet build DataTransPorter.Cli -c Release → DataTransPorter.Cli\bin\Release\net48\dmt.exe. The project's nuget.config restores through the Microsoft package feed proxy.

Tests

dotnet test DataTransPorter.Tests -c Release (after building the tool) runs unit tests for CSV escaping, dependency ordering, field rules / masking, the value codec, journal hashing, data packages and job loading, plus UI smoke tests that open and close every code-built dialog.

FetchXML examples

Migrate every record of the chosen entity:

<fetch>
  <entity name="mash_service">
    <all-attributes />
  </entity>
</fetch>

Migrate a filtered subset:

<fetch>
  <entity name="mash_service">
    <all-attributes />
    <filter type="and">
      <condition attribute="statecode" operator="eq" value="0" />
      <condition attribute="mash_category" operator="eq" value="100000001" />
    </filter>
  </entity>
</fetch>

Use <all-attributes /> (rather than naming columns) so every migratable field is carried over. Columns that cannot be written (calculated, rollup, audit, etc.) are filtered out automatically against the target schema.

How GUID preservation works

For each source record the tool:

1. Builds a target Entity with Id = source.Id and the migratable attributes.
2. Checks existence in bulk (IN query by id, or the skip-unchanged snapshot read).
   • Not found → Create(entity) – Dataverse honours the supplied Id, so the record is created with the same GUID. If a create is rejected as a duplicate, the exact id is re-checked: if it now exists (a create committed by a transport retry) it is re-sent as an update; otherwise it is a real alternate-key/duplicate collision and is reported as failed.
   • Found → Update(entity) (only fields valid for update, and only changed fields when Skip unchanged is on).

Notes & limitations

• Reference order – references (lookups) resolve by GUID. Lookups to records in the same migrated set are deferred and set after all records exist. Lookups to other tables must already be on the target: migrate parent/lookup entities first (or tick the [N:1] relationship).
• Null values – a blank source value does not clear an existing target value (patch semantics).
• Parallel writes & throttling – parallel batches increase load on the target; if you see many "transient retries" in the summary, lower Parallel batches or Batch size. Test on a non-production target first.
• Related records (N:1 / 1:N) – checking a [N:1] relationship migrates the referenced parent records before the main set (so the lookup resolves); checking [1:N] migrates the child records after. Only custom related tables are listed (system tables like team/user/currency are excluded). One level deep — it does not recurse into the related records' own relationships.
• N:N associations – only links where the migrated record is one side are replicated, and the other side must already exist on the target (otherwise that link is reported as failed). Links that already exist on the target are detected and counted as "already linked" (idempotent). Self-referential N:N is handled best-effort using the Referencing role.
• Status – statecode/statuscode are never written as part of the create/update; they are applied separately via SetStateRequest. A status reason of -1 lets the platform pick the default status for the chosen state.
• Owner / business unit – owning business unit/team/user are system-managed and are never copied; only ownerid is migrated, and only when the option is enabled.
• Counting loads the matching records to count them; for very large tables prefer a tight FetchXML filter.

Project layout