migman 1.0.3

Migman

Migman is a CLI tool for migrating and verifying MariaDB/MySQL tables using dump/load workflows and checksums.

Install (global tool)

dotnet tool install --global migman

Update

dotnet tool update --global migman

Usage

migman --help
migman run --dumper
migman run --loader
migman run --checksum
migman run --sync
migman init --docker
migman init --sample
migman init --docker --sample --force
migman run --checksum --loglevel info

Init options

• --docker: create docker-compose.yml for MariaDB/MySQL.
• --sample: create sample SQL scripts and connection helpers.

Configuration

migman.yml example:

source:
  type: mariadb
  host: localhost
  port: 3307
  user: root
  password: secret
  database: source_db
target:
  type: mysql
  host: localhost
  port: 3308
  user: root
  password: secret
  database: target_db
migration_mode: dump
dump_path: ./dumps
log_path: ./migman.log
log_level: info
command_timeout: 300
threads: 4
tables:
  - name: sample_pk_split
    target_name: sample_pk_split_copy
    compress: false
    drop_table: true
    include_indexes: true
    chunking:
      type: pk_split
      key: id
      size: 20
  - name: sample_pk_offset
    target_name: sample_pk_offset_copy
    compress: false
    drop_table: true
    include_indexes: true
    chunking:
      type: pk_offset
      keys:
        - order_id
        - line_id
      order_by:
        - column: order_id
          direction: asc
        - column: line_id
          direction: asc
      size: 20
  - name: sample_index_range
    target_name: sample_index_range_copy
    compress: false
    drop_table: true
    include_indexes: true
    chunking:
      type: index_range
      key: id
      index_name: PRIMARY
      size: 20
  - name: sample_index_split_datetime
    target_name: sample_index_split_datetime_copy
    compress: false
    drop_table: true
    include_indexes: true
    chunking:
      type: index_split_datetime
      key: created_at
      index_name: idx_created_at
      period_type: day
  - name: sample_cursor_index
    target_name: sample_cursor_index_copy
    compress: false
    drop_table: true
    include_indexes: true
    chunking:
      type: cursor_index
      index_name: idx_sku
      tiebreaker:
        - id
      size: 20

migration_mode values: dump or sync (sync compares checksums and inserts/updates target rows and deletes target rows missing in source when mismatched). Use migman run --sync when migration_mode: sync. If the target table is missing, sync creates it from the source schema. tables[].target_name overrides the target table name (defaults to tables[].name). drop_table: true is only applied in loader mode; sync mode rejects it. include_indexes: true keeps non-primary indexes in the dump SQL CREATE TABLE (default: true). command_timeout sets the MySQL command timeout in seconds (default: 300). index_name: PRIMARY forces use of the primary key index; cursor_index defaults to PRIMARY KEY columns as the tiebreaker when tiebreaker is omitted.

Configuration options

Top-level options:

• source: source database connection settings.
• target: target database connection settings.
• migration_mode: dump or sync. In sync, use migman run --sync.
• dump_path: directory for dump files.
• log_path: log file path (optional).
• log_level: error, warning, info, debug, trace (default: info).
• --loglevel: CLI override for log_level on migman run (same values).
• command_timeout: MySQL command timeout in seconds (default: 300).
• threads: number of worker threads (default: 4).
• tables: list of table configs.

source / target fields:

• type: mariadb or mysql.
• host: hostname or IP.
• port: integer port.
• user: username.
• password: password.
• database: database name.

tables[] fields:

• name: source table name (required).
• target_name: target table override (optional).
• compress: true to write .tsv.gz, false for .tsv.
• drop_table: true to drop target before load; ignored in sync (error if set).
• include_indexes: true to keep non-primary indexes in dump SQL CREATE TABLE (default: true).
• chunking: chunking config (required).

chunking for pk_split:

• type: pk_split (required).
• key: numeric PK column (required).
• size: rows per chunk (required).
• where: optional filter applied to range/chunk queries.
• order_by: optional override for sort direction (default: asc).
• sleep_ms: optional sleep between chunks in dump mode (milliseconds).

chunking for pk_offset:

• type: pk_offset (required).
• keys: list of PK columns (required; at least 1).
• size: rows per chunk (required).
• where: optional filter applied to chunk queries.
• order_by: optional list of key columns with direction (asc or desc).
  • Must include the same columns as keys in the same order.
  • Mixed directions are supported (e.g., sid desc, dt_create desc).
• sleep_ms: optional sleep between chunks in dump mode (milliseconds).

chunking for index_range:

• type: index_range (required).
• key: numeric column for range bounds (required).
• index_name: index name to force for range scans (required).
• size: rows per chunk (required).
• where: optional filter applied to range/chunk queries.
• order_by: optional override for sort direction (default: asc).
• sleep_ms: optional sleep between chunks in dump mode (milliseconds).

chunking for index_split_datetime:

• type: index_split_datetime (required).
• key: datetime column for range bounds (required).
• index_name: index name to force for range scans (required).
• period_type: day or hour (required).
• where: optional filter applied to range/chunk queries.
• order_by: optional override for sort direction (default: asc).
• sleep_ms: optional sleep between chunks in dump mode (milliseconds).

chunking for cursor_index:

• type: cursor_index (required).
• index_name: secondary index name to drive pagination (required).
• index_columns: optional list of index columns (defaults to index definition order).
• tiebreaker: optional list of columns to guarantee stable ordering (defaults to PRIMARY KEY).
• size: rows per chunk (required).
• where: optional filter applied to chunk queries.
• order_by: optional list of index columns with direction (asc or desc).
  • Must include the same columns as index_columns in the same order.
  • Tiebreaker columns always sort asc.
• sleep_ms: optional sleep between chunks in dump mode (milliseconds).

Chunking

Migman supports multiple chunking strategies for splitting large tables.

pk_split

Splits a table by a single numeric primary key range.

Required fields:

• chunking.type: pk_split
• chunking.key: primary key column
• chunking.size: rows per chunk Optional fields:
• chunking.where: filter condition applied to min/max range and chunk queries
• chunking.sleep_ms: sleep between chunks in dump mode (milliseconds)

Example:

chunking:
  type: pk_split
  key: id
  where: status = 'active'
  size: 20000

pk_offset

Splits a table by an ordered composite key (or single key) using keyset pagination.

Required fields:

• chunking.type: pk_offset
• chunking.keys: ordered list of key columns
• chunking.size: rows per chunk

Optional fields:

• chunking.order_by: must match chunking.keys order
• chunking.sleep_ms: sleep between chunks in dump mode (milliseconds)

Example:

chunking:
  type: pk_offset
  keys:
    - order_id
    - line_id
  order_by:
    - column: order_id
      direction: asc
    - column: line_id
      direction: asc
  size: 20000

index_range

Splits a table by numeric key ranges while forcing a specific index. index_name: PRIMARY forces use of the primary key index.

Required fields:

• chunking.type: index_range
• chunking.key: numeric range column
• chunking.index_name: index name to force
• chunking.size: rows per chunk Optional fields:
• chunking.where: filter condition applied to min/max range and chunk queries
• chunking.order_by: optional override for sort direction (default: asc)
• chunking.sleep_ms: sleep between chunks in dump mode (milliseconds)

Example:

chunking:
  type: index_range
  key: id
  index_name: PRIMARY
  size: 20000

index_split_datetime

Splits a table by time windows over a datetime column while forcing a specific index. period_type: day uses 24-hour windows, period_type: hour uses 1-hour windows.

Required fields:

• chunking.type: index_split_datetime
• chunking.key: datetime column
• chunking.index_name: index name to force
• chunking.period_type: day or hour Optional fields:
• chunking.where: filter condition applied to range and chunk queries
• chunking.order_by: optional override for sort direction (default: asc)
• chunking.sleep_ms: sleep between chunks in dump mode (milliseconds)

Example:

chunking:
  type: index_split_datetime
  key: created_at
  index_name: idx_created_at
  period_type: day

cursor_index

Splits a table by keyset pagination over a secondary index. When tiebreaker is omitted, the PRIMARY KEY columns are used as the default tie-breaker.

Required fields:

• chunking.type: cursor_index
• chunking.index_name: secondary index name
• chunking.size: rows per chunk Optional fields:
• chunking.index_columns: override index column order
• chunking.tiebreaker: additional columns to guarantee stable ordering
• chunking.where: filter condition applied to chunk queries
• chunking.order_by: must match chunking.index_columns order (tiebreaker columns always asc)
• chunking.sleep_ms: sleep between chunks in dump mode (milliseconds)

Example:

chunking:
  type: cursor_index
  index_name: idx_sku
  tiebreaker:
    - id
  size: 20000

Logging

log_level supports: error, warning, info, debug, trace. command_timeout sets the MySQL command timeout in seconds (default: 300).