Data migrations

The schema of an application developed in the Jetveo IDE follows its model by itself (see Database deployment). A data migration is for the other case: data that already exists has to change. It is a piece of C# that runs once on every instance, in the same transaction as the schema change of the deployment that brings it.

Typical cases:

Case	Example
A business change of data	Orders created before 2020 become Cancelled.
Moving rows to another codebook entry	Clients in state CANCELLED move to ARCHIVED, before CANCELLED is removed.
Filling or recomputing an attribute	FullName = Name + " " + Surname for every existing person.

A renamed entity or attribute needs no migration: it is renamed in the database with its data.

The file

A migration is a .jvmigration file anywhere in the project, for example src/migrations/Fill full names.jvmigration.

id: 3f2a9c1e-7b4d-4e1a-9c55-0d6b2f8e4a10
name: Fill full names
description: FullName is new in this release; compute it for the existing people.
code: |-
  (db, ctx) =>
  {
    foreach (var person in db.PersonSet.ToList())
      person.FullName = person.Name + " " + person.Surname;
  }

Key
id	Required. A new random GUID, written once when the file is created and never changed. It is the migration's identity: renaming the file or editing the code does not make it a new migration.
name	Optional. Shown in the instance log.
description	Optional. What the migration does and why.
after	Optional. Ids of migrations that must run before this one.
archived	Optional, false by default. See Archiving.
code	Required unless archived. The body: a (db, ctx) => { … } lambda.

The Language Server reports a duplicate id (JVE0102, archived migrations included), an after id that does not exist (JVE0200), a cycle in after (JVE0380) and an active migration without code (JVE0381). The body is compiled and checked like any other expression.

The body

The body is a (db, ctx) => { … } lambda. db is the application's database, with the same entity sets and codebooks as in a command. Changes are saved when the lambda returns; there is no need to call SaveChanges.

id: 9a9a9a9a-0000-0000-0000-000000000002
name: Clear negative totals
code: |-
  (db, ctx) =>
  {
    foreach (var invoice in db.InvoiceSet.Where(i => i.Total < 0))
      invoice.Total = 0;
  }

When and in what order migrations run

• Exactly once per instance. An instance records every migration it ran and never runs it again — not when the file is renamed, not when its code changes.
• After the schema change. At start, the deployment changes the schema, writes the codebook values and then runs the migrations the instance has not run yet, all in one transaction. A migration therefore sees the new model, new codebook entries included. Migrations run even when the model did not change.
• In order. A migration runs after every migration listed in its after. Migrations that do not depend on each other run in the order of their ids — do not rely on that; use after when the order matters.

id: 9a9a9a9a-0000-0000-0000-000000000003
name: Recompute order totals
after:
  - 9a9a9a9a-0000-0000-0000-000000000002
code: |-
  (db, ctx) => { … }

New instances

A new instance creates its database from the current model and runs no migrations: there is no old data to transform, and data a migration creates would end up where it does not belong. All migrations the application carries at that point are recorded as applied. Initial data for a new instance does not belong in a migration.

Migration mode

A migration does not run inside a user's request, and some of what a command does is switched off:

• No business events. Neither queued nor inline events fire, so no commands, notifications or e-mails they would send.
• No audit. The changes do not appear in the entity audit; the record of the migration in the instance is the only trace.
• No user. ctx has no signed-in user.

Documents can be read and written as usual.

A migration must not open its own connection or transaction: it runs in the deployment's.

Compiled against the new model

A C# migration is compiled with the version of the application that carries it, so it sees the model after the change. An attribute this release deletes is not there to read from — and its column is already gone when the migration runs.

Hence the rule for moving data out of something you are deleting: two releases.

1. Release 1 adds the new attribute (or codebook entry) and a migration that moves the data into it. The old one stays.
2. Release 2 deletes the old attribute (or entry).

For example, to move clients from CANCELLED to ARCHIVED: release 1 adds the ARCHIVED entry and a migration that moves every CANCELLED client to it; release 2 removes CANCELLED.

Deploy release 2 only to instances that already run release 1. In release 2 the migration refers to something that no longer exists, stops compiling and has to be archived. An instance that jumps from before release 1 straight to release 2 does not run it: it loses the old attribute's data (a Destructive: warning in the log), or, for a codebook entry its rows still refer to, keeps rows pointing nowhere (an error in the log).

Filling a new required reference

Adding a required reference (Direct, or Codebook without a default entry) to an entity with rows leaves every existing row pointing nowhere; the deployment reports those rows as an error in the instance log. Add the reference and a migration that fills it in the same release:

id: 0b7c4e2a-58d1-4f3b-b6a9-2e1d0c9f8a64
name: Assign existing projects to the internal client
code: |-
  (db, ctx) =>
  {
    var client = db.ClientSet.Single(c => c.Name == "Internal");
    foreach (var project in db.ProjectSet.ToList())
      project.Client = client;
  }

Without the migration — or if it misses a row — the deployment is rolled back with the table, the reference and the number of rows in the log. An instance whose table is empty needs nothing.

Changing a migration

After it ran somewhere, editing a migration changes nothing on the instances that already ran it; the next deployment there only logs the warning "Data migration … was changed after it was applied". If a migration ran successfully but did the wrong thing, repair the data with a new migration, under a new id.

When it fails, the whole deployment is rolled back: the schema change, the codebook values and every migration of that start. The database stays as it was, the failure is recorded for the instance, and the application does not start. Fix the code under the same id and deploy again. The fixed migration runs on the instances where it failed and is skipped on those where an earlier version already succeeded.

Archiving

Migrations are not deleted. Once every instance has run a migration, set archived: true:

id: 9a9a9a9a-0000-0000-0000-000000000001
name: Drop legacy notes
archived: true
code: |-
  …

An archived migration is no longer compiled or validated, never runs, and its code can stay as it was or be emptied. Its id stays reserved: no other migration may use it.

An instance that never ran an archived migration — for example one that skipped several releases — does not run it either. The deployment records it as skipped and logs the warning "Skipped data migration …: archived before this instance applied it". The data change of that migration is lost for that instance, so archive a migration only once you know every instance has it.

A migration listed in after that is archived counts as done.

Release labels and checkpoint releases, which will archive migrations automatically, are coming with the jv command line.