Database deployment
This page describes applications developed in the Jetveo IDE. Their database is brought up to date by the application itself, every time an instance starts: there are no schema migration files to write or keep in order. What you write by hand are data migrations, and only when existing data has to change.
The snapshot
Every build of the application carries a snapshot of its data model: the entities, codebooks with all their entries and values, attributes, references, indexes and package languages (they add translation columns to codebooks), with aspects already merged into the entities that inherit them. The database keeps the snapshot it was last deployed with.
At start, the instance compares the two and changes the database by their difference. Only the data model takes part: changes to pages, commands, expressions, menus or security never touch the database, and the start that follows them does nothing.
Entities, codebooks, attributes, references and codebook entries are matched by their id, never by name. Renaming an attribute is a change of its name under the same id, so the column is renamed and keeps its data — however many versions the instance skipped and in whatever order the changes were made in git. Deleting an attribute and creating a new one with the same name is a different thing: the old column and its data go away and an empty one is added.
Table names
Each entity and codebook has a table of the same name (Order, ProficiencyLevel), and a many-to-many reference a
join table Join_<Entity>_to_<Target>_<Reference>. The framework's own tables start with SYS_ and META_. An entity
or codebook whose name starts with SYS_, META_, sqlite_ or Join_ is reported by the IDE (JVE0104), because its
table would collide with one of those.
What happens at start
- The instance takes the deployment lock (a
.migration-lockfile next to the database). A lock left behind by a process that no longer runs is removed with a warning; a live one stops the start. - If the stored snapshot equals the application's own and every data migration has already run, the start goes on without touching the database.
- Otherwise, in one transaction: the framework's own tables are brought to the platform version first, then the schema changes, then the codebook values are written from the snapshot, then the data migrations run, then the database looks for references that point to no row.
- The new snapshot and the applied migrations are recorded and the transaction is committed.
While this runs, the application answers requests with a "migration in progress" message.
If anything fails — a schema change or a migration — the whole transaction is rolled back, the database stays exactly as it was before the start, and the application does not start: its users see a "version upgrade failed" page and the administration stops the instance. The reason is in the instance log and in the administration. Fix the cause and deploy again; nothing has to be cleaned up first.
A new instance gets its database created directly from the snapshot, with the codebook values filled in. Data migrations do not run there (see New instances).
Deploying an older version of the application works the same way, in the other direction. Data migrations are not undone.
What the database does for each change
The database is SQLite, whose ALTER TABLE can rename, add and drop, but cannot change an existing column. The
deployment picks the cheapest way for each change:
| Change in the model | What happens in the database |
|---|---|
| Rename an entity, codebook, attribute or reference | The table or column is renamed. No data is copied. |
| Add an entity or codebook | A new table. |
| Add an attribute or a reference | A new column. Existing rows get an empty value: null for a nullable one, otherwise the type's empty value (0, empty text, false, the minimum date); a codebook reference with a default entry gets that entry. |
| Delete an entity, codebook, attribute or reference | The table or column is dropped, with its data. A many-to-many reference drops its join table. |
| Change an attribute's type (see Type conversions) or nullability, a reference's target, or a codebook reference's default entry | Column rebuild: a new column is added, the values are copied over and converted, the old column is dropped and the new one takes its name. Indexes on the column are recreated. |
| Change of the primary key (does not happen through normal model edits) | Table rebuild: the table is copied into a new one. The instance log lists every rebuilt table with its row count. |
| Add or remove an index | The index is created or dropped; the data stays untouched. |
Changing an attribute between Document or User and any other type is not a conversion: it deletes the old column and adds a new, empty one.
Type conversions
| From → to | Result |
|---|---|
| Integer → Decimal | Converted exactly, within the Decimal range. |
| Integer or Decimal → String | The number as text; a Decimal always gets a decimal point (12.0). |
| Decimal → Integer | The decimal places are cut off. |
| String → Integer or Decimal | The number the text starts with; text that does not start with a number becomes 0. |
| String ↔ String (plain, Markdown, HTML), Date ↔ Date and Time | No conversion needed; the values stay as they are. |
| Boolean → Integer or Decimal | true becomes 1, false 0. |
| Boolean → String | True or False. |
| Date and Time → String | The stored text, yyyy-MM-dd HH:mm:ss. |
| Any other change to or from Boolean or Date and Time (other than Date ↔ Date and Time) | Refused: the deployment fails and the application does not start. |
To change a type the deployment refuses, add a new attribute, fill it from the old one in a data migration, and delete the old attribute in a later release.
Destructive changes
A change is destructive when it can lose data. Most of them are destructive only when there is data to lose; the
deployment checks the database before it starts and lists every one that applies in the instance log as a
Destructive: warning.
| Change | Destructive when |
|---|---|
| Delete an entity or codebook | its table has any row |
| Delete an attribute or reference | the column holds any value |
| Change an attribute's type, except to a wider one (Integer → Decimal, a number → String) | the column holds any value |
| Make a nullable attribute or reference required | the column holds any value |
| Point a reference at another entity | the column holds any value |
| Add a required reference (Direct, or Codebook without a default entry) | the table has any row |
| Remove a codebook entry | some row refers to it |
Renames, new entities and attributes, new optional references, making a required attribute optional and adding or removing indexes are never destructive.
Deleting an entity that has no rows, or an attribute that was never filled in, therefore passes without a warning.
Some of these changes leave rows pointing nowhere unless something takes care of the data. The deployment does not fail because of them: it reports every such reference as an error in the instance log, with the number of rows per column — in the deployment that removes a codebook entry also the entry's code — and goes on. Fix them in the release itself:
- A new required reference in a table with rows: every existing row starts with an empty reference. Fill it in a data migration in the same release — see Filling a new required reference.
- An optional reference made required while some rows have it empty: those rows get an empty reference. Fill them in a data migration in the same release, the same way.
- A removed codebook entry that rows still refer to: move those rows to another entry first, in a data migration of an earlier release.
References that were already broken before the deployment are reported the same way at every deployment until the data is repaired.
Destructive changes run on every instance and are reported in the log; nobody approves them. Making sure a release does not lose data that is still needed is the application developer's responsibility.
Codebook values
Codebook values are part of the snapshot. Whenever the model changes, all codebook rows are written again from it, so a codebook value is changed in the model, not by a data migration — a migration's change to a codebook row is overwritten by the next deployment.
- Date and Time values are wall-clock times without a time zone; see Codebooks.
- Decimal values are written with a decimal point (
1980.5), no thousands separators and no exponent. The database keeps 5 decimal places (see Database technical details), so do not use more. A value that is not a valid number is stored as 0.