Migrating XOS Models
This document describes an XOS toolchain that can be used to generate basic migrations for service models. The autogenerated migrations can later be extended with custom logic to support more complex scenarios.
NOTE: The following assumes you have downloaded the entire source code tree in
Installing the Toolchain
The XOS migration toolchain consists of a set of python libraries. There is a helper script to install the toolchain. Just run:
cd ~/cord/orchestration/xos bash scripts/setup_venv.sh
Once the toolchain is available, you will be able to generate migrations for
the services (or the core) using the
xos-migrate command. Execute the command
to see the available options:
usage: xos-migrate [-h] -s SERVICE_NAMES [-r REPO_ROOT] [-x XOS_ROOT] [--services-dir SERVICES_ROOT] [--check] [-v] XOS Migrations optional arguments: -h, --help show this help message and exit -r REPO_ROOT, --repo REPO_ROOT Path to the CORD repo root (defaults to '../..'). Mutually exclusive with '--xos'. -x XOS_ROOT, --xos-dir XOS_ROOT Path to directory of the XOS repo. Incompatible with ' --repo'. --services-dir SERVICES_ROOT Path to directory of the XOS services root. Incompatible with '--repo'.Note that all the services repo needs to be siblings --check Check if the migrations are generated for a given service. Does not apply any change. -v, --verbose increase log verbosity required arguments: -s SERVICE_NAMES, --service SERVICE_NAMES The name of the folder containing the service in cord/orchestration/xos-services
For example, if the code you want to migrate is in
~/Sites and you
want to generate migrations for
fabric, then run the
xos-migrate -r ~/Sites/cord -s core -s fabric
NOTE: The command above is equivalent to:
xos-migrate --xos-dir ~/Sites/cord/orchestration/xos --services-dir ~/Sites/cord/orchestration/xos-services/ -s core -s fabric
If no migrations were present for your service, you will see a new
migrations (a sibling of
models) that contains the file
0001-initial.py. If the service already had migrations you will see
a new file in that folder, for example:
NOTE: All the migration files need to be committed together with the code as they will be loaded into the core together with the models.
The autogenerated migrations operate on only the database tables. They do not make any modifications to the data in those tables. You can write custom code to make such changes, as illustrated in the following example.
Assume you already have a model called
FabricService that has two
last_name, and as part of your changes
you want to add a third property called
full_name, defined as
The following is the migration code that the tool will automatically generate:
class Migration(migrations.Migration): dependencies = [ ('fabric', '0001_initial'), ] operations = [ migrations.AddField( model_name='fabricservice', name='full_name', field=models.TextField(blank=True, null=True), ), ]
To migrate the data, you need to add a custom operation in your migration.
This can be done defining a custom method
def forwards(apps, schema_editor): MyModel = apps.get_model('myapp', 'MyModel') for row in MyModel.objects.all(): row.full_name = "%s %s" % (row.first_name, row.last_name) row.save(update_fields=['full_name'])
and adding it to the
The following is a complete example of the customized migration:
def forwards(apps, schema_editor): MyModel = apps.get_model('myapp', 'MyModel') for row in MyModel.objects.all(): row.full_name = "%s %s" % (row.first_name, row.last_name) row.save(update_fields=['full_name']) class Migration(migrations.Migration): dependencies = [ ('fabric', '0001_initial'), ] operations = [ migrations.AddField( model_name='fabricservice', name='full_name', field=models.TextField(blank=True, null=True), ), migrations.RunPython(forwards), ]
For more information about migrations you can refer to the official Django guide.
Additional information about migration can be found in the following:
It is sometimes necessary to make multiple changes to the models during development. In order to continuously upgrade the service to proceed with development we suggest you generate a new migration every time the models are changed. This is required to upgrade the service multiple times during the development loop (as the core expects new migrations).
This will probably lead to multiple migration files by the time your feature is complete, for example:
- 0003-modelA-fieldA.py - 0004-modelA-fieldB.py ... - 0007-modelB-fieldX.py
To maintain clarity, however, we suggest you submit a single
migration as part of a patch. To do that, simply remove all the
migrations you have generated as part of your development and run the
xos-migrate tool again. This will generate a single migration file
for all your changes.
Validating Migration Status
Once you are done with development, you should make sure that all the
necessary migrations are generated and checked in. To do that, run the
xos-migrate tool using the
--check flag. Here is an example:
xos-migrate -s fabric --check
The XOS core can be checked in place (without the entire source tree checked
out by the
repo tool) with:
xos-migrate -x xos -s core --check