ExampleService Tutorial

Summary

This section shows the steps to Add a Service, called ExampleService, to XOS.

We have also used ExampleService in a tutorial video and an accompanying slide deck, which are available online at:

Note that there are some differences between this section and the video/slides, most notably, this section uses the “devel” configuration, while the video uses the “cord-pod” configuration. Either will work, the main difference being that the former includes only ExampleService, while latter includes other CORD services in addition to ExampleService.

ExampleService is multi-tenant. It instantiates a VM instance on behalf of each tenant, and runs an Apache web server in that VM. This web server is then configured to serve a tenant-specified message (a string), where the tenant is able to set this message using the XOS administrative interface.

ExampleService has two major code components:

For this particular example, there are two unique additions to the base Service model:

When a user creates a new Slice and Tenant in the Admin website (constructed using Django), the data describing the service is stored in a database.

The Synchronizer runs on a recurring basis, obtains the service_message, tenant_message, and additional needed information from the database, and uses it to run an Ansible playbook that applies the configuration to the Instance.

Prepare Your Development Environment

In order to get started, you need a OpenStack host to deploy XOS on.

This can be CloudLab or a local DevStack installation, or one of the other Configurations that XOS supports.

Once you have that set up, do the following:

Development Loop

Once you’ve prepared your development environment as described above, the change/build/test development loop for service development on XOS is as follows:

  1. Make changes to your copy of XOS and propagate them to your OpenStack host.

  2. On your OpenStack host in XOS’s xos/configurations/devel directory, run: make rm, which will stop XOS and delete the Docker containers.

  3. Run make containers. This will rebuild all the Docker containers to include your changes. Initially this will take a long time as everything is rebuilt, but will take less time during subsequent runs as Docker saves intermediate state.

  4. Run: make in the same directory to start XOS running with the newly built containers.

  5. Test and verify your changes.

  6. Once you’re done testing, go back to step #1.

The process to do the above can be done with this command:

git pull && make rm && make containers && make

assuming you pull your changes from a development git repo.

Create the Django Components of a Service

XOS services are located in the xos/services directory in the XOS source tree. Create a new directory for your service.

In this example, the service will be named exampleservice and the example code is in the XOS repo at xos/services/exampleservice.

In your service directory, create an empty __init.py__ file. This is required for Python to recognize your service directory as a package, in order to avoid namespace conflicts.

Create a Django Model

Create a file named models.py in your service directory. The start of this file:

from core.models import Service, TenantWithContainer
from django.db import models, transaction

brings in the two XOS Core classes that will be extended by our Service, defines the model, and lets us atomically update the database on instance creation/removal.

The following uniquely identify and provide human readable names for the service in the admin web UI. For your own service, change these.

SERVICE_NAME = 'exampleservice'
SERVICE_NAME_VERBOSE = 'Example Service'
SERVICE_NAME_VERBOSE_PLURAL = 'Example Services'
TENANT_NAME_VERBOSE = 'Example Tenant'
TENANT_NAME_VERBOSE_PLURAL = 'Example Tenants'

Extending Service

We extend XOS Core’s Service class as follows:

class ExampleService(Service):

    KIND = SERVICE_NAME

    class Meta:
        app_label = SERVICE_NAME
        verbose_name = SERVICE_NAME_VERBOSE

XOS uses the KIND variable to uniquely identify each service (which is done internally using the provider_service variable).

The Meta options for app_label and verbose_name are used on the admin GUI.

In some cases, if you have no additional model fields you may want to add proxy = True to the class Meta, so it can use it’s super’s data model, per Django’s documentation.

We’re not using proxy in this example because we’re adding the following additional fields:

    service_message = models.CharField(max_length=254, help_text="Service Message to Display")

This uses Django’s Models to create a CharField in the data model. This field stores the message all Tenants of this Service will see. Think of this as a service-wide configuration parameter.

Extending TenantWithContainer

We extend XOS Core’s TenantWithContainer class, which is a Tenant that creates a VM instance:

class ExampleTenant(TenantWithContainer):

    KIND = SERVICE_NAME

    class Meta:
        verbose_name = TENANT_NAME_VERBOSE

as in Extending Service.

The following is the message that will be displayed on a per-Tenant basis:

    tenant_message = models.CharField(max_length=254, help_text="Tenant Message to Display")

Think of this as a tenant-specific (service intance specific) parameter.

When creating the Tenant, provide a default value of the first service available in the UI.

    def __init__(self, *args, **kwargs):
        exampleservice = ExampleService.get_service_objects().all()
        if exampleservice:
            self._meta.get_field('provider_service').default = exampleservice[0].id
        super(ExampleTenant, self).__init__(*args, **kwargs)

On save, you may need to create an Instance, which is done by calling the model_policy function (see below).

    def save(self, *args, **kwargs):
        super(ExampleTenant, self).save(*args, **kwargs)
        model_policy_exampletenant(self.pk)

On delete, you need to delete the instance created by this Tenant, which is done by cleanup_container().

    def delete(self, *args, **kwargs):
        self.cleanup_container()
        super(ExampleTenant, self).delete(*args, **kwargs)

Finally, if a TenantWithContainer is updated, call manage_container() to create or destroy the appropriate VMs.

def model_policy_exampletenant(pk):
    with transaction.atomic():
        tenant = ExampleTenant.objects.select_for_update().filter(pk=pk)
        if not tenant:
            return
        tenant = tenant[0]
        tenant.manage_container()

Create a Django Admin

Create a file named admin.py in your service directory. This file implements a graphical interface for your service. The start of this file:

from core.admin import ReadOnlyAwareAdmin, SliceInline
from core.middleware import get_request
from core.models import User

from django import forms
from django.contrib import admin

from services.exampleservice.models import *

Import the classes to extend, as well as other needed functions.

Also import the model we created, and the _NAME_ variables.

Extend Admin Classes for the Service and Tenant Classes

Specify that this Form will use the Service model we defined before.

class ExampleServiceForm(forms.ModelForm):

    class Meta:
        model = ExampleService

When creating the Form, set initial values for the fields as follows:

    def __init__(self, *args, **kwargs):
        super(ExampleServiceForm, self).__init__(*args, **kwargs)

        if self.instance:
            self.fields['service_message'].initial = self.instance.service_message

Save the validated data, for who created this Tenant and the message.

    def save(self, commit=True):
        self.instance.service_message = self.cleaned_data.get('service_message')
        return super(ExampleServiceForm, self).save(commit=commit)

Similar to Extending Service:

class ExampleServiceAdmin(ReadOnlyAwareAdmin):

    model = ExampleService
    verbose_name = SERVICE_NAME_VERBOSE
    verbose_name_plural = SERVICE_NAME_VERBOSE_PLURAL

and have this use the ExampleServiceForm defined above.

    form = ExampleServiceForm
    inlines = [SliceInline]

Display the Slice tab.

    list_display = ('backend_status_icon', 'name', 'service_message', 'enabled')
    list_display_links = ('backend_status_icon', 'name', 'service_message' )

Columns to display for the list of ExampleService objects, in the Admin GUI at /admin/exampleservice/exampleservice/.

    fieldsets = [(None, {
        'fields': ['backend_status_text', 'name', 'enabled', 'versionNumber', 'service_message', 'description',],
        'classes':['suit-tab suit-tab-general',],
        })]

    readonly_fields = ('backend_status_text', )
    user_readonly_fields = ['name', 'enabled', 'versionNumber', 'description',]

and rows displayed when viewing an ExampleService at /admin/exampleservice/exampleservice/<service id>/ with field privileges.

Render this page for Service admin users:

    extracontext_registered_admins = True

Order of the tabs, and additional Suit form includes are specified as:

    suit_form_tabs = (
        ('general', 'Example Service Details', ),
        ('slices', 'Slices',),
        )

    suit_form_includes = ((
        'top',
        'administration'),
        )

List only a user’s service objects in the Suit form_tabs:

    suit_form_tabs = (
    def queryset(self, request):
        return ExampleService.get_service_objects_by_user(request.user)

Register the ExampleServiceAdmin with Django:

admin.site.register(ExampleService, ExampleServiceAdmin)

Specify that this Form will use the Tenant model we defined before:

class ExampleTenantForm(forms.ModelForm):

    class Meta:
        model = ExampleTenant

Create a field later used to assign a user to this Tenant:

    creator = forms.ModelChoiceField(queryset=User.objects.all())

When creating the Form, set initial values for the fields:

    def __init__(self, *args, **kwargs):
        super(ExampleTenantForm, self).__init__(*args, **kwargs)

        self.fields['kind'].widget.attrs['readonly'] = True
        self.fields['kind'].initial = SERVICE_NAME

        self.fields['provider_service'].queryset = ExampleService.get_service_objects().all()

        if self.instance:
            self.fields['creator'].initial = self.instance.creator
            self.fields['tenant_message'].initial = self.instance.tenant_message

        if (not self.instance) or (not self.instance.pk):
            self.fields['creator'].initial = get_request().user
            if ExampleService.get_service_objects().exists():
                self.fields['provider_service'].initial = ExampleService.get_service_objects().all()[0]

Do the same as for ExampleServiceForm, but now for ExampleTenantForm:

    def save(self, commit=True):
        self.instance.creator = self.cleaned_data.get('creator')
        self.instance.tenant_message = self.cleaned_data.get('tenant_message')
        return super(ExampleTenantForm, self).save(commit=commit)

See notes above on ExampleServiceAdmin – this configures the fields for the Tenant Admin GUI.

class ExampleTenantAdmin(ReadOnlyAwareAdmin):

    verbose_name = TENANT_NAME_VERBOSE
    verbose_name_plural = TENANT_NAME_VERBOSE_PLURAL

    list_display = ('id', 'backend_status_icon', 'instance', 'tenant_message')
    list_display_links = ('backend_status_icon', 'instance', 'tenant_message', 'id')

    fieldsets = [(None, {
        'fields': ['backend_status_text', 'kind', 'provider_service', 'instance', 'creator', 'tenant_message'],
        'classes': ['suit-tab suit-tab-general'],
        })]

    readonly_fields = ('backend_status_text', 'instance',)

    form = ExampleTenantForm

    suit_form_tabs = (('general', 'Details'),)

    def queryset(self, request):
        return ExampleTenant.get_tenant_objects_by_user(request.user)

admin.site.register(ExampleTenant, ExampleTenantAdmin)

Install the Service in Django

So that Django loads your Service, you need to add it to the list of INSTALLED_APPS.

This is set in xos/xos/settings.py:

INSTALLED_APPS = (
    ...
    'services.exampleservice',
    ...
)

Next, so any data migrations get run if the data model of your service changes, you need to tell XOS to run that migration when it comes up. This is done by adding a line to xos/tools/xos-manage.

function makemigrations {
    ...
    python ./manage.py makemigrations exampleservice
    ...
}

Test Your Admin Interface

Go through the development loop to include your service in XOS. During the final make step, you may want to run docker logs -f devel_xos_1 and look out for any errors which may occur when you first run the code. If so, fix them and restart the loop.

Once XOS is up, go to http://<ip_or_dns_name_of_host>:9999/admin/exampleservice, and you should see the Admin interface:

ExampleService Administration
ExampleService Administration

Select “Change” next to “Example Services”, and you’ll see list of Example Services (empty for now):

Click on “Add Example Service”, and you’ll see options for configuring a service.

Fill in the “Name:”, “VersionNumber:”, and “Service Message”, fields, then click the “Slices” tab at top, then “Add another slice”.

Fill in the slice name, then select “mysite” in the Site popdown, then click “Save”.

You should see a message similar to this saying that adding the service was successful.

The slice configuration may not set a default OS image to be created for instances of this slice. To set this, go to Slices in the left side navigation, select the slice you created, and next to Default Image select trusty-server-multi-nic, which is an ubuntu VM created for use with XOS instances.

Go back to the main ExampleService admin page at /admin/exampleservice and next to “ExampleTenants” click “Add”.

Fill in a “Tenant Message”, then click Save. You should then see a message that “Success! The Example Tenant “exampleservice-tenant-1” was added successfully.”, and a list of Tenants with your message listed.

Create a Synchronizer

Synchronizers are processes that run continuously, checking for changes to the Tenant model and applying them to the running Instances. In this case, we’re using TenantWithContainer, which creates a Virtual Machine Instance for us.

XOS Synchronizers are located in the xos/synchronizers directory in the XOS source tree. It’s customary to name the synchronizer directory with the same name as your service. The example code given below is in the XOS repo at xos/synchronizers/exampleservice.

Create a file named model-deps with the contents: {}.

NOTE: This is used to track model dependencies using tools/dmdot, but that tool currently isn’t working.

Create a file named exampleservice-synchronizer.py:

#!/usr/bin/env python

# Runs the standard XOS synchronizer

import importlib
import os
import sys

synchronizer_path = os.path.join(os.path.dirname(
    os.path.realpath(__file__)), "../../synchronizers/base")
sys.path.append(synchronizer_path)
mod = importlib.import_module("xos-synchronizer")
mod.main()

This is boilerplate. It loads and runs the default xos-synchronizer module in it’s own Docker container.

To configure this module, create a file named exampleservice_config, which specifies various configuration and logging options:

# Required by XOS
[db]
name=xos
user=postgres
password=password
host=localhost
port=5432

# Required by XOS
[api]
nova_enabled=True

# Sets options for the synchronizer
[observer]
name=exampleservice
dependency_graph=/opt/xos/synchronizers/exampleservice/model-deps
steps_dir=/opt/xos/synchronizers/exampleservice/steps
sys_dir=/opt/xos/synchronizers/exampleservice/sys
logfile=/var/log/xos_backend.log
pretend=False
backoff_disabled=True
save_ansible_output=True
proxy_ssh=False

NOTE: Historically, synchronizers were named “observers”, so s/observer/synchronizer/ when you come upon this term in the XOS code/docs.

Create a directory within your synchronizer directory named steps. In steps, create a file named sync_exampletenant.py:

import os
import sys
from django.db.models import Q, F
from services.exampleservice.models import ExampleService, ExampleTenant
from synchronizers.base.SyncInstanceUsingAnsible import SyncInstanceUsingAnsible

parentdir = os.path.join(os.path.dirname(__file__), "..")
sys.path.insert(0, parentdir)

Bring in some basic prerequities, Q to perform complex queries, and F to get the value of the model field. Also include the models created earlier, and SyncInstanceUsingAnsible which will run the Ansible playbook in the Instance VM.

class SyncExampleTenant(SyncInstanceUsingAnsible):

    provides = [ExampleTenant]

Used by XOSObserver : sync_steps to determine dependencies.

    observes = ExampleTenant

The Tenant that is synchronized.

    requested_interval = 0

    template_name = "exampletenant_playbook.yaml"

Name of the ansible playbook to run.

    service_key_name = "/opt/xos/synchronizers/exampleservice/exampleservice_private_key"

Path to the SSH key used by Ansible.

    def __init__(self, *args, **kwargs):
        super(SyncExampleTenant, self).__init__(*args, **kwargs)

    def fetch_pending(self, deleted):

        if (not deleted):
            objs = ExampleTenant.get_tenant_objects().filter(
                Q(enacted__lt=F('updated')) | Q(enacted=None), Q(lazy_blocked=False))
        else:
            # If this is a deletion we get all of the deleted tenants..
            objs = ExampleTenant.get_deleted_tenant_objects()

        return objs

Determine if there are Tenants that need to be updated by running the Ansible playbook.

    def get_exampleservice(self, o):
        if not o.provider_service:
            return None

        exampleservice = ExampleService.get_service_objects().filter(id=o.provider_service.id)

        if not exampleservice:
            return None

        return exampleservice[0]

Find the ExampleService that this Tenant belongs to, by calling get_service_objects with the object’s provider_service.id.

    # Gets the attributes that are used by the Ansible template but are not
    # part of the set of default attributes.
    def get_extra_attributes(self, o):
        fields = {}
        fields['tenant_message'] = o.tenant_message
        exampleservice = self.get_exampleservice(o)
        fields['service_message'] = exampleservice.service_message
        return fields

Find the tenant_message and service_message variables, to pass to the Ansible playbook.

Create Ansible Playbooks

In the same steps directory, create an Ansible playbook named exampletenant_playbook.yml which is the “master playbook” for this set of plays:

---
# exampletenant_playbook

- hosts: "{{ instance_name }}"
  connection: ssh
  user: ubuntu
  sudo: yes
  gather_facts: no
  vars:
    - tenant_message: "{{ tenant_message }}"
    - service_message: "{{ service_message }}"

This sets some basic configuration, specifies the host this Instance will run on, and the two variables that we’re passing to the playbook.

  roles:
    - install_apache
    - create_index

This example uses Ansible’s Playbook Roles to organize steps, provide default variables, organize files and templates, and allow for code reuse. Roles are created by using a set directory structure.

In this case, there are two roles, one which installs Apache, and one which creates the index.html file from a Jinja2 template.

Create a directory named roles inside steps, then create two directories named for your roles, install_apache and create_index.

Within install_apache, create a directory named tasks, then within that directory, a file named main.yml. This will contain the set of plays for the install_apache role. To that file add the following:

---
- name: Install apache using apt
  apt:
    name=apache2
    update_cache=yes

This will use the Ansible apt module to install Apache.

Next, within create_index, create two directories, tasks and templates. In templates, create a file named index.html.j2, with the contents:

ExampleService
 Service Message: "{{ service_message }}"
 Tenant Message: "{{ tenant_message }}"

These Jinja2 Expressions will be replaced with the values of the variables set in the master playbook.

In the tasks directory, create a file named main.yml, with the contents:

---
- name: Write index.html file to apache document root
  template:
    src=index.html.j2
    dest=/var/www/html/index.html

This uses the Ansible template module to load and process the Jinja2 template then put it in the dest location. Note that there is no path given for the src parameter - Ansible knows to look in the templates directory for templates used within a role.

As a final step, you can check your playbooks for best practices with ansible-lint if you have it available.

Create a Docker container to run the Synchronizer

Synchronizers run in their own Docker containers, and these containers are defined in the Docker Compose files in each configuration. For the devel configuration, we’ll need to modify xos/configurations/devel/docker-compose.yml.

Using the xos_synchronizer_openstack as an example, create a new section as follows:

...
xos_synchronizer_exampleservice:
    image: xosproject/xos-synchronizer-openstack
    command: bash -c "sleep 120; python /opt/xos/synchronizers/exampleservice/exampleservice-synchronizer.py -C /opt/xos/synchronizers/exampleservice/exampleservice_config"
    labels:
        org.xosproject.kind: synchronizer
        org.xosproject.target: exampleservice
    links:
        - xos_db
    extra_hosts:
        - ctl:${MYIP}
    volumes:
        - ../common/xos_common_config:/opt/xos/xos_configuration/xos_common_config:ro
        - ../setup/id_rsa:/opt/xos/synchronizers/exampleservice/exampleservice_private_key:ro
...

We’ll use the same synchronizer image, xosproject/xos-synchronizer-openstack, as it is suitable in most cases. The command is a path to the Synchronizer and it’s config file. The org.xosproject.target label should be updated as well.

For Ansible to communicate with the VM, it requires an SSH key in order to communicate with the the Instance VM. This is added read-only as a Docker volume: - ../setup/id_rsa:/opt/xos/synchronizers/exampleservice/exampleservice_private_key:ro

Remember to rebuild your docker containers (make rm && make containers && make) after making these changes. Then verify that your new container is running with sudo docker ps, in addition to the other 3 devel configuration containers.

In the Admin web UI, navigate to the Slice -> <slicename> -> Instances, and find an IP address starting with 10.11.X.X in the Addresses column (this address is the “nat” network for the slice, the other address is for the “private” network).

Run curl <10.11.X.X address>, and you should see the display message you entered when creating the ExampleTenant.

user@ctl:~/xos/xos/configurations/devel$ curl 10.11.10.7
ExampleService
 Service Message: "Example Service Message"
 Tenant Message: "Example Tenant Message"

After verifying that the text is shown, change the message in the “Example Tenant” and “Example Service” sections of the Admin UI, wait a bit for the Synchronizer to run, and then the message that curl returns should be changed.

Debugging

XOS isn’t coming up after making changes

Verify that the docker containers for XOS are running with:

sudo docker ps

If you need to see log messages for a container:

sudo docker logs <docker_container>

There’s also a shortcut in the makefile to view logs for all the containers: make showlogs

If you want to delete the containers, including the database, and start over, run:

make rm

Which will delete the containers.

“500 Internal Server Error” when navigating the admin webpages

This is most likely Django reporting a problem in admin.py or model.py.

Django’s debug log is located in in /var/log/django_debug.log on the xos container, so run make enter-xos and then look at the end of that logfile.

“Ansible playbook failed” messages

The logs messages for when the Synchronizer runs Ansible are located in /opt/xos/synchronizers/<servicename>/sys in its synchronizer container. There are multiple files for each Tenant instance, including the processed playbook and stdout/err files . You can run a shell in the docker container with this command to access those files:

sudo docker exec -it devel_xos_synchronizer_<servicename>_1 bash

Ansible log messages for the OpenStack Synchronizer are put in /opt/openstack/*, if you’re seeing failures in the devel_xos_synchronizer_openstack_1 container.