Software on SlapOS is divided into two parts. First is the Software Release. It represents the entire installation of a software - without configuration files. Because configuration is missing, a software is not usable from just a Software Release, eg Wordpress is installed but no disk image or specific configuration exits.

This is created using the Software Instance. It will reuse the installed Software Release by creating wrappers, configuration files and anything specific to an instance, eg the missing disk image and config for Wordpress. During instantiation SlapOS only creates what is needed on a machine. As the Software is already installed on the machine, SlapOS can create instances using the Software Release and adding only what is missing. This way the installation can be shared across all instances saving space and not requiring to reinstall the complete software every time. The number of instances available to be instantiated on a machine depends on the number of available computer partitions.

Machines in a SlapOS system are called Nodes. Not servers. Not computers. A Software Release is a link to a git repository and defines a software through the software profile (can also be called buildout profile). It is used to install (supply) a software on a node while the instance profile used for instantiation.

SlapOS provides Software Instances of an installed software. It uses Service(s) which are not considered instances of a software as many services can be used to run a software. A software instance is deployed inside a Computer Partition (or Node partition). Partitions are containers in which a software instance resides.

Use Branches

When working on a software release, create your own branch! Create separate branches for a specific, atomic, feature and name the branch based on that (don't use your name!). When associated tests are passing, ask for review and merge to master. The development profile should be located at software/[name]/development.cfg (optional).

Good Example

wordpress, wordpress-upgrade-3.5, erp5-multihomed, resiliency-cleanup

Bad Example

Cedric

Never copy / paste more than 30 lines of code

This increases the cost of maintaining code in multiple locations, discourages code reuse and efforts of making generic software components that can be used in different recipes.

Always PIN versions of eggs/products

SlapOS / Buildout will try to install the most recent versions of eggs available by default. If versions aren't pinned, there's no guarantee that you will get the expected version as you might get the newest one which might break the system.

Good Example

[versions]
Flask-Auth = 0.85
apache-libcloud = 1.2.1
cns.recipe.symlink = 0.2.3

Bad Example

[versions]

Tag Software Release for Production

If you deploy a Software Release to a production system it's completely forbidden to use untagged one like git's HEAD. The reason is that in this case your software release will likely change in time and any software build can have unexpected results.

Good Example

https://lab.nexedi.cn/nexedi/slapos/blob/1.0.13/software/erp5/software.cfg

Bad Example

https://lab.nexedi.cn/nexedi/slapos/raw/master/software/erp5/software.cfg

Only release a (tagged), precompiled "frozen" Software Release for production

If you want to release a software release to the public, it's mandatory to pre-compile it and sign it in shacache.org for a minimum set of operating system that you plan to support. Without shacache everything will be compiled from scratch, which may take several hours for a big ERP5 release and can sometimes depend on environmental variables in OS which have a great chance to break the compilation process. With shacache the time to install is almost equal to time to download the already pre-compiled package which is much more user friendly.

Note, that Software Releases are frozen once correctly installed. They won't be processed anymore, because SlapGrid will ignore them. However all software instances are supposed to be processed (i.e. Buildout is run) at least once per day. There should be no exception.

In consequence, here are a few conventions while writing profiles and recipes:

Stable (in production) Software Release profiles should be frozen in a git tag.

Good Example

https://lab.nexedi.com/nexedi/slapos/blob/slapos-0.137/software/kvm/software.cfg

Bad Example

https://lab.nexedi.com/nexedi/slapos/raw/master/software/kvm/software.cfg

Instance Profiles/Recipes should be Promise-based and not break/alter existing data

You should always check for existing content before injecting/altering content.

Bad Example

# Recipe code that blindly sets application password without checking if it has been changed
def install(self):
   set_password()

Good Example

def install(self):
  if not password_already_set:
    set_password()

Every Instance requires at least one Promise script testing Connectivity

Good Example

Promise testing "Does port YYY answer HTTP 200"

Bad Example

No Promise

Use different ports for all instances composing a service

Good Example

# Wordpress Service composed from Apache and MariaDB instance
# Apache: httpd process listing on port W (IPv4) and stunnel on port X (IPv6)
# MariaDB: mysqld process listening on port X (IPv4) and stunnel on port Z (IPv6)
=> W,X and X,Z should be different

Bad Example

1) Port collusion
2) Service using reserved Webrunner IPv6 ports (30000, 50000, 2222)

Builout's contribution guidelines apply for SlapOS Buildout Profiles, too, especially:

[Section] and parameter names use dash not underscore

Good Example

[install-foo-bar]
# unless section is based on file name foo_bar.conf => [install-foo_bar.conf]

Bad Example

[test_foo_bar]

Section name must be representative of what it does

Good Example

# part deploying kvm => "kvm"
# part creating directories => "directory"

Bad Example

# part deploying kvm => "virtual-machine-deployment"

Falsy Parameters (empty string, None, []) should be equivalent to non-existence of parameter

To ease integration with Buildout

Good Example

# does not exist :)

Bad Example

parameter_foo = []
property = ${section:value_set_to_none_for_any_reason}

Parameters must be Atomic

For maintenance and security reasons, it should NOT be possible to pass a configuration file as instance parameter. Only atomic parameters are allowed.

Good Example

host = ...
ip = ...
port = 123
url = ...
password = Foo

Bad Example:

dict = {"host": ..., "ip": ... }

Maintain common parameter names

Good Example

Reuse host, ip, port, url, ...

Bad Example

Use hostname, ip-address, port-number, access-url instead

Defining all required/possible parameters in a Software Release can be tricky, resulting in a non-flexible Software Release. For practical reasons, giving a configuration file in such a case is accepted for the first versions of the Software Release. This should then disappear as new atomic possible parameters are added.

Component name matches file name

Good Example

Component "foo v1.2.3" => component/foo/buildout.cfg

Bad Example

Component "foo v1.2.3" => component/foo.cfg

The main section is named like the Component

Good Example

Component "foo v1.2.3" => [foo]

Bad Example

Component "foo v1.2.3" => [main]

Main section extends to latest component version

Several incompatible versions ("foo v1.2", "foo v1.3") can live in the same Buildout profile (component/foo/buildout.cfg). A main section ([foo]) must extend the latest one. The different versions should be in different sections ("[foo v1.2.3]", "[foo v1.2.4]").

Good Example

[foo]
<= foo-1.2

[foo-1.2]
recipe = slapos.recipe.cmmi
url = http://www.foo.com/foo-1.2.6.tar.gz
md5sum = abcdef012345679

[foo-1.3]
recipe = slapos.recipe.cmmi
url = http://www.foo.com/foo-1.3.2.tar.gz
md5sum = 987654321fedcba

Bad Example

Patches are kept in separate files

Good Example

# patch is in: component/foo/my-patch.patch
[my-patch.patch]
recipe = hexagonit.recipe.download
filename = ${:_buildout_section_name_}
url = ${:_profile_base_location_}/${:filename}
md5sum = ac06cbaa298ac686d0b0c04bc03e6ad8
    download-only = true

Bad Example

No "parts" should be defined in [buildout] section

Good Example

Bad Example

Minor dependencies can be kept in the component buildout profile

Good Example

Bad Example

Reuse existing "generic" recipes

The golden rule: reuse recipes like "create directory" (slapos.cookbook: makedirectory) or "create a wrapper" (slapos.cookbook:wrapper). Only highly customized Software Releases should write custom recipes. Obvious duplicates should be avoided. Example: ERP5 Software Release and Wordpress Software Release should use the same recipe to deploy MySQL.

Good Example

# use slapos.cookbook:makedirectory

Bad Example

# roll your own

Fail early and with understandable Messages

Errors should be obvious and their origin easy to understand. This means "fail early" and not silently continue until another recipe raises with a different error message. Expected errors (like raising because of a not-yet ready child instance) should be easy to understand, and should not be mistaken with an unexpected error.

Good Example

raise AttributeError("attribute %s is not defined" % (key))

Bad Example

pass

Recipes should be named as such and extend from GenericBaseRecipe (slapos.recipe.librecipe.GenericBaseRecipe)

Good Example

class Recipe(GenericBaseRecipe):

Bad Example

class MakeDirectory(SomeOtherRecipe):

Get SLAP parameters from Buildout Instance Profile

Recipes should take their SLAP instance parameters from the Buildout Instance Profile, as parameters. They should NOT (except in case of special need) retrieve parameters by calling the slap library. If they need to get/set SLAP parameters for cases not covered (example: slave handling), they must extend slapos.recipe.librecipe.GenericSlapRecipe.

Good Example

Bad Example

Publish SLAP connection parameters using Pubish recipe

Publishing SLAP connection parameters should be done using slapos.cookbook:publish in the Buildout Instance Profile, not directly from the code in a recipe.

Good Example

Bad Example:

ERP5 is a special case in slapos.git. In order to prevent ERP5 related branches to not diverge too much from other branches based on master, the following workflow is mandatory:

• Merge Master into erp5-component. Use temporary branch to verify tests pass
• Merge erp5-component into erp5 when stable. Use temporary branch to verify tests pass
• Merge erp5 into master when stable. Use temporary branch to verify tests pass

In principle this approach should be applied for all branches.

Instances generated from a Software Release take parameters (typically to customise instantiation and instance behaviour) and publish results (typically allowing access to requester). The structure of these values is constrained by how the Software Release was implemented, and must be documented so it can be used. Instance descriptors are intended to provide such documentation in a form allowing automated generation of a user interface to consult and provide parameters, and to consult published results (see also Instance descriptors).

One way of creating forms for users to define instance parameters is by defining JSON schemas.

The JSON entry point is used to list the available software types and forms the user can choose from, for example whether to instantiate a normal or resilient Webrunner (2 software types) or choosing between a slave and token (also 2 different software types.

By convention the JSON file is named by appending .json to the software.cfg, so:

• Software Release: https://lab.nexedi.com/nexedi/slapos/raw/master/software/re6stnet/software.cfg
• JSON Entry Point: https://lab.nexedi.com/nexedi/slapos/raw/master/software/re6stnet/software.cfg.json

There can be multiple forms for the same Software Type having different definitions (if it's a slave or token). Important entries are:

• "title" is what is displayed to the user on the form
• "software-type"and "shared" are what will be sent when the instance is requested. software type defines type and shared defines if it is slave or not.
• "request" is the link (relative link) to the file containing a json schema. This JSON Schema will be used to render the form to the user.

An online editor can be found here and the sample schema is a good place to start as is the example below:

{
    "name": "NAME GOES HERE",
    "description": "DESCRIPTION GOES HERE",
    "serialisation": "xml",
    "software-type": {
        "default": {
            "title": "DEFAULT",
            "description": "DESCRIPTION",
            "software-type": "default",
            "request": "instance-NAME-input-schema.json",
            "response": "instance-NAME-output-schema.json",
            "index": 0
        }
    }
}

With the entry point set you can start writing the schemas for the different software types. It is advised to use a simple flat approach (1 level deep). Also keep in mind:

• Keep it short, the more parameters, the longer and complex is the form to the user.
• Don't include entries which can be automaticaly generated or determinated by a convention (port numbers) as well as default values that don't requires changes.
• Don't include values which depend on other instances if these values can be taken automatically.
• When using two different software on the same Hosting Subscription (eg Apache and MariaDB), use prefixes on the variable names to keep track.
• Title and Description have to be meaningful, as this is what the user will see on the form.

Once the JSON schemas are published, the form will be rendered automatically by SlapOS Master. Note, that schemas are not stored, the get pulled in by URL if available.

Examples of software.cfg.json

• https://lab.nexedi.com/nexedi/slapos/raw/master/software/erp5/software.cfg.json
• https://lab.nexedi.com/nexedi/slapos/raw/master/software/slaprunner/software.cfg.json
• https://lab.nexedi.com/nexedi/slapos/raw/master/software/kvm/software.cfg.json
• https://lab.nexedi.com/nexedi/slapos/raw/master/software/apache-frontend/software.cfg.json

Examples of schemas:

• https://lab.nexedi.com/nexedi/slapos/raw/master/software/re6stnet/instance-re6stnet-input-schema.json
• https://lab.nexedi.com/nexedi/slapos/raw/master/software/re6stnet/instance-re6stnet-slave-input-schema.json
• https://lab.nexedi.com/nexedi/slapos/raw/master/software/apache-frontend/instance-slave-apache-input-schema.json
• https://lab.nexedi.com/nexedi/slapos/raw/master/software/apache-frontend/instance-apache-input-schema.json
• https://lab.nexedi.com/nexedi/slapos/blob/master/software/caucase/instance-caucase-input-schema.json

As mentioned earlier, the instance profile instance.cfg is run by Buildout when SlapGrid asks for deployment of an instance. This is a simple Buildout profile showing how to deploy the instance:

• Create a directory
• Generate MySQL configuration
• Create a wrapper from the mysqld binary installed with the Software Release
• ...

The SlapOS node processes (by running Buildout) each contained instance at least once per day. This means that instance profiles and recipes have to be "Promise-based", shoudn't blindly initialize the instance and take into account that they will be run regularly. One practical implication of this is to check for existing data before overwriting anything that can probably break an instance. (Pseudocode) Example:

def install(self):
if not password_already_set:
  set_password()

Here, it is safe to run often buildout for the instance because it won't overwrite your password.

A software instance (or service) is composed of one or more exectuables (eg Apache, Mariadb, Varnish, ...) that are run by the operating system. SlapOS allows to define such exectuables that will be run when starting the instance and terminated when stopping the instance. There are two types of executables:

• ${buildout:directory}/etc/services: All executables that are needed for the service (mysqld, apache, ...) should go to ${buildout:directory}/etc/services. If one of them exits, it will trigger the sending of an alert (bang) to the SlapOS Master and cause buildout to run for this instance.
• ${buildout:directory}/etc/run: All executable scripts designed to run for a short time and exit when done without disturbing the service should go to ${buildout:directory}/etc/run. Scripts are usually required to bootstrap the service but are not necessary for the service itself. Examples include generation of SSL keys, helper tools needed to inject commands to the executable acting as service, etc.

Executables can be anything, for example shell scripts, Python scripts or symbolic links to executable located in the Software Release directory (/opt/slapgrid/0123456789abcdef). Usually it's a simple shell script that runs (exec) an executable located in the Software Release directory providing all the needed arguments like location of a configuration file or the option to stay in foreground

After the SlapOS node has run buildout for a Software Instance it will add all executables present in the above two directories to the Supervisor (process manager) configuration who will then request the exectuables to start or stop. Supervisor is also responsible for reporting any suspect process exit to the SlapOS node.

Users are usually required to specifiy a number of parameters for their instance. Parameters can be arbitrary, for example the instantiation of a KVM allows to define amount of RAM, CPU, disk etc. For making sure an instance works at all times, default parameters have to be provided. These serve two purposes:

• Fallback in case user doesn't provide a parameter
• Prevent Buildout from exiting with Parameter not found with an empty value

The general rule is that an empty parameter is equivalent to not providing a parameter at all. For example, in the KVM instance profile, it is possible to add a [slap-parameter] section representing all parameters given by the user:

[slap-parameter]
# Default value if no second disk is specified. Will be ignored by the kvm recipe
second-disk-location =
# Default value for RAM amount, in MB
ram-size = 1024

This way, if the user defines ram-size the given value will just be overwritten.