This article explains how to deploy your software on an edge computing infrastructure operated by Rapid.Space... or by yourself with the free software "SlapOS". To reproduce the steps in the article, you will be given access to an instance of a service called Theia, deployed by SlapOS and running on a machine somewhere. That machine can be a server in a data center or just a simple Raspberry Pi sitting next you and connected to your Internet box . That is the promise of SlapOS: to work just as well in a traditional cloud data center as in a machine you own, located in your home or your office, and not necessarily more powerful than a simple Raspberry Pi. That last scenario is what we call edge computing.

We will take as an example the Matomo application, an alternative to Google Analytics with the advantage of being open source and RGPD compatible. You can then automatically create an instance of Matomo inside your Theia and access it from anywhere in the world using Rapid.Space's Content Delivery Network (CDN).

Beyond Rapid.Space, you can also automatically deploy any number of Matomo instances on any public or private cloud infrastructure that is already compatible with SlapOS: Teralab, OVH, Scaleway, AWS, Alicloud, VMWare, Vates, etc. You can also add your own infrastructure regardless of the platform (x86, ARM) or Linux distribution (Debian, Ubuntu, Red Hat, etc.).

From cloud to edge

For some developers, the cloud is "someone else's computer" hosted in a large data center. For Richard Stallman it is "a trap to force more people to buy locked-down, proprietary systems that will cost them more and more over time". And more recently for David Heinemeier Hansson, the creator of "Ruby on Rails", these are "obscene" bills.

But these stereotypes obscure the fact that the cloud is primarily a technology for automating the deployment and operation of an IT service. The cloud is to the combine what the system administrator is to the farm worker: an unstoppable industrial revolution that replaces manual work with automation.

The cloud is therefore above all the software that makes it possible to do without system administrators. When you order a database service from a hyperscaler, or from Rapid.Space, your order is taken into account in a central management system, a sort of cloud ERP. It is then executed by daemons in each server of the cloud infrastructure. Your database service is then delivered, monitored, backed up, repaired if necessary and invoiced... without any human intervention.

There is nothing to prevent you from running the daemons that automate the cloud on servers located at home or in your office and on a machine that belongs to you: a Raspberry Pi for example. It is then possible to benefit from all the advantages of the cloud in terms of automation without depending on someone else's computer, whether for IaaS, PaaS or SaaS.

This is the idea of edge computing: an infrastructure consisting of thousands of sites with a few servers rather than a few sites with thousands of servers. This idea, conceived in 2008 by the founders of Rapid.Space with the creation of the open source software "SlapOS", has since been successfully implemented for Airbus, Kyorin, the city of Munich, SANEF, Stellantis, Toyota, etc. on critical services, some of which are used by more than two million users. Rapid.Space's edge is also deployed in autonomous drone swarms, 5G telecommunication infrastructure, machine tools, etc.

And if the cloud software, infrastructure plans and operating procedures are all released under an open license, there is no longer any possible "trickery" or "obscene billing". This is the idea of "Fully Open": you are free to set up your own edge cloud provider within weeks if you are not satisfied with Rapid.Space's pricing or service. Everything you need is published on https://handbook.rapid.space.

So with edge computing and Fully Open offerings there is no reason not to adopt cloud technologies.

Theia: an IDE for the edge

Theia is a web-based IDE that makes it very easy to deploy applications on the Rapid.Space edge. Theia was created by a German company called Typefox. It is now part of the Eclipse Foundation's software portfolio. It is used by companies such as SAP, Ericsson, etc. It is now well integrated with SlapOS and financially supported by Rapid.Space. You can learn more about it at theia-ide.org.

Access to Theia

Go to your given Theia URL, and enter the corresponding password.

Enter the username "admin" and the password previously copied to access your Theia.

If small dialogue windows open in the bottom right corner, you can accept or close them.

Change the colour theme (optional)

In the rest of this tutorial we use the dark theme of Theia. If you want to do the same, change the theme with File > Preferences > Color Theme and choose "Dark (Theia)".

You can also access all settings with File > Preferences > Open Settings.

Open the file explorer

Open the file explorer by clicking on the icon of a double sheet of paper at the top left-hand edge.

In the "slapos" directory, the sub-folder "software" contains a list of all software versions available in Theia. The list is based on the official SlapOS software repository.

Open a terminal

You can close the "Problems" tab which sometimes opens at the same time as your first terminal.

You can drag-and-drop the terminal wherever you want, for example to the right sidebar, which allows you to minimize it by clicking on the icon.

Buildout: everything to automate

Buildout is the language used by SlapOS to automate all facets of Matomo deployment on the edge:

1. build (the Matomo binary code that will be executed)
2. allocate (resources in the infrastructure)
3. instantiate (Matomo with sufficient resources)
4. configure (Matomo according to the parameters provided by the user)
5. orchestrate (Matomo by automatically connecting PHP to MariaDB)
6. monitor (the state of Matomo through an extensible list of counters and alarms)
7. self-healing (Matomo by optimising its configuration or reallocating the service elsewhere)
8. plan technical continuity (of Matomo by restoring a backup on another infrastructure node)
9. count (the resources used by Matomo that cannot be allocated to another service)
10. billing (resources used by Matomo that need to be billed to someone)
11. track incidents (of Matomo and how they were resolved)
12. testing (new versions of Matomo and their compliance with all requirements before upgrading the infrastructure)

This article only covers the steps from construction (1) to monitoring (6). The other six steps would be required for a commercial deployment of Matomo as an edge or cloud service.

We will now teach SlapOS how to build Matomo. This is the purpose of the software.cfg file.

One of the magic things about SlapOS is that it is platform and distribution independent. It can even, in theory, support operating systems other than Linux: FreeBSD, NuttX, etc. And anything built by buildout is guaranteed not to call any missing system calls or CPU instructions, unlike what sometimes happens with binary distributions, including Docker images.

This requires describing and building all the dependencies... right down to the glibc. By defining all the dependencies, you also ensure that you can rebuild exactly the same thing even 20 years later. And since we keep a copy of the downloaded sources in shacache (shacache.nexedi.com), we can rebuild even if the source code has been depublished.

Fortunately, most of this work has already been done and can be reused using the "extends" statement in buildout.

Create software.cfg

Each software in Slapos starts with a software configuration file. Slapos will first download and compile the necessary components according to the contents of the software.cfg file.

The "slapos" directory contains several subfolders, three of which are of interest to us: "slapos/software", "slapos/component" and "slapos/stack":

• "slapos/software" contains software configuration files that correspond to complete services
• "slapos/component" contains configuration files for many of the individual components on which the software in "slapos/software" depends
• "slapos/stack" contains generic configuration files for particular features used by the software in "slapos/software

Create a new folder named matomo-tutorial in "slapos/software/" (right click on the software folder, select New Folder) to put all files related to the deployment.

Then create a new file software.cfg in this folder (right click on the folder "matomo-tutorial", select New File. Make sure when you name the file that it is in "slapos/software/matomo-tutorial/". If this is not the case, cancel and then click elsewhere before right-clicking on "matomo-tutorial" again).

Write the following code in the software.cfg file:

[buildout]
# stack/lamp installs Apache, MariaDB and PHP
# and deploys the specified web application.
extends =
  ../../stack/lamp/buildout.cfg

[application]
# Download the web application that stack/lamp will run.
url = https://builds.matomo.org/matomo-4.13.3.tar.gz
md5sum = 6fb7750818e7371b0d624e4d24538952
archive-root = matomo

The Matomo application is based on a set of open source software commonly used to build web servers and commonly referred to as LAMP: Linux, Apache, MySQL, PHP. To install Matomo, we therefore also need to install PHP, the Apache web server and an SQL database.

SlapOS already provides a configuration to install LAMP, with MariaDB as database instead of MySQL (MariaDB is originally a fully open source fork of MySQL). You just need to extend it with the "extends" buildout statement. This LAMP configuration is even designed to be extended, and provides customisation points that allow you to define in a few lines which web application to deploy - Matomo in this case - and how to get it. This is the role of the [application].

Customise the MariaDB database (optional)

The previous configuration is already sufficient to deploy Matomo, but before doing so let's add a small improvement:

By default, the MariaDB database deployed by "stack/lamp" has "lamp" as its name and "lamp" as its username as well. To customize this to "matomo" and "matomo", add the following lines to the end of the file:

[custom-application-deployment]
# Customize the database name and username
db-name = matomo
db-user = matomo

Install Matomo

These few lines are enough to deploy Matomo. Let's prove it!

In a terminal, run :

slapos supply ~/srv/project/slapos/software/matomo-tutorial/software.cfg slaprunner

In this command, "~/srv/project/slapos/software/matomo-tutorial/software.cfg" indicates the location of a buildout configuration and "slaprunner" is the name of a cloud infrastructure node. The effect of this command is to populate information in a cloud infrastructure database: this information indicates that the "slaprunner" node must have "~/srv/project/slapos/software/matomo-tutorial/software.cfg" installed.

In a commercial cloud infrastructure such as Rapid.Space's, this database is none other than the "cloud ERP" described at the beginning of this article, and the node is one of the servers in the cloud infrastructure. This command is issued by a cloud administrator to install on this node the set of software needed for a service. When a user later requests an instance of that service, it can then be allocated on that server.

In Theia, the database is in fact a simple SQLite database embedded in the Theia instance, and the "slaprunner" node is contained in the "~/srv/runner" sub-folder. This is a sort of scaled-down mini cloud infrastructure, with a single node (the node inside Theia is always called "slaprunner"), entirely contained in a single service (Theia) in a single machine. This makes it easy to develop a new buildout configuration to deploy a cloud service, testing the deployment locally in Theia. Later it will work the same way in a large scale commercial cloud infrastructure.

This is also what allows us to use a local path to identify our configuration, instead of a universally accessible URL.

In a second step, the infrastructure node will query the database, discover the new entry, retrieve the configuration file and install the required software using buildout. Only then does the installation take place. To start this process manually, run :

slapos node software

In a commercial infrastructure node, this is done automatically in the background at regular intervals (if there is no new input, the node does nothing). This asynchronous approach is a fundamental part of the SlapOS philosophy: it decentralises responsibilities. The database is a passive component that simply lists the desired state of each node. The nodes are the active components that independently converge on the desired state.

In the "slaprunner" node of your Theia, this automatic background execution has been previously disabled: we will have to modify and reinstall "software.cfg" during this tutorial, and the normal automatic operation of the node assumes that the configuration never changes (a fully justified assumption for a production service). For development purposes, this would be more of a hindrance than anything else. You can always re-enable this setting later from the Rapid.Space panel.

Wait for the command to finish running. This can take several hours since all the software in the LAMP stack has to be compiled. Occasionally the installation may abort with an error, for example if a download fails due to a temporary network problem. In this case restart the command until it succeeds.

To restart the command at regular intervals until it succeeds, run :

while ! slapos node software; do sleep 20; done

The software package required for Matomo will be installed in the "slaprunner" node, in a sub-folder of "~/srv/runner/software/". To find the Matomo installation sub-folder, you can run :

find ~/srv/project/runner/software -name matomo

You should see a list of several files displayed.

Instantiate Matomo

Run :

slapos request matomo-1 ~/srv/project/slapos/software/matomo-tutorial/software.cfg

This command allows a client to allocate an instance of a service. Here, "matomo-1" is the name chosen for the instance, and "~/srv/project/slapos/software/matomo-tutorial/software.cfg" identifies the service. Like slapos supply, this command only interacts with the cloud infrastructure database. It will first find a node on which the service has been previously provisioned with slapos supply, and then allocate the service on that node, i.e. it fills in information in the database to indicate that this node should have instantiated the desired service. The instance is identified only by its name and the client that requested it: a client cannot have multiple instances of the same name.

In a commercial cloud infrastructure, this action is more often done through a web interface such as the Rapid.Space panel. In Theia, the only node is "slaprunner" and we just provided "~/srv/project/slapos/software/matomo-tutorial/software.cfg" in the previous step, so that's where the service will be instantiated.

Then, as for the installation, the node will then query the database, discover this new allocation and instantiate the service.

To start this process manually, run :

slapos node instance

This command is used to instantiate services allocated with slapos request, just as slapos node software is used to install software needed for a service, provided with slapos supply. The design of SlapOS intentionally separates the build step from the instantiation step: it allows you to install the software needed for a service on a machine once, and then deploy multiple instances of that service on that machine.

The first execution will end with an error. This is normal, the instantiation requires resources that are not ready immediately. These include frontends in the Rapid.Space CDN, which are used here to provide a domain name for your Matomo web service: without this the service is only accessible through a public IPv6. But thanks to the Rapid.Space CDN, your Matomo instance will be accessible from anywhere in the world in IPv4, even if your local network uses private addresses. However, it takes time for the request to succeed and for a domain name to be allocated.

Rerun the command at regular intervals until it succeeds. This may take several minutes.

To avoid having to restart the command manually, you can run a loop that will stop when the command succeeds, such as

while ! slapos node instance; do sleep 20; done

In a node of a commercial infrastructure, slapos node instance runs automatically in the background at regular intervals, like slapos node software.

The command slapos node instance performs several functions:

• query the database to obtain the list of services allocated on the node
• apply the buildout configuration of the instance (create configuration files, scripts, etc)
• start the processes that run the service (defined by the buildout configuration)
• Regularly check that the instance is running properly (using tests defined by the buildout configuration)

As the conditions of correct operation are specific to each service, the instance defines a set of correct operation tests ("promises") which are checked at regular intervals. For example: can we connect to the MariaDB database? can we access the frontend URL? This is the monitoring function (point 6) mentioned at the beginning. If the instance diverges from its normal operation (for example, if a process dies unexpectedly and this causes a promise to fail), the buildout configuration is eventually applied again to converge the instance to the desired operation.

Only proceed once slapos node instance is running successfully.

Review deployed services

Each instance is allocated to a dedicated folder that will contain its configuration files and instance-specific data. These are called "partitions" of an infrastructure node. A "partition" also has resources dedicated to the instance, such as publicly accessible IPv6 addresses. These partitions are created during the initial configuration of the node.

In Theia, instance data is stored in sub-folders of "~/srv/runner/instance": each instance will have its own sub-folder "~/srv/runner/instance/slappart0", "~/srv/runner/instance/slappart1", "~/srv/runner/instance/slappart2"...

In a node like the on in which your Theia is running, each instance also corresponds to a dedicated user, who has no access to the data of the other instances. The "partitions" are then generally located in "/srv/slapgrid". It is in such a partition that your Theia service is located. You can see this with the pwd command which indicates the path of the current directory.

The LAMP stack will actually deploy 3 instances: the "matomo-1" instance that you requested in turn requests two sub-instances. These are an instance "Mariadb" which will contain the database and an instance "apache-php" which will deploy the web application. This forms a tree of instances with "matomo-1" at the root. The latter simply connects "apache-php" to "Mariadb". This is the orchestration function (point 5) mentioned at the beginning.

To see the processes in each partition, run

slapos node status

You will notice that each instance starts several processes to keep the service running. Some have finished running ("EXITED"): they have only a one-time function. Others are running ("RUNNING"): these are the ones that ensure the continuous operation of the different aspects of the Matomo service (web server, database, monitoring...).

Access Matomo

When the instance instantiates, it passes information to the database: typically the connection parameters that will allow access to the service.

To obtain these parameters, run the command again:

slapos request matomo-1 ~/srv/project/slapos/software/matomo-tutorial/software.cfg

This command is idempotent: as the query is already in the database, it will have no effect. But it has a second function: it displays the connection parameters stored by the instance in the database.

At the first run the instance did not exist yet, so there were no parameters to display. Now you should get something like :

{'backend-url': 'https://[2001:67c:1254:135::9ccf]:9988/',
 'mariadb-url-list': "['mysql://matomo:QOKKGVLhToj4sqbp@10.0.3.25:2099/matomo']",
 'monitor-base-url': 'https://softinst180275.host.vifib.net',
 'monitor-setup-url': 'https://monitor.app.officejs.com/#page=settings_configurator&url=https://softinst180275.host.vifib.net/public/feeds&username=admin&password=NPKYesweTIkdZesA',
 'url': 'https://softinst180276.host.vifib.net'}

The "backend-url" parameter corresponds to the IPv6 access of the service. The "url" parameter corresponds to the domain name of the "frontend" provided by the Rapid.Space CDN, which redirects to "backend-url". The "monitor-base-url" and "monitor-setup-url" parameters allow access to the status of the service promises ("monitoring"). The "mariadb-url-list" parameter contains a URL to access the MariaDB database.

Follow the link provided by "url" (ctrl + click on the link in the terminal, or copy it into a new tab).

This will take you to the Matomo web service, where you will be presented with a configuration form. Click on next until you get to step 3 "Database installation".

The parameters to be entered can be obtained by unpacking the URL of the MariaDB database: in order, in mysql://matomo:QOKKGVLhToj4sqbp@10.0.3.25:2099/matomo :

• the first matomo is the user name
• QOKKGVLhToj4sqbp is the password
• 10.0.3.25:2099 is the server address
• the last matomo is the name of the database

But instead of entering these parameters manually, we can enhance our configuration file to pre-populate these fields in the form!

Improve software.cfg

Complete software.cfg so that you get :

[buildout]
# stack/lamp installs Apache, MariaDB and PHP
# and deploys the specified web application.
extends =
  ../../stack/lamp/buildout.cfg

[application]
# Download the web application that stack/lamp will run.
url = https://builds.matomo.org/matomo-4.13.3.tar.gz
md5sum = 6fb7750818e7371b0d624e4d24538952
archive-root = matomo

[custom-application-deployment]
# Customize the database name and username
db-name = matomo
db-user = matomo
# Include custom configuration in the instance deployed by stack/lamp
path = ${custom-instance-configuration:output}

[custom-instance-configuration]
# Define custom configuration for the instance deployed by stack/lamp
recipe = slapos.recipe.template
output = ${buildout:directory}/custom-matomo-instance-configuration.cfg
inline =
  [apache-php-service]
  # Pre-fill the database installation form with the required values
  environment =
    MATOMO_DATABASE_HOST=$${mariadb-urlparse:host}:$${mariadb-urlparse:port}
    MATOMO_DATABASE_ADAPTER=mysql
    MATOMO_DATABASE_TABLES_PREFIX=matomo_
    MATOMO_DATABASE_USERNAME=$${mariadb-urlparse:username}
    MATOMO_DATABASE_PASSWORD=$${mariadb-urlparse:password}
    MATOMO_DATABASE_DBNAME=$${mariadb-urlparse:path}

The additional configuration is a bit more complicated and relies on more knowledge of the stack/lamp implementation (which can be obtained by reading the files in stack/lamp) and of buildout.

Firstly, the line path = ${custom-instance-configuration:output} allows to provide a file containing a configuration snippet which will be included in the configuration of the instance deployed by "stack/lamp".

Then the [custom-instance-configuration] section will create the file which will contain the configuration snippet to be included, from the contents of "inline". The [apache-php-service] section is already defined in the configuration of the instance deployed by "stack/lamp": it is in charge of launching the Apache web server which will deploy the Matomo application. It is extended here to modify the environment variables defined at the time of launching the Apache process: Matomo uses these environment variables (when they are defined) to pre-fill the form. Expressions such as $${mariadb-urlparse:host} will be replaced by the various fragments of the MariaDB URL.

Rebuild and redeploy Matomo

We have just modified the software.cfg file after having already installed and deployed the service defined by the previous content. By default, SlapOS will assume that the contents of the file have not changed and slapos node software and slapos node instance will ignore the changes.

To force SlapOS to reinstall the service, run :

slapos node software --all

Once the command finishes without errors, run :

slapos node instance --all

until the command succeeds, several times if necessary.

There is no need to run slapos supply and slapos request this time as the entries are already in the database.

One could instead have defined a new file with a new name, e.g. "software-v2.cfg" instead of modifying "software.cfg", but in that case one would have to call slapos supply again with the path of "software-v2.cfg", then install everything again with slapos node software, then call slapos request with the new path but with the same instance name "matomo-1" (which would update the existing instance instead of requesting a new one), and then slapos node instance. This is in fact how services deployed in production are updated - pretty much. But it's much less convenient for developing a service that is constantly being modified.

To account for the change in the "apache-php" process, restart the processes with :

slapos node restart all

(in reality you only need to restart the process whose name contains "apache-php"; you can list the processes with  slapos node status).

Finalise the configuration of Matomo

Reload the page: the fields will be pre-filled automatically. You can now click on "next" and finalize the configuration of your Matomo service.

In step 5, you will need to create a super user account with a username and password of your choice, and a valid email.

Then in step 6 you can enter the first website you want to analyse with Matomo. Matomo will then provide you with a small Javascript script to integrate into the HTML of your site. Now it's your turn!

Automate your slaprunner

Your Theia is delivered with the automatic execution of slapos node software and slapos node instance in the background disabled. Now that Matomo is ready to use, you can enable this feature so that the service is regularly checked for proper operation, and the instance always automatically converges to the desired state. In particular, if a service process stops unexpectedly, it will be restarted automatically. This is the operating framework for a service deployed on a node in production.

Go back to your Theia service page in the Rapid.Space panel, and change the setting called "Automatically Run Software/Instance" to "running".

Access the monitoring of your instance

Your Matomo instance, in addition to defining its running promises, also deploys a dedicated monitoring web server that publishes the status and history of the promises. The web application https://monitor.app.officejs.com allows you to subscribe to such servers to view and organise this data. The set of servers you monitor is simply stored locally in the browser, so you do not need an account.

Follow the URL given by the "monitor-setup-url" connection parameter seen earlier (ctrl + click on the link in the terminal, or copy it into a new tab). This will open monitor.app.officejs.com and subscribe to Matomo monitoring. Thereafter, as long as your browser's local data is preserved, simply access monitor.app.officejs.com to retrieve the status and promise history of all monitored instances.

Here we find the promises of the "matomo-1" instance, as well as its two sub-instances "Mariadb" and "apache-php".

Become a resilient and sovereign SaaS operator

In this tutorial we have shown how to automate the first 6 facets ("build" to "monitor") of what could eventually become a Matomo SaaS (Software as a Service) capable of providing a sovereign alternative to Google Analytics. Thanks to Rapid.Space's edge computing, all you need is a small server behind a consumer fibre connection to get started. And for this service to be resilient, all you need to do is duplicate this modest infrastructure at a partner or friend's site, taking care to encrypt the data partition to prevent any leakage, which is what most Linux distributions offer.

To finalise such a SaaS, there are still 6 facets ("self-healing" to "testing") to be added to the buildout configuration. 5 of these 6 facets are already provided by the free SlapOS software in a standardised way and only require a configuration effort. The only facet that requires constant effort is "self-healing".

The life of an edge or cloud system is littered with incidents. All clouds, including the most famous ones, are down on average about 8 hours a year. Each breakdown is the occasion of a quality approach by analysis of the origin of the breakdown, as in a car factory, according to the method of the five whys. Why did Matomo crash? Because there was no more disk space. Why didn't the promise of predicting the end of disk space work? Because disk space growth accelerated due to a peak in traffic. Why isn't peak usage taken into account? Because it costs too much in resources. And so on.

It will then be necessary to define how to avoid such an incident in the future: by creating a resource usage limit, by charging more for a better storage space forecast, by automatically deleting the oldest data, by moving Matomo to another server, etc. The choice of method will depend as much on commercial criteria as on technical ones. Each incident thus taken into account by an automatism will make it possible to reduce the operating costs of the SaaS and thus to propose it to more users, thus leading to new incidents. Improving a SaaS is a continuous effort... and a sign of its success.

The fact that Rapid.Space's SlapOS technology is 100% open source has an advantage in this effort: incidents induced by the edge cloud environment do not need to be "worked around" as they can be directly resolved by modifying SlapOS. It's a bit like with operating systems: when Windows has a bug, you have to work around it or wait for it to be fixed, whereas with Linux, everyone is free to modify the kernel source code immediately. Moreover, in the cloud as in operating systems, a 100% free offer is also a guarantee that you will not be subject to a sudden price increase that you cannot oppose.