Table of Contents

• Introduction
• Testing
  • Using Test Suites And Automated Tests
  • Test Suite Configuration
  • Selenium UI Tests
  • Live tests and production environment
• Testing infrastructure
  • Architecture
  • Nexedi ERP5 Test Suite Distribution
  • Nexedi ERP5 Task API
  • Test Nodes
  • Nexedi ERP5 synchronization
• Commits
  • Checklist
  • The Case of Business Templates

Introduction¶

This guideline summarizes conventions covering testing and committing code to the ERP5 codebase, but it should also be applied to publishing quality code in general.

Testing¶

ERP5 is developed in a test-driven approach. This means that before the first line of code is written, a failing test should be made. Then the code to make this test pass is created. Fast forward to complete ERP5 development and large scale projects and there will be the need to automate to ensure that any code submitted does not compromise the existing codebase. Having tests for is very important, because tests allows to save large chunks of develoment of time. They allows checking some parts of the code quickly, automate regression tests and doing critical refactoring of code. Without tests, it's impossible to reach good quality of code and services.

Using Test Suites And Automated Tests¶

A large ERP5 project uses "test suite" documents accessible via the Test Suite Module to run automated tests within ERP5 and output results to the Test Result Module. Inside an ERP5 repository:

/tests/__init__.py
/product/*
/bt5/*

tests/__init__.py defines a test suite with a list of tests to be executed in a project specific manner (ERP5 repo itself has a class "ERP5" for unit and a "PERF" class for performance tests.

Test Suite Configuration¶

Several parameters can be defined on a test suite level:

• Title: must be unique and is used to retrieve results in Test Result Module.
• Test Suite: name of suite, like "ERP5" or "PERF".
• Additional bt5 path list: List of bt5 paths which are not part of the softwae release of ERP5 (this means missing for example custom bt5s).
• Project Title: link to associated project.
• Distributor: Should be ERP5-Projects, used to select different clouds of testnodes.
• Priority: Number of cores required. Be reasonable. 15 cores for small projects makes no sense and pollutes available infrastructure.

Once a test suite is set you can add test suite repository objects for every repository relevant for this test (usually only one repo is needed) specifying the following parameters:

• Url: url of the repository, it might contain password for private repositories (for private repository, you could use nxdtestbot:opah9Fo9Vi username and password.
• Branch: which branch of this repository should be used.
• Buildout section id: what is the section name of the software to override (the section in the software getting this repository).
• Profile Path: path, in first given repository, to a buildout configuration file describing the environment to test.

The automated test suite will checkout every repository in vcs_repository_list, and as soon as there is a commit in one of them, the list of revision of all these repositories will be given to Nexedi ERP5. Then Nexedi ERP5 can check if there are other automated test suites running on other computers and will either create a new test result or ask the current node to synchronize.

Once the testnode knows that it must run tests, it first run slapos commands in order to update/create software and to create new instance (so we have software and instance inside erp5testnode partition). Once the software is built and the instance is created, the command runTestSuite is launched.

In order to define where email notification will be sent, please set the field "Stakeholder" on a project. If you are using a mailing list, you can create an organisation representing the mailing list (which is a group of person).

Selenium UI Tests¶

Running UI tests happens with help of firefox which is run inside a virtual X screen using Xvfb. As these test simulate user interaction with system it is required to make sure that always node-quantity=1. If developer wants to see how firefox is working he can use following command on the machine running tests (or use ssh tunneling if appropriate):

x11vnc -display :123
vncviewer localhost:5900
If tests are on remote machine he can use ssh tunnel as follows:

ssh -t -L 5900:localhost:5900 user@machine_ip-474 'x11vnc -localhost -display :123'
vncviewer localhost:5900

Live tests and production environment¶

Live tests are really good candidates to check if a production server is running correctly. Some advices above can help, but there is some risk. Like it can be possible to show to users data entered by tests. Some messages could be wrongly sent to external software (like bank money transfer !). Some data entered by users could be altered.

Here we will publish more informations once we have more experience with it.

Testing¶ infrastructure

Architecture¶

Unit test are run by launching one or several nodes, each launching one or more parallel tests. All nodes are synchronized through Nexedi ERP5 which is in charge of distributing the work and to make sure every node is testing same revision.

Each node is installed thanks to Slapos tools, soon it will be even managed through Slapos Master (Vifib). Code is handled by git, and nodes are in charge of checking if there is new commit, and if there is new commit they can ask Nexedi ERP5 if there is some work to do.

The trick is that some projects might depends on several repositories. You can check below how Nexedi ERP5 is doing synchronization.

Nexedi ERP5 Test Suite Distribution¶

Test Nodes doesn't know in advance on which project they are going to work on. Therefore every test node is defined with the url of a distributor. The test node will call "startTestSuite" on the distributor and it will get all needed parameters to work on one or many projects.

The first time a test node calls startTestSuite, Nexedi ERP5 is going to look if this test node already exists. If not, then it will be created under test node module.

From time to time, an alarm (task_distributor_alarm_optimize) looks at all defined test suites and available test nodes and distribute the work to do. This alarm avoid moving test suite from a test node to another uselessly. In the same time, this alarm is checking for all test node that seems dead (10 hours without sending message) and invalidate them. Like this test suite allocated to a dead test node will be moved to another test node automatically.

Nexedi ERP5 Task API¶

Nexedi ERP5 provide an API to start, stop unit test. This API also allows to report failures, results (tests passed or failed), status and allows to know if there still ongoing work. The API is available here.

Any tool test running reporting results on Nexedi ERP5 must use this API. Soon a library will exist to allow using it easily. This API allow to handle from simple cases (like one unit test at a time on a single server) to complex cases (an unit test running on several servers, with different number of cores on each server). For now a working example using this library is available in runTestSuite and ERP5TypeTestSuite.

Test Nodes¶

Test nodes are created on vifib.net using nexed_development_service account. Their configuration is quite trivial, please see existing examples if you wish to create a new test node. A typical configuration is:

<?xml version='1.0' encoding='utf-8'?>
<instance>
  <parameter id="test-node-title">COMP533-3Nodes-ERP5PROJECT1</parameter>
  <parameter id="node-quantity">3</parameter>
  <parameter id="test-suite-master-url">https://webbuildbot:a2fgt5e1@www.tiolive.com/nexedi/portal_task_distribution/erp5_project</parameter>
</instance>

Parameters:

• test-node-title: title of the test node, must be unique and should give idea of the computer used.
• node-quantity: how many parallel test to run
• test-suite-master-url: url of the distributor, make sure to select the right one depending on wich "cloud" you want your test node to belongs to (erp5_project, erp5_performance, erp5_scalability ?)

For a 4 core server (8 with hyperthreading), it is good to create only 2 test nodes with a node-quantity of 3. Like this at most 6 threads are used. Also having only 2 test nodes is important, if we use 6 testnodes of 1 core, then we need to install much more software than if we have only 2 nodes. So to save disk space, it is better to not have too much test nodes.

Nexedi ERP5 synchronization¶

Let's say we have 2 testing nodes daemon (A and B) running on two different computers. Each daemon is doing checkout or update of repositories. Since A and B can run with several minutes or even more of interval, Nexedi ERP5 needs to make sure that both A and B are testing same revision. Therefore testnode A (assuming no test is already started) will do:

• Checkout/update repository 1, get revision x, checkout/update repository 2, get revision y.
• Ask Nexedi ERP5 to start unit test for a particular test_suite title with following pair of repository and revisions: [('repository 1',x), ('repository 2',y)].
• And then Nexedi ERP5 check if there is already running test, if not create new one and then returns pair to tests. So here we assume there was not test, so it returns [('repository 1',x), ('repository 2',y)].
• Then runTestSuite is launched for [('repository 1',x), ('repository 2',y)]

And then testnode B will do (running a little bit after testnode A):

• checkout/update repository 1, get revision X, checkout/update repository 2, get revision Y.
• Ask Nexedi ERP5 to start unit test for a particular test_suite title with following pair of repository and revisions: [('repository 1',X), ('repository 2',Y)].
• And then Nexedi ERP5 check if there is already running test, see there is one, so returns previous commit x, and y;[('repository 1',x), ('repository 2',y)].
• Then runTestSuite reset git repositories and launch test for [('repository 1',x), ('repository 2',y)].

Like this we are sure that all computers running the same test suite will be synchronized.

Commits¶

Checklist¶

The following applies only to the commit itself. It is expected that, changes to be committed have been tested, which means

• You checked that system is usable after your change (instance is able to start, rendering is working, etc).
• You used specific tools to validate files (for python minimum is to run pyflakes/pylint EVEN after fixing a typo)
• You read your patch before committing.
• You admit being responsible of the code you commit and its effects.
• You never commit any pdb-related code (import, call to set_trace, ...).
• You follow the rules stated in Subversion Community Guide.

The Case of Business Templates¶

Since Business Templates are issued from automatically-generated files, the developer's control on their content is limited. Things like indentation changes happen independently from developer will. Some of the following rules can't be strictly applied to Business Templates. Still, the developer must stick to them as much as possible, for example by committing separately his changes and file format changes (indentation, ...).