This documents how different parts of ERP5 should be tested, as well as what
tools are available. For information on how to use tools, please take a look
at HowTo.
Table of Contents
Testing Frameworks¶
ERP5TypeTestCase¶
ERP5TypeTestCase is based on ZopeTestCase, and it provides an unit testing framework mainly.
ERP5TypeTestCase generates a portal site on the fly, and executes one or more test
suites on it, then discards the temporary site.
The advantage of using ERP5TypeTestCase is that you can obtain fine-grained control
of behaviours, because you write Python code in unrestricted environment. The
disadvantage is that the framework is often too low-level, if you want to test
a feature closer to the user.
SecurityTestCase¶
SecurityTestCase is a class extending ERP5TypeTestCase, and adding tools for
permissions and security checks.
Assertions such as:
assertUserCanAccessDocument
assertUserCanAddDocument
assertUserCanChangeLocalRoles
and many others are defined there to ease your work.
ERP5 Test Tool¶
ERP5 Test Tool is a tool available under a portal site, named portal_tests.
This is based on Zelenium, which, in turn, is based on
Selenium. You can
make as many as test suites you wish to have under portal_tests, and you can run
tests manually or semi-automatically by passing ?auto=true to portal_tests on
a web browser.
The advantage in ERP5 Test Tool is that you can interact with ERP5 in a visible
form, and it is relatively easy to write tests on Selenium. The disadvantage is
that Selenium is basically one-directional, thus you cannot put complex
expressions in it. Therefore, it is often necessary to write Python Scripts
separately, and call them from tests.
erp5mechanize¶
erp5mechanize is based on mechanize, and provides a tool to control a browser-like
object from Python.
Although writing tests with erp5mechanize is not as intuitive as with Selenium,
erp5mechanize is more flexible and less resource consuming. Thus erp5mechanize
is suitable to perform complex scenarios, simulate many users, and profile real
performance.
Simple Scripting¶
This is not very specific to ERP5, but it is also possible to simply run a batch
script from a command line, for example, by using wget or the urllib module
in Python.
Test Cases¶
Test Cases¶ are documents stored in Test Case Module, and managed
by the Business Template erp5_consulting. Each Test Case documents
how to test a business process with ERP5, in order to ensure that an implementation
follows project requirements.
The original purpose of Test Cases was to specify how ERP5 is configured for a
user, but this is considered a part of testing frameworks in ERP5.
How to Select a Testing Framework¶
Which testing framework should be used depends on what you want to test.
It is critical to select the most appropriate one in a long run.
Unit Testing¶
When you want to test a unit -- a small component of a system -- you should use
ERP5TypeTestCase. There is no alternative which gives you the same level of control.
Functional Testing¶
This depends on a meaning of functional testing. In ERP5, functional testing
means cross-units testing, sometimes known as integration testing. For this
purpose, you should use ERP5TypeTestCase. Although ERP5TypeTestCase mainly
targets at unit testing, there is no much difference between unit testing
and functional testing when writing tests in ERP5. Thus reusing the same
framework makes sense.
User Interface Testing¶
User interface testing is often assumed to be a part of functional testing, but
we distinguish them for clarity. When you want to test a feature that interacts
with the user directly, such as ListBox, it is an user interface testing.
For most UI testing, you should use ERP5 Test Tool, because this is the most
effective way to describe and check an UI. But if you need to describe a complex
scenario with which involves multiple actors on a module, you should use
erp5mechanize, because Selenium is not that powerful.
Performance Testing¶
Testing performance is a difficult issue, and there is no simple answer for this.
If you test the performance of use cases, you should use erp5mechanize, because
the other methods are not flexible enough or too much resource-sensitive.
If you test the performance of a small component of a system, you can use
ERP5TypeTestCase for this purpose.
If you test extremely simple cases, such as stressing a system with high loads,
simple scripting is the most efficient.
System Testing¶
Umigumi has a tool called
umitest which is suitable for system-level testing. This way, you can test if
ERP5 works well in a given installation of a whole operating system, but it is
likely that you need to combine other testing frameworks to perform precise
checks, especially ERP5 Test Tool.
Manual Testing¶
The last resort is manual testing, namely, testing a feature manually by human.
Unfortunately, not all tests can be completely automated or too hard to be
automated, thus using manual tests is inevitable from case to case. For example,
if you want to make sure that a page is rendered with conformance to a user's
preference in a given browser, it is nearly impossible to determine whether a
page is rendered appropriately in a mechanical way.
You should write Test Cases for manual testing. But note that this is the last
resort. Manual testing is not automated by computers, so it costs, and it won't
be performed regularly. Whenever possible, you should use other methods. Use
manual testing, only if you may not avoid it by any means.
When You Should Write Tests¶
In principle, whenever. Whenever that makes sense.
If you find a feature which is not tested, you should write one or more tests for it.
If you find a bug which is not tested, you must write a test before fixing it.
If you write a new feature, you should write one or more tests for it.
Also, do not forget to run existing tests when you modify anything, in order to
make sure that you would not break any existing feature. If you introduce
incompatibilities, you must update all tests which are affected.
Related Articles¶