Page Content

Overview

In this tutorial we will learn how to:
- Install and configure Wendelin on a KVM.
- Ingest data into Wendelin using Fluentd. (open source data collector used to ingest data to Wendelin) Fluentd Website.
- Manipulate data in Wendelin using Jupyter Notebook. Jupyter Website.
- Present data in Wendelin on a web site using Renderjs and jIO javascript libraries.

1. Getting Wendelin on your KVM

Request a KVM

For this tutorial, you will use services provided by vifib.com.
On Vifib request a KVM:
Go to services.
Click on Add button.
Choose KVM software and the latest version

More info: How to instantiate a KVM.

Request a KVM

Choose a name for your KVM and enter parameters:
- 4 GB of RAM
- at least 25 GB of disk space.
- list of NAT rules: 22, 80, 433, 20000, 20001, 20002, 20003 (Ports from 20000 to 20003 will be used for Wendelin services.)
Click Proceed.

More info: How to instantiate a KVM.

Request a KVM

Wait for monitoring status to turn green and connection parameters to appear.
Click on url parameter and install debian.
From software to install choose only SSH server!

More info: How to instantiate a KVM.

Access your KVM from terminal

Under connection parameters of your KVM look for IPv6.
Go to terminal. (Can be anywhere, you just need IPv6 access.)
From terminal access your KVM on port 10022 via ssh:
ssh username@IPv6 -p 10022

Authenticate and install Wendelin

Authenticate
Switch to root with: sudo -su , authenticate again, install with:
wget http://deploy.nexedi.cn/wendelin-standalone && bash wendelin-standalone
This is one line code used to install Wendelin.

Monitor Installaton process

Wendelin-ERP5 Monitor SlapOS Software Log

First time, you run the installer, you will probably get an error with a message: "Your software is still building, be patient it can take a while." Do not panic. This is expected. Installation is now running.
You can check that installation is going ok by following SlapOS software installation log:
tail -f /opt/slapos/log/slapos-node-software.log

Be patient. Installation can take up to several hours to complete.

Check Software Installaton Status

Wendelin-ERP5 Check Software Installation Status

Check status of Wendelin Installation With command erp5-show -s
Once done, it will return you an internal IPv4, username and password. Note them down.
Run bash wendelin-standalone again to make sure there are no errors.

Socat Bind

Next step is to bind the services to the correct ports of the KVM. Execute following commands:
apt-get install net-tools
wget https://lab.nexedi.com/nexedi/wendelin/raw/master/utils/wendelin-standalone-bind.sh && bash wendelin-standalone-bind.sh
This will activate urls provided to access Jupyter/ERP5.
Check if socat works: ps aux | grep socat

Request frontend

Same way like you requested KVM, now request Frontend on Vifib with parameters Backend URL: http://[!YOUR_IPv6!]:30001 and Backend Type: Zope.
Wait for connection parameters to appear and click on the URL parameter

Login

Click on Zope management interface
Login using username and password
go to: http://softinstxxxx.host.vifib.net/erp5

Configure Site

Go to: My Favorites > Configure your Site .

Your ERP5 instance is now ready, but it has only very generic core library. This core can be specialized through different business templates for different purposes.
This can be done manually, installing individual business templates that provide your ERP5 instance with utilities that you need.
That can be done in My Favorites > Manage Business Templates
But Wendelin uses a lot of different business templates and that would be a lot of work, so configurator is provided that makes that work for you and installs just the business templates that are important for Wendelin.

Install Configuration

Select Wendelin Configuration.

IF there is no Wendelin Configurator, you have to install it:
- Go to My Favourites > Manage Business Templates .
- Click on Import/Export button (blue and red arrows).
- Select Exchange > Install Business Templates from Repositories .
- Check box before erp5_wendelin_configurator business template and click Install Business Templates from Repositories.
Configuration will take some minutes.

Install Configuration

On next screen make sure that the boxes set wendelin tutorial and set up Data Notebook module for Jupyter integration are checked.
Click Set of data Notebook module and on the next screen Install .

Wait

Installation may take a few minutes
Once down, click "Start using your ERP5 System"
Login again.

Main Interface

Your instance is now ready to use

The start screen shows a list of modules (data-types) directly accessible.
You can also access them through Modules selection tab at the top of the screen.
Probably you noticed that after configuration there are a lot more of them.
Modules can be contain anything from Persons, Organizations to Data Streams.
Modules prefixed with Portal are not displayed (e.g. portal ingestion policies)

2. Simulate Sensor and ingest data via Fluentd

Create Ingestion Policy

Now we will setup Wendelin to receive data from fluentd.
Goto: http://softinstxxxx.host.vifib.net/erp5/portal_ingestion_policies/
Ingestion policy is security Setting to prevent arbitrary stream from being sent

Currently fluentd and Wendelin are setup to receive streams of data
A data stream is a file, created by ingestion policy, to which data is continually appended

Fast Input

Hit the "Green Runner" to create new Ingestion Policy
Enter "pydata" as reference name and "Pydata" as title and click Create Ingestion Policy .
This creates a new ingestion policy.

The ingestion policy includes default scripts to be called on incoming data.
If you want to modify data handling, you could now write your own script.

Wendelin in Production

First Wendelin prototype in production was used to monitor wind turbines.
Wendelin is used to collect data and manage wind parks.
Allowed to use Machine Learning for failure prediction, structural health monitoring.

Photo: Fotolia.fr - David Hense

Wendelin in Production

Each wind turbine equipped with multiple sensors per blade and PC collecting sensor data
Each PC including fluentd for ingesting into Wendelin (network availability!)

Basic FluentD

FluentD is open source unified data collector
Has a source and destination, generic, easy to extend
Can handle data collection even under poor network conditions
More info: FluentD Website

Complex FluentD

Setup of FluentD can be much more complex.
In Wendelin production, every turbine has its own FluentD.

Record Audio

For demostration of data ingestion we will use simple audio .vaw file.
You can record it on your own, use another SlapOS Webrunner hosting only a webpage:
https://softinst56756.host.vifib.net/public/project/hyperconvergence/
Click "Record" to Start/Stop audio and "Save" to save
Or you can download sample.wav

Webrunner using fluentD

For this part of tutorial you will need a webrunner using Fluentd:
- Request a new service on vifib, choose webrunner, and wait for connection parameters to appear.
- Click on the URL, and enter username and password provided.
- Click Open Softwate release, choose slapos > software > fluentd and click Open Software.
- Click green arrow and wait for installation to finnish.

Upload to Monitor

Wendelin-ERP5 Monitor Webrunner File Upload

Go to the Editor, click "This Project"
Make a folder for this tutorial
Left click and select upload file
Upload your .vaw file.

Forward File to fluentD

Wendelin-ERP5 Webrunner FluentD Configuration

In the Editor, open folder that you created.
Create a new file, name it YOUR_NAME.cfg
Code on next slide (please copy&paste)

We are now creating a configuration file to pass to fluentd
The file contains all parameters for fluentD regarding data source and destination
Normally this is set upfront, but for the tutorial we hardcode

FluentD Configuration File (Gist)

<source>
    @type bin
    format none
    path /srv/slapgrid/slappart!!X!!/srv/runner/PUT_YOUR_WAV_HERE/!!YOURNAME!!*.wav
    pos_file /srv/slapgrid/slappart!!X!!/srv/runner/!!YOURNAME!!.pos
    enable_watch_timer false
    read_from_head true
    tag pydata
</source>
<match pydata>
    @type wendelin
    @id wendelin_out
    streamtool_uri http://!!URL_TO_YOUR_ZOPE!!/erp5/portal_ingestion_policies/pydata
    user zope
    password !!insecure!!
    buffer_type memory
    flush_interval 1s
    disable_retry_limit true
</match>

Copy this script to new file that you created and edit paths to the vaw file and to the pos file, uri to your wendelin and username and password.
Fluentd configuration file has two parts.
Source part contains path to file that you wish to upload, path to pos file which is used to track position while sending the data, and a tag, which is used to match source to the output
Output part is a match tag. It must match a tag given to the data in the source part. It contains uri to your ingestion policy, and username and password for your Wendelin
Make sure all the paths are correct.

For more information on writing Fluentd configuration files check out: Fluentd configuration documentation.

Save and send from Terminal

Wendelin-ERP5 Webrunner FluentD File Transfer

Switch to the terminal and run fluentd with:
software/YOUR SOFT NUMBER/bin/fluentd -c path/to/YOUR_NAME.cfg
To run fluents with your configuration including what file to send to Wendelin.
This will create new Data Stream in your Wendelin instance, and send data to it.

Notice that Fluentd is running until you interrupt it and waiting for new input.
In our case, we had just a small .vaw file, but we could also have continuous stream of data from the sensor and fluentd would be continuously appending it to Data Stream.

Check created Data Stream

Wendelin-ERP5 - Verify File was received

Head back to Wendelin/ERP5.
In the Data Stream Module, check the file size of the pydata stream.
It should show a file size larger than 0.

3. Work with Ingested Data

Out-of-Core

Wendelin.Core enables computation beyond limits of existing RAM
We have integrated Wendelin and Wendelin.Core With Jupyter
In Jupyter we can use ERP5 Kernel (out-of-core compliant) vs. Python 2 Kernel (default Jupyter)

Enable Data Notebook

Head to My Favourites > Preferences.
Click Default System Preferences and open Data Notebook tag.
Check the box to enable Data Notebook and click the save icon.

Head to Jupyter

Request a frontend with Backand URL: https://[IPv6]:30000 and Backend Type: notebook. Wait for connection parameters and click on secure_access_url.
For password check the file knowledge0.cfg on a partition, where Jupyter is installed in your KVM (first use: slapos node | grep jupyter to check partition, and then: cat /srv/slapgrid/slappart__/knowledge0.cfg)
Go to: https://softinstxxxxxx.host.vifib.net/tree
Start a new ERP5 Notebook. This will make sure you use the ERP5 Kernel .

If you have IPv6 access, you can also access URL https://[IPv6]:30000 directly in your browser.
The Python 2 Kernel is the default Jupyter Kernel
Using Python 2 will disregard Wendelin and Wendelin.Core, so it's basic Jupyter.
Using ERP5 Kernel will use Wendelin.core in the background.
To make good use of it, all code written should be Out-of-core "compatible"
For example you should not just load a large file into memory (see below).

Learn ERP5 Kernel

Help yourself with Notebook

Note that your ERP5_URL in this case should be your internal url
You can retrieve it by running erp5-show -s in your KVM
Note, outside of the tutorial we would set the external IPv6 adress of ZOPE

Getting Started

Authenticate and set arbitrary reference for your notebook.

This is always the first step when you start a new Notebook with ERP5 Kernel.
It makes sure that you are connected to ERP5/Wendelin instance any you can work with objects on it.
It also creates Data Notebook object on your ERP5/Wendelin instance.
You can go to Data Notebook Module and see that your Data Notebook object is now saved there.

Accessing Objects

Type context, this will give you the Wendelin/ERP5 Object.
Type context.data_stream_module["1"] to get your uploaded sound file.

Accessing data works the same ways throughout [IPv6]:30002/erp5/[module_name]/[id].
All modules you see on the Wendelin/ERP5 start page can be accessed like this.
Once you have an object you can manipulate it.
Note that accessing a file by internal id (1) is only one way.
The standard way would be using the reference of the respective object, which will also allow to user portal_catalog to query.

Import libraries

Import necessary libs for later.

Accessing Data Itself

Try to get the length of the file using getData and via iterate
Note then when using ERP5 kernel all manipulations should be "Big Data Aware"
Just loading a file via getData() works for small files, but will break with volume

It's important to understand that manipulations outside of Wendelin.Core need to be Big Data "compatible"
Internally Wendelin.Core will run all manipulations "context-aware"
An alternative way to work would be to create your scripts inside Wendelin/ERP5 and call them from Juypter
Scripts/Manipulations are stored in Data Operations Module

Compute Fourier

Proceed to fetch data using getData for now
Extract one channel, save it back to Wendelin and compute FFT

We can call methods from Wendelin/ERP5.
Wendelin/ERP5 has a system of method acquistion. Every module can come with its own module specific methods and method names are always context specific ([object_name]_[method_name] ). Base methods on the other hand are core methods of Wendelin/ERP5 and applicable to more than one object.

Display Fourier

Check the rendered Fourier graphs of your recorded sound file

Save Image

Save the image back to Wendelin/ERP5.
Close figure with plt.close(), otherwise it will show on all outputs.

Create BigFile Reader

Add a new class BigFileReader
Allows to pass out-of-core objects

Rerun using Big File Reader

Rerun using the Big File Reader
Now one more step is out of core compliant
Verify graphs render the same

We are now showing how to step by step convert our code to being Out-of-Core compatible
This will only be possible for code we write ourselves
Whenever we have to rely on 3rd party libraries, there is no guarantee that data will be handled in the correct way. The only option to be truly Out-of-Core is to either make sure the 3rd party methods used are compatible and fixing them accordingly/committing back or to reimplement a 3rd party library completely.

Redraw from Wendelin

This is the way to redraw the plot directly from data stored in Wendelin/ERP5
Imidiatelly after you create content, you cannot redraw it. You must wait for the object to be catalogued.

Verify Images are Stored

Head back to Wendelin/ERP5
Go to Image module and verify your stored images are there.

Verify Data Arrays are Stored

Switch to the Data Array module
Verify all computed files are there.

4. Visualize, Display computed data

Running Web Sites from Wendelin

Last step is to display results in a web app
Head back to main section in Wendelin/ERP5
Go to Website Module

One of the modules in erp5 is Web Site Module.
We will use it to create simple Web Site for presentation of our result.

WebSite Module

Website Module contains websites
Open renderjs_runner - ERP5 gadget interface

Front end components are written with two frameworks, jIO and renderJS
jIO (Gitlab) is used to access documents across different storages
Storages include: Wendelin, ERP5, Dropbox, webDav, AWS, ...
jIO includes querying, offline support, synchronization
renderJS (Gitlab) allows to build apps from reusable components
Both jIO/renderJS are asynchronous using promises

Renderjs Runner

Parameters for website module
see ERP5 Application Launcher - base gadget
Open new tab: http://softinstxxxx/erp5/web_site_module/renderjs_runner/
It is important not to forget / at the end of the url, otherwise it will not work

Apps from gadgets are built as a tree structure, the application launcher is the top gadget
All other gadgets are child gadgets of this one
RenderJS allows to publish/aquire methods from other gadget to keep functionality encapsulated

Renderjs Web App

ERP5 interface as responsive application
We will now create an application like this to display our data

Clone Website

Go back to renderjs_runner website
Clone the website

Rename Website

Change id to pydata_runner
Change name to PyData Runner
Save

Publish Website

Select action Publish and publish the site
This changes object state from embedded to published
Try to access: http://softinstxxxx/erp5/web_site_module/pydata_runner/

Every object in ERP5 has a state (For example draft, published,...).
Workflows are used to change the state of objects.
A workflow in this case is to publish a webpage, which means changing its status from Embedded to Published.
Workflows (among other properties) can be security restricted. For example, everybody can see Web Site in published state, but only its creator can see it while it is still in draft state.
This concept applies to all documents in ERP5.

Layout Properties

Change to Tab "Layout Properties tab"
Update front page gadget to pydata
Refresh your app (disable cache), it will be broken, as pydata page gadget doesn't exist

One advantage working with an aync promise-chain based framework like renderJS is the ability to capture errors
It is possible to capture errors on client side, send report to ERP5 (stack-trace, browser) and not fail the app
Much more fine-grainded control, we currently just dump to screen/console

Web Page Module

Change to web page module
Search for reference %worklist%

The web page module includes html, js and css files used to build the frontend UI
The usual way of working with static files is to clone a file, rename its reference and publish it alive (still editable)

Clone Worklist gadgets

Open both files in new tabs, clone, change title.
Replace "worklist" in references and titles with "pydata", save and publish alive.
When published alive, object are still editable.
We will now edit both files to display our graph.

Pydata Gadget HTML

Go to edit tab on html gadget.
Copy and paste this script to the contents of the gadget.

This is a default gadget setup with some HTML.
Gadgets should be self containable so they always include all dependencies
RenderJS is using a custom version of RSVP for promises (we can cancel promises)
The global gadget includes promisified event binding (single, infinite event listener)
We are using RenderJS and jIO javascript libraries.
RenderJS Website
jIO Website
This is a default gadget setup with some HTML.
Gadgets should be self containable so they always include all dependencies
RenderJS is using a custom version of RSVP for promises (we can cancel promises)
The global gadget includes promisified event binding (single, infinite event listener)

Pydata Gadget JS

Same with javascript gadget.
Copy and paste this script to the javascript gadget (in the edit tab).

Save, refresh web app

Once you saved your files, go back to the web app and refresh
You should now have a blank page with header set correctly
This are just default template gadgets.
We will now update our gadgets to fetch our graph and display it

Update Pydata Gadget HTML

Update html gadget with this script.

Took from existing project, HTML was created to fit a responsive grid of graphs
Added JS library for multidimensional arrays: NDArray
Added JS libarary for displaying graphs: Dygraph

Pydata Gadget JS (1)

Update js gadget with this script (screenshots on this and next slides).

First we only defined options for the Dygraph plugin
In production system these are either set as defaults or stored along with respective data

Pydata Gadget JS (2)

Add methods outside of the promise chain
Simplified (removed actual creation of date objects)

Pydata Gadget JS (3)

Edit url variable for your instance and for id of your spectrum2 Data Array!

"ready" triggered once gadget is loaded
define gadget specific parameters
"render" called by parent gadget or automatically
we hardcode url parameter, by default it would be URL based

Pydata Gadget JS (4)

Orchestrated process starting with a cancellable promise queue
First step requesting the full file (NOT OUT-OF-CORE compliant - we load the whole file)
Return file converted into ndarray
Convert data into graph compatible format, store onto gadget
"declareService" triggered once UI is built
Graph will be rendered there.

Refresh Web Application

Example computes client-side as project requires to work offline "in the field"

Summary: What did we do?

We installed Wendelin on our KVM and configured it.
We ingested data with Fluentd.
We work with data using Jupyter Notebook.
We presented data in Wendelin using RenderJS and jIO javascript libraries.

Wendelin Tutorial - Installing and Using Wendelin on KVM

Overview

1. Getting Wendelin on your KVM

Request a KVM

Request a KVM

Request a KVM

Access your KVM from terminal

Authenticate and install Wendelin

Monitor Installaton process

Check Software Installaton Status

Socat Bind

Request frontend

Login

Configure Site

Install Configuration

Install Configuration

Wait

Main Interface

2. Simulate Sensor and ingest data via Fluentd

Create Ingestion Policy

Fast Input

Wendelin in Production

Wendelin in Production

Basic FluentD

Complex FluentD

Record Audio

Webrunner using fluentD

Upload to Monitor

Forward File to fluentD

FluentD Configuration File (Gist)

Save and send from Terminal

Check created Data Stream

3. Work with Ingested Data

Out-of-Core

Enable Data Notebook

Head to Jupyter

Learn ERP5 Kernel

Getting Started

Accessing Objects

Import libraries

Accessing Data Itself

Compute Fourier

Display Fourier

Save Image

Create BigFile Reader

Rerun using Big File Reader

Redraw from Wendelin

Verify Images are Stored

Verify Data Arrays are Stored

4. Visualize, Display computed data

Running Web Sites from Wendelin

WebSite Module

Renderjs Runner

Renderjs Web App

Clone Website

Rename Website

Publish Website

Layout Properties

Web Page Module

Clone Worklist gadgets

Pydata Gadget HTML

Pydata Gadget JS

Save, refresh web app

Update Pydata Gadget HTML

Pydata Gadget JS (1)

Pydata Gadget JS (2)

Pydata Gadget JS (3)

Pydata Gadget JS (4)

Refresh Web Application

Summary: What did we do?