Setup: Getting Wendelin

Setup: Explained

• Public Accessable IPv6, IPv4 tutorial fallback
• Load Wendelin via ShaCache
• Port forwarding via Socat

Setup: Undressed

• "Naked" Machine with a KVM
• Can be hosted anywhere (OVH, AWS, ...) any architecture (ARM, Intel...)
• Public IPv4 and IPv6 (Res6t)

Setup: Tutorial Preparation

• We use Slapos to deploy a cluster of VMs for the tutorial
• Using Two machines, 20 partitions
• Each partition with KVM Virtual Machine

Setup: SlapOS "OUTSIDE" - Manage Access

• SlapOS manages partitions and installs virtual machine
• Port forwarding from Public IPv6 to KVM private IPv4
• SlapOS simplifies all access by always going through IPv6

Setup: SlapOS "INSIDE" - Manage Wendelin

• SlapOS is installed on KVM, too (used to manage cluster and VM)
• Shortcut: Wendelin download pre-compiled via ShaCache
• Shortcut: Socat to automatically bind, port forward KVM

Todo: Access Virtual Machine

• Browser only tutorial (no IPv6)
• Go to https://softinst67162.host.vifib.net/shell
• Don't click "Green Button" please.

Todo: Access Terminal

• Click the terminal tab
• Enter SSH/VNC provided (like SSH slapos@2001:67c:1254:e:1b::XXXX -p 10022 )

Todo: Authenticate and install Wendelin

• Authenticate
• Switch to root sudo -su , authenticate again, install via:
• wget http://deploy.nexedi.cn/wendelin-standalone && bash wendelin-standalone

Todo: Monitor Installaton process

• See SlapOS software installation log:
• tail -f /opt/slapos/log/slapos-node-software.log

Todo: Check Software Installaton Status

• Check status of Wendelin/ERP5 installation
• watch -n 30 erp5-show -s
• Once done it will return you an internal IPv4. Note this down

Todo: Socat Bind

• wget https://lab.nexedi.com/nexedi/wendelin/raw/master/utils/wendelin-standalone-bind.sh && bash wendelin-standalone-bind.sh
• This will activate urls provided to access Jupyter/ERP5, note them down for later use.

Todo: Test Bindings

• Check if socat works: ps aux | grep socat
• Check processes: ps xa | grep runzope
• Check who does what: grep node-id /srv/slapgrid/slappart*/etc/zope-0.conf

Todo: Configure Wendelin - Login

• Go to your Zope 2 Url (vm[x]30002-url )
• Login using zope/insecure
• My Favorites > Check Site Consistency

Todo: Fix Consistency

• Select All and Fix Consistencies

Todo: Configure Site

• Once done (!), continue
• Select Configure Site for Wendelin

Todo: Install Configuration

• Select Configuration for Wendelin
• Click "Install" on next screen

Todo: Wait

• Installation may take a few minutes
• Once down, click "Start using your ERP5 System"
• Login again with zope/insecure

Todo: HOTFIX: Add Business Template

• Goto My Favorites > Manage Business Templates
• HOTFIX are addable in live system via business templates
• Business Templates are standard way of adding functionality in ERP5/Wendelin

Todo: Download Business Template

• Click the Export Symbol (red/blue arrows)
• Select Download Business Template from URL

Todo: Enter Business Template URL

• Enter the HOTFIX Url provided and click to download
• http://www.wendelin.io/wendelin05_release_hotfix-0.5.bt5/Base_download
• Click download

Todo: Install Business Template

• Once download finishes, go to the business template page
• From actions, select Install Business Template

Todo: Confirm HOTFIX additions

• The dialog will show a list of objects to be added with the installation
• Confirm and install

Todo: Main Interface

• Go back to the main interface
• Your instance is now ready to use

Todo: Create Ingestion Policy

• We now setup Wendelin to receive data from fluentd
• Goto: http://softinstxxxx.host.vifib.net/erp5/portal_ingestion_policies/
• Security Setting to prevent arbitrary stream from being sent

Todo: Fast Input

• Hit the "Green Runner" for fast input
• Enter "pydata" as our reference name and save
• This creates a new ingestion policy and corresponding data stream

Todo: Check Data Streams

• Switch to Data Stream Module
• You will now have an empty stream, setup to ingest data from fluentd

Audio: Simulate IOT Sensor

Wendelin in Production

• First Wendelin prototype in production to monitor wind turbines.
• Wendelin used to collect data and manage wind parks.
• Machine Learning for failure prediction, structural health monitoring.

Wendelin in Production

• Turbine equipped with multiple sensors per blade, PC collecting sensor data
• PC includes fluentd for ingesting into Wendelin (network availability!)
• Tutorial: we use same setup, sensor - Fluentd - Wendelin

Basic FluentD

• FluentD is open source unified data collector
• Has a source and destination, generic, easy to extend
• Can handle data collection even under poor network conditions

Complex FluentD

• Setup of FluentD can be much more complex
• In Wendelin production, every turbine has its own FluentD
• Tutorial simulates the same, all VM have their own FluentD

Todo: Record Audio

• We use another SlapOS Webrunner hosting only a webpage:
• https://softinst56756.host.vifib.net/public/project/hyperconvergence/
• Click "Record" to Start/Stop audio and "Save" to save

Todo: Access Monitor using fluentD

• Access Monitor at: https://softinst67162.host.vifib.net/
• Go to the Editor, click "This Project"

Todo: Upload to Monitor

• We already made a folder for uploads for this tutorial
• Left click and select upload file

Todo: Forward File to fluentD

• In the Editor, open folders project > slapos
• Create a new file, give it your name
• Code on next slide (please copy&paste)

Todo: FluentD Configuration File (Gist)

    <source>
            @type bin
            format none
            path
            /srv/slapgrid/slappart9/srv/runner/PUT_YOUR_WAV_HERE/!!YOURNAME!!*.wav
            pos_file /srv/slapgrid/slappart9/srv/runner/!!YOURNAMEGOESHERE!!.pos
            enable_watch_timer false
            read_from_head true
            tag pydata
            </source>
            <match pydata>
            @type wendelin
            @id wendelin_out
            streamtool_uri
            http://!!URL_TO_YOUR_ZOPE!!/erp5/portal_ingestion_policies/pydata
            user zope
            password insecure
            buffer_type memory
            flush_interval 1s
            disable_retry_limit true
            </match>
          

Todo: Save and send from Terminal

• Switch to the terminal and enter:
• software/bd283b01bd665a31e41dce567f199491/bin/fluentd -c project/YOUR_NAME.cfg
• To pass your configuration including what file to send to Wendelin

Todo: Save and send from Terminal

• Head back to Wendelin/ERP5
• In the Data Stream Module, check the file size of the pydata stream
• It should now show a file size larger than 0

Analyse: Work with Ingested Data

Out-of-Core

• Wendelin.Core enables computation beyond limits of existing RAM
• We have integrated Wendelin and Wendelin.Core With Jupyter
• ERP5 Kernel (out-of-core compliant) vs. Python 2 Kernel (default)

Todo: Head to Jupyter (Notebook)

• Head to Juypter http://[x].pydata-class.erp5.cn
• Start a new ERP5 Notebook
• This will make sure you use the ERP5 Kernel

Todo: Learn ERP5 Kernel (Notebook)

• Note you have to connect to Wendelin/ERP5
• The reference you set will store your notebook in the Date Notebook Module
• Passing login/password will authenticate Juypter with Wendelin/ERP5

Todo: Getting Started (Notebook)

• Connect, set arbitrary reference and authenticate

Todo: Accessing Objects (Notebook)

• Import necessary libs
• Type context , this will give you the Wendelin/ERP5 Object
• Type context.data_stream_module["1"] to get your uploaded sound file

Todo: Accessing Data Itself (Notebook)

• Try to get the length of the file using getData and via iterate
• Note then when using ERP5 kernel all manipulations should be "Big Data Aware"
• Just loading a file via getData() works for small files, but will break with volume

Todo: Compute Fourier (Notebook)

• Proceed to fetch data using getData for now
• Extract one channel, save it back to Wendelin and compute FFT
• Note, that ERP5 kernel at this time doesn't support %matplotlib inline

Todo: Display Fourier (Notebook)

• Check the rendered Fourier graphs of your recorded sound file

Todo: Save Image (Notebook)

• Save the image back to Wendelin/ERP5.

Todo: Create BigFile Reader (Notebook)

• Add a new class BigFileReader
• Allows to pass out-of-core objects

Todo: Rerun using Big File Reader (Notebook)

• Rerun using the Big File Reader
• Now one more step is out of core compliant
• Verify graphs render the same

Todo: Redraw from Wendelin (Notebook)

• Redraw the plot directly from data stored in Wendelin/ERP5

Todo: Verify Images are Stored

• Head back to Wendelin/ERP5
• Go to Image module and verify your stored images are there.

Todo: Verify Data Arrays are Stored

• Switch to the Data Array module
• Verify all computed files are there.

Visualize: Display computed data

Running Web Sites from Wendelin

• Last step is to display results in a web app
• Head back to main section in Wendelin/ERP5
• Go to Website Module

WebSite Module

• Website Module contains websites
• Open renderjs_runner - ERP5 gadget interface

Renderjs Runner

• Parameters for website module
• see ERP5 Application Launcher - base gadget
• Open new tab: http://softinstxxxx/erp5/web_site_module/renderjs_runner/

Renderjs Web App

• ERP5 interface as responsive application
• We will now create an application like this to display our data

Todo: Clone Website

• Go back to renderjs_runner website
• Clone the website

Todo: Rename Website

• Change id to pydata_runner
• Change name to PyData Runner
• Save

Todo: Publish Website

• Select action Publish and publish the site
• This changes object state from embedded to published
• Try to access: http://softinstxxxx/erp5/web_site_module/pydata_runner/

Todo: Layout Properties

• Change to Tab "Layout Properties tab"
• Update the router to custom_gadget_erp5_router.html
• Refresh your app (disable cache), it will be broken, as this file doesn't exist

Todo: Web Page Module

• Change to web page module
• Search for reference "router"

Todo: Clone Web Pages

• Open both the html and javascript file in a new tab
• Clone both, prefix the references with custom_ and publish alive
• Click edit tab on the html page

Todo: Prefix JS file to display

• Prefix the Javascript file to load to the correct reference
• Save, switch to the Javascript file in the other tab
• Click edit tab here as well

Todo: Update Default Gadget

• Look for "worklist" and change it to "pydata"
• Save, we now have a new default gadget to load
• Go back to web page module

Todo: Clone Worklist gadgets

• Search for %worklist%, open both files in new tabs, clone, change title
• Replace "worklist" in references with "pydata", save and publish
• We will now edit both files to display our graph

Todo: Graph Gadget HTML (Gist)

    <html>
            <head>
            <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
            <meta name="viewport" content="width=device-width, user-scalable=no" />
            <title>PyData Graph</title>
            <script src="rsvp.js" type="text/javascript"></script>
            <script src="renderjs.js" type="text/javascript"></script>
            <script src="gadget_global.js" type="text/javascript"></script>
            <script src="gadget_erp5_page_pydata.js" type="text/javascript"></script>
            </head>
            <body>
            </body>
            </html>
          

Todo: Graph Gadget JS (Gist)

    /*global window, rJS, RSVP, URI */
            /*jslint nomen: true, indent: 2, maxerr: 3 */
            (function (window, rJS, RSVP, URI) {
            "use strict";
            rJS(window)
            // Init local properties
            .ready(function (g) {
            g.props = {};
            })
            // Assign the element to a variable
            .ready(function (g) {
            return g.getElement()
            .push(function (element) {
            g.props.element = element;
            });
            })
            // Acquired methods
            .declareAcquiredMethod("updateHeader", "updateHeader")
            // declared methods
            .declareMethod("render", function () {
            var gadget = this;
            return gadget.updateHeader({
            page_title: 'PyData'
            })
            });
            }(window, rJS, RSVP, URI));
          

Todo: Save, refresh web app

• Once you saved your files, go back to the web app and refresh
• You should now have a blank page with header set correctly
• We will now add fetch our graph and display it

Todo: Update Graph Gadget HTML (Gist)

    <!DOCTYPE html>
            <html>
            <head>
            <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
            <meta name="viewport" content="width=device-width, user-scalable=no" />
            <title>PyData Graph</title>
            <!-- renderjs -->
            <script src="rsvp.js" type="text/javascript"></script>
            <script src="renderjs.js" type="text/javascript"></script>
            <!-- custom script -->
            <script src="dygraph.js" type="text/javascript"></script>
            <script src="gadget_global.js" type="text/javascript"></script>
            <script src="gadget_erp5_page_pydata.js" type="text/javascript"></script>
            </head>
            <body>
            <div class="custom-grid-wrap">
            <div class="custom-grid ui-corner-all ui-body-inherit ui-shadow ui-corner-all"></div>
            </div>
            <div data-gadget-url="gadget_ndarray.html"
            data-gadget-scope="ndarray"
            data-gadget-sandbox="public">
            </div>
            </body>
            </html>
          

Todo: Graph Gadget JS (1) (Gist)

    /*global window, rJS, console, RSVP, Dygraph */
            /*jslint indent: 2, maxerr: 3 */
            (function (rJS) {
            "use strict";
            var ARRAY_VALUE_LENGTH = 8,
            OPTION_DICT = {
            start_date: 0,
            time_factor: 1000,
            resolution: 1,
            xlabel: 'x',
            ylabel: 'y',
            key_list: ["Channel 1", "Date"],
            label_list: ["Date", "Channel 1"],
            series_dict: {
            "Channel 1": {
            axis : "y",
            color: "#00884B",
            pointSize: 1,
            visible : true,
            connectSeparatedPoints: true
            }
            },
            axis_dict: {
            y: {position : "left", axisLabelColor: "grey", axisLabelWidth : 40, pixelsPerLabel : 30},
            x: {drawAxis : true, axisLabelWidth : 60, axisLabelColor: "grey", pixelsPerLabel : 30}
            },
            connectSeparatedPoints: true
            };
            ...
          

Todo: Graph Gadget JS (2) (Gist)

    function generateInitialGraphData(label_list) {
            var i,
            data = [[]];
            for (i = 0; i < label_list.length; i += 1) {
            data[0].push(0);
            }
            return data;
            }
            function convertDateColToDate(gadget, array) {
            var label_list = gadget.property_dict.option_dict.label_list,
            time_factor = gadget.property_dict.option_dict.time_factor,
            time_offset = gadget.property_dict.option_dict.time_offset || 0,
            i,
            k;
            for (k = 0; k < label_list.length; k += 1) {
            if (label_list[k] === "Date") {
            for (i = 0; i < array.length; i += 1) {
            array[i] = [i, array[i]];
            }
            }
            }
            return array;
            }
            ...
          

Todo: Graph Gadget JS (3) (Gist)

    rJS(window)
            .ready(function (gadget) {
            gadget.property_dict = {};
            return gadget.getElement()
            .push(function (element) {
            gadget.property_dict.element = element;
            gadget.property_dict.option_dict = OPTION_DICT;
            });
            })
            .declareAcquiredMethod("jio_getAttachment", "jio_getAttachment")
            // render gadget
            .declareMethod('render', function () {
            var gadget = this,
            interaction_model = Dygraph.Interaction.defaultModel,
            option_dict = {},
            url;
            url = "http://softinstxxxx/erp5/web_site_module/pydata_runner/hateoas/data_array_module/[your_id]";
            ...
          

Todo: Graph Gadget JS (4) (Gist)

    return new RSVP.Queue()
            .push(function () {
            return gadget.jio_getAttachment("erp5", url, {
            start : 0,
            format : "array_buffer"
            });
            })
            .push(function (buffer) {
            var array_length,
            length,
            array,
            array_width = 1;
            array_length = Math.floor(
            buffer.byteLength / array_width / ARRAY_VALUE_LENGTH
            );
            length = buffer.byteLength - (buffer.byteLength % ARRAY_VALUE_LENGTH);
            if (length === buffer.byteLength) {
            array = new Float64Array(buffer);
            } else {
            array = new Float64Array(buffer, 0, length);
            }
            return nj.ndarray(array, [array_length, array_width]);
            })
            ...
          

Todo: Graph Gadget JS (5) (Gist)

    .push(function (result) {
            var i,
            data = [],
            ndarray = result,
            label_list =  gadget.property_dict.option_dict.label_list,
            key_list = gadget.property_dict.option_dict.key_list;
            for (i = 1; i < label_list.length; i += 1) {
            data = data.concat(
            nj.unpack(
            ndarray.pick(
            null,
            key_list.indexOf(label_list[i])
            )
            )
            );
            }
            data = convertDateColToDate(gadget, data);
            gadget.property_dict.data = data;
            return gadget
            });
            })
            .declareService(function () {
            var gadget = this;
            return gadget.property_dict.graph = new Dygraph(
            gadget.property_dict.element,
            gadget.property_dict.data,
            gadget.property_dict.option_dict
            );
            });
            }(rJS));
          

Todo: Refresh Web Application

• Not out-of-core compliant, but jIO already allows to fetch ranges
• Example computes client-side as project requires to work offline "in the field"
• Ongoing process to fix libraries to work asynchronously and Big Data aware

Summary: Hyperconvergence?

• Generic Hardware, specialized software services
• No more files, only services for compute, storage, networking, virtualization
• Manage through common toolset, scale by adding nodes

Summary: What did we do?

• SlapOS manages everything as a Service
• Including Deployment of VM Cluster, hosting of Audio Recording website
• Everything is a process