Skip to content

Building a Custom Insight Service

Overview

ExoSense supports a way to plug-in custom functions into the asset signal pipeline. These functions can be used for transforming data, streaming analytics, rules, and actions. This plug-in type service is referred to as Insights or Insight Modules. The Insight Module itself is a service that exposes the defined interfaces ExoSense requires and includes one or many functions

High level overview of an Insight transform function

An Insight Module may include many functions of type transform and/or rule, the available functions are dynamically loaded when users go to add a new transform (shows available transform functions) or rule (shows available rule functions).

High level overview of an Insight rule function

End users are presented with a set of transform and rule functions in the Asset Configuration window, this information is loaded dynamically when the user goes to add a new transform or rule.

Example showing a custom function configuration

Exosite's Insight Interface Concepts

The Insight functions are Streaming -- they are inserted into the data pipeline flow and process every input signals' value.

Insight functions are of type transform, rule, or action. Transforms output another signal, Rules output a Status, and Actions process rule statuses.

At their core, Insights are not expected to be stateful. Examples include Exosite's off the shelf transformations like Join and Linear Gain, they operate on the latest value of Signal data. For a given Join on Signal A and Signal B, each time a piece of data comes in from either of those Signals, the Join runs (and returns a joined value). The Join Transform uses the last seen value of any Signal not present in the data sent to the Join at any given time.

ExoSense supports two tpyes of Insight modules to create your oww functions. Either path will allow your end-users to insert these functions into their Asset configuration based on the definitions for inputs, constants, and outputs.

Type Description
Inline Insights Functions run directly in the ExoSense pipeline as native functions. This makes it easy to add and build within ExoSense. As these are natively supported, they are limited in using the supported language expressions and logic.
External insights Are a specific type of service exposed through Exosite's Marketplace for use with ExoSense. Operate as a self contained externally hosted software application that exposes HTTP endpoints for ExoSense to interace with. Provides the most flexibility for hosting, environement, processing, and languages

Advanced

The ability to create, host, and use custom insights is available at specific Exosite business account and ExoSense application tiers.

To learn more about creating and using Inline Insights, please use ExoSense Inline Insights

The rest of this document focuses on External Insights but many of the topics are related and applicable.

External Insights

An external insight module is a stand-alone software service that runs as a webservice, supporting the specified HTTP endpoints as defined by the Exosite Insight Schema.

To summarize, this webservice implements functions for:

  • Getting high-level information about the Insight
  • Getting a list of available functions, types, and input/output parameters which allows ExoSense to dynamically load the user interface
  • Getting info about a specific function, along with its required parameters
  • Process incoming Signal data (i.e. "run the function")

Hosting

The custom Insight services can be hosted anywhere that public HTTPS endpoints are available, including:

  • Exosite's Murano Application Platform (A template is provided)
  • Amazon Web Services
  • Microsoft Azure
  • Google Cloud Products
  • Heroku

Please contact Exosite support for more information on this advanced topic. Templates are available in Marketplace and through support.

The code behind the functions in Insight service is entirely up to its creator as long as it conforms to the Swagger definition and Insight Schema. The code that makes up an Insight can be written in any language available within the chosen host.

Getting Started

The typical steps to create an external insight are as follows:

  1. Decide on where your funciton code will run and HTTPS webservice endpoints will be served.
  2. Implement the webservice required functions, per the Insight Schema.
  3. Publish a new ExoSense Insight Service in Exosite's Marketplace for your business account, using the Swagger template found below.
  4. Add your Insight service from the IoT Marketplace to your ExoSense instance to test.
  5. Add and update your functions.
  6. Note that the Insight service in the marketplace does not need updating unless the hosting location has changed. ExoSense pulls in the list of functions, function types, and information about the funcitons dynamically.

Using Murano to Host your Insight Service

You can build and host your custom Insight Module service as a Murano application. Note: You must have an advanced platform business account to build custom Murano applications.

The following template can be deployed from the Exosite IoT Marketplace. Lua-based example built to run in Murano

Look for ExoSense® Insight Module Template under the Murano Applications category.

Contact support for templates to run in other infrastructure.

Publishing

An Insight must be published in Marketplace (private or public) before it can be used within ExoSense

An External Insight service is published as a specific type of service (ExoSense Insight Servce) in the Exosite IoT Marketplace using Murano's OpenAPI Swagger support.

Hint

When using the Murano Insight Module Template, the swagger YAML defintion is automatically published to the endpoint /interface. For example https://mycompanyuniqueinsights.apps.exosite.io/interface

If you host outside of Murano, you need to have your swagger yaml file available for publishing the Insight Service.

Learn more about Publishing to Exchange IoT Marketplace

ExoSense Insight Swagger Notes

  • Very little needs to be changed from the template swagger, as the function end points are already defined.
  • It must use the element tag in the swagger definition as shown below for ExoSense to identify it as a service it can consume

    tags:
      - name: insight
        description: Insight Module
    
  • To add additional Solution level configuration parameters such as API keys and tokens to be used by the service, use the x-exosite-config-parameters key in the swagger file, info section.

    x-exosite-config-parameters: # optional exosite service configuration parameters
      - name:         auth_token
        description:  Describe this exosite service level configuratin parameter, like auth token
        type:         string
        format:       password
        required:     false
    
Example Insight Swagger Template

Download this example swagger template file

swagger: "2.0"

info:
  version: "1.0"
  title: My Nifty Insight
  description: |
    This describes an Insight Module that can be added to Murano Exchange.  An Insight Module can
    implement one or more Insight Functions.  These Insight Functions can then be added to Assets
    as transform, rule, and action functions.
  contact:
    name: Your Name Here
    email: YOU@BIZ.com
  x-exosite-config-parameters: # optional exosite service configuration parameters
    - name:         auth_token
      description:  Describe this exosite service level configuratin parameter, like auth token
      type:         string
      format:       password
      required:     false
host: a.real.host # Set this to the host your function is on
basePath: / # Set this or the path according to your function
################################################################################
# Do not edit below
################################################################################
tags:
  - name: insight
    description: Insight Module
schemes:
  - https  # Only https is supported.
consumes:
  - application/json # Only JSON is supported.
produces:
  - application/json # Only JSON is supported.

################################################################################
#                                Operations                                    #
################################################################################
paths:
  /info:
    get:
      operationId: info
      description: Get some info about this Insight
      responses:
        "200":
          description: Insight Functions successfully listed
          schema:
            $ref: "#/definitions/InsightInfoResults"
        default:
          description: Errors
          schema:
            $ref: '#/definitions/ErrorResponse'

  /insights:
    post:
      operationId: listInsights
      description: Get a list of available Insight Functions and info about them
      parameters:
      - name: body
        in: body
        description: Get a list of available insight functions
        required: true
        schema:
          $ref: '#/definitions/InsightsFilterParams'
      responses:
        "200":
          description: Insight Functions successfully listed
          schema:
            $ref: "#/definitions/InsightListResults"
        default:
          description: Errors
          schema:
            $ref: '#/definitions/ErrorResponse'

  /insight/{function_id}:
    get:
      operationId: infoInsight
      description: Get info about one Insight Function
      parameters:
        - name: function_id
          type: string
          in: path
          description: Identifier of function
      responses:
        "200":
          description: Insight Functions successfully listed
          schema:
            $ref: "#/definitions/InsightInfo"
        default:
          description: Errors
          schema:
            $ref: '#/definitions/ErrorResponse'

  /lifecycle:
    post:
      operationId: lifecycle
      description: |
        Notifications of when a linkage that will call the process function is created or deleted.
      parameters:
        - name: body
          in: body
          description: Lifecycle event
          required: true
          schema:
            $ref: '#/definitions/LifecycleEvent'
      responses:
        default:
          description: The response to this is ignored.

  /process: # The actual path can be anything.
    post:
      operationId: process
      description: |
        Your function to process a bunch of Signal Data.

        It is good form to copy all the fields of a source SignalData into the new SignalData,
        then update the properties according to your function.
      parameters:
        - name: body
          in: body
          description: Data to process and arguments on how to process it
          required: true
          schema:
            $ref: "#/definitions/SignalDataObjectArray"
      responses:
        "200":
          description: Data successfully
          schema:
            $ref: "#/definitions/SignalDataArrayArray"
        default:
          description: Error that stops the pipeline.
            Non-fatal errors should produce new signals on a separate outlet.
          schema:
            $ref: '#/definitions/ErrorResponse'

################################################################################
#                                 Definitions                                  #
################################################################################
definitions:
  InsightInfoResults:
    type: object
    description: Info about this Insight Module
    properties:
      group_id_required:
        type: boolean
        description: True if the listInsights requires group_id to be something to list functions.
      name:
        type: string
        description: Presented name
      description:
        type: string
        description: Additional text about this insight. Markdown can be used.
      wants_lifecycle_events:
        type: boolean
        description: |
          True if this insight wants to be notified when linkages using it are created and deleted.
          Defaults to false.
      translations:
        type: object
        description: Translations of the name and description fields
        additionalProperties:
          type: object
          description: Keyed by each translated language code 
          properties:
            name:
              type: string
              description: Presented name
            description:
              type: string
              description: Additional text about this insight. Markdown can be used.
    required: [group_id_required]

  InsightsFilterParams:
    type: object
    description: Filter options for insight functions
    properties:
      group_id:
        type: string
        description: |
          Which group of insight functions to get.  The meaning of the contents is up to the Insight Module.
          Including ignoring it and always returning all.
      limit:
        type: integer
        description: Limit how many insights to fetch
      offset:
        type: integer
        description: Offset into the returned insights
    required: [group_id]

  SignalTypeInputInfo:
    type: object
    description: |
      What data type the incoming Signal must have.  One of `primitive_type`, `data_type` or `data_unit`
      should be specified.
    required: [name]
    properties:
      primitive_type:
        type: string
        enum: [NUMERIC, STRING, BOOLEAN, JSON]
        description: If present, the primitive type that the connecting signal must have
      data_type:
        type: array
        description: If present, the type that the connecting signal must have
        items:
          type: string
          description: A required data type
      data_unit:
        type: array
        description: If present, the units that the connecting signal must have
        items:
          type: string
          description: A required unit.
      tag:
        type: string
        description: Optional tag to add to the SignalData for inlet matching
      name:
        type: string
        description: Name of this signal
      description:
        type: string
        description: Optional text to help users.
      translations:
        type: object
        description: Translations of the name and description fields
        additionalProperties:
          type: object
          description: Keyed by each translated language code 
          properties:
            name:
              type: string
              description: Presented name
            description:
              type: string
              description: Optional text to help users.

  SignalTypeOutputInfo:
    type: object
    description: |
      What data type the incoming Signal must have.  One of `primitive_type`, `data_type` or `data_unit`
      should be specified.
    properties:
      primitive_type:
        type: string
        enum: [NUMERIC, STRING, BOOLEAN, JSON]
        description: If present, the primitive type that the connecting signal must have
      data_type:
        type: array
        description: If present, the type that the connecting signal must have
        items:
          type: string
          description: A required data type.
      data_unit:
        type: array
        description: If present, the unit that the connecting signal must have
        items:
          type: string
          description: A required unit.
      name:
        type: string
        description: Name of this outlet
      description:
        type: string
        description: Optional text to help users.
      suggested_name:
        type: string
        description: |
          A suggestion for what to name signals that get created for this outlet.
          This may include `{}` to indicate where to place the function name.
      translations:
        type: object
        description: Translations of the name and description fields
        additionalProperties:
          type: object
          description: Keyed by each translated language code 
          properties:
            name:
              type: string
              description: Presented name
            description:
              type: string
              description: Optional text to help users.
            suggested_name:
              type: string
              description: |
                A suggestion for what to name signals that get created for this outlet.
                This may include `{}` to indicate where to place the function name.

  ConstantInfo:
    type: object
    description: A constant that needs to be provided by the user
    properties:
      name:
        type: string
        description: The name of this constant; This is not shown to users and is the code identifier for this constant.
      type:
        type: string
        enum: [string, number, boolean]
        description: What kind of value this constant accepts
      description:
        type: string
        description: Some words for the user about this constant
      default:
        type: [string, number, boolean]
        description: Optional value to use if user doesn't specify one.
      enum:
        type: array
        description: An array of allowed values.
      enum_presented:
        type: array
        description: If present, the strings to display instead of the values in `enum`.  The length of this must match `enum`.
        items:
          type: string
          description: Presentable name for one value
      maximum:
        type: number
        description: For number type, the largest it can be
      minimum:
        type: number
        description: For number type, the smallest it can be
      multiple:
        type: boolean
        description: |
          When this is true, this constant will be an array of values instead of a single value.
          The UI will present the user a way to enter this constant multiple times.
      translations:
        type: object
        description: Translations of the name and description fields
        additionalProperties:
          type: object
          description: Keyed by each translated language code 
          properties:
            description:
              type: string
              description: Some words for the user about this constant
            enum_presented:
              type: array
              description: If present, the strings to display instead of the values in `enum`.
              items:
                type: string
    required: [name, type]

  InsightInfo:
    type: object
    description: One insight function
    properties:
      id:
        type: string
        description: The unique id for this insight function
      name:
        type: string
        description: A human friendly name for this insight function
      description:
        type: string
        description: A description of this insight function to display to users.
      translations:
        type: object
        description: Translations of the name and description fields
        additionalProperties:
          type: object
          description: Keyed by each translated language code 
          properties:
            name:
              type: string
              description: A human friendly name for this insight function
            description:
              type: string
              description: A description of this insight function to display to users.
      type:
        type: string
        enum: [transform, rule, action]
        description: |
          What kind of insight this function is.  Determines how it gets used and how the UI
          presents it.  If omitted, then 'transform' is used.
      history:
        type: object
        description: |
          When this object is present, a timeseries query will be called and the results sent along
          to the function call.  The properties here will be used to augment the query.  Only the
          Signals connected to the calling of this function will be queried.
        properties:
          include_from:
            type: string
            enum: [INLETS, OUTLETS, BOTH]
            description: When fetching history, which connected signals to fetch from.  Defaults to BOTH.
        additionalProperties:
          type: object
          description: An additional parameter to the query.  Use only one property. (value, constant, template)
          properties:
            value:
              type: [string, number]
              description: The exact value to use
            constant:
              type: string
              description: Which of this function's constants to use the value of here
            template:
              type: string
              description: |
                Which of this function's constants to use the value of here, fitted into a template
                string.  The constants must be surrounded with '{{}}'.  For example '-{{days}}d'
      constants:
        type: array
        description: Additional user specified values to pass to this insight function
        items:
          $ref: '#/definitions/ConstantInfo'
      constants_multiple_maximum:
        type: number
        description: |
          If there are constants with multiple set to true, this states the maximum number of times the
          constants can be repeated. (The maximum size of the arrays.)
          If unspecified then the UI can apply any limit, including none.
      inlets:
        type: array
        description: The kinds of data that can be connected. (This is an unordered set)
        maxItems: 5
        items:
          $ref: '#/definitions/SignalTypeInputInfo'
      outlets:
        type: array
        description: What kind of data each output will produce. (This is an ordered array)
        maxItems: 5
        items:
          $ref: '#/definitions/SignalTypeOutputInfo'
    required:
      - id
      - name
      - description

  InsightListResults:
    type: object
    description: The found insight functions with count info
    properties:
      total:
        type: integer
        description: The total number of insight functions in this group
      count:
        type: integer
        description: Number of insight functions returned
      insights:
        type: array
        description: The insight functions
        items:
          $ref: '#/definitions/InsightInfo'
    required:
      - insights
      - total
      - count

  LifecycleEvent:
    type: object
    description: A Lifecycle event
    properties:
      event:
        type: string
        description: What action caused this event
        pattern: "(create|delete)"
      id:
        type: string
        description: The unique id for the linkage that has the event.  This id will match in the calls to process.
      args:
        type: object
        description: The arguments for this linkage that will be also passed to each call of process.

  SignalData:
    type: object
    description: A single instance of a piece of data inside a signal.
    properties:
      tags:
        type: object
        description: |
          Tag-Value pairs to help ID the data. Used to tie the data back to an
          Asset or Device or other things.
        additionalProperties:
          type: string
          description: Value for a tag
          minLength: 1
      ts:
        type: integer
        description: Unix timestamp in microseconds of when the data originated
      gts:
        type: integer
        description: Unix timestamp in microseconds of when this SignalData was generated
      value:
        type: [string, number, object]
        description: The value for this instance of data
      origin:
        type: string
        description: The original Publishing ID
        minLength: 1
      generated:
        type: string
        description: The Publishing ID that created this SignalData.
        minLength: 1
      ttl:
        type: integer
        description: Value used to help prevent data from infinitely cycling
    required:
      - ts
      - value
      - origin
      - generated

  HistoricalData:
    type: object
    description: The history of signals.  Each signal id is the key, with an array of timestamp, value pairs.
    additionalProperties:
      type: array
      description: The history of signals.  Each signal id is the key, with an array of timestamp, value pairs.
      items:
        type: object
        description: One point in history
        properties:
          ts:
            type: integer
            description: Unix timestamp in microseconds of when the data was stored
          value:
            type: [string, number, object]
            description: The value for this instance of data
          tags:
            type: object
            description: Tag-Value pairs to help ID the data.  (Tags here are a subset of those in SignalData)
            additionalProperties:
              type: string
              description: Value for a tag
              minLength: 1
        required: [ts, value]

  CallbackInfo:
    type: object
    description: URL and auth token to use to send results if not immediately available.
    properties:
      url:
        type: string
        description: The URL to post the results to.
      token:
        type: string
        description: The token to put in the AUTHORIZATION header.

  SignalDataArray:
    type: array
    description: Array of Signal Data
    items:
      $ref: '#/definitions/SignalData'

  SignalDataArrayArray:
    type: array
    description: Array of Array of Signal Data.  An array for each outlet, each outlet can have many signals.
    items:
      $ref: '#/definitions/SignalDataArray'

  SignalDataObjectArray:
    type: object
    description: The data and arguments to be processed.
    properties:
      id:
        type: string
        description: A unique ID for this instance of this function in an asset pipeline.
      data:
        $ref: '#/definitions/SignalDataArray'
      history:
        $ref: '#/definitions/HistoricalData'
      cbi:
        $ref: '#/definitions/CallbackInfo'
      args:
        type: object
        description: Object of instance arguments.
        properties:
          function_id:
            type: string
            description: Which insight function to run
          group_id:
            type: string
            description: Which group_id this is under
          constants:
            type: object
            description: Additional constant parameters for the selected function


  ErrorResponse:
    type: object
    description: Error response that stops the pipeline.
    properties:
      message:
        type: string
        description: Error message
      status:
        type: number
        description: A numerical code for the error
      name:
        description: Error type
        type: string

Last update: September 3, 2021
Back to top