Device HTTP API¶

The purpose of this API is to provide device software and firmware with the ability to communicate with Murano. Devices use this API to perform actions such as authentication, posting sensor and status data, and retrieving configuration and control updates.

General¶

Authentication¶

A device may authenticate with a TLS Client Certificate or with the Authorization HTTP request header, using a secret Token or by using a Password. More information about the authentication settings may be found in the Provisioning section.

$ curl -i 'https://<IOT_CONNECTOR_FQDN>/onep:v1/stack/alias' \
-u '' \
-d 'data_in={"001":73.16492}'

$ curl -i 'https://<IOT_CONNECTOR_FQDN>/onep:v1/stack/alias' \
-u '<identity>' \
-d 'data_in={"001":73.16492}'

$ curl -i 'https:<IOT_CONNECTOR_FQDN>/onep:v1/stack/alias' \
-u ':<token>'
-d 'data_in={"001":73.16492}'

$ curl -i 'https:<IOT_CONNECTOR_FQDN>/onep:v1/stack/alias' \
-u '<identity>:<password>'
-d 'data_in={"001":73.16492}'

Product Domain¶

Each Murano Product is assigned a unique API domain for devices to connect to. In this document, we will use the example domain of <IOT_CONNECTOR_FQDN> as an example Product domain. Replace the example domain with the Endpoint shown in the associated IoT Connector page of your Murano solutions UI.

HTTP Responses¶

Typical HTTP response codes include:

Libraries and Sample Code¶

Sample code, libraries, and client applications that uses this API can be found on the Device API Libraries / Reference page.

Encoding¶

As of Content-Type: application/x-www-form-urlencoded, non-alphanumeric characters from both part of the equal sign must be percent encoded as defined by the standard https://www.w3.org/TR/html401/interact/forms.html.

The Content-Encoding and Accept-Encoding headers as defined by the RFC 2616 standard, are supported. The following encoding types are implemented:

• gzip: A format using the Lempel-Ziv coding (LZ77), with a 32-bit CRC.
• identity: No encoding is applied. This is the default when headers are omitted.

If the caller specifies a value for Accept-Encoding or Content-Encoding that is not supported by Murano, a 406 or 415 response will be returned, respectively.

POST /onep:v1/stack/alias HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Authorization: token <token>
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Content-Encoding: gzip
Content-Length: <length>

<gzipped 'data_in={"001":73.16492}'>

HTTP/1.1 204 No Content
Date: <date>
Server: Murano
Connection: Keep-Alive

GET /onep:v1/stack/alias?<alias_1> HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Authorization: token <token>
Accept: application/x-www-form-urlencoded; charset=utf-8
Accept-Encoding: identity;q=0.5, gzip;q=1.0

HTTP/1.1 200 OK
Date: <date>
Server: Murano
Connection: Close
Content-Length: <length>
Content-Encoding: gzip

<gzipped 'alias_1=value_1'>

TLS Connection Requirements¶

When making a secure TLS connection attempt to the API domain, it is required to specify the domain as the SNI field in the TLS connection request. The HTTP "Host" header must also be set to the domain name. Any mismatch or use of an invalid domain will result in the connection being terminated without response. Also read the Device API connectivity TLS Specs..

Bandwidth considerations¶

TLS handshake has a significant request size and latency overhead for small HTTP payloads. This is because the server needs to provide the certificate chain (typically 3~4kb) to the device for verification. HTTP keep-alive can be used to mitigate most of the TLS overhead, especially when a device sends multiple small requests.

From our tests, sending three sequential write requests (single resource) without HTTP keep-alive used ~16kb bandwidth. The same requests with HTTP keep-alive enabled, over a single connection, used ~6kb bandwidth and had %40 lower latency.

Restrictions¶

There is no specific bandwidth or message size limitation, however each individual resource size MUST respect the maximum size of 1048576 bytes.

Provisioning¶

Murano IoT Connectors are configured to allow devices to provision their own identities or to only allow whitelisted device identities to be provisioned. This configuration option is available via the IoT Connector Settings tab as the Allow devices to register their own identity checkbox. When checked (default), any device using this API with the assigned IoT Connector FQDN Endpoint can provision themselves without the need to be whitelisted in advance. If this option is disabled (unchecked), then device identities must be pre registered (whitelisted) in advance in order to provision and communicate with the Murano connector.

To connect your device remember to download and install the Murano Root server CA. Otherwise your firmware will probably raise a security error during connection.

For development on your operating system, use the Digicert Root CA which is already known by your system. (So no need of downloading and installing it manually.)

Once activated a devices will be visible on the Murano IoT-Connector devices page with its IP set.

Provision¶

Provision requests take the following forms. The Content-Type and body will depend on the type of authentication being provisioned.

Certificate¶

When an IoT Connector is configured for Certificate authentication, provisioning will happen automatically when the device first connects. The request the device will make on its initial connection, is irrelevant and is not part of the provisioning process. That said, the initial connection results in two subsequent steps but separate operations:

Token Auth¶

When an IoT Connector is configured for Token authentication, calling this endpoint will provision credentials for the given <identity> and return the secret <token> that the device must use for all subsequent API requests. Identities must also conform to the identity format specification defined in the Settings tab of the IoT Connector management UI.

POST /provision/activate HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Content-Length: <length>

id=<identity>

Response

HTTP/1.1 200 OK
Date: <date>
Server: Murano
Connection: Keep-Alive
Content-Length: <length>
Content-Type: text/plain; charset=utf-8

<token>

The response should return a 40-character string token.

$ curl -i 'https://<IOT_CONNECTOR_FQDN>/provision/activate' \
    -H 'Content-Type: application/x-www-form-urlencoded; charset=utf-8' \
    -d 'id=12345678'

HTTP/1.1 200 OK
Date: Fri, 05 May 2017 00:11:22 GMT
Server: Murano
Connection: Keep-Alive
Content-Length: 40
Content-Type: text/plain; charset=utf-8

22596363b3de40b06f981fb85d82312e8c0ed511

import json
import requests
​
requests.post(
  "https://<IOT_CONNECTOR_FQDN>/provision/activate",
  headers={ "Content-Type": "application/x-www-form-urlencoded; charset=utf-8" },
  data={ "id": "12345678" }
)

HTTP/1.1 200 OK
Date: Fri, 05 May 2017 00:11:22 GMT
Server: Murano
Connection: Keep-Alive
Content-Length: 40
Content-Type: text/plain; charset=utf-8

22596363b3de40b06f981fb85d82312e8c0ed511

Password Auth¶

When an IoT Connector is configured for Password authentication, calling this endpoint will provision credentials for the given <identity> and <password> (at least 20 characters) that the device must use for all subsequent API requests. Identities must also conform to the identity format specification defined in the Settings tab of the IoT Connector management UI.

POST /provision/activate HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Content-Length: <length>

id=<identity>
password=<password>

Response

HTTP/1.1 204 OK
Date: <date>
Server: Murano
Connection: Keep-Alive

Provision Responses¶

Provision specific response codes include:

Reprovision¶

The idea of reprovisioning is that, in case device credentials get compromised or for other reasons, new credentials need to be set for a device already existing in the database. The reprovisioning process is the same as provisioning but requires that the device authenticate first. So the "activate" endpoint described above is called the same way but with authentication credentials provided in the call.

POST /provision/activate HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Content-Type: application/x-pem-file; charset=utf-8
Content-Length: <length>

<certificate_in_PEM_format>

HTTP/1.1 204 OK
Date: <date>
Server: Murano
Connection: Keep-Alive

Writing And Reading Data¶

Write¶

Write one or more resources identified by <alias> with the given <value>. The connecting device is identified and authenticated with the provided <token>, <password> or <certificate>. If the Murano Product has defined resources with matching <alias>es, the <value>s are stored as the device state at timestamp the data was received by Murano. If multiple aliases are specified, they are written at the same timestamp. This method generates an event type of "data_in" on the Murano Device2 service and may be used to, for instance, store the data in the Tsdb database or alert users or other services of the event.

Request

POST /onep:v1/stack/alias HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Authorization: token <token>
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Content-Length: <length>

<alias_1>=<value_1>&<alias_2...>=<value_2...>&<alias_n>=<value_n>

Response

HTTP/1.1 204 No Content
Date: <date>
Server: Murano
Connection: Keep-Alive

• See HTTP Responses for a full list of responses.

$ curl -i 'https://<IOT_CONNECTOR_FQDN>/onep:v1/stack/alias' \
    -u '' \
    -d 'data_in={"001":73.16492}'

import json
import requests
​
requests.post(
  f"https://<IOT_CONNECTOR_FQDN>/onep:v1/stack/alias",
  headers={
    "Content-Type": "application/x-www-form-urlencoded; charset=utf-8",
    "Authorization": "token <token>" 
  },
  data={
    "config_io": json.dumps({"channels":{"001":{"display_name":"Temperature","description":"Ambient tmperature reading","properties":{"data_type":"TEMPERATURE","data_unit":"DEG_FAHRENHEIT","precision":2},"protocol_config":{"report_rate":30000,"timeout":60000}}}}),
    "data_in": json.dumps({"001":73.16492})
  }
)

HTTP/1.1 204 No Content
Date: Fri, 05 May 2017 00:11:22 GMT
Server: Murano
Connection: Keep-Alive

Read¶

Read the most recent value from one or more resources with <alias>. If at least one <alias> resource definition exists, the set value will be returned.

Request

GET /onep:v1/stack/alias?<alias_1>&<alias_2...>&<alias_n> HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Authorization: token <token>
Accept: application/x-www-form-urlencoded; charset=utf-8

Response

HTTP/1.1 200 OK
Date: <date>
Server: Murano
Connection: Close
Content-Length: <length>

<alias_1>=<value_1>&<alias_2...>=<value_2...>&<alias_n>=<value_n>

• Response may also be HTTP/1.1 204 No Content if either none of the aliases are found or the device state for the given aliases are empty
• See HTTP Responses for a full list of responses

$ curl -i 'https://<IOT_CONNECTOR_FQDN>/onep:v1/stack/alias?config_io' \
    -u '' \
    -H 'Accept: application/x-www-form-urlencoded; charset=utf-8'

HTTP/1.1 200 OK
Date: Fri, 05 May 2017 00:11:22 GMT
Server: Murano
Connection: Keep-Alive
Content-Length: 13
Content-Type: application/x-www-form-urlencoded; charset=utf-8

config_io=%7b%22channels%22%3a+%7b%22001%22%3a+%7b%22description%22%3a+%22URL+string%22%2c+%22display_name%22%3a+%22Raw+URL+String%22%2c+%22properties%22%3a+%7b%22data_type%22%3a+%22STRING%22%7d%2c+%22report_rate%22%3a+10000%2c+%22sample_rate%22%3a+10000%7d%7d%7d

Hybrid Read Write¶

Write one or more values with <alias_w> with given <value> and also read the most recent values from <alias_r>.

Note: All writes occur before all reads.

Request

POST /onep:v1/stack/alias?<alias_r1>&<alias_r2...>&<alias_rn> HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Authorization: token <token>
Accept: application/x-www-form-urlencoded; charset=utf-8
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Content-Length: <length>

<alias_w1>=<value_1>&<alias_w2...>=<value_2...>&<alias_wn>=<value_n>

Response

HTTP/1.1 200 OK
Date: <date>
Server: Murano
Connection: Close
Content-Length: <length>

<alias_r1>=<value_1>&<alias_r2...>=<value_2...>&<alias_rn>=<value_n>

• Response may also be HTTP/1.1 204 No Content if either none of the aliases are found or the device state for the given aliases are empty
• See HTTP Responses for a full list of responses

$ curl 'https://<IOT_CONNECTOR_FQDN>/onep:v1/stack/alias?config_io' \
    -u '' \
    -H 'Accept: application/x-www-form-urlencoded; charset=utf-8' \
    -H "Content-Type: application/x-www-form-urlencoded; charset=utf-8" \
    -d 'data_in={"001":73.16492}'

HTTP/1.1 200 OK
Date: Fri, 05 May 2017 00:11:22 GMT
Server: Murano
Connection: Keep-Alive
Content-Length: 13
Content-Type: application/x-www-form-urlencoded; charset=utf-8

config_io=%7b%22channels%22%3a+%7b%22001%22%3a+%7b%22description%22%3a+%22URL+string%22%2c+%22display_name%22%3a+%22Raw+URL+String%22%2c+%22properties%22%3a+%7b%22data_type%22%3a+%22STRING%22%7d%2c+%22report_rate%22%3a+10000%2c+%22sample_rate%22%3a+10000%7d%7d%7d

Long Polling¶

The read endpoint also supports long polling. Long polling is a method of getting a server push without the complexities of setting up publicly accessible HTTP server endpoints on your device. As the name suggests, long polling is similar to normal polling of an HTTP resource, but instead of requiring the client to make a new request to the server constantly, the server will wait to return until it has new information to return to the client (or a timeout has been reached).

To perform a request with long polling, simply add the header Request-Timeout: <milliseconds> to your request. The server will then wait until a new datapoint is written to the given resource and will then immediately return the value. If no datapoint is written before that time, a 304 Not Modified is returned and the client may make another long polling request to continue monitoring that resource.

You may also optionally add an If-Modified-Since header to specify a start time to wait. This is exactly the same as the alias.last semantics in scripting. You will want to use this if it's important that you receive all updates to a given resource; otherwise it is possible to miss points written between long polling requests.

Note: Only one resource may be read at a time when using long polling.

Request

GET /onep:v1/stack/alias?<alias_1> HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Authorization: token <token>
Accept: application/x-www-form-urlencoded; charset=utf-8
Request-Timeout: <timeout>
If-Modified-Since: <timestamp>

• <alias> is the alias you monitor for new datapoints.
• Request-Timeout specifies how long to wait on changes. <timeout> is a millisecond value and cannot be more than 300 seconds (300,000 ms).
• If-Modified-Since specifies waiting on aliases since the <timestamp>. <timestamp> can be timestamp seconds since 1970-01-01 00:00:00 UTC or standard <a href=http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html>HTTP-Date format. If this is not specified, it defaults to "now."

Response

When the device set state is updated:

HTTP/1.1 200 OK
Date: <date>
Server: Murano
Connection: Close
Content-Length: <length>
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Last-Modified: <value-set-date>

<alias>=<value>

If the device state is not updated before timeout:

HTTP/1.1 304 Not Modified
Date: <date>
Server: Murano
Connection: Close
Content-Length: <length>

When the resource is written and a value is returned, a Last-Modified header is included. When it is vital for your application to receive all updates to a resource, you can pass the Last-Modified header value back as the If-Not-Modified-Since header in your next request to make sure you do not miss any points that may have been written since the last request returned.

$ curl 'https://<IOT_CONNECTOR_FQDN>/onep:v1/stack/alias?config_io' \
    -u '' \
    -H 'Accept: application/x-www-form-urlencoded; charset=utf-8' \
    -H 'Request-Timeout: 30000' \
    -H 'If-Modified-Since: 1408088308'

HTTP/1.1 200 OK
Date: Fri, 05 May 2017 00:11:22 GMT
Server: Murano
Connection: Keep-Alive
Content-Length: 13
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Last-Modified: 1408088412

config_io=%7b%22channels%22%3a+%7b%22001%22%3a+%7b%22description%22%3a+%22URL+string%22%2c+%22display_name%22%3a+%22Raw+URL+String%22%2c+%22properties%22%3a+%7b%22data_type%22%3a+%22STRING%22%7d%2c+%22report_rate%22%3a+10000%2c+%22sample_rate%22%3a+10000%7d%7d%7d

Record¶

Report data to one or more historical timestamps of one or more aliases. If the Murano Product has defined resources with matching <alias>es, the most recent <value>s are stored as the device state with the timestamp the data was received by Murano. The connecting device is identified and authenticated with the provided <token>.

timestamp1=1551674686.339 s -> 1551674686339000 us
timestamp2=1551674686.339876 s -> 1551674686339876 us

Acceptable values for timestamps are as follows:

• when positive, the value is the number of seconds since 1970.01.01 00:00:00 UTC.
• when negative, the value is a time offset in seconds relative to 'now'. For example, -3600 means one hour ago.

Request

This example records to <alias_1> in <timestamp_1> and <timestamp_2>, and records to <alias_2> in <timestamp_3> and <timestamp_4>.

POST /onep:v1/stack/record HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Authorization: token <token>
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Content-Length: <length>

alias=<alias_1>&<timestamp_1>=<value_1>&<timestamp_2>=<value_2>&alias=<alias_2>&<timestamp_3>=<value_3>&<timestamp_4>=<value_4>

Response

HTTP/1.1 204 No Content
Date: <date>
Server: <server>
Connection: Close
Content-Length: 0

• See HTTP Responses for a full list of responses.
• If <timestamp_1> and <timestamp_2>, <timestamp_3> and <timestamp_4> have less than one-second difference:

HTTP/1.1 409 Conflict
Date: <date>
Server: <server>
Connection: Close
Content-Length: <length>

<alias_1>=<timestamp_1>&<alias_2>=<timestamp_2>

$ curl https://<IOT_CONNECTOR_FQDN>/onep:v1/stack/record \
    -u '' \
    -H 'Content-Type: application/x-www-form-urlencoded; charset=utf-8' \
    -d 'alias=data_in&1506708551={"001":73.16492}&1506708552.127={"001":74.78193}&1506708553.264391={"001":75.35915}'

Static Content Files¶

This chapter explains how to interact with static files provided by Murano's Content micro-service.

List Available Content¶

List available content <content-id>s.

GET /provision/download HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Authorization: token <token>

Response

HTTP/1.1 200 OK
Date: <date>
Server: Murano
Connection: Keep-Alive
Content-Length: <length>
Content-Type: text/csv; charset=utf-8

<content-id_1>
<content-id_2...>
<content-id_n>

Response may also be:

• See HTTP Responses for a full list of responses

$ curl -i 'https://<IOT_CONNECTOR_FQDN>/provision/download' \
    -u ''

HTTP/1.1 200 OK
Date: Fri, 05 May 2017 00:11:22 GMT
Server: Murano
Connection: Keep-Alive
Content-Length: 71
Content-Type: text/csv; charset=utf-8

MANIFEST
fw_20150519_rev01b.bin
fw_20160101_rev02.bin
splash01.png
splash02.png

Get Content Info¶

Retrieve meta information (content-type, size, updated timestamp, description) for the specified <content-id>.

GET /provision/download?id=<content-id>&info=true HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Authorization: token <token>

Response

HTTP/1.1 200 OK
Date: <date>
Server: Murano
Connection: Keep-Alive
Content-Length: <length>
Content-Type: text/csv; charset=utf-8

<content-type>,<byte-size>,<updated-timestamp>,<description>

Response may also be:

• See HTTP Responses for a full list of responses

$ curl -i 'https://<IOT_CONNECTOR_FQDN>/provision/download?id=splash01.png&info=true' \
    -u ''

HTTP/1.1 200 OK
Date: Fri, 05 May 2017 00:11:22 GMT
Server: Murano
Connection: Keep-Alive
Content-Length: 45
Content-Type: text/csv; charset=utf-8

image/png,23427,1462500951,Boot splash screen

Download Content¶

Download the content <content-id> in full or in part. To request chunks of the content, use the header Range: bytes=<range-specifier>. <range-specifier> takes the form of X-Y where both X and Y are optional but at least one of them must be present. X is the start byte position to return. Y is the end position. Both are 0 based. If X is omitted, Y will request the last Y count of bytes of the content. If Y is omitted, it will default to the end of the content. The response Content-Type header will be as defined in the content meta info.

GET /provision/download?id=<content-id> HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Authorization: token <token>
{Range: bytes=<range-specifier>}

Response

HTTP/1.1 200 OK
Date: <date>
Server: Murano
Connection: Keep-Alive
Content-Length: <number of bytes being returned>
Content-Type: <content-type>
Transfer-Encoding: chunked
{Accept-Ranges: bytes}
{Content-Range: bytes <first_position>-<last_position>/<total_length>}

<blob>

Response may also be:

• HTTP/1.1 206 Partial Content if the response is partial.
• See HTTP Responses for a full list of responses

$ curl -i 'https://<IOT_CONNECTOR_FQDN>/provision/download?id=splash01.png' \
    -u ''

HTTP/1.1 200 OK
Date: Fri, 05 May 2017 00:11:22 GMT
Server: Murano
Connection: Keep-Alive
Content-Length: 23427
Content-Type: image/png
Transfer-Encoding: chunked

<23427-byte-blob>

$ curl -i -r 0-1023 https://<IOT_CONNECTOR_FQDN>/provision/download?id=splash01.png \
    -u ''

HTTP/1.1 200 OK
Date: Fri, 05 May 2017 00:11:22 GMT
Server: Murano
Connection: Keep-Alive
Content-Length: 1024
Content-Type: image/png
Transfer-Encoding: chunked
Accept-Ranges: bytes
Content-Range: bytes 0-1023/23427

<1024-byte-blob>

Upload Content¶

The device API provides the ability to upload files from the device into Murano Content Store. Uploaded files will be accessible on the Content Service under the device2/<device identity-id>/ folder. Files uploaded by a given device will be available for download to that device only and will not be available to other devices. This method generates an event type of "content_in" on the Murano Device2 service. It carries all the meta information about the uploaded content but not the actual content blob. See the device2 service reference.

PUT /onep:v1/content/<content-id>?tags=<tags> HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Authorization: token <token>
Content-Length: <length in bytes>
{Content-Type: <media type of file>}
{Content-MD5: <base64-encoding of MD5 sum value>}

<blob as body payload>

is the url encoding of a JSON key/value object. Eg. {"mytag1":"myvalue1","mytag2":"myvalue2"}

Response

HTTP/1.1 204 No Content
Date: <date>
Server: Murano
Connection: Keep-Alive

$ curl -i -X PUT https://<IOT_CONNECTOR_FQDN>/onep:v1/content/logs/error.log?tags=%7B%22abc%22%3A123%2C%22xyz%22%3A456%7D \
    -H 'Content-Type: text/plain' \
    -H 'Content-Length: 13' \
    -H 'Content-MD5: 7naXGYTSUTihmayQVTQB3Q==' \
    -u '' \
    -d @error.log

import os
import json
import requests
import base64
import hashlib
import urllib.parse

def checksum(filename, hash_factory=hashlib.md5, chunk_num_blocks=128):
    h = hash_factory()
    with open(filename,'rb') as f:
        for chunk in iter(lambda: f.read(chunk_num_blocks*h.block_size), b''):
            h.update(chunk)
    return h.digest()

connector_fqdn = "<IOT_CONNECTOR_FQDN>"
token = "<TOKEN>"
file_name = "myfile.csv"
file_path = f"./{file_name}"
encoded_name = urllib.parse.quote(file_name)
upload_url = f"https://{connector_fqdn}/onep:v1/content/{encoded_name}"

checksum = checksum(file_path)
checksum = base64.b64encode(checksum).decode('utf-8')
content_length = os.path.getsize(file_path)
headers = {
    'Content-Type': 'application/x-www-form-urlencoded',
    'Authorization': 'token '+ token,
    'Content-Type': 'text/csv',
    'Content-Length': str(content_length),
    'Content-MD5': checksum
}

# Streaming uploads, which allow you to send large files or streams without reading into memory
with open(file_path, 'rb') as file:
    r = requests.put(
        upload_url,
        headers=headers,
        data=file)
    print(r, r.text)

HTTP/1.1 204 OK
Date: Tue, 02 Jul 2019 15:14:22 GMT
Server: Murano
Connection: Keep-Alive

Upload Content in Parts¶

If content is smaller than 100MB in size, it can be uploaded in a single push, as described in (Upload Content). For files over 100MB, Murano provides a way to upload them in multiple parts. Such multi-part upload involves three steps:

1. The initiation of the upload
2. The uploading of one or more part
3. The optional step of completing the upload

An upload starts by sending a request to initiate the upload, specifying a content-id, an optional length and optional tags. This creates a placeholder for the actual content blob to be uploaded to.

The second step is sending one or more request, each containing a separate part of the content blob with a range specifier so that Murano can assemble all parts into one whole. The parts don’t need to be in sequential order. For each part, an MD5 sum can be specified which, if provided, Murano will validate.

If an upload attempt specifies a range that overlaps that of another part already uploaded, the request will fail.

If length was given in the initial request, the upload completes automatically when a contiguous range of as many bytes have been uploaded. If length was not initially specified, the upload must be finalized by sending a 0-byte part.

If any of the requests, including initiation, part upload and finalization, fails, the request can be retried. Also, the initiation and part upload requests are idempotent - they can be repeated without side effects - except for the final part (if it automatically completes the upload) and the optional 0-byte finalization request. Once an upload has been finalized, attempts to re-upload parts of it or re-finalize it, will fail.

If a finalization attempt is made when not enough bytes have been uploaded or there is a gap in the uploaded ranges, the finalization will fail.

Status information about content that is in the process of being uploaded, can be obtained through the 'Get Content Info' endpoint. It will return range and status information about the parts that have already been, or are being, uploaded.

Once successfully finalized, content uploaded in parts will show up as just another content item in the device2/<identity>/<content-id> area and a content_in event is triggered.

Below are the actual requests involved in the process.

Initiate an upload

POST /onep:v1/content/ HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Authorization: token <token>
Content-Length: <length in bytes>
{Content-Type: application/json}

{
  "id":<content-id>,
  "length":<byte-count>,
  "type":<media type of file>,
  "tags":{"mytag1":"myvalue1","mytag2":"myvalue2"}
}

Response

HTTP/1.1 204 No Content
Date: <date>
Server: Murano
Connection: Keep-Alive

Upload a part

PATCH /onep:v1/content/<content-id> HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Authorization: token <token>
Content-Length: <length in bytes>
MD5: <base-64 encoded MD5 sum value of payload>
{Content-Type: application/octet-stream}
Content-Range: bytes <first byte position>-<last byte position>/<total size or *>

<blob as body payload>

Response

HTTP/1.1 204 No Content
Date: <date>
Server: Murano
Connection: Keep-Alive

Finalize an upload

PATCH /onep:v1/content/<content-id> HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Authorization: token <token>
Content-Length: 0

Response

HTTP/1.1 204 No Content
Date: <date>
Server: Murano
Connection: Keep-Alive

Get Upload Info

GET /onep:v1/content/<content-id> HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>
Authorization: token <token>
Content-Length: 0

Response

HTTP/1.1 200 OK
Date: <date>
Server: Murano
Connection: Close
Content-Length: <length>

{
  "id":"device2/<identity>/<content-id>",
  "incomplete_parts":[[12000000,17999999]],
  "length":25000000,
  "ranges":[[0,5999999],[6000000,11999999]],
  "type":"text/plain"
}

Debug events

Besides the content_in event (device2 murano service) indicating a successful upload, to aid development debug events are sent to the product debug web socket endpoint as follows:

• body_size_too_large: the part was greater than 100MB in size
• content_notfound: the content identified by content-id in a part upload or finalization attempt was not found
• incomplete_part: an attempt was made to finalize an upload with not enough bytes uploaded or gap in the uploaded ranges
• incomplete_range: the Content-Range header specified a range longer than bytes provided in the body
• invalid_content_id: the content id is somehow invalid
• invalid_range: the Content-Range header was invalid
• invalid_tags: the tags included with the initiation request was not proper JSON or not key/value pairs
• invalid_md5: the provided MD5 header contained a value that did not successfully validate for the body payload
• overlapping_range: a part was attempted to be uploaded with Content-Range overlapping a previously uploaded part
• upload_part_too_small: the part being uploaded was less than 5MB in size
• internal: an error internal to Murano occurred

Content over MQTT port¶

If your device is set to use MQTT connection you can still interact with content files using the HTTP API over the same port (eg. 8883). However client MUST set the ALPN header to enforce server-side protocol selection. Some client such as Curl send ALPN by default and nothing need to be done.

Example

File upload over MQTT port using Python.

import os
import json
import http.client as client
import base64
import hashlib
import urllib.parse
import ssl

def checksum(filename, hash_factory=hashlib.md5, chunk_num_blocks=128):
    h = hash_factory()
    with open(filename,'rb') as f:
        for chunk in iter(lambda: f.read(chunk_num_blocks*h.block_size), b''):
            h.update(chunk)
    return h.digest()

def ssl_alpn():
    try:
        print(f"OpenSSL version:{ssl.OPENSSL_VERSION}")
        ssl_context = ssl.create_default_context()
        alpn_protocols = ['http/1.1']
        ssl_context.set_alpn_protocols(alpn_protocols)

        return  ssl_context
    except Exception as e:
        print("Exception in ssl_alpn()")
        raise e

connector_fqdn = ""
token = ""
file_name = "test.csv"
file_path = f"./{file_name}"
encoded_name = urllib.parse.quote(file_name)

# Use port 8883 for MQTT, could be changed to port 443 if configured from Murano product settings
host = f"{connector_fqdn}:8883"
endpoint = f"/onep:v1/content/{encoded_name}"


checksum = checksum(file_path)
checksum = base64.b64encode(checksum).decode('utf-8')
content_length = os.path.getsize(file_path)
headers = {
    'Authorization': 'token '+ token,
    'Content-Type': 'text/csv',
    'Content-Length': str(content_length),
    'Content-MD5': checksum
}

# This is the important option enabling use of HTTP over MQTT port
context = ssl_alpn()
conn = client.HTTPSConnection(host, context=context)

try:
    with open(file_path, 'rb') as f:
        data = f.read()
        conn.request('PUT', endpoint, body=data, headers=headers, encode_chunked=False)
        r = conn.getresponse()

        print(f"STATUS: {str(r.status)}\nREASON: {r.reason}")
finally:
    conn.close()

Misc¶

Timestamp¶

Get the current time according to the server.

Request

GET /timestamp HTTP/1.1
Host: <IOT_CONNECTOR_FQDN>

Response

HTTP/1.1 200 OK
Date: <date>
Server: Murano
Connection: Keep-Alive
Content-Length: <length>
Content-Type: text/plain; charset=utf-8

<timestamp>

• See HTTP Responses for a full list of responses

$ curl -i https://<IOT_CONNECTOR_FQDN>/timestamp

HTTP/1.1 200 OK
Date: Fri, 05 May 2017 00:11:22 GMT
Server: Murano
Connection: Keep-Alive
Content-Length: 10
Content-Type: text/plain; charset=utf-8

1408088308