Time Series Storage Service¶

Summary¶

Write, record, and query device data in our Timeseries database optimized for IoT data.

TSDB (time series database) service provides a storage layer dedicated for metric data produced and evolving over time. For example, TSDB can be used to store periodic sensor data sent from your devices.

Data is associated with a timestamp and optional tags which can be used later for query purposes. Finally, the TSDB service provides a robust aggregation system enabling data-mining capabilities.

Data retention policy can be used to automatically delete data after a certain time period (days, weeks, months, years). When metric data is older than the specified retention period it will be permanently deleted during the next GC period. Because deletions happens periodically it is possible for queries to sometimes retrieve data older than the configured retention policy. The default retention policy applies to all metrics that do not have a per metric retention policy set.

Operations

Events

Configuration parameters¶

Operations¶

addConfig¶

Description

This operation is meant to set dynamic metric name retention which value won't be reflected to the service configuration. Will add or replace (if existing) metric retention configurations. For fixed metrics name set value from the service configuration instead.

Arguments

Responses

• Returns nil for Config updated

• Returns {object} when Error

  The response to the caller {object}

  Name	Type	Description
  error	string	Error Message in case of failure
  result	object	Result message

Example

local result = Tsdb.addConfig({
  default_retention = "3m",
  metrics_retention = {
    metric1 = "3d",
    metric2 = "10d"
  }
})
response.message = result

deleteAll¶

Description

delete all data of a given solution

Responses

• Returns nil for All data deleted

• Returns {object} when Error

  The response to the caller {object}

  Name	Type	Description
  error	string	Error Message in case of failure
  result	object	Result message

Example

-- Delete all data of a given solution
local out = Tsdb.deleteAll()
response.message = out

export¶

Description

Start an export job.
An event of TSDB export will be triggered once the job finished in any state.
You can define a 'tsdb' event handler in Lua for this 'exportJob' event in your solution.

Arguments

Responses

• Returns {object} when Job successfully started

  The information of job ID {object}

  Name	Type	Description
  job_id	string	Job ID

• Returns {object} when Error

  The response to the caller {object}

  Name	Type	Description
  error	string	Error Message in case of failure
  result	object	Result message

Example

-- Query constraints for export
local metrics = {
  "temperature",
  "humidity",
  "switch",
  "host"
}
local tags = {
  region = "us",
  city = "minneapolis"
}
local query = {
  metrics = metrics,
  tags = tags
}
-- Some other possible dt_format examples: -- %Y-%m-%dT%H:%M:%S.%f%z (the default) -- %Y-%m-%d %H:%M:%S.%f (a human readable version with microseconds) -- %Y-%m-%d Winnipeg (you can append any string that works for your use case)
local format = {
  temperature = {
    {round=3},
    {rename="Temp"}
  },
  timestamp = {
    {datetime={offset=-5, dt_format="%Y-%m-%d %H:%M:%S UTC%Z America/Chicago"}}
  },
  tags = {
    {normalize={"city", "region"}}
  },
  host = {
    {replace={
      match="/(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}):(\\d{1,5})/",
      to="Port: \\2 and IP: \\1"
    }}
  }
} -- Start a new export job
local job_id = Tsdb.export({
  query = query,
  filename = "export_mlps_20170321",
  format = format
})
response.message = job_id

exportJobInfo¶

Description

Query the information of an export job, including status.

Arguments

Responses

• Returns {object} when Job info successfully returned

  The information for export job {object}

  Name	Type	Description
  error	string	Error message if job failed
  query	object	Query arguments
  state	string	State of the job (enqueued, expired, in-progress, completed or failed)
  format	object	The data format rules. Property name should be the field name which are the metrics name, "timestamp" and "tags".
  format.^[a-zA-Z0-9_]+$	[ object ]	Functions to format the this field. the rules will be applied by its order in array.
  format.^[a-zA-Z0-9_]+$[].label	string	Prepend the given string to field value. Metrics field only.
  format.^[a-zA-Z0-9_]+$[].round	integer	Round the field value with given value. Metrics field only.
  format.^[a-zA-Z0-9_]+$[].rename	string	Rename a field to the given value. Metrics field only.
  format.^[a-zA-Z0-9_]+$[].discard	boolean	Remove the field when the value is true. The field parameter should be "tags" and only support "tags" field.
  format.^[a-zA-Z0-9_]+$[].replace	object	Replace the field value which matching the pattern to the new value. Metrics field only.
  format.^[a-zA-Z0-9_]+$[].replace.to	string	The replacement value. Using \{n} to specify capture group. {n} is number of group.
  format.^[a-zA-Z0-9_]+$[].replace.match	string	String or regular expression.
  format.^[a-zA-Z0-9_]+$[].datetime	nil	-
  format.^[a-zA-Z0-9_]+$[].normalize	[ string ]	Normalize the given list of tag names, it will fill into different columns. The tags are not specified will be dropped. The field parameter should be "tags" and only support "tags" field.
  job_id	string	Job ID
  length	string	The total length of export file in bytes
  filename	string	File name of the exported CSV file
  content_id	string	Content ID of the job to Content service
  context_id	string	Solution id
  start_time	string	Start time of the job
  update_time	string	Last updated time of the job

• Returns {object} when Error

  The response to the caller {object}

  Name	Type	Description
  error	string	Error Message in case of failure
  result	object	Result message

Example

local job_info = Tsdb.exportJobInfo({
  job_id = "xxyyzz"
})
response.message = job_info

exportJobList¶

Description

List export job records of a given solution in descending timestamp order

Arguments

Responses

• Returns {[ object ]} for List of export job

  Name	Type	Description
  state	string	State of the job (enqueued, expired, in-progress, completed or failed)
  job_id	string	Job ID
  start_time	string	Start time of the job

• Returns {object} when Error

  The response to the caller {object}

  Name	Type	Description
  error	string	Error Message in case of failure
  result	object	Result message

Example

local job_list = Tsdb.exportJobList()
response.message = job_list

getConfig¶

Description

Get retention configuration of solution.

Responses

• Returns {object} when The retention configuration successfully returned

  The configuration attributes for TSDB. {object}

  Name	Type	Description
  default_retention	^\(\\d\+\(d|w|m|y\)|infinity)$	Default retention time with unit. Default value is 3m. Supported units: d (days), w (weeks), m (months), y (years) e.g. 3d, 5w or infinity Default: "3m"
  metrics_retention	object	The retention time for metrics, property name should be the metric name.
  metrics_retention.^[a-zA-Z0-9_]+$	^\(\\d\+\(d|w|m|y\)|infinity|default)$	Retention time for metric with unit. To use default for following default value and infinity for keeping metric forever. Supported units: d (days), w (weeks), m (months), y (years) e.g. 3d, 5w, default or infinity

• Returns {object} when Error

  The response to the caller {object}

  Name	Type	Description
  error	string	Error Message in case of failure
  result	object	Result message

Example

local config = Tsdb.getConfig()
response.message = config

listMetrics¶

Description

list metrics of a given solution

Arguments

Responses

• Returns {object} when Metrics information retrieved

  Metric list {object}

  Name	Type	Description
  next	string	Cursor for getting next page
  total	integer	Total number of items
  metrics	[ string ]	List of metrics

• Returns {object} when Error

  The response to the caller {object}

  Name	Type	Description
  error	string	Error Message in case of failure
  result	object	Result message

Example

-- Get a list of created metrics for a given solution
local out = Tsdb.listMetrics({limit = 10})
response.message = out

-- Use next cursor to fetch next page if found in the result of previous query
local out = Tsdb.listMetrics({limit = 10, next = out.next})
response.message = out

listTags¶

Description

list tags of a given solution

Arguments

Responses

• Returns {object} when Tags information retrieved

  Tag list {object}

  Name	Type	Description
  next	string	Cursor for getting next page
  tags	object	Map of tags
  total	integer	Total number of items

• Returns {object} when Error

  The response to the caller {object}

  Name	Type	Description
  error	string	Error Message in case of failure
  result	object	Result message

Example

-- Get a list of created tags for a given solution
local out = Tsdb.listTags({limit = 10})
response.message = out

-- Use next cursor to fetch next page if found in the result of previous query
local out = Tsdb.listTags({limit = 10, next = out.next})
response.message = out

multiWrite¶

Description

Write data points to one or many metrics with an optional set of tags and a timestamp down to microsecond precision.

Note that if multiple data points are written with exactly the same timestamp, only the last one will be kept and it overwrites the others.

Each metric value has a limited size which depends on the number of tags. (Number of tags + 1) multiplies the size of metric value can't over 480KB. A write request will be rejected without partial writes if exceeding the limit. If any of data points are invalid, request will be rejected without partial writes.

To prevent from a synchronized service call take too long time to response, there are some limitations. Total number of data entries in multiple write at most 2,000 (Reference to the limit of single write). Number of data entries per datapoint: "Number of metrics * (Number of tag pairs) + 1"

If succeeds, it turns a list string of write timestamp in microseconds.

To improve write performance, the recent functionality can also be disabled by setting the recent_function_disable option to true.

Arguments

Responses

• Returns {[ object ]} when Data successfully inserted

  List of result of each data point. {object}

  Name	Type	Description
  write_timestamp	string	The timestamp of data point written to TSDB (in microseconds)

• Returns nil for Data successfully inserted

• Returns {object} when Error

  The response to the caller {object}

  Name	Type	Description
  error	string	Error Message in case of failure
  result	object	Result message

Example

-- Write multiple datapoints of metrics with tags
-- If timestamp is not provided, it will use the received time in microseconds from server side
local metrics1 = {
  temperature = 37.2,
  humidity = 73,
  switch = "on"
}
local metrics2 = {
  temperature = 31.2,
  humidity = 55,
  switch = "off"
}
local tags1 = {
  pid = "pzomp8vn4twklnmi",
  identity = "000001",
  region = "us",
  city = "minneapolis"
}
local tags2 = {
  pid = "lvwpoj19hp7k0000",
  identity = "000002",
  region = "tw",
  city = "taipei"
}
local out = Tsdb.multiWrite({
  datapoints = {
    {
      metrics = metrics1,
      tags = tags1
    },
    {
      metrics = metrics2,
      tags = tags2
    }
  },
  return_ts = true
})
response.message = out

query¶

Description

Query data points by using any metrics and tags. Support absolute (start_time, end_time) or relative (relative_start, relative_end) time parameters. The end time MUST be bigger or equal to the start time (end_time >= start_time or relative_end >= relative_start).
The first element in returned data point array is always the timestamp.
You can use fill argument to indicate the imputation of missing values.
The metric names of columns property in the response will always be in the order specified in the query, except for the timestamp column which is always the first one.

If no time constraints are specified, it will return recent data points up to the maximum limit in recent one week.

Note that only unique timestamped data will be returned, which means if multiple data points were written with exactly the same timestamp in the response, only the last one will be kept. The logic OR operator is only supported in basic query and cannot be used in down-sampling or aggregation queries.

Arguments

Responses

• Returns {object} when Error

  The response to the caller {object}

  Name	Type	Description
  error	string	Error Message in case of failure
  result	object	Result message

• Returns {object} when Query results retrieved in merge mode.

  Response data in merge mode. {object}

  Name	Type	Description
  tags	object	The “AND” operator tags specified in query arguments.
  values	[ [ number*, string, *object ] ]	The data points list.
  columns	[ string ]	The column names for mapping the column in values property.
  metrics	[ string ]	Metrics which are specified in query arguments.

• Returns {object} when Query results retrieved in split mode.

  Response data in split mode. {object}

  Name	Type	Description
  tags	object	The “AND” operator tags specified in query arguments.
  values	object	The data points list for each metric, the property name is the metric name specified in query arguments.
  values.^[a-zA-Z0-9_]+$	[ [ number*, string, *object ] ]	The data points list.
  columns	[ string ]	Its always empty array in this mode.
  metrics	[ string ]	Metrics which are specified in query arguments.

• Returns {object} when Query results retrieved in merge mode with OR operation.

  Response data in merge mode with OR operation. {object}

  Name	Type	Description
  tags	object	The “AND” operator tags specified in query arguments.
  values	object	The data points list of specific tag name, the property name is tag name.
  values.^[a-zA-Z0-9_]+$	object	The data points list of specific tag name and tag value, the property name is tag value.
  values.^[a-zA-Z0-9_]+\(.^\[a-zA-Z0-9_\]\+\)	[ [ number*, string, *object ] ]	The data points list.
  columns	[ string ]	The column names for mapping the column in values property.
  metrics	[ string ]	Metrics which are specified in query arguments.

• Returns {object} when Query results retrieved in split mode with OR operation.

  Response data in split mode with OR operation. {object}

  Name	Type	Description
  tags	object	The “AND” operator tags specified in query arguments.
  values	object	The data points list of specific tag name, the property name is tag name.
  values.^[a-zA-Z0-9_]+$	object	The data points list of specific tag name and tag value, the property name is tag value.
  values.^[a-zA-Z0-9_]+\(.^\[a-zA-Z0-9_\]\+\)	object	The data points list for each metric, the property name is the metric name specified in query arguments.
  values.[a-zA-Z0-9_]+\(.^\[a-zA-Z0-9_\]\+\).[a-zA-Z0-9_]+$	[ [ number*, string, *object ] ]	The data points list.
  columns	[ string ]	Its always empty array in this mode.
  metrics	[ string ]	Metrics which are specified in query arguments.

• Returns {object} when Query results retrieved in aggregation mode.

  Response data in split mode or aggregate funs specified in query arguments. {object}

  Name	Type	Description
  tags	object	The “AND” operator tags specified in query arguments.
  values	object	The data points list for each metric, the property name is the metric name specified in query arguments.
  values.^[a-zA-Z0-9_]+$	object	The aggregation of data points.
  values.^[a-zA-Z0-9_]+$.avg	number	The average of metric values.
  values.^[a-zA-Z0-9_]+$.max	number	The maximum of metric values.
  values.^[a-zA-Z0-9_]+$.min	number	The minimum of metric values.
  values.^[a-zA-Z0-9_]+$.sum	number	The sum of metric values.
  values.^[a-zA-Z0-9_]+$.count	number	The count of metric values.
  columns	[ string ]	Its always empty array in this mode.
  metrics	[ string ]	Metrics which are specified in query arguments.

Example

-- Query by Absolute Time Constraint --
-- Get temperature and humidity data points between 2016-08-01 (inclusive) and 2016-09-01 (exclusive) from devices in Minneapolis city and US region
local metrics = {
  "temperature",
  "switch"
}
local tags = {
  region = "us",
  city = "minneapolis"
}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  start_time = "2016-08-01T00:00:00Z",
  end_time = "2016-09-01T00:00:00Z",
  fill = "null",
  limit = 50
})
response.message = out

-- Query by Relative Time Constraint --
-- Get temperature data points in recent 3 hours from devices in Taipei city and Asia region (with timestamp in milliseconds format)
local metrics = {"temperature"}
local tags = {
  region = "asia",
  city = "taipei"
}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  relative_start = "-3h",
  epoch = "ms",
  fill = "null",
  limit = 50
})
response.message = out
-- Get temperature data points between 3 to 10 days ago from devices in Taipei city and Asia region (with timestamp in milliseconds format)
local metrics = {"temperature"}
local tags = {
  region = "asia",
  city = "taipei"
}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  relative_start = "-10d",
  relative_end = "-3d",
  epoch = "ms",
  fill = "null",
  limit = 50
})
response.message = out

-- Query without Time Constraint --
-- Get most recent 5 temperature and humidity data points from devices in Minneapolis city and US region
local metrics = {
  "temperature",
  "humidity"
}
local tags = {
  region = "us",
  city = "minneapolis"
}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  limit = 5
})
response.message = out

-- Query by Downsampling --
-- Get humidity data points in recent two days from devices in Taipei city, downsampled by 4-hours time slots
local metrics = {
  "humidity"
}
local tags = {
  city = "taipei"
}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  relative_start = "-2d",
  sampling_size = "4h",
  fill = "none",
  epoch = "ms"
})
response.message = out

-- Aggregation by Downsampling --
-- Get average and count of tire pressure data between 2016-08-15 (inclusive) and 2016-09-01 (exclusive) from devices in Minneapolis city, downsampled by 30-minutes time slots
local metrics = {
  "tire_pressure"
}
local tags = {
  city = "minneapolis"
}
local aggregate = {"avg", "count"}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  start_time = "2016-08-01T00:00:00Z",
  end_time = "2016-09-01T00:00:00Z",
  aggregate = aggregate,
  sampling_size = "30m",
  fill = "none"
})
response.message = out
-- Query by Fill in Custom String --
-- To fill in the empty time slots with "Empty" string
local metrics = {
  "temperature",
  "switch"
}
local tags = {
  region = "us",
  city = "minneapolis"
}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  fill = "~s:Empty",
})
response.message = out
-- Query by OR tags operator --
-- Get temperature data points which belongs to (sn = dev1 or sn = dev2) and dev = pump
local metrics = {
  "temperature"
} local tags = {
  dev = "pump",
  sn = {"dev1", "dev2"}
}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  start_time = "2016-08-01T00:00:00Z",
  end_time = "2016-09-01T00:00:00Z",
  limit = 50
})
response.message = out

recent¶

Description

Get the most recent data point of a particular set of metrics and tag values.

If you want to use advanced metric query, you need to specify an inner Lua table as an element
inside the Lua table of metrics parameter, i.e. metrics = {"m1","m2",{"m3","m4"}}
where {"m3","m4"} is an advanced metric query that query the most recent data
of m3 and also report the value of m4 that written together with m3.

To improve write request performance, this functionality can be disabled by setting the recent_function_disable option to true. If disabled the using the 'recent' query will return an error.

Arguments

Responses

• Returns nil for Operation successfully returned

• Returns {object} when Error

  The response to the caller {object}

  Name	Type	Description
  error	string	Error Message in case of failure
  result	object	Result message

Example

-- Get latest data of metric vibration and humidity from devices with tag sn=123 or sn=456
local out = Tsdb.recent({
  metrics = {"vibration","humidity"},
  tag_name = "sn",
  tag_values = {"123", "456"}
})
response.message = out

-- Get latest data of metric warning and critical from devices with tag sn=123 or sn=456
-- Together with the corresponding text for warning and critical metrics
local out = Tsdb.recent({
  metrics = { {"warning", "text"}, {"critical", "text"} },
  tag_name = "sn",
  tag_values = {"123", "456"}
})
response.message = out

write¶

Description

Write data point to one or many metrics with an optional set of tags and a timestamp down to microsecond precision.

Note that if multiple data points are written with exactly the same timestamp, only the last one will be kept and it overwrites the others.

Each metric value has a limited size which depends on the number of tags. (Number of tags + 1) multiplies the size of metric value can't over 480KB. A write request will be rejected without partial writes if exceeding the limit.

If succeeds, it returns a json of write timestamp in microseconds.

Important note on Tag usage: Tags applies a multiplication factor on the metric (eg. a metric with 4 tags -> 4x data write) and will have an impact on both performance and storage usage.

To improve write performance, the recent functionality can also be disabled by setting the recent_function_disable option to true.

Arguments

Responses

• Returns {object} when Data successfully inserted

  The response of write {object}

  Name	Type	Description
  write_timestamp	string	The timestamp of data point written to TSDB (in microseconds)

• Returns nil for Data successfully inserted

• Returns {object} when Error

  The response to the caller {object}

  Name	Type	Description
  error	string	Error Message in case of failure
  result	object	Result message

Example

-- Write data point of metrics with tags
-- If timestamp is not provided, it will use the received time in microseconds from server side
local metrics = {
  temperature = 37.2,
  humidity = 73,
  switch = "on",
  host = "8.168.1.24:443"
}
local tags = {
  pid = "pzomp8vn4twklnmi",
  identity = "000001",
  region = "us",
  city = "minneapolis"
}
local out = Tsdb.write({
  metrics = metrics,
  tags = tags
})
response.message = out

-- Write data points of metrics with tags and timestamp
local metrics = {
  temperature = 37.2,
  humidity = 73
}
local tags = {
  identity = "000002"
}
local out = Tsdb.write({
  metrics = metrics,
  tags = tags,
  ts = "1476243965s"
})
response.message = out

Events¶

exportJob¶

Description

An event message containing the export task result.

Arguments

Example

function handle_tsdb_exportJob (job)

 -- Your logic comes here 

end