Heroic Query Language

Heroic uses a JSON-based language to define queries. This can also be expressed using an experimental DSL called HQL (Heroic Query Language).

Queries have the following structure. 


<aggregation>
  [from <source>]
  [where <filter>]
  [as <key>=<value>[,<key>=<value>]]
  [with <key>=<value>[,<key>=<value>]];

References have the following structures. 


let $<name> = <query>;

Complex Queries are queries referencing other queries through a reference. The following is an example of this 


let $a = average by host | sum by site;

$a / shift($a, -7d) - 1.0
  where what = cpu-usage and role = heroic;

The following is a complete example of a HQL-based query: 


average by host | sum by site
  from points(1d)
  where role=heroic and what=cpu-idle
  with size=5m

JSON vs HQL

JSON is typically used when a query is built programatically because the structure is unambigious in terms of precedence and escaping. There is also a ton of language support for it, and it meshes well with restful APIs.

The HQL was developed to make it easier for humans to express queries or filters in a manner which is more convenient. The language is infix, and simple strings do not have to be escaped (e.g. host vs. "host")

A primary Goal of the HQL is that it should act as a complement to the JSON queries. Any query can be expressed either in JSON or HQL.

The following is an example filter expressed both in a JSON, and in the HQL. 


$key = "hello kitty" and host = foo.example.com

["and", ["$key", "hello kitty"], ["=", "host", "foo.example.com"]]

To test this principle you can fire up the Heroic Shell and run the following commands: 


$ tools/heroic-shell

heroic> parse-query --no-indent "average by host | sum by site"
{"aggregation":{"type":"chain","chain":[{"type":"group","of":["host"],"each":{"type":"sum"}},{"type":"average"}]}}

Aggregations

Main Article: Aggregations

Aggregations are expressed as function calls, or typed documents. Aggregations can be done on both tags and resource identifiers. See the following example using the sum aggregation. 


      sum(3m)
      

      {"type": "sum", "size": "3m"}

Filters

A filter reduces the number of selected time series, if no filter (or the true filter) is used, then it is implied that all time series in the database is selected.

Available Filters

Boolean Operators

Description Parsed JSON
And Matches if all child statements match.true if empty. 
<a> and <b>
 
["and", <a>, <b>]
Or Matches if any child statements match.false if empty. 
<a> or <b>
 
["or", <a>, <b>]
Not Matches if child statement does not match. 
!<a>
 
["not", <a>]
Override Grouping Override grouping. 
(<a> and <b>) and <c>
 
["and", ["and", <a>, <b>], <c>]

Operators

Description Parsed JSON
Tag Matches Tag foo equals to bar. 
<foo> = <bar>
 
["=", <foo>, <bar>]
Tag Does Not Match Tag foo does not equal to bar. 
<foo> != <bar>
 
["not", ["=", <foo>, <bar>]]
Tag In Tag foo equals to either bar or baz. 
<foo> in [<bar>, <baz>]
 
["or", ["=", <foo>, <bar>], ["=", <foo>, <baz>]]
Tag Not In Tag foo does not equal to either bar or baz. 
<foo> not in [<bar>, <baz>]
 
["not", ["or", ["=", <foo>, <bar>], ["=", <foo>, <baz>]]]
Tag Exists Tag foo exists. 
+<foo>
 
["+", <foo>]
Tag Does Not Exists Tag foo does not exist. 
!+<foo>
 
["not", ["+", <foo>]]
Tag Prefixed With Tag foo is prefixed with bar. 
<foo> ^ <bar>
 
["^", <foo>, <bar>]
Tag Not Prefixed With Tag foo is not prefixed with bar. 
<foo> !^ <bar>
 
["not", ["^", <foo>, <bar>]]
True True. 
true
 
true
False False. 
false
 
false
RegEx On Tag Regular expressions on tags. 
<a> ~ <b>
 
["~", <a>, <b>]
Custom Filters Details in the next section. 
q <a>
 
["q", <a>]

Custom Filters

When you need to select series using a more complex syntax than that available through the basic tag suggestions, you can create custom filters. These can take advantage of the HQL format of the above filters.

Some examples:

Suppose you want to look at unsuccessful incoming requests. How do we select the unsuccessful ones from all the other incoming requests? One way is to create a custom filter like this:

result in ["fail", "drop"]

Suppose you want to exclude a single host from your query. The way to accomplish this is to use a custom filter like this:

host != "myhost.example.net"

If you want to exclude more than one host, you could use a not in expression to exclude multiple values for the same tag, like this:

host not in ["myhost1.example.net", "myhost2.example.net"]

The HQL Language

Primitives

Simple String String made up of a limited set of characters for convenience. It must not match a parsed keyword. 
hello.world
Quoted String String which is quoted to support any set of characters. Supports the same escape sequences as Java. 
"hello world"

Arithmetic Expressions

Addition 
<a> + <b>
 Valid operands are: <string> + <string>, and <number> + <number>.
Subtraction 
<a> - <b>
 Valid operands are: <number> - <number>.

Special Variables

$key Can be used in most places a tag is expected. It indicates that the given expression should match the special field key instead of a tag.
$now Expands to the current (server-side) timestamp in milliseconds.

Durations

Durations are represented as a numeric component with a suffix, like 3H

Valid suffixes are:

  • ms - for milliseconds
  • s - for seconds
  • m - for minutes
  • H - for hours
  • d - for days
  • w - for weeks

Durations support arithmetic expressions with each other and numbers.

The following is a valid expression: 


$now - (1d + 1H)