To recap, Heroic consists of two components:
The API node is responsible for getting metrics out. It receives user traffic over HTTP, fetches metrics from the backend, and aggregations the results.
The consumer node is responsible for getting metrics in. It uses a message queue to receive metrics produced by ffwd or a similar system and writes the metrics to a durable backend. For each metric there are 3 different writes made for full API functionality. A consumer can perform all the write functions or be split up.
A typical production installation deploys the API and consumer on separate instances.
The rest of the getting started docs describes how to configure the various backends along with some best practices learned along the way.
There are 2 durable metric backends, Bigtable and Cassandra.
If you want to use Google Cloud Bigtable to store metrics, you can configure it with the following command using the the Heroic shell. This will create tables and column families.
tools/heroic-shell -P bigtable \ -X bigtable.project=<project> \ -X bigtable.instance=<instance> \ -X bigtable.credentials=default \ -X bigtable.configure ... heroic> configure
Heroic by default uses the
heroic keyspace, which can be configured using the Heroic shell.
tools/heroic-shell -P cassandra -X cassandra.seeds=<seeds> -X datastax.configure ... heroic> configure
Both the API and consumer nodes connect to Elasticsearch.
warning This is not a complete list of best practices of running Elasticsearch
When running Elasticsearch on k8s, a headless service should be configured in order for per-pod dns to be assigned and for cluster discovery to work. Sniffing should be enabled so as the k8s deployment is updated the new pods are discovered by Heroic. See #477 for more info.
Elasticsearch is also configured using the Heroic shell or having
configure: true in the config for the backend.
tools/heroic-shell -P elasticsearch-suggest -P elasticsearch-metadata -X elasticsearch.seeds=<seeds> ... heroic> configure
TODO: Add more here.
Pubsub is fully managed by Google. In order for the topics/subscriptions to be automatically created by Heroic is needs to have the proper PubSub IAM permissions to do so.