Styx API specification
This document is written using api-blueprint
Usage
-
install dredd:
npm install -g dredd
-
package project:
mvn package
-
run tests:
styx-api-service/bin/api-test.sh
To generate an html version of this spec, use aglio
aglio -i doc/api.apib -o api.html --theme-template triple --theme-variables streak
Authentication
Starting from V3, any request except GET
requires Google Authentication.
HTTP header Authorization: Bearer <id_token>
with a valid Google ID token is required
in those requests.
To know more details, please check Google documentation.
Authorization
If authorization is enabled for the Styx service, any workflow with a service_account
configured can only be updated by an authorized user of that service_account
.
Updates include deploying, enabling/disabling and triggering the workflow.
In order to authorize an account, you need to grant it the configured role
for the workflow service_account
. For more information, see Granting Roles to Service Accounts.
Workflows ¶
Resources related to the Workflows registered in Styx.
Workflows ¶
Headers
Content-Type: application/json
Body
[
{
"component_id": "styx-canary",
"workflow_id": "StyxCanary",
"component_uri": "file:///etc/styx/schedule.yaml",
"configuration": {
"id": "StyxCanary",
"schedule": "hours",
"offset": null,
"docker_image": null,
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_termination_logging": false,
"env": {
"FOO": "bar"
},
"resources": []
}
}
]
Headers
Content-Type: application/json
Body
[
{
"workflow": {
"component_id": "styx-canary",
"workflow_id": "StyxCanary",
"component_uri": "file:///etc/styx/schedule.yaml",
"configuration": {
"id": "StyxCanary",
"schedule": "hours",
"offset": null,
"docker_image": null,
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_termination_logging": false,
"env": {
"FOO": "bar"
},
"resources": [],
"running_timeout": "PT2H"
}
},
"state": {
"enabled": "true",
"next_natural_trigger": "2017-01-01T01:00:00Z",
"next_natural_offset_trigger": "2017-01-01T02:00:00Z"
}
}
]
Get WorkflowsGET/{version}/workflows
- version
string
(required) Example: v3API version
Choices:
v3
- full
boolean
(optional) Default: false Example: trueReturn the state of the workflow
Workflows ¶
Headers
Content-Type: application/json
Body
[
{
"component_id": "styx-canary",
"workflow_id": "StyxCanary",
"component_uri": "file:///etc/styx/schedule.yaml",
"configuration": {
"id": "StyxCanary",
"schedule": "hours",
"offset": null,
"docker_image": null,
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_termination_logging": false,
"env": {
"FOO": "bar"
},
"resources": []
}
}
]
Get Workflows for componentGET/{version}/workflows/{component_id}
- version
string
(required) Example: v3API version
Choices:
v3
- component_id
string
(required) Example: styx-canaryWorkflow Component
Headers
Content-Type: application/json
Body
{
"id": "StyxCanary",
"schedule": "days",
"offset": null,
"docker_image": "styx-canary-dummy:dummy",
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_termination_logging": false,
"env": {
"FOO": "bar"
},
"service_account": null,
"resources": [],
"running_timeout": "PT2H",
"retry_condition": "#tries < 3"
}
Headers
Content-Type: application/json
Body
{
"component_id": "styx-canary",
"workflow_id": "StyxCanary",
"component_uri": "file:///etc/styx/schedule.yaml",
"configuration": {
"id": "StyxCanary",
"schedule": "days",
"offset": null,
"docker_image": "styx-canary-dummy:dummy",
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_termination_logging": false,
"env": {
"FOO": "bar"
},
"service_account": null,
"resources": [],
"running_timeout": "PT2H",
"retry_condition": "#tries < 3"
},
"__from_api": "V3"
}
Create or update WorkflowPOST/{version}/workflows/{component_id}
Accepts a Workflow definition as a JSON object. Returns 200 OK if the Workflow was registered or updated. Note that a non-null value is required for docker_image.
- version
string
(required) Example: v3API version
Choices:
v3
- component_id
string
(required) Example: styx-canaryWorkflow Component
Workflow ¶
Headers
Content-Type: application/json
Body
{
"component_id": "styx-canary",
"workflow_id": "StyxCanary",
"component_uri": "file:///etc/styx/schedule.yaml",
"configuration": {
"id": "StyxCanary",
"schedule": "hours",
"offset": null,
"docker_image": null,
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_termination_logging": false,
"env": {
"FOO": "bar"
},
"resources": [],
"running_timeout": "PT2H",
"retry_condition": "#tries < 3"
}
}
Get WorkflowGET/{version}/workflows/{component_id}/{workflow_id}
- version
string
(required) Example: v3API version
Choices:
v3
- component_id
string
(required) Example: styx-canaryWorkflow Component
- workflow_id
string
(required) Example: StyxCanaryWorkflow ID
Delete WorkflowDELETE/{version}/workflows/{component_id}/{workflow_id}
Returns 204 No Content if the Workflow was found and deleted, 404 Not Found otherwise.
- version
string
(required) Example: v3API version
Choices:
v3
- component_id
string
(required) Example: styx-canaryWorkflow Component
- workflow_id
string
(required) Example: StyxCanaryWorkflow ID
Workflow with state ¶
Headers
Content-Type: application/json
Body
{
"workflow": {
"component_id": "styx-canary",
"workflow_id": "StyxCanary",
"component_uri": "file:///etc/styx/schedule.yaml",
"configuration": {
"id": "StyxCanary",
"schedule": "hours",
"offset": null,
"docker_image": null,
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_termination_logging": false,
"env": {
"FOO": "bar"
},
"resources": [],
"running_timeout": "PT2H"
}
},
"state": {
"enabled": "true",
"next_natural_trigger": "2017-01-01T01:00:00Z",
"next_natural_offset_trigger": "2017-01-01T02:00:00Z"
}
}
Get Workflow with StateGET/{version}/workflows/{component_id}/{workflow_id}/full
- version
string
(required) Example: v3API version
Choices:
v3
- component_id
string
(required) Example: styx-canaryWorkflow Component
- workflow_id
string
(required) Example: StyxCanaryWorkflow ID
Workflow Instances ¶
Query can be done in three different styles: offset+limit
, start+stop
or tail+limit
.
By using offset
and limit
, the oldest workflow instances is specified by offset
, and
the max number of workflows instances is specified by limit
.
By using start
and stop
, the oldest workflow instance is specified by start
, and the latest is
specified by stop
; start
is required while stop
is optional.
When using tail
and limit
, the newest (naturally triggered) instances are returned. The maxium number of instances
returned is specified by limit
but fewer instances may be returned even though more instances exist.
Headers
Content-Type: application/json
Body
[
{
"workflow_instance": {
"workflow": {
"component_id": "styx-canary",
"id": "StyxCanary"
},
"parameter": "2016-07-13T08"
},
"triggers": [
{
"trigger_id": "UNKNOWN",
"timestamp": "2016-07-13T09:00:23.941Z",
"complete": true,
"executions": [
{
"execution_id": "styx-run-qcp3r",
"docker_image": "UNKNOWN",
"commit_sha": "19c1a6",
"runner_id": "production",
"statuses": [
{
"timestamp": "2016-07-13T09:00:45.882Z",
"status": "SUBMITTED"
},
{
"timestamp": "2016-07-13T09:00:49.076Z",
"status": "STARTED"
},
{
"timestamp": "2016-07-13T09:00:49.313Z",
"status": "SUCCESS"
}
]
}
]
}
]
}
]
Get Workflow InstancesGET/{version}/workflows/{component_id}/{workflow_id}/instances{?offset,limit,tail,start,stop}
- version
string
(required) Example: v3API version
Choices:
v3
- component_id
string
(required) Example: styx-canaryWorkflow Component
- workflow_id
string
(required) Example: StyxCanaryWorkflow ID
- offset
string
(required) Example: 2017-01-01the offset parameter
- limit
number
(required) Example: 10max number of instances
- start
string
(required) Example: 2017-01-01the start parameter
- stop
string
(required) Example: 2017-01-03the stop parameter (exclusive)
- tail
boolean
(optional) Example: trueReturn the newest workflow instances?
Workflow Instance ¶
Headers
Content-Type: application/json
Body
{
"workflow_instance": {
"workflow": {
"component_id": "styx-canary",
"id": "StyxCanary"
},
"parameter": "2016-07-13T08"
},
"triggers": [
{
"trigger_id": "UNKNOWN",
"timestamp": "2016-07-13T09:00:23.941Z",
"complete": true,
"executions": [
{
"execution_id": "styx-run-qcp3r",
"docker_image": "UNKNOWN",
"commit_sha": "19c1a6",
"runner_id": "production",
"statuses": [
{
"timestamp": "2016-07-13T09:00:45.882Z",
"status": "SUBMITTED"
},
{
"timestamp": "2016-07-13T09:00:49.076Z",
"status": "STARTED"
},
{
"timestamp": "2016-07-13T09:00:49.313Z",
"status": "SUCCESS"
}
]
}
]
}
]
}
Get Workflow InstanceGET/{version}/workflows/{component_id}/{workflow_id}/instances/{instance}
- version
string
(required) Example: v3API version
Choices:
v3
- component_id
string
(required) Example: styx-canaryWorkflow Component
- workflow_id
string
(required) Example: StyxCanaryWorkflow ID
- instance
string
(required) Example: 2016-07-13T08Worfklow Instance parameter
Workflow state ¶
Headers
Content-Type: application/json
Body
{
"enabled": "true",
"next_natural_trigger": "2017-01-01T01:00:00Z",
"next_natural_offset_trigger": "2017-01-01T02:00:00Z"
}
Get Workflow stateGET/{version}/workflows/{component_id}/{workflow_id}/state
- version
string
(required) Example: v3API version
Choices:
v3
- component_id
string
(required) Example: styx-canaryWorkflow Component
- workflow_id
string
(required) Example: StyxCanaryWorkflow ID
Headers
Content-Type: application/json
Body
{
"enabled": "false",
"next_natural_trigger": "2017-01-01T01:00:00Z",
"next_natural_offset_trigger": "2017-01-01T02:00:00Z"
}
Headers
Content-Type: application/json
Body
{
"enabled": "false",
"next_natural_trigger": "2017-01-01T01:00:00Z",
"next_natural_offset_trigger": "2017-01-01T02:00:00Z"
}
Set Workflow statePATCH/{version}/workflows/{component_id}/{workflow_id}/state
- version
string
(required) Example: v3API version
Choices:
v3
- component_id
string
(required) Example: styx-canaryWorkflow Component
- workflow_id
string
(required) Example: StyxCanaryWorkflow ID
Resources ¶
Resources related to Styx Resources registered in the scheduler. Styx Resources can be used to limit the maximum concurrent executions for certain Workflows.
Resources ¶
Headers
Content-Type: application/json
Body
{
"resources": [
{
"concurrency": 20,
"id": "styx-canary-resource"
}
]
}
Get ResourcesGET/{version}/resources
- version
string
(required) Example: v3API version
Choices:
v3
Headers
Content-Type: application/json
Body
{
"concurrency": "20",
"id": "styx-canary-resource"
}
Headers
Content-Type: application/json
Body
{
"concurrency": 20,
"id": "styx-canary-resource"
}
Create ResourcePOST/{version}/resources
- version
string
(required) Example: v3API version
Choices:
v3
Resource ¶
Headers
Content-Type: application/json
Body
{
"resources": [
{
"concurrency": 20,
"id": "styx-canary-resource"
}
]
}
Get ResourceGET/{version}/resources/{resource}
- version
string
(required) Example: v3API version
Choices:
v3
- resource
string
(required) Example: styx-canary-resourceResource ID
Delete ResourceDELETE/{version}/resources/{resource}
- version
string
(required) Example: v3API version
Choices:
v3
- resource
string
(required) Example: styx-canary-resourceResource ID
Headers
Content-Type: application/json
Body
{
"concurrency": "30",
"id": "styx-canary-resource"
}
Headers
Content-Type: application/json
Body
{
"concurrency": 30,
"id": "styx-canary-resource"
}
Modify ResourcePUT/{version}/resources/{resource}
- version
string
(required) Example: v3API version
Choices:
v3
- resource
string
(required) Example: styx-canary-resourceResource ID
Backfills ¶
Resources related to the Backfills in Styx. This group allows to operate on Backfill entities registered in Styx and check detailed status of each single Workflow Instance belonging to a Backfill entity.
Backfills ¶
Headers
Content-Type: application/json
Body
{
"backfills": [
{
"backfill": {
"all_triggered": true,
"concurrency": 10,
"end": "2017-01-01T01:00:00Z",
"halted": false,
"id": "backfill-1489054446085-53384",
"next_trigger": "2017-01-01T01:00:00Z",
"schedule": "hours",
"start": "2017-01-01T00:00:00Z",
"trigger_parameters": {
"env": {
"FOO": "bar"
}
},
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
},
"created": "2017-01-01T00:00:00Z",
"last_modified": "2019-01-01T00:00:00Z"
},
"statuses": {
"active_states": [
{
"initial_timestamp": 1536838486145,
"latest_timestamp": 1536838486185,
"state": "DONE",
"state_data": {
"consecutive_failures": 0,
"execution_description": {
"commit_sha": "f043333085fa87738ac24f04d64fb58ecc845111",
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_image": "styx-canary-dummy:dummy",
"docker_termination_logging": false
},
"execution_id": "styx-run-6b1962c6-23ba-4245-a13b-ec3baa4b2133",
"last_exit": 0,
"messages": [
{
"level": "INFO",
"line": "Exit code: 0"
}
],
"retry_cost": 0,
"retry_delay_millis": null,
"tries": 1,
"trigger": {
"@type": "backfill",
"trigger_id": "backfill-1489054446085-53384"
},
"trigger_id": "backfill-1489054446085-53384"
},
"workflow_instance": {
"parameter": "2017-01-01T00",
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
}
}
}
]
}
}
]
}
Get BackfillsGET/{version}/backfills{?component,workflow,status,showAll}
- version
string
(required) Example: v3API version
Choices:
v3
- component
string
(optional) Example: styx-canaryWorkflow Component to filter with
- workflow
string
(optional) Example: StyxCanaryWorkflow ID to filter with
- status
boolean
(optional) Default: false Example: trueIf to return the status list per backfill
- showAll
boolean
(optional) Default: false Example: trueIf to return also halted backfills and backfills whose instances have been all triggered
Headers
Content-Type: application/json
Body
{
"start": "2017-01-01T00:00:00Z",
"end": "2017-01-02T00:00:00Z",
"component": "styx-canary",
"concurrency": "10",
"reverse": true,
"description": "fixed bug and reproduce data",
"trigger_parameters": {"env": {"FOO": "bar"}},
"workflow": "StyxCanary",
}
Headers
Content-Type: application/json
Body
{
"all_triggered": false,
"concurrency": 10,
"end": "2017-02-00T00:00:00Z",
"description": "fixed bug and reproduce data",
"halted": false,
"id": "backfill-1489054446085-52684",
"next_trigger": "2017-01-01T00:00:00Z",
"schedule": "hours",
"start": "2017-01-01T00:00:00Z",
"trigger_parameters": {"env": {"FOO": "bar"}},
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
},
"created": "2017-01-01T00:00:00Z",
"last_modified": "2019-01-01T00:00:00Z",
}
Create BackfillPOST/{version}/backfills{?allowFuture}
- version
string
(required) Example: v3API version
Choices:
v3
- allowFuture
boolean
(optional) Default: false Example: falseallow backfilling future partitions
Backfill ¶
Headers
Content-Type: application/json
Body
{
"backfill": {
"all_triggered": true,
"concurrency": 10,
"description": "fixed bug and reproduce data",
"end": "2017-01-01T01:00:00Z",
"halted": false,
"id": "backfill-1489054446085-53384",
"next_trigger": "2017-01-01T01:00:00Z",
"schedule": "hours",
"start": "2017-01-01T00:00:00Z",
"trigger_parameters": {"env": {"FOO": "bar"}},
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
},
"created": "2017-01-01T00:00:00Z",
"last_modified": "2019-01-01T00:00:00Z",
},
"statuses": {
"active_states": [{
"initial_timestamp": 1536838486145,
"latest_timestamp": 1536838486185,
"state": "DONE",
"state_data": {
"consecutive_failures": 0,
"execution_description": {
"commit_sha": "f043333085fa87738ac24f04d64fb58ecc845111",
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_image": "styx-canary-dummy:dummy",
"docker_termination_logging": false
},
"execution_id": "styx-run-6b1962c6-23ba-4245-a13b-ec3baa4b2133",
"last_exit": 0,
"messages": [
{
"level": "INFO",
"line": "Exit code: 0"
}
],
"retry_cost": 0.0,
"retry_delay_millis": null,
"tries": 1,
"trigger": {
"@type": "backfill",
"trigger_id": "backfill-1489054446085-53384"
},
"trigger_id": "backfill-1489054446085-53384"
},
"workflow_instance": {
"parameter": "2017-01-01T00",
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
}
}
}]
}
}
Get BackfillGET/{version}/backfills/{backfill_id}{?status}
- version
string
(required) Example: v3API version
Choices:
v3
- backfill_id
string
(required) Example: backfill-1489054446085-52684Backfill ID
- status
boolean
(optional) Default: true Example: trueIf to return the status list per backfill
Headers
Content-Type: application/json
Body
{
"concurrency": 20,
"description": "fixed bug and reproduce data",
"id": "backfill-1489054446085-52684",
}
Headers
Content-Type: application/json
Body
{
"all_triggered": false,
"concurrency": 20,
"end": "2017-02-00T00:00:00Z",
"description": "fixed bug and reproduce data",
"halted": false,
"id": "backfill-1489054446085-52684",
"next_trigger": "2017-01-01T00:00:00Z",
"schedule": "hours",
"start": "2017-01-01T00:00:00Z",
"trigger_parameters": {"env": {"FOO": "bar"}},
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
},
"created": "2017-01-01T00:00:00Z",
"last_modified": "2019-01-01T00:00:00Z",
}
Modify BackfillPUT/{version}/backfills/{backfill_id}
- version
string
(required) Example: v3API version
Choices:
v3
- backfill_id
string
(required) Example: backfill-1489054446085-52684Backfill ID
Delete BackfillDELETE/{version}/backfills/{backfill_id}
- version
string
(required) Example: v3API version
Choices:
v3
- backfill_id
string
(required) Example: backfill-1489054446085-52684Backfill ID
Status ¶
Resources related to inspect detailed status of Workflow Instances.
Active States ¶
Headers
Content-Type: application/json
Body
{
"active_states": [
{
"latest_timestamp": 1536838486185,
"state": "RUNNING",
"state_data": {
"consecutive_failures": 0,
"execution_description": {
"commit_sha": "f043333085fa87738ac24f04d64fb58ecc845111",
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_image": "styx-canary-dummy:dummy",
"docker_termination_logging": false
},
"execution_id": "styx-run-6b1962c6-23ba-4245-a13b-ec3baa4b21d5",
"last_exit": 20,
"messages": [
{
"level": "WARNING",
"line": "Exit code: 20"
}
],
"retry_cost": 0.2,
"retry_delay_millis": 600000,
"tries": 2,
"trigger": {
"@type": "natural"
},
"trigger_id": "natural-trigger"
},
"workflow_instance": {
"parameter": "2017-01-01T01",
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
}
}
}
]
}
Get Active StatesGET/{version}/status/activeStates{?component,workflow,components}
- version
string
(required) Example: v3API version
Choices:
v3
- component
string
(optional) Example: styx-canaryComponent ID
- workflow
string
(optional) Example: StyxCanaryWorkflow ID - requires Component ID to also be specified
- components
string
(optional) Example: styx-canary,styx-canary1,styx-canary3Component IDs, IDs should be comma separated. If provided, two other parameters will be ignored.
Events ¶
Headers
Content-Type: application/json
Body
{
"events": [
{
"event": {
"@type": "triggerExecution",
"trigger": {
"@type": "natural"
},
"workflow_instance": "styx-canary/StyxCanary/2017-01-01T01",
"parameters": {
"env": {
"FOO": "bar"
}
}
},
"timestamp": 1483228800000
},
{
"event": {
"@type": "dequeue",
"workflow_instance": "styx-canary/StyxCanary/2017-01-01T01"
},
"timestamp": 1483228800005
}
]
}
Get Workflow Instance eventsGET/{version}/status/events/{component_id}/{workflow_id}/{instance}
- version
string
(required) Example: v3API version
Choices:
v3
- component_id
string
(required) Example: styx-canaryWorkflow Component
- workflow_id
string
(required) Example: StyxCanaryWorkflow ID
- instance
string
(required) Example: 2017-01-01T01Worfklow Instance parameter
Scheduler ¶
Resources related to operations that act on the current state of the scheduler.
Retry ¶
Headers
Content-Type: application/json
Body
{
"workflow_instance": "styx-canary/StyxCanary/2017-01-01T01"
}
Headers
Content-Type: application/json
Body
{
"workflow_instance": "styx-canary/StyxCanary/2017-01-01T01"
}
Send RetryPOST/{version}/scheduler/retry{?delay}
- version
string
(required) Example: v3API version
Choices:
v3
- delay
long
(optional) Default: 0 Example: 0The new delay value to set for the workflow instance
Halt ¶
Headers
Content-Type: application/json
Body
{
"workflow_instance": "styx-canary/StyxCanary/2017-01-01T01"
}
Headers
Content-Type: application/json
Body
{
"workflow_instance": "styx-canary/StyxCanary/2017-01-01T01"
}
Send HaltPOST/{version}/scheduler/halt
- version
string
(required) Example: v3API version
Choices:
v3
Trigger ¶
Headers
Content-Type: application/json
Body
{
"parameter": "2017-01-01T01",
"trigger_parameters": {
"env": {
"FOO": "bar"
}
},
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
}
}
Headers
Content-Type: application/json
Body
{
"parameter": "2017-01-01T01",
"trigger_parameters": {
"env": {
"FOO": "bar"
}
},
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
},
"trigger_id": "UNKNOWN"
}
Trigger Workflow InstancePOST/{version}/scheduler/trigger{?allowFuture}
- version
string
(required) Example: v3API version
Choices:
v3
- allowFuture
boolean
(optional) Default: false Example: falseallow triggering future partition
Styx Config ¶
Resources related to global configurations for Styx.
Global Enabled ¶
Headers
Content-Type: application/json
Body
{
"enabled": "true"
}
Get Global EnabledGET/{version}/config
- version
string
(required) Example: v3API version
Choices:
v3
Headers
Content-Type: application/json
Body
{
"enabled": "true"
}
Set Global EnabledPOST/{version}/config{?enabled}
- version
string
(required) Example: v3API version
Choices:
v3
- enabled
boolean
(required) Example: trueGlobal enabled value
Generated by aglio on 12 Apr 2023