Styx API specification
This document is written using api-blueprint
Usage
-
install dredd:
npm install -g dredd -
package project:
mvn package -
run tests:
styx-api-service/bin/api-test.sh
To generate an html version of this spec, use aglio
aglio -i doc/api.apib -o api.html --theme-template triple --theme-variables streak
Authentication
Starting from V3, any request except GET requires Google Authentication.
HTTP header Authorization: Bearer <id_token> with a valid Google ID token is required
in those requests.
To know more details, please check Google documentation.
Authorization
If authorization is enabled for the Styx service, any workflow with a service_account
configured can only be updated by an authorized user of that service_account.
Updates include deploying, enabling/disabling and triggering the workflow.
In order to authorize an account, you need to grant it the configured role
for the workflow service_account. For more information, see Granting Roles to Service Accounts.
Workflows ¶
Resources related to the Workflows registered in Styx.
Workflows ¶
Headers
Content-Type: application/jsonBody
[
{
"component_id": "styx-canary",
"workflow_id": "StyxCanary",
"component_uri": "file:///etc/styx/schedule.yaml",
"configuration": {
"id": "StyxCanary",
"schedule": "hours",
"offset": null,
"docker_image": null,
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_termination_logging": false,
"env": {
"FOO": "bar"
},
"resources": []
}
}
]Headers
Content-Type: application/jsonBody
[
{
"workflow": {
"component_id": "styx-canary",
"workflow_id": "StyxCanary",
"component_uri": "file:///etc/styx/schedule.yaml",
"configuration": {
"id": "StyxCanary",
"schedule": "hours",
"offset": null,
"docker_image": null,
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_termination_logging": false,
"env": {
"FOO": "bar"
},
"resources": [],
"running_timeout": "PT2H"
}
},
"state": {
"enabled": "true",
"next_natural_trigger": "2017-01-01T01:00:00Z",
"next_natural_offset_trigger": "2017-01-01T02:00:00Z"
}
}
]Get WorkflowsGET/{version}/workflows
- version
string(required) Example: v3API version
Choices:
v3- full
boolean(optional) Default: false Example: trueReturn the state of the workflow
Workflows ¶
Headers
Content-Type: application/jsonBody
[
{
"component_id": "styx-canary",
"workflow_id": "StyxCanary",
"component_uri": "file:///etc/styx/schedule.yaml",
"configuration": {
"id": "StyxCanary",
"schedule": "hours",
"offset": null,
"docker_image": null,
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_termination_logging": false,
"env": {
"FOO": "bar"
},
"resources": []
}
}
]Get Workflows for componentGET/{version}/workflows/{component_id}
- version
string(required) Example: v3API version
Choices:
v3- component_id
string(required) Example: styx-canaryWorkflow Component
Headers
Content-Type: application/jsonBody
{
"id": "StyxCanary",
"schedule": "days",
"offset": null,
"docker_image": "styx-canary-dummy:dummy",
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_termination_logging": false,
"env": {
"FOO": "bar"
},
"service_account": null,
"resources": [],
"running_timeout": "PT2H",
"retry_condition": "#tries < 3"
}Headers
Content-Type: application/jsonBody
{
"component_id": "styx-canary",
"workflow_id": "StyxCanary",
"component_uri": "file:///etc/styx/schedule.yaml",
"configuration": {
"id": "StyxCanary",
"schedule": "days",
"offset": null,
"docker_image": "styx-canary-dummy:dummy",
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_termination_logging": false,
"env": {
"FOO": "bar"
},
"service_account": null,
"resources": [],
"running_timeout": "PT2H",
"retry_condition": "#tries < 3"
},
"__from_api": "V3"
}Create or update WorkflowPOST/{version}/workflows/{component_id}
Accepts a Workflow definition as a JSON object. Returns 200 OK if the Workflow was registered or updated. Note that a non-null value is required for docker_image.
- version
string(required) Example: v3API version
Choices:
v3- component_id
string(required) Example: styx-canaryWorkflow Component
Workflow ¶
Headers
Content-Type: application/jsonBody
{
"component_id": "styx-canary",
"workflow_id": "StyxCanary",
"component_uri": "file:///etc/styx/schedule.yaml",
"configuration": {
"id": "StyxCanary",
"schedule": "hours",
"offset": null,
"docker_image": null,
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_termination_logging": false,
"env": {
"FOO": "bar"
},
"resources": [],
"running_timeout": "PT2H",
"retry_condition": "#tries < 3"
}
}Get WorkflowGET/{version}/workflows/{component_id}/{workflow_id}
- version
string(required) Example: v3API version
Choices:
v3- component_id
string(required) Example: styx-canaryWorkflow Component
- workflow_id
string(required) Example: StyxCanaryWorkflow ID
Delete WorkflowDELETE/{version}/workflows/{component_id}/{workflow_id}
Returns 204 No Content if the Workflow was found and deleted, 404 Not Found otherwise.
- version
string(required) Example: v3API version
Choices:
v3- component_id
string(required) Example: styx-canaryWorkflow Component
- workflow_id
string(required) Example: StyxCanaryWorkflow ID
Workflow with state ¶
Headers
Content-Type: application/jsonBody
{
"workflow": {
"component_id": "styx-canary",
"workflow_id": "StyxCanary",
"component_uri": "file:///etc/styx/schedule.yaml",
"configuration": {
"id": "StyxCanary",
"schedule": "hours",
"offset": null,
"docker_image": null,
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_termination_logging": false,
"env": {
"FOO": "bar"
},
"resources": [],
"running_timeout": "PT2H"
}
},
"state": {
"enabled": "true",
"next_natural_trigger": "2017-01-01T01:00:00Z",
"next_natural_offset_trigger": "2017-01-01T02:00:00Z"
}
}Get Workflow with StateGET/{version}/workflows/{component_id}/{workflow_id}/full
- version
string(required) Example: v3API version
Choices:
v3- component_id
string(required) Example: styx-canaryWorkflow Component
- workflow_id
string(required) Example: StyxCanaryWorkflow ID
Workflow Instances ¶
Query can be done in three different styles: offset+limit, start+stop or tail+limit.
By using offset and limit, the oldest workflow instances is specified by offset, and
the max number of workflows instances is specified by limit.
By using start and stop, the oldest workflow instance is specified by start, and the latest is
specified by stop; start is required while stop is optional.
When using tail and limit, the newest (naturally triggered) instances are returned. The maxium number of instances
returned is specified by limit but fewer instances may be returned even though more instances exist.
Headers
Content-Type: application/jsonBody
[
{
"workflow_instance": {
"workflow": {
"component_id": "styx-canary",
"id": "StyxCanary"
},
"parameter": "2016-07-13T08"
},
"triggers": [
{
"trigger_id": "UNKNOWN",
"timestamp": "2016-07-13T09:00:23.941Z",
"complete": true,
"executions": [
{
"execution_id": "styx-run-qcp3r",
"docker_image": "UNKNOWN",
"commit_sha": "19c1a6",
"runner_id": "production",
"statuses": [
{
"timestamp": "2016-07-13T09:00:45.882Z",
"status": "SUBMITTED"
},
{
"timestamp": "2016-07-13T09:00:49.076Z",
"status": "STARTED"
},
{
"timestamp": "2016-07-13T09:00:49.313Z",
"status": "SUCCESS"
}
]
}
]
}
]
}
]Get Workflow InstancesGET/{version}/workflows/{component_id}/{workflow_id}/instances{?offset,limit,tail,start,stop}
- version
string(required) Example: v3API version
Choices:
v3- component_id
string(required) Example: styx-canaryWorkflow Component
- workflow_id
string(required) Example: StyxCanaryWorkflow ID
- offset
string(required) Example: 2017-01-01the offset parameter
- limit
number(required) Example: 10max number of instances
- start
string(required) Example: 2017-01-01the start parameter
- stop
string(required) Example: 2017-01-03the stop parameter (exclusive)
- tail
boolean(optional) Example: trueReturn the newest workflow instances?
Workflow Instance ¶
Headers
Content-Type: application/jsonBody
{
"workflow_instance": {
"workflow": {
"component_id": "styx-canary",
"id": "StyxCanary"
},
"parameter": "2016-07-13T08"
},
"triggers": [
{
"trigger_id": "UNKNOWN",
"timestamp": "2016-07-13T09:00:23.941Z",
"complete": true,
"executions": [
{
"execution_id": "styx-run-qcp3r",
"docker_image": "UNKNOWN",
"commit_sha": "19c1a6",
"runner_id": "production",
"statuses": [
{
"timestamp": "2016-07-13T09:00:45.882Z",
"status": "SUBMITTED"
},
{
"timestamp": "2016-07-13T09:00:49.076Z",
"status": "STARTED"
},
{
"timestamp": "2016-07-13T09:00:49.313Z",
"status": "SUCCESS"
}
]
}
]
}
]
}Get Workflow InstanceGET/{version}/workflows/{component_id}/{workflow_id}/instances/{instance}
- version
string(required) Example: v3API version
Choices:
v3- component_id
string(required) Example: styx-canaryWorkflow Component
- workflow_id
string(required) Example: StyxCanaryWorkflow ID
- instance
string(required) Example: 2016-07-13T08Worfklow Instance parameter
Workflow state ¶
Headers
Content-Type: application/jsonBody
{
"enabled": "true",
"next_natural_trigger": "2017-01-01T01:00:00Z",
"next_natural_offset_trigger": "2017-01-01T02:00:00Z"
}Get Workflow stateGET/{version}/workflows/{component_id}/{workflow_id}/state
- version
string(required) Example: v3API version
Choices:
v3- component_id
string(required) Example: styx-canaryWorkflow Component
- workflow_id
string(required) Example: StyxCanaryWorkflow ID
Headers
Content-Type: application/jsonBody
{
"enabled": "false",
"next_natural_trigger": "2017-01-01T01:00:00Z",
"next_natural_offset_trigger": "2017-01-01T02:00:00Z"
}Headers
Content-Type: application/jsonBody
{
"enabled": "false",
"next_natural_trigger": "2017-01-01T01:00:00Z",
"next_natural_offset_trigger": "2017-01-01T02:00:00Z"
}Set Workflow statePATCH/{version}/workflows/{component_id}/{workflow_id}/state
- version
string(required) Example: v3API version
Choices:
v3- component_id
string(required) Example: styx-canaryWorkflow Component
- workflow_id
string(required) Example: StyxCanaryWorkflow ID
Resources ¶
Resources related to Styx Resources registered in the scheduler. Styx Resources can be used to limit the maximum concurrent executions for certain Workflows.
Resources ¶
Headers
Content-Type: application/jsonBody
{
"resources": [
{
"concurrency": 20,
"id": "styx-canary-resource"
}
]
}Get ResourcesGET/{version}/resources
- version
string(required) Example: v3API version
Choices:
v3
Headers
Content-Type: application/jsonBody
{
"concurrency": "20",
"id": "styx-canary-resource"
}Headers
Content-Type: application/jsonBody
{
"concurrency": 20,
"id": "styx-canary-resource"
}Create ResourcePOST/{version}/resources
- version
string(required) Example: v3API version
Choices:
v3
Resource ¶
Headers
Content-Type: application/jsonBody
{
"resources": [
{
"concurrency": 20,
"id": "styx-canary-resource"
}
]
}Get ResourceGET/{version}/resources/{resource}
- version
string(required) Example: v3API version
Choices:
v3- resource
string(required) Example: styx-canary-resourceResource ID
Delete ResourceDELETE/{version}/resources/{resource}
- version
string(required) Example: v3API version
Choices:
v3- resource
string(required) Example: styx-canary-resourceResource ID
Headers
Content-Type: application/jsonBody
{
"concurrency": "30",
"id": "styx-canary-resource"
}Headers
Content-Type: application/jsonBody
{
"concurrency": 30,
"id": "styx-canary-resource"
}Modify ResourcePUT/{version}/resources/{resource}
- version
string(required) Example: v3API version
Choices:
v3- resource
string(required) Example: styx-canary-resourceResource ID
Backfills ¶
Resources related to the Backfills in Styx. This group allows to operate on Backfill entities registered in Styx and check detailed status of each single Workflow Instance belonging to a Backfill entity.
Backfills ¶
Headers
Content-Type: application/jsonBody
{
"backfills": [
{
"backfill": {
"all_triggered": true,
"concurrency": 10,
"end": "2017-01-01T01:00:00Z",
"halted": false,
"id": "backfill-1489054446085-53384",
"next_trigger": "2017-01-01T01:00:00Z",
"schedule": "hours",
"start": "2017-01-01T00:00:00Z",
"trigger_parameters": {
"env": {
"FOO": "bar"
}
},
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
},
"created": "2017-01-01T00:00:00Z",
"last_modified": "2019-01-01T00:00:00Z"
},
"statuses": {
"active_states": [
{
"initial_timestamp": 1536838486145,
"latest_timestamp": 1536838486185,
"state": "DONE",
"state_data": {
"consecutive_failures": 0,
"execution_description": {
"commit_sha": "f043333085fa87738ac24f04d64fb58ecc845111",
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_image": "styx-canary-dummy:dummy",
"docker_termination_logging": false
},
"execution_id": "styx-run-6b1962c6-23ba-4245-a13b-ec3baa4b2133",
"last_exit": 0,
"messages": [
{
"level": "INFO",
"line": "Exit code: 0"
}
],
"retry_cost": 0,
"retry_delay_millis": null,
"tries": 1,
"trigger": {
"@type": "backfill",
"trigger_id": "backfill-1489054446085-53384"
},
"trigger_id": "backfill-1489054446085-53384"
},
"workflow_instance": {
"parameter": "2017-01-01T00",
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
}
}
}
]
}
}
]
}Get BackfillsGET/{version}/backfills{?component,workflow,status,showAll}
- version
string(required) Example: v3API version
Choices:
v3- component
string(optional) Example: styx-canaryWorkflow Component to filter with
- workflow
string(optional) Example: StyxCanaryWorkflow ID to filter with
- status
boolean(optional) Default: false Example: trueIf to return the status list per backfill
- showAll
boolean(optional) Default: false Example: trueIf to return also halted backfills and backfills whose instances have been all triggered
Headers
Content-Type: application/jsonBody
{
"start": "2017-01-01T00:00:00Z",
"end": "2017-01-02T00:00:00Z",
"component": "styx-canary",
"concurrency": "10",
"reverse": true,
"description": "fixed bug and reproduce data",
"trigger_parameters": {"env": {"FOO": "bar"}},
"workflow": "StyxCanary",
}Headers
Content-Type: application/jsonBody
{
"all_triggered": false,
"concurrency": 10,
"end": "2017-02-00T00:00:00Z",
"description": "fixed bug and reproduce data",
"halted": false,
"id": "backfill-1489054446085-52684",
"next_trigger": "2017-01-01T00:00:00Z",
"schedule": "hours",
"start": "2017-01-01T00:00:00Z",
"trigger_parameters": {"env": {"FOO": "bar"}},
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
},
"created": "2017-01-01T00:00:00Z",
"last_modified": "2019-01-01T00:00:00Z",
}Create BackfillPOST/{version}/backfills{?allowFuture}
- version
string(required) Example: v3API version
Choices:
v3- allowFuture
boolean(optional) Default: false Example: falseallow backfilling future partitions
Backfill ¶
Headers
Content-Type: application/jsonBody
{
"backfill": {
"all_triggered": true,
"concurrency": 10,
"description": "fixed bug and reproduce data",
"end": "2017-01-01T01:00:00Z",
"halted": false,
"id": "backfill-1489054446085-53384",
"next_trigger": "2017-01-01T01:00:00Z",
"schedule": "hours",
"start": "2017-01-01T00:00:00Z",
"trigger_parameters": {"env": {"FOO": "bar"}},
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
},
"created": "2017-01-01T00:00:00Z",
"last_modified": "2019-01-01T00:00:00Z",
},
"statuses": {
"active_states": [{
"initial_timestamp": 1536838486145,
"latest_timestamp": 1536838486185,
"state": "DONE",
"state_data": {
"consecutive_failures": 0,
"execution_description": {
"commit_sha": "f043333085fa87738ac24f04d64fb58ecc845111",
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_image": "styx-canary-dummy:dummy",
"docker_termination_logging": false
},
"execution_id": "styx-run-6b1962c6-23ba-4245-a13b-ec3baa4b2133",
"last_exit": 0,
"messages": [
{
"level": "INFO",
"line": "Exit code: 0"
}
],
"retry_cost": 0.0,
"retry_delay_millis": null,
"tries": 1,
"trigger": {
"@type": "backfill",
"trigger_id": "backfill-1489054446085-53384"
},
"trigger_id": "backfill-1489054446085-53384"
},
"workflow_instance": {
"parameter": "2017-01-01T00",
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
}
}
}]
}
}Get BackfillGET/{version}/backfills/{backfill_id}{?status}
- version
string(required) Example: v3API version
Choices:
v3- backfill_id
string(required) Example: backfill-1489054446085-52684Backfill ID
- status
boolean(optional) Default: true Example: trueIf to return the status list per backfill
Headers
Content-Type: application/jsonBody
{
"concurrency": 20,
"description": "fixed bug and reproduce data",
"id": "backfill-1489054446085-52684",
}Headers
Content-Type: application/jsonBody
{
"all_triggered": false,
"concurrency": 20,
"end": "2017-02-00T00:00:00Z",
"description": "fixed bug and reproduce data",
"halted": false,
"id": "backfill-1489054446085-52684",
"next_trigger": "2017-01-01T00:00:00Z",
"schedule": "hours",
"start": "2017-01-01T00:00:00Z",
"trigger_parameters": {"env": {"FOO": "bar"}},
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
},
"created": "2017-01-01T00:00:00Z",
"last_modified": "2019-01-01T00:00:00Z",
}Modify BackfillPUT/{version}/backfills/{backfill_id}
- version
string(required) Example: v3API version
Choices:
v3- backfill_id
string(required) Example: backfill-1489054446085-52684Backfill ID
Delete BackfillDELETE/{version}/backfills/{backfill_id}
- version
string(required) Example: v3API version
Choices:
v3- backfill_id
string(required) Example: backfill-1489054446085-52684Backfill ID
Status ¶
Resources related to inspect detailed status of Workflow Instances.
Active States ¶
Headers
Content-Type: application/jsonBody
{
"active_states": [
{
"latest_timestamp": 1536838486185,
"state": "RUNNING",
"state_data": {
"consecutive_failures": 0,
"execution_description": {
"commit_sha": "f043333085fa87738ac24f04d64fb58ecc845111",
"docker_args": [
"luigi",
"--module",
"canary_job",
"CanaryJob"
],
"docker_image": "styx-canary-dummy:dummy",
"docker_termination_logging": false
},
"execution_id": "styx-run-6b1962c6-23ba-4245-a13b-ec3baa4b21d5",
"last_exit": 20,
"messages": [
{
"level": "WARNING",
"line": "Exit code: 20"
}
],
"retry_cost": 0.2,
"retry_delay_millis": 600000,
"tries": 2,
"trigger": {
"@type": "natural"
},
"trigger_id": "natural-trigger"
},
"workflow_instance": {
"parameter": "2017-01-01T01",
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
}
}
}
]
}Get Active StatesGET/{version}/status/activeStates{?component,workflow,components}
- version
string(required) Example: v3API version
Choices:
v3- component
string(optional) Example: styx-canaryComponent ID
- workflow
string(optional) Example: StyxCanaryWorkflow ID - requires Component ID to also be specified
- components
string(optional) Example: styx-canary,styx-canary1,styx-canary3Component IDs, IDs should be comma separated. If provided, two other parameters will be ignored.
Events ¶
Headers
Content-Type: application/jsonBody
{
"events": [
{
"event": {
"@type": "triggerExecution",
"trigger": {
"@type": "natural"
},
"workflow_instance": "styx-canary/StyxCanary/2017-01-01T01",
"parameters": {
"env": {
"FOO": "bar"
}
}
},
"timestamp": 1483228800000
},
{
"event": {
"@type": "dequeue",
"workflow_instance": "styx-canary/StyxCanary/2017-01-01T01"
},
"timestamp": 1483228800005
}
]
}Get Workflow Instance eventsGET/{version}/status/events/{component_id}/{workflow_id}/{instance}
- version
string(required) Example: v3API version
Choices:
v3- component_id
string(required) Example: styx-canaryWorkflow Component
- workflow_id
string(required) Example: StyxCanaryWorkflow ID
- instance
string(required) Example: 2017-01-01T01Worfklow Instance parameter
Scheduler ¶
Resources related to operations that act on the current state of the scheduler.
Retry ¶
Headers
Content-Type: application/jsonBody
{
"workflow_instance": "styx-canary/StyxCanary/2017-01-01T01"
}Headers
Content-Type: application/jsonBody
{
"workflow_instance": "styx-canary/StyxCanary/2017-01-01T01"
}Send RetryPOST/{version}/scheduler/retry{?delay}
- version
string(required) Example: v3API version
Choices:
v3- delay
long(optional) Default: 0 Example: 0The new delay value to set for the workflow instance
Halt ¶
Headers
Content-Type: application/jsonBody
{
"workflow_instance": "styx-canary/StyxCanary/2017-01-01T01"
}Headers
Content-Type: application/jsonBody
{
"workflow_instance": "styx-canary/StyxCanary/2017-01-01T01"
}Send HaltPOST/{version}/scheduler/halt
- version
string(required) Example: v3API version
Choices:
v3
Trigger ¶
Headers
Content-Type: application/jsonBody
{
"parameter": "2017-01-01T01",
"trigger_parameters": {
"env": {
"FOO": "bar"
}
},
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
}
}Headers
Content-Type: application/jsonBody
{
"parameter": "2017-01-01T01",
"trigger_parameters": {
"env": {
"FOO": "bar"
}
},
"workflow_id": {
"component_id": "styx-canary",
"id": "StyxCanary"
},
"trigger_id": "UNKNOWN"
}Trigger Workflow InstancePOST/{version}/scheduler/trigger{?allowFuture}
- version
string(required) Example: v3API version
Choices:
v3- allowFuture
boolean(optional) Default: false Example: falseallow triggering future partition
Styx Config ¶
Resources related to global configurations for Styx.
Global Enabled ¶
Headers
Content-Type: application/jsonBody
{
"enabled": "true"
}Get Global EnabledGET/{version}/config
- version
string(required) Example: v3API version
Choices:
v3
Headers
Content-Type: application/jsonBody
{
"enabled": "true"
}Set Global EnabledPOST/{version}/config{?enabled}
- version
string(required) Example: v3API version
Choices:
v3- enabled
boolean(required) Example: trueGlobal enabled value
Generated by aglio on 12 Apr 2023