Chapter 34. Simple HTTP API

The Simple HTTP API is a basic Server automation tool that lets you control the Server from external applications using simple HTTP calls.

Most of operations is accessible using the HTTP GET method and return plain text. Thus, both “request” and “response” can be conveniently sent and parsed using very simple tools (wget, grep, etc.).

If global security is “on” (on by default), Basic HTTP authentication is used. Authenticated operations will require valid user credentials with corresponding permissions.

Note that the ETL graph-related operations "graph_run", "graph_status" and "graph_kill" also work for jobflows and Data Profiler jobs.

The generic pattern for a request URL:

http://[domain]:[port]/[context]/[servlet]/[operation]?[param1]=[value1]&[param2]=[value2]...

example: http://localhost:8080/clover/simpleHttpApi/help

[Note]Note

For backward compatibility, you can also use http://localhost:8080/clover/request_processor/help.

For a wget client, you can use the following command line:

wget --user=$USER --password=$PASS -O ./$OUTPUT_FILE $REQUEST_URL

The HTTP API is enabled by default. You can disable it with configuration property http.api.enabled. In server GUI, switch to ConfigurationSetup and add line

http.api.enabled=false

to the properties file.

Operation help

parameters

no

returns

list of possible operations and parameters with its descriptions

example

http://localhost:8080/clover/simpleHttpApi/help

Operation graph_run

Call this operation to start execution of the specified job. The operation is called graph_run for backward compatibility, however it may execute ETL graph, jobflow or profiler job.

parameters

Table 34.1. Parameters of graph_run

parameter namemandatorydefaultdescription
graphIDyes-A file path to the job file, relative to the sandbox root.
sandboxyes-Text ID of sandbox.
additional job parametersno Any URL parameter with "param_" prefix is passed to executed job and may be used in transformation XML as a placeholder, but without the "param_" prefix. e.g. "param_FILE_NAME" specified in URL may be used in the XML as ${FILE_NAME}. These parameters are resolved only during loading of XML, so it cannot be pooled.
additional config parametersno URL Parameters prefixed with "config_" can set some of the execution parameters. For graphs, the following parameters are supported:
  • config_skipCheckConfig - when set to "false", graph configuration will be checked before the execution.

  • config_logLevel - log level of the executed graph, one of OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, ALL.

  • config_clearObsoleteTempFiles - when set to "true", temp files of previous runs of this graph will be deleted before the execution.

  • config_debugMode - when set to "true", debug mode for given graph will be enabled. See Job Config Properties for more info.

nodeIDno-In cluster mode it's ID of a node which should execute the job. However it's not final. If the graph is distributed, or the node is disconnected, the graph may be executed on some another node.
verbosenoMESSAGEMESSAGE | FULL - how verbose should possible error message be.

returns

run ID: incremental number, which identifies each execution request

example

http://localhost:8080/clover/simpleHttpApi/graph_run?graphID=graph/graphDBExecute.grf&sandbox=mva

Operation graph_status

Call this operation to obtain status of specified job execution. The operation is called graph_status for backward compatibility, however it may return status of an ETL graph or jobflow.

parameters

Table 34.2. Parameters of graph_status

parameter namemandatorydefaultdescription
runIDyes-Id of each graph execution
returnTypenoSTATUSSTATUS | STATUS_TEXT | DESCRIPTION | DESCRIPTION_XML
waitForStatusno-Status code which we want to wait for. If it is specified, this operation will wait until the graph is in the required status.
waitTimeoutno0If waitForStatus is specified, it will wait only the specified amount of milliseconds. Default 0 means forever, but it depends on an application server configuration. When the specified timeout expires and graph run still isn't in required status, the server returns code 408 (Request Timeout). 408 code may be also returned by an application server if its HTTP request timeout expires before.
verbosenoMESSAGEMESSAGE | FULL - how verbose should possible error message be.

returns

Status of a specified graph. It may be number code, text code or a complex description in dependence on optional parameter returnType. Description is returned as plain text with a pipe as a separator, or as XML. A schema describing XML format of the XML response is accessible on CloverETL Server URL: http://[host]:[port]/clover/schemas/executions.xsd In dependence on waitForStatus parameter it may return result immediately or wait for a specified status.

example

http://localhost:8080/clover/simpleHttpApi/graph_status?runID=123456&returnType=DESCRIPTION&waitForStatus=FINISHED&waitTimeout=60000

Operation graph_kill

Call this operation to abort/kill job execution. The operation is called graph_kill for backward compatibility, however it may abort/kill ETL graph, jobflow or profiler job.

parameters

Table 34.3. Parameters of graph_kill

parameter namemandatorydefaultdescription
runIDyes-Id of each graph execution
returnTypenoSTATUSSTATUS | STATUS_TEXT | DESCRIPTION
verbosenoMESSAGEMESSAGE | FULL - how verbose should possible error message be.

returns

Status of the specified graph after an attempt to kill it. It may be number code, text code or a complex description in dependence on optional parameter.

example

http://localhost:8080/clover/simpleHttpApi/graph_kill?runID=123456&returnType=DESCRIPTION

Operation server_jobs

parameters

no

returns

List of runIDs of currently running jobs.

example

http://localhost:8080/clover/simpleHttpApi/server_jobs

Operation sandbox_list

parameters

no

returns

List of all sandbox text IDs. In next versions will return only accessible ones.

example

http://localhost:8080/clover/simpleHttpApi/sandbox_list

Operation sandbox_content

parameters

Table 34.4. Parameters of sandbox_content

parameter namemandatorydefaultdescription
sandboxyes-text ID of sandbox
verbosenoMESSAGEMESSAGE | FULL - how verbose should possible error message be.

returns

A list of all elements in the specified sandbox. Each element may be specified as a file path relative to the sandbox root.

example

http://localhost:8080/clover/simpleHttpApi/sandbox_content?sandbox=mva

Operation executions_history

parameters

Table 34.5. Parameters of executions_history

parameter namemandatorydefaultdescription
sandboxyes-text ID of sandbox
fromno Lower datetime limit of start of execution. The operation will return only records after (and equal) this datetime. Format: "yyyy-MM-dd HH:mm" (must be URL encoded).
tono Upper datetime limit of start of execution. The operation will return only records before (and equal) this datetime. Format: "yyyy-MM-dd HH:mm" (must be URL encoded).
stopFromno Lower datetime limit of stop of execution. The operation will return only records after (and equal) this datetime. Format: "yyyy-MM-dd HH:mm" (must be URL encoded).
stopTono Upper datetime limit of stop of execution. The operation will return only records before (and equal) this datetime. Format: "yyyy-MM-dd HH:mm" (must be URL encoded).
statusno Current execution status. The operation will return only records with specified STATUS. Meaningful values are RUNNING | ABORTED | FINISHED_OK | ERROR
sandboxno Sandbox code. The operation will return only records for graphs from the specified sandbox.
graphIdno Text Id, which is unique in a specified sandbox. The file path is relative to the sandbox root.
orderByno Attribute for list ordering. Possible values: id | graphId | status | startTime | stopTime. By default, there is no ordering.
orderDescendnotrueA switch which specifies ascending or descending ordering. If it is true (which is default), ordering is descending.
returnTypenoIDsPossible values are: IDs | DESCRIPTION | DESCRIPTION_XML
indexno0Index of the first returned records in whole record set. (starting from
recordsnoinfiniteMax amount of returned records.
verbosenoMESSAGEMESSAGE | FULL - how verbose should possible error message be.

returns

List of executions according to filter criteria.

For returnType==IDs returns a simple list of runIDs (with new line delimiter).

For returnType==DESCRIPTION returns complex response which describes current status of selected executions, their phases, nodes and ports.

execution|[runID]|[status]|[username]|[sandbox]|[graphID]|[startedDatetime]|[finishedDatetime]|[clusterNode]|[graphVersion]
phase|[index]|[execTimeInMilis]
node|[nodeID]|[status]|[totalCpuTime]|[totalUserTime]|[cpuUsage]|[peakCpuUsage]|[userUsage]|[peakUserUsage]
port|[portType]|[index]|[avgBytes]|[avgRows]|[peakBytes]|[peakRows]|[totalBytes]|[totalRows] 

example of request

http://localhost:8080/clover/simpleHttpApi/executions_history?from=&to=2008-09-16+16%3A40&status=&sandbox=def&graphID=&index=&records=&returnType=DESCRIPTION

example of DESCRIPTION (plain text) response

execution|13108|FINISHED_OK|clover|def|test.grf|2008-09-16 11:11:19|2008-09-16 11:11:58|nodeA|2.4
phase|0|38733
node|DATA_GENERATOR1|FINISHED_OK|0|0|0.0|0.0|0.0|0.0
port|Output|0|0|0|0|0|130|10
node|TRASH0|FINISHED_OK|0|0|0.0|0.0|0.0|0.0
port|Input|0|0|0|5|0|130|10
node|SPEED_LIMITER0|FINISHED_OK|0|0|0.0|0.0|0.0|0.0
port|Input|0|0|0|0|0|130|10
port|Output|0|0|0|5|0|130|10
execution|13107|ABORTED|clover|def|test.grf|2008-09-16 11:11:19|2008-09-16 11:11:30
phase|0|11133
node|DATA_GENERATOR1|FINISHED_OK|0|0|0.0|0.0|0.0|0.0
port|Output|0|0|0|0|0|130|10
node|TRASH0|RUNNING|0|0|0.0|0.0|0.0|0.0
port|Input|0|5|0|5|0|52|4
node|SPEED_LIMITER0|RUNNING|0|0|0.0|0.0|0.0|0.0
port|Input|0|0|0|0|0|130|10
port|Output|0|5|0|5|0|52|4

For returnType==DESCRIPTION_XML returns complex data structure describing one or more selected executions in XML format. A schema describing XML format of the XML response is accessible on CloverETL Server URL: http://[host]:[port]/clover/schemas/executions.xsd

Operation suspend

Suspends server or sandbox (if specified). Suspension means, that no graphs may me executed on suspended server/sandbox.

parameters

Table 34.6. Parameters of suspend

parameter namemandatorydefaultdescription
sandboxno-Text ID of a sandbox to suspend. If not specified, it suspends the whole server.
atonceno If this param is set to true, running graphs from suspended server (or just from sandbox) are aborted. Otherwise it can run until it is finished in common way.

returns

Result message

Operation resume

parameters

Table 34.7. Parameters of resume

parameter namemandatorydefaultdescription
sandboxno-Text Id of a sandbox to resume. If not specified, the server will be resumed.
verbosenoMESSAGEMESSAGE | FULL - how verbose should the possible error message be.

returns

Result message

Operation sandbox_create

This operation creates a specified sandbox. If it is sandbox of "partitioned" or "local" type, it also creates locations by "sandbox_add_location" operation.

parameters

Table 34.8. Parameters of sandbox create

parameter namemandatorydefaultdescription
sandboxyes-Text Id of sandbox to be created.
pathno-Path to the sandbox root if server is running in standalone mode.
typenosharedSandbox type: shared | partitioned | local. For standalone server may be left empty, since the default "shared" is used.
createDirsnotrueSwitch whether to create directory structure of the sandbox (only for standalone server or "shared" sandboxes in cluster environment).
verbosenoMESSAGEMESSAGE | FULL - how verbose should possible error message be.

returns

Result message

Operation sandbox_add_location

This operation adds a location to the specified sandbox. Can be only used with partitioned or local sandboxes.

parameters

Table 34.9. Parameters of sandbox add location

parameter namemandatorydefaultdescription
sandboxyes-Sandbox which we want to add location to.
nodeIdyes-Location attribute - node which has direct access to the location.
pathyes-Location attribute - path to the location root on the specified node.
locationno-Location attribute - location storage ID. If it's not specified, new one will be generated.
verbosenoMESSAGEMESSAGE | FULL - how verbose should possible error message be.

returns

Result message

Operation sandbox_remove_location

This operation removes a location from the specified sandbox. Only sandboxes of type partitioned or local can have locations asociated.

parameters

Table 34.10. Parameters of sandbox add location

parameter namemandatorydefaultdescription
sandboxyes-Removes specified location from its sandbox.
locationyes-Location storage ID. If the specified location isn't attached to the specified sandbox, sandbox won't be changed.
verbosenoMESSAGEMESSAGE | FULL - how verbose should possible error message be.

returns

Result message

Operation download_sandbox_zip

This operation downloads content of a specified sandbox as a ZIP archive.

parameters

Table 34.11. Parameters

parameter namemandatorydefaultdescription
sandboxyes-Code of the sandbox to be downloaded.

returns

Content of the specified sandbox as a ZIP archive

example

 wget --http-user=username --http-password=password http://localhost:8080/clover/simpleHttpApi/download_sandbox_zip?sandbox=my-sandbox

Operation upload_sandbox_zip

This operation uploads content of a ZIP archive into a specified sandbox.

parameters

Table 34.12. Parameters

parameter namemandatorydefaultdescription
sandboxyes-Code of the sandbox the ZIP file will be expanded to.
zipFileyes-The ZIP archive file.
overwriteExistingnofalseIf true, the files already present in the sandbox will be overwritten.
deleteMissingnofalseIf true, the files not present in the ZIP file will be deleted from the sandbox.
fileNameEncodingnoUTF-8The encoding that was used to store file names in the ZIP archive.

returns

Result message

example of request (with using curl CLI tool (http://curl.haxx.se/))

curl -u username:password -F "overwriteExisting=true"
    -F "zipFile=@/tmp/my-sandbox.zip"
    http://localhost:8080/clover/simpleHttpApi/upload_sandbox_zip

Operation cluster_status

This operation displays cluster's nodes list.

parameters

no

returns

Cluster's nodes list.

Operation export_server_config

This operation exports a current server configuration in XML format.

parameters

Table 34.13. Parameters of server configuration export

parameter namemandatorydefaultdescription
includenoallSelection which items will be included in the exported XML file; the parameter may be specified multiple times. Possible values are:
  • all - include items of all types
  • users - include list of users
  • userGroups - include list of user groups
  • sandboxes - include list of sandboxes
  • jobConfigs - include list of job configuration parameters
  • schedules - include list of schedules
  • eventListeners - include list of event listeners
  • launchServices - include list of launch services
  • tempSpaces - include list of temporary spaces

returns

Current server configuration as an XML file.

example

wget --http-user=username --http-password=password http://localhost:8080/clover/simpleHttpApi/export_server_config

Operation import_server_config

This operation imports server configuration.

parameters

Table 34.14. Parameters of server configuration import

parameter namemandatorydefaultdescription
xmlFileyes-An XML file with server's configuration.
dryRunnotrueIf true, a dry run is performed with no actual changes written.
verbosenoMESSAGEMESSAGE | FULL - how verbose should the response be: MESSAGE for a simple message, FULL for a full XML report.
newOnlynofalseIf true only new items will imported to the server; the items already present on the server will be left untouched.
includenoallSelection which items will be imported from the XML; the parameter may be specified multiple times. Possible values are:
  • all - import items of all types
  • users - import users
  • userGroups - import user groups
  • sandboxes - import sandboxes
  • jobConfigs - import job configuration parameters
  • schedules - import schedules
  • eventListeners - import listeners
  • launchServices - import launch services
  • tempSpaces - import temporary spaces

returns

Result message or XML report

example of request (with using curl CLI tool (http://curl.haxx.se/))

curl -u username:password -F "dryRun=true" -F "verbose=FULL"
    -F "xmlFile=@/tmp/clover_configuration_2013-07-10_14-03-23+0200.xml"
    http://localhost:8080/clover/simpleHttpApi/import_server_config