Using /results¶
In this example we are going to explore deeply the options available when collecting and uploading benchmark results at EOSC Performance. This example was created using Jupyter notebook, click here to the original notebook file.
Create the environment¶
To do so, we select an API endpoint and collect a token from our configuration. We also need an access token, in this example we use oidc-agent to get one.
[1]:
eosc_perf_api="https://performance.services.fedcloud.eu/api/v1"
access_token=$(oidc-token egi-prod)
Search for results¶
Configure and limit your search using the multiple arguments and terms available.
Search for the benchmark id that produced our result¶
You can get a list of all available benchmarks using GET /benchmarks.
[2]:
benchmarks=$(curl -X 'GET' "$eosc_perf_api/benchmarks?docker_image=thechristophe/openbench-c-ray")
benchmark=$(echo $benchmarks | jq '.items[0]')
echo $benchmark | jq '.json_schema = "..."'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1441 100 1441 0 0 11620 0 --:--:-- --:--:-- --:--:-- 11528
{
"description": "Compare cpu perf with multithreaded raytracing",
"docker_image": "thechristophe/openbench-c-ray",
"docker_tag": "latest",
"id": "1cc7814e-131f-4002-803a-434a287cf135",
"json_schema": "...",
"upload_datetime": "2022-02-01T07:58:17.555822"
}
[3]:
benchmark_id=$(echo $benchmark | jq -r '.id')
curl -X 'GET' "$eosc_perf_api/results?benchmark_id=$benchmark_id" \
-H 'accept: application/json' | jq '.items[].id'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 74245 100 74245 0 0 895k 0 --:--:-- --:--:-- --:--:-- 895k
"07decfe3-8f03-4dcf-b8ee-b47090322e0c"
"c2517c02-e195-4bad-8d8d-707ffa8cff9b"
"1196b021-7923-44bf-a9f1-d7abd683fe5a"
"c45d2d84-2bd7-4fb7-83a7-b9a25d2d034a"
"2a04564b-adad-40f0-9c86-062546bf15dd"
"f5cdf98c-28db-4c8e-97f8-459ae1e94ee6"
"7ddecf46-75c6-46b1-abc7-f566b38de10c"
"dd74701b-052d-4302-9d76-96ddc0a11f0c"
"f97528bd-f350-427a-8cf0-44dbb2eaf487"
"138f4780-1cd5-4d4d-bb36-2b4c86f61147"
"a6dde02e-fbf4-4342-ba29-729802aa7b36"
"ddad3d39-eae3-43d4-a4b5-36fd9e427e3c"
"f8f9beee-705d-4096-870e-6b037ebab86b"
"eb2ba92d-479b-420d-bdda-4ac0fda780b4"
"244d5872-cfc4-4359-850e-a74314edbe65"
"615539ce-c123-4b2c-9de9-aa68dde88055"
"768b83de-58a4-426b-ac6a-3d7b1ee1db51"
"5c6a7dc1-14be-4b18-af26-cb7089f0607c"
"96e95029-c24b-468b-b659-4a0ef6886be2"
"0a1d1595-f4d4-4364-99a3-7e1cb2e642e3"
"105357bb-8054-4174-9bda-b0020a824ef9"
"7f66e8d3-2218-47ad-b697-726343bffb27"
"fb0ee217-5f19-4d24-bb86-a1baaf6262d0"
Search for the site id used to run our benchmark¶
You can get a list of all available benchmarks using GET /sites.
[4]:
sites=$(curl -X 'GET' "$eosc_perf_api/sites?name=CESNET-MCC")
site=$(echo $sites | jq '.items[0]')
echo $site | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 223 100 223 0 0 22300 0 --:--:-- --:--:-- --:--:-- 22300
{
"address": "unknown",
"id": "17fb17c9-107c-4571-ab1e-120299878342",
"name": "CESNET-MCC",
"upload_datetime": "2022-02-01T07:57:42.821704"
}
[5]:
site_id=$(echo $site | jq -r '.id')
curl -X 'GET' "$eosc_perf_api/results?site_id=$site_id" \
-H 'accept: application/json' | jq '.items[].id'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 13839 100 13839 0 0 409k 0 --:--:-- --:--:-- --:--:-- 409k
"2a04564b-adad-40f0-9c86-062546bf15dd"
"f5cdf98c-28db-4c8e-97f8-459ae1e94ee6"
"7ddecf46-75c6-46b1-abc7-f566b38de10c"
"dd74701b-052d-4302-9d76-96ddc0a11f0c"
Search for the flavor id used to run our benchmark¶
If needed you can also get a list of all available flavors in that site using GET /sites{id}/flavors.
[6]:
site_id=$(echo $site | jq -r '.id')
flavors=$(curl -X 'GET' "$eosc_perf_api/sites/$site_id/flavors:search?terms=hpc.8core-16ram")
flavor=$(echo $flavors | jq '.items[0]')
echo $flavor | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 225 100 225 0 0 15000 0 --:--:-- --:--:-- --:--:-- 15000
{
"description": "",
"id": "01210775-cc77-472d-a061-c98760dbb883",
"name": "hpc.8core-16ram",
"upload_datetime": "2022-02-01T07:57:48.652778"
}
[7]:
flavor_id=$(echo $flavor | jq -r '.id')
curl -X 'GET' "$eosc_perf_api/results?flavor_id=$flavor_id" \
-H 'accept: application/json' | jq '.items[].id'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3684 100 3684 0 0 179k 0 --:--:-- --:--:-- --:--:-- 179k
"dd74701b-052d-4302-9d76-96ddc0a11f0c"
Search for the tags to relate your result¶
Collect the tags you want to link to your result so users can find it easily.
[8]:
tag_gpu=$(curl -X 'GET' "$eosc_perf_api/tags?name=gpu" | jq '.items[0]')
echo $tag_gpu | jq
tag_hpc=$(curl -X 'GET' "$eosc_perf_api/tags?name=hpc" | jq '.items[0]')
echo $tag_hpc | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 191 100 191 0 0 27285 0 --:--:-- --:--:-- --:--:-- 27285
{
"description": "Result executed using gpu",
"id": "08156712-4607-469d-903a-d3033b44ab9d",
"name": "gpu"
}
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 195 100 195 0 0 16250 0 --:--:-- --:--:-- --:--:-- 16250
{
"description": "Result executed in hpc system",
"id": "92f7443d-1991-4bee-8b58-a4189c586a08",
"name": "hpc"
}
If you do not know the name of the tag you can use :search as generic filter.
[9]:
tag_1=$(echo $tag_hpc | jq -r '.id')
curl -X 'GET' "$eosc_perf_api/results?tags_ids=$tag_1" \
-H 'accept: application/json' | jq '.items[].id'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3684 100 3684 0 0 133k 0 --:--:-- --:--:-- --:--:-- 133k
"dd74701b-052d-4302-9d76-96ddc0a11f0c"
Search for results between dates¶
If is also possible to filter results by upload and execution date.
[10]:
upload_before='2023-01-01'
execution_after='2000-01-01'
curl -X 'GET' "$eosc_perf_api/results?upload_before=$upload_before&execution_after=$execution_after" \
-H 'accept: application/json' | jq '.items[].id'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 74245 100 74245 0 0 1050k 0 --:--:-- --:--:-- --:--:-- 1050k
"07decfe3-8f03-4dcf-b8ee-b47090322e0c"
"c2517c02-e195-4bad-8d8d-707ffa8cff9b"
"1196b021-7923-44bf-a9f1-d7abd683fe5a"
"c45d2d84-2bd7-4fb7-83a7-b9a25d2d034a"
"2a04564b-adad-40f0-9c86-062546bf15dd"
"f5cdf98c-28db-4c8e-97f8-459ae1e94ee6"
"7ddecf46-75c6-46b1-abc7-f566b38de10c"
"dd74701b-052d-4302-9d76-96ddc0a11f0c"
"f97528bd-f350-427a-8cf0-44dbb2eaf487"
"138f4780-1cd5-4d4d-bb36-2b4c86f61147"
"a6dde02e-fbf4-4342-ba29-729802aa7b36"
"ddad3d39-eae3-43d4-a4b5-36fd9e427e3c"
"f8f9beee-705d-4096-870e-6b037ebab86b"
"eb2ba92d-479b-420d-bdda-4ac0fda780b4"
"244d5872-cfc4-4359-850e-a74314edbe65"
"615539ce-c123-4b2c-9de9-aa68dde88055"
"768b83de-58a4-426b-ac6a-3d7b1ee1db51"
"5c6a7dc1-14be-4b18-af26-cb7089f0607c"
"96e95029-c24b-468b-b659-4a0ef6886be2"
"0a1d1595-f4d4-4364-99a3-7e1cb2e642e3"
"105357bb-8054-4174-9bda-b0020a824ef9"
"7f66e8d3-2218-47ad-b697-726343bffb27"
"fb0ee217-5f19-4d24-bb86-a1baaf6262d0"
Search using custom filters¶
If is also possible to filter results specific values inside the result (if you know the result structure).
[11]:
benchmark_id='1cc7814e-131f-4002-803a-434a287cf135'
filter_1='machine.cpu.count%20%3E%204'
filter_2='machine.cpu.count%20%3C%2020'
curl -X 'GET' "$eosc_perf_api/results?benchmark_id=$benchmark_id&filters=$filter_1&filters=$filter_2" \
-H 'accept: application/json' | jq '.items[].id'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 25917 100 25917 0 0 377k 0 --:--:-- --:--:-- --:--:-- 383k
"c2517c02-e195-4bad-8d8d-707ffa8cff9b"
"1196b021-7923-44bf-a9f1-d7abd683fe5a"
"dd74701b-052d-4302-9d76-96ddc0a11f0c"
"ddad3d39-eae3-43d4-a4b5-36fd9e427e3c"
"f8f9beee-705d-4096-870e-6b037ebab86b"
"768b83de-58a4-426b-ac6a-3d7b1ee1db51"
"105357bb-8054-4174-9bda-b0020a824ef9"
"7f66e8d3-2218-47ad-b697-726343bffb27"
If you do not want results that does not include the field in the response, we recommend that you use a benchmark id.
Configure sorting and use pagination¶
For all the previous options, it is possible to sort the results using the following fields:
id: Result id.
upload_datetime: EOSC Performance upload datetime of the result.
json: Result json values.
execution_datetime: Execution date of the benchmark.
benchmark_id: Benchmark id used to obtain the result.
flavor_id: Favor id used to run the benchmark.
site_id: Site id where the benchmark was executed.
In addition, sometimes you might get more results that expected. In such case you will have to use pagination to collect all the items. To do so you can use the following parameters:
per_page: The number of items to be displayed on a page (maximum 100)
page: The return page number (1 indexed)
[12]:
curl -X 'GET' "$eosc_perf_api/results?per_page=4&page=2&sort_by=%2Bid" \
-H 'accept: application/json' | jq '.items[].id'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 13086 100 13086 0 0 399k 0 --:--:-- --:--:-- --:--:-- 399k
"138f4780-1cd5-4d4d-bb36-2b4c86f61147"
"244d5872-cfc4-4359-850e-a74314edbe65"
"2a04564b-adad-40f0-9c86-062546bf15dd"
"5c6a7dc1-14be-4b18-af26-cb7089f0607c"
Upload results¶
(Conditional) Register, if not done already¶
To use our service as user, first we need to accept the terms of usage and register. Make sure to read the terms and conditions.
[ ]:
curl -X 'POST' \
"$eosc_perf_api/users:register" \
-H "Authorization: Bearer $access_token"
Search for the benchmark id that produced our result¶
You can get a list of all available benchmarks using GET /benchmarks.
[14]:
benchmarks=$(curl -X 'GET' "$eosc_perf_api/benchmarks?docker_image=thechristophe/openbench-c-ray")
benchmark=$(echo $benchmarks | jq '.items[0]')
echo $benchmark | jq '.json_schema = "..."'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1441 100 1441 0 0 234k 0 --:--:-- --:--:-- --:--:-- 234k
{
"description": "Compare cpu perf with multithreaded raytracing",
"docker_image": "thechristophe/openbench-c-ray",
"docker_tag": "latest",
"id": "1cc7814e-131f-4002-803a-434a287cf135",
"json_schema": "...",
"upload_datetime": "2022-02-01T07:58:17.555822"
}
Benchmarks are public to the Internet, access token is not needed.
Search for the flavor id used to run our benchmark¶
First you need to find the site where the benchmark was run. Once the site id is collected, it is possible to access and select the site flavors.
[15]:
sites=$(curl -X 'GET' "$eosc_perf_api/sites?name=CESNET-MCC")
site=$(echo $sites | jq '.items[0]')
echo $site | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 223 100 223 0 0 31857 0 --:--:-- --:--:-- --:--:-- 31857
{
"address": "unknown",
"id": "17fb17c9-107c-4571-ab1e-120299878342",
"name": "CESNET-MCC",
"upload_datetime": "2022-02-01T07:57:42.821704"
}
In this example we will use :search endpoint to find the flavor.
[16]:
site_id=$(echo $site | jq -r '.id')
flavors=$(curl -X 'GET' "$eosc_perf_api/sites/$site_id/flavors:search?terms=hpc.8core-16ram")
flavor=$(echo $flavors | jq '.items[0]')
echo $flavor | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 225 100 225 0 0 25000 0 --:--:-- --:--:-- --:--:-- 25000
{
"description": "",
"id": "01210775-cc77-472d-a061-c98760dbb883",
"name": "hpc.8core-16ram",
"upload_datetime": "2022-02-01T07:57:48.652778"
}
Search for the tags to relate your result¶
Collect the tags you want to link to your result so users can find it easily.
If you do not know the name of the tag you can use :search as generic filter.
[17]:
tag_hpc=$(curl -X 'GET' "$eosc_perf_api/tags?name=hpc" | jq '.items[0]')
echo $tag_hpc | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 195 100 195 0 0 8863 0 --:--:-- --:--:-- --:--:-- 8863
{
"description": "Result executed in hpc system",
"id": "92f7443d-1991-4bee-8b58-a4189c586a08",
"name": "hpc"
}
Upload your result¶
Use an execution datetime and the collected site_id and flavor_id to upload a result.
[18]:
execution_datetime="2022-01-01T10:00:00.00000Z"
benchmark_id=$(echo $benchmark | jq -r '.id')
flavor_id=$(echo $flavor | jq -r '.id')
tag1_id=$(echo $tag_hpc | jq -r '.id')
result_json='{"arguments": "Total Time - 4K, 16 Rays Per Pixel", "machine": {"cpu": {"count": 4}}, "result": {"all_results": "155.995:157.626:156.195", "score": 156.605}, "test": "pts/c-ray-1.2.0", "units": "Seconds"}'
[19]:
query="execution_datetime=$execution_datetime&benchmark_id=$benchmark_id&flavor_id=$flavor_id&tags_ids=$tag1_id"
curl -X 'POST' "$eosc_perf_api/results?$query" \
-H 'accept: application/json' \
-H "Authorization: Bearer $access_token" \
-H 'Content-Type: application/json' \
-d "$result_json" | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2299 100 2095 100 204 1847 179 0:00:01 0:00:01 --:--:-- 2027
{
"benchmark": {
"description": "Compare cpu perf with multithreaded raytracing",
"docker_image": "thechristophe/openbench-c-ray",
"docker_tag": "latest",
"id": "1cc7814e-131f-4002-803a-434a287cf135",
"json_schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"arguments": {
"const": "Total Time - 4K, 16 Rays Per Pixel"
},
"machine": {
"properties": {
"cpu": {
"properties": {
"arch": {
"description": "Processor architecture",
"suggestToUser": true,
"type": "string"
},
"bits": {
"description": "Processor address size",
"type": "number"
},
"brand_raw": {
"description": "Human-readable processor branding",
"type": "string"
},
"count": {
"description": "Processor core count",
"suggestToUser": true,
"type": "number"
},
"hz_actual_friendly": {
"description": "Human-readable maximum processor frequency",
"type": "string"
}
},
"required": [
"count"
],
"type": "object"
}
},
"type": "object"
},
"result": {
"description": "The execution results",
"properties": {
"all_results": {
"description": "Concatenated string of all scores",
"type": "string"
},
"score": {
"description": "Average time in seconds for benchmark completion (lower is better)",
"suggestToUser": true,
"type": "number"
}
},
"required": [
"all_results",
"score"
],
"type": "object"
},
"test": {
"const": "pts/c-ray-1.2.0"
},
"units": {
"const": "Seconds"
}
},
"required": [
"arguments",
"machine",
"result",
"test",
"units"
],
"type": "object"
},
"upload_datetime": "2022-02-01T07:58:17.555822"
},
"execution_datetime": "2022-01-01T10:00:00",
"flavor": {
"description": "",
"id": "01210775-cc77-472d-a061-c98760dbb883",
"name": "hpc.8core-16ram",
"upload_datetime": "2022-02-01T07:57:48.652778"
},
"id": "76c5773a-52d7-408b-961b-e6d3ad06c640",
"json": {
"arguments": "Total Time - 4K, 16 Rays Per Pixel",
"machine": {
"cpu": {
"count": 4
}
},
"result": {
"all_results": "155.995:157.626:156.195",
"score": 156.605
},
"test": "pts/c-ray-1.2.0",
"units": "Seconds"
},
"site": {
"address": "unknown",
"id": "17fb17c9-107c-4571-ab1e-120299878342",
"name": "CESNET-MCC",
"upload_datetime": "2022-02-01T07:57:42.821704"
},
"tags": [
{
"description": "Result executed in hpc system",
"id": "92f7443d-1991-4bee-8b58-a4189c586a08",
"name": "hpc"
}
],
"upload_datetime": "2022-02-23T14:32:02.859960"
}