CLI
Contents
CLI¶
Workflow¶
You can use SkyPilot Benchmark by simply replacing your sky launch command with sky bench launch:
# Launch mytask on a V100 VM and a T4 VM
$ sky bench launch mytask.yaml --gpus V100,T4 --benchmark mybench
The second command will launch mytask.yaml on a V100 VM and a T4 VM simultaneously, with a benchmark name mybench.
After the task finishes, you can check the benchmark results using sky bench show:
# Show the benchmark report on `mybench`
$ sky bench show mybench
CLUSTER RESOURCES STATUS DURATION SPENT($) STEPS SEC/STEP $/STEP EST(hr) EST($)
sky-bench-mybench-0 1x GCP(n1-highmem-8, {'V100': 1}) FINISHED 12m 51s 0.6317 - - - - -
sky-bench-mybench-1 1x AWS(g4dn.xlarge, {'T4': 1}) FINISHED 16m 19s 0.1430 - - - - -
In the report, SkyPilot shows the duration and cost of mybench on each VM.
The VMs can be terminated by either sky bench down or sky down:
# Terminate all the clusters used for `mybench`
$ sky bench down mybench
# Terminate all the clusters used for `mybench` except `sky-bench-mybench-0`
$ sky bench down mybench --exclude sky-bench-mybench-0
# Terminate individual clusters as usual
$ sky down sky-bench-mybench-0
Note
Each cluster launched by sky bench launch will automatically stop itself 5 minutes after the task is finished.
However, you don’t have to restart those clusters.
Regardless of the status of the clusters, sky bench show will provide the benchmark results.
Note
SkyPilot Benchmark does not consider the time/cost of provisioning and setup.
The columns (such as DURATION and SPENT($)) in the report indicate the time/cost spent in executing the run section of your task YAML.
Note
Here, the columns other than DURATION and SPENT($) are empty.
To get a complete benchmark report, please refer to SkyCallback.
Managing benchmark reports¶
sky bench ls shows the list of the benchmark reports you have:
# List all the benchmark reports
$ sky bench ls
BENCHMARK TASK LAUNCHED CANDIDATE 1 CANDIDATE 2 CANDIDATE 3 CANDIDATE 4
bert bert_qa 2022-08-10 10:07:27 1x Standard_NC6_Promo (K80:1) 1x g4dn.xlarge (T4:1) 1x g5.xlarge (A10G:1) 1x n1-highmem-8 (V100:1)
mybench mytask 2022-08-10 11:24:27 1x n1-highmem-8 (V100:1) 1x g4dn.xlarge (T4:1)
To delete a benchmark report, use sky bench delete:
# Delete the benchmark report on `mybench`
$ sky bench delete mybench