YellowDog Launches AMD Confidential VMs at Scale with Google Cloud
May 25, 2022 — YellowDog has launched 1,100 Confidential VMs (C2D-standard-112 instances) across 4 regions (Asia-East, Asia-Southeast, Europe, and Central US) on Google Cloud.
This has created one of the world’s largest secure C2D cloud clusters to date with 123,200 vCPUs fully provisioned in 19 minutes. As each instance was brought online they were issued tasks by the YellowDog Scheduler and commenced work.
Spot Virtual Machines were used, and these are provided at a lower cost than regular instances but can be withdrawn (pre-emption) at short notice.
YellowDog is self-healing and handles pre-emptions automatically by discovering and provisioning additional virtual machines to make up for the shortfall and re-distributing incomplete tasks to ensure that deadlines are met, whilst also giving customers the benefit of the lowest compute costs. The two graphs below show the VM locations, and the time taken to provision, schedule and start running a workload.
YellowDog also used Google Cloud Storage for both seeding and collecting workload data across these regions.
The key to creating clusters at scale is to ensure that VMs start working on tasks as soon as they have been provisioned. Throughout the workload execution, YellowDog ensures the cluster is used to its maximum capacity and any instances lost due to pre-emptions are automatically replaced.
The Workload
The workload used in this cluster is Molecular Docking, a key stage in the Drug Discovery pipeline in Life Sciences which lends itself to being run at scale. Using Confidential VMs makes perfect sense, given the large investment Life Sciences companies make in new drug lines. This compute option massively increases the security options available and is of huge benefit.
Harvard University’s Real Database contains over 1.4 billion commercially available on-demand molecules. This database can be used by scientists to find potential hit compounds for the development of new drugs and therapeutics. YellowDog used this database, AutoDock Vina, an open-source application for molecular docking and Open Babel (also open-source) to convert and store data from the docking simulations.
Prior to job execution files representing “virtual” molecules are uploaded to Google Cloud Storage (GCS) whilst after completing each analysis the docking scoring results, and job outputs are returned to GCS using YellowDog’s Object Store service. Note, GCS is mounted in the working directory of the instance to reduce data transfer activity.
During execution of the workload, instances were provisioned and issued tasks by the YellowDog Scheduler which utilized each vCPU at 100% until completion.
Comparison
Separately, YellowDog also executed a smaller workload to compare job duration between confidential and non-confidential VMs using a single region and instance type. There was no discernible difference in performance as identified in job durations. This is important as in this use case there is minimal additional cost in deploying this significantly increased level of security.
To read the full details of this news, visit this link.
Source: YellowDog
