kskilling2017Leave a Comment on VCF 3.10 Adding a cluster to an existing WLD using the API

VCF 3.10 Adding a cluster to an existing WLD using the API

In my previous post https://kskilling.com/2020/10/14/vcf-3-10-deploying-workload-domains-wld-via-the-api/ I covered the steps needed to create a new WLD in VCF 3.10 via the API.

This post will cover the steps needed to expand that WLD by adding a new cluster.

My customer use case was to have a single WLD with multiple clusters to allow their virtual machines to be hosted separately based on their application type. Just like the first cluster deployed when the WLD was created, all hosts in the additional clusters would have four physical network adapters. These PNics would need to be split across two Distributed Virtual Switches (DVS) to allow management and virtual machine traffic to be separated.

As with my previous post I will use examples of the JSON from the VMware documentation and adapt them to match my use cases, as I can not directly share the information used for my customer.

Prerequisites

Just like the deployment of a WLD, creating a new cluster via the API still requires you to do the following:

  1. Create Network Pools for vMotion and vSAN/NFS the new cluster (if using dedicated IP ranges per cluster rather than per WLD)
  2. Prepare the ESXi hosts as per the VCF documentation
  3. Add the hosts to SDDC Manager via the Commission Hosts process and assign them to the correct Network Pool either via the GUI or the API.

You will also need the following tools:

  1. Tool to edit and display JSON files.
  2. Tool such as Curl or Postman to submit API calls. I will be using Curl on the SDDC Manager appliance due to restrictions in my customer environment.
  3. If using Curl on SDDC Manager a tool to upload files to the SDDC Manager appliance such as WinSCP.
  4. The credentials of the SDDC Manager REST API User (admin) and the SDDC Manager Super User (vcf).

Creating a Cluster

Payload JSON – PNic and vDS association

Reference: Section 2.8.1 of the VCF API documentation https://vdc-download.vmware.com/vmwb-repository/dcr-public/2d4955d7-fb6f-4a61-be78-64d95b951ccd/c6e26ae1-9438-4da0-bfc7-2e21d9046820/index.html#_create_a_cluster

Just like the payload JSON for creating a new WLD the JSON specification for creating a cluster contains a section to specify how the PNics of the hosts should be assigned to the virtual switches. This section needs to be updated to include separating the PNics onto multiple switches.

For each host that is listed in the hostSpecs section of the JSON the hostNetworkSpec section is amended to add a new vmNic entry for each physical adapter. The vdsName is then set to the name of the vDS you want to attach that adapter to, in my customer use case two PNics were to be assigned to each vDS, which gave a hostSpecs section like the example below:

"clusterSpecs" : [ {
        "name" : "Cluster1",
        "hostSpecs" : [ {
          "id" : "<Host1_ID>",
          "license":  "<License Key>",
          "hostNetworkSpec" : {
            "vmNics" : [ {
              "id" : "vmnic0",
              "vdsName" : "SDDC-Dswitch-Private1"
            }, {
              "id" : "vmnic1",
              "vdsName" : "SDDC-Dswitch-Private0"
            },{
              "id" : "vmnic2",
              "vdsName" : "SDDC-Dswitch-Private1"
            },{
              "id" : "vmnic3",
              "vdsName" : "SDDC-Dswitch-Private0"
            } ]
          }
        },

The second change needed is to the networkSpec section where the types of traffic associated with each vDS instance is specified. For my customer the traffic was to be separated so that all management related traffic used one vDS and the VM traffic would use a second vDS. . This gave a networkSpec section like the example below:

"networkSpec" : {
          "vdsSpecs" : [ {
            "name" : "SDDC-Dswitch-Private0",
            "portGroupSpecs" : [ {
              "name" : "SDDC-DPortGroup-Mgmt",
              "transportType" : "MANAGEMENT"
            }, {
              "name" : "SDDC-DPortGroup-VSAN",
              "transportType" : "VSAN"
            }, {
              "name" : "SDDC-DPortGroup-vMotion",
              "transportType" : "VMOTION"
            }]
          },
          {
            "name" : "SDDC-Dswitch-Private1",
            "portGroupSpecs" : [ {
              "name" : "SDDC-DPortGroup-VM",
              "transportType" : "PUBLIC"
            }]
          } ],
          "nsxClusterSpec" : {
            "nsxVClusterSpec" : {
              "vlanId" : 0,
              "vdsNameForVxlanConfig" : "SDDC-Dswitch-Private0"
            }
          }
        }
      } ]
    },

Payload JSON – Gathering the ESXi Host ID values

Reference section 2.5.2 of the VCF API documentation https://vdc-download.vmware.com/vmwb-repository/dcr-public/2d4955d7-fb6f-4a61-be78-64d95b951ccd/c6e26ae1-9438-4da0-bfc7-2e21d9046820/index.html#_usecase_gethosts

Within the payload JSON hostSpecs example in the previous section the id field of each host contained a placeholder value such as <Host1_ID>. The value required for this field is the unique id assigned by SDDC Manager to the ESXi host when it completes the Commission process. The id value is not displayed in the GUI as far as I could locate, and therefore must be retrieved via the API.

Note: This id value does not persist if a host is decommissioned and then added back to SDDC Manger.

To get the list of hosts you can perform a GET operation on SDDC Manager, however it is worth using a filtered API call to query just for hosts marked as having a status of UNASSIGNED_USEABLE. This status means a host is available for use and can be assigned to a new WLD or cluster, this filters out any hosts already assigned to WLDs. You could also use a filtered search to look for hosts that are capable with the WLD the cluster belongs to based on their PNic configuration, however I simply reused a previous api call to look for UNASSIGNED_USEABLE since all my hosts have the same PNic configuration.

The API url to use is

<SDDC Manager FQDN>/v1/hosts?status=UNASSIGNED_USABLE

When using curl on the SDDC Manager appliance I like to send the output to a file so that I can view it using a JSON viewer rather than as a single stream of text. This is also useful if I have multiple WLDs or tasks to complete and will need different hosts from SDDC Manager each time. The command to run is:

curl "https://<SDDC Manager FQDN>/v1/hosts?status=UNASSIGNED_USABLE" -i -u "admin:<SDDC Manager API User Password>" -X GET -H "Accept: application/json" > <json output file name>

The output of the command contains details about the specification of each host matching the status filter. If you remove the header text:

HTTP/1.1 200 OK 
Content-Type: application/json 
Content-Length: 1424

from the file you can open it with a standard JSON viewer. The values you are interested in are the id fields of each host. Expand the elements section and each numbered entry represents a single host. Locate the hosts you want to use for the new cluster and copy the value from the id field. Paste the value into the payload JSON for the new cluster to replace the appropriate placeholder value.

Payload JSON – Gathering the Domain ID

As the cluster being created will belong to an existing WLD, the payload JSON needs to specify which WLD the cluster is to be added to. Just like the ESXi hosts, this value needs to be specified as an id rather than a name, and is not shown in the SDDC Manager GUI as far as I can tell.

Reference section 2.7.2 of the VCF API documentation https://vdc-download.vmware.com/vmwb-repository/dcr-public/2d4955d7-fb6f-4a61-be78-64d95b951ccd/c6e26ae1-9438-4da0-bfc7-2e21d9046820/index.html#_get_the_domains

The process to retrieve the WLD ids via the API is similar to getting the host ids, a simple GET command is used to return a list of ids.

The API to use is:

<SDDC Manager FQDN>/v1/domains

Just like when I gather the host information I like to send the output to a file so that I can view it using a JSON viewer rather than as a single stream of text, however if you only have one WLD then you can easily copy the id value from the results without performing this additional step. The command to run is:

curl "https://<SDDC Manager FQDN>/v1/domains" -i -u "admin:<SDDC Manager API User Password>" -X GET -H "Accept: application/json" > <json output file name>

As with the hosts, remove the header text from the file and you can view it as a JSON file. Each WLD is listed as a separate entry and the value you need for the domain id is the value of the id field.

Paste the value into the payload JSON for the new cluster to replace the appropriate placeholder value for the domain id near the top of the specification.

Validating the Payload JSON

Reference section 2.8.1 of the VCF API documentation https://vdc-download.vmware.com/vmwb-repository/dcr-public/2d4955d7-fb6f-4a61-be78-64d95b951ccd/c6e26ae1-9438-4da0-bfc7-2e21d9046820/index.html#_create_a_cluster

Fill in the remaining values in the JSON such as vlan ids and your JSON is ready for validation. I also included settings for the advancedOptions section to define the evcMode and highAvailability mode for the cluster. My final spec looked like the following example:

{
  "clusterCreationSpec" : {
    "domainId" : "b8e5899c-a0df-40ee-8e3c-078a0613a819",
    "computeSpec" : {
      "clusterSpecs" : [ {
        "name" : "Cluster1",
        "hostSpecs" : [ {
          "id" : "<Host1_ID>",
          "license":  "<License Key>",
          "hostNetworkSpec" : {
            "vmNics" : [ {
              "id" : "vmnic0",
              "vdsName" : "SDDC-Dswitch-Private1"
            }, {
              "id" : "vmnic1",
              "vdsName" : "SDDC-Dswitch-Private0"
            },{
              "id" : "vmnic2",
              "vdsName" : "SDDC-Dswitch-Private1"
            },{
              "id" : "vmnic3",
              "vdsName" : "SDDC-Dswitch-Private0"
            } ]
          }
        },{
          "id" : "<Host2_ID>",
          "license":  "<License Key>",
          "hostNetworkSpec" : {
            "vmNics" : [ {
              "id" : "vmnic0",
              "vdsName" : "SDDC-Dswitch-Private1"
            }, {
              "id" : "vmnic1",
              "vdsName" : "SDDC-Dswitch-Private0"
            },{
              "id" : "vmnic2",
              "vdsName" : "SDDC-Dswitch-Private1"
            },{
              "id" : "vmnic3",
              "vdsName" : "SDDC-Dswitch-Private0"
            } ]
          }
        },{
          "id" : "<Host3_ID>",
          "license":  "<License Key>",
          "hostNetworkSpec" : {
            "vmNics" : [ {
              "id" : "vmnic0",
              "vdsName" : "SDDC-Dswitch-Private1"
            }, {
              "id" : "vmnic1",
              "vdsName" : "SDDC-Dswitch-Private0"
            },{
              "id" : "vmnic2",
              "vdsName" : "SDDC-Dswitch-Private1"
            },{
              "id" : "vmnic3",
              "vdsName" : "SDDC-Dswitch-Private0"
            } ]
          }
        }],
       "datastoreSpec" : {
          "vsanDatastoreSpec" : {
            "failuresToTolerate" : 1,
            "licenseKey" : "XXXX-XXXX",
            "datastoreName" : "vSanDatastore"
          }
        },
        "networkSpec" : {
          "vdsSpecs" : [ {
            "name" : "SDDC-Dswitch-Private0",
            "portGroupSpecs" : [ {
              "name" : "SDDC-DPortGroup-Mgmt",
              "transportType" : "MANAGEMENT"
            }, {
              "name" : "SDDC-DPortGroup-VSAN",
              "transportType" : "VSAN"
            }, {
              "name" : "SDDC-DPortGroup-vMotion",
              "transportType" : "VMOTION"
            }]
          },
          {
            "name" : "SDDC-Dswitch-Private1",
            "portGroupSpecs" : [ {
              "name" : "SDDC-DPortGroup-VM",
              "transportType" : "PUBLIC"
            }]
          } ],
          "nsxClusterSpec" : {
            "nsxVClusterSpec" : {
              "vlanId" : 0,
              "vdsNameForVxlanConfig" : "SDDC-Dswitch-Private0"
            }
          }
    },
        "advancedOptions" : {
          "evcMode" : "",
          "highAvailability" : {
            "enabled" : true
          }
        }
      } ]
    }
  }
}

If you can check the JSON structure is valid first using a JSON formatter or viewer, this eliminates missing quotes or curly braces. When you have a valid JSON file upload it to the SDDC Manager appliance (if using curl on the appliance) using the vcf user account.

To validate that the JSON file matches the expected format for SDDC Manager it can be submitted via an API call using the URL

<SDDC Manager FQDN>/v1/clusters/validations

The full curl command to use, parsing the JSON from a file rather than on the command line is

curl "https://<SDDC Manager FQDN>/v1/clusters/validations" -i -u "admin:<SDDC Manager API User Password>" -X POST -H "Accept: application/json" -d @ <json input file name>

If the command returns an executionStatus value of COMPLETED and a resultStatus value of SUCCEEDED the JSON is a valid cluster spec file. In the case of an error it will highlight in the error message the cause of the failure, such as an expected field is missing.

Submitting the Payload JSON

Reference section 2.8.1 of the VCF API documentation https://vdc-download.vmware.com/vmwb-repository/dcr-public/2d4955d7-fb6f-4a61-be78-64d95b951ccd/c6e26ae1-9438-4da0-bfc7-2e21d9046820/index.html#_create_a_cluster

The final stage of the cluster creation process is to submit the validated JSON to create the task for the creation of the cluster. Before making the final API call the JSON file used for the validation API call needs to be modified in the same way as we did for the WLD. The JSON spec used for the cluster creation task does not contain the clusterCreationSpec field at the start of the file and you need to remove the curly braces associated with it as well.

Using the example spec from the API documentation for the validation stage the final file looks like this:

{
    "domainId" : "b8e5899c-a0df-40ee-8e3c-078a0613a819",
    "computeSpec" : {
      "clusterSpecs" : [ {
        "name" : "Cluster1",
        "hostSpecs" : [ {
          "id" : "<Host1_ID>",
          "license":  "<License Key>"
          "hostNetworkSpec" : {
            "vmNics" : [ {
              "id" : "vmnic0",
              "vdsName" : "SDDC-Dswitch-Private1"
            }, {
              "id" : "vmnic1",
              "vdsName" : "SDDC-Dswitch-Private0"
            },{
              "id" : "vmnic2",
              "vdsName" : "SDDC-Dswitch-Private1"
            },{
              "id" : "vmnic3",
              "vdsName" : "SDDC-Dswitch-Private0"
            } ]
          }
        },{
          "id" : "<Host2_ID>",
          "license":  "<License Key>"
          "hostNetworkSpec" : {
            "vmNics" : [ {
              "id" : "vmnic0",
              "vdsName" : "SDDC-Dswitch-Private1"
            }, {
              "id" : "vmnic1",
              "vdsName" : "SDDC-Dswitch-Private0"
            },{
              "id" : "vmnic2",
              "vdsName" : "SDDC-Dswitch-Private1"
            },{
              "id" : "vmnic3",
              "vdsName" : "SDDC-Dswitch-Private0"
            } ]
          }
        },{
          "id" : "<Host3_ID>",
          "license":  "<License Key>"
          "hostNetworkSpec" : {
            "vmNics" : [ {
              "id" : "vmnic0",
              "vdsName" : "SDDC-Dswitch-Private1"
            }, {
              "id" : "vmnic1",
              "vdsName" : "SDDC-Dswitch-Private0"
            },{
              "id" : "vmnic2",
              "vdsName" : "SDDC-Dswitch-Private1"
            },{
              "id" : "vmnic3",
              "vdsName" : "SDDC-Dswitch-Private0"
            } ]
          }
        }],
       "datastoreSpec" : {
          "vsanDatastoreSpec" : {
            "failuresToTolerate" : 1,
            "licenseKey" : "XXXX-XXXX",
            "datastoreName" : "vSanDatastore"
          }
        },
        "networkSpec" : {
          "vdsSpecs" : [ {
            "name" : "SDDC-Dswitch-Private0",
            "portGroupSpecs" : [ {
              "name" : "SDDC-DPortGroup-Mgmt",
              "transportType" : "MANAGEMENT"
            }, {
              "name" : "SDDC-DPortGroup-VSAN",
              "transportType" : "VSAN"
            }, {
              "name" : "SDDC-DPortGroup-vMotion",
              "transportType" : "VMOTION"
            }]
          },
          {
            "name" : "SDDC-Dswitch-Private1",
            "portGroupSpecs" : [ {
              "name" : "SDDC-DPortGroup-VM",
              "transportType" : "PUBLIC"
            }]
          } ],
          "nsxClusterSpec" : {
            "nsxVClusterSpec" : {
              "vlanId" : 0,
              "vdsNameForVxlanConfig" : "SDDC-Dswitch-Private0"
            }
          }
    },
        "advancedOptions" : {
          "evcMode" : "",
          "highAvailability" : {
            "enabled" : true
          }
        }
      } ]
    }
  }

Upload the new JSON file to the SDDC Manager if using curl, and submit the request using the following command:

curl "https://<SDDC Manager FQDN>/v1/clusters" -i -u "admin:<SDDC Manager API User Password>" -X POST -H "Accept: application/json" -d @ <json input file name>

The response will include an id field, which is the task id for the submitted cluster creation task. You can monitor the progress of the task via the SDDC GUI if you have access, or you can use the task id from the output and monitor it via the api using the command:

curl "https://<SDDC Manager FQDN>/v1/tasks/<Task ID>" -i -u "admin:<SDDC Manager API User Password>" -X GET -H "Accept: application/json"

If the task fails and you need to resubmit it using the same JSON spec as before you can use the command:

curl "https://<SDDC Manager FQDN>/v1/tasks/<Task ID>" -i -u "admin:<SDDC Manager API User Password>" -X PATCH -H "Accept: application/json"

The task to add a new cluster is quicker than creating a full WLD, the time required for it to complete will obviously depend on the size of the cluster you are creating.

In my final post for this mini series I’ll cover the steps to add hosts to an existing cluster.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this:
search previous next tag category expand menu location phone mail time cart zoom edit close