Evolution of nsx-t-gen, nsx-t-ci-pipeline and canned-pks toolkits

I had been working and supporting a set of tools aimed at easing and automating install and use of VMware's NSX-T SDN product with Pivotal's PAS (Pivotal Application Service) and PKS (Pivotal Container Service - k8s offering) product versions on vSphere platforms. I wanted to share the evolution of these tools and how they work together.

nsx-t-gen

nsx-t-gen is a toolkit that helps automate the install of VMware NSX-T SDN product on vSphere platform. It uses Concourse pipelines to make a set of tasks easy and automated while keeping the various pieces together towards the objective of fully automation of NSX-T.



The various components used in this toolkit are:

• Github repos for the pipeline definitions, tasks and scripts
• NSX-T and other binary install bits (ovftool)
• User supplied params for vCenter, networks, credentials etc.
• NSX-T Mgmt and runtime configuration for Edges, Mgr, Ctrl along with routers/switches, ip blocks, pools, nat rules, security group etc.

The end result is a fully installed NSX-T Mgmt plane running on a managed cluster (with edges) while using single or multiple compute clusters as transport nodes. Various Cloud native product offerings (like PAS or PKS or others) can be installed and managed on top of this NSX-T managed network infrastructure.

The current version of NSX-T supported against vSphere are (each version is in its own branch):

• 2.3 (most recent release supported with PKS 1.2)
• 2.2
• 2.1

All of these pipelines use similar set of pipeline parameters and tasks while abstracting the version dependencies from the end user to provide the same zero-touch install experience.

Things handled by nsx-t-gen:

• Deploy the VMware NSX-T Manager, Controller and Edge ova images
• Configure the Controller cluster and add it to the management plane
• Configure hostswitches, profiles, transport zones
• Configure the Edges and ESXi Hosts to be part of the NSX-T Fabric
• Create T0 Router (one per run, in HA vip mode) with uplink and static route
• Configure arbitrary set of T1 Routers with logical switches and ports
• NAT Rules setup for T0 Router
• Container IP Pools and External IP Blocks
• Self-signed cert generation and registration against NSX-T Manager
• Route redistribution for T0 Router
• HA Spoofguard Switching Profile
• Load Balancer (with virtual servers and server pool) creation
• Security groups to associate with job types or vms or others (leveraged for PAS to allow Load Balancer server pool to be dynamically linked to GoRouter or TCPRouter or SSH Proxy VM instances from PAS) 

nsx-t-ci-pipeline

The nsx-t-ci-pipelines is aimed at install of Pivotal Products with NSX-T integration with fully automation including and not limited to:

• Installing Pivotal Ops Mgr with NSX-T mode enabled, configuring the Bosh Director
• Installing the NSX-T Add-on tile along with PAS 2.x tile with external cni provider configurations
• Installing PKS tile (v1.0, 1.1.x, and most recent v1.2) with NSX-T enablement, auto-configuring the PKS Super User creds to be used against NSX-T.

• Handle dynamic integration of PAS components with Load balancer (like GoRouter or TcpRouter or Diego Brains) using security group association with the related job groups and using pre-configured server groups tied to security group for membership (handled by nsx-t-gen pipeline), compared to old way of using static IPs for job types and associating them with the Load Balancer server pool membership.
• Automatically configuring NAT rules to allow the PKS API Controller to be reachable externally.
• Automatically creating and configuring the PKS CLI user 
• Install Harbor Tile and automatically configure NAT rules to expose Harbor outside.

Canned PKS

One of the challenges most customers or field people face relates to installing these products in completely offline or isolated environments (no online access to any external resource).

The Concourse pipelines implicitly assume an online, interconnected setup to pull down various resources, github repos and docker images that are internally used as part of a task definition. This makes the whole isolated, offline working model break it.

I helped build a model of a pipeline that was aimed at truly offline pipeline execution to install NSX-T and PKS (Pivotal Container Service) in such limited restricted environments. The outcome was canned-pks.

 

There are two portions to using the canned-pks toolkit:

1. Capture all the materials required for executing the pipeline in purely offline mode by saving or downloading the materials in online mode into a s3 compatible store (minio is a simple, easy and free s3 equivalent that anyone can run on any machine)
• The BOM file has the bill of materials (github repos, docker images, install bits, including Pivotal Tiles and stemcells or VMware NSX-T install bits)
• Tools to download and upload the saved bits into the store 
• All of the above mentioned bits are downloaded and saved using scripts.
2. Run the offline equivalent of the pipeline bundled in the canned-pks repo against the saved resources. The pipeline extracts them into a form that can then be used to execute the actual steps that installs the product, without requiring any online access. 

The complete steps are detailed in the canned-pks repo. It supports an opinionated install view of NSX-T (single compute cluster vs arbitrary # of compute clusters supported in base nsx-t-gen pipeline and certain precooked configs) to achieve a fast, simple and easy install of NSX-T with PKS in offline environments.

Also, I am happy to announce that the nsx-t-gen pipeline codebase has now been adopted by VMware and would be maintained going forward in form of their nsx-t-datacenter-ci-pipelines repo. 

All of these tools should help users get an more easier and faster ramp up experience with NSX-T and PAS/PKS product stack.

Just a note of caution: All of the above tools are not officially supported but free to use at user's own discretion and risk.