Monthly Archives: June 2019

Kubernetes and Container Registry

In this blog we are going to show how to set up an Azure Kubernetes Cluster, an Azure Container Registry and run a container on AKS.

I’m presuming we all know what Kubernetes is, because going into that subject will take a few extra episodes of the blog. For more information take a look at the following:

kubernetes.io
AKS Documentation
ACR Documentation

In this blog we are going to create everything from scratch to setup a new Container Registry and a Kubernetes Cluster.

We need the following items.

  • Resource group
  • Key Vault
  • Service Principle
  • Container Registry
  • Kubernetes Cluster
  • Docker image

We also need to following software installed:
KUBECTL
if you have AZ CLI tools installed you can do this via:

az aks install-cli

otherwise it can be downloaded from:

https://kubernetes.io/docs/tasks/tools/install-kubectl/

DOCKER Desktop
Can be downloaded from https://hub.docker.com

In the code we’ll be using the newer AZ commands instead of AzureRM. The ARM templates that are used are mostly the default ones provided by Microsoft but we’ve added that creation of some default tags to it.
The ARM templates and source code for this blog can be downloaded from:
https://github.com/rexdekoning/kubernetesblog

First connect to your subscription

$Cred = Get-Credential  
Connect-AzAccount -SubscriptionId "[your sub id]" -Credential $Cred

Now we set some default parameters:

$ResourceGroupName        = 'rdk-akstest'
$ResourceGroupLocation    = 'WestEurope'
$ServicePrincipalName     = 'AKSClusterDemo'
$KeyVaultName             = 'AKSKeyVaultRDK'
$ClusterName              = 'rdkakscluster'
$Owner                    = 'Rex de Koning'
$RegistrySKU              = 'Basic'
$RegistryName             = 'methosregistry'
$DockerEmail              = 'rex@methos.nl'
$AgenVMSize               = 'Standard_DS2_v2'
$KubernetesVersion        = '1.12.8'
$NetworkPlugin            = 'kubenet'
$AgentCount               = 1
$Tags =  @{ Owner="$Owner" };
$ServicePrincipalName += $ResourceGroupName

First we make sure our Resource Group exists:

#Create resource group
$resourceGroup = Get-AzResourceGroup -Name $ResourceGroupName -ErrorAction SilentlyContinue
if(!$resourceGroup) {
    New-AzResourceGroup -Name $resourceGroupName -Location $ResourceGroupLocation -Tag $Tags
}

After this we create our KeyVault:

$keyVault = Get-AzKeyVault -VaultName $KeyVaultName -Tag $Tags
if (!$keyVault) {
    $keyVault = New-AzKeyVault -VaultName $KeyVaultName -ResourceGroupName $ResourceGroupName -Location $ResourceGroupLocation -EnabledForTemplateDeployment -Tag $Tags
}

When we have the KeyVault we can create our service principle and store the data in our KeyVault.

In this demo we create a serviceprincipal without defining any fine-grained roles and/or rights. By only supplying the Displayname a serviceprinciple without any specific rights is created and an ApplicationID is generated.

$servicePrincipal = Get-AzADServicePrincipal -DisplayName $ServicePrincipalName
if (!$servicePrincipal) {
    $servicePrincipal = New-AzADServicePrincipal -DisplayName $ServicePrincipalName
    $Ptr = [System.Runtime.InteropServices.Marshal]::SecureStringToCoTaskMemUnicode($servicePrincipal.Secret)
    $result = [System.Runtime.InteropServices.Marshal]::PtrToStringUni($Ptr)
    Set-AzKeyVaultSecret -VaultName $KeyVaultName -Name $servicePrincipal.ApplicationId -SecretValue $servicePrincipal.Secret -Tag $Tags
    $ServicePrincipalSecret = $result
} else {
    $ServicePrincipalSecret = (Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name $servicePrincipal.Id).SecretValueText
}

We can now create our Container Registry, this will also return the admin credentials that are created, and we also store those in our KeyVault:

$CRSParameters = @{
    "registryName"     = $RegistryName
    "registryLocation" = $ResourceGroupLocation
    "registrySku"      = $RegistrySKU
    "adminUserEnabled" = $true
}
$UserName = ""
$Password = ""
$Server = ""
$CRSDeploy = New-AzResourceGroupDeployment -Name "Deployment" -ResourceGroupName $ResourceGroupName -TemplateFile .\crs.json -TemplateParameterObject $CRSParameters #-Verbose 
$CRSDeploy.Outputs.GetEnumerator() | ForEach-Object {
    $myObject = $_
    switch($_.Key) {
        "registryUsername" { $UserName = $myObject.value.Value; break }
        "registryPassword" { $Password = $myObject.value.Value; break }
        "registryServer"   { $Server   = $myObject.value.Value; break }
        default { break }
    }
}

$Password = ConvertTo-SecureString -String $Password -AsPlainText -Force
Set-AzKeyVaultSecret -VaultName $KeyVaultName -Name $UserName -SecretValue $Password -Tag $Tags

Now we can create our AKS cluster. For this demo we our only going to create 1 node ( $AgentCount ). Normally you would create at least 3 nodes. Also the serviceprincipal that is used for this deployment has no specific rights as mentioned before. Normally you would create a serviceprincipal with specific rights so that the serviceprincipal has access to the container registry, create a loadbalancer when needed.

In one of the next blogs we will explain more about serviceprincipals and rights/role assignment.

$DeployParameters = @{
    "resourceName"                 = "$ClusterName"
    "location"                     = "$ResourceGroupLocation"
    "dnsPrefix"                    = "$ClusterName"
    "agentCount"                   = $AgentCount
    "agentVMSize"                  = "$AgenVMSize"
    "servicePrincipalClientId"     = "$($servicePrincipal.ApplicationId)"
    "servicePrincipalClientSecret" = "$ServicePrincipalSecret"
    "kubernetesVersion"            = "$KubernetesVersion"
    "networkPlugin"                = "$NetworkPlugin"
    "enableRBAC"                   = $true
    "enableHttpApplicationRouting" = $false
    "Owner"                        = "$Owner"
}
$Deployment = New-AzResourceGroupDeployment -Name "Deployment" -ResourceGroupName $ResourceGroupName -TemplateFile .\aks.json -TemplateParameterObject $DeployParameters

We can also output the created clustername:

$Deployment.Outputs.GetEnumerator() | ForEach-Object {
    Write-Output "$($_.Key) : $($_.value.Value)"
}

At this time we can begin to work with our cluster. First we need our credentials. We have our normal credentials which we can get via:

# Get AKS Cluster Credentials for kubectl
Import-AzAksCredential -ResourceGroupName $ResourceGroupName -Name $ClusterName -Force

If needed for any reason it is also possible to get the admin user via:

# Get Admin user
#Import-AzAksCredential -ResourceGroupName $ResourceGroupName -Name $ClusterName -Admin -Force

After we imported the AKS credentials we can check if our node(s) are up using:

#Check if our nodes are up
kubectl get nodes --output=wide

When our nodes are up we can start connecting to our container registry, for this we first get the password from the KeyVault

#Get dockerpassword from Vault
$DockerPassword = Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name $UserName

Write-Output "Login to registry"
$DockerPassword.SecretValueText | docker login $server -u $UserName --password-stdin

Now that we have everything in place we can start creating a docker image or re-use an existing one. For now we are going to re-use an existing NGINX Demo docker image which contains Hello World

Write-Output "Download default Hello-World image"
docker pull nginxdemos/hello

Write-Output "Re-tag image"
docker tag nginxdemos/hello $server/hello:1.0

Write-Output "Push Image to CRS"
docker image push $server/hello:1.0

We now re-tagged an existing imaged and pushed it to our own private container registry.

After this it is time to link our AKS Cluster to our Container Registry

#Create secret to Link AKS to CRS
kubectl create secret docker-registry $server --docker-server=$server --docker-username=$UserName --docker-password=$($DockerPassword.SecretValueText) --docker-email=$DockerEmail

We can issue a command to check the contents of the now created secret

#Check the secret
kubectl describe secret

We can create our kubernetes yaml to start our own pods. In this case we fill a variable with the content but it could also be a file.

$yaml = @"
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: my-api
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: my-api
    spec:
      containers:
      - name: my-api
        image: $server/hello:1.0
        ports:
        - containerPort: 80
      imagePullSecrets:
      - name: $server
---
apiVersion: v1
kind: Service
metadata:
  name: my-api
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: my-api
"@

Now that we have the YAML we can create a deployment to Kubernetes.
For this demo we use the output of $yaml via piping to STDIN as input for kubectl create -f as specified by ‘-‘.

#create deployment using yaml content via STDIN
$yaml | kubectl create -f -

If you would like to use a file with YAML content you can use:

kubectl create -f .\file.name

We can get the deployment status via:

kubectl get service/my-api

When the deployment is ready the external IP will be visible and it can be used to open our demo page in the browser:

in my current case:

start http://23.100.7.99

So this shows, that in little time and with little effort several services can be spun up in Azure and you can ran your own docker images on an Azure Kubernetes Cluster.

It is also possible to connect to a web-based dashboard for our AKS Cluster. For this we issue the following commands:

#Get Kubernetes Dashboard
kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
kubectl proxy

http://localhost:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/#!/login

When first accessing the dashboard it will ask for the .kubeconfig file. Under windows this is: %userprofile%\.kube\config

To clean up the resources we just creates we can issue the following commands:

# Clean up Azure resources
Remove-AzAks -ResourceGroupName $ResourceGroupName -Name $ClusterName -Force
Remove-AzADServicePrincipal -DisplayName $ServicePrincipalName -Force
Remove-AzADApplication -DisplayName $ServicePrincipalName -Force
Remove-AzResourceGroup -Name $ResourceGroupName -Force