Preparing and Joining your Nodes to the Controller
Preparing your Nodes
Once you’ve set up your Controller & Registry, you’ll want to install Anka Build Virtualization software onto a macOS machine and join it to your Build Cloud.
This guide has a few options that are typically not available on MDM controlled machines. You may need to ask your system administrators/local IT team to allow them.
Be sure to reboot the host after applying these changes.
Host Preparation (Optional, but Recommended)
Enable
Automatic Login
for the current user: Go to Preferences > Users > Enable Automatic Login for the current user. Or, usesudo sysadminctl -autologin set -userName {USERTOENABLE} -password {PASSWORD}
.Disable spotlight on the host and also inside of the VM:
If you cannot perform launchctl commands, you can execute these commands from the command-line:
# Disable indexing volumes sudo defaults write ~/.Spotlight-V100/VolumeConfiguration.plist Exclusions -array "/Volumes" || true sudo defaults write ~/.Spotlight-V100/VolumeConfiguration.plist Exclusions -array "/Network" || true sudo killall mds || true sleep 60 sudo mdutil -a -i off / || true sudo mdutil -a -i off || true sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist || true sudo rm -rf /.Spotlight-V100/* rm -rf ~/Library/Metadata/CoreSpotlight/ || true killall -KILL Spotlight spotlightd mds || true sudo rm -rf /System/Volums/Data/.Spotlight-V100 || true
MDS can be disable entirely with
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist
, but only if SIP is disabled on the host (not recommended).
You may also want to have your nodes restart on host level failure:systemsetup -setrestartpowerfailure on
&systemsetup -setrestartfreeze on
Joining Prerequisites
Add the Registry
This is not needed on all Nodes connecting to the Controller. Most users have a “builder machine” that has the registry connection and is used to create and push VM Templates/Tags.
Assuming you haven’t changed the default port configuration, your Registry is serving requests on port 8089
.
We now need to configure the Registry on this machine so we can push/upload the local VM Template so other machines connected to the Anka Build Cloud. Uploading the Template to the Registry makes it possible to download and run it from other nodes.
sudo anka registry add <registry name> http://<ip>:8089
Verify the configuration:
sudo anka registry list-repos
++
++
<registry name you set> (default)
+--------+------------------+
| host | <the ip you set> |
+--------+------------------+
| scheme | http |
+--------+------------------+
| port | 8089 |
+--------+------------------+
Then, confirm the registry list command shows your VM Template:
sudo anka registry list
Push the VM to the Registry
sudo anka registry -r http://anka.registry:8089 push 12.X -t base
In the example above,-r {URL}
is used, but is not required if you’ve added the Registry usinganka registry add
.
After the push completes, you should see your new Template in the “Templates” section of the controller UI.
Join to the Controller & Registry
Be sure to run ankacluster as root.
Avoid using underscores in your domainnames/urls.
❯ sudo ankacluster join http://<ip>
Password:
Testing connection to controller...: Ok
Testing connection to the registry...: Ok
Ok
Cluster join success
- Replace
<ip>
with the IP of the machine hosting your controller: - If you changed the default port for the controller from 80, you’ll need to use the new port at the end of the IP. Otherwise, leave it off.
The command may hang for a few moments and then display Cluster join success
. Please report any errors you find to support@veertu.com.
The Anka agent is listening on a socket to provide status information at runtime.
You can override the path of the socket by setting the ANKA_AGENT_SOCKET
env var.
❯ sudo ankacluster join --help
Joins the current machine to one or many Anka Build Cloud Controllers
Usage:
ankacluster join CONTROLLER_ADDRESS[,CONTROLLER_ADDRESS2] [flags]
Flags:
--api-key-file string The (non-base64) API Key file path. Takes precedence over api-key-string
--api-key-id string API Key identifier
--api-key-string string API Key string
-c, --cacert string Specify the path to your Root CA Certificate (PEM/X509)
-M, --capacity-mode string Set the capacity mode (resource or number) the Node will use when calculating if it can start a VM task - 'resource' will look at total unused CPU cores and RAM available - 2GB (see --vcpu-override and --ram-override) & 'number' will only accept if --max-vm-count isn't already met (default 'number') (default "number")
-C, --cert string Specify the path to the Node's certificate file (PEM/X509)
-K, --cert-key string Specify the path to the Node's certificate key file (PEM/X509)
--cli-start-timeout duration Timeout for anka start command (default 1m30s)
--dial-timeout duration http dial timeout (default 15s)
--drain-mode Join node with Drain Mode
--dump-network-meter Dump aggregation of http stats to a file (default true)
--dump-network-meter-file-name string Filepath to dump http stats to, dir is log dir (default "http-dump.json")
--enable-vm-monitor Enabled unresponsive VM monitoring. This will throw a failure when the VM becomes unresponsive for longer than the --vm-stuck-timeout
-f, --force-no-sudo Force the anka_agent to start without sudo
-g, --global DEPRECATED! Install agent into system domain
-G, --groups string Specify group name (or multiple names sepearated by ',') to add the current Node to
--heartbeat duration Set the duration between status updates the Node sends to the Anka Cloud Cluster (default: 5 seconds)
-h, --help help for join
-H, --host string Set the address (IP or Hostname) of the Node that the Anka Cloud Cluster will use when communicating with CI tools/plugins. This is useful when your CI tool cannot connect to the Node's local IP address (the default value of --host), but does have access to an external IP or hostname for it (proxy, load balancer, etc).
--ignore-arm-vm-limit Ignore ARM VM Limit
-k, --keystore string Specify the path to your certificate keystore (PEM/PKCS12)
-p, --keystore-pass string Specify the password for your certificate keystore
--max-idle-connections-per-host int mac idle connections per host (default 50)
-m, --max-vm-count int Set the maximum number of VMs this Node is allowed to run (default: 2) (default 2)
-n, --name string Set a custom Node name (default: hostname)
--no-central-logging Disable sending logs to central logging location
--node-id string Custom node id
--num-http-sample-threshold int The agent will aggregate http stats after this amount of requests (default 1000)
--num-workers int The number of concurrent worker executing tasks (default 2)
--optimization-threshold int Number of times to retry to start VM before starting disk space optimization (default 5)
-R, --ram-override int Override ram limit for resource based capacity (default: {total ram} - 2GB)
--request-timeout duration http request timeout (default 1m0s)
--reserve-space string Disk space to reserve when pulling. Number followed by magnitude (1024B, 10KB, 140MB, 45GB...) (default: 20% of disk size)
-r, --root-cert string Identical to --cacert
--skip-pull-error string pass no-download-bytes for ignoring pull errors when download bytes is 0
--skip-tests Disable testing the connection before starting the agent
--skip-tls-verification Skip TLS verification
-t, --tls Enable TLS for communicating with the Anka Cloud Cluster
--tls-handshake-timeout duration tls handshake timeout (default 5s)
-V, --vcpu-override int Override vcpu limit for resource based capacity (default: {current physical(intel)/performance(arm) cpu count} * 2)
--verbosity string verbosity level, 0 - 10 (higher number - more chatty)
--vm-stuck-delay duration The time between unresponsive VM checks (default: 30s - Duration examples: 3500s, 20m, 5h)
--vm-stuck-timeout duration The time to wait until the VM is considered unresponsive (default: 10s - Duration examples: 3500s, 20m, 5h)
Check Join Status
You can check the status of the Anka agent using the sudo ankacluster status
command.
❯ sudo ankacluster status
status: running
config:
vm_limit: 2
optimization_threshold: 5
num_workers: 2
controller_addresses:
- http://anka.controller.dev
version: 1.13.0-6cd34a2c
capacity_mode: number
heartbeat: 5s
node_name: MyMacMiniNode
vm_stuck_check_delay: 30s
vm_stuck_check_timeout: 10s
Start a VM instance using the Controller UI
Go to your Controller dashboard and click on the Instances tab:
Click on Create Instance(s), and the Create New Instances view displays:
Select the VM Template and click Start. The Create New Instances view closes and returns you to the Instances view. You should now see the Instance in a Scheduling or Pulling State:
After the Scheduling and Pulling finishes, the VM starts on one of the Nodes and shows a Started State in the Controller UI:
You can now confirm the Instance is running from inside the Node:
- JSON output is available for scripting/automation using
anka --machine-readable
sudo anka --machine-readable list | jq
{
"status": "OK",
"body": [
{
"status": "suspended",
"name": "catalina",
"stop_date": "2020-04-01T21:30:59.798697Z",
"creation_date": "2020-04-01T00:00:13.656296Z",
"version": "base",
"uuid": "10c720eb-dcce-46f7-baa3-28bacef0ec0f"
},
{
"status": "running",
"name": "mgmtManaged-catalina-1585776660490226000",
"stop_date": "2020-04-01T21:36:11.742662Z",
"creation_date": "2020-04-01T21:31:01.055250Z",
"version": "",
"uuid": "dcbeb319-421a-4d30-8466-194eb7fa5f75"
}
],
"message": ""
}
Timestamp format: https://www.ietf.org/rfc/rfc3339.txt
Disjoining
You don’t need to disjoin nodes to upgrade the Anka Virtualization package.
❯ sudo ankacluster disjoin
Disjoined from the Anka Cloud Cluster
Answers to Frequently Asked Questions
- Errors like the following are typically the cause of an older version of the agent on your node. You’ll want to visit http://downloads.veertu.com, download the agent version matching your controller’s version, and then try your ankacluster join command again:
> sudo ankacluster join http://anka.controller --reserve-space 20GB Testing connection to controller...: Ok Testing connection to the registry...: Ok Ok exit status 2 Log file created at: 2021/07/16 12:36:10 Running on machine: hostname1 Binary: Built with gc go1.14.3 for darwin/amd64 Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg I0716 12:36:10.164826 8172 log_cleaner.go:34] clean /var/log/veertu/anka_agent*... I0716 12:36:10.166385 8172 log_cleaner.go:56] don't touch: /var/log/veertu/anka_agent.hostname1.root.log.INFO.20210716-123610.8172 I0716 12:36:10.172275 8172 main.go:173] starting ankaAgent I0716 12:36:10.172282 8172 main.go:174] Anka Agent version: 1.7.1-9545c9f5 I0716 12:36:10.965625 8172 runner.go:43] Anka version is 2.4.1 I0716 12:36:10.965806 8172 runner.go:46] Agent Version is v2 Joining cluster failed
- The Controller will be checking disk space on every pull/preparation of a VM. If not enough disk space is available, it will automatically delete VM Templates/tags from the Node based on which has the oldest last used timestamp until there is enough space for the VM Template/Tag. This cannot be disabled at the moment. This can be modified to some extent by using –reserve-space flag (see explanation above)
- The registry
External URL
(seen in the controller UI/config) is used by the Nodes to pull down templates and tags. Be sure to set this URL properly in your Build Cloud Configuration and ensure firewalls allow communication.