Running Kazoo VMs with KVM/QEMU
I've been running Kazoo in CentOS VMs to replicate production setups to solve support tickets. But getting my laptop talking to the VMs running on my dev server was non-obvious to me. I've finally got it working for an all-in-one Kazoo server and learned a little bit too.
The setup
I have three systems to connect:
| Computer | Network IP |
| My laptop (or any computer on my LAN really) | 192.168.1.5 |
| My dev server (beefy blade) | 192.168.1.10 |
| My Kazoo VM (using libvirt) | 192.168.122.21 |
So the dev server running libvirt defines a default network of 192.168.122.0/24 and assigns IP addresses to the VMs from that pool.
By default, the dev server (the host) can talk to the VMs on the subnet and the VMs can talk to each other but computers on my LAN cannot talk to the VM subnet.
What to do?
Setup the laptop
The dev server knows how to route to the VMs so we need the laptop to send packets destined for the VM to the dev server. This is actually straight-forward to accomplish by adding a route to the kernel routing table:
First, check out the routing table:
sudo ip route show
default via 192.168.1.1 dev eth0 proto dhcp metric 100
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.5 metric 100
Now add the new route for the VM subnet IPs to route to the dev server:
sudo ip route add 192.168.122.0/24 via 192.168.1.10 dev eth0
Verify the routes:
sudo ip route show
default via 192.168.1.1 dev eth0 proto dhcp metric 100
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.5 metric 100
192.168.122.0/24 via 192.168.1.10 dev eth0
Setup the dev server
The main issue on the dev server was the iptables wasn't setup to accept NEW connections to the subnet.
iptables -L FORWARD
...
ACCEPT all -- anywhere 192.168.122.0/24 ctstate RELATED,ESTABLISHED
Just RELATED and ESTABLISHED.
I needed to add NEW to that list:
sudo iptables -I FORWARD -m state -d 192.168.122.0/24 --state NEW,RELATED,ESTABLISHED -j ACCEPT
You may also need to modify the POSTROUTING to masquerade:
iptables -t nat -A POSTROUTING -j MASQUERADE -o eth0
Setup the VM
On the VM side all I really did was turn of firewalld:
systemctl stop firewalld
systemctl disable firewalld
No plans to open this to the greater Internet so for now this is acceptable to me. :)
Testing
Now that the three servers should be communicating properly, let's take a look. On each server, start a tcpdump:
sudo tcpdump -vv port 8000
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
In this case, look for all traffic on port 8000 (sending or receiving) as that is Kazoo's default API port.
Now, from the laptop, query the base API URL:
laptop$ curl -v http://192.168.122.21:8000
In the various TCP dumps you should see something like:
# Laptop
11:06:24.242471 IP (tos 0x0, ttl 64, id 21252, offset 0, flags [DF], proto TCP (6), length 60)
laptop.49560 > VM.8000: Flags [S], cksum 0xfcf3 (incorrect -> 0x04e3), seq 1995735410, win 29200, options [mss 1460,sackOK,TS val 3284285253 ecr 0,nop,wscale 7], length 0
11:06:24.243341 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
VM.8000 > laptop.49560: Flags [S.], cksum 0x73bf (correct), seq 2588808479, ack 1995735411, win 28960, options [mss 1460,sackOK,TS val 83877269 ecr 3284285253,nop,wscale 7], length 0
11:06:24.243385 IP (tos 0x0, ttl 64, id 21253, offset 0, flags [DF], proto TCP (6), length 52)
laptop.49560 > VM.8000: Flags [.], cksum 0xfceb (incorrect -> 0x12c6), seq 1, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 0
11:06:24.243452 IP (tos 0x0, ttl 64, id 21254, offset 0, flags [DF], proto TCP (6), length 135)
laptop.49560 > VM.8000: Flags [P.], cksum 0xfd3e (incorrect -> 0xe0bf), seq 1:84, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 83
11:06:24.244169 IP (tos 0x0, ttl 63, id 51382, offset 0, flags [DF], proto TCP (6), length 52)
VM.8000 > laptop.49560: Flags [.], cksum 0x1274 (correct), seq 1, ack 84, win 227, options [nop,nop,TS val 83877270 ecr 3284285254], length 0
11:06:24.445887 IP (tos 0x0, ttl 63, id 51383, offset 0, flags [DF], proto TCP (6), length 2798)
VM.8000 > laptop.49560: Flags [P.], cksum 0x07a6 (incorrect -> 0x7cb2), seq 1:2747, ack 84, win 227, options [nop,nop,TS val 83877472 ecr 3284285254], length 2746
11:06:24.445911 IP (tos 0x0, ttl 64, id 21255, offset 0, flags [DF], proto TCP (6), length 52)
laptop.49560 > VM.8000: Flags [.], cksum 0xfceb (incorrect -> 0x05f9), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285456 ecr 83877472], length 0
11:06:24.446377 IP (tos 0x0, ttl 64, id 21256, offset 0, flags [DF], proto TCP (6), length 52)
laptop.49560 > VM.8000: Flags [F.], cksum 0xfceb (incorrect -> 0x05f7), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285457 ecr 83877472], length 0
11:06:24.447064 IP (tos 0x0, ttl 63, id 51385, offset 0, flags [DF], proto TCP (6), length 52)
VM.8000 > laptop.49560: Flags [F.], cksum 0x0622 (correct), seq 2747, ack 85, win 227, options [nop,nop,TS val 83877473 ecr 3284285457], length 0
11:06:24.447090 IP (tos 0x0, ttl 64, id 21257, offset 0, flags [DF], proto TCP (6), length 52)
laptop.49560 > VM.8000: Flags [.], cksum 0xfceb (incorrect -> 0x05f4), seq 85, ack 2748, win 272, options [nop,nop,TS val 3284285458 ecr 83877473], length 0
# Dev machine (host)
laptop.49560 > 192.168.122.21.8000: Flags [S], cksum 0x04e3 (correct), seq 1995735410, win 29200, options [mss 1460,sackOK,TS val 3284285253 ecr 0,nop,wscale 7], length 0
18:06:24.240212 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
192.168.122.21.8000 > laptop.49560: Flags [S.], cksum 0xfcf3 (incorrect -> 0x73bf), seq 2588808479, ack 1995735411, win 28960, options [mss 1460,sackOK,TS val 83877269 ecr 3284285253,nop,wscale 7], length 0
18:06:24.240672 IP (tos 0x0, ttl 64, id 21253, offset 0, flags [DF], proto TCP (6), length 52)
laptop.49560 > 192.168.122.21.8000: Flags [.], cksum 0x12c6 (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 0
18:06:24.240702 IP (tos 0x0, ttl 64, id 21254, offset 0, flags [DF], proto TCP (6), length 135)
laptop.49560 > 192.168.122.21.8000: Flags [P.], cksum 0xe0bf (correct), seq 1:84, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 83
18:06:24.241046 IP (tos 0x0, ttl 63, id 51382, offset 0, flags [DF], proto TCP (6), length 52)
192.168.122.21.8000 > laptop.49560: Flags [.], cksum 0xfceb (incorrect -> 0x1274), seq 1, ack 84, win 227, options [nop,nop,TS val 83877270 ecr 3284285254], length 0
18:06:24.442439 IP (tos 0x0, ttl 63, id 51383, offset 0, flags [DF], proto TCP (6), length 2798)
192.168.122.21.8000 > laptop.49560: Flags [P.], cksum 0x07a6 (incorrect -> 0x7cb2), seq 1:2747, ack 84, win 227, options [nop,nop,TS val 83877472 ecr 3284285254], length 2746
18:06:24.443185 IP (tos 0x0, ttl 64, id 21255, offset 0, flags [DF], proto TCP (6), length 52)
laptop.49560 > 192.168.122.21.8000: Flags [.], cksum 0x05f9 (correct), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285456 ecr 83877472], length 0
18:06:24.443660 IP (tos 0x0, ttl 64, id 21256, offset 0, flags [DF], proto TCP (6), length 52)
laptop.49560 > 192.168.122.21.8000: Flags [F.], cksum 0x05f7 (correct), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285457 ecr 83877472], length 0
18:06:24.443935 IP (tos 0x0, ttl 63, id 51385, offset 0, flags [DF], proto TCP (6), length 52)
192.168.122.21.8000 > laptop.49560: Flags [F.], cksum 0xfceb (incorrect -> 0x0622), seq 2747, ack 85, win 227, options [nop,nop,TS val 83877473 ecr 3284285457], length 0
18:06:24.444356 IP (tos 0x0, ttl 64, id 21257, offset 0, flags [DF], proto TCP (6), length 52)
laptop.49560 > 192.168.122.21.8000: Flags [.], cksum 0x05f4 (correct), seq 85, ack 2748, win 272, options [nop,nop,TS val 3284285458 ecr 83877473], length 0
# VM
18:06:24.243655 IP (tos 0x0, ttl 63, id 21252, offset 0, flags [DF], proto TCP (6), length 60)
laptop.49560 > vm.8000: Flags [S], cksum 0x04e3 (correct), seq 1995735410, win 29200, options [mss 1460,sackOK,TS val 3284285253 ecr 0,nop,wscale 7], length 0
18:06:24.243713 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
vm.8000 > laptop.49560: Flags [S.], cksum 0xfcf3 (incorrect -> 0x73bf), seq 2588808479, ack 1995735411, win 28960, options [mss 1460,sackOK,TS val 83877269 ecr 3284285253,nop,wscale 7], length 0
18:06:24.244510 IP (tos 0x0, ttl 63, id 21253, offset 0, flags [DF], proto TCP (6), length 52)
laptop.49560 > vm.8000: Flags [.], cksum 0x12c6 (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 0
18:06:24.244543 IP (tos 0x0, ttl 63, id 21254, offset 0, flags [DF], proto TCP (6), length 135)
laptop.49560 > vm.8000: Flags [P.], cksum 0xe0bf (correct), seq 1:84, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 83
18:06:24.244555 IP (tos 0x0, ttl 64, id 51382, offset 0, flags [DF], proto TCP (6), length 52)
vm.8000 > laptop.49560: Flags [.], cksum 0xfceb (incorrect -> 0x1274), seq 1, ack 84, win 227, options [nop,nop,TS val 83877270 ecr 3284285254], length 0
18:06:24.445989 IP (tos 0x0, ttl 64, id 51383, offset 0, flags [DF], proto TCP (6), length 2798)
vm.8000 > laptop.49560: Flags [P.], cksum 0x07a6 (incorrect -> 0x7cb2), seq 1:2747, ack 84, win 227, options [nop,nop,TS val 83877472 ecr 3284285254], length 2746
18:06:24.446930 IP (tos 0x0, ttl 63, id 21255, offset 0, flags [DF], proto TCP (6), length 52)
laptop.49560 > vm.8000: Flags [.], cksum 0x05f9 (correct), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285456 ecr 83877472], length 0
18:06:24.447336 IP (tos 0x0, ttl 63, id 21256, offset 0, flags [DF], proto TCP (6), length 52)
laptop.49560 > vm.8000: Flags [F.], cksum 0x05f7 (correct), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285457 ecr 83877472], length 0
18:06:24.447502 IP (tos 0x0, ttl 64, id 51385, offset 0, flags [DF], proto TCP (6), length 52)
vm.8000 > laptop.49560: Flags [F.], cksum 0xfceb (incorrect -> 0x0622), seq 2747, ack 85, win 227, options [nop,nop,TS val 83877473 ecr 3284285457], length 0
18:06:24.448032 IP (tos 0x0, ttl 63, id 21257, offset 0, flags [DF], proto TCP (6), length 52)
laptop.49560 > vm.8000: Flags [.], cksum 0x05f4 (correct), seq 85, ack 2748, win 272, options [nop,nop,TS val 3284285458 ecr 83877473], length 0
Continuing
Instead of having to edit the routing table on every device on the network to forward packets to the dev machine, I setup my router to do it for me. Consult your router's manual for how to add static routes, then add something along the lines of:
| Network/HostIP | 192.168.122.0 | Subnet |
| Netmask | 255.255.255.0 | |
| Gateway | 192.168.1.10 | Dev Server IP |
| Metric | 0 | |
| Interface | LAN |
Naming may vary but hopefully it is clear enough to get you started.
Next Steps
- Setup a cluster of Kazoo VMs with multiple zones and see how it goes
- Orchestrate it all with…?