Running Kazoo VMs with KVM/QEMU

I've been running Kazoo in CentOS VMs to replicate production setups to solve support tickets. But getting my laptop talking to the VMs running on my dev server was non-obvious to me. I've finally got it working for an all-in-one Kazoo server and learned a little bit too.

The setup

I have three systems to connect:

Computer Network IP
My laptop (or any computer on my LAN really) 192.168.1.5
My dev server (beefy blade) 192.168.1.10
My Kazoo VM (using libvirt) 192.168.122.21

So the dev server running libvirt defines a default network of 192.168.122.0/24 and assigns IP addresses to the VMs from that pool.

By default, the dev server (the host) can talk to the VMs on the subnet and the VMs can talk to each other but computers on my LAN cannot talk to the VM subnet.

What to do?

Setup the laptop

The dev server knows how to route to the VMs so we need the laptop to send packets destined for the VM to the dev server. This is actually straight-forward to accomplish by adding a route to the kernel routing table:

First, check out the routing table: 

sudo ip route show
default via 192.168.1.1 dev eth0 proto dhcp metric 100
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.5 metric 100

Now add the new route for the VM subnet IPs to route to the dev server: 

sudo ip route add 192.168.122.0/24 via 192.168.1.10 dev eth0

Verify the routes: 

sudo ip route show
default via 192.168.1.1 dev eth0 proto dhcp metric 100
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.5 metric 100
192.168.122.0/24 via 192.168.1.10 dev eth0

Setup the dev server

The main issue on the dev server was the iptables wasn't setup to accept NEW connections to the subnet. 

iptables -L FORWARD
...
ACCEPT     all  --  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED

Just RELATED and ESTABLISHED.

I needed to add NEW to that list: 

sudo iptables -I FORWARD -m state -d 192.168.122.0/24 --state NEW,RELATED,ESTABLISHED -j ACCEPT

You may also need to modify the POSTROUTING to masquerade: 

iptables -t nat -A POSTROUTING -j MASQUERADE -o eth0

Setup the VM

On the VM side all I really did was turn of firewalld: 

systemctl stop firewalld
systemctl disable firewalld

No plans to open this to the greater Internet so for now this is acceptable to me. :)

Testing

Now that the three servers should be communicating properly, let's take a look. On each server, start a tcpdump: 

sudo tcpdump -vv port 8000
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

In this case, look for all traffic on port 8000 (sending or receiving) as that is Kazoo's default API port.

Now, from the laptop, query the base API URL: 

laptop$ curl -v http://192.168.122.21:8000

In the various TCP dumps you should see something like: 

# Laptop
11:06:24.242471 IP (tos 0x0, ttl 64, id 21252, offset 0, flags [DF], proto TCP (6), length 60)
    laptop.49560 > VM.8000: Flags [S], cksum 0xfcf3 (incorrect -> 0x04e3), seq 1995735410, win 29200, options [mss 1460,sackOK,TS val 3284285253 ecr 0,nop,wscale 7], length 0
11:06:24.243341 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    VM.8000 > laptop.49560: Flags [S.], cksum 0x73bf (correct), seq 2588808479, ack 1995735411, win 28960, options [mss 1460,sackOK,TS val 83877269 ecr 3284285253,nop,wscale 7], length 0
11:06:24.243385 IP (tos 0x0, ttl 64, id 21253, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > VM.8000: Flags [.], cksum 0xfceb (incorrect -> 0x12c6), seq 1, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 0
11:06:24.243452 IP (tos 0x0, ttl 64, id 21254, offset 0, flags [DF], proto TCP (6), length 135)
    laptop.49560 > VM.8000: Flags [P.], cksum 0xfd3e (incorrect -> 0xe0bf), seq 1:84, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 83
11:06:24.244169 IP (tos 0x0, ttl 63, id 51382, offset 0, flags [DF], proto TCP (6), length 52)
    VM.8000 > laptop.49560: Flags [.], cksum 0x1274 (correct), seq 1, ack 84, win 227, options [nop,nop,TS val 83877270 ecr 3284285254], length 0
11:06:24.445887 IP (tos 0x0, ttl 63, id 51383, offset 0, flags [DF], proto TCP (6), length 2798)
    VM.8000 > laptop.49560: Flags [P.], cksum 0x07a6 (incorrect -> 0x7cb2), seq 1:2747, ack 84, win 227, options [nop,nop,TS val 83877472 ecr 3284285254], length 2746
11:06:24.445911 IP (tos 0x0, ttl 64, id 21255, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > VM.8000: Flags [.], cksum 0xfceb (incorrect -> 0x05f9), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285456 ecr 83877472], length 0
11:06:24.446377 IP (tos 0x0, ttl 64, id 21256, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > VM.8000: Flags [F.], cksum 0xfceb (incorrect -> 0x05f7), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285457 ecr 83877472], length 0
11:06:24.447064 IP (tos 0x0, ttl 63, id 51385, offset 0, flags [DF], proto TCP (6), length 52)
    VM.8000 > laptop.49560: Flags [F.], cksum 0x0622 (correct), seq 2747, ack 85, win 227, options [nop,nop,TS val 83877473 ecr 3284285457], length 0
11:06:24.447090 IP (tos 0x0, ttl 64, id 21257, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > VM.8000: Flags [.], cksum 0xfceb (incorrect -> 0x05f4), seq 85, ack 2748, win 272, options [nop,nop,TS val 3284285458 ecr 83877473], length 0

# Dev machine (host)
    laptop.49560 > 192.168.122.21.8000: Flags [S], cksum 0x04e3 (correct), seq 1995735410, win 29200, options [mss 1460,sackOK,TS val 3284285253 ecr 0,nop,wscale 7], length 0
18:06:24.240212 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.122.21.8000 > laptop.49560: Flags [S.], cksum 0xfcf3 (incorrect -> 0x73bf), seq 2588808479, ack 1995735411, win 28960, options [mss 1460,sackOK,TS val 83877269 ecr 3284285253,nop,wscale 7], length 0
18:06:24.240672 IP (tos 0x0, ttl 64, id 21253, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > 192.168.122.21.8000: Flags [.], cksum 0x12c6 (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 0
18:06:24.240702 IP (tos 0x0, ttl 64, id 21254, offset 0, flags [DF], proto TCP (6), length 135)
    laptop.49560 > 192.168.122.21.8000: Flags [P.], cksum 0xe0bf (correct), seq 1:84, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 83
18:06:24.241046 IP (tos 0x0, ttl 63, id 51382, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.122.21.8000 > laptop.49560: Flags [.], cksum 0xfceb (incorrect -> 0x1274), seq 1, ack 84, win 227, options [nop,nop,TS val 83877270 ecr 3284285254], length 0
18:06:24.442439 IP (tos 0x0, ttl 63, id 51383, offset 0, flags [DF], proto TCP (6), length 2798)
    192.168.122.21.8000 > laptop.49560: Flags [P.], cksum 0x07a6 (incorrect -> 0x7cb2), seq 1:2747, ack 84, win 227, options [nop,nop,TS val 83877472 ecr 3284285254], length 2746
18:06:24.443185 IP (tos 0x0, ttl 64, id 21255, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > 192.168.122.21.8000: Flags [.], cksum 0x05f9 (correct), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285456 ecr 83877472], length 0
18:06:24.443660 IP (tos 0x0, ttl 64, id 21256, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > 192.168.122.21.8000: Flags [F.], cksum 0x05f7 (correct), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285457 ecr 83877472], length 0
18:06:24.443935 IP (tos 0x0, ttl 63, id 51385, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.122.21.8000 > laptop.49560: Flags [F.], cksum 0xfceb (incorrect -> 0x0622), seq 2747, ack 85, win 227, options [nop,nop,TS val 83877473 ecr 3284285457], length 0
18:06:24.444356 IP (tos 0x0, ttl 64, id 21257, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > 192.168.122.21.8000: Flags [.], cksum 0x05f4 (correct), seq 85, ack 2748, win 272, options [nop,nop,TS val 3284285458 ecr 83877473], length 0

# VM
18:06:24.243655 IP (tos 0x0, ttl 63, id 21252, offset 0, flags [DF], proto TCP (6), length 60)
    laptop.49560 > vm.8000: Flags [S], cksum 0x04e3 (correct), seq 1995735410, win 29200, options [mss 1460,sackOK,TS val 3284285253 ecr 0,nop,wscale 7], length 0
18:06:24.243713 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    vm.8000 > laptop.49560: Flags [S.], cksum 0xfcf3 (incorrect -> 0x73bf), seq 2588808479, ack 1995735411, win 28960, options [mss 1460,sackOK,TS val 83877269 ecr 3284285253,nop,wscale 7], length 0
18:06:24.244510 IP (tos 0x0, ttl 63, id 21253, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > vm.8000: Flags [.], cksum 0x12c6 (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 0
18:06:24.244543 IP (tos 0x0, ttl 63, id 21254, offset 0, flags [DF], proto TCP (6), length 135)
    laptop.49560 > vm.8000: Flags [P.], cksum 0xe0bf (correct), seq 1:84, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 83
18:06:24.244555 IP (tos 0x0, ttl 64, id 51382, offset 0, flags [DF], proto TCP (6), length 52)
    vm.8000 > laptop.49560: Flags [.], cksum 0xfceb (incorrect -> 0x1274), seq 1, ack 84, win 227, options [nop,nop,TS val 83877270 ecr 3284285254], length 0
18:06:24.445989 IP (tos 0x0, ttl 64, id 51383, offset 0, flags [DF], proto TCP (6), length 2798)
    vm.8000 > laptop.49560: Flags [P.], cksum 0x07a6 (incorrect -> 0x7cb2), seq 1:2747, ack 84, win 227, options [nop,nop,TS val 83877472 ecr 3284285254], length 2746
18:06:24.446930 IP (tos 0x0, ttl 63, id 21255, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > vm.8000: Flags [.], cksum 0x05f9 (correct), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285456 ecr 83877472], length 0
18:06:24.447336 IP (tos 0x0, ttl 63, id 21256, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > vm.8000: Flags [F.], cksum 0x05f7 (correct), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285457 ecr 83877472], length 0
18:06:24.447502 IP (tos 0x0, ttl 64, id 51385, offset 0, flags [DF], proto TCP (6), length 52)
    vm.8000 > laptop.49560: Flags [F.], cksum 0xfceb (incorrect -> 0x0622), seq 2747, ack 85, win 227, options [nop,nop,TS val 83877473 ecr 3284285457], length 0
18:06:24.448032 IP (tos 0x0, ttl 63, id 21257, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > vm.8000: Flags [.], cksum 0x05f4 (correct), seq 85, ack 2748, win 272, options [nop,nop,TS val 3284285458 ecr 83877473], length 0

Continuing

Instead of having to edit the routing table on every device on the network to forward packets to the dev machine, I setup my router to do it for me. Consult your router's manual for how to add static routes, then add something along the lines of:

Network/HostIP 192.168.122.0 Subnet
Netmask 255.255.255.0  
Gateway 192.168.1.10 Dev Server IP
Metric 0  
Interface LAN  

Naming may vary but hopefully it is clear enough to get you started.

Next Steps

  • Setup a cluster of Kazoo VMs with multiple zones and see how it goes
  • Orchestrate it all with…?

Spellcheck projects using aspell and Makefiles

Spellcheck using Aspell and Makefiles

With an international community surrounding Kazoo it is important to provide tools that help everyone stay on the same page with the development cycle.

One of the challenges, as a native speaker of English, is how easily our brains 'fix' misspelt words without us realizing it. Often enough, PRs are accepted where a key into a JSON structure is misspelled and goes unnoticed, time passes until someone realizes the error, and now JSON documents using that key must be supported to prevent previously working data from failing to work now (ask about Kazoo's history with 'wednesday' in the temporal routes callflow action!).

Fortunately there's a nice command line tool, GNU Aspell, that provides spell checking capabilities and custom dictionaries. Aspell does a great job suggesting alternatives when finding a potentially misspelt word, making it easy to find the correct word (or add a word to a personal dictionary).

Personal dictionary

A personal dictionary is helpful to maintain a word list separate and apart from the "default" English dictionary Aspell provides, especially concerning acronyms and jargon specific to the project. When you spellcheck files and find words that are correct but unknown to Aspell, you can add them so Aspell knows not to highlight them again. These words are added to the personal dictionary. Being a plaintext file, the personal dictionary is also easily included in source control so you can distribute it with your project.

Personal replacements

The replacement file can contain automatic replacements; when Aspell encounters a word in the replacement file, it will replace it automatically with the replacement word. Really handy for oft-misspelt words.

Installation

Aspell should be available through most GNU/Linux distros.

Makefile setup

You can easily add a make target to spellcheck a project. Create a splchk.mk file and add: 

.PHONY = splchk

DICT = .aspell.en.pws
REPL = .aspell.en.prepl

$(DICT):
	@$(file >$(DICT),personal_ws-1.1 en 0 utf-8)

$(REPL):
	@$(file >$(REPL),personal_repl-1.1 en 0 utf-8)

splchk: $(DICT) $(REPL) $(addsuffix .chk,$(basename $(shell find . -name "*.md" )))

%.chk: %.md
	aspell --home-dir=. --personal=$(DICT) --repl=$(REPL) --lang=en -x check $<

Replace the $(shell find . -name "*.md") with whatever you want to use to find files to spellcheck.

When you run make splchk you will get the opportunity to check the spelling for each file found (if the file doesn't have any issues you won't deal with it). If you find a word that is correct but Aspell highlights it for repair, you can 'a'dd it which will include it in your personal dictionary.

Comparing Kazoo and other CPaaS solutions

Introduction

First, what is a CPaaS? Communication Platform as a Service is the expanded acronym but it boils down to APIs and programmatic control of the phone system.

The power of CPaaS is giving you the control of each call in a way that makes sense for your (or your business') needs. It lets you access information outside of the phone system (such as CRMs and other databases) to make decisions about how to route a particular call.

Another feature of CPaaS is the ability to get events out of the phone system - via webhooks or over a websocket typically. This gives you the ability to track a call's progress through the callflow, updating information in CRMs and other databases and lets you build robust applications on top of the phone network.

CPaaS providers often offer API access to manipulate running calls (such as programmatically hanging up a call, initiating a transfer, etc), fetching CDRs (call detail records - typically used when billing for calls), initiating new calls, and more.

A newer feature, a requirement of CPaaS providers these days, is having WebRTC-based phones (or a phone in your browser), minimizing the software needed to be installed and allowing companies to provide a branded phone in their web apps.

Most CPaaS providers allow you to process phone calls without being a telecom junkie by providing a simplified set of actions that may be performed on a given call. No longer do you need to know what the PSTN is or how SIP or SDP work - you get to build on top of an abstracted phone system and focus more on your business needs.

The competitive landscape

When you view the competitive landscape in telecom, it is easy to see that the big players (traditional telcos and UCaaS providers like Avaya, Broadsoft, and others) are recognizing the value of adding a CPaaS-type offering by purchasing CPaaS providers like Zang and Nexmo (via Cisco acquisition). Twilio stands apart as an independent CPaaS provider that IPO'd in 2016.

The market has shifted and CPaaS + UCaaS is the norm going forward; UCaaS companies are racing to catch up while CPaaS companies are trying to expand into more traditional telephony environments.

Technical Evaluation

Let's dive into a bit of the nitty-gritty and see how a handful of providers compare.

Real-time Call Control

Most CPaaS offer a small list of actions/verbs that can be returned from your program. The basics are covered by most providers.

Connecting callers to endpoints

When a caller calls in, typically you want them to talk to someone (or several someones). Below is a quick chart on the various endpoints supported by CPaaS providers for connecting the caller:

Action Kazoo Twilio Plivo Nexmo Tropo
Registered Device ✔ X X X X
SIP Endpoint ✔ ✔ ✔ ✔ ✔
Conference ✔ ✔ ✔ X ✔
DID(s) ✔ ✔ ✔ ✔ ✔
Call Queue - ✔ X X X
User (find-me/follow-me) ✔ X X X X
Group ✔ X X X X
Page group ✔ X X X X

Some notes:

  • Kazoo is the only platform that allows devices to register directly to the platform to receive calls. Registration is the phone's way of saying "Send calls for me to this IP/port please". Devices can then move around (say a softphone on a laptop) but still be able to receive calls easily. The SIP endpoint, in contrast, is typically in a fixed location and is harder to move once setup. An existing PBX is the typical destination for SIP endpoints.
  • Twilio is the only provider that offers a "Queue" option in the public docs. Kazoo has a community-supported queuing applications called ACDc as well as a commercial offering called Qubicle, which can be leveraged for call queues. You can also simulate queues with conferences.
  • The 'User' endpoint is the ability to associate multiple devices with a user and ring all relevant devices when the user is dialed. Most providers allow you to specify multiple endpoints to dial but that association must be tracked on your side. With Kazoo, you can create a User and use that in your callflow instead.

  • The 'Group' endpoint is similar to the User in that you can add devices and users to a group and all associated devices will be rung.

    The biggest differentiate among the providers in this case comes down to the need for registration, insight into why a call failed to connect (some providers are quite opaque about the reason for failure), and additional features offered on each endpoint type (like setting custom ringtones).

Caller interactions

Most providers allow you to interact with the caller by playing audio files, generating dynamic text to read to the caller (TTS - text to speech), and collecting touch tones (DTMF) from the caller.

Action Kazoo Twilio Plivo Nexmo Tropo
Collect DTMF ✔ ✔ ✔ ✔ ✔
Play Audio (MP3 or WAV) ✔ ✔ ✔ ✔ ✔
TTS ✔ ✔ ✔ ✔ ✔

Other basics offered

Action Kazoo Twilio Plivo Nexmo Tropo
Hangup a call ✔ ✔ ✔ X ✔
Pause execution ✔ ✔ ✔ X ✔
Record the caller ✔ ✔ ✔ ✔ ✔
Redirect to different URL ✔ ✔ ✔ X ✔
Reject a call ✔ ✔ ✔ X ✔

Kazoo Differentiators

While CPaaS allows you to interact with phone calls at a higher level than processing SIP packets, the features above are fairly low-level as far as business concerns go. They make for great demos and getting a prototype out the door quickly.

Invariably, once you gain traction, feature requests come in and more functionality is needed. The question to ask is, do you have to do the work to implement the functionality or is it provided already?

For instance, say your prototype connects the caller, based on some fancy criteria, to an agent/sales person/fortune teller. You gain traction and now your callees (the agents/sales people/fortune tellers) would love to have a voicemail service in case they can't answer the phone (since most are using a personal mobile phone and don't want the caller to hear their personal voicemail).

Do you want to build a voicemail box?

With Kazoo, you can configure a voicemail box via API and add it to the callflow with minimal effort. With the other providers, you're looking at having to get the IVR right, get the prompts recorded, test all the paths through a voicemail box, figure out storage of the voicemail recordings, build an API for providing the voicemail messages, build voicemail->email and attach the recording, archive/delete old/deleted messages, and more. This is not quick to build and, frankly, most people don't consider voicemail a feature worth paying much, if anything, for; is it a good use of your time?

Kazoo offers a number of callflow actions, like voicemail, that allow you to add higher level, business-focused, actions to your call processing capabilities. A short list of highlights:

  • Call Parking
  • Directed / Group Call Pickup
  • Eavesdrop
  • Faxing
  • Hotdesking
  • IVRs
  • Manual presence updates
  • Time-of-day routing
  • Voicemail

Each of these features represents days to weeks of work to implement yourself (if you even can given the primitives offered by the other providers) and each has its own set of edge cases and intricacies that come with it. Instead, why not use the pre-built, heavily-tested callflow actions provided by Kazoo?

API control

The single largest difference between Kazoo's REST API and other providers is the sheer breadth of APIs exposed by Kazoo. Remember, Kazoo is both a CPaaS and UCaaS platform; on top of that, everything is API driven so you while you might start with the CPaaS-specific features, you can easily grow into UCaaS type services (like office PBX features).

Call Control APIs

Each provider allows you to list and control active calls. What controls you have vary; some allow you to redirect the executing callflow to your server for new callflow instructions; others expose the actions you can overlay on the call via the API and that's it.

Similarly, conferences and conference participants can be managed through most providers' APIs.

All in all, there isn't much that differentiates the basics among providers.

Kazoo API Differentiators

  • Multi-tenancy

    Kazoo is designed to be multi-tenant; you can have as deep or as broad of an account tree as you like. Most providers, like Twilio, Nexmo, and Tropo, don't provide multi-tenancy at all (Plivo does appear to though). As a service integrator, you will have to manage that hierarchy on your side. This means you have to build infrastructure to manage billing, resource usage, configuration, etc.

  • Carrier management

    Kazoo allows you to define your own upstream carriers, so even if you're using a hosted instance of Kazoo (versus running it yourself), you should be able to add your own carriers to you account and use them instead of the system's carriers. You can even setup hybrid situations where you route certain destinations to your carriers and use the system's as the default carrier group.

    None of the other CPaaS providers offer BYOC (Bring Your Own Carrier). This means that as you grow your voice traffic, you will not have levers to pull to manage costs. With BYOC, when it makes sense to, you can find your own deals on connectivity and seamlessly switch to your own carriers to save costs.

  • Finer-grained features

    Each API entity (such as devices or conferences) in Kazoo offers significantly more functionality compared to the other providers' equivalent entities. Of course there are some similarities but, for instance looking at a "device" in Kazoo, you can optionally define:

    • Call forwarding settings, including for failover (or just straight call-forwarding)
    • Call recording (both inbound and outbound) for the device
    • Call restrictions based on number classifiers (can call US DIDs, can't call international, etc)
    • Caller ID settings
    • Dialplan settings (allow users to dial shorter numbers and "fix" them - for instance the US used to allow 7-digit dialing and the area code was assumed based on the phone's location).
    • Do Not Disturb settings
    • Codecs available (both audio and video)
    • Custom music on hold
    • Custom presence ID (instead of the SIP username)

    There are other Kazoo-specific features on devices as well. Since users can own devices, you can set similar properties on the user and have them propagate to all devices (and define properties on the devices to override user settings).

  • Class-4 and Class-5 Switching

    CPaaS providers are typically concerned with a handful of phone system features - dialing endpoints, conferences, call recording, call queue, and phone number management most commonly.

    Kazoo, being a hybrid class-4 and class-5 switch, offers so much more:

    PBX functionality includes configuring users, devices, and callflows, blacklists, directories, faxes and faxboxes, IVRs, voicemail, and more.

    Configure connectivity features like carriers (resources in Kazoo parlance), ratedecks, PBX trunking.

    Configure reseller features like branding email notification templates and white-labeling Kazoo services.

Questions to ask a CPaaS provider

These are some good questions to ask your CPaaS provider(s) and to ask yourself how important the answers are to your needs:

  • Where are the voice servers located, geographically?
  • Where are the API servers located? Hint, some providers may have voice spread out around the world but API servers only in geographic location.
  • Where are the servers located (the data centers) relative to your users? To your carriers?
  • How is the networking built? Some hosted infrastructure providers, like AWS/GCE aren't well suited to VoIP traffic - are CPaaS providers piggybacking off that infrastructure or are they building out their own racks?
  • Is the CPaaS provider reselling another provider's service with better lipstick?
  • Can you extend the platform?

Conclusion

The market has spoken and they want UCaaS+CPaaS offered under one roof. The big UCaaS players have bought their CPaaS plays and are busy integrating them into the fold with varying success. Meanwhile Twilio remains an outlier and looks to be doubling down on being the CPaaS for the masses.

Kazoo offers you CPaaS + UCaaS from day one in a cohesive system. You can use as many or as few APIs and callflow actions as you need to accomplish your goals, while knowing that as you grow and need new functionality, Kazoo is there ready to provide it. On the off chance Kazoo isn't able to provide a feature, you can rest easy knowing the open-source nature of the project gives you the option to build it yourself and keep it private, build it and release it upstream for all to benefit, or support the project by contracting with 2600Hz to build the functionality for you.

Ready to kick the tires?

Site improvements - Headers and Certs edition

Analyzing jamesaimonetti.com

After reading Julia Evans' post on the HSTS header, I felt like I should add it to this site. This lead me to Mozilla's Observatory that will analyze your site and grade it. Like Julia, I received a fun F - the biggest dings were missing headers and an invalid certificate chain. Let's look at how I improved my site's scoring.

The initial site

My blog is a Nikola-powered site. Since the pages are fronted by Apache (for the time being) most of the changes need to be applied there (as Apache controls the response headers). Let's add some headers!

Setting the HTTP Strict Transport Security (HSTS)

This post talks more about preload and other considerations.

On the https (port 443) virtual host, its a simple one-line addition: 

Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"

The max_age is in seconds (63072000 is 2 years).

Setting the Content Security Policy (CSP) header

Mozilla has great help text about the header here. 

Header always set Content-Security-Policy "default-src 'none'; font-src 'self'; img-src 'self'; object-src 'none'; script-src 'self'; style-src 'self'"

I've opted to keep everything in-house-only. If I start hosting images or fonts on other sites I can add them in as exceptions.

Certificate issues when redirecting

The redirection error seemed inappropriate as I use a permanent redirect from http to https: 

<VirtualHost *:80>
	ServerName jamesaimonetti.com
	Redirect permanent / https://jamesaimonetti.com
</VirtualHost>

So I went digging a bit more.

One of the ideas was to support HTTP public key pinning (HPKP). A cursory view of Let's Encrypt discussion suggested that this might not be worth the trouble for a relatively low-value site like mine.

As it turns out, the addition of the HSTS header will likely improve this metric! Once we reload the site and rescan it with Observatory, we'll find out.

Setting the X-Content-Type-Options header

Pretty straightforward explanation and easily added: 

Header always set X-Content-Type-Options "nosniff"

Setting the X-Frame-Options header

I don't want my site in an iframe (see here) so in it goes: 

Header always set X-Frame-Options "DENY"

This also adds a setting to the CSP header: 

Header always set Content-Security-Policy "frame-ancestors 'none'"

Setting the X-XSS-Protection header

Seems mostly for legacy browsers that don't support CSP (see here). 

Header always set X-XSS-Protection "1; mode=block"

Testing the setup

Initial Errors

Once the headers were set, I reloaded the site in Apache and reloaded it in my browser. I verified the response headers were included but now I had a bunch of console errors to handle: 

Content Security Policy: The page’s settings blocked the loading of a resource at self (“script-src https://jamesaimonetti.com”). Source: onfocusin attribute on DIV element.  jamesaimonetti.com
Content Security Policy: The page’s settings blocked the loading of a resource at self (“script-src https://jamesaimonetti.com”). Source: $('a.image-reference:not(.islink) img:no....  jamesaimonetti.com:637
Content Security Policy: The page’s settings blocked the loading of a resource at self (“script-src https://jamesaimonetti.com”). Source:
    moment.locale("en");
    fancydates....  jamesaimonetti.com:637
Content Security Policy: The page’s settings blocked the loading of a resource at self (“script-src https://jamesaimonetti.com”). Source:
  (function(i,s,o,g,r,a,m){i['GoogleAna....  jamesaimonetti.com:640

Google Analytics

Looking at the CSP FAQ about adding the Google Analytics, it looks like I need to whitelist Google in the img-src portion of my CSP. 

Header always set Content-Security-Policy "img-src 'self' www.google-analytics.com"

This in and of itself did not address the issue as the inline javascript used to do the analytics isn't executed. The recommended way is to load the analytics javascript in its own file (vs inline in the html page). Currently in my conf.py I've added a BODY_END snippet with the google analytics setup. I've moved that snippet to /files/assets/js/ga.js and included it in my EXTRA_HEAD_DATA config: 

EXTRA_HEAD_DATA = """
<script src="/assets/js/ga.js" type="text/javascript"/>
"""

Now I need to add google-analytics to my CSP script-src as well: 

Header always set Content-Security-Policy "script-src 'self' www.google-analytics.com"

Improvement

With these changes above, scorecard improved to a solid B!

Still getting dinged with the HSTS header and certificate chain issues.

Turns out I had only included 2 of the three SSL-related directives in Apache. The third line below was added: 

SSLCertificateKeyFile "/etc/letsencrypt/live/jamesaimonetti.com/privkey.pem"
SSLCertificateFile "/etc/letsencrypt/live/jamesaimonetti.com/cert.pem"
SSLCertificateChainFile "/etc/letsencrypt/live/jamesaimonetti.com/chain.pem"

chain.pem includes the intermediate certificates needed to complete the chain. See SSL Labs' SSLTest to find out more about your site.

Forward secrecy

For funzies, adding forward secrecy to Apache to bump those scores up (yay gamification!).

Disable RC4

RC4 is broken; don't offer it!

I added a ssl.conf in conf.d with this: 

SSLProtocol all -SSLv2 -SSLv3
SSLHonorCipherOrder on
SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS"