Stateless Property-based testing with Erlang and PropEr

In preparation for my talk at CodeBEAM, I thought I'd give an introductory example of some stateless property-based tests as an introduction to property-based testing.

Fine reading and viewing options

First things first, get yourself a copy of the Property-Based Testing book.

Second, there are a number of blog posts and videos online that talk about property-based testing that are far better resources than I would write. In no particular order:

  1. An introduction to property-based testing (using F#) - I like the approach of an adversarial relationship between the code writer and the test writer. How can the test writer ensure the code writer is properly implementing a function without cheating on the implementation just to get tests to pass?
  2. Code checking automation - John Hughes breaks down testing some SMS processing code using Quickcheck and finding an inconsistency in the spec.
  3. Don't write tests! - John Hughes talks about why testing is hard and how property-based testing can make it easier (and maybe more fun!).
  4. Testing web services with QuickCheck - Using QuickCheck (and stateful property testing) to test web services.
  5. The Hitchhiker's Guide to the Unexpected - Fred Hebert talks about using property testing to test his supervisor hierarchy and how it reacts to failure (see here specifically).

Basically any video where John Hughes or Thomas Arts talk QuickCheck or property-based testing is worth a watch!

PropEr

We use the open source, QuickCheck-inspired, testing tool PropEr to develop our tests. The homepage for the project has links to tutorials and other helpful information specifically related to using PropEr.

The Code

Briefly, we want to test the higher-level properties of a function instead of explicitly enumerating the inputs and outputs we expect.

Let us consider the function camel_to_snake/1 which takes a Camel case string (such as ThisIsCamelCase) and converts it to Snake case (such as this_is_snake_case).

camel_to_snake(<<First, Bin/binary>>) ->
    iolist_to_binary([to_lower_char(First)
		     ,[maybe_camel_to_snake(Char) || <<Char>> <= Bin]
		     ]).

maybe_camel_to_snake(Char) ->
    case is_upper_char(Char) of
	'false' -> Char;
	'true' ->
	    [$_, to_lower_char(Char)]
    end.

is_upper_char(Char) ->
    Char >= $A andalso Char =< $Z.

to_lower_char(Char) when is_integer(Char), $A =< Char, Char =< $Z -> Char + 32;
to_lower_char(Char) -> Char.

Pretty simple implementation - if the character is uppercase, convert it to an underscore and the lowercase version of the character.

Now, if we were doing unit tests alone, we might write:

camel_to_snake_test_() ->
    Tests = [{<<"Test">>, <<"test">>}
	    ,{<<"TestKey">>, <<"test_key">>}
	    ,{<<"testKey">>, <<"test_key">>}
	    ,{<<"TestKeySetting">>, <<"test_key_setting">>}
	    ,{<<"testKeySetting">>, <<"test_key_setting">>}
	    ,{<<"TEST">>, <<"t_e_s_t">>}
	    ],
    [?_assertEqual(To, camel_to_snake(From))
     || {From, To} <- Tests
    ].

And for a relatively trivial function like this, we're reasonably confident those tests are adequate:

make eunit
...
  module camel_tests'
    camel_tests:40: camel_to_snake_test_...ok
    camel_tests:40: camel_to_snake_test_...ok
    camel_tests:40: camel_to_snake_test_...ok
    camel_tests:40: camel_to_snake_test_...ok
    camel_tests:40: camel_to_snake_test_...ok
    camel_tests:40: camel_to_snake_test_...ok
    [done in 0.018 s]
...

Let's see if we can get some more coverage of the input space with property tests!

Properties to test

We have a pretty basic property here - take a string and convert uppercase characters to _{lowercase}.

So how can we generate inputs and outputs that will help test the conversion?

Well, we can generate the CamelCase string character by character and build the snakecase version as we go.

First, the high level property test:

prop_morph() ->
    ?FORALL({Camel, Snake}
	   ,camel_and_snake()
	   ,?WHENFAIL(io:format("~nfailed to morph '~s' to '~s'~n", [Camel, Snake])
		     ,Snake =:= camel_to_snake(Camel)
		     )
	   ).

So `camelandsnake/0` is a generator that will return a 2-tuple with the built strings, will compare the call to `cameltosnake/1` against the generated Snake, and if the property fails (evals to false), we output the failing pair for analysis.

camel_and_snake() ->
    camel_and_snake(30). % we don't want overly huge strings so cap them to 30 characters for now

camel_and_snake(Length) ->
    ?LET(CamelAndSnake
	,camel_and_snake(Length, [])
	,begin
	     %% we create a list of [{CamelChar, SnakeChar}]; unzipping results in {CamelChars, SnakeChars} as iolist() data
	     {Camel, Snake} = lists:unzip(lists:reverse(CamelAndSnake)),
	     {list_to_binary(Camel), list_to_binary(Snake)}
	 end
	).

%% Create a list of [{CamelChar, SnakeChar}]
camel_and_snake(0, CamelAndSnake) ->
    CamelAndSnake;
camel_and_snake(Length, CamelAndSnake) ->
    CandS = oneof([upper_char()
		  ,lower_char()
		  ]),

    camel_and_snake(Length-1
		   ,[CandS | CamelAndSnake]
		   ).

%% Lower chars are easy - whatever the camel gets the snake gets too
lower_char() ->
    ?LET(Lower
	 ,choose($a,$z)
	 ,{Lower, Lower}
	).

%% Uppercase just requires a little math
upper_char() ->
    ?LET(Upper
	,choose($A,$Z)
	,{Upper, [$_, Upper+32]}
	).

Running it the first time we see:

(prop_morph)......!
Failed: After 4 test(s).
{<<70,74,101,101,81,90,81,72,112,117,84,74,106,100,100,67,90,98,98,122,80,107,81,77,99,79,113,82,84,99>>,<<95,102,95,106,101,101,95,113,95,122,95,113,95,104,112,117,95,116,95,106,106,100,100,95,99,95,122,98,98,122,95,112,107,95,113,95,109,99,95,111,113,95,114,95,116,99>>}

failed to morph 'FJeeQZQHpuTJjddCZbbzPkQMcOqRTc' to '_f_jee_q_z_q_hpu_t_jjdd_c_zbbz_pk_q_mc_oq_r_tc'

Shrinking ............................................(44 time(s))
{<<65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65>>,<<95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97>>}

failed to morph 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' to '_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a'

Ah, right, if the first character is uppercase, we don't want to introduce the underscore. Let's fix the generator to account for the first character situation:

%% First, we need to choose a different start
camel_and_snake(0, CamelAndSnake) ->
    CamelAndSnake;
camel_and_snake(Length, []) ->
    CandS = oneof([first_upper_char()
		  ,lower_char()
		  ]),
    camel_and_snake(Length-1, [CandS]);
camel_and_snake(Length, CamelAndSnake) ->
    CandS = oneof([upper_char()
		  ,lower_char()
		  ]),

    camel_and_snake(Length-1
		   ,[CandS | CamelAndSnake]
		   ).

%% We define first_upper_char to not include the underscore for the snake version
upper_char() ->
    ?LET(Upper
	,choose($A,$Z)
	,{Upper, [Upper+32]}
	).

Running this through PropEr gives us:

proper_test_ (prop_morph) .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
OK: Passed 500 test(s).

Ship it!

Not quite ready

One of the joys of KAZOO is getting to learn about how telecom works in the wider world. This forces Unicode front and center when dealing with user input (on the plus side, we can use emojis for phone extensions!).

Looking at our code, we see we've naively handled just the Latin-based alphabet. For inspiration, we open up string.erl to see how it handles upper/lowercase conversions:

%% ISO/IEC 8859-1 (latin1) letters are converted, others are ignored
%%

to_lower_char(C) when is_integer(C), $A =< C, C =< $Z ->
    C + 32;
to_lower_char(C) when is_integer(C), 16#C0 =< C, C =< 16#D6 ->
    C + 32;
to_lower_char(C) when is_integer(C), 16#D8 =< C, C =< 16#DE ->
    C + 32;
to_lower_char(C) ->
    C.

to_upper_char(C) when is_integer(C), $a =< C, C =< $z ->
    C - 32;
to_upper_char(C) when is_integer(C), 16#E0 =< C, C =< 16#F6 ->
    C - 32;
to_upper_char(C) when is_integer(C), 16#F8 =< C, C =< 16#FE ->
    C - 32;
to_upper_char(C) ->
    C.

Let's adjust our generators first to see failing cases:

lower_char() ->
    ?LET(Lower
	,union([choose($a,$z)
	       ,choose(16#E0,16#F6)
	       ,choose(16#F8,16#FE)
	       ])
	,{Lower, [Lower]}
	).

first_upper_char() ->
    ?LET(Upper
	,union([choose($A,$Z)
	       ,choose(16#C0,16#D6)
	       ,choose(16#D8,16#DE)
	       ])
	,{Upper, [Upper+32]}
	).

upper_char() ->
    ?LET(Upper
	,union([choose($A,$Z)
	       ,choose(16#C0,16#D6)
	       ,choose(16#D8,16#DE)
	       ])
	,{Upper, [$_, Upper+32]}
	).

Running this, we get some nice failures:

 proper_test_ (prop_morph)...!
Failed: After 1 test(s).
{<<222,240,240,198,253,220,75,212,233,76,248,110,77,83,229,99,195,88,216,250,246,67,227,237,103,240,217,253,220,221>>,<<254,240,240,95,230,253,95,252,95,107,95,244,233,95,108,248,110,95,109,95,115,229,99,95,227,95,120,95,248,250,246,95,99,227,237,103,240,95,249,253,95,252,95,253>>}
failed to morph 'ÞððÆýÜKÔéLønMSåcÃXØúöCãígðÙýÜÝ' to 'þðð_æý_ü_k_ôé_løn_m_såc_ã_x_øúö_cãígð_ùý_ü_ý'

Shrinking ..............................................................(62 time(s))
{<<192,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65>>,<<224,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97,95,97>>}
failed to morph 'ÀAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' to 'à_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a_a'

You can also see that, because we wanted 30-character strings, PropEr generates a shrunk version that is still 30 characters long. Let's inform PropEr that we want to constrain the length of the strings but be a little more flexible so PropEr can generate better failing test cases:

prop_morph() ->
    ?FORALL({Camel, Snake}
	   ,resize(20, camel_and_snake())
	   ,camel_and_snake()
	   ,?WHENFAIL(io:format('user', "~nfailed to morph '~s' to '~s'~n", [Camel, Snake])
		     ,Snake =:= camel_to_snake(Camel)
		     )
	   ).

camel_and_snake() ->
    ?SIZED(Length, camel_and_snake(Length)).

You can read more about resize/2 and the ?SIZED macro but basically they let PropEr know to constrain the size a bit but with more flexibility than a static length.

Running the tests now:

proper_test_ (prop_morph)...!
Failed: After 1 test(s).
{<<220>>,<<252>>}

failed to morph 'Ü' to 'ü'

Shrinking ..(2 time(s))
{<<192>>,<<224>>}

failed to morph 'À' to 'à'

Much easier!

Let's adjust our implementation to account for these upper/lower bounds:

is_upper_char(Char) ->
    (Char >= $A andalso Char =< $Z)
	orelse (16#C0 =< Char andalso Char =< 16#D6)
	orelse (16#D8 =< Char andalso Char =< 16#DE).

to_lower_char(Char) when is_integer(Char), $A =< Char, Char =< $Z -> Char + 32;
to_lower_char(Char) when is_integer(Char), 16#C0 =< Char, Char =< 16#D6 ->
    Char + 32;
to_lower_char(Char) when is_integer(Char), 16#D8 =< Char, Char =< 16#DE ->
    Char + 32;
to_lower_char(Char) -> Char.

And the tests now pass nicely:

 proper_test_ (prop_morph) .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
OK: Passed 500 test(s).

Honing the skill

Thinking in properties is not necessarily intuitive at first. There are many choices for what properties to choose to test, so pick and choose which apply to the particulars of the code you're testing. Another helpful thing is to keep the properties as simple as possible at first. As you build confidence in the generators and property tests, you can layer on more complex properties to ensure the tests are encapsulating the properties of your code.

As you progress down the road of property testing, you will hopefully find that you force yourself to think more deeply about your code and hopefully head off issues before the tests locate them. A great exercise is to pair with a teammate, have one write the implementation and one write the property tests, and compete to see if the tester can find bugs in the implementation.

Running Kazoo VMs with KVM/QEMU

I've been running Kazoo in CentOS VMs to replicate production setups to solve support tickets. But getting my laptop talking to the VMs running on my dev server was non-obvious to me. I've finally got it working for an all-in-one Kazoo server and learned a little bit too.

The setup

I have three systems to connect:

Computer Network IP
My laptop (or any computer on my LAN really) 192.168.1.5
My dev server (beefy blade) 192.168.1.10
My Kazoo VM (using libvirt) 192.168.122.21

So the dev server running libvirt defines a default network of 192.168.122.0/24 and assigns IP addresses to the VMs from that pool.

By default, the dev server (the host) can talk to the VMs on the subnet and the VMs can talk to each other but computers on my LAN cannot talk to the VM subnet.

What to do?

Setup the laptop

The dev server knows how to route to the VMs so we need the laptop to send packets destined for the VM to the dev server. This is actually straight-forward to accomplish by adding a route to the kernel routing table:

First, check out the routing table:

sudo ip route show
default via 192.168.1.1 dev eth0 proto dhcp metric 100
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.5 metric 100

Now add the new route for the VM subnet IPs to route to the dev server:

sudo ip route add 192.168.122.0/24 via 192.168.1.10 dev eth0

Verify the routes:

sudo ip route show
default via 192.168.1.1 dev eth0 proto dhcp metric 100
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.5 metric 100
192.168.122.0/24 via 192.168.1.10 dev eth0

Setup the dev server

The main issue on the dev server was the iptables wasn't setup to accept NEW connections to the subnet.

iptables -L FORWARD
...
ACCEPT     all  --  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED

Just RELATED and ESTABLISHED.

I needed to add NEW to that list:

sudo iptables -I FORWARD -m state -d 192.168.122.0/24 --state NEW,RELATED,ESTABLISHED -j ACCEPT

You may also need to modify the POSTROUTING to masquerade:

iptables -t nat -A POSTROUTING -j MASQUERADE -o eth0

Setup the VM

On the VM side all I really did was turn of firewalld:

systemctl stop firewalld
systemctl disable firewalld

No plans to open this to the greater Internet so for now this is acceptable to me. :)

Testing

Now that the three servers should be communicating properly, let's take a look. On each server, start a tcpdump:

sudo tcpdump -vv port 8000
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

In this case, look for all traffic on port 8000 (sending or receiving) as that is Kazoo's default API port.

Now, from the laptop, query the base API URL:

laptop$ curl -v http://192.168.122.21:8000

In the various TCP dumps you should see something like:

# Laptop
11:06:24.242471 IP (tos 0x0, ttl 64, id 21252, offset 0, flags [DF], proto TCP (6), length 60)
    laptop.49560 > VM.8000: Flags [S], cksum 0xfcf3 (incorrect -> 0x04e3), seq 1995735410, win 29200, options [mss 1460,sackOK,TS val 3284285253 ecr 0,nop,wscale 7], length 0
11:06:24.243341 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    VM.8000 > laptop.49560: Flags [S.], cksum 0x73bf (correct), seq 2588808479, ack 1995735411, win 28960, options [mss 1460,sackOK,TS val 83877269 ecr 3284285253,nop,wscale 7], length 0
11:06:24.243385 IP (tos 0x0, ttl 64, id 21253, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > VM.8000: Flags [.], cksum 0xfceb (incorrect -> 0x12c6), seq 1, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 0
11:06:24.243452 IP (tos 0x0, ttl 64, id 21254, offset 0, flags [DF], proto TCP (6), length 135)
    laptop.49560 > VM.8000: Flags [P.], cksum 0xfd3e (incorrect -> 0xe0bf), seq 1:84, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 83
11:06:24.244169 IP (tos 0x0, ttl 63, id 51382, offset 0, flags [DF], proto TCP (6), length 52)
    VM.8000 > laptop.49560: Flags [.], cksum 0x1274 (correct), seq 1, ack 84, win 227, options [nop,nop,TS val 83877270 ecr 3284285254], length 0
11:06:24.445887 IP (tos 0x0, ttl 63, id 51383, offset 0, flags [DF], proto TCP (6), length 2798)
    VM.8000 > laptop.49560: Flags [P.], cksum 0x07a6 (incorrect -> 0x7cb2), seq 1:2747, ack 84, win 227, options [nop,nop,TS val 83877472 ecr 3284285254], length 2746
11:06:24.445911 IP (tos 0x0, ttl 64, id 21255, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > VM.8000: Flags [.], cksum 0xfceb (incorrect -> 0x05f9), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285456 ecr 83877472], length 0
11:06:24.446377 IP (tos 0x0, ttl 64, id 21256, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > VM.8000: Flags [F.], cksum 0xfceb (incorrect -> 0x05f7), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285457 ecr 83877472], length 0
11:06:24.447064 IP (tos 0x0, ttl 63, id 51385, offset 0, flags [DF], proto TCP (6), length 52)
    VM.8000 > laptop.49560: Flags [F.], cksum 0x0622 (correct), seq 2747, ack 85, win 227, options [nop,nop,TS val 83877473 ecr 3284285457], length 0
11:06:24.447090 IP (tos 0x0, ttl 64, id 21257, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > VM.8000: Flags [.], cksum 0xfceb (incorrect -> 0x05f4), seq 85, ack 2748, win 272, options [nop,nop,TS val 3284285458 ecr 83877473], length 0

# Dev machine (host)
    laptop.49560 > 192.168.122.21.8000: Flags [S], cksum 0x04e3 (correct), seq 1995735410, win 29200, options [mss 1460,sackOK,TS val 3284285253 ecr 0,nop,wscale 7], length 0
18:06:24.240212 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.122.21.8000 > laptop.49560: Flags [S.], cksum 0xfcf3 (incorrect -> 0x73bf), seq 2588808479, ack 1995735411, win 28960, options [mss 1460,sackOK,TS val 83877269 ecr 3284285253,nop,wscale 7], length 0
18:06:24.240672 IP (tos 0x0, ttl 64, id 21253, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > 192.168.122.21.8000: Flags [.], cksum 0x12c6 (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 0
18:06:24.240702 IP (tos 0x0, ttl 64, id 21254, offset 0, flags [DF], proto TCP (6), length 135)
    laptop.49560 > 192.168.122.21.8000: Flags [P.], cksum 0xe0bf (correct), seq 1:84, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 83
18:06:24.241046 IP (tos 0x0, ttl 63, id 51382, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.122.21.8000 > laptop.49560: Flags [.], cksum 0xfceb (incorrect -> 0x1274), seq 1, ack 84, win 227, options [nop,nop,TS val 83877270 ecr 3284285254], length 0
18:06:24.442439 IP (tos 0x0, ttl 63, id 51383, offset 0, flags [DF], proto TCP (6), length 2798)
    192.168.122.21.8000 > laptop.49560: Flags [P.], cksum 0x07a6 (incorrect -> 0x7cb2), seq 1:2747, ack 84, win 227, options [nop,nop,TS val 83877472 ecr 3284285254], length 2746
18:06:24.443185 IP (tos 0x0, ttl 64, id 21255, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > 192.168.122.21.8000: Flags [.], cksum 0x05f9 (correct), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285456 ecr 83877472], length 0
18:06:24.443660 IP (tos 0x0, ttl 64, id 21256, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > 192.168.122.21.8000: Flags [F.], cksum 0x05f7 (correct), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285457 ecr 83877472], length 0
18:06:24.443935 IP (tos 0x0, ttl 63, id 51385, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.122.21.8000 > laptop.49560: Flags [F.], cksum 0xfceb (incorrect -> 0x0622), seq 2747, ack 85, win 227, options [nop,nop,TS val 83877473 ecr 3284285457], length 0
18:06:24.444356 IP (tos 0x0, ttl 64, id 21257, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > 192.168.122.21.8000: Flags [.], cksum 0x05f4 (correct), seq 85, ack 2748, win 272, options [nop,nop,TS val 3284285458 ecr 83877473], length 0

# VM
18:06:24.243655 IP (tos 0x0, ttl 63, id 21252, offset 0, flags [DF], proto TCP (6), length 60)
    laptop.49560 > vm.8000: Flags [S], cksum 0x04e3 (correct), seq 1995735410, win 29200, options [mss 1460,sackOK,TS val 3284285253 ecr 0,nop,wscale 7], length 0
18:06:24.243713 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    vm.8000 > laptop.49560: Flags [S.], cksum 0xfcf3 (incorrect -> 0x73bf), seq 2588808479, ack 1995735411, win 28960, options [mss 1460,sackOK,TS val 83877269 ecr 3284285253,nop,wscale 7], length 0
18:06:24.244510 IP (tos 0x0, ttl 63, id 21253, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > vm.8000: Flags [.], cksum 0x12c6 (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 0
18:06:24.244543 IP (tos 0x0, ttl 63, id 21254, offset 0, flags [DF], proto TCP (6), length 135)
    laptop.49560 > vm.8000: Flags [P.], cksum 0xe0bf (correct), seq 1:84, ack 1, win 229, options [nop,nop,TS val 3284285254 ecr 83877269], length 83
18:06:24.244555 IP (tos 0x0, ttl 64, id 51382, offset 0, flags [DF], proto TCP (6), length 52)
    vm.8000 > laptop.49560: Flags [.], cksum 0xfceb (incorrect -> 0x1274), seq 1, ack 84, win 227, options [nop,nop,TS val 83877270 ecr 3284285254], length 0
18:06:24.445989 IP (tos 0x0, ttl 64, id 51383, offset 0, flags [DF], proto TCP (6), length 2798)
    vm.8000 > laptop.49560: Flags [P.], cksum 0x07a6 (incorrect -> 0x7cb2), seq 1:2747, ack 84, win 227, options [nop,nop,TS val 83877472 ecr 3284285254], length 2746
18:06:24.446930 IP (tos 0x0, ttl 63, id 21255, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > vm.8000: Flags [.], cksum 0x05f9 (correct), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285456 ecr 83877472], length 0
18:06:24.447336 IP (tos 0x0, ttl 63, id 21256, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > vm.8000: Flags [F.], cksum 0x05f7 (correct), seq 84, ack 2747, win 272, options [nop,nop,TS val 3284285457 ecr 83877472], length 0
18:06:24.447502 IP (tos 0x0, ttl 64, id 51385, offset 0, flags [DF], proto TCP (6), length 52)
    vm.8000 > laptop.49560: Flags [F.], cksum 0xfceb (incorrect -> 0x0622), seq 2747, ack 85, win 227, options [nop,nop,TS val 83877473 ecr 3284285457], length 0
18:06:24.448032 IP (tos 0x0, ttl 63, id 21257, offset 0, flags [DF], proto TCP (6), length 52)
    laptop.49560 > vm.8000: Flags [.], cksum 0x05f4 (correct), seq 85, ack 2748, win 272, options [nop,nop,TS val 3284285458 ecr 83877473], length 0

Continuing

Instead of having to edit the routing table on every device on the network to forward packets to the dev machine, I setup my router to do it for me. Consult your router's manual for how to add static routes, then add something along the lines of:

Network/HostIP 192.168.122.0 Subnet
Netmask 255.255.255.0  
Gateway 192.168.1.10 Dev Server IP
Metric 0  
Interface LAN  

Naming may vary but hopefully it is clear enough to get you started.

Next Steps

  • Setup a cluster of Kazoo VMs with multiple zones and see how it goes
  • Orchestrate it all with…?

Spellcheck projects using aspell and Makefiles

Spellcheck using Aspell and Makefiles

With an international community surrounding Kazoo it is important to provide tools that help everyone stay on the same page with the development cycle.

One of the challenges, as a native speaker of English, is how easily our brains 'fix' misspelt words without us realizing it. Often enough, PRs are accepted where a key into a JSON structure is misspelled and goes unnoticed, time passes until someone realizes the error, and now JSON documents using that key must be supported to prevent previously working data from failing to work now (ask about Kazoo's history with 'wednesday' in the temporal routes callflow action!).

Fortunately there's a nice command line tool, GNU Aspell, that provides spell checking capabilities and custom dictionaries. Aspell does a great job suggesting alternatives when finding a potentially misspelt word, making it easy to find the correct word (or add a word to a personal dictionary).

Personal dictionary

A personal dictionary is helpful to maintain a word list separate and apart from the "default" English dictionary Aspell provides, especially concerning acronyms and jargon specific to the project. When you spellcheck files and find words that are correct but unknown to Aspell, you can add them so Aspell knows not to highlight them again. These words are added to the personal dictionary. Being a plaintext file, the personal dictionary is also easily included in source control so you can distribute it with your project.

Personal replacements

The replacement file can contain automatic replacements; when Aspell encounters a word in the replacement file, it will replace it automatically with the replacement word. Really handy for oft-misspelt words.

Installation

Aspell should be available through most GNU/Linux distros.

Makefile setup

You can easily add a make target to spellcheck a project. Create a splchk.mk file and add:

.PHONY = splchk

DICT = .aspell.en.pws
REPL = .aspell.en.prepl

$(DICT):
	@$(file >$(DICT),personal_ws-1.1 en 0 utf-8)

$(REPL):
	@$(file >$(REPL),personal_repl-1.1 en 0 utf-8)

splchk: $(DICT) $(REPL) $(addsuffix .chk,$(basename $(shell find . -name "*.md" )))

%.chk: %.md
	aspell --home-dir=. --personal=$(DICT) --repl=$(REPL) --lang=en -x check $<

Replace the $(shell find . -name "*.md") with whatever you want to use to find files to spellcheck.

When you run make splchk you will get the opportunity to check the spelling for each file found (if the file doesn't have any issues you won't deal with it). If you find a word that is correct but Aspell highlights it for repair, you can 'a'dd it which will include it in your personal dictionary.

Comparing Kazoo and other CPaaS solutions

Introduction

First, what is a CPaaS? Communication Platform as a Service is the expanded acronym but it boils down to APIs and programmatic control of the phone system.

The power of CPaaS is giving you the control of each call in a way that makes sense for the your (or your business') needs. It lets you access information outside of the phone system (such as CRMs and other databases) to make decisions about how to route a particular call.

Another feature of CPaaS is the ability to get events out of the phone system - via webhooks or over a websocket typically. This gives you the ability to track a call's progress through the callflow, updating information in CRMs and other databases and lets you build robust applications on top of the phone network.

CPaaS providers often offer API access to manipulate running calls (such as programmatically hanging up a call, initiating a transfer, etc), fetching CDRs (call detail records - typically used when billing for calls), initiating new calls, and more.

A newer feature, a requirement of CPaaS providers these days, is having WebRTC-based phones (or a phone in your browser), minimizing the software needed to be installed and allowing companies to provide a branded phone in their web apps.

Most CPaaS providers allow you to process phone calls without being a telecom junkie by providing a simplified set of actions that may be performed on a given call. No longer do you need to know what the PSTN is or how SIP or SDP work - you get to build on top of an abstracted phone system and focus more on your business needs.

The competitive landscape

When you view the competitive landscape in telecom, it is easy to see that the big players (traditional telcos and UCaaS providers like Avaya, Broadsoft, and others) are recognizing the value of adding a CPaaS-type offering by purchasing CPaaS providers like Zang and Nexmo (via Cisco acquisition). Twilio stands apart as an independent CPaaS provider that IPO'd in 2016.

The market has shifted and CPaaS + UCaaS is the norm going forward; UCaaS companies are racing to catch up while CPaaS companies are trying to expand into more traditional telephony environments.

Technical Evaluation

Let's dive into a bit of the nitty-gritty and see how a handful of providers compare.

Real-time Call Control

Most CPaaS offer a small list of actions/verbs that can be returned from your program. The basics are covered by most providers.

Connecting callers to endpoints

When a caller calls in, typically you want them to talk to someone (or several someones). Below is a quick chart on the various endpoints supported by CPaaS providers for connecting the caller:

Action Kazoo Twilio Plivo Nexmo Tropo
Registered Device X X X X
SIP Endpoint
Conference X
DID(s)
Call Queue - X X X
User (find-me/follow-me) X X X X
Group X X X X
Page group X X X X

Some notes:

  • Kazoo is the only platform that allows devices to register directly to the platform to receive calls. Registration is the phone's way of saying "Send calls for me to this IP/port please". Devices can then move around (say a softphone on a laptop) but still be able to receive calls easily. The SIP endpoint, in contrast, is typically in a fixed location and is harder to move once setup. An existing PBX is the typical destination for SIP endpoints.
  • Twilio is the only provider that offers a "Queue" option in the public docs. Kazoo has a community-supported queuing applications called ACDc as well as a commercial offering called Qubicle, which can be leveraged for call queues. You can also simulate queues with conferences.
  • The 'User' endpoint is the ability to associate multiple devices with a user and ring all relevant devices when the user is dialed. Most providers allow you to specify multiple endpoints to dial but that association must be tracked on your side. With Kazoo, you can create a User and use that in your callflow instead.
  • The 'Group' endpoint is similar to the User in that you can add devices and users to a group and all associated devices will be rung.

    The biggest differentiate among the providers in this case comes down to the need for registration, insight into why a call failed to connect (some providers are quite opaque about the reason for failure), and additional features offered on each endpoint type (like setting custom ringtones).

Caller interactions

Most providers allow you to interact with the caller by playing audio files, generating dynamic text to read to the caller (TTS - text to speech), and collecting touch tones (DTMF) from the caller.

Action Kazoo Twilio Plivo Nexmo Tropo
Collect DTMF
Play Audio (MP3 or WAV)
TTS

Other basics offered

Action Kazoo Twilio Plivo Nexmo Tropo
Hangup a call X
Pause execution X
Record the caller
Redirect to different URL X
Reject a call X

Kazoo Differentiators

While CPaaS allows you to interact with phone calls at a higher level than processing SIP packets, the features above are fairly low-level as far as business concerns go. They make for great demos and getting a prototype out the door quickly.

Invariably, once you gain traction, feature requests come in and more functionality is needed. The question to ask is, do you have to do the work to implement the functionality or is it provided already?

For instance, say your prototype connects the caller, based on some fancy criteria, to an agent/sales person/fortune teller. You gain traction and now your callees (the agents/sales people/fortune tellers) would love to have a voicemail service in case they can't answer the phone (since most are using a personal mobile phone and don't want the caller to hear their personal voicemail).

Do you want to build a voicemail box?

With Kazoo, you can configure a voicemail box via API and add it to the callflow with minimal effort. With the other providers, you're looking at having to get the IVR right, get the prompts recorded, test all the paths through a voicemail box, figure out storage of the voicemail recordings, build an API for providing the voicemail messages, build voicemail->email and attach the recording, archive/delete old/deleted messages, and more. This is not quick to build and, frankly, most people don't consider voicemail a feature worth paying much, if anything, for; is it a good use of your time?

Kazoo offers a number of callflow actions, like voicemail, that allow you to add higher level, business-focused, actions to your call processing capabilities. A short list of highlights:

  • Call Parking
  • Directed / Group Call Pickup
  • Eavesdrop
  • Faxing
  • Hotdesking
  • IVRs
  • Manual presence updates
  • Time-of-day routing
  • Voicemail

Each of these features represents days to weeks of work to implement yourself (if you even can given the primitives offered by the other providers) and each has its own set of edge cases and intricacies that come with it. Instead, why not use the pre-built, heavily-tested callflow actions provided by Kazoo?

API control

The single largest difference between Kazoo's REST API and other providers is the sheer breadth of APIs exposed by Kazoo. Remember, Kazoo is both a CPaaS and UCaaS platform; on top of that, everything is API driven so you while you might start with the CPaaS-specific features, you can easily grow into UCaaS type services (like office PBX features).

Call Control APIs

Each provider allows you to list and control active calls. What controls you have vary; some allow you to redirect the executing callflow to your server for new callflow instructions; others expose the actions you can overlay on the call via the API and that's it.

Similarly, conferences and conference participants can be managed through most providers' APIs.

All in all, there isn't much that differentiates the basics among providers.

Kazoo API Differentiators

  • Multi-tenancy

    Kazoo is designed to be multi-tenant; you can have as deep or as broad of an account tree as you like. Most providers, like Twilio, Nexmo, and Tropo, don't provide multi-tenancy at all (Plivo does appear to though). As a service integrator, you will have to manage that hierarchy on your side. This means you have to build infrastructure to manage billing, resource usage, configuration, etc.

  • Carrier management

    Kazoo allows you to define your own upstream carriers, so even if you're using a hosted instance of Kazoo (versus running it yourself), you should be able to add your own carriers to you account and use them instead of the system's carriers. You can even setup hybrid situations where you route certain destinations to your carriers and use the system's as the default carrier group.

    None of the other CPaaS providers offer BYOC (Bring Your Own Carrier). This means that as you grow your voice traffic, you will not have levers to pull to manage costs. With BYOC, when it makes sense to, you can find your own deals on connectivity and seamlessly switch to your own carriers to save costs.

  • Finer-grained features

    Each API entity (such as devices or conferences) in Kazoo offers significantly more functionality compared to the other providers' equivalent entities. Of course there are some similarities but, for instance looking at a "device" in Kazoo, you can optionally define:

    • Call forwarding settings, including for failover (or just straight call-forwarding)
    • Call recording (both inbound and outbound) for the device
    • Call restrictions based on number classifiers (can call US DIDs, can't call international, etc)
    • Caller ID settings
    • Dialplan settings (allow users to dial shorter numbers and "fix" them - for instance the US used to allow 7-digit dialing and the area code was assumed based on the phone's location).
    • Do Not Disturb settings
    • Codecs available (both audio and video)
    • Custom music on hold
    • Custom presence ID (instead of the SIP username)

    There are other Kazoo-specific features on devices as well. Since users can own devices, you can set similar properties on the user and have them propagate to all devices (and define properties on the devices to override user settings).

  • Class-4 and Class-5 Switching

    CPaaS providers are typically concerned with a handful of phone system features - dialing endpoints, conferences, call recording, call queue, and phone number management most commonly.

    Kazoo, being a hybrid class-4 and class-5 switch, offers so much more:

    PBX functionality includes configuring users, devices, and callflows, blacklists, directories, faxes and faxboxes, IVRs, voicemail, and more.

    Configure connectivity features like carriers (resources in Kazoo parlance), ratedecks, PBX trunking.

    Configure reseller features like branding email notification templates and white-labeling Kazoo services.

Questions to ask a CPaaS provider

These are some good questions to ask your CPaaS provider(s) and to ask yourself how important the answers are to your needs:

  • Where are the voice servers located, geographically?
  • Where are the API servers located? Hint, some providers may have voice spread out around the world but API servers only in geographic location.
  • Where are the servers located (the data centers) relative to your users? To your carriers?
  • How is the networking built? Some hosted infrastructure providers, like AWS/GCE aren't well suited to VoIP traffic - are CPaaS providers piggybacking off that infrastructure or are they building out their own racks?
  • Is the CPaaS provider reselling another provider's service with better lipstick?
  • Can you extend the platform?

Conclusion

The market has spoken and they want UCaaS+CPaaS offered under one roof. The big UCaaS players have bought their CPaaS plays and are busy integrating them into the fold with varying success. Meanwhile Twilio remains an outlier and looks to be doubling down on being the CPaaS for the masses.

Kazoo offers you CPaaS + UCaaS from day one in a cohesive system. You can use as many or as few APIs and callflow actions as you need to accomplish your goals, while knowing that as you grow and need new functionality, Kazoo is there ready to provide it. On the off chance Kazoo isn't able to provide a feature, you can rest easy knowing the open-source nature of the project gives you the option to build it yourself and keep it private, build it and release it upstream for all to benefit, or support the project by contracting with 2600Hz to build the functionality for you.

Ready to kick the tires?

Site improvements - Headers and Certs edition

Analyzing jamesaimonetti.com

After reading Julia Evans' post on the HSTS header, I felt like I should add it to this site. This lead me to Mozilla's Observatory that will analyze your site and grade it. Like Julia, I received a fun F - the biggest dings were missing headers and an invalid certificate chain. Let's look at how I improved my site's scoring.

The initial site

My blog is a Nikola-powered site. Since the pages are fronted by Apache (for the time being) most of the changes need to be applied there (as Apache controls the response headers). Let's add some headers!

Setting the HTTP Strict Transport Security (HSTS)

This post talks more about preload and other considerations.

On the https (port 443) virtual host, its a simple one-line addition:

Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"

The max_age is in seconds (63072000 is 2 years).

Setting the Content Security Policy (CSP) header

Mozilla has great help text about the header here.

Header always set Content-Security-Policy "default-src 'none'; font-src 'self'; img-src 'self'; object-src 'none'; script-src 'self'; style-src 'self'"

I've opted to keep everything in-house-only. If I start hosting images or fonts on other sites I can add them in as exceptions.

Certificate issues when redirecting

The redirection error seemed inappropriate as I use a permanent redirect from http to https:

<VirtualHost *:80>
	ServerName jamesaimonetti.com
	Redirect permanent / https://jamesaimonetti.com
</VirtualHost>

So I went digging a bit more.

One of the ideas was to support HTTP public key pinning (HPKP). A cursory view of Let's Encrypt discussion suggested that this might not be worth the trouble for a relatively low-value site like mine.

As it turns out, the addition of the HSTS header will likely improve this metric! Once we reload the site and rescan it with Observatory, we'll find out.

Setting the X-Content-Type-Options header

Pretty straightforward explanation and easily added:

Header always set X-Content-Type-Options "nosniff"

Setting the X-Frame-Options header

I don't want my site in an iframe (see here) so in it goes:

Header always set X-Frame-Options "DENY"

This also adds a setting to the CSP header:

Header always set Content-Security-Policy "frame-ancestors 'none'"

Setting the X-XSS-Protection header

Seems mostly for legacy browsers that don't support CSP (see here).

Header always set X-XSS-Protection "1; mode=block"

Testing the setup

Initial Errors

Once the headers were set, I reloaded the site in Apache and reloaded it in my browser. I verified the response headers were included but now I had a bunch of console errors to handle:

Content Security Policy: The page’s settings blocked the loading of a resource at self (“script-src https://jamesaimonetti.com”). Source: onfocusin attribute on DIV element.  jamesaimonetti.com
Content Security Policy: The page’s settings blocked the loading of a resource at self (“script-src https://jamesaimonetti.com”). Source: $('a.image-reference:not(.islink) img:no....  jamesaimonetti.com:637
Content Security Policy: The page’s settings blocked the loading of a resource at self (“script-src https://jamesaimonetti.com”). Source:
    moment.locale("en");
    fancydates....  jamesaimonetti.com:637
Content Security Policy: The page’s settings blocked the loading of a resource at self (“script-src https://jamesaimonetti.com”). Source:
  (function(i,s,o,g,r,a,m){i['GoogleAna....  jamesaimonetti.com:640

Google Analytics

Looking at the CSP FAQ about adding the Google Analytics, it looks like I need to whitelist Google in the img-src portion of my CSP.

Header always set Content-Security-Policy "img-src 'self' www.google-analytics.com"

This in and of itself did not address the issue as the inline javascript used to do the analytics isn't executed. The recommended way is to load the analytics javascript in its own file (vs inline in the html page). Currently in my conf.py I've added a BODY_END snippet with the google analytics setup. I've moved that snippet to /files/assets/js/ga.js and included it in my EXTRA_HEAD_DATA config:

EXTRA_HEAD_DATA = """
<script src="/assets/js/ga.js" type="text/javascript"/>
"""

Now I need to add google-analytics to my CSP script-src as well:

Header always set Content-Security-Policy "script-src 'self' www.google-analytics.com"

Improvement

With these changes above, scorecard improved to a solid B!

Still getting dinged with the HSTS header and certificate chain issues.

Turns out I had only included 2 of the three SSL-related directives in Apache. The third line below was added:

SSLCertificateKeyFile "/etc/letsencrypt/live/jamesaimonetti.com/privkey.pem"
SSLCertificateFile "/etc/letsencrypt/live/jamesaimonetti.com/cert.pem"
SSLCertificateChainFile "/etc/letsencrypt/live/jamesaimonetti.com/chain.pem"

chain.pem includes the intermediate certificates needed to complete the chain. See SSL Labs' SSLTest to find out more about your site.

Forward secrecy

For funzies, adding forward secrecy to Apache to bump those scores up (yay gamification!).

Disable RC4

RC4 is broken; don't offer it!

I added a ssl.conf in conf.d with this:

SSLProtocol all -SSLv2 -SSLv3
SSLHonorCipherOrder on
SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS"

Org-Babel and Erlang Escripts

I recently wrote an escript for work that used the wonderful getopt application for processing command-line arguments. Part of that script includes printing the script usage if passed the `-h` or `–help` flag.

As part of my initiative to use orgmode's literate programming capabilities, I wrote a `README.org` for the script that described how to install pre-requisites, what the script does, etc. I also wanted to include the usage text as part of the doc, since part of what I've been doing was adding more flags to tune the operations of the script.

The tricky thing is that the real usage text and what was in the doc frequently grows apart as development occurs. Wouldn't it be great to have the README.org stay in sync with what the current usage flags are?

Org-babel SRC blocks to the rescue!

With the appropriate SRC block, you can capture the results as part of the doc when exporting (or just capture the results in place) to a new format.

For my use case, I used a simple shell SRC block (with appropriate headers) to run the script with the appropriate flag:

./my_escript -h

Evaluating the SRC block with `C-cC-c` creates the attending `#+RESULTS:` section below. Except…it was empty. I couldn't figure out why the script's output wasn't showing up. Simple shell commands like `pwd` showed up just fine. What do?

STDERR vs STDOUT

Ah yes; it turns out that, by default, getopt prints usage to stderr instead of stdout. A quick fix to call getopt:usage/3 with `standard_in` as the third argument and the results showed up as expected.

I'm sure there are some other tricks to getting org-babel to get stderr output into the results but from my cursory searching it looked a bit more convoluted and not worth the time when I could fix it with such a small change.

Formatting tangled output in org-mode

I have been creating training materials to help people get comfortable with Kazoo. Part of those training materials include Erlang source code, JSON schemas, CouchDB design documents, and more. I have been using a literate programming style to create these materials, which include source blocks for the Erlang or JSON pieces. However, when "tangling" (extracting) the source blocks into their own files, the formatting (especially for the JSON) was terrible.

I immediately saw that ob-tangle.el has a hook, org-babel-post-tangle-hook, that executes after the tangling has occurred. What I didn't really understand, and took many false starts to figure out, was how this hook executed and in what context.

What I learned about org-babel-post-tangle-hook

After many false starts and brute-force attempts to make it work (so many lambdas), I took a day away from the issue and came back with both fresh eyes and a beginner's heart.

Here's the process of tangling, as far as I have gleaned thus far:

  1. run `org-babel-tangle` (`C-cC-vt` for most)
  2. The source blocks are tangled into the configured file(s) and saved
  3. When there are functions added to `org-babel-post-tangle-hook`, a buffer is opened with the contents of the file and the functions are mapped over.
  4. Once the functions have finished, the buffer is killed.

Read the source, Luke

Looking at the source:

(when org-babel-post-tangle-hook
  (mapc
   (lambda (file)
     (org-babel-with-temp-filebuffer file
       (run-hooks 'org-babel-post-tangle-hook)))
   (mapcar #'car path-collector)))

First, we see that nothing will be done if org-babel-post-tangle-hook is nil. `(mapcar #'car path-collector)` takes the `path-collector` list and applies `#'car` to each element (I'm not sure what `#'` before `car` is for). The resulting list will be used by `mapc` which we can read is like `lists:foreach/2` in Erlang - applying a function to each element in the list for its side-effects. That anonymous function (lambda) takes the element from the list (the filename I believe) and calls `org-babel-with-temp-filebuffer` with that and an expression for running the hooks.

Summarizing, if there are hooks to be run, call a function for each file that was tangled.

So what does 'org-babel-with-temp-filebuffer` do? From the lovely Help for the function: "Open FILE into a temporary buffer execute BODY there like ‘progn’, then kill the FILE buffer returning the result of evaluating BODY."

(defmacro org-babel-with-temp-filebuffer (file &rest body)
  "Open FILE into a temporary buffer execute BODY there like
`progn', then kill the FILE buffer returning the result of
evaluating BODY."
  (declare (indent 1))
  (let ((temp-path (make-symbol "temp-path"))
	(temp-result (make-symbol "temp-result"))
	(temp-file (make-symbol "temp-file"))
	(visited-p (make-symbol "visited-p")))
    `(let* ((,temp-path ,file)
	    (,visited-p (get-file-buffer ,temp-path))
	    ,temp-result ,temp-file)
       (org-babel-find-file-noselect-refresh ,temp-path)
       (setf ,temp-file (get-file-buffer ,temp-path))
       (with-current-buffer ,temp-file
	 (setf ,temp-result (progn ,@body)))
       (unless ,visited-p (kill-buffer ,temp-file))
       ,temp-result)))

Here we get deeper into Emacs Lisp than I really know, so I'll shoot a bit in the dark (dusk perhaps) about the functionality. `file` is our tangled file from the lambda and `body` is the `run-hooks` expression (not yet evaluated!). Basically, this code loads the contents of `file` into a buffer and folds `body` over that buffer. When finished, it kills the buffer.

Killed, you say?

Yes! So any formatting you may have applied to that temporary buffer is lost.

What to do?

We need a way to format the buffer, in a mode-aware way, and have that persist to the tangled file before the buffer is killed at the end of processing the hook.

The blessing and curse of Emacs is that all is available if you have the time and inclination! :)

The current implementation

A couple pieces of the puzzle, arrived at independently, put me on the right path.

First, with all of my Kazoo work, I wanted to ensure that the source files were all properly indented according to our formatting tool (which, of course, uses Erlang-mode's formatting!). Using a couple hooks to accomplish this gave me:

(add-hook 'Erlang-mode-hook
          (lambda ()
            (add-hook 'before-save-hook 'Erlang-indent-current-buffer nil 'make-it-local)))

So now I have an Erlang-mode specific `before-save-hook` action to indent the buffer prior to saving the buffer to the file.

I can, in turn, apply a similar hook into the js-mode (or js2-mode or JSON-mode) and use `JSON-pretty-print-buffer-ordered` to format the buffer. As long as I have a function that formats the current buffer properly, I can create the `before-save-hook` to ensure the buffer has formatting applied prior to saving.

The final piece was to figure out how to tie all this into `org-babel-post-tangle-hook`:

(defun my/run-before-save-hooks ()
  (run-hooks 'before-save-hook)
  (save-buffer)
  )

(add-hook 'org-babel-post-tangle-hook 'my/run-before-save-hooks)

What I finally came to was that, now that I had hooks available before saving for each major-mode I was interested in, and the buffer opened after tangling had the associated major-mode applied, all I needed was a way to run the `before-save-hook` hook and save the buffer before ceding control back to the `org-babel-post-tangle-hook` and `org-babel-with-temp-filebuffer` progn.

I am pretty happy with the result as so far I haven't encountered any glaring issues or performance problems. I hope others will find this useful, provide feedback on better, more idiomatic ways to accomplish the task, but overall I'm happy with the solution.

Unexpected wins

For whatever reason, when creating my .Emacs and attending customizations, I was mostly copy/pasting snippets I found (chalk it up to those dark PHP days). I didn't really take the time to grok what was happening, and most were `setq` or `define-key` anyway, so pretty simplistic. I haven't worked with Lisp in any real capacity since college (over a decade ago now - yikes) but with the last 8 or so years immersed in Erlang, I was familiar and comfortable with functional programming.

Having to dig into these functions because no one had written an easy guide to copy/paste was just the kick I needed to realize that, holy cow, I really did kind of understand what's going on here! There was a specific moment where I was ruminating on the code running the post-tangle hook, and reading the `mapc` description "Apply FUNCTION to each element of SEQUENCE for side effects only." clicked in my head as "Hey, that's lists:foreach/2"!

Navigating Emacs' help system, specifically 'describe function' (`C-hf`), has also been a boon, and I am more than grateful to the developers who've not only written the docs but the foresight to have it so easily accessible in Emacs. It has given me the genesis of an idea for a non-trivial Elisp project I want to attempt.

Finally

I haven't had a challenge like this in quite some time. Granted, on the surface it is a pretty silly, simple thing to have accomplished. On its own merit, probably not worth the time I spent. But the confidence gained in at least reading and comprehending Elisp code a little more fully, and the spark of an idea to try out, should more than compensate for that time. Now I just need to follow through and hopefully contribute a little back to the Emacs community.

The Great Migration of 2016

The Great Migration of 2016

It has been a long time since I've blogged for fun. A lot has changed and a lot has remained.

It is my goal to start writing more about what I'm up to, as much for an archive for my kids, family, and friends to read as it is to just flex the writing muscles. Most posts will continue on the nerdy theme as relates to computers and programming. However, I do plan to write up summaries about activities that involve the family and friends, for when nostalgia or curiosity about a time in life comes up.

Initially, though, I will start to write up a series of posts about how I'm consolidating as much as possible of my digital life into servers and services that I run, and using Emacs to interact with those services as much as possible.

Blog migration

I was a happy Wordpress user and developer when blogging first became a "thing". Cranking out plugins helped pay the bills out of college and blogging about technical things is ostensibly why I got a job at 2600Hz in 2010.

There are certainly ways to interact with Wordpress installations via Emacs but in the end, I wasn't happy with them and wanted something more streamlined. Through some series of events, I came across Nikola and appreciated the minimalist nature of the default installation, the ease with which I migrated existing posts from Wordpress, and the ability to manage posts using Emacs' org-mode, with which I've made a conscious effort to learn this year as well.

Git migration

Part of the appeal is that I can now put my posts, as they're static files, into version control. I've setup Gogs on my server and am transitioning my personal repos to it (and off of GitHub). I also have the full power of the command line (grep, awk, sed, etc) to work with my blog's corpus.

Going forward

I'm excited by the prospects of these (and other) changes. My goal has been to reduce the applications I use with regularity to two: Emacs and a browser. The more I can accomplish in Emacs, the less friction there is to me getting things done, which is part of why I'm so excited. Emacs is a tool that has gotten out of my way to the point that I don't even think about most keybindings I use. Emacs has become a natural extension of my thought process, and as long as my fingers can keep up with my mind, there's no impedance from my editor.

Hopefully this is the restart of my blog; no excuses aside from laziness now!

Using ibrowse to POST form data

It is not immediately obvious how to use ibrowse to send an HTTP POST request with form data (perhaps to simulate a web form post). Turns out its pretty simple:

    ibrowse:send_req(URI, [{"Content-Type", "application/x-www-form-urlencoded"}], post, FormData)

Where URI is where you want to send the request ("http://some.server.com/path/to/somewhere.php") and FormData is an iolist() of URL-encoded values ("foo=bar&fizz=buzz"). There's obviously a lot more that can be done, but for a quick snippet, this is pretty sweet.

Emulating Webmachine's {halt, StatusCode} in Cowboy

At 2600Hz, we recently converted our REST webserver from Mochiweb/Webmachine to Cowboy, with cowboy\_http\_rest giving us a comparable API to process our REST requests with. One feature that was missing, however, was an equivalent to Webmachine's {halt, StatusCode} return. While there has been chatter about adding this to cowboy\_http\_rest, we've got a function that emulates the behaviour pretty well (this is cleaned up a bit from our actual function, removing project-specific details).

    -spec halt/4 :: (#http_req{}, integer(), iolist(), #state{}) -> {'halt', #http_req{}, #state{}}.
    halt(Req0, StatusCode, RespContent, State) ->
        {ok, Req1} = cowboy_http_req:set_resp_body(Content, Req0),
        {ok, Req2} = cowboy_http_req:reply(StatusCode, Req1),
        {halt, Req2, State}.

Obviously you can omit setting the response body if you don't plan to return one.