Chewing The Cud

I’m currently working on a small startup project, for one to meet a need of some acquaintances, but more importantly to learn me some Erlang with regards to the web.

While I’m further along than I actually expected to be, I thought I’d begin documenting the steps I’ve taken towards building this app.

The current nerdities I’m using:

• Emacs – Editor of choice
• Erlang – Functional programming language
• Webmachine – REST toolkit
• CouchDB – Document store
• Couchbeam – Couchdb framework for Erlang
• ErlyDTL – Templating engine
• ESTMP – Email sending
• Tsung – Stress-testing

Installation of all of these on a GNU/Linux system is pretty straightforward, so I won’t cover that here. Defaults were used for Erlang. I installed the other libraries/applications in ~/dev/erlang/lib and pointed $ERL_LIBS there in my .bashrc.

I did follow this guide for setting up Tsung. The BeeBole site has several other pages worth reading for developing web applications in Erlang.

Once installed, build the webmachine project:

$WEBMACHINE_HOME/scripts/new_webmachine.erl wm_app /path/to/root
cd /path/to/roow/wm_app
make
./start.sh

You now have a working project! Of course, I like to have my Erlang shell inside of emacs while I’m developing, so I added a comment to the start.sh script that contained the shell parameters. My start.sh looks like this:

#!/bin/sh
 
# for emacs C-c C-z flags:
# -pa ./ebin -pa ./priv/templates/ebin -boot start_sasl -s wm_app
 
cd `dirname $0`
exec erl -pa $PWD/ebin $PWD/deps/*/ebin $PWD/deps/*/deps/*/ebin $PWD/priv/templates/ebin -boot start_sasl -s wm_app

I currently have all of my dependencies in $ERL_LIBS; when I deploy this to production, I’ll add the libs to the wm_app/deps as either a symlink or copied into the directory.

To have the custom shell means you need the .emacs code to start an Erlang shell with custom flags.

Important note: If you need to specify multiple code paths in the -pa arg, you have to use a -pa for each path, unlike in the shell command version where any path after the -pa (or -pz) is added.

Another caveat: when starting the Erlang shell within emacs, if you’re currently in a erlang-related buffer (.erl, .hrl, etc), the default shell is started without the option to set flags. I typically have the start.sh open anyway to copy the flags so I don’t run into this much anymore; I’m documenting it here just in case anyone stumbles on it.

Now you have a shell within which to execute commands against your webmachine app, load updated modules, etc.

Coming up, I’ll talk about how I’m using ErlyDTL to create templates and using CouchDB/Couchbeam for the document store.

While working on a script today that had been working, I couldn’t for the life of me figure out why it was failing. It uses the PHP curl_* functions to make various requests and processes the results. Turns out when you send a POST body with the CURLOPT_POSTFIELDS and a value field begins with an at symbol(@), you have to escape it (\@). The reason is the at symbol is used by curl to denote a file upload path (“@/path/to/upload.file”). So escape the at symbol and you should be back to good with the curling.

Working with symfony, especially when adding to the schema and generating the model, form, and filter classes, it becomes tedious to add each of the new files to your subversion repository. Here’s a succinct line to add all un-versioned files to your repo:

#!/bin/sh
 
svn add `svn st | grep ? | head | awk '{print $2}'`

The key is in the tick-marked section. It takes the output of svn st(atus) and pipes it to grep, selecting only the un-versioned files (denoted by the ?), pipes that to head which outputs the first 10 (by default) lines, and pipes that to awk which prints the second column containing the file path to be added.

But what if you have more than 10 files to add? You can easily pass a -n NUM switch to the head command to increase the number of lines it reads in at a time. I’ll leave it as an exercise to the reader to modify the script to allow a user to pass in what NUM should be.

So when your “svn st” output is filled with un-versioned files, all of which you need, run this little guy and have it done speedily.

I have a bulk import process that needs running nightly. Currently there are around 4,200 “rows” to process, which actually can encompass many tables so row is not entirely appropriate. The problem is that the script poops out after ~200 “rows” with memory limit errors. While increasing the memory limit is do-able, I am not interested in that as a solution currently.

First, I recorded some numbers to benchmark where the script was in memory consumption. The first number is the “row”, the peak is how much memory the script has allocated, and the mem is the delta in memory allocation per row.

0 peak: 41,103,408 mem: 29,884,416
1 peak: 42,440,264 mem: 1,310,720
2 peak: 43,613,848 mem: 1,310,720
3 peak: 43,893,960 mem: 262,144
4 peak: 44,223,040 mem: 262,144
5 peak: 44,896,296 mem: 786,432
6 peak: 45,671,560 mem: 786,432
7 peak: 45,865,952 mem: 0
8 peak: 46,418,272 mem: 786,432
9 peak: 47,917,888 mem: 1,310,720
10 peak: 48,566,312 mem: 786,432

After the initialization phase, memory allocation increases fairly steadily and it is not long until the 128M memory limit is reached. This is unacceptable as I know some “rows” should be much closer to 0 as nothing is imported on the majority of rows.

My first solution was to disable logging:

sfConfig::set('sf_logging_enabled', FALSE);

The initial memory allocation was decreased, but the running deltas remained higher than expected.

Second, I inserted a ton of unset() calls in the various functions. This dropped my deltas a little:

0 peak: 30,611,472 mem: 20,971,520
1 peak: 31,969,760 mem: 1,310,720
2 peak: 32,180,112 mem: 262,144
3 peak: 32,333,216 mem: 262,144
4 peak: 32,525,144 mem: 262,144
5 peak: 32,656,616 mem: 0
6 peak: 32,863,736 mem: 262,144
7 peak: 33,103,264 mem: 262,144
8 peak: 33,455,544 mem: 262,144
9 peak: 33,754,288 mem: 262,144
10 peak: 33,984,976 mem: 262,144

But allocation still killed the import before it could complete. Browsing through various sites, I discovered Propel had a hard time cleaning up circular references, which meant PHP couldn’t garbage-collect that memory. However, to combat this, Propel 1.3 offers a static method disableInstancePooling that allowed me to override Propel’s desire to keep instances around.

Adding:

Propel::disableInstancePooling();

to the beginning of the import gave me these results:

0 peak: 26,569,632 mem: 17,301,504
1 peak: 28,582,152 mem: 2,097,152
2 peak: 30,455,352 mem: 1,835,008
3 peak: 30,536,176 mem: 0
4 peak: 30,536,176 mem: 0
5 peak: 31,517,088 mem: 1,048,576
6 peak: 31,534,152 mem: 0
7 peak: 31,552,120 mem: 0
8 peak: 31,589,632 mem: 0
9 peak: 31,695,504 mem: 0
10 peak: 31,695,504 mem: 0

Now new memory was allocating only when the import was actually doing something of significance. In fact, watching the import proceed with the deltas displayed, I could observe the memory decreasing at times, prolonging the life of the script by orders of magnitude. Whereas before the script was processing ~200 “rows”, it currently processes the whole batch (4,237 “rows” currently) in one go.

As the importable “rows” increase, I know I won’t be butting up against memory limits for some time.