Switching to aufs storage seemed to resolve it.

I used a base image box from phusion which seems to optimize for docker.