You can improve the passive cooling with an aluminum box around the vesc, just having more metal to absorb the heat. I think this build by @jkoljo is doing just that:
Another thing you can try is filling the box with CorrosionX and having the esc submerged in the stuff. This would also offer some protection from water intrusion.