There are lots of reasons this error message can occur, see: http://www.google.com/search?q=sharepoint+Server+Out+Of+Memory. In my case it happened when a new site was being provisioned using a custom site definition during the ApplyWebTemplate call.
Turned out the there were subtle errors in my onet.xml. In particular I was referencing some document templates using an incorrect path:
<Project Title="$Resources:onet_TeamWebSite;" xmlns="http://schemas.microsoft.com/sharepoint/" Revision="2" ListDir="$Resources:core,lists_Folder;" xmlns:ows="Microsoft SharePoint">
...
<DocumentTemplates>
<DocumentTemplate Path="INCORRECTPATH" Name=""
Edward Wilde 