The error message gave an important clue...you ran out of memory. Please type in an xterm
spectre -h rfmemory
like it suggests in the error message. There are a list of suggestions for you to go through. If you continue to have difficulty, I suggest contacting Cadence Customer Support.
Staff Support AE, Global Customer Support
Cadence Design Systems, Inc.