If the only large signal is the VCO (i.e. you have no other large signal driven inputs), then you would simulate the mixer just as you would if you were simulating an oscillator - i.e. tell PSS that it's an oscillator and point at the output of the oscillator. Then you can have a small-signal input to the mixer, and look at the transfer function to the output of the mixer.
If you want to have a large signal (driven) signal to the mixer, to investigate (say) distortion of the input signal through the mixer, in the presence of the VCO, then you can also do this with the "hb" analysis (you need to use "hb" rather than "pss" in "harmonic balance" mode). The hb analysis allows what we call semi-autonomous analysis. If you pick this, turn on the "oscillator" checkbox and select the oscillator outputs as usual. Then for tone 1, specify the estimate of the oscillator frequency, and number of harmonics, and then specify the frequency of the other driven signals. Hopefully the use model is reasonably clear...
If this doesn't make sense, please ensure you include the subversion of icfb/virtuoso you're using (Help->About... in the CIW), and the subversion of spectre you're using (this will appear in the spectre.out displayed in ADE, or by typing "spectre -W" on the UNIX command line).