Shawn & Xujing,
I'm not convinced that these suggestions are necessary. You should not have to mess with maxstep for a driven circuit, and you should not need to give tstab to be where it has settled - at least for a circuit with "linear" settling. PSS (both shooting and harmonic balance) should solve for the steady state, and do not have to be at the steady state before they start. I have simulated cases where it would have taken 10,000 cycles to settle without needing a tstab. Generally tstab can help, but is mainly useful when the start up behaviour is not simple RC-type settling.
There are a few things that look odd. First of all, the picture of the psin form (BTW, I would generally recommend using the "port" component from analogLib rather than the rather outdated "psin" component) doesn't show the value for "Frequency 2". I suspect it is not set - which is why on the PSS form itself it only shows a single tone (at 5GHz). So only a single frequency is going into the LNA. If this had been set correctly, the auto-calculate would have shown the common frequency between the two inputs. Setting the common frequency to 40M won't help if you only have inputs at 5G because those sub-harmonics won't appear.
Assuming that you meant to have a second frequency at frf+40M, then 127 harmonics of 40M would only give you up to 5.08GHz. That's fine - because shooting PSS would give you spectral accuracy by default up to 4 times the largest harmonic requested (this is more for small-signal analyses that might follow, but the maxstep during the shooting interval would be set to something like 9.84ps (1/(PssFund*maxharm*4*5)). So you shouldn't have to do anything special - this should be fine - if smaller timesteps are needed to follow the waveforms, the simulator will do that anyway. You could set maxacfreq higher (say to 15G or something like that) if needed - but it shouldn't be needed. The maxstep that was reported in the PDF was actually from the tstab part, and that has looser timestep requirements (doesn't need to be so accurate).
Conservative is a good idea if you're trying to look at IM3 products (say), because these can be quite small and you need the resolution.
However, something is fishy - the fact that it doesn't simulate in transient is a cause for concern. As Shawn says, if it doesn't simulate in transient it won't in shooting PSS either. The convergence issue you showed appears during the initial transient (probably - because you sent screen shots of parts of the log - it would have been better if you'd attached the spectre.out file and also the input.scs to be sure what you're actually simulating). Maybe there's a problem with the circuit itself which is causing convergence failure - or maybe the inductors are making it unstable and it is taking short timesteps. Hard to know without seeing what you're actually simulating.
In general I would suggest using harmonic balance for this type of two-tone simulation. An LNA should be reasonably linear, and so a two-tone harmonic balance simulation (rather than trying to simulate the common frequency in shooting PSS) would be more efficient and more accurate (assuming you ask for a sufficient number of harmonics of each tone). That said, I'd be worried if I couldn't simulate it in transient...