There are 2 ways to run Calibre:
a. Export schematic as cdl netlist (ciw:File->Export->Cdl) and layout as gds (ciw:File->Export->Stream). Specify both inputs in the calibre drc/lvs control file and run in batch mode.
b. Run drc/lvs from within Virtuoso using Calibre menu. The basic steps to set up the "Calibre" menu in the layout editor are as follows. For help in troubleshooting the setup, I think you should contact Mentor Graphics because the menu SKILL file is provided by them. It is not from Cadence.
Add the following line to your cshrc file:
setenv MGC_HOME <path-to-your-calibre-software>
Add the following commands to your .cdsinit file:
=== IC5141 ===
load( strcat( getShellEnvVar("MGC_HOME") "/shared/pkgs/icv/tools/queryskl/calibre.skl" ))
=== IC613 ===
load( strcat( getShellEnvVar("MGC_HOME") "/shared/pkgs/icv/tools/queryskl/calibre.OA.skl" ))