To answer your last question, you can define the procedures you are using in the menu before or after the menu item is defined. It is independent of the item definition and will only cause an error if it is not available when the menu button is pushed.
Your initial issue is because there is already user triggers set for maskLayout. You have to get that information and add your trigger to it.Note that if there is more than one procedure defined for a trigger, you will only get the first procedure.
appInfo = deGetAppInfo( "maskLayout" ) ;returns a property list where:
userApp = appInfo->userAppTrigList ; This is the user pre install trigger
userMenu = appInfo->userMenuTrigList ; This is the user menu trigger
userPost = appinfo->userPostInstallTrigList ; This is the post intall list
deUnRegUserTriggers("maskLayout") ; Unregister the triggers
deRegUserTriggers( "maskLayout" userApp userMenu 'trPostUserInstallTrigger)
You probably can find more info on these in the Virtuoso Design Environment SKILL Reference.