The "doneProc" for enterPolygon automatically passes the windowId the points and a t/nil (done) to indicate if the routine was completed or not. Cadence provides hooks for doneProcs or callbacks to some of their interactive routines and these are usually passed a fixed set of arguments. The documentation should explain what these arguments are. If you are having trouble finding the doc (or are too lazy to look it up), you can use the following trick to find out how many arguments are needed and what type of arguments they are:
procedure(myPolygonDp(@optional a b c d)
println(list(a b c d))
This will take an optional number of arguments and default the rest to nil. It will then print out their values. In the preceding example, it would be easy to see that the windowId and points are passed to the routine. It would be less clear what the second argument (done) is since you would simply get a t. There is usually enough information for me to write the routine. The code given in the documentation is self-explanatory since it checks the value of done and draws a polygon in w using pts.
As for your second question, I have observed the same as you (the last object created appears first in the selectedSet for all objects). I do not know if this will work all the time. A safer approach could be to get a list of all objects in the cell. Call the leHiCreatePolygon code. Then get a new list of all the objects in the cell and identify which object is not in the original set.
cv = geGetEditCellView()
polygons = setof(x cv~>shapes x~>objType=="polygon")
newPolygon = car(setof(x cv~>shapes x~>objType=="polygon" && !memq(x polygons)))