The layers y0-y9 are only used for plotting signals (from ADE). This is because the same layers are used for probing signals in the schematic - the idea is that the graph trace matches the schematic probe - and so you can see which signal is which. The y0-y9 layers therefore come from the technology library attached to the schematic library, and it will use the packets associated with those layers in that technology library to determine how they are drawn.
For expressions (which is what the Direct Plot forms/menus generate), it will use the standard ViVA colours and linestyles. For the colours, the default set and order are fixed (I think) - although you can change them after plotting. The default line style is set via:
envSetVal("viva.trace" "lineStyle" 'string "solid")
I believe (and I think this is in the documentation) that the allowed valis are: solid, dot, dash, dashdot, dashdotdot
You also have the "lineThickness" which can be set to "fine", "medium", "thick", "extrathick"