When you check the Use custom form layout checkbox, you can define a layout which will be used on the editing form. You can use the WYSIWYG editor and you can also insert a field label, input, validation label and/or submit button.
Detailed information about document types and transformations can be found in Developer's Guide -> Development -> Document types.
Page url: http://devnet.kentico.com/docs/6_0/contexthelp/index.html?form_tab3.htm