The Monsters option allows certain monster rules to be applied. The checkmark by each option signifies that it is selected to be used for PC creation.
The Use Default Monsters uses the default monster skills, feats etc from the LST data files.
The Use HD +/- instead of monster classes means that you add Hit Dice to monsters as opposed to Monster classes.
The Ignore Monster HD Cap will allow the user to exceed the maximum HD as defined in the data monster files.
*** Note
The Default Monster Mode used when this option is checked is deprecated in version 5.10, it will be removed in version 5.12. The feature is being replaced with the use of Default Monster Kits.