There's no easy answer to this one. It's a lot of trial and error. You need to make sure you're running the right protocol for your mouse.
Many three button mice are "dual protocol" which means that they have both a 2-button and 3-button mode. The way to get the mouse to switch into 3-button mode (which usually then uses MouseSystems protocol) varies between different models.
You may need to slide a switch on the mouse or hold down the middle button when starting the server. Other methods of switching modes can be done by the server, you just have to find the right combination of settings for your mouse. See the Pointer section of the XF86Config man page for a complete list of settings.