Ask questions and get answers from experienced industry professionals
The killer mistake that people make is they let money become too big a deciding criteria, and then suffer through the whole project because of it. First, don't do that! Don't let the money become a bigger part of the process than the quality of service you want. Reputable contractors in any given area are buying from mostly the same vendors, so there aren't major differences in pricing. Interview the firms you are interested, find one that you feel a high level of trust for. If you "feel" one is going to be more expensive than another, examine that feeling and ask yourself if it isn't just because that firm might have a better handle on the entire scope of your project and their responsibilities within that scope than firms that might tend to gloss over things like that. Decide on the firm based on if you want to work with THAT firm or not.
That said, also make sure to write out a list of all the things you'd like to have when all is said and done, and then prioritize them, so you don't forget or bail on something important to you. And then- have fun!
Do your homework. You can never ask enough questions. Take your time. There is alot of fantastic information on the internet. In construction, there are many ways and opinions on materials, processes and costs. Only deal with reputable contractors. Check their credentials, get references and ask to see work they have done. With the last down turn in the economy, usually the good contractors are left standing, so use one that has been around for awhile. You could also hire a construction management consulting firm. They will know what questions to ask your contractor and can even help manage your project during the building process and confirm the project is on schedule and being executed as promised, using accepted building practices.
The biggest mistake you can make is to focus on price. Interview people that have had a custom home built. Ask them, "what would you do different?".... there is your answer of what mistakes to avoid!
Ask your friends about their personal experiences as a starting point. I am sure you will both good and bad stories. You want to make sure that you ask as many questions as possible upfront to reduce surprises during the process. Take your time, this process should not be rushed. Make sure the builder know that you want to stay on budget. Get a list of allowance items and shop the stores to make sure the allowances fit your home.
Most important to me would be to ask the builder for references and go talk to previous clients.
Good luck!