A Good Mediator
Smooth mediation starts with the right mediator. Of course, this means someone with relevant training and experience in both the relevant divorce law and in mediation. It also means someone with the right temperament. Most effective mediators maintain a calm but assertive temperament: calm enough not to stoke raw emotions, but assertive enough to be able to convince the participants they are being unreasonable.
A good mediator also does a significant amount of preparation ahead of time. At the very least, both sides should be required to submit a written explanation of their opinion. The best mediators will talk to both sides ahead of time, going back and forth if necessary, to understand what the issues and the pressure points are. A good mediator in a divorce with financial issues will expect a list of all the assets and debts, each party's opinion of the value of each asset, and the basis for that valuation.
When I do mediation, I usually do it in two parts. First, each side meets with me for an hour or two and presents their case. Then we all meet together (if possible) and I usually give everyone homework. Sometimes we need documents. Sometimes we need an appraisal of a house. It may seem surprising, but many attorneys come to mediation without having done the basic work of getting a list and valuation of the assets.
The Participants
Start by being prepared. If your mediator hasn't required you to provide a list of assets and debts, do it anyway. Be ready to explain the basis for your valuation of each asset and debt. If the issue is custody, come with options and alternatives for what you think is best for the children.
Take positions that are just aggressive enough. Some people go into mediation asking for the world, and the other side is immediately turned off and adopts the belief that mediation is worthless because the other side is being ridiculous. On the other hand, don't give your bottom line right away: keep some wiggle room. Each side will want to feel like they got something in the mediation, so let them argue you down. If you start with your bottom line, you've nowhere to go but out the door.
Try to be creative and open minded in your solutions to problems. There is usually something for both sides to gain by coming to a resolution now rather than fighting in court. For instance, I once had a husband agree to let the wife have property in the divorce, as long as he had the option to use it for several years if he made certain payments to her as 'rent.' (With the payments structured in a certain way as to not be taxable under IRS guidelines.) I have seen former spouses agree to a time-sharing agreement on each other's houses. Think about what you really want and what your spouse really wants: maybe it is possible for everyone to get what they want.
Be realistic about what you can get at court. This is usually the BATNA - Best Alternative To a Negotiated Agreement. If the agreement is better than your BATNA, then obviously you should take it. Try to understand what the other side can expect in court as well - don't expect them to compromise for less than they can get in court. When I meet with clients before mediation, I usually discuss with them in detail what I believe they can get in court. It is easier to keep a clear head and think rationally a few days before the mediation starts. I write it down so that the client and I know what we will accept and when we will walk away. That doesn't mean you should aim to get just what you can get in court - aim for a bit more (what those trained in negotiation sometimes call an 'Aspirational Value.') But be ready to fall back to your BATNA if necessary.
Keep calm and view the mediation as a business transaction. Treat the other side with courtesy and respect throughout. It is probably the biggest business transaction of your life. Don't let emotions mess it up.
The Result
The best agreements are usually those where everyone is equally unhappy. In any case, if you think you 'won' then I suggest you keep it to yourself. You don't want the other side to know you would have compromised for less: that just gives them motivation to want to back out.
I cannot emphasize this enough: get your agreement in writing, signed by everyone. I don't know about other states, but in Nevada verbal divorce agreements cannot be enforced. Besides that, if there isn't a writing, it is almost guaranteed that both sides will come out thinking they agreed to something different. If the mediator isn't going to do a written agreement, then do it yourself. Mediation without a written agreement signed before everyone leaves is likely just a waste of time.