Assume that you have two broken and two unbroken units in close proximity to your leader and wish to rally the broken units.
You start by activating your command team and may command up to three units in command range. Your leader commands the two unbroken units to advance. After both commanded activations are complete, your leader may act himself. You decide not to advance with the other troops but instead move closer towards the broken units.

You have to command other units and activate the command team before you can rally other units. This will often force your leaders to stay behind if they want to rally broken units. The range for rallying units is equal to the command range and is higher for Elite units.
After your leader has completed his action, you decide how many and which troops to rally. You can choose any combination of units as long as the total number does not exceed three.
You desperately need every unit available and thus decide to rally both broken units. Rallying two units means a +1 modifier on your required dice roll - that makes a 4+ to succeed.
Fortunately, one of the units has an Elite skill rating. That makes things easier for your leader. Subtract -1 from your required dice roll. You end up with a final target number of 3+ so a roll of 3, 4, 5 or 6 will be successful.
You roll a 5 - congratulation, you've done it. Both units have been rallied and from that moment on do not count as broken any more. If you had failed, none of the units would have been rallied and virtually nothing would have happened.
Both rallied units are free to activate in that turn, but note that the leader who rallied them cannot command them - he has finished his activation with the Rally Action. Your newly rallied troops may either be commanded by another leader or try to act off command (risking to break again).
There is also an editor's note on rally explaining the concept.
Relevant rules are described in the command chapter of the infantry combat rules.
More rules covering broken units have an extra chapter in the rules.