Invasion is the process of taking over an enemy system. You cannot invade a system if you are at peace with the owner. You must be at war to travel to another player's colony (as opposed to an outpost).
After traveling to the system, you must first defeat any fleets the defender owns orbiting the system (and any competing invading fleets, if necessary). Then you may blockade and invade the system. Once you have begun invading, newly arrived enemy ships will not interrupt your progress.
Invading isolates the system from the rest of the defender's systems. It cannot benefit from any resources on other systems, nor they from its. Besieged planets still produce FIDS, but suffer heavy approval penalties from both the blockade and invasion.
If you wish to take over the system, the siege power of your invading fleets have to exceed the system's defense. The greater you exceed it, the faster the invasion will go, to a cap of ~5 turns (experiments welcome, depends on game speed). If the defenses of the system ever exceed the siege power of your fleets, the remaining turns will display 999.
You can weaken a well defended system with bombardment. Some bombs target improvements, others population. Both reduce the system's defense.
Another option is a troop invasion. Troop invasions have a chance of success from 0% to 100%. If a troop invasion is successful, you instantly take over the system.
Colonies have a defense value for defending against invasions, depending on population, improvements, and traits.