Invasion is the process of taking over an enemy system. You cannot invade a system if you are at peace with the owner. You must be at war to travel to another player's colony (as opposed to an outpost).
After traveling to the system, you must first defeat any fleets the defender owns orbiting the system (and any competing invading fleets, if necessary). Then you may blockade and invade the system. Once you have begun invading, newly arrived enemy ships will not interrupt your progress.
Invading isolates the system from the rest of the defender's systems. It cannot benefit from any resources on other systems, nor they from its. Besieged planets still produce FIDS, but suffer heavy approval penalties from both the blockade and invasion.
If you wish to take over the system, the siege power of your invading fleets have to exceed the system's defense. The greater you exceed it, the faster the invasion will go, to a cap of ~5 turns (experiments welcome, depends on game speed). If the defenses of the system ever exceed the siege power of your fleets, the remaining turns will display 999. Siege power is a sum of military power (MP) of all besieging fleets in system, multiplied by empire's siege bonus. Siege weapons use their siege power (much larger than MP of "standard " weapons at the same tech level) for siege strength calculation. This makes specialized siege cruisers very viable option.
When siege completes, a player gets full control over the system, however approval there will be very low and will also rise very slowly, if not aided by warfare-branch planetary improvements High Energy Array, Community Defense and Gravity Warping (pre-Disharmony tech).
Invasion with only siege weapons works much slower. Player needs to weaken a well defended system with ships that carry bombers, fighters and a support module with appropriate bomb. Some bombs target improvements, others population. Both reduce the system's defense.
Another option is a troop invasion. Troop invasions have a chance of success from 0% to 100%. If a troop invasion is successful, you instantly take over the system.
Colonies have a defense value for defending against invasions, depending on population, improvements, and traits.