Colombia midfielder Carlos Sanchez was sent off in the third minute for a deliberate handball - the second fastest red card in World Cup history.
Shinji Kagawa scored the resulting penalty to put Japan 1-0 up in the sixth minute, but Juan Quintero equalised with a low free kick which squeezed over the line shortly before halftime.
After the break, Japan pressed a tired Colombia side and Yuya Osako grabbed the winner with a glancing header in the 73rd minute.
Colombia attacking midfielder James Rodriguez, who was left out of the starting line-up due to muscle pain, replaced Quintero in the 59th minute but failed to change the course of the match.