The action comes on the heels of the police shooting of Jacob Blake, a Black man, in Kenosha, Wisconsin, on Sunday.
"Myself and our players and our organization are very disturbed by what's happening in Kenosha," Bucks head coach Mike Budenholzer said.
"It's a great challenge to have an appreciation and a desire for change and to want something different and better in Kenosha and Milwaukee and Wisconsin and then go out and play a game."
The Magic, who trail 3-1 in the best-of-seven, first-round series, left the court when it was clear the Bucks were not going to play.
The Athletic reported that the Bucks players were in the locker room attempting to reach Wisconsin attorney general Josh Kaul. Citing sources, the online publication said the Magic would not accept a forfeit of the game.
The report added that the Houston Rockets and Oklahoma City Thunder were also planning to boycott their Game 5, which was scheduled to be played later today.
The Bucks did not immediately respond to a request for comment.
