An assistant coach of the Japan national side, Kiwi Brown also fears much of the goodwill towards rugby could be lost in the country unless it can forge a place in a revamped international club competition.
The Sunwolves concept is effectively over after the announcement last week the franchise wouldn't be included in an Australian Super Rugby competition because of travel restrictions and other logistics.
It prematurely ended what was their fifth and final season in Super Rugby's Australian Conference, having learned last year that 2020 would be their last campaign.
While the team struggled for results throughout their existence, they continued to attract healthy crowds, in contrast to other more established teams.
Brown has previously been critical of South Africa's role in the Sunwolves' demise but took his attack to a new level after the team was officially shuttered.
He is adamant the Sunwolves were targeted after Japan voted for France, rather than South Africa, in a closely fought race to stage the 2023 Rugby World Cup.
"I just feel as though the relationship between South Africa and Japan was pretty nasty there when South Africa were really trying to get the Sunwolves kicked out of the competition," said Brown, now an assistant coach at the Highlanders in New Zealand.
"I've mentioned previously the hatred South Africa had for the Sunwolves and Japan rugby around them not voting for South Africa to host the World Cup.
"I think that's where it started and now we've ended up with not having the Sunwolves involved and a little bit of grey around what role Japan will play in the future of Super Rugby."
Brown said it was important for the sport's sake that Japan is involved in any top-flight international club or franchise competition that evolves next year.
He was heartened by reports that Japan's leading Top League clubs were being considered to join an international play-off series, believing they would be competitive in such an environment.
He said interest in rugby and the Top League had soared after Japan's hosting of last year's Rugby World Cup but that sentiment was under threat following the COVID-19 hiatus.
Brown and head coach Jamie Joseph helped mastermind the Brave Blossom's charge to the quarter-finals.
They were knocked out, ironically by South Africa, who went on to clinch the title.
Brown said that outcome and the goodwill shown to the Springboks by the Japanese public may have healed rugby relations between the nations.