Christopher Pyne says the debate ahead of the upcoming ballot, pending the outcome of High Court challenges, must be respectful.
His comments follow the distribution of posters by neo-Nazis depicting a child cowering below two people brandishing rainbow belts.
They cite widely discredited research linking same-sex relationships to higher levels of abuse.
While Mr Pyne agreed Australia has a robust democracy, he also decried "lurid and mad" claims.
"There are crazy people out there who do say mad things," he told ABC radio on Wednesday.
"But we also shouldn't give them an airing."
The frontbencher supports same-sex marriage but not because he is a "small l" liberal.
"I'm in favour of it because as a conservative, I want more people to share in the institution of marriage," he said.
"So let's have a respectful debate but let's not give an airing to people with wicked views."
Prime Minister Malcolm Turnbull, who is also backing the "yes" vote, has urged Australians to ignore "extreme and unpleasant" campaign material.
But Labor wants him to go further and personally condemn the posters.
Nationals senator Matt Canavan on Wednesday defended traditional marriage and argued changing the Marriage Act would be a radical reordering of Australian socieity.
"What happens to the family when we no longer connect marriage to the lifelong commitment of raising a child, but to the more temporary and ephemeral state of love," he wrote in The Australian.
"Advocates for it should be asked for a better explanation than five words and the title of a Beatles song - All You Need Is Love."
