Most Victorians have been doing the right thing after it was made mandatory last week for people in locked-down Melbourne and the Mitchell Shire to wear face coverings when outside, Mr Andrews said.
Other Victorians are expected to wear masks when the 1.5m social distancing can't be done.
"If you are just making a selfish choice that your alleged personal liberty, quoting something you've read on some website - this is not about human rights," Mr Andrews said.
"There are 10 families that are going to be burying someone in the next few days. Wear a mask."
Videos released on social media of a woman challenging Bunnings workers who asked her to wear a mask to enter a store and calling out a post office worker have also sparked community outrage.
Stores like Bunnings operate on private property and have a right to make requests of customers, including that they wear masks.
The woman in question told the Bunnings worker she was being discriminated against and the request was against the law and her human rights.
Australia's Deputy Chief Health Officer Nick Coatsworth said it was "disturbing" to see people attacking others who are doing their job.
"It is not a human rights issue to decide not to wear a mask," he told the Today Show on Monday.
"I just can't see how it takes away someone's human rights or individual liberty to just have to put a face covering on at the request of government. It is simple. We know it's effective. We know it's the time to do it."
Dr Coatsworth praised the overall efforts of Victorians to stay home or wear masks when out in public.
"It sounds so simple. We realise it's difficult. The quicker we do that together the quicker the curve will bend down the other side," he said.
Police said on Sunday that 126 fines were issued in the past 24 hours.
Twenty $200 fines were handed out to people for failing to wear a mask, including to a man and a woman who refused to give police details when they were stopped.
The state reported a record 10 new deaths for COVID-19, taking the state's toll to 61 and the country's to 155.
