The state recorded its second highest tally of 627 new cases on Friday, and eight deaths, down from 723 cases and 13 deaths on Thursday.
Melbourne and Mitchell Shire are about halfway through stage three restrictions which are due to end in mid-August.
"We cannot open up with these numbers, we cannot open up with significantly less than these numbers," Premier Daniel Andrews said on Friday.
His warning came after it was revealed that an alarming number of Victorians with COVID-19 are not staying home, with more than 100 referred to police.
Australian Defence Force personnel and public health officials visited more than 500 homes of positive cases on Thursday, but one in four were not there.
Mr Andrews said there was no excuse for people who have tested positive not to isolate.
Opposition leader Michael O'Brien said there should be harsher punishments for breaching isolation rules such as monitoring bracelets and higher fines.
All 30 remaining residents at the hard-hit St Basil's aged care facility in Fawkner were moved to hospitals as replacement staff tested positive.
The move came after six St Basil's staff tested positive, prompting all other staff to go into self-isolation.
A Northern Hospital doctor in his 30s is battling the virus in intensive care at that hospital, making him one of three Melbourne doctors reportedly in ICU.
There are 5743 active cases and Victoria's total deaths now number 113, bringing the national figure to 197.
The postcode with the most active cases is 3029 with 346 cases across Hoppers Crossing, Tarneit and Truganina in Melbourne's west.
Meanwhile, a midwifery professor has criticised a Melbourne private hospital for inadequately caring for its premature babies because of its COVID-19 measures.
Mothers of babies in the Epworth Freemasons special care nursery are currently allowed to be with their baby for just one hour, twice per day.
Sydney University midwifery academic Karleen Gribble said the reason seemed to be because there wasn't much space in the nursery, but this wasn't a good enough reason.
"If you are preventing a mother from being with her premature infant then you are not providing adequate care to that baby," Prof Gribble said.
Research proves that mother and baby need each other for physiological development, bringing in the mother's milk and for bonding, she said.
In a statement, Epworth Freemasons said it had to limit people and their length of visit but was "working with each family to ensure there is the right amount of time together for feeding and bonding".
