'I’ll have to leave': Residents say brutal rates rise will drive them out