Ant colonies don’t stay the same size. A healthy nest has one or more queens laying eggs constantly, and a mature colony can hold thousands of workers. Warm, officer-summer conditions speed that up considerably, so a small nest in spring can become a serious problem by the following season.
Think of it like a snowball rolling downhill. Early on, it’s tiny and easy to stop, but every metre adds more mass and momentum. The longer you wait, the more workers there are, the wider they forage, and the harder the colony becomes to eliminate.