Higher town rating means faster growth. Also you should make sure that there is as much free space as possible in all directions from the center of your town. Blocking one part off with a big train station for example is a bad idea.
The growth is a bit random at times though, it is especially annoying when one of your big buildings gets replaced by a cottage early in the game as it will set you back for a long time. When it comes to deliveries, all that matters is if all cargos are delivered or not. Any extra supplies doesn't make a difference.
Silly Sad wrote: +1
and i want to add (may be it will be the first step to understanding the problem) that towns are not only growing -- towns suffers from population loss at the same time.
and more specific question is: what are the factors of population loss????
What I've experienced is that towns desire a balance between building types. So what you should be looking for when selecting a town, is the amount of buildings, not the population itself. A town with many small houses is a better choice than a town with fewer, but bigger buildings, because the first alternative is likely to grow big buildings, while the latter one is likely to destroy them.