Technically it is only the singularity at the center which doesn't grow when seen from a frame of reference on the outside. But things will still fall in below

Technically it is only the singularity at the center which doesn't grow when seen from a frame of reference on the outside. But things will still fall in below the event horizon when seen from the outside. A black hole forms when the density (of energy or matter) at some point adds up to a gravity that light can't escape. Colliding two neutron stars would likely instantly cause a high enough density in a volume such that a black hole and its event horizon can form, which would envelop a very large mass in much less than a second.