The backlinks won’t immediately drop off because you changed the domain’s servers. It could take quite a while, if ever, for the linking sites to notice the change and update their links. The backlinks should remain in place until the sites actively recheck and remove the dead links.
