Could father get custody?

Asked: Could father get custody?

My son is almost 3 months old. For my entire pregnancy, the father of the child wanted nothing to do with me or the baby & threatened me constantly. I stayed away to remain healthy during my preg and have been raising my child alone since birth. Now, the baby's paternal grandfather has found me and come forward, wanting to be in the baby's life. That is fine…I'm worried, however, about the father now following through on his threats of taking my baby away. I'm a great mother, I have a good job etc. My question is IS THERE ANY CHANCE HE CAN TAKE MY BABY AWAY WITHOUT PROOF I'M AN UNFIT MOM!? I'm scared because his family has money & theyre not very nice…