Working out in my first trimester

I have one beautiful little girl. Before her I had a miscarriage and then after her I had two more. Now I'm pregnant again, but scared I'm to blame for the other three. I exercised a lot with all three. My doctor said miscarriages are not caused by working out. I still felt scared so got a second opinion. That advice nurse told me to stay away from all working out and caffeine because because of how many miscarriages ive had. Now I'm not sure what to do you work out or not.