The COVID-19 pandemic severely affected the lives of families and the well-being of both parents and their children. Various factors, including prenatal stress, dysregulated stress response systems, and genetics may have influenced how the stress caused by the pandemic impacted the well-being of different family members. The present work investigated if emotional well-being during the COVID-19 pandemic could be predicted by developmental stress-related and genetic factors. Emotional well-being of 7–10 year-old children (n = 263) and mothers (n = 241) (participants in a longitudinal German birth cohort (POSEIDON)) was assessed during the COVID-19 pandemic using the CRISIS questionnaire at two time periods (July 2020–October 2020; November 2020–February 2021). Associations of the children’s and mothers’ well-being with maternal perceived stress, of the children’s well-being with their salivary and morning urine cortisol at 45 months, and polygenic risk scores (PRSs) for depression, schizophrenia, loneliness were investigated. Lower emotional well-being was observed in both children and mothers during compared to before the pandemic, with the children’s but not the mothers’ emotional well-being improving over the course of the pandemic. A positive association between the child and maternal emotional well-being was found. Prenatally assessed maternal perceived stress was associated with a lower well-being in children, but not in mothers. Cortisol measures and PRSs were not significantly associated with the children’s emotional well-being. The present study confirms that emotional well-being of children and mothers are linked, and were negatively affected by the COVID-19 pandemic, with differences in development over time.