A colleague of mine recently asked one of his classes whether they think it’s OK for women to give birth out of wedlock. Almost all said yes. Next they were asked if their parents would be upset if they got pregnant or impregnated someone? Again, almost all said yes.

Is this hypocritical? Social conservatives have long condemned those vocal elites on the left who espouse a do-as-I-say-not-as-I-do ethos when it comes to sex and relationships: I’m married and monogamous, but it’s OK for you to be a libertine. As the late James Q. Wilson and others have suggested, this tends to work out fine for the middle and upper classes—they can smoothly transition to stable relationships after they’ve sowed their oats—but not so well for the under-privileged, whose romantic entanglements are more likely to yield out-of-wedlock births than stable relationships.

Casting the students’ ostensibly conflicting opinions as polar opposites strikes me as unreasonable. The issue is not hypocrisy, but moral complexity.

Almost everyone knows that unmarried families aren’t great for kids. Princeton sociologist Sara McLanahan knows it, bleeding heart liberal New York Times columnist Nicholas Kristof knows it; only a marginal fringe believes otherwise. They can rant and rave all they want, but there’s 30 years of mainstream sociology against them. I’ll also take for granted that single motherhood offers challenges that can’t easily be fixed. Even a massive program of income redistribution under a Bernie Sanders presidency wouldn’t drive down nonmarital fertility to 1960s levels because some of its antecedents are purely cultural. (If you believe that culture hasn’t changed, perhaps you also question the existence of modern contraception and its impact on sexual behavior?) Nor would such a program make things OK for the remaining kids raised without two parents, as Sara McLanahan and others inform us. Better? Absolutely. Completely OK? No.

Yet to express vociferous disapproval toward single motherhood is to cast aspersions on people who already have many, many problems. (My work with Haverford College sociologist Matthew McKeever shows just how profound these problems are.) It makes it more difficult to engage with them, as W. Bradford Wilcox and I learned in conducting research for our new book, Soul Mates: Religion, Sex, Love, and Marriage among African Americans and Latinos. We spoke to many pastors who minister to flocks full of single mothers. Although they frequently promoted right behavior—kindness, supportiveness, fidelity—in romantic relationships, they rarely spoke explicitly about marriage. Doing so could alienate the very people they’re trying to help. Certainly there’s a middle ground between endorsing marriage and wholesale condemnation of those who forsake it, but elite discourse struggles to stay there.

As Kathy Edin and Maria Kefalas show in their magisterial Promises I Can Keep, children have inestimable value to the underprivileged. Young people who realistically despair of their chances for a meaningful career or a stable marriage can still find fulfillment in becoming parents. For this reason, disapproval of unwed motherhood is often construed as a prejudicial value judgment. And disapproval has an unseemly history of marginalization, stigma, and ostracism. Of course, in the past, this disapproval helped to keep nonmarital fertility uncommon, but it also made things awful for the few women who did give birth out of wedlock—and at this point the genie is out of the bottle. Moreover, disapproval of unwed motherhood has been lumped in with other forms of prejudice, like racism, that have been rejected by almost everybody. This has been a real problem for American conservatives.

To express vociferous disapproval of single motherhood is to cast aspersions on people who already have many problems.

Indeed, we can’t even say that corrosive forms of disapproval have entirely vanished from our discourse, culture, policy, and governance. To take just one example: aspersions baked into the administration of means-tested benefits. There’s far more scrutiny of apocryphal earned income tax credit fraud than there is of dubious high-end tax write-offs. And consider the awful new movement to drug-test beneficiaries of public aid (though I see a bit of wiggle room here if commensurate treatment opportunities are vastly expanded and aid recipients who test positive aren’t immediately disqualified).

And of course the voices of disapproval continue to be loudest among the very people who least have the interests of the underclass at heart. Conservative politicians have led the charge toward demonizing single mothers, while simultaneously championing economic policies that largely benefit the wealthy (this of course helps explain why single mothers tilt heavily Democratic). Perhaps this orientation was most on display in Ronald Reagan’s infamous denunciation of a supposed “welfare queen.” This level of discourse reduces the credibility of those issuing the denunciations and makes it harder for people to be receptive toward constructive policies aimed at single mothers. In recent years reformist conservatives like Reihan Salam and David Frum have promoted policies that would be more beneficial to all working-class families, but their ideas remain outside the GOP mainstream.

For all of these reasons, I find much of the negative discourse about single mothers immoral. In my mind, accepting them is a stronger moral imperative than condemning them. Condemnation, as it’s been practiced over the past 40 years, hasn’t accomplished anything and has produced considerable ugliness. I freely concede that there’s been unreasonableness about family issues on the political left (the vituperation directed at Daniel Patrick Moynihan remains the best example), but that is in the past for reasonable people. Consider how many thousands of scholars have produced scholarship showing that growing up in non-intact families is bad for kids.

It’s worth reiterating that liberal sentiment is more heterogeneous than is often acknowledged. Most progressives neither advocate nor practice the kind of sexual conduct that sometimes leads to nonmarital fertility. Yet both the left and the right tend to erroneously identify elites—educated, mostly white progressives who live on the coasts or near large universities—as representing all liberal opinion (and even among this group, few are advocating messy personal lives). This formulation leaves out minorities and “Reagan Democrats,” many of whom have more moderate views on single motherhood.

I suspect, too, that conservative animus toward putative liberal hypocrisy about single motherhood is mostly directed at a small and noisy claque of extremists. Of course, this is a favored polemical strategy on both sides of the aisle: find someone on the other side saying something stupid and unrepresentative, and write against them. Needless to say, this doesn’t produce a reasonable or persuasive exchange of ideas. Scholars of unwed motherhood should take this lesson to heart.

Finally, I should acknowledge those fantasy icons of the academic left: the affluent professional woman who has a fatherless child (what were called “Murphy Brown moms” back in the 1990s). Matt McKeever and I have shown that these women represent a tiny fraction of the women who give birth out of wedlock. Never-married mothers who make more money than 90% of women in their demographic, for instance, still have annual incomes several thousand dollars lower than the median American family. I share the right’s contempt for the lefties who try to depict these unicorns as being somehow representative of women who give birth out of wedlock (like this risible book does). But a few of these women do exist, and they require a much different policy discourse.

Let me attempt to summarize. My attitude towards nonmarital fertility occupies a narrow middle. It’s bad for kids, but so too is paternalistic condemnation. For years, this condemnation has been part and parcel of injurious mistreatment directed towards single mothers. Nonmarital childbearing is problematic, but that’s not the same as saying it’s wrong, and attempts to discourage it need to be considerably more sensitive and constructive than has traditionally been the case.

This essay has been about what we shouldn’t do when it comes to single mothers. What would a positive agenda look like? For starters, we need to ensure that opportunity exists in America for everyone. In short, this means access to education, better jobs, and an adequate safely net. This would reduce the prevalence of single motherhood, and make things markedly better for women who do give birth out of wedlock.

What about the cultural front? Prudish conservative objections aside, the success of TV shows like 16 and Pregnant in reducing teen birth rates has been well documented. However, more direct messaging is trickier. Simply telling people to not have children out of wedlock is an act of folly, and often leads to the sort of counterproductive moralizing I dismissed earlier in this article. There may be a place for positive messaging, but it faces an uphill battle—and it should never be used in the place of real assistance for the less privileged Americans.

Perhaps it’s best to worry first about the material lot of less fortunate Americans. Once they’re better off they may be more receptive to suggestions about their personal lives.

Nicholas H. Wolfinger is Professor of Family and Consumer Studies and Adjunct Professor of Sociology at the University of Utah. His new book, Soul Mates: Religion, Sex, Children, and Marriage among African Americans and Latinos, coauthored with W. Bradford Wilcox, is published by Oxford University Press.