When we first moved to the small, white, working-class town in southwest Ohio where we interviewed young adults for the Love and Marriage in Middle America Project, my wife and I felt more scared living there than we ever did in Queens. In this town that was supposed to be the epitome of charming, small-town America, locals warned us about sex offenders and druggies and rapists; outsiders warned us about the KKK, fights on the streets, and grinding poverty.

The place was dirt poor, people told us, and had been at least since Appalachian migrants started settling into the neighborhood in the mid-twentieth century. The town in the valley was the “devil’s thumbprint,” as one interviewee who grew up and still lived there characterized it, and the only way to escape it was to go up the hill and get out.

But the narrative of poverty collided with the facts about the place. For instance, as far back as 1960, its median household income tracked closely with the national median. By 2000, at just under $36,000, the median household income had dropped below the national median (about $42,000)—reflecting the falling fortunes of the working class in America overall, but still not the profile of a town gripped by poverty.

The tales of violence and chaos that we heard also collided with the reality that we encountered in the town. The tidy homes that defined the neighborhood were much smaller than the new McMansions on the hill, but most residents (many of them homeowners) kept their lawns mowed and their properties neat. Admittedly, there were signs that not all was well, like the police blotter documenting a steady stream of burglaries and drug possession charges, the vacant home that had once been a meth lab, a few men loitering around the homeless shelter. Most neighbors, however, were kind and affable.

The street on which we lived makes the point. Most of our neighbors were homeowners who either had retired or worked full-time: a small business owner, a few factory workers, a Postal Service worker, a mechanic, an accountant, a member of the armed forces, a custodian, to name just some of them. When a college-educated friend from a nearby subdivision found out about this, she was shocked: she had assumed that everyone on our street was dependent on welfare. She was also shocked to learn that only about one-third of the American adult population has a four-year college degree. In other words, it was news to our friend that our mostly non-college-educated neighbors were not a struggling underclass but the norm in America.

The narrative about our working-class community collided with the facts.

As our friend’s mistake illustrates, the perception of the town’s residents as “poor white trash” persists, and the stigma affects the confidence of the residents themselves. It is particularly damaging in high school, where kids from the “poor” neighborhood are far outnumbered by the kids from the middle- and upper-middle-class subdivisions (the school district is relatively affluent). Cassie, 25, remembers standing in high school choir, warming up at a concert, when a classmate told her to “Go back to your box on the side of the highway. No one wants you.” Traumatized, she ran out of the classroom crying. Her father, who made almost $20 an hour as a truck driver—and that doesn’t even count his wife’s income as an accounts payable clerk for a Fortune 500 company—was so angry about the incident that he stormed into the principal’s office demanding to meet the bully. When Cassie’s father met him, he delivered a lesson in simple living: after telling him about how much money he made, he said “The only reason we don’t spend the money like you and your mommy do is because we don’t want to. We live better than that.”

Alex, 23, who also grew up in the working-class valley, remembers his hometown as an idyllic Mayberry, where the church bell rang every day and kind old ladies greeted each other on front porches. He and his friends thought they had it all: they could walk to the local elementary school in the morning and in the afternoon ride their bikes all over town until dinnertime. When he entered junior high, though, and mingled for the first time with kids from the more affluent neighborhoods, he discovered something: “We were poor, apparently.” “I started learning that there was different economic levels of prosperity,” he said. Suddenly, juxtaposed to the new subdivisions of his classmates, the modest one-story home that Alex’s factory-working grandparents owned was a sign of poverty—and the kids were relentless in letting Alex know about it.

Alex’s grandparents were not poor; Cassie’s parents were not poor. But in an affluent era, the working class becomes increasingly invisible; they are reduced to “the poor,” because compared to the expanding and increasingly affluent class of college-educated Americans, that’s what they look like. Michael Harrington’s descriptions of the “invisible poor” in his 1962 classic The Other America rattled Americans who were proud of their broadly shared prosperity. But in our era of growing inequality and socioeconomic segregation, it’s becoming easier to conflate the working class with the poor—and it’s the working class who risk becoming invisible.

For a social scientist, it’s convenient to describe America in terms of thirds: the top third with a four-year college degree or more are “rich,” the middle third with some postsecondary education are in the “middle,” and the bottom third with no more than a high school education are the “poor.” That’s an understandable categorization for a social scientist trying to describe broad trends. But in real life, we don’t walk around with our educational attainment branded on our foreheads, nor do we always fit those broad trends.

For instance, in Cassie and Alex’s town, the person with a cosmetology license lives next to the small business owner with no more than a high school education, who lives next to the unemployed young man who dropped out of high school. In this example, it’s only the unemployed young man who is truly materially poor—but because all three individuals live in a town stigmatized (wrongly) by the more affluent residents as a hotbed of poverty, the cosmetologist and the small business owner are also branded as “poor” people.

In our era of growing inequality and socioeconomic segregation, it’s the working class who risk becoming invisible.

The falling economic and family fortunes of the working class are, indeed, serious problems that require our collective attention. Yet framing the class as poor makes it easier for us to imagine its members as a problem to solve, rather than as friends and neighbors and co-workers with complementary strengths and gifts to offer society. It’s easier to label the “other” as “poor”—and therefore suffering from a problem—than it is to acknowledge the “working class” and to respect legitimate cultural differences and preferences that that term implies (for instance, a preference for blue-collar work and building a family in the years immediately after high school instead of attending college and delaying family formation).

Our friends and neighbors who are truly materially poor bear a heavy stigma: they face the judgment of the non-poor who wonder why they “can’t seem to get their acts together,” especially in an age of fabulous wealth for some. How we think and talk about poverty is its own challenge that we need to address—along with the reality that many working-class young adults whose parents and grandparents were solidly working- and middle-class are finding it increasingly difficult to achieve the same stability, both at work and in their families.

Instead of branding people like Cassie and Alex with the stigma of poverty, we should recognize that many Americans without college degrees are the heirs of a particular American sensibility: that you can work hard, raise a family, live in a modest home, and live a good life even if you never get rich. Many of the working-class people we know still bear that sensibility with pride. That’s not a mindset or preference that needs to be fixed; if anything, many of us could learn from it.