Homophily: The Urban History of an Algorithm

The word “homophily” was coined in a highly-cited 1954 essay by Paul F. Lazarsfeld and Robert K. Merton on friendship in a mixed-race housing project in Pittsburgh, Pennsylvania. The researchers were suspicious of the “familiar and egregiously misleading question: do birds of a feather flock together?”1 They focused on “racial attitudes,” and concluded that friendships form and persist not simply on the basis of shared identities but also thanks to shared values and beliefs.

They imagined two possible social phenomena: “a tendency for friendships to form between those who are alike in some designated respect,” which they termed homophily, and a countervailing pattern, “a tendency for friendships to form between those who differ in some designated respect,” termed heterophily.2 The axes of likeness, they hypothesized, could run along lines of “status” (race, educational level, gender, etc) or “values or opinions.”

The values that Merton and Lazarsfeld studied in Pittsburgh were not randomly chosen. It is not surprising that the issue of integration would be divisive and a deciding factor for friendships in a mixed-race housing project in 1947. Yet it is an irony of history that this highly contextually and historically specific empirical observation has subsequently been taken as a law of human behavior. The irony is further complicated by the fact that Merton and Lazarfeld never actually published the report and data from which their construction of homophily emerged, which would have complicated the simple uptake of homophily.

Homophily is not the only concept central to network science to emerge from mid-twentieth century US housing projects, slum clearance, and planning. Mark Granovetter’s influential notion of “weak ties” emerged in part from a study of the destruction of Boston’s traditional West End via “urban renewal.”3 Jeffrey Travers and Stanley Milgram’s equally important notion of “small worlds” stems from an experiment in which arbitrarily selected people in Nebraska and Boston were asked to direct mail to a Boston stockbroker.4 Behind so many of these abstract concepts are planned communities: wives of married students in MIT Housing; Decatur housewives; residents of Japanese internment camps to name a few.5

This project is thus only the beginning of a longer one needed to unpack, trace, and enliven these connections and specters. Yet homophily is perhaps one of the most urgent places to start. The afterlife of the concept has been remarkable, effectively reconstructing social worlds in its image. Today, the assumption that homophily is a rule also underlies online social and economic interactions, as platforms reinforce the axiom that “similarity breeds connection.” What began as descriptions or questions about social life have become a rule for algorithms shaping social interactions online.

While smart city discourses predict calculable cities using data and maps as engines of change, researching maps and data and their long urban histories can uncover the dangers lurking in what’s called progress. The ties between network science, urban planning, and social engineering are deeply historical, conceptual, and bi-directional. Network science is haunted by the consequences of urban planning, and vice versa—smart cities are just the latest manifestation of this intricate web of influence.

What follows is a series of probes into the archives of Lazarsfeld and Merton to uncover the history of the concept of homophily and its influence on urbanism and network science.6 Their archive is not simply something from the past. It speaks directly to our present, our segregated cities and our polarized platforms, where the effects of research in a housing project now reverberate at much greater scale in networks and networked cities.

Charles “Teenie” Harris, Small parade with boys seated in wagons and girls holding sign inscribed “Dramatic” on dirt road with housing project buildings in background, possibly Bentley Drive, Terrace Village, ca. 1940–1945, Heinz Family Fund, 2001.35.34030, © Carnegie Museum of Art, Pittsburgh, Charles “Teenie” Harris Archive.

Charles “Teenie” Harris, Man feeding pigeons in yard of Terrace Village, with “keep clean” sign, low fence, and children walking on sidewalk, ca. 1950–1965, Heinz Family Fund, 2001.35.27985, © Carnegie Museum of Art, Pittsburgh, Charles “Teenie” Harris Archive.

John R. Shrader, Terrace Village, ca. 1940–1945. Source: Allegheny Conference on Community Development Photographs, 1875–1981, Detre Library & Archives, Heinz History Center, Pittsburgh.

Charles “Teenie” Harris, Small parade with boys seated in wagons and girls holding sign inscribed “Dramatic” on dirt road with housing project buildings in background, possibly Bentley Drive, Terrace Village, ca. 1940–1945, Heinz Family Fund, 2001.35.34030, © Carnegie Museum of Art, Pittsburgh, Charles “Teenie” Harris Archive.

1. Defining Homophily: Raced History

Under the United States Housing Act of 1937, local governments had the authority to decide whether or not to build mixed-race housing. One result of this was that racially-segregated public housing developments were largely the norm.7 However, some social scientists and housing advocates made major efforts to change this pattern of segregation.8

In a 1951 essay commissioned by the Columbia-Lavanburg Studies of the Social Organization of Housing Communities project, prominent progressive housing advocate Catherine Bauer questioned the assumption that cohesive neighborhoods were dependent on “a high degree of social similarity” and echoed the liberal optimism of the time that social science research would help in reversing patterns of segregation.9 She reflected: “throughout the North today, the color line is being seriously challenged, due to the new hope and political prowess of the Negroes and to the pricks of democratic conscience.”10 She argued that “our new housing and planning tools do not inherently produce social segregation. Indeed, they can be used to produce the opposite result quite as effectively.”11 She presented an ambitious agenda for social research that could help counter segregation: “One of the long-standing precepts that has furthered segregation of all kinds … is that property values and neighborhood ‘stability’ depend on social homogeneity.”12 She asked, in effect, what actually makes a neighborhood from a social perspective. Citing an earlier Columbia-Lavanburg Studies research of Addison Terrace in Pittsburgh as a starting point, Bauer asks: “Does it require a high degree of social similarity, or can a strong neighborhood consciousness develop from a heterogeneous group?”13

Robert K. Merton, a sociologist, directed the Columbia-Lavanburg Studies project. Beginning in 1947, Columbia University researchers conducted a survey of residents of Addison Terrace, in Pittsburgh, to try to understand the effects of “tenant morale” on the success of new public housing projects. At the time of the study, the confluence of postwar housing shortages, New Deal era responses to the foreclosure and unemployment crises of the Great Depression, and increasing awareness of urban slums had made public funding for housing a national priority for the first time.14 Addison Terrace was constructed by the new Pittsburgh Housing Authority with federal funding made available through the Housing Act of 1937. Designed by the Pittsburgh firm Marlier, Boyd, Lee, and Prack, the project was dedicated by President Franklin Delano Roosevelt in 1940. It was demolished in 2015.

Merton, together with researchers Patricia Salter West and Marie Jahoda, code-named Addison Terrace in their study “Hilltown.” Hilltown was chosen for study because it was segregated yet mixed-raced, and because it was divided almost evenly between black and white residents. With its electric stoves, hot water, and in-apartment washrooms, it also represented a vast improvement in living standards for all residents.

The Columbia-Lavanburg survey explored the social dynamics of the planned public housing community through residents’ responses to over one hundred different questions. They tabulated patterns of friendship between people with different professions, educational attainments, and places of birth. They inquired about how residents used common spaces of the housing project, and whether they belonged to neighborhood social organizations. They asked about residents’ expectations prior to moving to the housing project, who they associated with most frequently while living there, and what their impressions were of the level of political engagement in their community.

Despite this preliminary range of questions, when its results were finally published in 1954 by Merton and Lazarsfeld, they focused on only two of those questions:

25. Do you think colored and white people should live together in housing projects?
26. On the whole, do you think that colored and white in the Village get along pretty well, or not so well?

The answers to these two questions helped Merton and Lazarsfeld divide the residents of Addison Terrace into three types: “liberals,” who believed that housing projects should be integrated and that people there “got along pretty well”; “illiberals,” who supported segregation and thought that people there failed to get along; and “ambivalent” residents, who believed that “the races should not be allowed to live in the same project” but nonetheless admitted that integration worked in Addison Terrace.15

Visualization of responses by white residents to survey at Addison Terrace.

Visualization of the responses of black residents at Addison Terrace which were omitted from the analysis in Merton and Lazarsfeld, “Friendship as Social Process” (1954).

Roughly 51% of Addison Terrace’s residents and survey participants were black (Merton and Lazarsfeld used the term “Negro,” conventional at the time). Yet the way this population responded to the survey differed greatly from the way the other 49%, the project’s white residents, responded. The white population surveyed gave responses that were distributed between the three categories (34% liberal, 45% ambivalent, and 21% illiberal), while the black population responded as overwhelmingly (88%) liberal. The researchers concluded that, since there were so few ambivalent or illiberal black residents with friends in Hillltown, they could not reliably compare the results. Lazarsfeld and Merton therefore excluded the responses from the black residents. In a footnote, they say: “we must here confine our inquiry to white residents, since there are too few illiberal and ambivalent Negroes with friends in Hilltown to allow comparative analysis.”16 The concept of homophily is therefore haunted, from the beginning, by racial segregation.

The Hilltown survey was not limited to status and opinions—it also asked respondents to name their “three closest friends,” excluding family members and colleagues, casual acquaintances, and neighbors. Combining the specific information about individual friendships with the conclusions about “racial attitudes” from survey questions 25 and 26, Merton and Lazarsfeld were then able to ask whether in fact there were statistically significant patterns of friendship along value lines.

Visualization of friendships and “racial attitudes” among Addison Terrace’s white population.

Among the white residents, Lazarsfeld and Merton concluded that “there are definite patterns of friendship in terms of racial values.”17 Liberals were more likely to be close friends with liberals, and illiberals with illiberals. (The ambivalent residents, who comprised the largest category of white residents, “expressed their uncertain and inconsistent racial values” through friendships with members of all three groups.) In short, the authors said that “value homophily prevails.”18

Scan of research notes from Robert K. Merton Archive showing table of friendships with observed homophily circled.

To arrive at this conclusion, the researchers showed that the relationship between friendship and shared values was not random: of the 34% of white residents categorized as liberal, 45% had friendships with other liberals, while 34% had ambivalent friendships, and only 20% illiberal. This showed that liberals tended to “overselect” other liberals as friends. Likewise, of the 21% categorized as illiberal, 24% said they were friends with other illiberals, 56% with ambivalents, and only 20% with liberals. These disproportionate percentages demonstrated, as well as defined, homophily, despite the fact that they only represent a minority of the friendships observed by the survey, and that in total there are more heterophilous friendships than homophilous ones. And as the authors of the original survey note in their unpublished report, “it is often the absolute number of homophilous relationships, and not their relative proportion, that matters.”19

Explanation of overselection of friendships among Addison Terrace’s white population in Merton and Lazarsfeld, “Friendship as Social Process” (1954).

2. Simplifying Sociality: Formalizing Homophily

Asking only about the residents’ “three closest friends” was a choice that had significant implications. While this limited view of society made the subsequent analysis possible, it fell short of representing the many ways that people share information and develop values within a community. Thus, from the start, observing homophily required a rigid social network—a deliberately engineered notion of social space. This made the concept well suited to formalization, and has allowed homophily to be scaled up and applied widely in today’s digital ecosystems.

Triadic closure, another concept from network science, is a key illustration of this. Like homophily it assumes that friendships form out of harmony and mutual connection. The notion of triadic closure assumes that if person A is linked to both person B and person C, then person B and person C are also likely to be linked. Person B and person C are likely to be friends, in other words, on the basis of their shared friendship with person A. The result of a group formed in this way is a dense network of mutual “friendships.” Facebook, the key example of contemporary social networks, guides this process and expands the network by identifying and then suggesting “friends of friends.”

Regardless of initial conditions, the way a network is designed and engineered can result in radically different structures. When networks are designed so that they increase the likelihood of forming connections across values (a network with a high tolerance), they encourage new connections with others in different groups, allowing integration to be configured and maintained. However, when networks are designed so that there is a low probability of links forming between nodes with different values (a network with low tolerance), they promote connections between those who are already similar, initially integrated configurations give way to segregation.

Yet in networks with both high and low tolerances, triadic closure encourages new links to be formed in formulations that match the existing structures of the network. In cases where networks have a low tolerance and are already segregated, the addition of new relationships based on triadic closure reinforces the existing structure and does little to remedy segregation. In cases where networks have relatively high tolerance and are integrated across values, the addition of new relationships based on triadic closure creates clusters based on existing connections.

Facebook’s algorithms are still a black box. However it is clear that the homogenization of friend recommendations and personal news feeds into echo chambers and filter bubbles is not the only possible outcome. Rather, these phenomena are the result of algorithmic dynamics aided by opaque interfaces, which originate in Merton and Lazarsfeld’s concept of homophily. What if these networks were designed to encourage links between those who are different? Would they display heterophily instead?

Comparison between text written in Merton, Jahoda, and Salter, Pattern of Social Life (1949) and that published in Merton and Lazarsfeld, “Friendship as Social Process” (1954).

3. Methods and Topics: Naturalizing Homophily

Lazarsfeld and Merton’s 1954 essay, where the term “homophily” was publicly introduced, was titled “Friendship as Social Process.” The paper was divided into two parts. The first, written by Merton, contained substantive descriptions of patterns of friendship based on responses to the two questions about racial dynamics in Addison Terrace from the Columbia-Lavanburg survey. The second, written by Lazarsfeld, transformed Merton’s notably limited scenarios into a general model of friendship formation and dissolution, complete with estimates of statistical probability for specific types of friendships between people who either agreed or disagreed on some particular value.20 Together, the two parts defined an abstract principle, a prescriptive or constitutive rule of human interaction, rather than a model of one possible thing that people do among many others.

In the 1954 essay, Merton and Lazarsfeld declare: “it may be assumed, finally, that the near future will see the development of a more comprehensive model incorporating, into one scheme of analysis, the processes giving rise both to status-homophily—the pattern of friendship prevailing between persons occupying similar statuses—and to value-homophily—the pattern of friendship tending to form between persons holding similar values. When this is achieved, sociologists will at long last have met the challenge implicit in Aristotle’s doubts about the adequacy of the adage, ancient by his time, that “birds of a feather flock together.’”21

However, several years earlier, in 1949, Merton, with researchers Jahoda and Salter, drafted a book-length manuscript on the Columbia-Lavanburg survey entitled Pattern of Social Life. In Chapter 7 they define homophily, but in very different terms than the 1954 essay (even as some of the language was taken verbatim and used in the later 1954 essay). As Merton, Salter, and Jahoda define it, homophily is a complex phenomenon, not always present, and largely the result of social forces rather than individual choice.22 They argued that homophily is “not a fixed tendency, it does not spontaneously occur under all manner of social conditions, and it does not disappear only under deliberate coercion. Quite the contrary: the pattern of homophily is the resultant of underlying social and cultural forces, appearing in those special circumstances which foster parochialism and curb a sense of community.”

They then added: “[A]ll this is thoroughly obscured by the millennia-old proverb that takes the special and, in some communities, exceptional, case for the generality. And reaching into the skies for a figure of speech, holds that bird of like feather flock together. But, as we have just seen, for the earthborn, it may be quite otherwise.”23 For Merton, Jahoda, and Salter, homophily was highly contingent, contextual, and only sometimes relevant as a description of how social relationships form and are maintained. Yet this manuscript, this data, and these conclusions were never published.

The turn from homophily as observation in the unpublished manuscript to homophily as structuralist methodology in the 1954 paper is also reflected in the progression of scientific and popular examples that utilize the term. We studied the appearance of the term “homophily” in scholarly papers between 1964 and 2018, and found that it was used with increasing frequency from 1990 onward, with a dramatic increase in the late 2000s. “Homophily” has been the topic of 1,777 academic papers. The fields that make use of this term most frequently are computer science, sociology, behavioral sciences, other social sciences, psychology, and business economics.24

“Homophily” has been the topic of 1,777 academic papers since 1954, most frequently in the fields of computer science, sociology, behavioral sciences, other social sciences, psychology, and business economics.

4.a. Anticipating Homophily: The Self-Fulfilling Prophecy

Two other projects that Merton and Lazarsfeld worked on independently of one another offer insights into aspects of homophily’s afterlife in the digital networks and control systems related to them today.

Illustration of Merton’s 1948 definition of a “Self-Fulfilling prophecy.” 

Drawing on findings from the Columbia-Lavanburg study and before his essay with Lazarsfeld, Merton also coined the phrase “self-fulfilling prophecy” to describe the effects and persistence of racial prejudices. “Many Americans of goodwill are (sometimes reluctantly) brought to retain enduring ethnic and racial prejudices. They experience these beliefs, not as prejudices, not as prejudgements, but as irresistible products of their own observation. ‘The facts of the case’ permit them no other conclusion”25 And yet, Merton argues, they fail to see that they “have produced the very ‘facts’ which [they] observe.”26 The essay provides insights into Merton’s interest in studying and defining homophily: groups exhibiting homophily around particular values (say, racial attitudes) only serve to exacerbate these dynamics, reinforcing each other’s belief in faulty assumptions.

A self-fulfilling prophecy as Merton articulated it is an apt description of some of the most pernicious impacts of the use of homophily in social networks and algorithmic decision-making today. “The self-fulfilling prophecy is, in the beginning, a false definition of the situation evoking a new behavior which makes the originally false conception come true. The serious validity of the self-fulfilling prophecy perpetuates a reign of error.”27

Merton’s solution to this “reign of error” was large-scale institutional change. Describing the survey of Addison Terrace residents, he showed that while many white residents had anticipated that there would be racial tension, the majority felt after living at Addison Terrace that “the races get along fairly well.”28 Thus, he argues that institutions like mixed-race housing can reverse the feelings of animosity and apprehension that lead to racism: “under appropriate institutional and administrative conditions the experience of interracial amity can supplant the fear of interracial conflict.”29 If Addison Terrace had been built following the logic of homophily (that illiberals are most likely to be friends with other illiberals), then the assumption of segregation would likely be replicated and maintained by the institution itself, according to the logic of the self-fulfilling prophecy. But instead, the project was intentionally structured to encourage and model co-existence and encounters across racial lines.

Examples of self-fulfilling prophecies used by Merton in 1948 essay.

Examples of self-fulfilling prophecies used by Merton in 1948 essay.

Present-day examples of self-fulfilling prophecies, showing feedback loop between physical and digital networked spaces.

Present-day examples of self-fulfilling prophecies, showing feedback loop between physical and digital networked spaces.

4.b. Ignoring Ambiguity: The Like Button

Lazarsfeld directed the Princeton Radio Research Project from 1937–1948, an investigation of the effects of electronic media on society. Among the methods of data collection pioneered by the Radio Research Project were “like” and “dislike” buttons that enabled the tabulation of momentary emotional responses from the members of a focus group.

Stanton, Lazarsfeld, and their invention, in 1942. Source: CBS Photo Archive, Getty Images, New York.

Theodor Adorno, Lazarsfeld’s collaborator in the radio project, disputed the notion that culture could be reduced to such a binary and worried about the implicit suggestion of the research itself: that media could be crafted to have specific effects on groups of people.30 Yet as we can see on social networks, homophily is also unknowingly activated today by the clicks of “like” buttons that quantify opinions and emotional reactions. As the results of “likes” are tallied and displayed, they can guide the opinions expressed by others—the self-fulfilling prophecies of algorithmic decision-making today.

The “like” button as we know it, and as it functions today, r­einforces Adorno’s concern about binaries and influence. Despite the like button’s more recent allowance for anger, sadness, or surprise, it still does not allow for nuances of ambiguity or ambivalence, thus increasing the likelihood that recommendation engines will perpetuate homophilic preferences rather than opening up our networks of friendships to people with values different from our own.

5. Producing Polarization: Modeling Homophily

In 1971, the economist Thomas Schelling developed a model to simulate the dynamics of residential segregation. It was an early, influential, and specifically spatial application of the concept of homophily. Schelling represented two races with pennies and nickels arranged randomly on a grid. The assumption of homophily, as well as its racial history, was built into the basic rule of the simulation: individuals were unhappy if they were a racial minority (for example, a penny surrounded by too many nickels), and if an individual was unhappy, they would move to an open square. People move, he assumed, to homophilic locations. The logic seemed inexorable. Playing this out (with the help of many grad students), Schelling showed that even this slight individual preference always resulted in marked segregation.

Schelling originally used pennies and nickels to run simulations through which he argued that segregation could be explained by individual choices.

Schelling used this mathematical analysis as an explanation for “neighborhood tipping,” which was defined as the transition of an area’s population from majority white to majority black.31 Merton’s reminder about “appropriate institutional and administrative conditions” echoes loudly here. For “white flight” in American cities was never simply a matter of unhappy individuals: it was fueled by both the overt and structural racism of real estate speculators and federal housing policies, and the relatively greater economic mobility of white households. These are complex processes and are not easily captured by a mathematical model. In neglecting these forces, Schelling’s model makes the social process of structural inequality disappear and recasts institutional racism as innocent personal preference.

Today the principles of Schelling’s model have been translated from a description of urban neighborhoods to the design of online worlds. Geographic proximity has been replaced by network proximity. As digital platforms, more or less invisibly, use homophily to guide us to people, purchases, destinations, and ideas, they help to produce a social world in which previously held identities and positions are reinforced and concentrated rather than challenged or hybridized.

Conclusion

Let’s return to Addison Terrace, where white and black residents lived together. Lazarsfeld and Merton discovered that people who valued racial integration tended to be friends with each other, as did people who valued segregation. They omit a great deal of data to reach this conclusion. The data from black residents spoke loudly in support of the value and success of integration—but those results were excluded from the analysis, along with the significant percentage of black and white residents who declared that they have friends and/or acquaintances of the other race. Lazarsfeld and Merton included the voice of just a single resident in their published paper, who stands out for her forthright proclamation of an ethical and political decision. “Mrs. Marsh,” whom Merton and Lazarsfeld identify as white and liberal, told the story of inviting some of her black neighbors to dinner. Some of her white neighbors protested, but her voice is clear: “It doesn’t bother me. I just say to them, ‘Well, they’re better friends to me than you,’ and let it go at that… I don’t care what anyone thinks of me, as long as I know I’m doing right.”32

Mrs. Marsh’s story was evidence of value-homophily: her friends, her better friends, were those who shared with her the value of interracial community, “living together” and “getting along,” in the survey’s terms.

It’s a perverse irony of history, and by no means an inevitable one, that the concept to which Mrs. Marsh offered in her testimony has engendered a process that has resulted in the naturalization of the values that she so eloquently rejected. Unpacking the black box of homophily shows that it’s not only math that drives algorithms, but also concepts and “institutional and administrative conditions.” Homophily has turned into a generative formula that segregates cities and polarizes networks, rather than encouraging their integration and internal differentiation. The algorithms can be re-engineered, with higher tolerances and structures that privilege difference and inclusion. Who will have the courage of Mrs. Marsh, not only to state this, but to put it into practice?