Rapid urbanization and the issue of environmental degradation compel developing countries to find a cleaner approach to replace or complement the traditional on-site construction method. Off-site construction (OSC) with its various advantages has been regarded as a promising approach to address the problems confronted in developing countries. However, promoting the adoption of OSC is indeed a tough task that still faces multiple barriers. It is stakeholder collaboration by integrating diverse resources within the construction industry that can potentially address these barriers. Although multiple studies have been conducted from various perspectives, the issue of stakeholder collaboration, as the key to promoting OSC adoption, has been overlooked. This research aims to explore stakeholders’ influencing power over such barriers using two-mode social network analysis. Firstly, 13 barriers to the OSC adoption, as well as 15 stakeholders with the power to overcome these barriers, were identified based on literature and expert judgments. Subsequently, a questionnaire survey was conducted comprising 39 responses from experts with rich experience in OSC in Chongqing, China. Centrality and core-periphery structure analyses were adopted to study the network. The results show that the government and developers have the highest degree centrality, betweenness centrality, and eigenvector centrality, and thus are the most influential stakeholders in the network. Similarly, the lack of knowledge and expertise, and the dominant conventional project process have the highest values of centrality among all barriers. With the density of interaction being 0.814, core stakeholders and barriers have intensive relationships, and the network shows an apparent core-periphery structure. Theoretically, this study demonstrates the use of the two-mode social network in construction research, and practically, this study sheds lights on how to effectively drive the adoption of OSC in developing countries through promoting stakeholder collaboration.