The development of prefabricated buildings is a crucial technical pathway for the construction industry to transition towards green and low-carbon practices. It represents an effective approach to achieving “carbon peak” and “carbon neutrality” in the construction sector. In order to further advance the progress of prefabricated buildings in Ji’an city and facilitate the shift towards green and low-carbon construction, 17 key driving factors have been identified through questionnaire surveys and other methods based on the current status of prefabricated building development in Ji’an city. Utilizing the ISM model, this study examines the influence mechanism and hierarchical relationship among these driving factors, analyzing surface, middle, and deep driving factors. The research reveals that deep driving factors for prefabricated building development in Ji’an city primarily stem from government perspectives such as high resource utilization rate, as well as developer perspectives including component factory production and key technology innovation.