Construction workers typically undertake highly demanding physical tasks involving various types of stresses from awkward postures, using excessive force, highly repetitive actions, and excessive energy expenditure, which increases the likelihood of unsafe actions, productivity loss, and human errors. Biomechanical models have been developed to estimate joint loadings, which can help avoid strenuous physical exertion, potentially enhancing construction workforce productivity, safety, and well-being. However, the models used are mainly in 2D, or to predict static strength ignored their velocity and acceleration or using marker-based method for dynamic motion data collection. To address this issue, this paper proposes a novel framework for investigating the mechanical energy expenditure (MEE) of workers using a 3D biomechanical model based on computer vision-based techniques. Human 3D Pose Estimation algorithm based on 2D videos is applied to approximate the coordinates of human joints for working postures, and smart insoles are used to collect foot pressures and plantar accelerations, as input data for the biomechanical analyses. The results show a detailed MEE rate for the whole body, at which joints the maximum and minimum values were obtained to avoid excessive physical exertion. The proposed method can approximate the total daily MEE of construction tasks by summing the assumed cost of individual tasks (such as walking, lifting, and stooping), providing suggestions for the design of a daily workload that workers can sustain without developing cumulative fatigue.