Fabric structure parameters have a significant impact on the comfort of heat and moisture transfer in garments. Previous numerical simulations required extensive mathematical calculations and mostly investigated one- or two-dimensional models of fiber assembly without considering the weave structure, which is a key parameter, that significantly influences the porosity of the woven structure and consequently its heat and moisture management. While the finite element method supports the simulation of the optimal shape and material properties with better visibility, previous finite element models focused on heat transfer and neglected water vapor transfer in fabrics. In this article, the finite element simulation of heat and moisture transfer in woven fabrics is established based on the testing principle of thermal resistance and moisture resistance tester using COMSOL Multiphysics software. In this simulation, three-dimensional parametric geometrical models of the fabric are created using curve interpolation methods by acquiring the control point coordinates of different weaves (plain, 2/2 balanced twill, and 4/1 unbalanced twill weaves). Heat and moisture transfer properties of fabric models in the horizontal and vertical directions were analyzed, including the heat flux, moisture resistance, water vapor permeability, and water vapor concentration. The article also deals with the effects of weave structure and fabric cover in a range of 72.1–85.1% on the fabric heat flux and water vapor concentration. Comparison between model and experimental results revealed that the three-dimensional simulation can accurately predict the impact of weave pattern and fabric cover on the fabric heat and moisture transfer performance. In addition, this model can be utilized to study the distribution of heat and water vapor within fabrics, providing a theoretical foundation for optimizing heat and moisture comfort in woven fabrics.