The interplay of lattice, electronic, and spin degrees of freedom at epitaxial complex oxide interfaces provides a route to tune their magnetic ground states. Unraveling the competing contributions is critical for tuning their functional properties. We investigate the relationship between magnetic ordering and magnetic anisotropy and the lattice symmetry, oxygen content, and film thickness in compressively strained LaMnO$_3$/LaCrO$_3$ superlattices. Mn-O-Cr antiferromagnetic superexchange interactions across the heterointerface resulting in a net ferrimagnetic magnetic structure. Bulk magnetometry measurements reveal isotropic in-plane magnetism for as-grown oxygen-deficient thinner thin samples due to equal fractions of orthorhombic a+a-c-, and a-a+c- twin domains. As the superlattice thickness is increased, in-plane magnetic anisotropy emerges as the fraction of the a+a-c- domain increases. On annealing in oxygen, the suppression of oxygen vacancies results in a contraction of the lattice volume, and an orthorhombic to rhombohedral transition leads to isotropic magnetism independent of the film thickness. The complex interactions are investigated using high-resolution synchrotron diffraction and X-ray absorption spectroscopy. These results highlight the role of the evolution of structural domains with film thickness, interfacial spin interactions, and oxygen-vacancy-induced structural phase transitions in tuning the magnetic properties of complex oxide heterostructures.