Abstract Background Our objectives were first to determine the optimal coronary computed tomography angiography (CTA) protocol for the quantification and detection of simulated coronary artery cross-sectional area (CSA) differences in vitro, and secondly to quantitatively compare the performance of the optimized CTA protocol with a previously validated radial coronary cardiovascular magnetic resonance (CMR) technique. Methods 256-multidetector CTA and radial coronary CMR were used to obtain images of a custom in vitro resolution phantom simulating a range of physiological responses of coronary arteries to stress. CSAs were automatically quantified and compared with known nominal values to determine the accuracy, precision, signal-to-noise ratio (SNR), and circularity of CSA measurements, as well as the limit of detection (LOD) of CSA differences. Various iodine concentrations, radiation dose levels, tube potentials, and iterative image reconstruction algorithms (ASiR-V) were investigated to determine the optimal CTA protocol. The performance of the optimized CTA protocol was then compared with a radial coronary CMR method previously developed for endothelial function assessment under both static and moving conditions. Results The iodine concentration, dose level, tube potential, and reconstruction algorithm all had significant effects (all p < 0.001) on the accuracy, precision, LOD, SNR, and circularity of CSA measurements with CTA. The best precision, LOD, SNR, and circularity with CTA were achieved with 6% iodine, 20 mGy, 100 kVp, and 90% ASiR-V. Compared with the optimized CTA protocol under static conditions, radial coronary CMR was less accurate (− 0.91 ± 0.13 mm2 vs. -0.35 ± 0.04 mm2, p < 0.001), but more precise (0.08 ± 0.02 mm2 vs. 0.21 ± 0.02 mm2, p < 0.001), and enabled the detection of significantly smaller CSA differences (0.16 ± 0.06 mm2 vs. 0.52 ± 0.04 mm2; p < 0.001; corresponding to CSA percentage differences of 2.3 ± 0.8% vs. 7.4 ± 0.6% for a 3-mm baseline diameter). The same results held true under moving conditions as CSA measurements with CMR were less affected by motion. Conclusions Radial coronary CMR was more precise and outperformed CTA for the specific task of detecting small CSA differences in vitro, and was able to reliably identify CSA changes an order of magnitude smaller than those reported for healthy physiological vasomotor responses of proximal coronary arteries. However, CTA yielded more accurate CSA measurements, which may prove useful in other clinical scenarios, such as coronary artery stenosis assessment.