The separation of crop types is essential for many agricultural applications, particularly when within-season information is required. Generally, remote sensing may provide timely information with varying accuracy over the growing season, but in small structured agricultural areas, a very high spatial resolution may be needed that exceeds current satellite capabilities. This paper presents an experiment using spectral and textural features of NIR-red-green-blue (NIR-RGB) bands data sets acquired with an unmanned aerial vehicle (UAV). The study area is located in the Swiss Plateau, which has highly fragmented and small structured agricultural fields. The observations took place between May 5 and September 29, 2015 over 11 days. The analyses are based on a random forest (RF) approach, predicting crop separation metrics of all analyzed crops. Three temporal windows of observations based on accumulated growing degree days (AGDD) were identified: an early temporal window (515−1232 AGDD, 5 May−17 June 2015) with an average accuracy (AA) of 70−75%; a mid-season window (1362−2016 AGDD, 25 June−22 July 2015) with an AA of around 80%; and a late window (2626−3238 AGDD, 21 August−29 September 2015) with an AA of <65%. Therefore, crop separation is most promising in the mid-season window, and an additional NIR band increases the accuracy significantly. However, discrimination of winter crops is most effective in the early window, adding further observational requirements to the first window.