The signal-to-noise ratio of optical emission spectroscopy (OES) data has decreased as the plasma etching process has advanced. As a result, not only the advanced endpoint detection method was required, but also the selection of more informative wavelengths. This paper proposes an improved endpoint detection algorithm by combining data-driven wavelength selection and a Gaussian mixture model (GMM). The data-driven wavelength selection algorithm finds the correlation between training data and a sigmoid function of time. Then, using the fitted GMM of the training data in latent space, the endpoint of the test data is determined in real-time. The proposed algorithm’s performance was evaluated using real OES data, comprised of seven operations. The correlation-based wavelength selection algorithm significantly reduced detection error by 70.2% when compared to the conventional method, which selects a few wavelengths manually based on prior knowledge. Additionally, the GMM detection method clustered OES data from low open area wafers much more clearly than the recently proposed method using GMM. This demonstrates that combining correlation-based wavelength selection with GMM is an effective method for detecting endpoints during plasma etching of small open area wafers.