Cleaning Extremely Noisy Soil Spectroscopy Data

Two-Criterion for Outlier Removal

We have been diving deep into soil spectroscopy to predict key soil properties such as organic carbon, total carbon, nitrogen, clay, sand, silt, and bulk density. We have built a complete workflow to clean, preprocess, and model these highly complex datasets.

We performed PCA and calculated Mahalanobis distances. All spectra were preprocessed using: SNV and 2nd derivative Savitzky–Golay (window=11, polyorder=2). Because of the exceptionally noisy nature of the dataset, we applied two thresholds:

Criterion 1: Automatic Statistical Threshold

The first criterion, removed samples whoseMahalanobis distance exceeded the mean plus three times the standard deviation. This approach captured most outliers based on the distribution of the dataset. It adapted to the dataset’s variability and identified statistically extreme spectra.

Criterion 2: Conservative Hard Threshold

The second criterion, a threshold with a maximum Mahalanobis distance of 26, acted as a safety net for extremely noisy datasets.

In noisy datasets, extreme outliers inflated the mean and standard deviation, allowing some abnormal samples to pass the statistical filter. While a hard threshold worked well to cut extreme values, it was less flexible: it could remove samples that would have been statistically normal in a clean dataset, and it did not adapt to the real variability of the data. A fixed limit of 30 was sometimes too strict in a dataset with little variation or too relaxed if the dataset had a different scale.

In clean datasets, a single criterion might have been sufficient. However, in extremely noisy datasets, the first filter (mean + 3×std) captured most outliers adaptively based on the dataset’s statistics, while the second filter (≤30) ensured that any extreme outliers that escaped the first filter were removed, especially in datasets with long-tailed distributions. Without both filters, the model could have retained physically implausible or highly noisy spectra, compromising robustness and predictive accuracy.

In conclusion, for noisy datasets like this one, both criteria were complementary, not redundant: one was adaptive and statistical, the other conservative and fixed, ensuring that only valid spectra were retained for modeling.

Dataset: https://lnkd.in/eX68uCWz