Prediction of thermal conductivity of natural rock materials using LLE-transformer-lightGBM model for geothermal energy applications

https://ift.tt/6xs4DmV

4.1. Prediction results of the models

The prediction results of different models for the thermal conductivity of rocks are shown in Fig. 5. compares the predictive performance of different models, with the data points of the LLE-Transformer-LightGBM model showing the closest fit to the reference diagonal line, indicating significantly higher predictive accuracy compared to traditional methods (e.g., SVR and BPNN), whose data points exhibit greater dispersion. The tightly clustered data points in the LLE-Transformer-LightGBM model highlight its ability to capture both linear and nonlinear data relationships. In geothermal energy development, such accuracy is critical for optimizing resource extraction efficiency and ensuring the safety of engineering designs. The LLE-Transformer-LightGBM model demonstrates the best performance, exhibiting high prediction accuracy, with most of the training and testing data points closely clustered around the optimal fit line (gray dashed line). To comprehensively evaluate the model, a detailed analysis was conducted on the consistency and discrepancies between the tabular results and the visualized data. This approach facilitates a deeper understanding of the model’s predictive performance and its practical implications. The results in Table 2 (e.g., R² = 0.952, RMSE = 0.086, MAE = 0.0739) quantitatively demonstrate the precision and generalization capability of the LLE-Transformer-LightGBM model. These metrics align closely with the visualization trends in Fig. 5, where the predicted values are tightly clustered along the perfect prediction diagonal. While the overall performance metrics indicate high accuracy, certain differences are observed in specific data subsets. These discrepancies, though not directly reflected in the tabular summary, reveal the model’s limitations in generalization under extreme conditions. Addressing these issues is critical for applications such as geothermal systems, where accurate predictions under extreme compositional conditions are essential. The consistency between tabular metrics and visualization patterns highlights the hybrid model’s strengths in capturing complex relationships and maintaining generalization capability. However, the observed discrepancies also illustrate the limitations of relying solely on numerical summaries to evaluate model performance. Integrating visual analysis helps identify specific data regions that require further attention, such as adding data for underrepresented conditions or optimizing the model to handle extreme scenarios. These insights not only improve the current model but also provide broader guidance for predictive modeling of complex systems. This model showcases outstanding predictive capability among all the evaluated models, reflected by the concentrated distribution of data points. In contrast, the red points in the SVR model are more scattered and deviate significantly from the ideal diagonal line, indicating higher prediction errors. Additionally, the other four intelligent prediction models perform between these two extremes on the test set, showing a certain degree of practical application value. Overall, the LLE-Transformer-LightGBM model is the best-performing model, while the SVR model requires further optimization to improve its generalization performance.

Table 2. Calculation results of model evaluation indicators.

Model	Number of data samples	R²	RMSE	MAE	VAF(%)
LLE-Transformer-LightGBM	79	0.952	0.086	0.0739	95.2
Transformer-LightGBM	79	0.932	0.091	0.0755	92.8
Transformer	79	0.910	0.119	0.0904	91.0
LightGBM	79	0.898	0.123	0.0948	87.2
BPNN	79	0.860	0.148	0.1015	85.4
SVM	79	0.821	0.141	0.0992	82.0
BP (Ting et al., 2021)	6	0.993	0.098	\	\
ANFIS (Singh et al., 2005)	15	\	\	0.0226	\
XRD (Jiajia et al., 2023)	70	0.868	0.25	\	\

Fig. 6 illustrates the error distribution, further emphasizing the robustness of the LLE-Transformer-LightGBM model. The errors for both the training and testing sets are concentrated near zero, indicating strong generalization capability. In contrast, traditional models such as SVR and BPNN exhibit wider error distributions, reflecting their limited ability to generalize to unseen data. In real-world scenarios, such as deep engineering projects, a model with reliable predictive capabilities can significantly reduce the risks of structural instability and failure caused by geological condition variations. The training and testing error distributions of the six machine learning models for predicting the thermal conductivity of rocks are shown in the six graphs. The LLE-Transformer-LightGBM and Transformer-LightGBM models exhibit concentrated error distributions near zero for both the training and testing datasets, indicating high prediction accuracy and strong generalization capabilities. Among these, the LLE-Transformer-LightGBM model demonstrates the best performance of all the models. In comparison, while the Transformer and LightGBM models show relatively balanced error distributions between the training and testing datasets, their errors are noticeably larger than those of the first two models. Furthermore, traditional models such as SVR and BPNN exhibit more scattered absolute errors across both the training and testing datasets, with more significant errors overall.

From Fig. 7, illustrates the relative error distribution for the training and testing sets. The LLE-Transformer-LightGBM model demonstrates the lowest median error and the narrowest error range, indicating exceptional stability. In contrast, traditional models such as SVR and BPNN exhibit greater error fluctuations and significant outliers, reflecting their susceptibility to overfitting and lack of robustness. In a broader scientific context, these findings align with recent research emphasizing the integration of feature engineering and advanced machine learning algorithms to improve the predictive performance of complex systems. it can be observed that in the training set, the LLE-Transformer- LightGBM model exhibits the lowest median error, with a concentrated and narrow error distribution range, indicating its strong fitting ability to the training data. In contrast, the Transformer-LightGBM and Transformer models have similar average errors, but their error distributions are slightly wider than that of the LLE-Transformer-LightGBM model. The LightGBM model shows a larger error distribution range, with average errors higher than the aforementioned three models. The BPNN and SVR models have the largest error ranges and more dispersed distributions, with significant outliers, demonstrating weaker fitting ability to the training data.

In the testing set, the LLE-Transformer-LightGBM model also demonstrates excellent generalization ability, with the smallest error range and lowest median error. The Transformer-LightGBM and Transformer models follow, with slightly wider error distributions, indicating their prediction accuracy on the testing set is slightly inferior to the best-performing model. The LightGBM model exhibits an even larger error range on the testing set compared to its performance on the training set. The other models, particularly the BPNN and SVR models, show significantly expanded error ranges and more dispersed distributions on the testing set, highlighting their poor generalization ability.

Therefore, the LLE-Transformer-LightGBM model achieves the lowest median error and the smallest error range on both the training and testing sets, indicating that it not only has strong fitting ability but also superior generalization performance compared to other models. These results suggest that the LLE-Transformer-LightGBM model is capable of more accurately predicting rock thermal conductivity.

4.2. Comparison of predictive performance

The study employed various evaluation metrics to compare the performance of machine learning models, including the coefficient of determination (R²), root mean square error (RMSE), mean absolute error (MAE), and variance accounted for (VAF). R² reflects the goodness of fit between predicted and actual values, with values closer to 1 indicating stronger predictive ability; RMSE represents the square root of the mean squared prediction error, with lower values indicating higher prediction accuracy; MAE directly reflects prediction bias through the mean absolute error; and VAF evaluates the proportion of variance in actual values explained by the predictions, with higher values indicating better performance (Lin et al., 2025, Xie et al., 2024b, Xie et al., 2024c, Wang et al., 2025). Table 2 presents the specific calculation results for these metrics across models, clearly demonstrating differences in prediction accuracy and error control.

Based on the results in Table 2, Fig. 8 visualizes the evaluation metrics for each model, making their strengths and weaknesses more intuitive. A comprehensive analysis of these metrics aids in accurately identifying the best-performing model and provides a scientific basis for model selection and optimization. Through this multidimensional quantitative comparison approach, the study offers significant insights for improving machine learning models and applying them in practice.

The comparison between Table 2 and Fig. 8 reveals significant differences in the performance metrics of various models. The LLE-Transformer-LightGBM model stands out as the best performer, achieving an R² of 0.952, a variance accounted for (VAF) of 95.2 %, and the lowest RMSE and MAE values of 0.086 and 0.0739, respectively. This indicates superior predictive accuracy and error control. In contrast, the Transformer-LightGBM model ranks second with an R² of 0.932 and a VAF of 92.8 %, though its error metrics are slightly higher.

The Transformer model and the LightGBM model show relatively close performance, with R² values of 0.910 and 0.898 and VAFs of 91.0 % and 87.2 %, respectively. Although their performance is not as strong as the top two models, their error metrics remain within acceptable ranges and may have specific advantages in certain applications. In contrast, the traditional BPNN and SVM models demonstrate significantly weaker performance, with R² values of 0.860 and 0.821 and higher error metrics, highlighting their limitations in handling complex prediction tasks.

The study evaluated the computational efficiency of the LLE-Transformer-LightGBM model by comparing its performance with BPNN and SVR. The analysis focused on training and prediction speed. The training time for the LLE-Transformer-LightGBM model is approximately 1.2 times longer than BPNN and 1.6 times longer than SVR, primarily due to its additional feature extraction and deep learning stages. However, its prediction speed is significantly faster, being 1.8 times faster than BPNN and 2 times faster than SVR. This demonstrates that while the hybrid model requires slightly more time during training, it offers superior efficiency in real-time prediction scenarios.

In summary, hybrid models (LLE-Transformer-LightGBM and Transformer- LightGBM) outperform single models significantly. This demonstrates that combining feature engineering with powerful machine learning algorithms can greatly enhance predictive accuracy and error control. These findings provide valuable references for practical model selection and suggest directions for further optimization.

The proposed LLE-Transformer-LightGBM model achieved significant improvements in accuracy and interpretability in the prediction of rock thermal conductivity through its hybrid approach. Comparisons with previous studies further clarified these advancements and highlighted the contributions of this research. Traditional models, such as linear regression and empirical formulas, often struggle to capture the inherent nonlinear relationships in thermal conductivity prediction. For example, Cho and Kwon (2010a) used empirical methods to study partially saturated granite, but their model’s accuracy was limited due to its inability to handle complex data structures. In contrast, the LLE-Transformer-LightGBM model achieved an R² of 0.952 and an RMSE of 0.086, demonstrating its capability to effectively address both linear and nonlinear dependencies. Machine learning models like Support Vector Regression (SVR) and Random Forest (RF) have also been explored for predicting thermal properties. Zhao et al. (2024), for instance, used SVR to analyze the thermal conductivity of granite, reporting an R² below 0.9. Similarly, RF models showed potential in sensitivity analysis but were prone to overfitting and lacked scalability. By integrating dimensionality reduction (LLE), feature extraction (Transformer), and efficient regression (LightGBM), the proposed model overcame these limitations, demonstrating superior generalization ability and robustness. Unlike many "black-box" machine learning models, this study emphasized interpretability through feature importance analysis. For instance, the inverse decomposition method quantified the contributions of original features during dimensionality reduction, revealing the importance of factors such as SiO₂ and surface distance. This approach aligns with Guo et al. (2020), who underscored the necessity of feature-level explanations in predicting thermal properties. However, this research extended interpretability through a more advanced predictive framework. Compared with other studies on geothermal applications (Zhang et al., 2024), this model directly addressed the prediction of thermal conductivity under varying geological conditions. This specificity enables more precise thermal management and material selection in practical applications, marking a significant advancement in the field. By addressing and extending the limitations of previous research, the LLE-Transformer-LightGBM model offers a comprehensive solution that not only achieves high prediction accuracy but also ensures the interpretability and adaptability of complex geological datasets. These comparisons underscore the relevance and innovation of this study in the field of thermal property prediction.

4.3. Inverse decomposition method for solving feature importance

For a composite prediction model with feature dimensionality reduction, the quantification of feature importance requires the use of the inverse decomposition method to map the importance calculation results back to the original features.

First, obtain the weight matrix from the LLE to determine the contribution of each high-dimensional input feature to the reduced-dimensional variables. Then, decompose the LLE weight matrix onto each original input feature to calculate the linear contribution proportion of each original feature to the reduced-dimensional variables.(15)Where is the j-th original feature; is the linear contribution weight of the j-th feature to the i-th variable after dimensionality reduction; d is the total number of original features.

Subsequently, extract the feature importance of the three reduced-dimensional variables from the trained LightGBM model using split gain information. The overall importance of each original feature can then be determined through regression-based allocation.(16)Where k is the total number of features after dimensionality reduction; represents the contribution weight of the original features to the dimensionality reduction variable ; is the importance of features in LightGBM features after dimensionality reduction.

As shown in Fig. 9, distance from the surface exhibited the highest feature importance, contributing 13.87 % to the prediction of thermal conductivity. This result aligns with geological expectations, as surface distance is directly correlated with environmental conditions such as temperature gradients and pressure. These factors significantly affect the thermal properties of rocks, especially in deep engineering applications like geothermal systems and underground storage.

The contributions of SiO₂ and CaO ranked second and third, with 9.24 % and 8.32 %, respectively. SiO₂, as a primary constituent of most rock types, plays a critical role in determining their thermal behavior due to its intrinsic thermal conductivity and abundance. Similarly, CaO influences the mineralogical composition and microstructure of rocks, indirectly affecting their heat transfer capabilities. These findings highlight the necessity of considering chemical compositions in thermal conductivity modeling.

MgO and MnO showed moderate contributions of 7.76 % and 7.53 %, respectively. While these oxides are present in smaller proportions compared to SiO₂ and CaO, they can still influence the thermal properties of rocks, particularly in specific geological conditions. MgO, for instance, is associated with certain high-temperature mineral phases, while MnO might affect the thermal characteristics through its role in microstructural changes.

In specific geological contexts, the thermal conductivity of rocks displays unique patterns influenced by varying factors, such as porosity, mineral composition, and environmental conditions. These context-specific patterns are crucial for accurate predictions. By applying techniques such as feature dimensionality reduction and model optimization, these patterns can be effectively identified and integrated into the predictive framework. This approach enhances the model’s accuracy and ensures its adaptability to diverse geological scenarios. Lower-ranked features, such as P₂O₅, Na₂O, and TiO₂, demonstrated relatively minor contributions to the prediction model. These features might have limited direct effects on thermal conductivity but could serve as secondary indicators under specific circumstances. For example, TiO₂ could influence the thermal behavior of certain mineral phases, while P₂O₅ may reflect compositional variability in unique geological formations.

This feature importance analysis underscores the necessity of integrating geological, chemical, and environmental factors in predictive models of thermal conductivity. The high importance of surface distance and major chemical components like SiO₂ and CaO indicates that both macroscopic and microscopic properties must be considered. Additionally, minor features should not be disregarded, as they may provide crucial insights in specialized scenarios. Such a holistic approach ensures a more robust and accurate prediction framework.

4.4. Discussion

4.4.1. Discussion on the Applicability of LLE

Using LLE as a preprocessing step significantly enhanced the model’s predictive performance. As shown in Fig. 10, after applying LLE, the features exhibit a more concentrated distribution pattern in the reduced-dimensional space, with significantly reduced overlap between features and a more compact arrangement. This reflects the advantage of LLE in capturing local structures. Overall, LLE not only improved the spatial distribution of features but also laid a solid foundation for the improvement in model performance. Consequently, the R² increased from 0.874 to 0.952, and the RMSE decreased from 0.118 to 0.086, fully demonstrating the practical value of LLE as a preprocessing step.

In this study, the differences between LLE and two alternative dimensionality reduction techniques—Principal Component Analysis (PCA) and t-distributed Stochastic Neighbor Embedding (t-SNE)—were explored to evaluate their effectiveness in feature representation. PCA, as a linear dimensionality reduction method, projects data onto orthogonal axes to maximize variance, demonstrating high computational efficiency and low computational cost. However, its ability to handle nonlinear relationships in high-dimensional data is limited. Experimental results showed that while PCA retained 85 % of variance within the first three principal components, it failed to capture the subtle nonlinear structures critical for predicting thermal conductivity.In contrast, t-SNE performed exceptionally well in visualizing high-dimensional data, particularly in emphasizing local structures. However, its inherent randomness and high computational cost limit its applicability in regression tasks, such as predicting thermal conductivity. Although t-SNE generated intuitive clustering results in experiments, it struggled to consistently reproduce feature representations across multiple runs.Against this backdrop, LLE demonstrated significant advantages by preserving nonlinear structures while maintaining stability and interpretability. LLE effectively revealed the critical roles of surface distance and chemical composition in influencing thermal conductivity. Overall, while PCA and t-SNE have their strengths in specific applications, LLE outperforms them by balancing nonlinear pattern preservation, consistency, and computational efficiency, making it the most suitable dimensionality reduction technique for this study.

4.4.2. Limitations of the hybrid prediction model

Although the LLE-Transformer-LightGBM model demonstrates significant advantages in predictive accuracy and generalization capability, it has limitations in feature interpretability. The combination of LLE and Transformer enhances feature extraction and dimensionality reduction, but the generated features often lack direct physical interpretability. In fields where a clear understanding of the relationship between input features and prediction outcomes is crucial (e.g., safety-critical engineering applications), this limitation could affect the practical value of the model. The abstract nature of the extracted features may pose challenges to decision-making based on domain knowledge, potentially reducing trust and transparency in the model’s results. Future research should focus on integrating domain knowledge with the model’s feature extraction process to improve its credibility and applicability in practical geological engineering scenarios. Specifically, for the abstract features generated by LLE and Transformer, incorporating geological constraints into the dimensionality reduction process—such as embedding prior knowledge of physical variables like mineral composition, porosity, and thermal conductivity—can provide the features with clearer physical significance. Additionally, future studies should emphasize cross-validation using multi-source data, such as combining laboratory measurements with in-situ field data, to verify the consistency and stability of the extracted features across different geological settings. Moreover, the development of visualization-based feature mapping tools could allow reduced-dimensional features to be traced back to their original physical variables, offering an intuitive interpretative framework for domain experts. These improvements would enhance the scientific rigor and transparency of the model, providing more reliable support for modeling the thermal properties of rocks. They would also further expand the model’s potential for engineering applications in complex geological environments.

February 11, 2025 at 07:45PM
https://ift.tt/ExXg5sY