Modified Newton’s method for solving parametric ν-support vector regression with Universum data

Bazikar, Fatemeh; Sadeghi Bigham, Bahram; Hemmati, Atefeh

doi:10.22034/cmde.2025.66530.3113

Modified Newton’s method for solving parametric ν-support vector regression with Universum data

Document Type : Research Paper

Authors

¹ Department of Computer Science, Faculty of Mathematical Sciences, Alzahra University, Tehran, Iran.

² Department of Computer Engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran.

10.22034/cmde.2025.66530.3113

Abstract

Universum, representing a third category distinct from the two primary classes in classification tasks, facilitates the incorporation of prior knowledge into the learning process. Extensive studies have confirmed its effectiveness in improving both supervised and semi-supervised classification models. Recently, Universum data has been introduced into parametric $\nu$-support vector regression (UPar-$\nu$-SVR) to enhance generalization performance. In this paper, we present a Newton-based method for solving UPar-$\nu$-SVR, with the objective of further improving its efficiency and accuracy. Our approach reformulates the problem into an unconstrained convex optimization framework and employs a generalized Newton’s method for its solution. To assess the effectiveness of our proposed method, we conduct comprehensive experiments on multiple UCI benchmark data sets. The experimental results indicate that our algorithm outperforms existing techniques, providing superior generalization capabilities and computational efficiency.

Keywords

Main Subjects

Computer aspects of numerical algorithms

References

[1] Z. Arabasadi, R. Alizadehsani, M. Roshanzamir, H. Moosaei, and A. A. Yarifard, Computer aided decision making for heart disease detection using hybrid neural network-Genetic algorithm, Computer Methods and Programs in Biomedicine, 141 (2017), 19–26.
[2] N. Ayoobi et al., Time series forecasting of new cases and new deaths rate for COVID-19 using deep learning methods, Results in Physics, 27 (2021), 104495.
[3] F. Bazikar, S. Ketabchi, and H. Moosaei, DC programming and DCA for parametric-margin ν-support vector machine, Applied Intelligence, 50 (2020), 1763–1774.
[4] O. Chapelle, A. Agarwal, F. Sinz, and B. Schölkopf, An analysis of inference with the universum, Advances in Neural Information Processing Systems, 20 (2007).
[5] F.H. Clarke, Optimization and Nonsmooth Analysis, SIAM, (1990).
[6] S. Ding, N. Zhang, X. Zhang, and F. Wu, Twin support vector machine: theory, algorithm and applications, Neural Computing and Applications, 28 (2017), 3119–3130.
[7] M. H. Doroudyan and S. T. A. Niaki, Pattern recognition in financial surveillance with the ARMA-GARCH time series model using support vector machine, Expert Systems with Applications, 182 (2021), 115334.
[8] D. Dua and C. Graff, UCI machine learning repository, (2019), URL https://archive.ics.uci.edu/ml.
[9] M. Ganaie, M. Tanveer, and J. Jangir, EEG signal classification via pinball universum twin support vector ma- chine, Annals of Operations Research, 328(1) (2023), 451–492.
[10] M. Ganaie, M. Tanveer, and A. s. D. N. Initiative, KNN weighted reduced universum twin SVM for class imbalance learning, Knowledge-Based Systems, 245(2022), 108578.
[11] D. Gupta, H. J. Sarma, K. Mishra, and M. Prasad, Regularized Universum twin support vector machine for classification of EEG Signal, in 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), IEEE, (2019), 2298–2304.
[12] P.-Y. Hao, New support vector algorithms with parametric insensitive/margin model, Neural Networks, 23(1) (2010), 60–73.
[13] B. Hazarika, D. Gupta, and B. Kumar, EEG signal classification using a novel universum-based twin parametric- margin support vector machine, Cognitive Computation, 16(4) (2024), 2047–2062.
[14] C.W. Hsu, C.C. Chang, C.J. Lin, et al., A practical guide to support vector classification, (2003), URL: https://www.csie.ntu.edu.tw/ cjlin/papers/guide/guide.pdf
[15] S. H. Javadi, H. Moosaei, and D. Ciuonzo, Learning wireless sensor networks for source localization, Sensors, 19(3) (2019), 635.
[16] W. Jianlin, F. Xuying, and Y. Tao, A geometric approach to support vector regression and its application to fermentation process fast modeling, Chinese Journal of Chemical Engineering, 20(4) (2012), 715–722.
[17] S. Ketabchi, H. Moosaei, M. Razzaghi, and P. M. Pardalos, An improvement on parametric ν-support vector algorithm for classification, Annals of Operations Research, 276(1) (2019), 155–168.
[18] R. Khemchandani and S. Chandra, Twin support vector machines for pattern classification, IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(5) (2007), 905–910.
[19] M. A. Kumar and M. Gopal, Least squares twin support vector machines for pattern classification, Expert Systems with Applications, 36(4) (2009), 7535–7543.
[20] B. Kumar and D. Gupta, Universum based Lagrangian twin bounded support vector machine to classify EEG signals, Computer Methods and Programs in Biomedicine, 208 (2021), 106244.
[21] B. Liu et al., Adaptive robust Adaboost-based twin support vector machine with universum data, Information Sciences, 609 (2022), 1334–1352.
[22] O. L. Mangasarian and E. W. Wild, Multisurface proximal support vector machine classification via generalized eigenvalues, IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(1) (2005), 69–74.
[23] B. Mei and Y. Xu, Multi-task least squares twin support vector machine for classification, Neurocomputing, 338 (2019), 26–33.
[24] H. Moosaei, S. Ketabchi, M. Razzaghi, and M. Tanveer, Generalized twin support vector machines, Neural Processing Letters, 53(2) (2021), 1545–1564.
[25] H. Moosaei and M. Hladík, A lagrangian-based approach for universum twin bounded support vector machine with its applications, Annals of Mathematics and Artificial Intelligence, 91(2) (2023), 109–131.
[26] H. Moosaei, F. Bazikar, and M. Hladík, Universum parametric ν-support vector regression for binary classification problems with its applications, Annals of Operations Research, (2023), 1–45.
[27] H. Moosaei, F. Bazikar, S. Ketabchi, and M. Hladík, Universum parametric-margin ν-support vector machine for classification using the difference of convex functions algorithm, Applied Intelligence, 52(3) (2022), 2634–2654.
[28] H. Moosaei, M. Ganaie, M. Hladík, and M. Tanveer, Inverse free reduced universum twin support vector machine for imbalanced data classification, Neural Networks, 157 (2023), 125–135.
[29] W. S. Noble, Support vector machine applications in computational biology, in Kernel Methods in Computational Biology, (2004), 71.
[30] X. Peng, Least squares twin support vector hypersphere (LS-TSVH) for pattern recognition, Expert Systems with Applications, 37(12) (2010), 8371–8378.
[31] X. Peng, A ν-twin support vector machine (ν-TSVM) classifier and its geometric algorithms, Information Sciences,180(20) (2010), 3863–3875.
[32] X. Peng, TPMSVM: a novel twin parametric-margin support vector machine for pattern recognition, Pattern Recognition, 44(10–11) (2011), 2678–2692.
[33] Z. Qi, Y. Tian, and Y. Shi, Twin support vector machine with universum data, Neural Networks, 36 (2012), 112–119.
[34] B. Richhariya and M. Tanveer, EEG signal classification using universum support vector machine, Expert Systems with Applications, 106(2018), 169–182.
[35] B. Richhariya, M. Tanveer, and A. S. D. N. Initiative, A fuzzy universum least squares twin support vector machine (FULSTSVM), Neural Computing and Applications, 34(14) (2022), 11411–11422.
[36] B. Richhariya and M. Tanveer, Universum least squares twin parametric-margin support vector machine, in 2020 International Joint Conference on Neural Networks (IJCNN), IEEE, (2020), 1–8.
[37] B. Richhariya, M. Tanveer, and Alzheimer’s Disease Neuroimaging Initiative, An efficient angle-based universum least squares twin support vector machine for classification, ACM Transactions on Internet Technology, 21(3) (2021), 1–24.
[38] Y. H. Shao, C. H. Zhang, X. B. Wang, and N. Y. Deng, Improvements on twin support vector machines, IEEE Transactions on Neural Networks, 22(6) (2011), 962–968.
[39] Y. H. Shao, Z. Wang, W. J. Chen, and N. Y. Deng, Least squares twin parametric-margin support vector machine for classification, Applied Intelligence, 39 (2013), 451–464.
[40] M. Tanveer, A. Sharma, and P. N. Suganthan, General twin support vector machine with pinball loss function, Information Sciences, 494 (2019), 311–327.
[41] M. Tanveer, A. Tiwari, R. Choudhary, and M. Ganaie, Large-scale pinball twin support vector machines, Machine Learning, (2022), 1–24.
[42] Y. Tian and Z. Qi, Review on: twin support vector machines, Annals of Data Science, 1 (2014), 253–277.
[43] Y. Tian, Q. Zhang, and D. Liu, ν-Nonparallel support vector machine for pattern classification, Neural Computing and Applications, 25 (2014), 1007–1020.
[44] Y. Tian, Z. Qi, X. Ju, Y. Shi, and X. Liu, Nonparallel support vector machines for pattern classification, IEEE Transactions on Cybernetics, 44(7) (2013), 1067–1079.
[45] V. Vapnik, The support vector method of function estimation, in Nonlinear Modeling: Advanced Black-Box Techniques: Springer, (1998), 55–85.
[46] V. Vapnik and A. Chervonenkis, Theory of pattern recognition, Nauka, Moscow, 1974.
[47] J. Weston, R. Collobert, F. Sinz, L. Bottou, and V. Vapnik, Inference with the universum, in Proceedings of the 23rd International Conference on Machine Learning, (2006), 1009–1016.
[48] X. Y. Wang, T. Wang, and J. Bu, Color image segmentation using pixel wise support vector machine classification, Pattern Recognition, 44(4) (2011), 777–787.
[49] Y. Xiao, J. Wen, and B. Liu, A new multi-task learning method with universum data, Applied Intelligence, 51 (2021), 3421–3434.
[50] Y. Xu, M. Chen, and G. Li, Least squares twin support vector machine with universum data for classification, International Journal of Systems Science, 47(15) (2016), 3637–3645.
[51] Z. Yang and Y. Xu, Laplacian twin parametric-margin support vector machine for semi-supervised classification, Neurocomputing, 171 (2016), 325–334.
[52] H. Zamani Sabzi, S. Abudu, R. Alizadeh, L. Soltanisehat, N. Dilekli, and J. P. King, Integration of time series forecasting in a dynamic decision support system for multiple reservoir management to conserve water sources, Energy Sources, Part A: Recovery, Utilization, and Environmental Effects, 40(11) (2018), 1398–1416.
[53] J. Zhao, Y. Xu, and H. Fujita, An improved non-parallel universum support vector machine and its safe sample screening rule, Knowledge-Based Systems, 170 (2019), 79–88.