Olive oil is one of the food products which immensely adulterated by cheaper substances, including animal fats or cheaper vegetable oils. It gives a concern since olive oil contains numerous healthy fats which are beneficial for the human body. However, the adulteration changes the fat composition in olive oil. The adulteration can lead to several life-threatening diseases, including coronary heart disease. Hence it is essential to do authentication of olive oil products. This paper presents the quantification method for determining the adulterant level in extra-virgin olive oil (EVOO) using a smartphone with a custom-built optical attachment. Sample sets are composed of binary blends of EVOO and palm oil with 11 different concentrations of palm oil from 0% to 100% (v/v) at an increment of 10%. The proposed method is based on fluorescence imaging of the samples by the excitation light of a laser pointer (λ = 405 nm). The images were obtained by a complementary metal-oxide-semiconductor (CMOS) sensor of a smartphone camera and processed by an image-processing algorithm to obtain the peak value of the Red (R), Green (G), and Blue (B) in RGB color space. Analysis of images of EVOO containing different levels of palm oil revealed a correlation between RGB and the level of palm oil in EVOO. The proposed sensor system has the best value at the analysis in the green color space with the linearity and the sensitivity of 97.32% and 2.892 color-scale/%, respectively. The proposed method offers low cost, is easy to operate, and could be a consumer-oriented solution for reducing olive oil adulteration.