Mental fatigue (MF) is a common phenomenon in our daily lives. In the workplace, mental fatigue can increase the risk of human errors. Moreover, if it is not detected and handled correctly, it can cause various problems. Electroencephalogram (EEG) is one of many modalities that is widely used by researchers for detecting mental fatigue. However, due to the difficulty in provoking fatigue condition during the EEG measurement, this study proposed two experimental designs for detecting mental fatigue, the first design is by using physical induction (PI) and the 2nd design is by using mental induction (MI). We obtained the EEG signals from 20 healthy participants, from 14 channels of wireless EEG headset. Each participant was given four types of cognitive tests in the form of a computer game including Trail, Span, Stroop, and Arithmetic. The subjective measurement of fatigue condition was measured by using the Swedish Occupational Fatigue Inventory (SOFI) questionnaires. The EEG signal was extracted by using power percentage (PP) features of alpha (\alpha), beta (\beta), and theta (\theta) bands of frequency from all channels. As for classification, we introduced the use Relevance Vector Machine (RVM) approach which is claimed to have better performance than SVM as the precedence method. According to the results, all of the cognitive tests has increased average value of response time (RT) and decreased correction score (CS) on post-induction compared to pre-induction. Another result showed that the best mental fatigue detection is obtained from mental induction rather than physical induction. The classification results demonstrate an accuracy of 93.7% (MI) and 92.6% (PI) are achieved by RVM approach. Furthermore, RVM has also less execution time compared to SVM.