In this paper, we proposed a hierarchical spatial pyramid pooling mechanism for improving fine-grained vehicle classification problems. Our proposed method created by removing the last layer of convolutional neural network (CNN) classifier, attaching the hierarchical spatial pyramid pooling at the end of the network, and fine-tune the CNN classifier. Our hierarchical spatial pyramid pooling consists of two independent spatial pyramid pooling that pooled the features from two different layers in the original CNN classifier. We modified ResNet50 and AlexNet CNN classifier using our proposed method and test it on fine-grained vehicle classification dataset. Unlike part-based classifier, which very popular method to tackle fine-grained classification problems, our method does not require any pre-processing or post-processing step in order to do the classification tasks. Experiments on BoxCars fine-grained vehicle classification dataset shows that our proposed method can increase the overall accuracy of the classifier and some of the results achieve state-of-the-art performance.