Gastrointestinal endoscopy has been an active field of research motivated by the large number of highly lethal GI cancers. Early GI cancer precursors are often missed during the endoscopic surveillance.   The high missed rate of such abnormalities during endoscopy is thus a  critical bottleneck. Lack of attentiveness due to tiring procedures, and requirement of training are few contributing factors. An automatic  GI  disease classification system can help reduce such risks by flagging suspicious frames and lesions. GI endoscopy consists of several multi-organ surveillance, therefore, there is need to develop methods that can generalize to various endoscopic findings. In this realm, we present a comprehensive analysis of the  Medico  GI  challenges:  Medical  Multimedia Task at MediaEval 2017, Medico Multimedia Task at MediaEval 2018,  and  BioMedia  ACM  MM  Grand  Challenge  2019.  These challenges are initiative to set-up  a  benchmark for different computer vision methods applied to the multi-class endoscopic images and promote to build new approaches that could reliably be used in clinics. We report the performance of 21 participating teams over a  period of three consecutive years and provide a  detailed analysis of the methods used by the participants, highlighting the challenges and shortcomings of the current approaches and dissect their credibility for the use in clinical settings.  Our analysis revealed that the participants achieved an improvement on maximum  Mathew correlation coefficient  (MCC)  from  82.68%  in  2017  to  93.98%  in  2018  and 95.20% in 2019 challenges, and a significant increase in computational speed over consecutive years.