Dipayan Biswas
I am a machine learning researcher and Ph.D. candidate in Computer Science at the University of Houston (expected December 2026), advised by Prof. Shishir Shah (now at the University of Oklahoma) and Prof. Jaspal Subhlok. I specialize in multimodal AI: vision-language models, LLM evaluation, and computer vision.
On the NSF-funded VideoPoints platform, I deployed a summarization system that condenses hour-long lectures into 5-minute summaries with supporting visuals, used by 750+ students in 10+ courses. My work also covers object detection in lecture videos, including the LVVO dataset, and fine-tuning VLMs with QLoRA. I have published 5 peer-reviewed IEEE papers.
Previously, I was an Assistant Professor / Lecturer at Khulna University of Engineering & Technology (KUET), Bangladesh, and a Software Engineer at Samsung R&D Institute Bangladesh. I hold an M.Sc. in Computer Science from the University of Houston (May 2026, awarded en route to the Ph.D.) and a B.Sc. in Electronics and Communication Engineering from KUET, where I was University Gold Medalist.
I am graduating in December 2026 and seeking full-time Applied Scientist, Research Scientist, and ML Engineer roles in multimodal AI, vision-language models, and computer vision.
University of Houston
- dipayan1109033@gmail.com
- Houston, TX
- GitHub
- Google Scholar
Research Interests
Technical Skills
News
- Sep 2026 Project Built Multi-Step LLM and VLM Pipelines: configurable prompt chaining for multi-step workflows on local or cloud models, with resumable experiments. [Code]
- Aug 2026 Project Built a Unified Inference Toolkit for Vision-Language Models: one interface for multimodal prompts on 8+ backends, from cloud APIs to vLLM, Ollama, and Hugging Face Transformers. [Code]
- Jun 2026 Project Fine-tuned Gemma 3 with QLoRA, raising ChartQA accuracy from 43.6% to 61.6% (4B) and 61.8% to 69.4% (12B). [Code]
- May 2026 Degree Received M.Sc. in Computer Science from the University of Houston, en route to the Ph.D.
- Mar 2026 Project Deployed multimodal video summarization on VideoPoints, condensing hour-long lectures into 5-minute summaries with supporting visuals.
- Feb 2026 Certificate Completed Generative AI with Large Language Models (DeepLearning.AI and AWS), Coursera. [Certificate]
- Jan 2026 Certificate Completed Machine Learning in Production (DeepLearning.AI), Coursera. [Certificate]
- Nov 2025 Award Received Audience's Choice Runner-Up at the Ph.D. Research Showcase, University of Houston. [Certificate]
- Nov 2025 Industry Completed the Machine Learning Fellowship at Fellowship.AI, building a multimodal real estate listing analyzer.
- Nov 2025 Course Completed the AWS Cloud Practitioner Essentials course (Amazon Web Services). [Certificate]
- Aug 2025 Talk Presented at IEEE MIPR 2025 (San Jose, CA): “Visual Content Detection in Educational Videos with Transfer Learning and Dataset Enrichment”.
- Aug 2025 Project Deployed Lecture Video Chaptering with LLMs on VideoPoints, splitting each newly uploaded lecture into titled chapters. [Details]
- Jul 2025 Course Completed Building and Evaluating Advanced RAG (DeepLearning.AI). [Course]
- Jul 2025 Service Concluded 3 years as Organizing Secretary, Graduate Student Association of Bangladesh (GSAB), University of Houston.
- Jun 2025 Dataset Released LVVO, a 4,000-frame dataset for visual object detection in lecture videos. [Paper] [Data]
- May 2025 Paper “Visual Content Detection in Educational Videos with Transfer Learning and Dataset Enrichment” accepted at IEEE MIPR 2025.
- Feb 2025 Project Built a Modular Object Detection Training Pipeline, a PyTorch codebase for reproducible training and fine-tuning of YOLO, Faster R-CNN, and DETR with Hydra configs. [Code]
- Aug 2022 Service Joined as Organizing Secretary, Graduate Student Association of Bangladesh (GSAB), University of Houston.
Experience
- Condensed hour-long lectures into 5-minute summaries with supporting visuals for review and navigation, used by 750+ students in 10+ courses; 78% of surveyed students rated them helpful.
- Designed a multimodal pipeline fusing speech (Whisper), slide text (OCR), and slides with Gemma 3, raising faithfulness by 19% (76.7% to 91.4%) and coverage by 15% in a blind evaluation by 8 faculty.
- Automated evaluation with an LLM-as-judge (DeepEval) that matched faculty agreement on faithfulness (0.68 vs. 0.59 rank correlation) and reproduced their 19% gain (judge: 18%).
- Showed that per-slide fusion lifts the coverage gain over transcript-only from 10.6% to 15.0%, beyond run-to-run noise.
- Developed zero-shot cross-modal selection of supporting visuals with SigLIP and Gemma 3; faculty rated it 82.8% (up from 70.3%) and kept 83% of the selected visuals.
- Released LVVO, a 4,000-frame dataset from 245 videos, planning the labeling guidelines and a multi-annotator protocol with expert conflict resolution (83.4% inter-annotator F1).
- Raised AP50 from 90.8% to 95.3% with semi-supervised auto-labeling, adding 3x more training frames to 1,000 expert-labeled frames without new manual labeling.
- Developed a graph-based method merging fragmented detections into coherent visuals, improving mAP by 15.8%.
- Developed a multimodal listing analyzer (Gemini API) that jointly scores real estate photos and descriptions to recommend pricing, staging, and content improvements across 3k listings.
- Engineered a weighted-KNN price prediction pipeline with outlier detection over 8K+ Redfin listings, flagging mispriced properties at 85% precision against held-out sale prices.
- Taught five undergraduate courses with labs, including Digital Image Processing, and supervised student research projects.
- Mentored 10+ undergraduates in ML research, co-authoring IEEE papers on plant disease (99.80% accuracy) and breast cancer (99.28%) classification with transfer learning and ensemble methods.
- Implemented 360-degree photo and video viewing modes, including panoramic and little-planet views, for the shipped Samsung Gear 360 iOS app using Objective-C, OpenGL ES, and OpenCV.
Selected Projects
Built configurable prompt chaining for multi-step workflows on local or cloud models, with resumable experiments. Made the workflow engine reusable across datasets, saving every step's output for review.
Built one interface for multimodal prompts on 8+ backends, from cloud APIs to vLLM, Ollama, and Hugging Face Transformers. Pre-configured 10+ open LLMs and VLMs, including Gemma 3, Qwen3-VL, and InternVL, each callable by name.
Fine-tuned Gemma 3 with QLoRA, raising ChartQA accuracy from 43.6% to 61.6% (4B) and 61.8% to 69.4% (12B). Trained on 28k chart questions using Hugging Face TRL and PEFT on single and multiple L40S GPUs via Slurm.
Deployed automatic chaptering that splits each newly uploaded lecture into titled chapters; in daily use since Aug 2025. Aligned Whisper transcripts with slide transitions and used Gemini structured output, auto-correcting gaps and overlaps.
Designed a PyTorch codebase for reproducible training and fine-tuning of YOLO, Faster R-CNN, and DETR, with Hydra configs and unified train, evaluate, and predict steps.
In a team project, trained and evaluated an IoU-aware RetinaFace with ResNet50 and MobileNet0.25 backbones on 4 V100 GPUs, raising AP on the medium and hard WIDER FACE sets.
Publications [Google Scholar]
This paper investigates how modern object detectors can be adapted to lecture videos, where visual content is often semantically complex and weakly structured. It shows that YOLO-based transfer learning, combined with semi-supervised dataset enrichment, raises AP50 from 90.8% to 95.3%.
This work presents LVVO, a dataset for visual object detection in educational videos, with 4,000 frames from 245 lecture videos. Its labels come from a multi-annotator protocol with expert conflict resolution, and it supports both supervised and semi-supervised research.
This paper presents LogiForm, a graph-based approach for identifying meaningful visual objects in lecture video frames, where slide visuals are often diverse, custom-designed, and composed of parts without clear boundaries. By merging fragmented detections using spatial, color, and local geometric cues, it improves mAP by 15.8%.
Develops a ResNet50 transfer-learning model for 38-class plant disease detection on PlantVillage, reaching 99.80% accuracy.
Builds a five-classifier voting ensemble for breast cancer prediction, reaching 99.28% accuracy.
Detects epileptic seizures with an artificial neural network (ANN) using selected features of different complexities.
Education
Certifications & Courses
Honors & Awards
- Ph.D. Research Showcase, Audience's Choice Runner-Up, University of Houston, 2025 [Certificate]
- Cullen Graduate Student Fellowship, University of Houston, 2021–2026
- University Gold Medalist (ranked 1st in class), KUET, 2018 [Certificate]