![\"\"](\"https:\/\/blogs.infosys.com\/topaz\/wp-content\/uploads\/2025\/06\/collage.png\")
<\/p>\n
<\/p>\n
Why Fine-Tuning Was Necessary<\/h3>\n\n- Complex Retail Environments:<\/strong>
\nRetail shelves are inherently dynamic. Variations in product arrangement, lighting, and camera angles can significantly impact detection accuracy.<\/li>\n- Subtle Visual Differences:<\/strong>
\nThe pre-trained model struggled to differentiate between intentional display gaps and genuine low-stock scenarios. This required the model to develop a more nuanced sensitivity.<\/li>\n- Data Specificity:<\/strong>
\nOur retail data contains specific visual characteristics that weren’t fully captured in the general pre-training dataset, making domain-specific fine-tuning essential.<\/li>\n<\/ul>\n <\/p>\n
\n <\/p>\n
Fine-Tuning Methods for VILA
\n![\"\"](\"https:\/\/blogs.infosys.com\/topaz\/wp-content\/uploads\/2025\/06\/Picture15.png\")
<\/h3>\n
\nArchitecture & Components<\/h3>\n
VILA employs an auto-regressive architecture that integrates visual and textual inputs seamlessly. Its design comprises three main components:<\/p>\n
\u25cf\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Vision Encoder (ViT):<\/strong> Converts images into feature-rich embeddings.<\/p>\n\u25cf\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Projector:<\/strong> Bridges the visual encoder and the LLM by translating image embeddings into a tokenized format that the LLM can understand.<\/p>\n\u25cf\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Large Language Model (LLM)<\/strong>: Processes both visual tokens and text, generating coherent, context-aware responses. This setup treats visual inputs like a “foreign language,” enabling the model to reason over multiple images and maintain strong text-only capabilities.<\/p>\n <\/p>\n
\nFine-Tuning Methods and Observations<\/h3>\n
The table below summarizes the approaches, highlighting \u00a0strengths and weaknesses:<\/p>\n
![\"\"](\"https:\/\/blogs.infosys.com\/topaz\/wp-content\/uploads\/2025\/06\/table-1.png\")
<\/p>\n
![\"\"](\"https:\/\/blogs.infosys.com\/topaz\/wp-content\/uploads\/2025\/06\/Picture16.png\")
<\/p>\n
<\/p>\n
Fine-Tuning Setup Using PEFT (ViT[1]\u00a0[2]\u00a0[3]\u00a0[4]\u00a0: FFT & LLM: LORA)<\/h3>\n
To enhance Smart Inventory Monitoring<\/strong>, we fine-tuned the VILA<\/strong> model using Parameter-Efficient Fine-Tuning (PEFT)<\/strong> method and applied full fine-tuning (FFT)<\/strong> on the vision encoder and LORA for the LLM . The pre-trained model initially misidentified empty spaces and low-stock areas, necessitating domain-specific adaptation.<\/p>\nDataset\u00a0<\/strong><\/h4>\nVideos Collected: 10 (8 for fine-tuning, 2 for testing)<\/span>\r\n\r\nCamera Setup: Multiple sources focusing on a single shelf<\/span>\r\n\r\nPreprocessing: ffmpeg re-encoding for compatibility<\/span>\r\n\r\nAnnotation Format: JSON (Segmented by events)<\/span>\r\n\r\nResolution: 2560x1440 <\/span>\r\n\r\n<\/pre>\nExample Annotation:<\/strong><\/h4>\n{<\/span>\r\n\r\n\"video_131_1\": {<\/span>\r\n\r\n\"duration\": 35,<\/span>\r\n\r\n\"subset\": \"training\",<\/span>\r\n\r\n\"recipe_type\": \"131\",<\/span>\r\n\r\n\"annotations\": [<\/span>\r\n\r\n{<\/span>\r\n\r\n\"segment\": [0, 5],<\/span>\r\n\r\n\"id\": \"1\",<\/span>\r\n\r\n\"sentence\": \"Top rack: 4 Pepsi bottles, 2 orange bottles (low stock), etc.\"<\/span>\r\n\r\n},<\/span>\r\n\r\n{<\/span>\r\n\r\n\"segment\": [5, 8],<\/span>\r\n\r\n\"id\": \"2\",<\/span>\r\n\r\n\"sentence\": \"Customer approaches the shelf.\"<\/span>\r\n\r\n}<\/span>\r\n\r\n]<\/span>\r\n\r\n}<\/span>\r\n\r\n}<\/span><\/pre>\nTraining Setup<\/h4>\nModel Fine-Tuned: VILA-1.5-34B<\/span>\r\n\r\nGPUs: 8 x NVIDIA H100 (80GB)<\/span>\r\n\r\nBatch Size: 8<\/span><\/pre>\nFine-Tuning Time<\/h4>\nTotal Training Time: ~2 hr 15mins<\/span>\r\n\r\nEpochs Completed: 50<\/span><\/pre>\nPrompt & Chunking Configuration<\/h2>\n\n- VLM Dense Captioning Prompt:<\/strong><\/li>\n<\/ul>\n
Continuously analyze the video to create detailed captions that monitor the inventory levels across 5 racks on a retail shelf over time. For each rack, count the visible products and clearly distinguish between intentional design gaps and actual shortages. If any rack shows 2 or fewer products (ignoring the designated gaps), immediately mark it with a ‘Low Stock Alert’; if a rack has zero visible products, mark it with an ‘Empty Rack Alert’ also add a short description of the scene. Alerts generated should contains only brief details about the scene. Ensure that every detection event is accompanied by an accurate timestamp and that the status is updated dynamically as the video plays. Finally, compile these frame-by-frame insights dense caption that provides a comprehensive overview of the shelf’s status changes over time, highlighting the specific racks and moments when alerts were triggered. Format the final output for storage in a vector database for efficient retrieval and analysis<\/span><\/em><\/p>\n\n- Caption Summarization (LLM) Prompt:<\/strong><\/li>\n<\/ul>\n
Review the dense captions generated for the retail shelf, ensuring each event is tagged with precise timestamps. Extract and condense the key insights by focusing on the number of visible products per rack, the occurrence of ‘Low Stock Alert’ or ‘Out of Stock Alert’ events, and brief scene descriptions. Generate a succinct summary that highlights the critical inventory status, pinpointing the exact time and rack where stock issues were detected, to inform potential restocking actions efficiently<\/span><\/em><\/p>\n\n- Summary Aggregation (LLM) Prompt:<\/strong><\/li>\n<\/ul>\n
Combine multiple caption summaries, each with their associated timestamps, into a comprehensive overview that reflects the overall stock status across the retail shelves. Aggregate the individual insights to identify trends, such as recurring low stock or empty shelf events, over time. Provide an aggregated summary that highlights key products at risk, overall inventory health, and includes relevant timestamps for tracking changes and triggering timely interventions. Ensure the summary is actionable and clearly outlines areas requiring immediate attention.<\/em><\/span><\/p>\n\n- Video Chunking Strategy:<\/strong><\/li>\n<\/ul>\n
Chunk Size= 30secs<\/span><\/em><\/p>\nResults & Observations<\/h2>\n
We evaluated our approach on our dataset using two configurations: a Zero-Shot<\/strong> configuration, where the pre-trained VILA model is used without any fine-tuning, and a PEFT (ViT: FFT & LLM: LORA)<\/strong> configuration, where full fine-tuning is applied to the vision encoder component and the LLM is fine-tuned using Low-Rank Adaptation (LoRA).<\/p>\nBefore fine-tuning, the zero-shot pre-trained VILA model frequently misidentified empty spaces as low-stock areas, \u00a0resulting in inventory monitoring inaccuracies. However, after applying PEFT on VILA,[5]\u00a0 we observed significant improvements in product detection accuracy and \u00a0fewer false alarms.<\/p>\n
Performance Improvement<\/strong><\/h3>\nZero-Shot Evaluation<\/h3>\n
![\"\"](\"https:\/\/blogs.infosys.com\/topaz\/wp-content\/uploads\/2025\/06\/table-2.png\")
<\/p>\n
PEFT (ViT: FFT & LLM: LORA)<\/h3>\n
![\"\"](\"https:\/\/blogs.infosys.com\/topaz\/wp-content\/uploads\/2025\/06\/table-3.png\")
<\/p>\n
<\/p>\n
Key Observations<\/h2>\n\n- Performance Boost:<\/strong> Fine-tuning more than doubled BLEU scores and significantly improved CIDEr, METEOR, and ROUGE_L, reflecting a notable enhancement in caption quality and alignment with ground truth.<\/li>\n
- Efficiency with Limited Data:<\/strong> These improvements were achieved using a relatively small dataset and without full-scale fine-tuning, showcasing the effectiveness of our PEFT approach.<\/li>\n
- Overall Impact:<\/strong> Our streamlined fine-tuning process significantly enhanced model accuracy and reliability in detecting low-stock conditions on retail shelves.<\/li>\n<\/ul>\n
Sample Alert Triggered<\/strong>
\nBelow is an alert generated when low stock or an empty shelf is detected:<\/p>\n![\"\"](\"https:\/\/blogs.infosys.com\/topaz\/wp-content\/uploads\/2025\/06\/Picture17.png\")
<\/p>\n
Future Possibilities with Infosys Video AI Agent<\/h2>\nAdvanced Inventory Management<\/h3>\n\n- Smart Reordering:<\/strong> Integrate with inventory systems to trigger automatic product reordering.<\/li>\n
- Dynamic Shelf Management:<\/strong> Use real-time data to adjust shelf layouts dynamically.<\/li>\n<\/ul>\n
Enhanced Customer Engagement<\/h3>\n\n- Personalized Marketing:<\/strong> Leverage customer behavior analytics to deliver targeted offers.<\/li>\n
- Augmented Reality (AR):<\/strong> Deploy AR interfaces for in-store staff, overlaying real-time VSS insights onto live feeds.<\/li>\n<\/ul>\n
Continuous Learning & Adaptability<\/h3>\n\n- Real-Time Fine-Tuning:<\/strong> Implement continuous fine-tuning to adapt to evolving retail scenarios.<\/li>\n
- Customization:<\/strong> Tailor VSS for diverse retail settings, from small shops to large hypermarkets.<\/li>\n<\/ul>\n
\nConclusion<\/h2>\n
By combining advanced video analysis with intelligent summarization and real-time alerting, Infosys Video AI Agent redefines retail store monitoring. The integration of a Video AI Agent with fine-tuning techniques (like LoRA) has resulted in a system that not only improves security and safety but also optimizes inventory management and enhances customer satisfaction. As we continue refining our models and exploring new integrations, Infosys Video AI Agent is poised to set a new standard in AI-driven retail solutions.<\/p>\n","protected":false},"excerpt":{"rendered":"
Inventory Management & Shelf Replenishment Smart Inventory Monitoring, which involves detecting empty shelves and […]<\/p>\n","protected":false},"author":865,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[12],"tags":[],"coauthors":[14],"class_list":["post-61","post","type-post","status-publish","format-standard","hentry","category-nvidia"],"acf":[],"_links":{"self":[{"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/posts\/61","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/users\/865"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/comments?post=61"}],"version-history":[{"count":3,"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/posts\/61\/revisions"}],"predecessor-version":[{"id":76,"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/posts\/61\/revisions\/76"}],"wp:attachment":[{"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/media?parent=61"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/categories?post=61"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/tags?post=61"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/coauthors?post=61"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nRetail shelves are inherently dynamic. Variations in product arrangement, lighting, and camera angles can significantly impact detection accuracy.<\/li>\n
\nThe pre-trained model struggled to differentiate between intentional display gaps and genuine low-stock scenarios. This required the model to develop a more nuanced sensitivity.<\/li>\n
\nOur retail data contains specific visual characteristics that weren’t fully captured in the general pre-training dataset, making domain-specific fine-tuning essential.<\/li>\n<\/ul>\n
<\/p>\n
\n
<\/p>\n
Fine-Tuning Methods for VILA
\n<\/h3>\n
\nArchitecture & Components<\/h3>\n
VILA employs an auto-regressive architecture that integrates visual and textual inputs seamlessly. Its design comprises three main components:<\/p>\n
\u25cf\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Vision Encoder (ViT):<\/strong> Converts images into feature-rich embeddings.<\/p>\n \u25cf\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Projector:<\/strong> Bridges the visual encoder and the LLM by translating image embeddings into a tokenized format that the LLM can understand.<\/p>\n \u25cf\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Large Language Model (LLM)<\/strong>: Processes both visual tokens and text, generating coherent, context-aware responses. This setup treats visual inputs like a “foreign language,” enabling the model to reason over multiple images and maintain strong text-only capabilities.<\/p>\n <\/p>\n The table below summarizes the approaches, highlighting \u00a0strengths and weaknesses:<\/p>\n <\/p>\n To enhance Smart Inventory Monitoring<\/strong>, we fine-tuned the VILA<\/strong> model using Parameter-Efficient Fine-Tuning (PEFT)<\/strong> method and applied full fine-tuning (FFT)<\/strong> on the vision encoder and LORA for the LLM . The pre-trained model initially misidentified empty spaces and low-stock areas, necessitating domain-specific adaptation.<\/p>\n Continuously analyze the video to create detailed captions that monitor the inventory levels across 5 racks on a retail shelf over time. For each rack, count the visible products and clearly distinguish between intentional design gaps and actual shortages. If any rack shows 2 or fewer products (ignoring the designated gaps), immediately mark it with a ‘Low Stock Alert’; if a rack has zero visible products, mark it with an ‘Empty Rack Alert’ also add a short description of the scene. Alerts generated should contains only brief details about the scene. Ensure that every detection event is accompanied by an accurate timestamp and that the status is updated dynamically as the video plays. Finally, compile these frame-by-frame insights dense caption that provides a comprehensive overview of the shelf’s status changes over time, highlighting the specific racks and moments when alerts were triggered. Format the final output for storage in a vector database for efficient retrieval and analysis<\/span><\/em><\/p>\n Review the dense captions generated for the retail shelf, ensuring each event is tagged with precise timestamps. Extract and condense the key insights by focusing on the number of visible products per rack, the occurrence of ‘Low Stock Alert’ or ‘Out of Stock Alert’ events, and brief scene descriptions. Generate a succinct summary that highlights the critical inventory status, pinpointing the exact time and rack where stock issues were detected, to inform potential restocking actions efficiently<\/span><\/em><\/p>\n Combine multiple caption summaries, each with their associated timestamps, into a comprehensive overview that reflects the overall stock status across the retail shelves. Aggregate the individual insights to identify trends, such as recurring low stock or empty shelf events, over time. Provide an aggregated summary that highlights key products at risk, overall inventory health, and includes relevant timestamps for tracking changes and triggering timely interventions. Ensure the summary is actionable and clearly outlines areas requiring immediate attention.<\/em><\/span><\/p>\n Chunk Size= 30secs<\/span><\/em><\/p>\n We evaluated our approach on our dataset using two configurations: a Zero-Shot<\/strong> configuration, where the pre-trained VILA model is used without any fine-tuning, and a PEFT (ViT: FFT & LLM: LORA)<\/strong> configuration, where full fine-tuning is applied to the vision encoder component and the LLM is fine-tuned using Low-Rank Adaptation (LoRA).<\/p>\n Before fine-tuning, the zero-shot pre-trained VILA model frequently misidentified empty spaces as low-stock areas, \u00a0resulting in inventory monitoring inaccuracies. However, after applying PEFT on VILA,[5]\u00a0 we observed significant improvements in product detection accuracy and \u00a0fewer false alarms.<\/p>\n <\/p>\n Sample Alert Triggered<\/strong> By combining advanced video analysis with intelligent summarization and real-time alerting, Infosys Video AI Agent redefines retail store monitoring. The integration of a Video AI Agent with fine-tuning techniques (like LoRA) has resulted in a system that not only improves security and safety but also optimizes inventory management and enhances customer satisfaction. As we continue refining our models and exploring new integrations, Infosys Video AI Agent is poised to set a new standard in AI-driven retail solutions.<\/p>\n","protected":false},"excerpt":{"rendered":" Inventory Management & Shelf Replenishment Smart Inventory Monitoring, which involves detecting empty shelves and […]<\/p>\n","protected":false},"author":865,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[12],"tags":[],"coauthors":[14],"class_list":["post-61","post","type-post","status-publish","format-standard","hentry","category-nvidia"],"acf":[],"_links":{"self":[{"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/posts\/61","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/users\/865"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/comments?post=61"}],"version-history":[{"count":3,"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/posts\/61\/revisions"}],"predecessor-version":[{"id":76,"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/posts\/61\/revisions\/76"}],"wp:attachment":[{"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/media?parent=61"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/categories?post=61"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/tags?post=61"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blogs.infosys.com\/topaz\/wp-json\/wp\/v2\/coauthors?post=61"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nFine-Tuning Methods and Observations<\/h3>\n
<\/p>\n<\/p>\nFine-Tuning Setup Using PEFT (ViT[1]\u00a0[2]\u00a0[3]\u00a0[4]\u00a0: FFT & LLM: LORA)<\/h3>\n
Dataset\u00a0<\/strong><\/h4>\n
Videos Collected: 10 (8 for fine-tuning, 2 for testing)<\/span>\r\n\r\nCamera Setup: Multiple sources focusing on a single shelf<\/span>\r\n\r\nPreprocessing: ffmpeg re-encoding for compatibility<\/span>\r\n\r\nAnnotation Format: JSON (Segmented by events)<\/span>\r\n\r\nResolution: 2560x1440 <\/span>\r\n\r\n<\/pre>\nExample Annotation:<\/strong><\/h4>\n
{<\/span>\r\n\r\n\"video_131_1\": {<\/span>\r\n\r\n\"duration\": 35,<\/span>\r\n\r\n\"subset\": \"training\",<\/span>\r\n\r\n\"recipe_type\": \"131\",<\/span>\r\n\r\n\"annotations\": [<\/span>\r\n\r\n{<\/span>\r\n\r\n\"segment\": [0, 5],<\/span>\r\n\r\n\"id\": \"1\",<\/span>\r\n\r\n\"sentence\": \"Top rack: 4 Pepsi bottles, 2 orange bottles (low stock), etc.\"<\/span>\r\n\r\n},<\/span>\r\n\r\n{<\/span>\r\n\r\n\"segment\": [5, 8],<\/span>\r\n\r\n\"id\": \"2\",<\/span>\r\n\r\n\"sentence\": \"Customer approaches the shelf.\"<\/span>\r\n\r\n}<\/span>\r\n\r\n]<\/span>\r\n\r\n}<\/span>\r\n\r\n}<\/span><\/pre>\nTraining Setup<\/h4>\n
Model Fine-Tuned: VILA-1.5-34B<\/span>\r\n\r\nGPUs: 8 x NVIDIA H100 (80GB)<\/span>\r\n\r\nBatch Size: 8<\/span><\/pre>\nFine-Tuning Time<\/h4>\n
Total Training Time: ~2 hr 15mins<\/span>\r\n\r\nEpochs Completed: 50<\/span><\/pre>\nPrompt & Chunking Configuration<\/h2>\n
\n
\n
\n
\n
Results & Observations<\/h2>\n
Performance Improvement<\/strong><\/h3>\n
Zero-Shot Evaluation<\/h3>\n
<\/p>\nPEFT (ViT: FFT & LLM: LORA)<\/h3>\n
<\/p>\nKey Observations<\/h2>\n
\n
\nBelow is an alert generated when low stock or an empty shelf is detected:<\/p>\n<\/p>\nFuture Possibilities with Infosys Video AI Agent<\/h2>\n
Advanced Inventory Management<\/h3>\n
\n
Enhanced Customer Engagement<\/h3>\n
\n
Continuous Learning & Adaptability<\/h3>\n
\n
\nConclusion<\/h2>\n