Lessons from benchmarking, fine-tuning, and operationalizing multimodal AI for manufacturing workflows

Ahead of the launch of NVIDIA Nemotron 3.5 Super VL, Infosys Topaz (Infosys’s AI-first set of services, solutions, and platforms built using generative AI technologies) received early access to the model and evaluated it against a manufacturing-specific benchmark spanning documents, engineering drawings, NVIDIA SCADA interfaces, inspection reports, charts, video, and tool-calling workflows.

NVIDIA Nemotron 3.5 Super VL is an open, customizable multimodal model built for long-running agents, combining reasoning, tool use, and text, image, and video understanding.

We used the benchmark to compare the model against a leading open-source vision-language model. This post shares what we measured, where the model’s advantage comes from, and what we learned about fine-tuning it well.

Why NVIDIA Nemotron 3.5 Super VL for manufacturing agents

NVIDIA Nemotron 3.5 Super VL uses a 120B-total, 12B-active architecture that combines reasoning, instruction following, and tool use with text and visual understanding.

For our manufacturing clients, three properties mattered most going in:

• One model for perception, reasoning, and tool calls. This avoids chaining a vision model to a separate reasoning model and losing context in between.

•  A hybrid reasoning mode. It can be switched on for calculation-heavy questions and off for simple extraction.

• Open weights that can be post-trained. Clients can adapt the model on their own data and deploy it on infrastructure they control.

Building an evaluation that would hold up

Public benchmarks tell you little about whether a model can total a bill of materials (BOM) or spot a station fault on a human-machine interface (HMI) screen. To assess performance in real manufacturing contexts, Infosys Topaz built a domain-specific evaluation around three design principles.

Labels are exact by construction. Every image and video in the benchmark is rendered from seed data, and every gold answer is computed from that same data. There is no human labelling error and no ambiguity about what “correct” means.

Scoring is programmatic. Five deterministic verifiers grade every response: exact match, numeric tolerance, abstention detection, JSON schema match, and tool-call match. No LLM-as-judge is used.

The test set is fully held out. Training, validation, and test data come from disjoint random seeds. We verified split integrity by hashing every rendered file across all three splits, and found zero shared media.

Figure 1. Sample rendered items from the evaluation data: an engineering drawing, a SCADA line-control screen, a bill of materials, and a rotated, noisy equipment nameplate used for robustness testing.

The resulting test set has 240 items across 70 rendered images and 5 rendered videos, covering 19 capabilities. These include document extraction from work orders, certificate of analysis (CoAs), and equipment nameplates, as well as engineering drawing dimensions and tolerances. HMI and supervisory control and data acquisition (SCADA) screens appear too, along with spec compliance on inspection reports, BOM cost totals, and BOM-to-JSON conversion. The set also covers statistical process control (SPC) and Pareto charts, overall equipment effectiveness (OEE) calculation, visual defect inspection, and conveyor video counting. Finally, it includes tool calling (for example, logging a nonconformance against the right part and lot) and multi-image comparison. Robustness variants rotate, blur, and add noise to documents. A dedicated abstention category tests whether the model says a field is absent rather than inventing one.

Every model ran through the identical harness, prompts, and serving stack (vLLM, temperature 0). We ran the base model twice and the runs differed by one item. We treat ±1 item as the noise floor, and every comparison below should be read against it.

Out of the box, without any customization, NVIDIA Nemotron 3.5 Super VL answered 231 of 240 manufacturing questions correctly. That beat a strong open-source baseline by roughly six points. The per-capability view shows where that lead comes from:

Where the advantage comes from and where it doesn’t
The two models disagree on 18 items. NVIDIA Nemotron 3.5 Super VL wins 16 and the open-source baseline wins 2, and the pattern is clear. NVIDIA Nemotron 3.5 Super VL’s wins are almost entirely multi-step arithmetic over visual data. It got all four BOM totals right, and on one BOM it matched the exact gold total where the baseline was off by about $150. It also got all four BOM-to-JSON conversions and all four OEE calculations right, plus SPC and Pareto items. These are the categories where we route the model through its reasoning mode.

With reasoning switched off, the two models tie at 90.4%. So we describe NVIDIA Nemotron 3.5 Super VL’s lead as a reasoning advantage, not a raw perception advantage. It is a commercially meaningful one: in manufacturing, the expensive errors are usually the wrong sum or the wrong rate, not the misread label. The baseline’s two wins were both perception items: counting red defect parts in a conveyor video and distinguishing a crack from a scratch.

For context on the comparison, the open-source baseline is a 32B dense model, while NVIDIA Nemotron 3.5 Super VL has 120B total and 12B active parameters. The baseline has no reasoning mode, which is why we report NVIDIA Nemotron 3.5 Super VL both with and without reasoning.

One result we would call out for enterprise deployments is abstention. NVIDIA Nemotron 3.5 Super VL scored 17/17 on the abstention items, correctly reporting when a requested field, such as a customer PO number, was not on the document instead of fabricating one.

Customizing NVIDIA Nemotron 3.5 Super VL with NVIDIA NeMo AutoModel
A model at 96% on a domain benchmark still has room to grow, and for an agent, specific misses matter more than averages. Out of the box, NVIDIA Nemotron 3.5 Super VL’s most consequential miss in our use case testing was on conveyor video. Asked how many red defect parts passed end-of-line QA, it answered zero when the answer was three, consistently across runs.

We set out to improve this through customization with parameter-efficient fine-tuning (LoRA) using NVIDIA NeMo AutoModel. Our training set had 444 examples, drawn from training seeds disjoint from the test set. Our setup was:

•  Starting point: NVIDIA’s reference NVIDIA Nemotron 3.5 Super VL PEFT recipe. Our reproduction matched NVIDIA’s published trainable-parameter count exactly (177M parameters, 0.15% of the model, at rank 64). Our final adapter used rank 16, about 44M trainable parameters (0.04%).

•  What was trained: LoRA adapters on 272 modules, covering the Mamba projections, attention, MoE latent projections, and shared experts. The base model, vision tower, projector, and LM head stayed frozen.

•  Infrastructure: FSDP2 with 8-way expert parallelism on a single node of 8× NVIDIA H100 80GB GPUs.

The result: no regression, plus the targeted improvement.

With post-training, the video defect-counting failure flipped from wrong to correct in every run and in both reasoning modes. We therefore treat it as real signal rather than noise, and it brings NVIDIA Nemotron 3.5 Super VL level with the baseline on video. Every other capability held.

We want to be precise about what this does not show. The overall gain of about 1.5 items is within reach of our ±1 noise floor, and some individual items moved in both directions. This is not a headline accuracy lift, and we are not presenting it as one. The benchmark was already near ceiling. With only 8–9 failures left, it can confirm non-regression but cannot demonstrate a large improvement. What it does demonstrate is the pattern our clients need: find a specific failure and fix it with a small, cheap adapter without breaking anything else. The final adapter changed the model’s weights by less than 1% in relative norm.

What it takes to fine-tune a multimodal agent model well
Our first adapter did not land cleanly, and the lessons from that run are probably the most useful part of this post for practitioners.

Match the recipe to the task shape. NVIDIA’s reference recipe was built for receipt extraction, where targets are long, structured outputs on single images. Our targets were mostly one to five tokens: a number, a yes/no, or a part number. Applying the reference hyperparameters unchanged to short-answer QA concentrated too much gradient on a handful of answer tokens. The model drifted toward the most frequent training labels and lost accuracy. A 10× lower learning rate (2e-5) and a lower LoRA rank (16, keeping the same alpha/rank scale) fixed this. With those settings, validation loss fell steadily across every checkpoint.

Keep every capability in the training mix. In our first run, our own data pipeline accidentally excluded the tool-calling examples. Tool calling then collapsed, because the model began answering tool prompts with a frequent abstention template from the rest of the data. Restoring those examples in the model’s native tool-call format fixed it. We also capped capabilities that were already at 100% at a small replay sample instead of dropping them. Even a small replay sample protects a capability.

Train the reasoning mode deliberately. The default data formatting prefixes every answer with an empty reasoning block. That effectively teaches the model to answer without thinking, which hurts a model deployed in hybrid reasoning mode. If reasoning is part of your deployment, include reasoning traces in training data for the calculation-heavy categories.

For video, serving configuration matters as much as weights. Our initial serving setup sampled too few frames for the model to count parts on a conveyor reliably. Configuring frame sampling to match the source (12 fps, up to 256 frames, with video token pruning disabled) was a prerequisite for meaningful video evaluation. We also raised the training-time frame budget so the adapter learned from the same density of evidence it would see in production.

How Infosys Topaz with NVIDIA Nemotron 3.5 Super VL can help manufacturers operationalize multimodal AI
The capabilities in this evaluation map directly to agent workflows our manufacturing clients are building. We took a quality nonconformance agent as an example. It reads an inspection report, checks each characteristic against the drawing tolerance, and identifies the failed part and lot. It then calls the quality management system to open a nonconformance with the right fields. A second example is a line-performance agent that reads HMI screens and OEE dashboards, computes losses correctly, and escalates faults. In both cases, the reasoning-heavy steps are where we saw the clearest advantage with NVIDIA Nemotron 3.5 Super VL.

The customization story matters just as much. Because NVIDIA Nemotron 3.5 Super VL is open and post-trainable with NVIDIA NeMo, clients can close their own specific failure modes on their own data. They can deploy the model on infrastructure they control, keeping drawings, quality records, and line video inside their environment.

What’s next
All the numbers in this post come from synthetic, rendered data. That was a deliberate choice, because it gave us exact labels and clean held-out splits, but it is also a limitation. Models fine-tuned on one generator’s rendering style may perform better on that style than on real plant imagery. Infosys Topaz will continue to

•  Validate on a hand-labelled set of real plant images, HMI captures, and line video, to be reported alongside the synthetic results

•  Build a larger and harder held out set focused on counting and visual inspection, so fine-tuning gains can be measured with more statistical power

Authored by:
Prakruthi B Gowda – Senior Solution Architect, NVIDIA, Meenakshi Venkatesan – Principal Architect, Infosys Topaz CoE, Shantanu Todkar – Specialist Programmer, Infosys Topaz CoE, Lokesh Mehta – Specialist Programmer, Infosys Topaz CoE
Contributors:
Guruprasad NV – AVP, Infosys Topaz CoE, Kiran Kori – Lead Consultant, Infosys Topaz CoE, Pratik Dongare – Specialist Programmer, Infosys Topaz CoE, Rajesh Jalda – Specialist Programmer, Infosys Topaz CoE