Our Approach

AdequateInfosoft assembled a specialized Jetson AI team and delivered the full pipeline in structured phases.

Phase 1 Multimodal DeepStream Pipeline

Real-time Multimodal AI Pipeline using the Jetson Orin NX and NVIDIA DeepStream SDK.

Built a real-time multimodal AI pipeline on Jetson Orin NX with NVIDIA DeepStream SDK to support three simultaneous inference streams, person detection and co-viewer count, gaze estimation (if eyes are in the TV zone or on secondary device), and pose estimation to differentiate between active and passive viewers.

Created custom GStreamer Plugins for ingesting and processing low-latency frames.

Phase 2 Fine-Tuning the Model with TAO Toolkit

Supervised conventional models perform poorly when used in the home environment, including conditions of low/light, large/distance and infrared input.

We used NVIDIA's TAO Toolkit to fine-tune these pre-trained detection models for each of these conditions using specifically designed datasets for each, thereby significantly increasing the accuracy for the intended environment.

Phase 3 Model Optimization to Meet 15W Power Budget

In order to conform to a strict power budget, we used TensorRT INT8 quantization and structural pruning on all three models to provide a direct means of creating a fully concurrent three-model, inference-only system that operated within a 15-Watt power envelope on Jetson Orin NX, with a less than 40 millisecond per frame imager latency.

Phase 4 Audio ACR Integration

We integrated passive audio fingerprinting using Chromaprint alongside a neural hashing layer for content matching. The audio pipeline ran as a parallel thread, identifying TV content in real time and tagging metadata without any audio storage.

Phase 5 Zero-Video Storage & Privacy Architecture

All frames were processed at RAM level, and were discarded immediately after being inferred. No video, no faces and no raw sensor data was ever saved to disk. The only data that was encrypted and sent to the client's cloud bidding engine via a secure MQTT pipeline were structured attention metadata (viewer count, attention score, content ID).

Results Produced

• Three-model concurrent pipeline operates in real-time at less than 15W on Jetson Orin NX
• Inferencing lag of less than 40ms on person detection, gaze, and pose
• Fully validated zero-video storage architecture contains no PII or raw frames
• Audio ACR scores of more than >92% for live broadcast or streaming content
• Production ready with the capability for over-the-air upgrades and encryption of metadata transport.

Why AdequateInfosoft

Expertise in building out DeepStream pipelines, TAO Toolkit model tuning, TensorRT optimization and building privacy-focused systems are hard to find together in one place. But that's exactly what AdequateInfosoft delivered - on time and on budget - for this project.

Contact AdequateInfosoft for a FREE consultation if you would like to explore building a similar edge AI system.

Our Approach & Infrastructure Design (Utilizing NVIDIA AI Technology)

We have developed a comprehensive AI service through NVIDIA's AI Enterprise suite of technologies to provide us with a unified, individualised platform that enables us to handle high volumes of inbound call traffic with natural and lifelike pacing:

Centralised Inference & LLM Orchestration

Leveraging NVIDIA NIM (Inference Microservices) allowed us to deploy and host a number of domain-specific optimized models. By running NVIDIA TensorRT-LLM inside of the NIM containers, we achieved the highest throughput possible on the underlying GPU hardware and crushed Time-To-First-Token (TTFT) latency to less than 400ms.

Speech-to-Text & Translation

The ability to detect multiple languages, and seamlessly switch between English, Arabic and Hindi at any time during a call, was enabled by NVIDIA Riva ASR (Automated Speech Recognition) and NMT (Neural Machine Translation) micro services.

This provided us with the ability to transcribe and match languages with ultra-low latency on-premise/private cloud, while maintaining complete control over data privacy.

High-Fidelity Audio Synthesis

The polished text outputs were sent to Deepgram and ElevenLabs API integrations in order to synthesise three unique, lifelike female sounding voices.

Multi-Agent Routing

n8n (open-source automation platform) was used as the logic engine to dynamically manage workflow state and pass context between our five specialised agents (Inbound Qualifier, Inbound Handler, Appointment Confirmer, No-Show Recovery, and High-Priority Safety/Escalation Agent).

The 14 day pipeline for execution

To meet the deadline, we achieved decoupled development using modular, containerized, NVIDIA microservices:

• Day 1-4 - Setup NVIDIA AI Enterprise environments, mapping SIP Trunks and GSM Gateways for 6 separate phones.
• Day 5-9 - Fine-tuning Safety Agent prompt guarding lines, configuring multi-lingual, Riva customized healthcare terminology vocabularies, Mapping n8n Routing triggers
• Day 10-14 - Stress Tests at 200 concurrent streaming connections to drive high traffic through the Triton Inference Server (Tracking ~50k minutes/month) and Fine-tuning weightings on prompt response.

Results & Impact on Business

On Day 14, the platform was up and running. The use of NVIDIA's accelerated architecture enabled the system to seamlessly and efficiently absorb 82% of typical Patient Queries/Booking Changes.

Additionally, due to using the NVIDIA Riva + TensorRT-LLM stack, the Safety/Escalation agent accurately parsed panic state voice transcripts in real time, without lag, and routed Life-threatening calls to human agents.

The no-show recovery campaigns that were run on behalf of our clients resulted in a 24% recovery of dormant leads in the first week. This success led the client to secure a continuing retainer contract for 24/7 support.