Jetson AGX Orin Autonomous Cart Development with Camera–LiDAR Sensor Fusion

The Operational Challenge

The aim was to design a self-driven indoor cart for transporting goods from one location to another while navigating hallways and places alongside pedestrians, trolleys, forklifts, and other machines.

The cart was supposed to function successfully under unstable conditions of the environment. Some examples of such circumstances include the following:

• Narrow shelves, loading bays, and points of intersection
• Glare from the floor, insufficient illumination, and difficulties arising out of temporary obstacles
• Humans walking on the route of the cart
• Moving pallets and shelves
• The cart losing connection with a Wi-Fi network from time to time
• The need for drivers to stop, rest, and take control of the cart.

Hence, an intelligent system was needed, which included more than just the visual recognition of objects. It must contain a well-coordinated robotics structure where the various components such as sensors, localization, mapping, navigation and control of wages and safety actions work together on a timely basis.

Architecture Plan: Implementing Jetson AGX Orin in Robotics

The developed system incorporates NVIDIA Jetson AGX Orin as AI and robotics 'main processor'.

The performance metrics indicate that Jetson AGX Orin can achieve the AI performance of up to 275 TOPS due to the use of an Ampere-based GPU equipped with Tensor Cores.

Such performance proves valuable when the autonomous robot is required to process several streams of images, perform neural network inference, and accomplish robotics tasks without cloud-based transfer of important perception data.

The autonomous cart architecture includes the following layers:

Layer	Main components	Purpose
Perception	Stereo/RGB cameras, 2D or 3D LiDAR, IMU, wheel encoders	Detect the environment and estimate movement
Edge compute	NVIDIA Jetson AGX Orin, NVMe storage, optional MCU	Run sensor processing, AI inference and ROS 2 nodes
Robot control	Motor drivers, safety controller, emergency stop, battery BMS	Control speed, steering, braking and safe stop states
Robotics software	ROS 2, localization, mapping, Nav2, obstacle layer	Plan and execute safe routes
Fleet and operations	Web dashboard, task API, logs and diagnostics	Dispatch jobs, monitor health and maintain the fleet

The Jetson platform utilizes the NVIDIA JetPack software which is based on Ubuntu, CUDA-based workloads, TensorRT model optimization, OpenCV processing together with ROS 2 application nodes.

A separate microcontroller can be used independently for real-time functions such as motor control, encoder data collection, monitoring of batteries and watchdog operations.

Camera–LiDAR Sensor Fusion Design

Camera and LiDAR sensors provide different but complementary information. LiDAR produces accurate distance measurements and geometric obstacle contours, often remaining useful where image texture is poor. Cameras provide semantic detail: they can help distinguish a person from a pallet, identify floor markings, interpret signs and classify relevant operating zones.

The sensor-fusion pipeline begins with physical sensor placement and calibration. LiDAR placement must minimize blind areas caused by the cart chassis, payload shelf or protective enclosure.

Cameras need a stable field of view and appropriate exposure control for bright doorways, low-light storage zones and flickering industrial lighting.

Engineers establish the geometric relationship between sensors through intrinsic and extrinsic calibration:

• Camera intrinsics define focal length, optical center and lens-distortion parameters.
• LiDAR-to-camera extrinsics define rotation and translation between coordinate frames.
• The IMU and wheel encoder frames are aligned with the robot base frame.
• Timestamps are synchronized so sensor observations correspond to the same physical moment.

LiDAR points can be converted into coordinates of the camera frame as well as be projected on the image.

This makes it possible for the system to give a person/object a distance value. In this way camera detections can be sorted based on their importance and urgency taking into account the distance, direction of movement, and potential collision threats posed by confirmed detections.

In order to guarantee navigation safety, the robot is being updated with a local cost map that contains data about LiDAR returns, detected objects and other keep-out areas that have been arranged.

In case a person steps in the predicted path of the cart, it can slow down, malfunction, or change its route following the rules set beforehand.

Localization, Mapping and Navigation

An autonomous cart relies on accurate localization. If a robot is able to recognize various obstacles on its way, yet it does not know its own position, then carrying out a delivery task cannot be accomplished in a secure manner.

For the first-time operation of the cart, the engineering team needs to create a map for the area in which the cart will operate. The map is created with the use of SLAM-technology based on LiDAR technology.

During the process of operation of the cart, it works based on localization and created map.

A practical ROS 2 software structure can include:

• robot_state_publisher for the cart's coordinate-frame model
• LiDAR, camera, IMU and encoder driver nodes
• Sensor fusion or state-estimation node for odometry
• SLAM or map-localization node
• Nav2 planner, controller, recovery and behavior-tree nodes
• Perception node for people, obstacles and zone awareness
• Motor-control bridge for commands, feedback and fault state reporting
• Fleet-management API client for tasks and telemetry

Nav2 provides the navigation framework for route planning and local obstacle avoidance. The global planner determines a route from the current location to the selected destination. The local controller continually updates motion commands using current sensor data.

This allows the cart to pause for a person, steer around a temporary trolley where permitted, or trigger a controlled recovery when the route is blocked.

For high-confidence use cases, the robot does not assume that every condition can be solved autonomously. It uses well-defined responses such as slow-down zones, stop zones, operator notifications, manual takeover, return-to-home behavior and safe fault states.

Edge AI Perception and Model Optimization

The Jetson AGX Orin platform provides the capability of executing perception models on the cart. In a common system, either YOLO object detection model or some specially trained model is used to detect people, pallets, vehicles, cones, or any delivery area signs.

The model used should comply with the working environment and cannot be considered as a universal model which doesn't require representative data and field validation.

As part of the proper Infosoft work on the AI implementation, we utilize image-data processing, model selection, development of the inference pipeline, conversion via TensorRT, and work on accuracy profiling and monitoring.

An FP16 or INT8 may be used depending on the latency and accuracy requirements.

The perception pipeline produces more than bounding boxes. It publishes tracked object locations, confidence values, estimated distance and motion context for use by the navigation and safety layers.

A human-detection event may reduce maximum speed before the cart reaches a stopping threshold. A persistent blocked aisle may generate a re-route request or an operator alert.

Field Testing and Validation

Field testing starts in an environment that is controlled before the real trial runs happen in operations.

The development team ensures that the various layers of hardware and software work in order. These layers include the following: sensor frames, timestamp synchronization, odometry drift, braking performance, emergency stop mechanism, recovery phase after network drop, and behavior of the system after detecting battery low.

Field validation plan includes the following elements:

• Testing with static obstacles and free-path
• Crossing pedestrians and approach with obstacle avoidance at different speeds
• Testing in low light and under glare/reflection conditions
• Navigating narrow corridors and loading areas
• Simulations with obstacles, change of routes and recoveries
• Behavior under Wi-Fi drop and restart of services
• Repeated routes testing with log and ticketing
• Handover, remote monitoring and incident procedures

Production readiness is measured against agreed acceptance criteria such as route completion, localization stability, stop behavior, detection quality, recovery time and maintainability.

Safety validation must reflect the specific cart, payload, speed, operating area and applicable regional requirements. AI detection alone is not a substitute for appropriately engineered safety controls.