This is the final part of the Project Ogre series. We've built the robot, created a digital twin, built maps with SLAM, and trained an RL policy. Now we bring it all together with Nav2 for autonomous waypoint navigation.

Project Goal: Navigate autonomously to user-specified waypoints while avoiding obstacles, using the saved map and trained policy.

📦 Full Source Code: Nav2 configuration and launch files are available at github.com/protomota/ogre-slam

The video below shows Nav2 autonomous navigation in action. On the left is RViz displaying the map, costmaps, and planned path. On the right is Isaac Sim where the robot navigates autonomously through the maze using the trained RL policy to execute velocity commands.

▶ Watch the video on YouTube

The Nav2 Stack

Nav2 (Navigation 2) is ROS2's navigation framework. It handles:

1. Localization: Where is the robot on the map?

2. Global Planning: What's the best path to the goal?

3. Local Planning: How do we follow that path while avoiding new obstacles?

4. Recovery: What to do when stuck?

For Project Ogre, the architecture looks like:

Saved Map → AMCL (localization)
                    ↓
User Goal → Global Planner → Local Planner → /cmd_vel
                                                  ↓
                                          Policy Controller
                                                  ↓
                                           Wheel Commands

The RL policy sits between Nav2's velocity commands and the actual wheel control, translating ideal commands into optimal wheel velocities.

Understanding Costmaps

Nav2 uses costmaps to represent navigable space. Each cell in the grid has a cost (0-254) indicating traversal difficulty:

Cost   Name            Meaning
-----  --------------  ----------------------------------------
0      FREE_SPACE      Safe to traverse
1-252  INFLATED        Near obstacles (increasingly expensive)
253    INSCRIBED       Inside robot's radius (collision likely)
254    LETHAL          Obstacle (certain collision)
255    NO_INFORMATION  Unknown space

Two Costmaps

Global Costmap (/global_costmap/costmap)

• Covers the entire map

• Used for path planning

• Updates from static map + sensors

Local Costmap (/local_costmap/costmap)

• Rolling 3m × 3m window around robot

• Used for reactive obstacle avoidance

• Updates continuously from sensors

Inflation

Obstacles are "inflated" by the robot's radius to prevent collision. The key parameters:

robot_radius: 0.20        # Inscribed radius
inflation_radius: 0.35    # How far to spread costs
cost_scaling_factor: 3.0  # How quickly costs drop off

If the robot clips walls, increase inflation_radius. If it can't navigate narrow passages, decrease it.

Nav2 Configuration

The full configuration lives in config/nav2_params.yaml. Here are the key sections:

AMCL (Localization)

AMCL (Adaptive Monte Carlo Localization) tracks the robot's position on the map using laser scan matching:

amcl:
  ros__parameters:
    base_frame_id: "base_link"
    odom_frame_id: "odom"
    scan_topic: /scan
    min_particles: 500
    max_particles: 2000
    update_min_d: 0.1  # Update after 10cm movement
    update_min_a: 0.2  # Update after ~11° rotation

Global Planner

We use Smac 2D for global path planning - it handles tight spaces better than the default NavFn:

planner_server:
  ros__parameters:
    planner_plugins: ["GridBased"]
    GridBased:
      plugin: "nav2_smac_planner/SmacPlanner2D"
      tolerance: 0.50  # Search 50cm around goal
      max_planning_time: 2.0

The tolerance parameter is crucial - it defines how close the robot needs to get to the exact goal point. Higher tolerance helps when goals are near obstacles.

Local Controller

The local controller follows the global path while avoiding obstacles. For mecanum robots, we use a velocity-based approach:

controller_server:
  ros__parameters:
    controller_plugins: ["FollowPath"]
    FollowPath:
      plugin: "dwb_core::DWBLocalPlanner"
      max_vel_x: 0.3
      max_vel_y: 0.3     # Strafing enabled for mecanum
      max_vel_theta: 1.0
      min_speed_xy: 0.0
      min_speed_theta: 0.0

Note max_vel_y - this enables strafing, which is what makes mecanum navigation special.

Behavior Server (Recovery)

When the robot gets stuck, recovery behaviors kick in:

behavior_server:
  ros__parameters:
    behavior_plugins: ["spin", "backup", "wait"]
    spin:
      plugin: "nav2_behaviors/Spin"
    backup:
      plugin: "nav2_behaviors/BackUp"

The robot will try spinning, backing up, or waiting before giving up on a goal.

Running Navigation

In Isaac Sim

Terminal 1: Start Isaac Sim

• Load usds/ogre.usd

• Verify ROS2 Context has a matching Domain ID

• Press Play

Terminal 2: Launch Navigation

cd ~/ros2_ws && source install/setup.bash

ros2 launch ogre_slam navigation.launch.py \
  map:=~/ros2_ws/src/ogre-slam/maps/isaac_sim_map.yaml \
  use_sim_time:=true

The use_sim_time:=true flag is critical - without it, TF transforms will time out because the simulation clock doesn't match wall clock.

Terminal 3: Launch Policy Controller

conda deactivate
source ~/ros2_ws/install/setup.bash

ros2 launch ogre_policy_controller policy_controller.launch.py use_sim_time:=true

Terminal 4: Launch RViz (optional, for remote visualization)

./scripts/launch_isaac_sim_rviz.sh

On the Real Robot

cd ~/ros2_ws && source install/setup.bash
ros2 launch ogre_slam navigation.launch.py \
  map:=~/ros2_ws/src/ogre-slam/maps/my_map.yaml

This launches:

• AMCL localization with the saved map

• RPLIDAR + RealSense D435 sensors

• Full Nav2 stack

• Web teleop for manual override

Navigating in RViz

With navigation running, RViz becomes your command interface:

Step 1: Set Initial Pose

Click 2D Pose Estimate in the toolbar, then click and drag on the map to indicate:

• Where the robot is

• Which direction it's facing

AMCL needs this initial hint to start tracking correctly. As the robot moves, localization will refine automatically.

Step 2: Send Navigation Goals

Click Nav2 Goal (or 2D Goal Pose), then click on the map where you want the robot to go. Drag to set the final orientation.

The robot will:

1. Plan a global path (green line in RViz)

2. Follow the path using local planning

3. Avoid obstacles detected in real-time

4. Execute recovery behaviors if stuck

Manual Override

On the real robot, the web teleop interface at http://10.21.21.45:8080 provides manual control. Any teleop commands override Nav2's autonomous control.

The Complete Data Flow

Let's trace a navigation command through the entire system:

1. User clicks goal in RViz
         ↓
2. Global Planner computes path using costmap
         ↓
3. Local Controller generates /cmd_vel following path
         ↓
4. Policy Controller receives velocity command
         ↓
5. Neural network computes optimal wheel velocities
         ↓
6. /joint_command sent to robot/sim
         ↓
7. Robot moves, sensors update, localization refines
         ↓
   (loop continues until goal reached)

The trained RL policy adds value at step 5 - it learns compensation for wheel slip, motor delays, and mechanical imperfections that pure kinematics can't handle.

3D Obstacle Avoidance

The RealSense D435 provides depth data that the RPLIDAR can't see - obstacles below the 2D scan plane like:

• Table legs

• Stairs

• Low furniture

• Pets

The depth camera publishes a PointCloud2 to /camera_points, which feeds into the local costmap's voxel layer:

local_costmap:
  voxel_layer:
    plugin: "nav2_costmap_2d::VoxelLayer"
    observation_sources: scan camera
    camera:
      topic: /camera_points
      sensor_frame: camera_link
      max_obstacle_height: 0.5
      min_obstacle_height: 0.0

Now the robot can detect and avoid 3D obstacles in real-time.

Troubleshooting Navigation

"Legal potential but no path"

Cause: Goal is in an inflated zone near obstacles

Solutions:

1. Click goals further from walls

2. Increase tolerance in planner config

3. Switch to SmacPlanner2D (handles this better than NavFn)

Robot Clips Walls

Cause: Inflation radius too small

Solution: Increase inflation_radius in both costmaps (try 0.40-0.50m)

Robot Can't Enter Narrow Spaces

Cause: Inflation radius too large

Solution: Decrease inflation_radius (try 0.30m)

Localization Drifts

Cause: AMCL losing track, often from rapid motion or featureless areas

Solutions:

1. Move slower during navigation

2. Increase max_particles

3. Ensure good initial pose estimate

TF_OLD_DATA Errors

Cause: Clock synchronization issues

Solutions:

• In simulation: Ensure use_sim_time:=true on ALL nodes

• On real robot: Check that odometry and SLAM are publishing current timestamps

Robot Spins Forever

Cause: Recovery behavior stuck in loop

Solution: Cancel the goal and try a different destination. Check if there's an obstacle blocking all paths.

Performance on Jetson

Running the full Nav2 stack on Jetson Orin Nano requires some optimization:

Memory Management

global_costmap:
  resolution: 0.10  # Coarser than SLAM (0.05) to save memory
local_costmap:
  resolution: 0.05  # Finer for local avoidance

CPU Optimization

# Enable max performance mode
sudo /usr/bin/jetson_clocks

# Monitor resources
tegrastats

Disable Unnecessary Visualization

Don't run RViz on the Jetson - use remote visualization instead. Save those CPU cycles for navigation.

What We Built

Over these five posts, we created a complete autonomous mobile robot:

1. Hardware: Mecanum drive robot with Jetson Orin Nano, RPLIDAR, and RealSense

2. Odometry: Calibrated encoders with EKF sensor fusion

3. Mapping: SLAM-generated occupancy grids

4. Learning: RL-trained velocity controller

5. Navigation: Full Nav2 stack with 3D obstacle avoidance

The robot can now navigate autonomously to any point on the map, avoiding both static obstacles (walls from the map) and dynamic obstacles (detected in real-time by sensors).

Future Improvements

The system works, but there's always room to improve:

• IMU Integration: Adding an IMU would dramatically improve odometry, especially during rotation

• Dynamic Obstacle Tracking: Currently obstacles are binary - tracking moving obstacles would enable smarter planning

• Sim-to-Real Transfer: Fine-tuning the RL policy on real-world data

---

Project Ogre Summary:

Component     Technology
------------  --------------------------------
Computing     Jetson Orin Nano
Drive         Mecanum wheels (omnidirectional)
2D Sensing    RPLIDAR A1
3D Sensing    RealSense D435
SLAM          slam_toolbox
Localization  AMCL
Planning      Nav2 (Smac2D + DWB)
Control       RL-trained policy
Simulation    NVIDIA Isaac Sim

Key Launch Commands:

# Mapping (simulation)
ros2 launch ogre_slam mapping.launch.py use_sim_time:=true

# Mapping (real robot)
./scripts/launch_mapping_session.sh

# Navigation (simulation)
ros2 launch ogre_slam navigation.launch.py \
  map:=~/ros2_ws/src/ogre-slam/maps/isaac_sim_map.yaml \
  use_sim_time:=true

# Navigation (real robot)
ros2 launch ogre_slam navigation.launch.py \
  map:=~/ros2_ws/src/ogre-slam/maps/my_map.yaml

# Policy controller
ros2 launch ogre_policy_controller policy_controller.launch.py

Thanks for following along with Project Ogre. If you build something similar or have questions, let me know in the comments.

Credits

• slam_toolbox: Steve Macenski

• robot_localization: Tom Moore

• RPLIDAR ROS: SLAMTEC

• Isaac Lab: NVIDIA

• Nav2: Open Robotics & Nav2 maintainers

This is Part 4 of the Project Ogre series. In Part 3, we built maps using SLAM. Now we'll train a neural network to optimally control the mecanum wheels using reinforcement learning in NVIDIA Isaac Lab.

Project Goal: Train an RL policy that converts velocity commands (vx, vy, vtheta) into optimal wheel velocities, learning the robot's real dynamics.

📦 Full Source Code: Training environment and scripts are available at github.com/protomota/ogre-lab

The video below shows Isaac Lab training 4096 parallel instances of the robot simultaneously. Each instance learns from its own experiences, accelerating the training process dramatically compared to single-environment training.

▶ Watch the video on YouTube

Why Use Reinforcement Learning?

Mecanum wheel kinematics have a clean theoretical model: given a desired body velocity (vx, vy, angular), you can compute the required wheel velocities using inverse kinematics. So why train a neural network?

The theory doesn't match reality.

Real robots have:

• Wheel slip on different surfaces

• Motor response delays and non-linearities

• Weight distribution asymmetries

• Mechanical imperfections in wheel alignment

An RL policy learns to compensate for these factors by experiencing thousands of trials in simulation. It discovers that "to go straight forward at 0.3 m/s, the right wheels need slightly higher velocity to compensate for weight imbalance."

The trained policy sits between Nav2's velocity commands and the actual wheel controllers, translating ideal commands into commands that achieve the desired behavior on the real robot.

Isaac Lab Architecture

Isaac Lab is NVIDIA's GPU-accelerated robot learning framework built on Isaac Sim. Key features:

• Parallel environments: Train on 1024+ robots simultaneously

• GPU physics: PhysX runs entirely on GPU for massive speedup

• RSL-RL integration: Works with the RSL-RL library for PPO training

The training loop runs at ~30Hz control rate with 120Hz physics, training policies in minutes rather than hours.

The Training Environment

The environment is defined in ogre_navigation_env.py. Let's break down the key components:

Observation Space (10 dimensions)

The policy sees:

Index  Description
-----  -----------------------------------------
0-2    Target velocity command (vx, vy, vtheta)
3-5    Current base velocity (vx, vy, vtheta)
6-9    Current wheel velocities (fl, fr, rl, rr)

This gives the policy enough information to understand both what it should achieve and what the robot is currently doing.

Action Space (4 dimensions)

The policy outputs wheel velocity targets for all four wheels:

• Front-left (FL)

• Front-right (FR)

• Rear-left (RL)

• Rear-right (RR)

Raw policy outputs are in [-1, 1] and scaled by action_scale (8.0 rad/s in the working config) to get actual wheel velocities.

Reward Function

The policy learns through reward signals:

# Main velocity tracking reward
vel_error = target_vel - current_vel
rew_vel_tracking = 2.0 * exp(-vel_error_norm / 0.25)

# Uprightness reward (+1 upright, -1 flipped)
rew_uprightness = 1.0 * up_z

# Wheel symmetry penalty (prevents veering during straight motion)
rew_symmetry = -1.0 * (left_avg - right_avg)^2

The velocity tracking reward uses exponential decay - perfect tracking gives maximum reward, with diminishing returns as error increases. The symmetry penalty discourages the policy from commanding asymmetric wheel velocities when the target is straight-line motion.

Episode Structure

• Duration: 10 seconds per episode

• Termination: Episode ends early if robot flips (base height < 0.02m)

• Reset: Random target velocities sampled each episode

Random targets force the policy to generalize across the full velocity space rather than memorizing a single trajectory.

Robot Configuration

The USD robot model must match the real hardware:

OGRE_MECANUM_CFG = ArticulationCfg(
    spawn=sim_utils.UsdFileCfg(
        usd_path="~/ros2_ws/src/ogre-slam/usds/ogre_robot.usd",
    ),
    init_state=ArticulationCfg.InitialStateCfg(
        pos=(0.0, 0.0, 0.04),  # Wheel axle height
    ),
    actuators={
        "wheels": ImplicitActuatorCfg(
            joint_names_expr=["fl_joint", "fr_joint", "rl_joint", "rr_joint"],
            effort_limit=10.0,
            stiffness=0.0,
            damping=10.0,
        ),
    },
)

Key parameters:

• wheel_radius: 0.040m (40mm)

• wheelbase: 0.095m (95mm)

• track_width: 0.205m (205mm)

The Wheel Sign Challenge

One non-obvious issue: the USD model has the right wheels (FR, RR) with opposite joint axis orientation from the left wheels. Sending [+10, +10, +10, +10] to all wheels makes the robot spin instead of moving forward.

The solution is sign correction applied in both training and deployment:

Training (_apply_action):

corrected_actions = self.actions.clone()
corrected_actions[:, 0] *= -1  # FR (right wheel)
corrected_actions[:, 1] *= -1  # RR (right wheel)
self.robot.set_joint_velocity_target(corrected_actions)

Deployment (Isaac Sim action graph): The same correction must be applied when running the trained policy - multiply FR and RR velocities by -1.

This creates a normalized action space where [+,+,+,+] always means "all wheels forward" regardless of joint axis conventions.

Running Training

Prerequisites

• Isaac Sim 5.0+

• Isaac Lab installed at ~/isaac-lab/IsaacLab

• Conda environment env_isaaclab

Step 1: Sync the Environment

The training environment must be copied to Isaac Lab's task directory:

cd ~/ogre-lab
./scripts/sync_env.sh

This copies ogre_navigation/ to Isaac Lab's source tree where it can be discovered and imported.

Step 2: Run Training

conda activate env_isaaclab
cd ~/ogre-lab
./scripts/train_ogre_navigation.sh

Default settings:

• 4096 parallel environments

• 1000 iterations

• Headless mode (no visualization)

For debugging, run with fewer environments and visualization:

./scripts/train_ogre_navigation.sh 64 100

Step 3: Monitor with TensorBoard

TensorBoard provides real-time visualization of training progress. In a separate terminal:

conda activate env_isaaclab
tensorboard --logdir ~/isaac-lab/IsaacLab/logs/rsl_rl/ogre_navigation/

Open http://localhost:6006 in your browser:

Key metrics to watch:

• Train/mean_reward: The average reward across all parallel environments. Should steadily increase as the policy learns. A healthy training run shows rewards climbing from near-zero to positive values.

• Train/mean_episode_length: How long episodes last before termination. Should approach the maximum (300 steps = 10 seconds at 30Hz). If episodes are short, robots are flipping or hitting termination conditions.

• Loss/value_function: The critic's prediction error. Should decrease over time as the value network learns to estimate expected returns. Spikes are normal early in training.

• Loss/surrogate: The PPO policy loss. Fluctuates during training but should generally trend downward.

TensorBoard is invaluable for diagnosing training issues. If rewards plateau, you might need to adjust reward weights. If episode lengths stay low, the action scale might be too aggressive.

What to Expect

Learning iteration 50/100
  Mean reward: 1.25        # Positive and increasing = good
  Mean episode length: 299  # Near 300 = robots staying upright

If rewards are very negative (-20000), the velocity targets are too aggressive - reduce max_lin_vel and max_ang_vel. If episode lengths are short, robots are flipping - reduce action_scale.

Exporting the Trained Policy

Once training completes, export the model for deployment:

cd ~/ogre-lab
./scripts/export_policy.sh

This runs Isaac Lab's play.py to verify the policy works, then exports:

• policy.onnx - For ONNX runtime inference

• policy.pt - PyTorch JIT format

The exported models go to the training run's exported/ directory.

Deploying to ROS2

Copy the trained model to the ROS2 workspace:

./scripts/deploy_model.sh --rebuild

This copies the latest exported model to ogre-slam/ogre_policy_controller/models/ and rebuilds the ROS2 package.

Testing in Isaac Sim

Terminal 1: Isaac Sim

• Load ogre.usd

• Press Play

Terminal 2: Policy Controller

conda deactivate  # Exit conda - ROS2 uses system Python
source ~/ros2_ws/install/setup.bash
ros2 launch ogre_policy_controller policy_controller.launch.py

Terminal 3: Test Commands

# Forward motion (0.3 m/s)
ros2 topic pub /cmd_vel_smoothed geometry_msgs/msg/Twist \
  "{linear: {x: 0.3, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}" -r 10

# Strafe left
ros2 topic pub /cmd_vel_smoothed geometry_msgs/msg/Twist \
  "{linear: {x: 0.0, y: 0.3, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}" -r 10

# Rotate
ros2 topic pub /cmd_vel_smoothed geometry_msgs/msg/Twist \
  "{linear: {x: 0.0, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 1.0}}" -r 10

Watch the robot in Isaac Sim. It should move smoothly in the commanded direction.

The Data Flow

Nav2 Planner → /cmd_vel → Policy Controller → /joint_command → Isaac Sim/Robot
                               ↓
                         policy.onnx
                               ↓
                    [FL, FR, RL, RR] wheel velocities

The policy controller subscribes to /cmd_vel_smoothed, runs the velocity command through the neural network, and publishes individual wheel velocities to /joint_command.

Troubleshooting Training

Robots Exploding/Flying

Cause: Physics instability, usually from collision issues or excessive forces

Solutions:

• Check USD model has proper collision meshes

• Reduce action_scale (try 6.0 instead of 8.0)

• Increase physics solver iterations

Very Negative Rewards

Cause: Velocity targets exceed what the robot can achieve

Solutions:

• Reduce max_lin_vel and max_ang_vel

• Check action_scale allows sufficient wheel velocity

Short Episode Lengths

Cause: Robots flipping over

Solutions:

• Reduce action_scale

• Add stronger uprightness reward

• Check wheel friction parameters

Policy Doesn't Generalize

Cause: Training converged to local minimum

Solutions:

• Increase training iterations

• Add curriculum learning (start easy, increase difficulty)

• Tune reward scales

---

Quick Reference

Key Files

File                       Location                                  Purpose
-------------------------  ----------------------------------------  --------------------
ogre_navigation_env.py     ogre-lab/isaaclab_env/                    Training environment
rsl_rl_ppo_cfg.py          ogre-lab/isaaclab_env/agents/             PPO hyperparameters
policy.onnx                ogre-slam/ogre_policy_controller/models/  Deployed model
policy_controller_node.py  ogre-slam/ogre_policy_controller/         ROS2 inference node

Commands

# Train policy
./scripts/train_ogre_navigation.sh

# Export trained model
./scripts/export_policy.sh

# Deploy to ROS2
./scripts/deploy_model.sh --rebuild

# Run policy controller
ros2 launch ogre_policy_controller policy_controller.launch.py

What's Next

With a trained policy deployed, Part 5 brings everything together with Nav2 navigation. The policy controller integrates seamlessly - Nav2 plans paths and generates velocity commands, the RL policy executes them optimally, and the robot navigates autonomously through the mapped environment.

This is Part 3 of the Project Ogre series. In Part 1, we built the mecanum drive robot, and in Part 2, we created a digital twin in Isaac Sim. Now we'll use both to build maps using SLAM (Simultaneous Localization and Mapping).

Project Goal: Create accurate occupancy grid maps using slam_toolbox that can be used for autonomous navigation with Nav2.

📦 Full Source Code: All ROS2 packages and configuration files are available at github.com/protomota/ogre-slam

The video below shows SLAM mapping in action. On the left is RViz displaying the occupancy grid map as it's being built in real-time. On the right is Isaac Sim where I'm teleoperating the simulated robot around the maze. As the robot explores, the LIDAR scans are processed by slam_toolbox to incrementally construct the map.

▶ Watch the video on YouTube

What is SLAM?

SLAM (Simultaneous Localization and Mapping) solves a chicken-and-egg problem: to know where you are, you need a map. To build a map, you need to know where you are. SLAM algorithms solve both simultaneously by matching sensor data against an incrementally-built map.

For Project Ogre, we use slam_toolbox - a robust 2D SLAM implementation that works well with LIDAR sensors. It takes laser scan data from the RPLIDAR and odometry estimates from our encoders, then produces:

1. A map (occupancy grid showing obstacles and free space)

2. A localization estimate (where the robot is on that map)

The Two-Computer Setup

Project Ogre uses a split architecture:

Host Computer (Development)

• Runs NVIDIA Isaac Sim for simulation

• Runs RViz for visualization

• Handles resource-intensive debugging

Jetson Orin Nano (Robot)

• Runs on the physical robot

• Handles real sensor data

• Executes navigation commands

Both computers use the same ROS_DOMAIN_ID to communicate over the network. This lets me visualize the robot's SLAM progress in RViz on my development machine while the actual processing happens on the Jetson.

Configuring slam_toolbox

The SLAM configuration lives in config/slam_toolbox_params.yaml. Here are the key parameters tuned for Project Ogre's hardware:

Frame Configuration

odom_frame: odom
map_frame: map
base_frame: base_link
scan_topic: /scan
mode: mapping

These define the TF frames SLAM uses and where it gets laser scan data.

Scan Processing

transform_publish_period: 0.02    # 50 Hz TF publishing
map_update_interval: 2.0          # Update map every 2 seconds
resolution: 0.05                  # 5cm grid cells
max_laser_range: 12.0             # RPLIDAR A1 max range
minimum_time_interval: 0.2        # Minimum time between scans

The map_update_interval and resolution are tuned for the Jetson's memory constraints. Publishing the map every 2 seconds instead of continuously reduces CPU load significantly.

Critical: Minimum Travel Thresholds

minimum_travel_distance: 0.0
minimum_travel_heading: 0.0

These are set to zero on purpose. By default, slam_toolbox only processes new scans when the robot has moved a minimum distance or rotated a minimum angle. With our noisy 2 PPR encoders, this caused issues - the map would stop updating because odometry drift made SLAM think the robot hadn't moved.

Setting both to 0.0 forces SLAM to process scans based on time only (minimum_time_interval), not odometry.

Scan Matching Parameters

correlation_search_space_dimension: 2.0
correlation_search_space_resolution: 0.01
distance_variance_penalty: 0.3
angle_variance_penalty: 0.5

These control how SLAM matches new scans against the existing map. Lower variance penalties mean "trust scan matching more than odometry." Given our encoder limitations, we want SLAM to rely more heavily on the LIDAR data.

Loop Closure

do_loop_closing: true
loop_search_maximum_distance: 6.0
loop_match_minimum_response_coarse: 0.45
loop_match_minimum_response_fine: 0.55

Loop closure happens when the robot revisits a previously-seen location. SLAM recognizes the overlap and corrects accumulated drift. These parameters balance between catching real loop closures and avoiding false positives.

Mapping in Isaac Sim

Before mapping with the real robot, I develop and test in Isaac Sim. This catches configuration issues without burning through battery life or risking hardware damage.

Step 1: Start Isaac Sim

Load the usds/ogre.usd scene and verify the ROS2 Context node has a matching Domain ID. Press Play to start the simulation.

Step 2: Launch SLAM

cd ~/ros2_ws
source install/setup.bash

ros2 launch ogre_slam mapping.launch.py \
  use_rviz:=true \
  use_teleop:=false \
  use_odometry:=false \
  use_ekf:=false \
  use_sim_time:=true

Note the flags:

• use_sim_time:=true - Sync with Isaac Sim's clock

• use_odometry:=false - Isaac Sim provides odometry directly

• use_ekf:=false - Simulated odometry is clean, no filtering needed

Step 3: Launch the Policy Controller

The policy controller converts Twist commands to individual wheel velocities:

ros2 launch ogre_policy_controller policy_controller.launch.py

Step 4: Drive and Map

ros2 run teleop_twist_keyboard teleop_twist_keyboard \
  --ros-args -r /cmd_vel:=/cmd_vel_smoothed

Drive slowly around the environment. Watch RViz as the map builds in real-time. Make sure to create loop closures by revisiting your starting position.

Step 5: Save the Map

ros2 run nav2_map_server map_saver_cli \
  -f ~/ros2_ws/src/ogre-slam/maps/isaac_sim_map

This creates two files:

• isaac_sim_map.yaml - Metadata (resolution, origin, etc.)

• isaac_sim_map.pgm - The actual map image

Mapping on the Real Robot

Once SLAM works in simulation, deploying to the Jetson is straightforward:

Step 1: Launch the Mapping Session

SSH into the robot and run:

cd ~/ros2_ws/src/ogre-slam
./scripts/launch_mapping_session.sh

This unified script launches:

• RPLIDAR driver

• Encoder-based odometry node

• EKF sensor fusion

• slam_toolbox (mapping mode)

• Web teleop interface

Step 2: Drive via Web Interface

Open a browser on any device on the network:

http://10.21.21.45:8080

The web interface provides joystick-style control. Drive slowly and smoothly - jerky movements confuse the encoder-based odometry.

Tips for Good Maps:

• Move at consistent, slow speeds

• Avoid wheel slippage (especially on smooth floors)

• Create multiple loop closures

• Drive the same area from different angles

Step 3: Save the Map

ros2 run nav2_map_server map_saver_cli \
  -f ~/ros2_ws/src/ogre-slam/maps/my_map

Remote Visualization

Running RViz on the Jetson consumes resources needed for SLAM. Instead, run RViz remotely:

On the robot:

ros2 launch ogre_slam mapping.launch.py use_rviz:=false

On your development machine:

cd ~/ros2_ws/src/ogre-slam
./scripts/remote_launch_slam_rviz.sh

The script sets up the ROS_DOMAIN_ID and launches RViz with pre-configured displays for:

• LaserScan (/scan)

• Map (/map)

• TF frames

• Robot footprint

Troubleshooting

Map Not Updating

Symptom: RViz shows an empty or frozen map

Cause: Usually odometry issues. slam_toolbox needs meaningful position changes to trigger scan processing.

Solution:

1. Verify /odom shows changing positions when driving:

   ros2 topic echo /odom --once | grep -A 3 "position"

1. Check gear_ratio calibration (Part 1)

2. Ensure minimum_travel_distance: 0.0 in config

SLAM Losing Track

Symptom: Map suddenly rotates or shifts incorrectly

Cause: Scan matching failed, possibly from driving too fast or odometry drift

Solution:

1. Drive slower

2. Increase scan_buffer_size for more reference scans

3. Lower link_match_minimum_response_fine to accept weaker matches

High CPU Usage

Symptom: Jetson running hot, sluggish response

Solution:

1. Increase map_update_interval (try 3.0 or 5.0 seconds)

2. Increase resolution (0.10 instead of 0.05)

3. Disable RViz on the Jetson

No Loop Closure

Symptom: Drift accumulates, map doesn't correct when revisiting areas

Solution:

1. Ensure do_loop_closing: true

2. Lower loop_match_minimum_response_coarse to detect more candidates

3. Increase loop_search_maximum_distance for wider search

Understanding the Map Output

This is the 2D SLAM map from the maze I created for my robot within Isaac Sim:

The saved map consists of:

isaac_sim_map.yaml:

image: isaac_sim_map.pgm
mode: trinary
resolution: 0.05
origin: [-10.0, -10.0, 0.0]
negate: 0
occupied_thresh: 0.65
free_thresh: 0.25

isaac_sim_map.pgm: A grayscale image where:

• White (255): Free space - safe to traverse

• Black (0): Occupied - obstacles detected

• Gray (205): Unknown - not yet observed

The resolution (0.05) means each pixel represents a 5cm × 5cm area. The origin defines where the map's (0,0) pixel is in world coordinates.

---

Quick Reference

ROS2 Topics

Topic               Purpose
------------------  -----------------------------------
/scan               RPLIDAR laser scans (input to SLAM)
/odom               Raw wheel odometry
/odometry/filtered  EKF-smoothed odometry
/map                Output occupancy grid

Commands

# Launch mapping (simulation)
ros2 launch ogre_slam mapping.launch.py use_sim_time:=true

# Launch mapping (real robot)
./scripts/launch_mapping_session.sh

# Save map
ros2 run nav2_map_server map_saver_cli -f ~/ros2_ws/src/ogre-slam/maps/my_map

# Check map publishing rate
ros2 topic hz /map

What's Next

With a map saved, the next step is Nav2 integration - but there's a problem. Nav2 sends velocity commands assuming the robot will execute them perfectly. Mecanum wheels are difficult to control precisely. The angled rollers that enable omnidirectional movement also introduce wheel slip, and each motor responds slightly differently. Simple kinematic equations don't account for these real-world factors. None of this crossed my mind when I picked up what looked like a cool robot kit. It is cool - just significantly more complicated than expected.

Part 4 covers training a reinforcement learning policy in NVIDIA Isaac Lab. The neural network learns the robot's actual dynamics through millions of simulated trials - compensating for wheel slip, uneven floors, and motor response differences. Setting up the Isaac Lab environment, defining reward functions, tuning hyperparameters, and iterating through training runs was a significant undertaking.

The trained policy sits between Nav2's velocity commands and the wheel motors, translating "go forward at 0.5 m/s" into the precise PWM signals each wheel needs to achieve that motion. Without it, the robot drifts, overshoots, and struggles to follow planned paths accurately.

This is Part 2 of the Project Ogre series. In Part 1, we built the physical mecanum drive robot. Now we'll create an accurate digital twin in NVIDIA Isaac Sim that lets us develop and test without touching the real hardware.

Project Goal: Build an accurate simulation of Project Ogre in Isaac Sim that uses the same ROS2 interface as the real robot, enabling simulation-first development.

📦 Full Source Code: The USD robot model and Isaac Sim configuration are available at github.com/protomota/ogre-slam

Why Simulation First?

In an ideal workflow, you'd build the simulation before touching hardware. I was too eager to get a working robot, so I built the physical platform first. I made progress on the real robot, but eventually hit enough issues that I realized constructing a digital twin would make debugging much easier.

The benefits of simulation-first development:

• No hardware damage risk - Test aggressive maneuvers without breaking motors or sensors

• Faster iteration - Reset the simulation in milliseconds vs. repositioning a physical robot

• Reproducible experiments - Same initial conditions every time

• Parallel development - Work on navigation code while the hardware is being built

Isaac Sim is NVIDIA's robotics simulation platform built on Omniverse. It provides:

• GPU-accelerated physics (PhysX 5)

• Photorealistic rendering with ray tracing

• Native ROS2 integration

• Sensor simulation (LIDAR, cameras, IMU)

The USD Robot Model

The robot is defined in a Universal Scene Description (USD) file: usds/ogre_robot.usd. This format captures geometry, physics properties, and joint definitions in a single file.

Creating the Model

I built the USD model by:

1. Importing the chassis - A simple box representing the steel frame

2. Adding wheel assemblies - Four mecanum wheels with proper roller geometry

3. Defining joints - Revolute joints connecting each wheel to the chassis

4. Setting up collision meshes - Simplified geometry for physics calculations

5. Adding sensors - LIDAR and camera mounts

Physical Dimensions

The USD model matches the real robot's measurements exactly:

Body: 200mm (L) x 160mm (W) x 175mm (H)
Wheel Radius: 40mm
Wheelbase: 95mm (front-to-rear axle distance)
Track Width: 205mm (left-to-right wheel centers)

Getting these dimensions right is critical - SLAM and navigation algorithms depend on accurate odometry, which depends on knowing exactly how far the wheels travel per rotation.

Wheel Configuration

Mecanum wheels require careful attention to roller angles:

M4 (FL) --- M1 (FR)
   |    X    |
M3 (RL) --- M2 (RR)

• Front-left and rear-right wheels have rollers angled one way

• Front-right and rear-left wheels have rollers angled the opposite way

This arrangement creates the characteristic X-pattern when viewed from above. If the roller angles are wrong, the robot won't strafe properly.

Joint Configuration

Each wheel joint needs proper physics parameters for stable simulation:

Joint Properties

# Wheel joint configuration
joint_type: "revolute"
axis: [1, 0, 0]  # Rotate around X-axis
damping: 10.0
friction: 0.1

Critical: Maximum Joint Velocity

Isaac Sim's default maximum joint velocity is extremely high (1,000,000 rad/s). This causes mecanum wheels to spin out of control during physics steps.

Solution: Set maximum velocity to 10,000 rad/s or lower:

drive:
  maxVelocity: 10000.0  # NOT the default 1000000!

This single parameter took hours to debug. The robot would work fine at low speeds but become unstable at higher velocities.

Wheel Sign Correction

The USD model has the right wheels (FR, RR) with opposite joint axis orientation from the left wheels. Sending [+10, +10, +10, +10] to all wheels makes the robot spin instead of moving forward.

The solution is sign correction in the action graph:

Left wheels (FL, RL): velocity as-is
Right wheels (FR, RR): velocity × -1

This normalization ensures [+,+,+,+] always means "all wheels forward" regardless of joint axis conventions.

Physics Settings

Stable mecanum simulation requires tuned physics parameters:

Simulation Timestep

physics_dt: 1/120  # 120 Hz physics
rendering_dt: 1/60  # 60 Hz rendering

The physics timestep must be faster than your control rate. With 30 Hz control, 120 Hz physics gives four physics steps per control step.

Wheel Friction

Mecanum wheel friction is counterintuitive. The angled rollers mean friction behaves differently along different axes:

# Ground material
static_friction: 0.8
dynamic_friction: 0.6

# Wheel material
static_friction: 0.5
dynamic_friction: 0.4

Too much friction and the wheels won't slide sideways. Too little and the robot drifts uncontrollably.

Joint Damping

damping: 1.0 to 10.0  # Range for stability

Damping prevents joint oscillation. Higher values make the wheels feel "heavier" but more stable. Start with 10.0 and reduce if response feels sluggish.

ROS2 Integration

Isaac Sim connects to ROS2 through Action Graphs - visual programming nodes that define data flow.

Setting Up the ROS2 Context

Every Isaac Sim scene with ROS2 needs a context node:

1. Create an Action Graph

2. Add "ROS2 Context" node

3. Set domain_id to match your terminals (default: 0)

The domain ID must match between Isaac Sim and your ROS2 terminals, or topics won't be visible.

Publishing Joint States

To publish wheel positions and velocities:

1. Add "Articulation State" node pointing to the robot

2. Connect to "ROS2 Publish Joint State" node

3. Set topic name: /joint_states

Subscribing to Commands

To receive wheel velocity commands:

1. Add "ROS2 Subscribe Joint State" node

2. Set topic: /joint_command

3. Connect to "Articulation Controller" node

4. Set control type: "velocity"

The Three Action Graphs

Rather than cramming everything into one monolithic action graph, I split the ROS2 integration into three focused graphs. Each handles a specific subsystem, making debugging easier and keeping the visual programming manageable.

Mecanum Drive Controller

This is the core locomotion graph that handles bidirectional communication between ROS2 and the robot's wheels.

What it does:

• Subscribes to /joint_command for wheel velocity targets

• Applies wheel sign correction (FR, RR × -1) to normalize joint axis conventions

• Drives the Articulation Controller with velocity commands

• Publishes current joint states to /joint_states for odometry feedback

Key nodes:

• ROS2 Context - Establishes the ROS2 bridge with matching domain ID

• ROS2 Subscribe Joint State - Receives velocity commands from the policy controller

• Articulation Controller - Applies velocities to the simulated joints

• Articulation State - Reads current joint positions and velocities

• ROS2 Publish Joint State - Broadcasts state for external nodes

The sign correction happens between the subscriber and the articulation controller. Without it, sending identical velocities to all wheels makes the robot spin instead of driving straight.

LIDAR Sensor Stream

This graph handles the simulated RPLIDAR A1 sensor, streaming 2D laser scan data to SLAM and navigation nodes.

What it does:

• Reads from the RTX Lidar sensor attached to the robot

• Converts the point cloud to a 2D LaserScan message

• Publishes to /scan at the configured rate (~8 Hz to match the real A1)

Key nodes:

• Isaac Read Lidar Point Cloud - Pulls data from the RTX Lidar sensor

• Isaac Create Render Product - Sets up the rendering pipeline for the sensor

• ROS2 Publish LaserScan - Outputs sensor_msgs/LaserScan to /scan

Configuration:

horizontal_fov: 360.0      # Full rotation
horizontal_resolution: 1.0  # 1 degree per sample
max_range: 12.0            # Matches RPLIDAR A1 spec
min_range: 0.15            # Minimum detection distance

The simulated LIDAR matches the real sensor's characteristics closely enough that slam_toolbox configuration works identically in both environments.

RealSense Depth Camera

This graph simulates the Intel RealSense D435 depth camera, providing both RGB and depth streams.

What it does:

• Captures RGB frames from the simulated camera

• Generates depth images from the camera's depth sensor

• Publishes synchronized image streams to standard RealSense topics

Published topics:

• /camera/color/image_raw - RGB image (sensor_msgs/Image)

• /camera/depth/image_rect_raw - Depth image (sensor_msgs/Image)

• /camera/camera_info - Camera intrinsics for 3D reconstruction

Key nodes:

• Isaac Create Render Product - Sets up the camera rendering pipeline

• ROS2 Camera Helper - Handles image encoding and publishing

• Isaac Read Camera Info - Extracts camera parameters

Configuration:

resolution: [640, 480]
frame_rate: 30
depth_range: [0.2, 10.0]  # Matches D435 spec

The depth stream feeds into Nav2's voxel layer for 3D obstacle detection - critical for spotting obstacles below the LIDAR's scan plane like table legs or stairs.

Why Three Separate Graphs?

Splitting into three graphs provides several advantages:

1. Isolation - A bug in the camera graph doesn't affect locomotion

2. Performance - Each graph can run at its optimal rate (wheels at 50Hz, LIDAR at 8Hz, camera at 30Hz)

3. Debugging - Easy to disable one sensor while testing another

4. Reusability - The mecanum drive graph works for any 4-wheeled robot

You could combine them into one graph, but the visual complexity becomes overwhelming quickly. Three focused graphs are much easier to maintain.

Testing the Digital Twin

Once the model is configured, test it incrementally:

Step 1: Basic Motion

# Terminal 1: Launch Isaac Sim, load ogre.usd, press Play

# Terminal 2: Send a velocity command
ros2 topic pub /joint_command sensor_msgs/msg/JointState \
  "{velocity: [1.0, 1.0, 1.0, 1.0]}" -r 10

All wheels should spin. If the robot spins in place instead of moving forward, check the wheel sign correction.

Step 2: Omnidirectional Motion

# Forward
ros2 topic pub /cmd_vel geometry_msgs/msg/Twist \
  "{linear: {x: 0.3}}" -r 10

# Strafe left
ros2 topic pub /cmd_vel geometry_msgs/msg/Twist \
  "{linear: {y: 0.3}}" -r 10

# Rotate
ros2 topic pub /cmd_vel geometry_msgs/msg/Twist \
  "{angular: {z: 0.5}}" -r 10

Test all motion primitives. The strafe test is the true mecanum validation - if it moves diagonally instead of purely sideways, check the wheel arrangement.

Step 3: Sensor Verification

# Check LIDAR
ros2 topic echo /scan --once

# Visualize in RViz
rviz2 -d ~/ros2_ws/src/ogre-slam/rviz/isaac_sim.rviz

The LIDAR should show walls and obstacles in the simulation environment.

Simulation vs. Real Robot

The key advantage of this setup: the same ROS2 code runs in both environments.

Component       Simulation         Real Robot
--------------  -----------------  --------------
LIDAR topic     /scan              /scan
Odometry        Isaac Sim physics  Encoder-based
Camera          Simulated RGB-D    RealSense D435
Joint commands  /joint_command     /joint_command

The only difference is launch parameters. The same SLAM, navigation, and control code works in both places.

Simulation-Only Flags

# Simulation
ros2 launch ogre_slam mapping.launch.py use_sim_time:=true

# Real robot
ros2 launch ogre_slam mapping.launch.py use_sim_time:=false

The use_sim_time flag tells ROS2 to use Isaac Sim's clock instead of wall clock. Without this, TF transforms time out because simulation time doesn't match real time.

Common Issues

Robot Falls Through Floor

Cause: Collision meshes not enabled or ground plane missing

Solution:

1. Ensure robot has collision enabled on all rigid bodies

2. Add a ground plane with collision

Wheels Spin But Robot Doesn't Move

Cause: Wheel friction too low or joint damping too high

Solution:

1. Increase ground friction

2. Reduce joint damping

3. Check wheel contact with ground

Robot Vibrates or Explodes

Cause: Physics instability from timestep or mass issues

Solution:

1. Reduce physics timestep (increase Hz)

2. Check mass distribution is reasonable

3. Reduce joint stiffness

Topics Not Visible

Cause: ROS_DOMAIN_ID mismatch

Solution:

1. Check Isaac Sim ROS2 Context domain_id

2. Match in terminal: export ROS_DOMAIN_ID=0

---

Quick Reference

Key Files

File                                   Purpose
-------------------------------------  ----------------------------------------
usds/ogre_robot.usd                    Robot model with physics
usds/ogre.usd                          Complete scene with environment
actiongraph_ros2_mecanum_drive_sm      Wheel control and joint state publishing
actiongraph_ros2_lidar_stream_sm       LIDAR sensor simulation
actiongraph_ros2_real_sense_camera_sm  RGB-D camera simulation
rviz/isaac_sim.rviz                    RViz config for simulation visualization

Isaac Sim Settings

Physics:
  timestep: 1/120 (120 Hz)
  solver_iterations: 4

Wheel Joints:
  max_velocity: 10000 rad/s
  damping: 10.0

Ground:
  static_friction: 0.8
  dynamic_friction: 0.6

What's Next

With the digital twin working, Part 3 covers using slam_toolbox to build maps. We'll run SLAM in both simulation and on the real robot, comparing results and tuning parameters for accurate mapping.

The ability to test SLAM in simulation before deploying to hardware saves significant time - you can iterate on configuration without draining the robot's battery or worrying about collisions.

Project Ogre is a mecanum drive robot platform designed for autonomous navigation using ROS2 and NVIDIA's robotics ecosystem. The name comes from the robot's stocky, powerful appearance with its four omnidirectional wheels. This is the first post in a five-part series covering the complete build:

1. Part 1: Building the Robot (this post)

2. Part 2: Building the Digital Twin in Isaac Sim

3. Part 3: SLAM Mapping with slam_toolbox

4. Part 4: Training in Isaac Lab

5. Part 5: Autonomous Navigation with Nav2

Project Goal: Build a mecanum drive robot powered by the Jetson Orin Nano 8GB Developer Kit, running entirely from a single battery, capable of omnidirectional movement, SLAM mapping, and autonomous waypoint navigation using ROS2 Nav2.

📦 Full Source Code: The teleoperation code we ran on the Jetson Orin Nano to test the motors and camera is available at github.com/protomota/ogre-teleop

Why Mecanum Wheels?

Honestly, I wasn't specifically looking for mecanum wheels. I wanted powerful motors with encoders for a 4-wheel drive robot - something that could power over small bumps and floor transitions where my JetBot would struggle. I was originally planning to just 3D print some standard wheels.

But while browsing for motors, I found a kit that changed my plans: four motors with encoders, four mecanum wheels pre-fitted, and a simple steel platform to build upon. The price was right, and it gave me a solid foundation to work with. So mecanum it was.

The bonus? Mecanum wheels unlock omnidirectional movement. Traditional differential drive robots can only move forward, backward, and rotate. With mecanum wheels, the robot can strafe sideways, move diagonally, and rotate while translating - all thanks to rollers mounted at 45-degree angles on each wheel.

The key insight is that when all four wheels spin together, the angled rollers create force vectors that can be combined to produce motion in any direction:

• All wheels forward → Robot moves forward

• Left wheels backward, right wheels forward → Robot rotates

• Front wheels inward, rear wheels outward → Robot strafes left

This flexibility turned out to be invaluable for navigation in tight spaces, though it does add complexity to both hardware and software.

▶ Watch the video on YouTube

Hardware Components

Core Computing

• NVIDIA Jetson Orin Nano Developer Kit (8GB) - The robot's brain. The Orin Nano provides significant computing power for running SLAM, Nav2, and sensor processing simultaneously. At 25W power mode, it handles everything smoothly.

Drive System

• 4x 25GA-370 DC Motors with Encoders - These geared motors provide the torque needed to move the robot. Each motor includes a 2 PPR (pulses per revolution) Hall effect encoder for odometry.

• 4x Mecanum Wheels (40mm radius) - The omnidirectional wheels that give the robot its unique mobility. Pay attention to the roller angles - you need two left-handed and two right-handed wheels arranged correctly.

• PCA9685 Motor Driver - A 16-channel PWM driver controlled via I2C. This handles motor speed control without consuming GPIO pins.

Sensors

• RPLIDAR A1 - A 360-degree 2D LIDAR scanner. It provides laser scan data for SLAM mapping and localization. The A1 is an excellent entry-level option with 12-meter range at ~8Hz scan rate.

• Intel RealSense D435 - A depth camera for 3D obstacle detection. While the LIDAR handles 2D mapping, the RealSense provides point cloud data for detecting obstacles below the LIDAR's scan plane (like table legs or stairs).

Physical Dimensions

These measurements are critical for accurate odometry and SLAM:

Body: 200mm (L) x 160mm (W) x 175mm (H)
Wheel Radius: 40mm
Wheelbase: 95mm (front-to-rear axle distance)
Track Width: 205mm (left-to-right wheel centers)
Total Footprint: ~310mm x 205mm x 300mm
Robot Weight: ~5kg (10 lb 15 oz)

The robot sits with its body positioned 20mm above the wheel axle height. The LIDAR is mounted on 65mm posts, placing it at 0.30m above the base_link frame.

System Architecture

The robot is organized into five distinct layers, from sensors at the top to the chassis at the bottom:

┌─────────────────────────────────────────────────────────────────────┐
│                        SENSOR LAYER (Top)                           │
│  ┌──────────────┐   ┌─────────────────────┐   ┌──────────────────┐  │
│  │  RPLIDAR A1  │   │  Intel RealSense    │   │   LiPo Battery   │  │
│  │              │   │       D435          │   │     Monitor      │  │
│  └──────────────┘   └─────────────────────┘   └──────────────────┘  │
└─────────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                         LOGIC LAYER                                 │
│  ┌────────────────────────┐  ┌─────────────┐  ┌──────────────────┐  │
│  │   Jetson Orin Nano     │  │   PCA9685   │  │  5V → 3.3V       │  │
│  │         8GB            │  │     PWM     │  │  Logic Level     │  │
│  │    (Main Controller)   │  │    Driver   │  │  Converter       │  │
│  └────────────────────────┘  └─────────────┘  └──────────────────┘  │
└─────────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                         POWER LAYER                                 │
│  ┌────────────────────┐   ┌─────────────────────────────────────┐   │
│  │   6S LiPo Battery  │   │   Buck Converters & Power           │   │
│  │                    │   │        Conditioner                  │   │
│  └────────────────────┘   └─────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    MOTOR CONTROLLER LAYER                           │
│  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌────────────┐  │
│  │   Motor      │ │   Motor      │ │   Motor      │ │   Motor    │  │
│  │ Controller 1 │ │ Controller 2 │ │ Controller 3 │ │ Controller │  │
│  │     (FL)     │ │     (FR)     │ │     (RL)     │ │   4 (RR)   │  │
│  └──────────────┘ └──────────────┘ └──────────────┘ └────────────┘  │
└─────────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    STEEL CHASSIS (Base)                             │
│                                                                     │
│    ┌───────────┐                              ┌───────────┐         │
│    │  MECANUM  │                              │  MECANUM  │         │
│    │   WHEEL   │  ════════════════════════    │   WHEEL   │         │
│    │   (FL)    │       25GA-370 Motor         │   (FR)    │         │
│    └───────────┘                              └───────────┘         │
│         ║                                          ║                │
│         ║              STEEL FRAME                 ║                │
│         ║                                          ║                │
│    ┌───────────┐                              ┌───────────┐         │
│    │  MECANUM  │                              │  MECANUM  │         │
│    │   WHEEL   │  ════════════════════════    │   WHEEL   │         │
│    │   (RL)    │       25GA-370 Motor         │   (RR)    │         │
│    └───────────┘                              └───────────┘         │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Motor Layout and Encoder Wiring

The motor arrangement follows the standard mecanum convention:

M4 (FL) --- M1 (FR)
   |    X    |
M3 (RL) --- M2 (RR)

Each motor's encoder connects to GPIO pins on the Jetson. I use BOARD pin numbering:

Motor  Position     GPIO Pins (BOARD)
-----  -----------  -----------------
M1     Front-Right  7, 11
M2     Rear-Right   13, 15
M3     Rear-Left    29, 31
M4     Front-Left   32, 33

The encoder reader uses interrupt-based counting to track wheel rotations. With only 2 PPR, the resolution is quite coarse, which is why sensor fusion (EKF) is essential for usable odometry.

Encoder Calibration: The gear_ratio Challenge

Getting accurate odometry from low-resolution encoders requires careful calibration. The gear_ratio parameter was the most critical value to tune.

The Problem: With 2 PPR encoders, each encoder pulse represents a significant distance traveled. The gear ratio converts raw encoder counts into actual wheel rotations. An incorrect gear_ratio causes massive position drift.

Calibration Process:

1. Start with an initial gear_ratio estimate (I started with 50.0)

2. Mark the robot's starting position with tape

3. Drive exactly 1.0 meter forward using a measuring tape

4. Check the reported odometry position

5. Calculate the correction factor

When I tested with gear_ratio=50.0, driving 1.0m forward reported 4.483m. The correction:

new_gear_ratio = 50.0 × (4.483 / 1.0) = 224.0

After setting gear_ratio=224.0, the robot reported 1.008m for 1.0m actual travel - an error of just 0.8%.

Odometry Configuration

The odometry parameters live in config/odometry_params.yaml:

wheel_radius: 0.040     # 40mm measured
wheel_base: 0.095       # 95mm measured
track_width: 0.205      # 205mm measured
encoder_ppr: 2          # Hall sensors: 2 PPR
gear_ratio: 224.0       # Calibrated for 25GA-370 motors
publish_rate: 50.0      # Hz

The covariance matrices are set high because the 2 PPR encoders are inherently noisy:

pose_covariance_diagonal: [1.0, 1.0, 0.0, 0.0, 0.0, 1.0]
twist_covariance_diagonal: [0.5, 0.5, 0.0, 0.0, 0.0, 0.5]

EKF Sensor Fusion

With only 2 PPR encoders, raw odometry is too noisy for SLAM. The robot_localization package implements an Extended Kalman Filter (EKF) that smooths the odometry data.

The data flow:

Raw Encoders → odometry_node → /odom (noisy)
                                   ↓
                            ekf_filter_node (smoothing)
                                   ↓
                          /odometry/filtered (clean)
                                   ↓
                        SLAM & Nav2 (accurate mapping/navigation)

The EKF combines the noisy encoder measurements with a motion model, weighting each based on configured covariances. The result is much smoother position estimates that SLAM can actually use.

Flask Teleoperation Server

Before diving into autonomous navigation, I needed a way to manually control the robot and view the camera feed. The solution was a Flask-based web interface that runs on the Jetson.

Why Web-Based Control?

A web interface has significant advantages over traditional approaches:

• Device agnostic - Control from any phone, tablet, or laptop on the network

• No installation - Just open a browser and navigate to the robot's IP

• Visual feedback - Live camera stream alongside controls

• Development friendly - Easy to extend with new features

Architecture

The Flask server exposes two main endpoints:

Camera Stream (/video_feed)

• MJPEG stream from the RealSense RGB camera

• Runs at ~15 FPS to balance quality and bandwidth

• Uses OpenCV to capture and encode frames

Motor Control (/cmd_vel)

• Accepts POST requests with velocity commands

• Translates to ROS2 Twist messages

• Implements dead zone for joystick drift

Web Joystick

The frontend uses a virtual joystick library that maps touch/mouse input to velocity commands:

• Vertical axis → Linear X velocity (forward/backward)

• Horizontal axis → Angular Z velocity (rotation)

• Smooth ramp-up/down for natural feel

Running the Server

cd ~/ros2_ws/src/ogre-teleop
python3 app.py

Then open a browser to http://<robot-ip>:8080. The interface shows the camera feed with an overlaid joystick control.

This web interface proved invaluable for testing motor calibration, verifying camera mounting angles, and generally getting a feel for how the robot handles before writing autonomous code.

TF Frame Structure

ROS2's TF system tracks the relationship between coordinate frames. For Project Ogre on the real robot:

map (from slam_toolbox or amcl)
  └─ odom (from EKF or odometry_node)
      └─ base_link (robot center at wheel axle height)
          ├─ laser (RPLIDAR: 0.30m up, 180° rotated)
          └─ camera_link (RealSense D435: 0.15m forward, 0.10m up)

Note that the LIDAR is mounted with a 180° rotation. This is common - just ensure your URDF/launch files account for it.

Power Considerations

The entire robot runs from a single 5000mAh 6S LiPo battery (~24V nominal). Three buck converters distribute power to different subsystems:

• 24V → 12V: Powers the Jetson Orin Nano (accepts 9-20V input)

• 24V → 9V: Powers the motor controllers

• 24V → 5V: Powers the 5V rail for logic-level components

This single-battery approach keeps the robot compact and simplifies charging. The 6S LiPo provides plenty of headroom for the Jetson's power demands while the buck converters ensure each subsystem receives clean, regulated voltage.

Weight Distribution

One challenge I didn't anticipate was weight balance. The 6S LiPo battery is heavy and sits at the rear of the robot, which created an uneven weight distribution. Without counterbalancing, too much torque to the motors would cause the robot to tip backward and fall on its back.

My solution was to 3D print small cylindrical barrels and fill them with fishing weights (lead sinkers). These counterweights mount at the front of the chassis to balance out the battery's mass. It works, but it's not elegant.

In hindsight, I would have tried harder to position the battery at the robot's center of mass from the start. Planning for weight distribution early in the design process would have saved me from adding dead weight to compensate later. Lesson learned for the next build.

What's Next

With the robot built and calibrated, Part 2 covers building an accurate digital twin in NVIDIA Isaac Sim. Creating a simulation environment first would have been the smarter approach, but I was too eager to get hardware running. The digital twin lets us:

• Test code changes without risking hardware damage

• Iterate faster with instant simulation resets

• Develop navigation algorithms in a controlled environment

• Debug issues that are hard to reproduce on real hardware

The simulation uses the same ROS2 interface as the real robot, so the same code runs in both environments.

Project Goal: Enable real-time control of the SO-ARM 101 digital twin in Isaac Sim using the physical leader arm.

📦 Full Source Code: All updated code, ROS2 integration, and configuration files are publicly available in my forked GitHub repository at github.com/protomota/lerobot-sim. My contributions are detailed in PROTOMOTA_README.md.

In the video below, you'll see a split-screen view: on the left is the physical 3D-printed leader arm from the SO-ARM 101, and on the right is a screen recording of Isaac Sim showing the digital twin responding in real-time to my movements. I'm using the leader arm as a controller for the simulated robot, with every joint movement mirrored instantly in the simulation.

To get this working, I built a ROS2 bridge between the LeRobot teleoperation framework and Isaac Sim. I wrote a custom teleoperation controller that reads joint positions from the physical leader arm and publishes them as ROS2 messages. On the Isaac Sim side, there's an action graph that subscribes to those ROS2 commands and converts them into movements within the simulation, creating seamless real-time control.

▶ Watch the video on YouTube

---

The Challenge with Separate Processes

I started with LycheeAI's approach, which uses separate processes for teleoperation and joint state publishing. It's a clean, logical design - separation of concerns at its finest. However, I ran into challenges getting it to work reliably with my hardware setup.

The core issue is that the Feetech STS3215 motors communicate over USB serial using half-duplex communication. When two processes try to access the same serial port simultaneously, you get packet collisions, corrupted data, and error messages like "There is no status packet" flooding your terminal.

I experimented with various solutions - mutex locks, reduced read frequencies, queue-based IPC - but couldn't achieve the reliability I needed for real-time teleoperation.

The Integrated Solution

The fix was elegant: integrate ROS2 publishing into the teleoperation loop itself. The loop already reads joint positions every cycle, so just publish them to ROS2 at the same time.

┌────────────────────────────────────────────────────────┐
│                   Teleoperation Loop                   │
│                                                        │
│   1. Read leader position (ACM0)                       │
│   2. Read follower position (ACM1)  ──► ROS2 Publish   │
│   3. Write to follower (ACM1)                          │
│                                                        │
└────────────────────────────────────────────────────────┘

No port conflicts. 60Hz publishing. Smooth simulation.

The new command is called lerobot-ros-teleoperate, and it supports two modes:

Full mode is when you have both the leader and follower arms connected. The leader controls the follower, and the follower's positions get published to ROS2.

Sim-only mode is when you only have the leader arm. The leader positions get published directly to ROS2, driving the simulation without needing physical follower hardware.

This is huge for development. You can iterate on your simulation setup using just the leader arm, without needing the full dual-arm rig powered up.

USB Connection Order Matters

One gotcha that tripped me up: USB port assignment is based on plug-in order. The first arm you plug in gets /dev/ttyACM0, the second gets /dev/ttyACM1.

I standardized on:

1. Leader first, gets ACM0

2. Follower second, gets ACM1

If you get weird behavior where the wrong arm is responding, unplug both and reconnect in the correct order.

Gripper Calibration

The gripper needed special handling. Once I had basic teleoperation working, I noticed the gripper wasn't behaving correctly in Isaac Sim. It would move, but not through its full range, and the closed position wasn't lining up.

After some debugging, I discovered the issue: the physical gripper outputs values from about 0.02 radians when closed to 1.72 radians when open. But Isaac Sim's Jaw joint needed to reach -11 degrees to fully close.

The fix was two-part:

1. Set Isaac Sim Jaw limits to -11 (lower) and 100 (upper)

2. Apply -0.21 rad offset in code

Now the gripper tracks perfectly.

Bridging to Isaac Sim

The lerobot-ros-teleoperate command publishes joint positions to /joint_states, which is the standard ROS2 topic for joint data. But the Isaac Sim USD file from LycheeAI has its action graph configured to listen on /isaac_joint_command.

Rather than modify either side, I used topic_tools relay, a standard ROS2 utility that republishes messages from one topic to another. It's a one-liner that bridges the two:

ros2 run topic_tools relay /joint_states /isaac_joint_command

This keeps the LeRobot side using standard conventions while matching what Isaac Sim expects.

Getting the Code

IMPORTANT: To get this working, you need to clone my forked repository that includes the integrated ROS2 teleoperation solution and all the modifications discussed in this post.

Clone the repository:

git clone https://github.com/protomota/lerobot-sim.git
cd lerobot-sim

Follow the installation instructions in the repository README to set up the environment and dependencies.

Quick Start

source /opt/ros/humble/setup.bash
conda activate lerobot

# Terminal 1: Start teleoperation (leader arm on ACM0)
lerobot-ros-teleoperate \
    --teleop.type=so101_leader \
    --teleop.port=/dev/ttyACM0 \
    --teleop.id=armatron_leader

# Terminal 2: Relay joint states to Isaac Sim's expected topic
ros2 run topic_tools relay /joint_states /isaac_joint_command

Then open Isaac Sim, load your USD file, and press Play. The simulated arm will mirror your physical leader arm in real-time.

What's Next

With teleoperation working, the next step is adding cameras. The goal is vision-based imitation learning, where the robot learns tasks by watching demonstrations.

The plan:

• Add camera feeds to capture what the robot "sees" during teleoperation

• Record demonstrations pairing camera frames with joint positions as the operator performs tasks

• Train vision policies that map camera input directly to robot actions

• Deploy to real hardware using the same camera setup on the physical follower arm

LeRobot already has infrastructure for this. The framework supports recording datasets with synchronized camera and joint data, and includes ACT (Action Chunking Transformer) and other imitation learning policies out of the box.

If you found this useful, let me know in the comments. And if you're building something similar, I'd love to hear about it.

Credits

• TheRobotStudio - Original SO-ARM 100 design and hardware

• LeRobot - Hugging Face's framework for real-world robotics

• LycheeAI - Isaac Sim URDF modifications, Action Graph info, and initial ROS2 bridge concepts

JetBot (jetbot.org) is an educational robot platform originally designed for the now-discontinued Jetson Nano. While the newer NVIDIA Jetson Orin Nano Super Developer Kit (8GB) offers significantly more computing power, the official JetBot project hasn't been updated since February 2021. The existing documentation only provides an outdated parts list for the Orin Nano, with no software support outside of some other forked repos.

Project Goal: Fully adapt the JetBot platform to work with the newer, more powerful NVIDIA Jetson Orin Nano, including all necessary hardware modifications, software updates, and Jupyter notebook compatibility.

📦 Full Source Code: All updated code, configuration files, and Docker containers are publicly available in my forked GitHub repository at github.com/protomota/jetbot-orin

After extensive testing and code modifications, I successfully adapted JetBot to run on the Orin Nano, including all the original Jupyter notebooks. This post shares my implementation to help others build their JetBots with the newer hardware.

JetBot in Action

Here's the modified JetBot Orin Nano demonstrating autonomous navigation on a toy roadway using vision-based training and machine learning. The robot uses its camera to detect road boundaries and navigate the course in real-time, showcasing the practical capabilities of the Orin Nano's enhanced computing power for inference and decision-making.

▶ Watch the video on YouTube

The Power Challenge

The biggest hurdle I encountered was the power requirements. The original Nano runs on 5V, while the Orin Nano requires 9-20V. Combined with the motor controller and motor requirements, standard 10,000mAh USB power banks can't deliver this higher voltage reliably. You can power one or the other, but not both. After testing numerous options (and making far too many returns to Amazon), I decided that the best and easiest solution would be dual power banks: one for the Orin Nano and one for the motors and controller. While this adds weight and complexity, the plus side is longer runtime. Until someone starts selling a custom Power Delivery Board desinged for the Orin Nano, this is the best solution I could come up with.

Model Redesign

This presented another problem - the original chassis model only fits one battery. So I needed to modify the battery compartment area of the model to support dual 10,000mAh batteries and then reprint it.

Download the modified chassis with printer settings: chassis-dual.3mf

Download STL only: chassis-dual.stl

Updated Bill of Materials for JetBot Orin Nano

While the JetBot community has updated the parts list for the Orin Nano, the current recommendations are partially outdated and don't properly address the power requirements. While most of the instructions in the parts list are accurate and you will need the parts, the instructions will only get you 80% there. There are many broken links and outdated recommendations. Here are my recommended improvements:

NVIDIA Jetson Orin Nano

• Developer Kit: NVIDIA Jetson Orin Nano Super Developer Kit (8GB) - Our robot brain.

Power Solution

• Power Banks: 2 x 35W 10,000mAh Power Bank - You need two of these. One for the Orin Nano, and one for the motors and motor controller. The larger wattage on this model is also essential for meeting the Orin Nano's higher power demands. 35W will also allow you to run the Orin Nano at full 25W power. This particular model is available under a couple of different names as it is a white label product. This one is particularly cool as the power reading buttons and readout conveniently fit perfectly in the windows of the JetBot.

• Power Adapter: USB-C to DC Adapter - Much cleaner and more compact solution without cable management headaches of the recommended 6-foot cable.

• USB-C Cable: Short USB-C Cable - This plugs into the battery and the power adapter.

Storage Upgrade

Instead of an SD card, I strongly recommend upgrading to an NVME M.2 SSD. This provides significantly better performance and reliability.

Important: Installing Jetpack on the Orin Nano with an SSD requires a host x86 PC running Ubuntu for the reflashing process. ARM-based systems like Raspberry Pi won't work for this.

If you don't have an x86 Ubuntu machine, you can purchase an affordable mini PC for about the same cost as a Raspberry Pi. Install Ubuntu alongside or instead of Windows for the setup process.

I had the idea that it might work in an Ubuntu virtual machine, but I don't know if that is supported. Something to try if you need to.

Getting Started

Follow the NVIDIA Jetpack SDK Installation Instructions to install the latest Jetpack on your Jetson Orin Nano.

Hardware Setup and Motor Controller Configuration

Before running the software, ensure your hardware is properly configured:

Motor Controller Power

The Adafruit MotorHAT/FeatherWing motor controller requires external power (6-12V) separate from the Jetson.

Important: Check that the motor controller has a green power LED lit at all times. Some power banks automatically shut down if they detect low current draw, thinking nothing is connected. If you're using a power bank:

• Use a power bank that supports low-current devices

• Monitor the green LED - if it turns off, your power source has shut down

• NVIDIA does not yet have an official recommendation or solution for powering the Orin Nano from a dedicated battery pack designed for robotics. It is possible, but outside the scope of this tutorial. Venture at your own risk as it has its own set of technical challenges.

Connecting to your NVIDIA Orin Nano

Open your terminal and SSH into your Orin Nano:

ssh jetbot@<jetbot_ip>

I2C Bus Configuration

Since the Orin Nano I2C bus is different (7, rather than the default 1), you need to set the JETBOT_I2C_BUS environment variable:

# Add to ~/.bashrc for persistence
echo 'export JETBOT_I2C_BUS=7' >> ~/.bashrc
source ~/.bashrc

Installing the JetBot Python Package

Create a directory called 'source' in your home directory:

cd ~
mkdir source

IMPORTANT: Clone my forked and updated repository that works with the Orin Nano hardware:

cd source
git clone https://github.com/protomota/jetbot-orin.git

Install the JetBot Python package:

cd jetbot-orin
python3 setup.py install --user

Configure System

First, run the scripts/configure_jetson.sh script to configure the power mode and other parameters.

cd ~/source/jetbot-orin
./scripts/configure_jetson.sh

Configure Docker Environment

Navigate to the docker directory and source the configure.sh script to configure environment variables.

cd docker
source configure.sh

Build Docker Containers

Build all JetBot containers from scratch:

./build.sh

This step is required before running the containers for the first time.

Enable and Start Containers

Enable Docker to start at boot and launch the JetBot containers:

./enable.sh $HOME

The directory you specify (e.g., $HOME) will be mounted as /workspace in the Jupyter container. All work saved in /workspace will persist across container restarts.

Access Jupyter Lab

Open your web browser and navigate to:

https://<jetbot_ip>:8888

The default password is jetbot.

Optional Configuration

Enable JetBot Python API Support

Run setup (optional, but recommended - only needed if you plan to use JetBot Python APIs outside of Docker):

cd ~/source/jetbot-orin
python3 setup.py install --user

Power Mode (Jetson Orin Nano 8GB)

For Jetson Orin Nano 8GB Developer Kit, we recommend lowering the power mode to 3W for optimal battery operation:

Common Orin Nano power modes:

• Mode 0: MAXN (15W/20W depending on your module)

• Mode 1: 10W mode (power-saving)

• Mode 2: 15W mode

• Mode 3: 7W mode

sudo nvpmodel -m 3

Then verify it worked:

sudo nvpmodel -q

Managing Containers

Stop All Containers

cd docker
./disable.sh

Rebuild Containers

cd docker
./disable.sh
./build.sh
./enable.sh $HOME

Important Notes

• The containers will restart automatically on boot

• Work saved outside of /workspace in the Jupyter container will be lost when the container restarts

• The NVIDIA runtime is automatically configured by configure.sh

• Docker daemon is automatically enabled at boot by configure.sh

Their pitch is "context over process." I think Saarinen is right about the diagnosis. But he's solving the wrong layer of the problem.

Process became the product a long time ago

If you've used Jira or Azure DevOps at a company with more than 50 engineers, you know what he's talking about. Jira's the one everyone loves to hate, but Azure DevOps is the quiet offender. Same disease in a Microsoft suit: work items linked to test plans linked to boards nobody opens, welded to your CI/CD so the process overhead becomes literally impossible to remove without breaking your builds.

Theo Browne talked about this — his Jira at Twitch took over two minutes to load. That's not a performance bug. That's what happens when every stakeholder gets to add a field.

I've lived both sides of this. For a stretch of my career I was a Jira admin at a 100-person company. It consumed about half my day. Not building. Not shipping. Tweaking workflows, fixing broken automations, cleaning up schemes that had drifted into nonsense. Half my working hours maintaining a tool whose purpose was supposed to be helping other people work. I also spent time as a product owner on a large enterprise team, basically living in Azure DevOps. Grooming backlogs, writing acceptance criteria, sitting in sprint ceremonies, managing boards. The work was real, but it wasn't building. It was managing the process around building.

At my core I'm a builder. I went back to building. Those process management roles no longer provide value in the ways we need to work today. The future belongs to builders, not process managers.

The Agile Manifesto said "working software over comprehensive documentation" in 2001. Then an industry of consultants, certification bodies, and framework vendors built a multi-billion-dollar empire around the ceremony of being agile. SAFe. Scrum certifications. "Agile transformations" led by people who haven't shipped code in a decade. Microsoft built an entire product around it. Atlassian built an entire company around it. The tools designed to make teams agile became the heaviest things in the building.

The agile industrial complex is real. The principles were never the problem. The industry that grew around them was.

What AI actually changes

Theo's approach at Twitch was to skip the spec and build a rough prototype to discover requirements. Before AI, that worked if you had the right engineer. But in the AI era, I think you actually need the spec more, not less. Not the 20-page Google Doc. A markdown file produced by the subject matter experts. What you're building, who it's for, what done looks like. Enough context to hand an agent and get a useful first pass.

You can't "skip the spec" when your builder is an LLM. The prompt IS the spec. The question is whether you write a thoughtful one that gets a working prototype on the first pass, or wing it and spend three hours course-correcting.

Write the spec, prompt the agent, get a prototype, refine. That's the loop. The cost of a first pass is approaching zero. When it costs an afternoon and a good prompt, you iterate faster than any sprint cycle could keep up with.

A new type of team is emerging

A solo founder with AI agents can ship what used to take 15 engineers. A team of three can operate at the scale of thirty. The leverage is compounding fast, and it changes what tools you need.

Jira and Azure DevOps exist because large teams need coordination overhead. Tickets, workflows, sprint boards, estimation rituals — all coordination tax. The cost of having a lot of humans in the same codebase. But the teams forming now don't look like that. They don't need sprint boards because there's no sprint. They don't need estimation because one person holds the whole context.

Smaller doesn't mean solo, though. There are unicorns who can design, code, manage product, and test. They exist. But they're rare, and even the best of them usually can't match the quality a small team of specialists produces. The unicorn gets you to market. The team gets you to quality.

The roles haven't gone away. You still need a designer's eye, an engineer's instinct for what breaks at scale, a PM who manages the project and works with clients, a QA engineer who thinks adversarially. What changes is headcount and throughput. A designer with AI explores ten directions in the time it used to take to mock up two. An engineer ships in a day what used to take a sprint. AI is a productivity accelerator, not a role eliminator. A team of five with the right specializations can operate like fifty used to.

That's the market Linear is actually chasing. Not better issue tracking for big teams. Lighter tools for smaller teams that punch above their weight.

The business model is breaking too

This isn't just a team structure problem. It's a pricing problem.

Software projects used to be measured in developer hours. Entire sales organizations were built around scoping engagements at X hours times Y rate. That math is collapsing. The honest unit of work is shifting from hours to tokens, and while nobody has a clean way to estimate token cost for a project yet, the direction is obvious: it's a lot less. That's all that matters.

Software can't be sold at the prices it was even two years ago. A project that a consultancy would have quoted at six months and half a million dollars can now be built by a small team in weeks for a fraction of the cost. The buyers are figuring this out. The sellers, a lot of them, haven't.

Large firms that only chase big deals are going to get their lunch taken by smaller indie dev shops that move faster, charge less, and ship better work with leaner teams. The overhead that justified premium pricing, the project managers, the Jira admins, the sprint ceremonies, the 30-person delivery teams, none of that scales the way it used to. It's just cost now.

Offshoring is facing the same math. The whole model was built on labor cost arbitrage: cheaper hourly rates in other markets. But when the unit of work shifts from hours to tokens, that gap shrinks fast. Tokens cost the same no matter where the developer sits. Meanwhile, the coordination costs haven't changed. Time zone gaps, communication overhead, context getting lost across languages and cultures, losing direct control over the taste and style of the output, the quality variance that comes from managing work at a distance. Those tradeoffs used to be worth it because the savings were significant. When a small local team with AI tools can compete on price, the economic case for offshoring gets a lot harder to make.

Sales teams are struggling and a lot of them don't understand why. The pipeline looks the same. The pitch decks look the same. But the deals aren't closing because the buyers can see the math changing. The teams that adapt, that price for the new reality and sell speed and quality instead of headcount, will win. The ones still quoting based on 2023 economics are going to have a rough year.

What AI doesn't change

You still don't know what your users want. Inflectra put it well: "If code shows up faster than requirements learning, you can ship the wrong thing even sooner."

AI makes shipping faster. It does not make learning faster. Skip the feedback loops and you just automate failure. The Agile principle of "optimize for learning" matters more now than it did in 2001. Short feedback loops aren't optional when your agent can ship a feature before lunch.

What actually dies is the handoff model. Not the roles, but the assembly line where each role produces an artifact for the next to consume. When your team is small enough that everyone shares context directly, you don't need the proxy. The teams that win will be the ones with taste — who know what's worth building and can spot "this is wrong" before users have to tell them.

When building is cheap, judgment becomes the bottleneck.

The manifesto was right

Issue tracking isn't dead. Bad issue tracking deserved to die. The bloated Jira instances, the Azure DevOps boards nobody opens, the velocity charts that measured activity instead of outcomes.

What survives is what the Agile Manifesto said 25 years ago. Build working software. Collaborate with your users. Respond to change. Value people over process. We buried those principles under dashboards and certifications and two-week rituals. AI is digging them back out.

The future isn't post-Agile. It's Agile without the industry that grew around it.

Build something. Learn from it. Do it again.

But every time I looked at my setup there was a small mental asterisk: I was running my local agent stack on weights trained by Alibaba. Not because I had any concern about the model, but because I'm an American indie builder and my favorite local model was coming out of Hangzhou. It felt slightly off-brand for what I'm trying to build.

Gemma 4 26B-A4B is the first genuinely capable small US open weights model I've been able to run on my own hardware. Not "capable for a small model." Capable, full stop.

Why 26B on an Orin should not work

The AGX Orin is a 275mm x 87mm module that draws about 60 watts under load. 64GB of unified memory shared between CPU and GPU. Ampere GPU, 2048 CUDA cores. It sits on my workbench next to my dev machine.

Gemma 4 26B-A4B is not a normal 26B model. It has 25.2 billion total parameters, but only 3.8 billion are active during any given inference pass. 128 experts per MoE layer, router picks 2 per token, the rest sit idle. The file is 16.8GB at Q4_K_M, but the compute per token is closer to a 4B model.

That's the whole trick. Training quality of a 26B, inference cost of a 4B. On hardware where every watt matters, that's the difference between "runs" and "doesn't."

NVIDIA's Jetson AI Lab lists the AGX Orin as a supported platform and ships a Docker container with llama.cpp pre-configured:

sudo docker run -it --rm --pull always --runtime=nvidia --network host \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  ghcr.io/nvidia-ai-iot/llama_cpp:gemma4-jetson-orin \
  llama-server -hf ggml-org/gemma-4-26B-A4B-it-GGUF:Q4_K_M

One line. Model takes ~24GB of RAM. On my 64GB Orin, that leaves plenty of room for the rest of the system.

What "runs" actually means

Set expectations. At Q4 on the Orin, Gemma 4 26B-A4B is not fast. I'm getting around 11 tokens per second, which I'd call "comfortable for non-interactive use." Background tasks, batch processing, offline analysis. Not real-time chat.

For my use cases this is fine. I'm not building a chatbot on the Orin. I'm running background agent tasks: log analysis, code review, config generation, structured extraction. None of those need sub-second latency. They need good output on short-to-medium prompts.

And the output is where 26B earns its place. Sub-10B models on this hardware work for simple stuff (summarize this, extract these fields, classify this log line), but they fall apart on anything that needs real reasoning. A 3B model asked to analyze a stack trace gives you something plausible that's wrong half the time. A 7B does better but hallucinates function names. Gemma 4 26B is a different animal. It understands code, handles tool calling, follows multi-step instructions without losing the thread.

The 256K context window doesn't hurt either. I can feed it a whole config file, a stack trace, and the relevant source all at once.

For the first time, I have a model on the Orin that I trust to do work I'd previously have to ship to an API.

Rough edges

Skip Ollama on Jetson. There's an open bug where gemma4:26b throws HTTP 500s on moderately long context on the Orin specifically. Looks like a memory management issue with Ollama's CUDA integration on Jetson. Stick with llama.cpp.

Long context is slow. The 256K window exists but filling it is painful. Prefill scales with context length, and the Orin's GPU takes a while on 50K+ tokens. I keep my prompts under 10K for anything that needs to respond in under a minute.

Multimodal is experimental. Vision through llama.cpp on Jetson isn't mature yet. The text capabilities are solid.

The punchline

I've been waiting for a model smart enough to be useful and small enough to run on hardware I actually own. Not a rented H100. Not a cloud API I'm paying per token for. Not a Mac Studio I'd have to buy specifically for this. A 60-watt box I already have on my desk.

Gemma 4 26B-A4B is the first model that meets both criteria on the Orin.

It's not going to replace Claude or GPT for complex, multi-turn reasoning. It's not going to write this newsletter. But for the work that needs to happen locally, without a cloud round-trip, it is the best option available today. And it's US open weights, which means I can finally stop the mental asterisk.

The interesting question isn't whether it's as good as a cloud model. It obviously isn't. The interesting question is whether it's good enough that you stop needing the cloud model for a meaningful chunk of your workload.

On my bench, the answer is yes. For the first time.

For a solo founder, it's a trap.

When you ship something broken and you're the only person who can fix it, you don't get to "learn from the market." You get buried in support emails, refund requests, and a reputation hit you can't delegate to anyone. There's no PM absorbing the fallout while you iterate. There's just you, triaging at midnight.

Shipping too early as a solo founder is worse than shipping too late, because the recovery tax falls entirely on one person. You can't parallelize damage control and feature work when you're the whole company.

The framework

Five questions before anything goes live. Not a checklist in a project management tool. Just five things worth thinking through before you hit deploy.

1. Can a new user finish the core loop without help?

Not "can they figure it out if they're technical." Can a regular person sign up, do the main thing, and get value without messaging you? If the answer is no, it's not ready. Doesn't matter how clever the architecture is.

That dead end on one screen size, the missing empty state, the confusing button label. Nobody complains on day one. By day seven you're answering the same question in every support thread.

2. What breaks if twice as many people use it as you expect?

Solo founders don't have an SRE team. If your database melts or your API rate limits get hit on launch day, you're the one SSHing in while your phone blows up. Spend 30 minutes thinking about what happens at 2x your optimistic traffic estimate. Usually the answer is "nothing, because my traffic estimates are delusional." But sometimes it catches a real problem: a missing index, a webhook that retries infinitely on failure.

3. Is there a way to undo this if it goes wrong?

Feature flags. Database backups verified within the last 24 hours. A rollback path that doesn't require rewriting migrations at 2 AM. If you can't reverse the deploy in under 10 minutes, maybe don't ship it on a Friday.

The trap here: migrations that are technically reversible but practically aren't, because the rollback would wipe user data created after the deploy. Test your rollback, don't just confirm it exists.

4. Are you shipping this because it's ready, or because you're tired of looking at it?

There's a specific kind of fatigue that hits around week three of working on a feature. You start telling yourself "good enough" when you mean "I'm sick of this." Those are not the same thing.

If you're excited to announce it, it's probably ready. If you're relieved to be done with it, you're probably trying to escape the project, not finish it.

5. What's the support burden for the first 48 hours?

Can you actually be available? Or are you shipping right before going offline for six hours? Solo founders don't get to ship and disappear. The three people who try it and hit a bug during that window will leave and never come back.

Block your calendar for 48 hours after any significant release. No deep work on other projects. Just monitoring, responding, and fixing whatever surfaces. Boring, but it's the difference between a launch and a mess.

When "just ship it" actually works

Some things genuinely benefit from speed over polish: internal tools only you use, blog posts, landing page copy, config changes. Anything where the blast radius is you and only you.

There's also a middle ground: give it away free during a beta period. People who aren't paying have a completely different tolerance for rough edges. They'll tell you what's broken instead of demanding a refund. They'll stick around through the awkward phase because they feel like they're part of building something, not customers who got shortchanged.

A free beta buys you the one thing solo founders don't have: room to be wrong. You still need to care about the experience, but the stakes on any single release drop significantly when nobody's credit card is attached.

If a stranger will interact with it, run the five questions. If only you will, ship it. If it's a free beta, you can be a little more aggressive — but don't skip question one. A broken core loop wastes everyone's time, paying or not.

The real problem with "just ship it"

The advice assumes you have buffer. Someone to catch what you miss.

Solo founders have none of that. Every hour spent recovering from a bad launch is an hour not spent building the next thing. The math is brutal when there's only one of you.

Five questions, maybe 15 minutes total. Worth it every time.

That thought has killed more promising products than bad code, bad marketing, and bad timing combined.

The platform trap

A tool solves a problem. A platform hosts other people's solutions to other people's problems. The difference in engineering effort, go-to-market strategy, and operational complexity is not 2x. It's 10x or more.

When Shopify was a tool, it helped merchants set up online stores. When it became a platform, it needed an app ecosystem, a developer relations team, an API stability guarantee, a review process for third-party apps, documentation for external developers, a partner program, and a fraud detection system for apps that abuse merchant data. Each of those is a company-sized problem on its own.

Shopify pulled it off because they had hundreds of millions in revenue and thousands of employees when they made that transition. They didn't start as a platform. They earned the right to become one.

Most indie products try to skip directly to platform. They add plugin systems before they have 100 users. They build APIs before anyone has asked for one. They design extensibility frameworks before the core product is stable. Then they spend six months maintaining infrastructure that nobody uses while the core experience rots.

Tools have gravity. Platforms have overhead.

A good tool attracts users because it solves their problem right now. No ecosystem required. No third-party developer community needed. Someone finds it, tries it, and either it works or it doesn't. The feedback loop is fast.

A platform attracts users only when the ecosystem around it has enough value to justify the switching cost. That's a chicken-and-egg problem that burns time and money. You need developers to build apps before users will adopt the platform, but developers won't build apps until there are users. Solving this requires either subsidizing developers (expensive) or building the first wave of apps yourself (which means you're back to building tools anyway).

The tool path: build something, ship it, get users, charge money. Six weeks to first revenue if the product is good.

The platform path: build something, build the developer tools around it, write documentation, recruit developers, hope they build things, hope users find those things. Twelve to eighteen months before the ecosystem generates value, if it ever does.

For a solo founder or small team, the platform path is almost always wrong. Not because platforms aren't valuable. Because you can't afford the timeline.

The premature API

The most common form of platform thinking in early-stage products: building an API before anyone has asked for one.

APIs are expensive to maintain. Once external developers depend on your API, every endpoint becomes a contract. Changing a response format breaks someone's integration. Deprecating a field requires a migration path. Version management becomes a permanent line item in your engineering budget.

If you have 50 users and zero of them have asked for API access, you don't need an API. You need a better core product. The time spent building and documenting an API could have gone toward fixing the three things your actual users complain about.

Build the API when someone emails you saying "I need to integrate this with my system and I'll pay more for it." That's demand. Everything before that is speculation.

The plugin system nobody uses

Same pattern. A product with 200 users adds a plugin architecture. Now the codebase has a plugin loader, a sandboxing layer, a configuration schema, and documentation for plugin developers. Total plugins written by external developers: zero.

The plugin system exists because the founder imagined an ecosystem. Not because users asked for extensibility. The fantasy is compelling: other people building features for your product, for free, that attract more users. In practice, plugin ecosystems only work at scale. WordPress has plugins because it has millions of sites. Figma has plugins because it has millions of designers. Your product with 200 users does not have the gravity to attract plugin developers.

Build the ten features your users actually need instead of building a system for other people to maybe build features someday.

When to actually become a platform

There are real signals that a product should add platform capabilities:

Users are building workarounds. They're scraping your UI, exporting CSVs and transforming them in scripts, or building unofficial integrations. This means the core product has value but doesn't connect to their workflow. An API makes sense here because the demand already exists.

Power users are asking for customization that would fracture the core product. If adding every custom request would turn the product into an unmanageable mess, a plugin or extension system lets power users solve their own edge cases without bloating the main product.

Revenue supports the investment. Platform infrastructure is ongoing cost. API maintenance, developer support, documentation updates, backwards compatibility testing. If the business can't absorb that cost for 12+ months with no direct revenue from the platform layer, it's too early.

A third party offers to build on top of your product and pay for the privilege. This is the clearest signal. Someone with money wants to extend your product for their own commercial purposes. That's real demand.

Everything else is founder fantasy about what the product could become if it were ten times bigger. Build for what it is now.

The tool-first path

Build a tool. Make it good. Charge for it. Get to the point where the tool generates enough revenue and has enough users that platform capabilities become a natural extension rather than a premature bet.

Basecamp was a project management tool for years before it became anything resembling a platform. Notion was a note-taking tool before it had an API. Linear was an issue tracker before it had integrations. Each of them nailed the core experience first. The platform layer came later, funded by tool revenue, pulled by user demand.

There's a version of your product that solves one problem really well for a specific group of people. That version is shippable in weeks, not months. It doesn't need an app store or a developer portal or an API reference. It needs to work, reliably, for the people who have that specific problem.

Ship that. Everything else is a distraction dressed up as ambition.

Neither has gone through a traditional validation process. They might fail. But they exist because I needed them, and that matters more than most founders think.

The validation trap

The standard startup advice goes like this: talk to 50 potential customers, identify a pain point, validate willingness to pay, build an MVP, test it with early adopters, iterate based on feedback.

This process is not wrong. It's just slow, expensive, and biased toward problems that are easy to articulate. The best products often solve problems people don't know how to describe yet. Nobody was asking for "a marketplace for AI agent skills" in 2024 because the concept barely existed. You can't validate demand for a category that hasn't been named.

The validation loop also selects for crowded markets. If 50 people can describe the problem clearly, 10 other teams are already building a solution. You've validated yourself into a competition.

What works instead

Build for yourself first. Not as a growth hack or a marketing angle. As a filter.

If you're building something you actually use, three things happen automatically:

You know the problem is real because you have it. Not "I think users might want this" but "I need this right now and nothing else does it." That's a different level of conviction. It survives the first bad week.

You're your own first tester. Every edge case, every friction point, every moment where the product falls short hits you before it hits anyone else. The feedback loop is instant. No surveys, no analytics dashboards, no waiting for support tickets. You just feel it.

You build faster because you skip the specification phase. When you're building for yourself, you already know the requirements. The PRD is in your head because you lived it. The "what should this do?" question has an obvious answer: whatever you need it to do right now.

The market research industrial complex

There's a whole ecosystem that profits from making founders feel like building is the risky part. Market research firms, customer discovery consultants, survey tools, validation frameworks. They all sell the same premise: the biggest risk is building the wrong thing, and their process reduces that risk.

The premise isn't entirely false. Building the wrong thing does happen. But for solo founders and small teams, the bigger risk is building nothing at all. Analysis paralysis kills more projects than bad product-market fit. You can pivot a launched product. You can't pivot a spreadsheet full of interview notes.

A founder who ships something ugly in two weeks and gets 5 real users has more useful information than a founder who spent three months interviewing 50 people and hasn't written a line of code.

When validation actually matters

There are situations where building without validation is genuinely reckless:

Enterprise software where each sale takes 6 months and costs $200K to close. You need to know the buyer exists and has budget before you invest a year of engineering.

Hardware products where the manufacturing minimum is 10,000 units and $500K. The cost of being wrong is bankruptcy, not wasted weekend hours.

Products that require regulatory approval. If you need FDA clearance or financial licensing, validate first because the compliance timeline is measured in years.

For software products built by a small team with low overhead? Just build it. The cost of being wrong is a few weeks of work. The cost of over-validating is months of delay while someone else ships the thing you were researching.

There are no new ideas

The other thing founders get wrong: chasing originality. Spending months looking for an idea nobody's had before. That's backwards.

If nobody has built it, there's usually a reason. Either the problem doesn't matter enough, the timing is wrong, or someone tried and failed for reasons you haven't discovered yet. Searching for a completely novel idea is one of the slowest ways to start a company.

The better approach: find something that already exists, that people already pay for, and build a version that's better for a specific group. Content scheduling tools exist. Dozens of them. But none of them were built for someone running five brands from one desk with AI agents handling the drafting. That's a niche. That's a differentiator. The category is validated. The specific angle is not.

Stripe didn't invent online payments. PayPal existed. Stripe built a payments API for developers instead of for merchants. Same category, different angle, better product for a specific audience.

Notion didn't invent note-taking or project management. They combined them in a way that attracted a specific kind of user who wanted flexibility over structure.

Linear didn't invent issue tracking. They built a version of it that was fast when everything else was slow, and opinionated when everything else tried to be configurable.

Competition is validation. If other companies are making money in the space, the demand is real. Your job is not to find a market with zero competitors. Your job is to find the gap in an existing market where nobody is serving a specific need well.

Look at what's already working. Find the group of users that existing products are ignoring or underserving. Build for them. That's faster, cheaper, and more likely to work than chasing a completely original idea.

The unfair advantage of building what you need

Products built for yourself have a quality that's hard to replicate: they're opinionated. They make decisions about how things should work instead of trying to accommodate every possible use case.

Basecamp was built because 37signals needed project management for their own client work. Stripe was built because the Collison brothers needed a payment API that didn't make them want to throw their laptop out a window. Slack was a chat tool built inside a gaming company that realized the chat was more valuable than the game.

None of these started with "let's research the project management / payments / enterprise chat market." They started with "this sucks, let me build something better." The product reflected specific, strong opinions about how work should happen. Those opinions attracted users who agreed and repelled users who didn't. That's not a bug. That's positioning.

Generic products built from surveys and focus groups tend toward the middle. They satisfy the average case and delight nobody. Products built from frustration tend toward extremes. They nail one workflow perfectly and ignore everything else. Users who have that exact workflow become devoted fans. Everyone else uses something else, and that's fine.

The practical version

If you're sitting on an idea and debating whether to validate or build:

Ask one question: do you personally have the problem this solves? If yes, build it this week. Use whatever tools get you to a working version fastest. Don't worry about scale, don't worry about the tech stack, don't worry about what happens at 10,000 users. You don't have 10,000 users. You have one: yourself.

Use your own product for two weeks. Write down every time it frustrates you, every time it's missing something, every time you work around a limitation. Fix the worst one. Repeat.

After a month of daily use, show it to three people who have the same problem. Not 50. Three. If two of them say "can I use this?" you have something. If all three shrug, you learned that in a month instead of three months of interviews.

The validation happened. It just happened through building and usage instead of through research and surveys. The information is better because it came from real behavior, not hypothetical answers to hypothetical questions.

The thing nobody tells you about market research

Survey responses are aspirational. Interview answers are performative. People describe the version of themselves they want to be, not the version that actually opens their laptop at 9 AM.

"Would you pay $20/month for a tool that does X?" gets a 70% yes rate in a survey. Actual conversion when the tool launches is closer to 3%. The gap between stated intent and real behavior is enormous, and no amount of careful survey design closes it.

Usage data is the only honest signal. And you can only get usage data from something that exists. Which means you have to build it first.

The entire market validation industry exists because of a reasonable fear (building the wrong thing) applied to the wrong solution (asking people what they want instead of watching what they do). Asking is cheap and feels productive. Building is expensive and feels risky. But building gives you the only data that actually predicts whether the product works.

Build the thing. Use the thing. Fix the thing. Show three people the thing.

That's the whole process.

The LinkedIn agent doesn't know what it posted yesterday. The security agent doesn't remember which servers it already patched. The publishing agent has no idea which drafts are staged and which were rejected. Every session starts from zero. Every session wastes its first few minutes rediscovering context that should have been obvious.

Memory is what turns a chatbot into something useful. Almost nobody builds it right.

What memory actually means in an agent system

When people say "memory" in the context of AI agents, they usually mean one of three things, and they're usually conflating all three.

Conversation history is what ChatGPT gives you. The model remembers what you said earlier in the thread. Cheapest form of memory, least useful for real work. It dies when the session ends. It fills the context window. It can't be shared across agents.

Retrieved context is RAG. Embed documents, store them in a vector database, retrieve relevant chunks at query time. Works for knowledge bases. Terrible for operational memory because relevance scoring doesn't understand time, recency, or task state. A vector search for "what's the current deploy status" might return a doc from three months ago because the words match.

Persistent state is what agents actually need. Not "what documents are similar to this query" but "what happened yesterday, what's in progress right now, and what decisions were already made." Most setups skip this entirely.

My system uses all three, but the third one does the heavy lifting.

The MEMORY.md pattern

The pattern that works: give every agent a MEMORY.md file. Plain markdown. Human-readable. Version-controlled by nature because it lives in a git repo.

A LinkedIn agent's MEMORY.md tracks voice guidelines, which posts performed well, which topics to avoid, and the current posting schedule. A security agent's MEMORY.md tracks which servers have been audited, what vulnerabilities were found, and which patches are pending.

Not a database. A text file the agent reads at session start and updates when something important changes. The format is just markdown:

# MEMORY.md - Agent Name

## Role
What this agent does.

## Current State
What's in progress right now.

## Decisions Made
Things that were decided and shouldn't be relitigated.

## Preferences
How the human wants things done.

Why markdown instead of a database?

Debuggability. When an agent does something wrong, open the MEMORY.md and read it. You can see exactly what the agent knew. No query logs, no embedding inspection, no "why did the retrieval return that chunk?" detective work. Just text.

Editability. Open the file and change it. If an agent has a wrong assumption baked into memory, fix it in thirty seconds. Try doing that with a vector store.

Portability. Every agent's memory is a text file in a git repo. You can grep across all agent memories in one command. Diff changes over time. Copy one agent's memory pattern to bootstrap a new agent.

Semantic search on top of flat files

MEMORY.md handles persistent state. But agents also need to search their memory when a question comes up that isn't covered by the top-level file.

Agents can also have a `memory/` directory with dated entries and topic-specific files. When an agent gets a question about something from last Tuesday, it runs a semantic search across MEMORY.md and everything in `memory/`. The search uses embeddings (text-embedding-3-small works fine) and returns the top snippets with file paths and line numbers.

The flow: agent gets a question, runs `memory_search`, gets back relevant snippets with citations, pulls the specific lines it needs with `memory_get`, and answers with full context.

This is where the "retrieved context" layer comes in, but it's searching the agent's own operational history, not a generic document corpus. A RAG system pointed at your company wiki retrieves information. This retrieves experience.

Why RAG alone fails for operational memory

A common mistake: building agent memory with RAG and nothing else. Embed all the docs, all the Slack messages, all the meeting notes into a vector store and point the agent at it.

It works for answering questions about static knowledge. "What's our refund policy?" gets the right answer because the refund policy doc is sitting in the index and the embedding similarity is high.

It fails for anything time-sensitive or state-dependent. "What did we decide about the pricing change?" might return four different documents from four different meetings because they all discuss pricing. The agent has no way to know which one is current. "What's the status of the deployment?" returns nothing useful because deployment status isn't a document, it's a state that changes every hour.

The fix isn't better embeddings or fancier retrieval. The fix is a separate memory layer that tracks state explicitly. MEMORY.md handles "what is true right now." Semantic search handles "what happened before that might be relevant." RAG handles "what do we know about this topic in general." Three layers, three purposes.

The memory lifecycle

Memory isn't write-once. It has a lifecycle.

Capture: Something important happens during a session. The agent writes it to memory. Not everything, just decisions, outcomes, and state changes. An agent that logs every API call to memory will drown in noise. An agent that logs "deployed v2.3 to production, all tests passing, monitoring for 24h" gives its future self exactly what it needs.

Recall: At the start of every session, the agent loads its MEMORY.md. This is automatic in my system. The agent's workspace files (SOUL.md, MEMORY.md, TOOLS.md) are injected into every session. For deeper recall, the agent runs semantic search when a question requires historical context.

Decay: Old memory needs to age out or get compressed. A MEMORY.md that grows forever becomes useless. I handle this with periodic consolidation: the agent reviews its memory, keeps what's still relevant, archives what isn't, and summarizes patterns. This happens on a schedule, not continuously.

Correction: Sometimes memory is wrong. The agent believed something that turned out to be false, or a decision was reversed. The human edits the MEMORY.md directly. This is why plain text matters. Correcting a vector embedding is a research project. Correcting a markdown file is a text edit.

What breaks when memory is wrong

Bad memory is worse than no memory.

An agent with no memory starts fresh every session. It's slow but safe. It asks questions it's asked before. It redoes work it's done before. Annoying but not dangerous.

An agent with wrong memory acts on false beliefs with full confidence. The security agent that "remembers" a server was patched when it wasn't. The publishing agent that "remembers" a draft was approved when it was actually rejected. The financial agent that "remembers" an invoice was sent when it's still in queue.

This is why MEMORY.md being human-readable matters. Review agent memories periodically. Not every day, but often enough to catch drift. When an agent starts making decisions that don't make sense, check its memory first. Nine times out of ten, something in the MEMORY.md is stale or wrong.

The cost math

Memory isn't free. Every MEMORY.md that gets loaded into a session consumes tokens. Semantic search costs embedding API calls. Storing files costs disk space (trivial) and git history (also trivial).

The real cost is context window usage. A 500-line MEMORY.md takes roughly 2,000 tokens. Across a fleet of agents running multiple sessions per day, that adds up. But prompt caching covers most of it. MEMORY.md files load at session start, which means they hit the cache on subsequent messages within the same session. First load costs full price. Everything after costs roughly 10% because the cached prefix matches.

Without memory, agents waste tokens rediscovering context. An agent that spends 500 tokens re-asking "what's the current status?" every session burns more than the 2,000-token memory file that would have answered it upfront. The net cost of memory is negative.

Getting started

If you're running agents and haven't built a memory layer, start here:

1. Create a MEMORY.md for your most important agent. Put the basics in it: what the agent does, what's currently in progress, and any decisions that shouldn't be repeated.

1. Tell the agent to update its MEMORY.md when something important changes. Not after every message. After outcomes: task completed, decision made, error encountered.

1. After a week, read the MEMORY.md. Is it useful? Does it contain things the agent actually needs to know? Trim the noise, keep the signal.

1. Add semantic search when you outgrow a single file. This usually happens when the agent has been running for a month and the memory directory has enough entries to make search worthwhile.

1. Set a reminder to review agent memories monthly. Catch stale beliefs before they cause problems.

The whole system is plain text files with a search layer on top. No specialized infrastructure. No vector database to manage (the embeddings are computed at query time or cached locally). No migration path to worry about because markdown doesn't have schema changes.

Model quality matters less than you think. Memory quality matters more than anyone talks about.

What I Built

A NVIDIA Jetson AGX Orin 64GB running Qwen 3.5 35B-A3B (MoE, custom quantized) as a local AI model provider, fully integrated into an OpenClaw agent stack. My Mac calls the Jetson over LAN using a simple alias (agx) and gets 35B-level reasoning back at ~30 tok/sec — $0/month, 60 watts.

Hardware & Specs

• Device: NVIDIA Jetson AGX Orin 64GB

• OS: Ubuntu, JetPack R36.4.7 (aarch64)

• CUDA: 12.6

• RAM: 64GB unified memory (CPU + GPU share it)

• Storage: 3.7TB NVMe

• Power: ~60W under load

• Model: Kbenkhaled/Qwen3.5-35B-A3B-quantized.w4a16 (w4a16 quantized, vLLM)

Why MoE matters

Qwen 3.5 35B-A3B is a Mixture of Experts model. 35B total parameters, but only ~3B active per token at inference. That's ~10x less memory bandwidth per inference compared to a dense model of the same size. The dense 27B Qwen variant is slower on the same hardware. MoE wins on edge hardware every time.

What NOT to Do: The Ollama Trap

The standard Ollama build of Qwen does not optimize for Orin's CUDA architecture the same way. If you want real performance out of the Jetson, skip Ollama and use a custom quantized build served via vLLM with CUDA acceleration.

NVIDIA's Jetson AI Lab documents this model officially — that's where I found it: 👉 jetson-ai-lab.com/models/qwen3-5-35b-a3b

The specific quantized build I used is the w4a16 variant optimized for Orin's architecture.

vLLM serves it with an OpenAI-compatible API on port 8000.

Step 1: Verify the Model is Running

SSH into your Jetson and confirm vLLM is serving:

# Check what's listening on port 8000
ss -tlnp | grep 8000
# Should show: LISTEN 0 2048 0.0.0.0:8000 0.0.0.0:*

# Verify the model API is responding
curl -s http://localhost:8000/v1/models | python3 -m json.tool

Test a completion:

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Kbenkhaled/Qwen3.5-35B-A3B-quantized.w4a16",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}],
    "max_tokens": 100
  }' | python3 -c "import json,sys; r=json.load(sys.stdin); print(r['choices'][0]['message']['content'])"

Step 2: Configure OpenClaw on the Jetson

Check what OpenClaw sees:

openclaw models list

Set the local model as the default:

openclaw config set agents.defaults.model qwen
openclaw gateway restart

Gotcha: openclaw config set model qwen doesn't work — model is not a root-level key. The correct path is agents.defaults.model.

Step 3: Clean Up Stale Model Entries

If you have leftover model entries (e.g., from an old Ollama provider or a stale alias), remove them:

# View current models config
openclaw config get models

# Remove stale provider (e.g. ollama pointing at port 11434 that isn't running)
python3 -c "
import json
with open('/home/agx/.openclaw/openclaw.json') as f:
    cfg = json.load(f)
cfg['models']['providers'].pop('ollama', None)
with open('/home/agx/.openclaw/openclaw.json', 'w') as f:
    json.dump(cfg, f, indent=2)
print('Done')
"

# Remove stale model alias from agents config
python3 -c "
import json
with open('/home/agx/.openclaw/openclaw.json') as f:
    cfg = json.load(f)
cfg['agents']['defaults']['models'].pop('kbenkhaled/Qwen3.5-35B-A3B-quantized.w4a16', None)
with open('/home/agx/.openclaw/openclaw.json', 'w') as f:
    json.dump(cfg, f, indent=2)
print('Done')
"

openclaw gateway restart

Gotcha: There's no openclaw models remove command. You have to edit the JSON directly.

Note on Ollama errors: OpenClaw has built-in Ollama auto-discovery that tries port 11434 at startup. If Ollama isn't running, you'll see Failed to discover Ollama models: TypeError: fetch failed in logs. This is cosmetic — it doesn't affect functionality. There's no config key to disable it in 2026.3.2.

Step 4: Allow Remote Exec on the Jetson Node

By default, agx requires approval for every system.run command from a remote session. To allow your main machine to run commands freely:

# Set security to full (no restrictions — fine for a trusted local node)
openclaw config set tools.exec.ask off
openclaw config set tools.exec.security full
openclaw gateway restart

Gotcha: security=allowlist without defined safeBins will give you allowlist miss errors. Use full for a local trusted node, or define your safeBins list explicitly.

Step 5: Add the Jetson as a Provider on Your Main Machine

First, confirm your Mac can reach the Jetson's model server:

# Run this on your Mac
curl -s http://YOUR_JETSON_IP:8000/v1/models | python3 -m json.tool | head -10

Then add it as a custom provider in OpenClaw (run on your Mac, or use the gateway tool):

openclaw config set models.providers.agx-qwen '{
  "baseUrl": "http://YOUR_JETSON_IP:8000/v1",
  "apiKey": "none",
  "api": "openai-completions",
  "models": [{
    "id": "Kbenkhaled/Qwen3.5-35B-A3B-quantized.w4a16",
    "name": "AGX Qwen3.5-35B (local)",
    "input": ["text"],
    "contextWindow": 16000,
    "maxTokens": 4096
  }]
}'

Replace YOUR_JETSON_IP with your Jetson's actual LAN IP:

# Check Jetson IP
hostname -I | awk '{print $1}'

Step 6: Add a Model Alias

# Run on your Mac
openclaw models aliases add agx agx-qwen/Kbenkhaled/Qwen3.5-35B-A3B-quantized.w4a16

Now you can reference it anywhere as agx.

Step 7: Test It from Your Mac

curl -s http://YOUR_JETSON_IP:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Kbenkhaled/Qwen3.5-35B-A3B-quantized.w4a16",
    "messages": [{"role": "user", "content": "Write a one-paragraph story about a robot."}],
    "max_tokens": 300
  }' | python3 -c "import json,sys; r=json.load(sys.stdin); print(r['choices'][0]['message']['content'])"

GPU Health Check

# Real-time stats (run on Jetson)
tegrastats

# Or one-shot summary
nvidia-smi --query-gpu=name,temperature.gpu,utilization.gpu,memory.used,memory.total,power.draw --format=csv,noheader,nounits

# System overview
free -h          # RAM
df -h /          # Disk
uptime           # Load average

Healthy idle numbers on the AGX 64GB running Qwen 3.5 35B-A3B:

• RAM: ~56GB used (model loaded in unified memory)

• GPU temp: ~46°C

• GPU utilization: ~10% idle, spikes during inference

• Power draw: ~3.5W idle

Gotchas Summary

• Issue: Unrecognized key: "model"

  Cause: Wrong config path

  Fix: Use agents.defaults.model not model

• Issue: SYSTEM_RUN_DENIED: approval required

  Cause: Default node security

  Fix: Run openclaw config set tools.exec.security full on the node

• Issue: SYSTEM_RUN_DENIED: allowlist miss

  Cause: security=allowlist with no bins defined

  Fix: Switch to full or define safeBins

• Issue: openclaw models remove not found

  Cause: Command doesn't exist

  Fix: Edit openclaw.json directly with python3

• Issue: Ollama errors at startup

  Cause: Built-in discovery, can't disable

  Fix: Ignore — cosmetic only

• Issue: Model output includes thinking chain

  Cause: Reasoning mode baked into model

  Fix: Add system prompt telling it to skip thinking, or disable reasoning in vLLM config

Hardware Note: Orin Nano Super (8GB)

Same approach works on an Orin Nano Super (8GB) — just use a smaller model. The Qwen 3.5 Small series (just released March 2026, 0.8B–9B range) is built for on-device/edge and fits the 8GB form factor. Methods are identical.

Why Bother?

• $0/month operating cost

• No API latency, no rate limits, no data leaving your network

• 60W power draw — runs all night on overnight tasks

• 35B-level reasoning for background jobs: research, batch processing, coding runs

• Full tool-use and thinking capabilities

• Still might become the brain for a robot someday

Originally set up March 6, 2026. All commands verified in production.

This isn't a bug. It's how language models work. Understanding it changed how I build everything.

Two kinds of workflows

Deterministic workflows (n8n, Zapier, custom code) follow the same path every run. Step 2 fires after Step 1. If it breaks, it breaks the same way, which means you can find it and fix it. The limitation: you specify everything upfront. Every edge case. Every branch.

AI skill workflows don't have a fixed path. The model reads instructions, reasons about the task, and decides what to do. It handles ambiguity and adapts to context you never anticipated.

The tradeoff: execution quality depends on the model doing the work. Same skill file. Different model. Different results.

How AI workflows actually fail

Traditional software fails with wrong logic. You find it, you fix it.

AI workflows fail by premature completion. The model decides the task is "done enough" and wraps up. No error. No warning. It just stops.

This happens because language models are trained to be helpful and responsive. Finishing feels like the right move. The longer and more complex the task, the more likely the model cuts a corner, especially on steps that feel administrative or repetitive.

I see this constantly. My daily content pipeline has 7 steps. A weaker model hits Step 4, feels like the main work is done, and summarizes. A stronger model follows through to Step 7 even when it's grinding through the fifth piece of content in a row.

Design for compliance, not just clarity

Tightening the skill file helps more than switching models (though model choice matters too).

Be explicit about completion. "DO NOT skip this step" and "This step is MANDATORY" aren't redundant. They counteract the model's natural tendency to treat later steps as optional.

State completion criteria. Instead of "write social posts," write "write social posts for all four platforms: Twitter, Instagram, TikTok, YouTube Shorts. All four must be present before this step is complete."

Use memory for standing rules. If a step gets skipped repeatedly, add it to memory with the date it was corrected. Models read prior corrections as high-priority context.

Verify outputs, not just completion. Don't check whether the model said it finished. Check whether the output files exist, the word counts are right, the required sections are present.

The hybrid architecture

The most practical solution for complex pipelines: use deterministic tools for orchestration and AI for content.

n8n handles the skeleton. Trigger at 8am, pass Step 1 output to Step 2, wait for approval gate, continue. The structure is reliable. AI fills in the variable parts: writing the summary, picking the angle, adapting tone.

The mental model: n8n is the project manager, AI is the writer. The project manager doesn't forget steps. The writer doesn't need to think about pipeline logic.

What to actually do

If you're running AI workflows today:

1. Audit your skill files for vague step language. "Write social content" is not a complete instruction.

2. Add explicit completion checks. List every required output, not just the task.

3. Test with a weaker model. If a cheaper model skips steps, your skill isn't tight enough. Fix the skill, not the model tier.

4. Consider hybrid architecture for any pipeline longer than 4-5 steps. The complexity cost of n8n pays off fast.

The goal isn't to make AI workflows as reliable as traditional software. They can't be. The goal is to design them so the model's flexibility fills the right gaps, and none of the mandatory ones.

Three engineers at StrongDM built production security software in 2025 under two rules: no human writes code, no human reviews code. They shipped it. It's running in production.

I've been following this story since they published their methodology in February. My reaction was something between "obviously this is where things are headed" and "I genuinely don't know how I feel about that."

The domain matters here. StrongDM isn't building a todo app. They're building access management software — the kind that controls who can touch what across Okta, Jira, Slack, and Google Drive. If it has a flaw, the blast radius is real. The fact that no human reviewed the code doesn't make it smaller.

The testing problem they actually solved

The part that stuck with me: agents cheat. Not deliberately, but effectively. If a test checks whether a function returns a specific value, the agent will hardcode that value. Test passes. Software is broken. The model found the shortest path to green and didn't care whether it was useful.

This isn't a new problem. Goodhart's Law has been around since 1975. What's new is that the cheater is your software, and it's faster at gaming metrics than you are at writing them.

StrongDM's fix: treat validation like a machine learning holdout set. Store test scenarios completely outside the codebase, where the agent can't read them. Their evaluation framework tests user-level outcomes — did the software do what the user needed, not did this function return the right value.

They call this measuring "satisfaction." I'd call it the right question.

The fake infrastructure play

They also built behavioral clones of every third-party service the software integrates with. Full replicas of Okta, Jira, Slack, Google Drive — their APIs, edge cases, observable behaviors — running locally with no rate limits and no production risk. They call it a Digital Twin Universe.

With it, they run thousands of test scenarios per hour. The setup lets them:

1. Simulate failure modes that would be dangerous to test against live systems

2. Run the same scenario thousands of times without rate limits

3. Have the agents building the software also build the testing environment

Six months ago, faithfully replicating even one major SaaS API was economically absurd. Now it's table stakes for this team.

The accountability question nobody has answered

When no human has read the code, who's responsible for what it does?

There's no good answer yet. Stanford Law flagged it two days after StrongDM's announcement. Existing software liability frameworks assume a human made decisions about what shipped. The legal infrastructure for "the model decided" doesn't exist.

This matters for anyone building AI-first. Your outputs have consequences regardless of whether a human touched the code.

The number

StrongDM's benchmark: if you're not spending at least $1,000 per engineer per day on tokens, your factory has room to improve.

That's $20K/month per engineer in inference before salaries. The math works if three engineers can build and maintain production security software without reviewers. It doesn't work for most teams today.

But the cost comes down as models get cheaper, and the methodology — scenario holdouts, digital twins, probabilistic validation — scales in both directions.

Whether this becomes standard practice is a question 2026 is answering right now. I'm watching closely.

Most people are still calling this "better prompting." I don't think that's right.

What's actually happening: we're moving from one-off instructions to software-defined environments. When I started building agent teams for my projects, I thought in terms of prompts. Now I think in terms of files. SKILL.md, CLAUDE.md, .clinerules, action schemas — each one is a deployable artifact with a specific scope and lifecycle.

The difference matters more than it sounds.

A prompt is temporary. A skill file is infrastructure. When I write a SKILL.md for one of my agents, I'm authoring behavior: what it does, in what order, under what constraints, and when it stops. That's configuration code. It belongs in version control, gets reviewed when it changes, and gets maintained like anything else in your stack.

I learned this the hard way. Early versions of my agent setups were sprawling instruction blobs I'd partially remember and inconsistently apply. New behavior introduced in one session would disappear in the next. The agent's reliability was inversely proportional to how long it had been since I'd touched the config.

Once I started treating these files like actual code — committed to git, organized by concern, with clear inheritance — everything got more predictable. Not because the models improved. Because the configuration did.

The layering pattern

A setup worth copying in 2026:

1. A root config for global defaults and guardrails

2. Skill modules for work you do repeatedly

3. Project-level overrides that inherit from the base

4. Memory files that persist context across sessions

One configuration for writing. Another for product work. Another for ops. Each tuned, versioned, owned by you.

The alternative is relying on vibes and hoping the agent remembers what you told it last week.

The security surface nobody's thinking about

If behavior lives in files, those files are part of your attack surface. Not theoretically — practically.

Third-party skill files should be treated like third-party dependencies:

• Review them before running them

• Scope permissions to what's actually needed

• Don't give an agent write access when it only needs to read

Most people haven't applied basic hygiene standards to their agent configs yet. That gap will close, probably after something goes wrong.

What to actually do

Put your config files in git. Write commit messages that explain why a rule changed, not just what changed. Build skills that do one thing and can be tested in isolation.

Think through your precedence model deliberately: what's global, what's project-specific, what can a single user override. Those answers exist — they just require thinking it through rather than letting the config accumulate by accident.

A well-built agent stack can be cloned, backed up, handed off, and iterated. It's your operating layer. Treat it like one.