Controlling Robots in NVIDIA Isaac Sim from ROS 2 Jazzy

Introduction

In the process of running the Techman Robot (TM5-700) in the latest environment, I encountered numerous errors. In particular, I had to adjust code provided for ROS 2 Jazzy and compensate for behaviors that differed from documentation due to Isaac Sim version upgrades. Here, I share some points that required significant trial and error.

Development Environment

OS: Ubuntu 24.04 LTS
ROS 2: Jazzy Jalisco
Simulator: Isaac Sim 5.1.0
Robot: Techman Robot (TM5-700) Repository here
Tool: MoveIt 2
Python 3.11 (Required for Isaac Sim 5.1.0, created environment using Miniconda)

Overall Workflow

Clone robot data and run colcon build
Launch RViz2 with the robot deployed using ros2 launch
Open Isaac Sim and place the robot
Plan motion in RViz2 and press "Execute"
Observe actual motion in Isaac Sim

However, I encountered several errors before reaching the actual motion verification stage. Below are records of their solutions.

🛠 Troubleshooting: Errors and Solutions

1. Missing Python Modules During Build

【Error Message】 ModuleNotFoundError: No module named 'catkin_pkg' or No module named 'yaml'

【Cause】 The Conda virtual environment (env_isaacsim) lacks Python libraries required for ROS 2 build and execution.

【Solution】 With the virtual environment activated, directly install missing libraries using pip.

pip install catkin_pkg PyYAML lark empy==3.3.4

2. ROS 2 and Python Version Incompatibility

【Error Message】 ModuleNotFoundError: No module named 'rclpy._rclpy_pybind11'

【Cause】 ROS 2 Jazzy expects Python 3.12, but if the virtual environment uses a different version like Python 3.11, binary compatibility is lost and modules cannot be loaded.

【Solution】

During the tutorial, use conda deactivate to use system Python, or recreate the environment with conda create -n env_name python=3.12 matching the OS version.
Continue running Isaac Sim in the conda Python 3.11 virtual environment as before.

3. MoveIt 2 Cannot Find Controllers

【Error Message】 Returned 0 controllers. Check that the controller manager is up and running.

【Cause】 Starting with ROS 2 Jazzy, YAML file parsing has become stricter, and syntax from older versions like Humble is no longer recognized.

【Solution】 Update the hierarchical structure of moveit_controllers.yaml to the latest format (including /** and ros__parameters structure).

4. Joint Name Mismatch

【Symptom/Error】 Warnings appear that joint names cannot be found on the Articulation Controller side.

【Cause】 Joint names in the Isaac Sim robot model (e.g., shoulder_1_joint) don't match MoveIt configuration (e.g., joint_1).

【Solution】

Since it's natural for the simulation environment to match the actual system, it's recommended to use Name Override in the ROS2 Subscribe Joint State node of each Joint property on the Isaac Sim side to align names with MoveIt.

5. File Structure

The following files were edited for this work:

CMakeLists.txt
package.xml
config/kinematics.yaml
config/moveit_controllers.yaml
config/ompl_planning.yaml
config/pilz_cartesian_limits.yaml
config/ros2_controllers.yaml
config/tm5-700.srdf
launch/demo.launch.py

tm5-700_moveit_config/
├── CMakeLists.txt           # Build configuration
├── package.xml              # Package metadata and dependencies
├── .setup_assistant         # MoveIt Setup Assistant configuration
├── config/                  # Configuration files
│   ├── tm5-700.urdf.xacro      # Robot model definition
│   ├── tm5-700.srdf            # Semantic description (groups, poses, etc.)
│   ├── tm5-700.ros2_control.xacro  # ROS2 Control configuration
│   ├── moveit_controllers.yaml  # MoveIt controller configuration
│   ├── ros2_controllers.yaml    # ROS2 controller configuration
│   ├── controllers.yaml         # Controller parameters
│   ├── kinematics.yaml          # Kinematics solver configuration (KDL)
│   ├── joint_limits.yaml        # Joint limits
│   ├── initial_positions.yaml   # Initial poses
│   ├── ompl_planning.yaml       # OMPL planner configuration
│   ├── chomp_planning.yaml      # CHOMP planner configuration
│   ├── stomp_planning.yaml      # STOMP planner configuration
│   ├── trajopt_planning.yaml    # TrajOpt planner configuration
│   ├── pilz_cartesian_limits.yaml  # Pilz Cartesian limits
│   ├── sensors_3d.yaml          # 3D sensor configuration
│   └── moveit.rviz              # RViz configuration file
└── launch/                  # Launch files
    ├── demo.launch.py           # Demo launch
    ├── move_group.launch.py     # move_group node launch
    ├── moveit_rviz.launch.py    # RViz launch
    ├── rsp.launch.py            # robot_state_publisher launch
    ├── spawn_controllers.launch.py  # Controller launch
    ├── static_virtual_joint_tfs.launch.py  # Virtual joint TF
    ├── warehouse_db.launch.py   # Database launch
    └── setup_assistant.launch.py # Setup Assistant launch

6. Isaac Sim 5.x Generation Notes

When researching how to accomplish similar tasks, I found many documents using the Action Graph ROS2 Follow Joint Trajectory node extensively. However, this node is not included in my Isaac Sim 5.1. To achieve the same behavior, the current standard approach is to connect ROS2 Context -> ROS2 Subscribe Joint State -> Articulation Controller.

On top of that, create a bridge in Python.

import rclpy
from rclpy.node import Node
from rclpy.action import ActionServer
from control_msgs.action import FollowJointTrajectory
from sensor_msgs.msg import JointState

class IsaacActionBridge(Node):
    def __init__(self):
        super().__init__('isaac_action_bridge')
        self._action_server = ActionServer(
            self,
            FollowJointTrajectory,
            '/tmr_arm_controller/follow_joint_trajectory', # Must match MoveIt configuration
            self.execute_callback)
        self.publisher_ = self.create_publisher(JointState, '/joint_command', 10)
        self.get_logger().info('Action Server Bridge started. Waiting for MoveIt...')

    def execute_callback(self, goal_handle):
        self.get_logger().info('Executing trajectory...')
        trajectory = goal_handle.request.trajectory
        
        # Send the points to Isaac Sim
        for point in trajectory.points:
            msg = JointState()
            msg.header = trajectory.header
            msg.name = trajectory.joint_names
            msg.position = point.positions
            self.publisher_.publish(msg)
            # Small sleep to simulate motion (Optional)
            # time.sleep(0.01)

        goal_handle.succeed()
        result = FollowJointTrajectory.Result()
        return result

def main(args=None):
    rclpy.init(args=args)
    node = IsaacActionBridge()
    rclpy.spin(node)
    node.destroy_node()
    rclpy.shutdown()

if __name__ == '__main__':
    main()

7. Summary

This workflow enables planning and executing motion in RViz2, which then simulates the arm accordingly in Isaac Sim.

TM Robot's provision of source code was incredibly helpful, but since there isn't comprehensive step-by-step documentation, it took some time to understand.

When Isaac Sim moved after pressing Execute in RViz2, it was the moment all the effort paid off. I hope this information is useful to someone.