Nvidia Isaac Sim: How to publish the state of a Surface Gripper to a ROS2 topic?

Introduction

In this article you will learn how to publish the state of a Surface Gripper to a ROS2 topic as JointState message.

As with most tasks in Isaac Sim, they can be accomplished using two methods:

1. Visually using the Isaac Sim application’s graphical user interface
2. Programmatically using the python omni.isaac.* packages

Why would I want the state published as a ROS2 message?

ROS stands for Robot Operation System. As seen on the website:

ROS is a set of software libraries and tools for building robot applications

This has become widely adopted across the industry and allows interop between different systems which makes it an attractive communication protocol. Specifically, the tooling allows recording robot states and replaying to apply those states using well-defined standards.

In most cases, to maximize the utility of your gripper, it will be attached to a robot arm such as the Franka or Panda. These arm models will come with formal joints, and you can use the native ROS2 JointStates extension included with Isaac Sim to publish their positions, velocities, and forces.

Using the ROS2 JointStates Utility

Given we want to control or set the complete state of the robot when replaying, we also need to publish the state of the Surface Gripper to make it available for recording.

Why can’t we use the same method for Surface Gripper?

Most end effectors having moving parts that apply mechanical forces to other objects based on friction or object collisions. These types of grippers have actual *Joint primitives in their hierarchy which can found and published.

Fixed and Revolute Joints of UR3E robot arm

However, Surface Grippers are constructed differently. They are not simulated using moving parts, thus do not have Joints with angle or force properties to be queried.

How do Surface Grippers work?

Surface grippers internally simulate a force towards the gripper and when the distance to other nearby objects drops below a set threshold, it dynamically applies a Fixed Joint between the two objects simulating the “grip” effect and causing the two objects to behave as a single object until the gripper is intentionally opened or the force exceeds a limit, and the grip is “broken”.

This is why Surface Grippers have these force related properties: Grip Threshold, Force Limit, Torque Limit, Bend Angle, Stiffness, Damping

In the real world, the force is most commonly created using suction, but you could also use electromagnets.

You can find the Surface Gripper extension at this location of your Isaac Sim installation. The location is dependent on your Operating System

• Windows: C:\Users\<user>\AppData\Local\ov\pkg\isaac-sim-4.0.0
• Linux: /home/<user>/.local/share/ov/pkg/isaac-sim-4.0.0

Surface Gripper extension path within installation path:
exts\omni.isaac.surface_gripper\omni\isaac\surface_gripper

If a Surface Gripper can only open or closed, why do we use a JointState type message instead of Boolean?

Our goal is to use the record states of robots during demonstrations and then replay them. Replaying means not only do we need the state, but the time the robot was in a certain state. Simply recording a Boolean would not be sufficient.

Using JointState message includes a standard Header with frame_id and Time in Seconds and Nanoseconds. JointState also has the added benefit of being consistent with traditional mechanical grippers and adaptable to more complex gripper types which have more than single state variable in the future.

High-level Objective

We will manually convert the Surface Gripper.closed Boolean into at JointState type message representing the True/False as joint Positions.

As I mentioned in the Introduction, we will show how to do this through GUI and using Python APIs.

In both cases, we will construct a basic scene that mimics or extends the Surface Gripper sample using a Cone to represent the gripper and a Cube to represent the gripped object. We will then add ActionGraph nodes to implement the conversion and ROS2 publishing with some extra nodes for user interaction and observability.

(This will make more sense when you see the demonstration)

Setup / Prerequisites

Launch Isaac Sim with ROS2 libraries

1. Launch Isaac Sim from the Omniverse Launcher
2. On the Isaac Sim App Selector
   Set “ROS Bridge Extension”: omni.isaac.ros2_bridge
   Set “Use Internal ROS2 Libraries”: humble

Isaac Sim App Selector with ROS2 Bridge and Humble libraries

Setup ROS2 tools on your machine

In order verify that we are actually publishing the state as we expect, we need to be able to observe the ROS2 information published from Isaac Sim. For this, you need the ROS2 CLI installed.

I installed this on Windows 11, within WSL using this guide:
https://docs.ros.org/en/humble/Installation/Ubuntu-Install-Debians.html

Verify your installation by entering ros2 in the terminal

Verify usage of ros2 cli tools in Ubuntu on WSL

Using the GUI

Create Scene

We will create a minimal scene with ground plane, cube, and cone.

Add Ground Plane
Create > Physics > Ground Plane

Add Cube
Right Click “World” > Create > Shape > Cube

Set Translate: 0, 0, 0.5
Enable Rigid Body with Collider Preset

Add Cone
(Follow similar steps steps as Cube, but select Cone)
Set Translate: 0, 0, 1.5
Enable Rigid Body and Collider Preset

Checkpoint

Your scene should look like this:

Create the Action Graph

Right Click “Cone” > Create > Visual Scripting > Action Graph

• With Action Graph active, Press “Tab” to open context dialog
• Search “Surfac Gripper”, drag Surface Gripper onto the Graph

• Select the Surface Gripper Node
• In the properties pane, set the ParentRigidBody to the Cone primitive you created abo

Following the same techniques demonstrated above for adding nodes to the graph and setting properties

• Add two more OnKeyboardInput nodes to the graph
  - One for Close (C) and one for Open (O)
• Connect each to Close and Open inputs of the Surface Gripper

At this point, it looks as if we have everything we need for the surface gripper to function; however, if you run the simulation, press “C” to force the gripper into closed state, and attempt to “grip” the cube, it will not work! Why is that?

Add the Grip Position

You may remember when you set the ParentRigidBody property on the Sufrace Gripper Node there was another property above called GripPosition. This is the missing component!

The grip position is acts as the origin of the attraction force towards the gripper. Without it, the calculations to qualify another object being gripped or not cannot be computed.

• Add Xform to ActionGraph
  Right Click “Action Graph” > Create > Xform

• Rename the Xform to GripPosition
• Set Translate to the bottom of the Cone
• Based on the documentation the force is generated in the X direction
  We must orient the primitive with X facing down, or out of our gripper

• On the Surface Gripper node, set GripPosition to the GripPosition Xform

Now re-test the Surface Gripper and it works! 🥳

Until this point in the article we have only recreated the basic Surface Gripper as shown in the official tutorial from Nvidia

Now, we want to move on the novel information this article introduces which is the conversion and ROS2 publishing.

Add nodes to publish ROS2 message

• Click the ActionGraph set to context.
• Press Tab to open the Quick Search menu
• Type “ros2 publisher”
• Drag that ROS2 Publisher item to the Action Graph

Note: You will use this quick search method to add nodes to action graph for the reminder of article

Select the node and set the following properties:

• messageName: JointState
• messsagePackage: sensor_msgs
• nodeNamespace: surface_gripper
• topicName: joint_state

Now will work backwards and set the other required inputs on the node: Exec In, Header Stamp, and Position

Add ROS2 Context, On Playback Tick, Isaac Read Simulation Time, and Isaac Time Splitter and connect them as seen below

Next we will perform the conversion from the SurfaceGripper.outputs.Closed bool to the ROS2Publisher.inputs:Position double[]

Create two Constant Doubles, Select If, and Make Array nodes and connect them as shown below.

Note: You can hover over the inputs and outputs of nodes to see the types they produce or expect.

Conversion from bool to double[]

Now we want to confirm publishing. Press Play to start simulation, Open another terminal and type ros2 topic list

This confirms the node is publishing to that topic. Now let’s inspect the data being published to confirm the JointState schema.

I have boxed one of the messages and underlined the Position of 1 when the gripper is in Closed state.

Congratulations! 🥳 🤖

You have successfully published the state the Surface Gripper to a ROS2 topic. The remaining steps are extra, nice to have features.

Add nodes to display the gripper state on the screen

Convert closed bool to a string, then insert that True/False value into “Closed: ” at index 8, and finally connect to Print Text node.

Add nodes to change the color of the cone based on gripper state

I chose to create OmniPBR materials with Green for open and Red for closed and assign them to the cone primitive based on gripper state.

Create two Constant Paths nodes referring to the material paths, and based on the bool state, write the Material path to the Gripper’s Material.

Final Result!

When open the text displays “Closed: False” and the gripper is green
When closed the text displays “Closed: True” and the gripper is red

Gripper with state printed to screen and material swapped based on state

Final GUI .usda file

You can download this scene .usda file here

Using Code

In the above setup section, we noted how to launch Isaac Sim with ROS2 bridge and humble libraries and also how to setup ROS2 tools on our Ubuntu distribution in WSL. If you skipped those to get to the code quickly, please ensure you have those requirements met before continuing.

When we write code to control Isaac Sim, we do this through Extensions.
In fact, Isaac Sim itself is an Omniverse Kit-based app that is composed of other extensions. The extension we write is programmatically executing all of the actions we did above through the GUI.

Create a New “Loaded Scenario” Extension

You can follow Nvidia’s official documentation for these steps:

Isaac Sim Workflows - Omniverse IsaacSim latest documentation

Isaac Sim Extension Templates - Omniverse IsaacSim latest documentation

I will call mine SurfaceGripperToROS2

Add the path to the extensions search paths and enable the extension

Checkpoint 1

Verify default extension is working

• Open the SurfaceGripper extension and click the “Load” button
• It should populate the scene with a UR10e robot arm.

Basic Extension and Editor Setup

While the default template works, in my opinion, it is a really poor default.
I think it has outdated coding practices, a mixing of concerns between ui_builder and scenario files which causes confusion about responsibility, and has poor tooling support.

I will walk through the steps to upgrade to a more modern setup. Although there are significant changes, and at the end of this section I include a link to a start branch you can clone with equivalent code changes.

Setup Git Repository

• git init
• Add standard Python.gitignore
• Create minimal, Initial Commit

Setup Python analysis paths and use Isaac Sim python interpreter

By default, VSCode’s python analysis will not work because Isaac Sim extension code is executed with paths dynamically added to PYTHONPATH to make other extensions/packages available. Such as omni.isaac.ui

We fix this by explicitly telling VS Code where to find these packages and to use them for analysis and auto complete.

• Create .vscode/settings.json with these contents

Note we use Isaac Sim installation path which is dependent on the OS, user’s name, and the Isaac Sim version (4.0.0, 4.1.0) and you should adjust accordingly

More standardization

• Add .vscode/extensions.json with these contents and install recommended extensions
• Add .gitattributes, .editorconfig, .flake8, and pyproject.toml with contents matching the code here
• Add .vscode/launch.json with these contents to enable debugging your extension

Code Cleanup

• Update README.md
• Removal of _physx_subscription in ui_builder
• Improve variable and function naming
• Make ScenarioTemplate an Abstract class
• Remove _setup_scene from ui_builder (Scene setup should be part of the scenario)
• Remove comments and copyright notices
• Make Scenario#setup_scenario async
• Add omni.log logging

Finally, we can get to actually implementing our extension! 😅

Setup the Scene

Add ground plane, dome light, cube, and cone.
The cube should be above the ground plane, and cone above the cube.
Note: I also updated the refinement level of default cone to 2. This should better match the default setting from the GUI

Checkpoint!

Similar to the setup we did using the GUI you should see something like this.

If the changes were unclear, you can get the completed by cloning the repo and checking out the start branch.

git clone https://github.com/mattmazzola/surface_gripper_to_ros2 -b start

Create the Action Graph

There is a significant amount of code to accomplish this step, and I won’t include it all; however, I will point out the notable operations.

Remember, we are recreating the same graph we created above so you can refer to it to understand all the nodes, values, and connections to make.

Use og.Controller.edit to create, set, and connect nodes

Controller Class - kit-omnigraph 1.26.3 documentation

keys = og.Controller.Keys

og_path = get_next_free_path("SurfaceGripperActionGraph", "/graph")
(graph_handle, nodes, _, _) = og.Controller.edit(
    {"graph_path": og_path, "evaluator_name": "execution"},
    {
        keys.CREATE_NODES: [
            # Inputs
            ("OnImpulseClose", "omni.graph.action.OnImpulseEvent"),
            ...,
        ]
    }
)

Note: Certain properties or connections of nodes cannot be set until other pre-requisite properties are applied which change the schema of node.

For example, we cannot connect IsaacTimeSplitter.outputs:seconds to ROS2Publisher.inputs:header:stamp:sec until the type of the ROS2Publisher message is set to sensor_msgs.msg.JointState

Also, I wasn’t able to re-use og.Controller.edit for the subsequent edits to the same graph. (There is probably a way, but I couldn’t find so I used the more fine-grained dedicated APIs og.Controller.connect

Controller Class - kit-omnigraph 1.26.3 documentation

og.Controller.connect(
    og.Controller.attribute(og_path + "/IsaacTimeSplitter.outputs:seconds"),
    og.Controller.attribute(og_path + "/ROS2Publisher.inputs:header:stamp:sec"),
)

Add Buttons to Control Gripper state

Previously we had only matched authoring experience using the GUI. Now, we will add an additional feature and useful technique which demonstrates how interacting with the UI can trigger flows in the Action graph.

You may have noticed the previous code added “Impulse” nodes. When the node is selected in the graph, the user can manually click to trigger impulse. However, we didn’t want to require having the node selected so that is why we added keyboard input. The impulse nodes can also be programmatically triggered, and this is how we accomplish our goal.

Add buttons and click handlers to programmatically send an impulse on the respective open and close nodes.

og.Controller.set(
    og.Controller.attribute("/graph/SurfaceGripperActionGraph/OnImpulseClose.state:enableImpulse"),
    True,
)

Using Buttons to Control Gripper State

Congratulations… Again! 🥳🤖

Run the simulation and test your gripper!
You should see something like below, if you don’t you can check the final code in the repository.

git clone https://github.com/mattmazzola/surface_gripper_to_ros2

Conclusions

We have demonstrated how to output the state of a Surface Gripper as a JointState message to a ROS2 topic.

I hope this article has helped you better utilize Surface Grippers in your Isaac Sim projects.

Let me know what you liked, didn’t like, or ideas for future expansion in the comments!