Comparing UI programming patterns between React and NVidia Omniverse
In this article I will show you how implement common UI patterns in NVidia Omniverse by associating with familiar patterns from the popular web application framework, React.
Notable patterns
- Components
- Derived or Computed Properties
- Conditional Rendering
- Styling and Themes
- Dialogs
- Sharing state between components
I will build small application in React demonstrating the patterns above. Then, I will show the code necessary to replicate that functionality in Omniverse. Specifically, we will build an Extension for Isaac Sim, but the larger programming style is applicable across the Omniverse Kit application platform.
Before we go into specific code for each pattern, I think it is valuable to prime your expectations for the article which I hope will make the understanding the implementations easier.
- Review Demos of the React Application and Omniverse Extension
To understand what are building and how the patterns are observed - Review a table mapping lower-level HTML, CSS, JavaScript of web applications to python in Omniverse Kit applications
Helps directly associate pairs of code snippets - Compare programming paradigms
To understand why the patterns are implemented differently
Application Demos
React
Code: CodeSandbox
Remember the goal of implementing the patterns above
- Counters are implemented as components
- Styles are applied to main UI, Components, and Modal
- Computed property from sum of counter values
- Conditional rendering “Small” or “Large” description of Total Value
- Dialog or Window with state shared, the total value, from the parent component or Window
— 3rd counter instance affects the shared total value, demonstrating live updates to shared state.
— 4th counter on demonstrates independent state
Omniverse
Similar to the React application, but with Windows arranged side by side to more easily see state updating in both windows. Also, there is an extra feature of menu bar and demonstration of those interactions.
Mapping of Concepts
What are the major differences between the programming paradigms?
Modern web frameworks make implementing these patterns easier because they allow programming in a declarative manner.
This means you describe or declare how the UI should look by writing templates, in the form of JSX or Handlebars, and data or state is applied to that template to output the final format to render such as HTML. The framework handles re-rending by detecting state changes and recomputing changes to the template.
This programming paradigm abstracts away the complexity of how and when to update which reduces both the programming and conceptual complexity. This declarative model is why those frameworks have become so popular.
In NVidia Omniverse extension development, there is still an imperative programming model. This means the responsibility of re-rendering in response to data changes must be explicitly programmed by the developer.
In my opinion, while imperative may allow better performance because you only render exactly what is necessary, in practice, it is almost always implemented incorrectly or incompletely. In other words, updates that should happen do not. This causes stale UI. It also decreases the speed of development is slower due to the extra complexity. This is why we see the heavy bias towards declarative paradigms.
Omniverse Documentation
Additionally, learning Omniverse extension development is made more challenging since NVidia’s documentation is fragmented and not well maintained. It can be difficult to find what you are looking for, and even when you do, it may contain broken links or incomplete markdown syntax that does not render supporting images.
In fact, parts of their omni.*source code have incorrectly documented variables in docstrings, missing or mismatched return types, missing python definitions for autocomplete. All which contribute to a poor developer experience. If you're coming from modern practices, you may lower your expectations. You have been warned. 😉
In my opinion, code in the UI samples they do provide is not well written. There are odd choices for variable or function names which can mislead the reader of their purpose causing them to not absorb the key idea that was intended to be presented.
The difficulties of programming UI in Omniverse are why I think this article can provide value.
Despite the drawbacks mentioned above, I still think there is value in learning Omniverse as the potential dominant platform for developing embodied AI models through simulation.
By leveraging the existing knowledge of the much larger audience of web developers and showing them how their skills can be applied to Omniverse, it may help lower the barrier to entry and encourage community growth. The networks effects will accelerate creation of resources available for everyone. 🎉
Reviewing Pattens
In these next sections I will go through how we implemented patterns in Omniverse. You can look at the source code for the React application, but I won’t highlight the code here. It is better to check out the official documentation if you are unfamiliar with React. Also, I was admittingly a little sloppy on the implementation of the state in React and know it’s not done in the most optimal manor 🫢
Creating Components
I will call this pattern “Components” to sound familiar, but Omniverse does not have a first-class concept of component. We look at the properties components provide and try to replicate them in Omniverse.
First, we look at the Counter component with only ui code.
You can think of the VStacks and HStacks as different div s with flexbox and flex-direction applied. Spacing is like gap and the Spacer elements are like placeholders with flex: 1 applied.
def counter_component(title: str = "Counter"):
# https://docs.omniverse.nvidia.com/kit/docs/omni.kit.documentation.ui.style/latest/styling.html#customize-the-selector-type-using-style-type-name-override
with ui.VStack(
style=counter_style,
spacing=10,
):
ui.Label(
title,
alignment=ui.Alignment.CENTER,
name="title",
)
with ui.HStack():
ui.Spacer()
with ui.HStack(
spacing=0,
width=200,
height=50,
alignment=ui.Alignment.CENTER,
):
ui.Button(
text="Dec",
name="dec",
style_type_name_override="DecrementButton",
)
counter_label = ui.Label(
str(counter_int_model.as_int),
alignment=ui.Alignment.CENTER,
name="value",
)
ui.Button(
text="Inc",
name="inc",
style_type_name_override="IncrementButton",
)
ui.Spacer()Now, let’s look at the component with the ui removed to focus the on_change callbacks. This is very similar to setting <button onClick={} /> callbacks.
In React, the counter values text/label would update automatically because we bind the content of a div to the value of the in JSX. In Omniverse, we need to register explicit add_value_changed(_update_label) which re-writes the text of the label.
It’s also worth noting that we chose to expose the int_model of this component by returning it from the function. In React, the state would either live in the parent and be passed in, or we would expose the change handlers through props. Exposing the state directly here may an anti-pattern, but it has been useful.
@dataclass
class CounterComponent:
int_model: ui.SimpleIntModel = field(default_factory=ui.SimpleIntModel)
def counter_component(title: str = "Counter"):
counter_int_model = ui.SimpleIntModel(0)
def _on_click_decrement():
counter_int_model.set_value(counter_int_model.as_int - 1)
def _on_click_increment():
counter_int_model.set_value(counter_int_model.as_int + 1)
def _update_label(model: ui.SimpleIntModel):
counter_label.text = str(model.as_int)
counter_int_model.add_value_changed_fn(_update_label)
counter_component = CounterComponent(int_model=counter_int_model)
return counter_componentDerived or Computed Properties
To demonstrate this pattern, we create two counter instances and derive or compute the sum of those values and store it in additional int model.
Whenever the derived changes we then change the associated label.
self._counter_1 = counter_component("Counter 1")
self._counter_2 = counter_component("Counter 2")
# Create computed value from the sum of the two counters
self._computed_int_model = ui.SimpleIntModel(0)
def update_computed_value(_: ui.SimpleIntModel):
total = self._counter_1.int_model.as_int + self._counter_2.int_model.as_int
if self._coupled_counter:
total += self._coupled_counter.int_model.as_int
self._computed_int_model.set_value(total)
self._counter_1.int_model.add_value_changed_fn(update_computed_value)
self._counter_2.int_model.add_value_changed_fn(update_computed_value)
self._computed_label = ui.Label(
f"Total: {self._computed_int_model.as_int}",
alignment=ui.Alignment.CENTER,
name="large",
)
def update_computed_label(model: ui.SimpleIntModel):
self._computed_label.text = f"Total: {model.as_int}"
self._computed_int_model.add_value_changed_fn(update_computed_label)Conditional Rendering
To demonstrate this pattern, we render a description based on the computed value above. If it is under a threshold, we display “Small”, if over a threshold we display “Large”, and otherwise do not display anything.
In Omniverse, conditional rendering is achieved by replacing the contents of a Frame. In React you can return null, in Omniverse you return nothing, or a ui.Spacer()
def create_value_description_frame(
int_model: ui.SimpleIntModel,
low_threshold=-5,
high_threshold=5,
):
description_frame = ui.Frame(height=40)
# Overwrite frame contents whenever the model changes
# Create Frame first, use reference inside the callback
# https://docs.omniverse.nvidia.com/kit/docs/omni.kit.documentation.ui.style/latest/containers.html#frame
def int_changed(model: ui.SimpleIntModel):
with description_frame:
if model.as_int <= low_threshold:
ui.Label(
"Low",
alignment=ui.Alignment.CENTER,
style={"font_size": 42},
)
elif model.as_int >= high_threshold:
ui.Label(
"High",
alignment=ui.Alignment.CENTER,
style={"font_size": 42},
)
else:
ui.Spacer()
int_model.add_value_changed_fn(lambda model: int_changed(model))Aside about poor Omniverse documentation quality from this Frame example:
- Example of what I consider poor naming:
_recreate_ui = ui.FrameThey assign a Frame, which is a passive element, to a name which sounds like a function which would recreate the UI. Misleading. - They use an odd pattern exposing code to function via default values
def changed(model, recreate_ui=self._recreate_ui):This is unnecessary because the frame is already available to the closure. I think it can only confuse the reader. They may think this is necessary, but no, it is added complexity. - Generically named
changedcallback. It is the Int model that is changing and happens re-draw the Frame, but someone who is unfamiliar, might think it is the callback for Frame that is changing which would be circular. - Example of the broken markdown images
You will see the codeinstead of the image. The means you have to imagine what effect the code would have instead of being able to see it.
Styling and Themes
As shown here, I used a themes folder with a colors.py file and reference those colors in the default.py styles.
There is not as much novel information in this section. Styling in Omniverse is mostly about learning the Omniverse selector syntax, style syntax, and limitations of support.
Selectors
Remember from the mapping table above. style_type_name_override are like #ids, names are like .classes , and states are like CSS :link pseudo-classes. Omniverse also supports 6 states.
Generic: TypeSelector::Name Selector:State Selector
Example: Button::Primary:hoveredStyles
Mostly a change of hyphens - for underscores _
CSS: background-color
Omniverse: background_colorColors are created via the ColorShade constructor.
CSS color has advanced in the recent years with trends to shift from HEX to RGBA to HSLA and now OKLCH. Omniverse seems to only support HEX and RGBA. Let’s hope for support of OKLCH will be added in the future! 🤞
Oddity about styling Buttons
On the web when you want to change the background color and font color of a <button> you can set both attributes in the same declaration like:
#my-button {
background-color: hsla(0deg 50% 50% / 0.5);
color: hsla(0deg 0% 0% / 1.0);
}In Omniverse, Button contain a Label so we must set both to have the same effect.
"MyButton": {
"background_color": color_red,
},
"MyButton.Label": {
"color": color_gray_dark,
},Refer to the Omniverse documentation for more details:
Dialogs / Windows
Similar to the Component pattern, let’s analyze the code with and without the UI elements to help focus.
On the first call, the window will not exist, and we create it. Then we populate the menu bar and the frame with a counter instance and similar elements as seen above in our Component.
def create_modal(modal_window: ui.Window | None, int_model: ui.SimpleIntModel):
if not modal_window:
modal_window = ui.Window(
"Example Modal",
width=280,
height=430,
flags=ui.WINDOW_FLAGS_MENU_BAR,
visible=False,
)
modal_window.menu_bar.style = menu_bar_style
with modal_window.menu_bar:
with ui.Menu("File"):
ui.MenuItem("Save")
ui.MenuItem("Export")
ui.Separator()
with ui.Menu("More Cameras"):
ui.MenuItem("This Menu is Pushed")
ui.MenuItem("and Aligned with a widget")
with ui.Menu("Window"):
ui.MenuItem(
"Hide",
# hide_on_click=True,
triggered_fn=hide_window,
)
with modal_window.frame:
with ui.VStack(spacing=20, height=0, style=modal_style):
int_label = ui.Label(
"",
name="large",
)
coupled_counter = counter_component("Coupled Counter")
counter_component("Modal Counter")
ui.Button(
"Show/Hide Menu",
clicked_fn=lambda: show_hide_menu(modal_window.menu_bar),
height=40,
)
ui.Button(
"Add New Menu",
clicked_fn=lambda: add_menu(modal_window.menu_bar),
height=40,
)
return modal_window, coupled_counterSharing state between components
We share state between the Counter components on the root extension Window and the Counter components in the Secondary Window
- We integrate with parent state by passing it as an argument. In this case, Total Value as
int_modeland register additional on change functions. - We declare an additional “coupled” counter and return it, to make the value available to the parent Total Value computation
- The independent counter is simply creating a new instance of the counter as shown above.
- The modal window is returned so that parent can control window visibility.
self._modal, self._coupled_counter = create_modal(self._modal, self._computed_int_model)def create_modal(modal_window: ui.Window | None, int_model: ui.SimpleIntModel):
coupled_counter = None
if not modal_window:
with modal_window.frame:
def on_int_changed(model: ui.SimpleIntModel):
int_label.text = f"Total Value: {model.as_int}"
int_model.add_value_changed_fn(on_int_changed)
on_int_changed(int_model)
coupled_counter = counter_component("Coupled Counter")
coupled_counter.int_model.add_value_changed_fn(on_int_changed)
counter_component("Modal Counter")
def show_hide_menu(menubar: ui.MenuBar):
menubar.visible = not menubar.visible
def add_menu(menubar: ui.MenuBar):
with menubar:
with ui.Menu("New Menu"):
ui.MenuItem("I don't do anything")
return modal_window, coupled_counterConclusion
I hope you enjoyed learning about Omniverse application development.
If you have ideas to better implement these patterns or think additional patterns are significant enough to be worth demonstrating, let me know in the comments!
What types of UI do you plan building in Omniverse?
