Transcript
[ Music ]
>> Good afternoon.
Welcome to our session on
Working with USD.
It's great to be here.
My name is Denis Kovacs and I'm
going to be joined on stage by
Chloe Moore.
Last year, we introduced AR
Quick Look on iOS.
It's the simplest way to explore
3D content in AR on your
devices.
For instance, you can view the
model of a product you'd like to
buy.
You can place it in your own
space and get a real sense for
its true size.
As you move your device around,
the model appears firmly
grounded in your environment.
And you can also get a good
sense of how the real object
would look in your home because
the model is rendered using real
lighting conditions of your
space.
Shiny materials like the
polished chrome on this lamp
even reflect your environment.
And AR is simply fun.
It's a really intuitive way to
explore and interact with your
beautiful 3D content.
Underneath all these AR
experiences is a file format we
introduced last year to our
platforms, USDZ.
USDZ support is deeply
integrated into our apps and
frameworks, so you can share AR
experiences with your friends
over Messages and Mail, you can
experience interactive news with
content coming to life in AR, or
you can find new experiences on
the web, and take them directly
into AR through Safari.
And you can build your own
beautiful apps that take
advantage of USDZ.
The technology behind USDZ is
USD and it is the main topic of
this session.
So here's our agenda for today.
First, we will look at what USD
is.
We'll discuss some workflows
related to creating and
converting to USDZ.
The rest of the talk, we will
spend building an intuition for
USD.
Starting with some essentials
common to most 3D file formats
and we will end with some
features that are unique to USD.
So, what is USD?
USD stands for Universal Scene
Description.
It's a 3D file format and a
supporting C++ library with
Python bindings developed by
Pixar.
The library can read and write
USD files.
It contains a powerful
composition engine and much,
much more.
The focus of this library is on
speed, scalability and
collaboration.
There are three file extensions
associated with USD, a plain
text version, USDA, that is
designed to be as easy to read
and understand as possible.
A binary version, USDC, that is
as efficient as possible to read
and write.
And the extension USD, that can
be either plain text or binary.
You can go back and forth
between these file types.
And then there is USDZ.
Let's take a closer look at
that.
So what is USDZ?
USDZ is the distribution format
for USD.
It contains all the data related
to a 3D scene packaged up in a
single compact file.
It's optimized for sharing and
it's the basis of AR Quick Look.
USDZ is supported on iOS, macOS
and tvOS.
If you look at the anatomy of a
USDZ file we'll see that it's an
uncompressed zip archive where
all the contained files are
aligned to 64-byte boundaries
for most efficient memory
mapping.
There's two types of files
contained in a USDZ archive.
The scene description files in
USDA, USD, or USDC or even USDZ
format so you can have nested
archives, and a set of textures
currently in PNG or JPEG format.
Let's take a rough comparison of
current file formats out there.
The basic-- The most basic
format is obj, which essentially
contains a single 3D model.
It has a limited material
support and no animations.
Then there's a large group of
different more modern file
formats.
They usually can contain
multiple models that can be laid
out in a scene graph, they can
support different sets of
material descriptions, and also
animation.
USD supports all of this, but
additionally it was designed to
be scalable, Pixar invented the
format for its large movie
scenes.
And it also supports
collaboration, which allows many
artists to work together on a
single scene without getting in
each other's way.
USDZ, as the archive package
inherits most of these features
as well.
Next, let's take a look at
workflows.
We can group these in two
categories, converting existing
assets and creating new ones.
Let's start with converting
existing assets.
So let's imagine you have your
assets on this in some of these
other formats, maybe FBX or
gltf, and you want to convert
them to use USDZ to explore--
deploy them to your app or AR
Quick Look.
Today, we introduced a new
command line tool called
usdzconvert.
It supersedes last year's Xcode
converter and it allows you to
convert many more file formats
such as FBX and gltf.
It also performs asset
validation.
So you can be sure that there
are no issues with the generated
USDZ files.
And it is Python based so it's
platform independent.
So it's easy for you to
integrate this converter into
your existing content pipelines
that might be on other platforms
in macOS.
Here are couple of sample
sessions.
Let's say you have a gltf file,
you can pass the file to
usdzconvert and it converts it
into USDZ.
You can also see that it
performs asset validation
afterwards.
If you want to know all the
command line options, you can
use this -h.
And you can also provide rich
material descriptions directly
from the command line.
This is especially handy for
file formats like obj that do
not have a rich material
definition.
Usdzconvert is a part of a
broader package that we are
providing to developers.
It also includes the precompiled
binaries of the USD library for
macOS and some other things as
well.
For example, the USD command
line tools.
Usdcat, for example, is great to
output a plain text
representation of your USDZ
file.
Usdtree is useful to see a
high-level structure of the
model hierarchy.
And usdchecker is USDZ's asset
validator.
We're also including a little
script called fixed capacity.
If you offered models with the
transparent materials for AR
Quick Look last year and those
materials look opaque in iOS 13,
use a script to fix that.
OK. For a little demo of all of
these, I'd like to invite Chloe
on stage.
[ Applause ]
>> Thank you, Denis.
As we have seen before, we
provide a precompiled USD Python
library that's available for
download here.
I have already downloaded and
unpacked it.
The USD folder contains the
precompiled USD Python library,
as well as a suite of a command
line tools such as usdtree and
usdcat provided by Pixar.
The usdzconvert folder contains
the usdzconvert tool and the fix
opacity tool.
We also provide a USD command
which sets up some environmental
variables for you.
When you double click it opens
up a terminal window.
Let's start with usdzconvert.
If you just run usdzconvert by
itself, it outputs our argument.
Now suppose you have some gltf
file that you want to convert.
You can simply run usdzconvert
with your file name.
Now, let's take a look at a
generated asset.
To generate to the file contains
all materials and animation.
As we can see here, usdzconvert
also performs additional asset
validation.
Next, let's say you have some
obj file without material, like
this tetrahedron model.
You can simply add the material
with additional arguments, for
instance constant color.
Here you can see material is
applied.
To see the high-level structure
of your model, you can use
usdtree.
Here, see the material and the
mesh.
To see the full plain text
representation of your model,
you can use usdcat.
Here's another common use case.
You have a obj file here
together with a rich set of
material textures.
Assigning textures to a material
is as simple as assigning
constants.
A USDZ file is an uncompressed
zip archive.
So a quick way to examine its
content is to use a zip info.
Here we can see the USDZ archive
contains a USDZ file and all
above textures.
Here you can see all material
textures are applied to the
model.
So to summarize, converting your
existing asset to USDZ is easy
with usdzconvert.
And there is a set of additional
tools that would help you to
examine your generated assets.
Back to you, Denis.
[ Applause ]
Thank you, Chloe.
So that covers the conversion
stage.
Now let's take a look at how to
create USDZ files from scratch.
You can of course start creating
your assets with your favorite
authoring tool.
And it will likely be able to
export to a format like FBX
already that you can then
convert to USDZ with the
usdzconvert tool that we just
saw.
But there's a growing number of
content creation applications
that support direct USD export.
That allows you to integrate
your content creation step more
easily into a USD pipeline.
And the conversion step with all
these potential translation
complications turns into a much
simpler archiving step.
Here are two examples of content
creation tools that support USD
export.
This one is Substance Painter by
Adobe, a great tool for creating
photorealistic materials for
your 3D objects.
It supports USD and USDZ export.
And here is Autodesk Maya, one
of the leading 3D modeling and
animation tools.
It supports USD and USDZ export
through a plugin that's written
and maintained by Pixar.
So as we've seen, a lot of these
applications even support direct
USDZ export further simplifying
your USDZ creation step.
And because of USD's Python
bindings you can also create
your own custom Python pipelines
that are tailored to your
particular content.
Next, we're making exporting
USDZ from SceneKit super easy.
All you have to do is to create
or load your SCNScene just as
before.
And when you're ready to export
to USDZ, all you need is one API
call, write to, and make sure
that the file extension is usdz.
In fact, we've integrated USDZ
export into Xcode's SceneKit
editor as well.
So, you can also export directly
through a command-- through a
user interface as well.
OK. So that covers the
workflows.
Now, we want to take a look at
USD's features.
Now, as we've mentioned in the
introduction, USD has a very
large feature set.
And it is impossible to even
mention all of them in this
talk.
But there are a few key concepts
that are easy to grasp.
So let's focus on those first.
We will talk about four
concepts, the File Structure,
the Scene Graph, Mesh Data, and
Materials.
A key differentiator of USD is
that its plain text form is
designed to be readable and
understandable.
So the format becomes less
mysterious and more transparent.
And when things go wrong, you
can more quickly pinpoint where
and why.
So let's focus on the file
structure first.
So this is an excerpt of a USDA
file.
The first thing you will notice
is that it contains these nested
containers.
In USD they're called prims.
These prims contain properties
that store the actual data.
And there's a set of metadata
attached to file level, prim
level, or property level.
Because of the nested structure,
every object can be accessed
through an object path.
In this case, the prim cube is
nested under simple mesh, so its
path becomes /simpleMesh/cube.
This also works with properties,
with the dot notation.
Now that we have seen the basic
file structure, let's take a
look at how it is used to store
scene data.
A scene graph defines the object
hierarchy.
Transforms on a parent also
affect its children.
In this case, we have a cube and
a sphere parented to the cube.
If you move the cube, the sphere
comes along.
If you move the sphere, the cube
stays put.
A scene graph is easy to store
with a nested prim structure of
USD.
You can see the ultra prim of
the parents with the two child
prims, cube and sphere.
You can also see that there are
other prims such as materials or
maybe animations that don't
belong to the scene graph.
So that covers scene graphs.
Now let's take a look at mesh
data.
Mesh data can be grouped roughly
in two categories.
Mesh attributes such as
positions, normals, and texture
coordinates.
And mesh connectivity, such as
the numbers of vertices per face
or which vertex indices belong
to a face.
Let's look at that at the simple
example of a tetrahedron.
We start with a set of points,
four points for the tetrahedron.
And each face contains three
vertices, so there are four
triangles.
And for each face, we will also
store the vertex indices.
Next, we'll store normals.
And in this case we choose to
store the normals per face.
So there are four normals.
And the USD language for
per-face attributes is uniform.
So the metadata is set to
uniform.
That's also store texture
coordinates.
These are going to be sort as
indexed, mesh attributes, and
the indices are per-face,
per-vertex, and in USDs language
studies face bearing.
Lastly, we'll make sure that
this tetrahedron is rendered as
a polygon and not as a
subdivision surface.
We'll come back to that later.
So now we've seen the scene
graph, and the mesh data, let's
take a look at how materials are
defined.
USD's material definition is
very rich.
It is meant for movie quality
output.
For realistic real-time
rendering, there's a smaller
subset called UsdPreviewSurface.
It's a physically-based material
description and it supports two
workflows, metallic-roughness
and specular-roughness.
We will focus on
metallic-roughness here.
So in this example, the same
gramophone that we saw in
Chloe's demo, we'll first assign
a constant, white color.
Next, we will set normal and
occlusion maps to bring out the
fine details.
Next, let's crank up the
metallic level so it becomes
really shiny.
And we will also add a roughness
map to dull out parts of the
object.
Lastly, we will also set diffuse
color and metallic textures.
So with these five textures, we
get the final beautiful
photorealistic look of this
mesh.
So let's see how this is
described in USD.
USD uses a shader node graph
that has separate shader nodes
that are connected to each
other.
UsdPreviewSurface has a very
simple shader node graph
structure, with only four node
types.
The main shader node that
defines all the PBR attributes,
we've seen five of them before.
A texture sampler that tells you
which texture to use, a mesh
attribute reader for example for
the texture coordinates, and a
UV transformed node that allows
you to scale or rotate your
texture coordinates.
Let's take a look at the
simplest material example with a
couple of constant-- constants
for the mesh attributes.
We can see here that the main
shader node's surface output is
wired to the material's surface
output.
Here is what it looks like in
USD.
You have the outer nested
material prim with main shader
node prim inside, and the output
of the shader node is wired to
the output of the material using
these object paths that we've
seen before.
Next, let's take a look at a
slightly more complicated
example of a texture assigned to
the diffuse color channel.
We will need a mesh attribute
reader for the texture
coordinates, that's the red
shader node, a texture sampler
for the texture that we want to
apply, it's the orange one, and
the main PBR shader node with
the diffuse color attributes.
This is what it looks like in
USD.
You have the outer material prim
again and then the three shader
nodes.
And the outputs are wired to
some of the inputs of the next
shader node.
OK. So that covers essentials in
USD.
All of these concepts that we
talked about are in a samples
folder where we provided little
sample scripts to create scene
graphs, mesh data, material and
mesh groups, so that you can
assign multiple materials to
single mesh, and even animations
for transform animations and for
skinning and skeletal
animations.
Now that we covered the basic
structure of USD, let's take a
look at the main advantages that
set it apart from other formats.
Scalability to complex scenes
and life composition and
collaboration.
For scalability, we will look at
something called subdivision
surfaces.
And for collaboration, we will
look at the composition engine.
So let's start with subdivision
surfaces.
Subdivision surfaces are an
inefficient representation of
curved surfaces.
In contrast, polygonal surface
descriptions are an
approximation of the real curved
surface.
And that approximation bakes in
some assumptions on what a good
approximation is, for example,
how closely you're going to be
to the actual object.
In contrast, subdivision
surfaces described the true
surface.
So you can do dynamic
tessellation, for example based
on the distance to the camera to
find a good enough approximation
of the true surface with small
polygons.
Subdivision surfaces are also
great for animated content.
And we have been using
subdivision surfaces in the past
couple years, for example, for
an emoji or memoji.
On the left-hand side, you can
see what the subdivision control
mesh would look like if you
rendered it as a polygonal
surface.
You can see that it's fairly
course, which is great in terms
of the memory footprint.
On the right-hand side, you can
see the final subdivided
surface, and you can see the
beautiful details in there.
The industry standard for
subdivision surfaces is
OpenSubdiv and it is also
developed by Pixar.
We have been working closely
with Pixar to provide metal
shaders that work optimally on
our platforms.
So the GPU evaluation is as fast
as possible.
And OpenSubdiv is the basis for
subdivision surfaces in
Scenekit.
And of course USD has great
support for subdivision surfaces
and it works beautifully with
OpenSubdiv.
In fact, USD has a unified
description for both polygonal
mesh data and subdivision
surface data.
It also has subdivision
specific-- support for
subdivision-specific properties,
such as creases or corners.
So, that concludes subdivision
surfaces, a great and efficient
way to store high quality
surfaces.
Now let's take a look at the
composition engine.
The composition engine is a
powerful authoring tool that
enables efficient collaboration
between artists.
And we'll take a look at an
example of references.
Here's again the scene graph
with two child objects parented
under a parent node.
And this is an efficient
representation, as long as those
two objects are different.
But what if you have multiple
copies of the same object in
your scene graph?
Then this nested scene graph
structure becomes inefficient
because it stores duplicate
data.
Most filed formats factor out
the duplicate data and store
references instead, and use the
support set as well.
But it goes further.
It also creates virtual objects
inside this nested scene graph
prim structure, and each of
these virtual objects has its
own unique object path, so that
you can go in and override some
of its materials-- some of its
properties.
For example, in this case, I
overrode the second spheres
material color to yellow without
affecting the first spheres
material.
You can also remove whole
subtrees from your scene graph.
And USD stores all of these
edits as efficient overrides to
the original data.
Let's take a look at what artist
workflows this enables.
This is the finder in macOS
Catalina.
And you can see here macOS Quick
Look rendering a scene, USD
file.
Let's say now, I'm a layout
artist that places all these
objects in the scene.
And at the same time, there's
another artist working on
refining some of these objects,
for example, this chair.
And this artist replaced this
chair object with this one.
Now, if you go back to the
layout file, you can see that
these objects have been
automatically replaced, the file
itself did not change, it pulled
in the references.
And in fact, I can do my own
edits.
For example, changing the color
without affecting the other
artists files.
So this allows us to decouple
our workflows from each other.
So this was a quick sneak peek
on all the new artist workflows
that USD can enable.
We have also included two more
examples of these more advanced
USD features in the samples
folder.
Subdivisions surfaces with
creases and references with
overrides.
So, in summary, we have looked
at universal scene description.
It is a file format and a
powerful library for authoring
all the way to deploying 3D
content.
We have looked at workflows to
create and convert assets to
USDZ.
We introduced a new tool called
usdxconvert that converts obj,
gltf, fbx and other file formats
to USDZ.
And you can download these
Python USDX tools, which include
usdzconver, a precompiled USD
library and the sample scripts
from our website.
For more information, please
check out the session page.
And we will have a lab on USD
and USDZ tomorrow at 3 p.m. And
also check out the advances in
AR Quick Look on Friday at 9:00
a.m.
Thank you.
[ Applause ]