WWDC2001 Session 404

Transcript

Kind: captions
Language: en
this is a session 404 OpenGL high
performance 2d and the first of five
specific OpenGL sessions running today
and tomorrow here at WWDC the way we
structured structure the demos is to
have each session addressing specific
OpenGL development aspects from 2d 3d
rendering and optimization so you'll be
able to pick the information you need
from the experts here presenting to you
for your application development what we
are not going to talk about is Hardware
specific acceleration features we're
gonna be general and so to prove to you
that our OpenGL implementation is
consistent across our graph platform
regardless of your the hardware
acceleration we have some really cool
stuff we're gonna see specially on this
demo here that OpenGL is incredibly
stable and scalable and versatile which
to us it's a great feature for again
across our product line and the I must
say that the OpenGL team has done a
great job including the DTS folks and
bringing about an employment a shin that
runs both Mac OS 9 and my quest and
consistently as off today so your
application obviously can run on both
our asses without glitches but this is
really only the beginning what we really
want to do is bring OpenGL to a new
height and what I mean by that is to
finally really prove to the industry
that an open source open standard
vendor-neutral 3d API is the best way to
go so we're going to see some really
cool stuff and what we really intend
here to do is to give the best of our
insight to you and to inspire you to
come up and design the best 2
and 3d applications based on OpenGL and
you know give us your feedback well
we're going to be here for an hour
providing you with a lecture we're
really looking forward to hearing back
from you
hopefully in them in a very candid
manner how do how do you we can help you
bring your applications to to the
platform and to leverage your investment
to really easier development and and
performance of your application using
OpenGL so to present section 404 OpenGL
high performance 2d I'm pleased to
welcome DTS engineer Jeff Stowell thanks
Sergio today I'm going to talk about how
to use the OpenGL pipeline to get real
impressive 2d performance out of the
platform this includes both 2d or
traditional viewing pictures those kind
of things and two and a half gee maybe
some work with sprites work with the
movies and those kind of things and
we'll see how you can really use the
OpenGL pipeline to display those 2d kind
of raster images so first we'll talk
about why are we talking about OpenGL a
3d API for 2d work then we'll move on to
implementing 2d with OpenGL things I'll
talk about first is how to display
images we'll talk about how to load
images into textures and use textures to
image from you can use this extrapolate
this to using anything you generate from
your program we've seen demos before in
the past with using PDFs as a source of
a texture and texturing that onto an
object you need to do the same
techniques there we'll move on and talk
about sprites and side-scrolling now use
side-scrolling kind of loosely with
OpenGL obviously there's no limitation
on whether you're going to use it for
vertical and horizontal scrolling know
where you're going to create the next
great game the next to the next route
creation of defender and then we'll talk
about liubao QuickTime integration and
beyond using just the QuickTime API for
loading textures how we can use the
QuickTime API and movie to integrate
with OpenGL to display them first why
why are we talking about OpenGL it
really is the fastest pipeline on our
system we've done a tremendous amount of
work
the engineering team has done some great
work to get great throughput to the
video card I would be hard-pressed and I
think that most people in this room
would be very hard-pressed to try and
duplicate that effort for your own
application unless you're an expert at
HP graphics you have access to all the
resources of the engineering team has
and understand the PowerPC new processor
inside and outside it'd be very
difficult to get the kind of performance
like six hundred and fifty megabytes per
second through that across that AGP bus
to the graphics card so I recommend as a
way to get that high performance
graphics to use a OpenGL it's also a
flexible API the API is not specifically
designed to work with 3d you can use it
for two and two and a half D by just
going to an orthographic context and
kind of forgetting about that Z value
and using that to display your 2d images
also hardware acceleration we have folks
from ATI and NVIDIA wandering around and
that they've basically bet their
companies on the fact that they can
provide world-class hardware
acceleration to the graphics pipeline
and I think they've done a hell of a job
so use that leverage that work that
they've done and the board today Apple
engineering is done for your
applications and use the OpenGL pipeline
for your apps and also there's lots of
sample code out there if you get stuck
there's tons of resources on the web
OpenGL is a it's a very stable API with
lots of code out there to teach you what
to do and help you when you get stuck
but I would be remiss if I didn't talk
about why not OpenGL what things about
OpenGL may turn you away what where
where may you not go down the right path
my pipeline here so to speak by thinking
about it using OpenGL for your
application first it works very well in
16 and 32-bit modes but it does have
some problems if you're trying to get
high performance in an 8-bit mode the
8-bit mode is gonna be you're gonna fall
into the software case and in a lot of
special cases where you have special law
you're doing a pallet animation for
example here you're doing pal animation
OpenGL doesn't have a real equivalent of
doing 8-bit pallet animation so I
suspect the your application could be
specially coated to handle that in a
case of Oh better than OpenGL so that
would be a case that you may want to not
think about using OpenGL and use a
special case of glitter of your own
also let's say you're doing fonts you do
you want to have very scalable fonts you
want to do specific kerning with fonts
or handle Unicode fonts OpenGL doesn't
have any built-in properties to do this
you could combine quartz and OpenGL to
do this but really if you're trying to
do like the next InDesign or page maker
or those those kind of applications if I
don't think about the using the courts
2d API because that really has a robust
feature set for handling images handling
fonts and those kind of things while
OpenGL really is where the
high-performance bit images can be
handled so what we're going to talk
about first went to hop in the image
display talk about texture loading and
talk about orthographic display and
scaling then we're talking about sprites
with alpha blending and alpha test it's
talked about using some matrix
manipulations to move them around the
screen then we'll work that right into
the side scroller showing it's showing a
large scaling background with sprites
flying around than that and lastly we'll
talk about QuickTime integration first
let's talk about image display one of
the key things here is how to load
textures how to get any text or any
image you want into your system we're
going to use a quicktime image
compression API it's a great API for
loading images you can bring up a nav
services dialog and select any any image
you want use QuickTime to load it you
didn't have to worry about where there
was a BMP a tiff I'll pick a JPEG
whatever it'll load it for you it'll put
it in a buffer you can use some of you
have looked at this may say well hold on
if I use new G world for that buffer I
have some padding problems I'll show you
how to get around that we'll show you
how to use any buffer that you can take
you're going to take a non padded buffer
and you can use that directly for
texturing to OpenGL with that you want
to use pack pixels the pack pixels
extension or the pack pixels that is in
OpenGL 1.2 drivers will allow you to use
a texture format directly from the image
to an off-screen gee world to a texture
without having to do any pixel Swizzle
in whatsoever so you don't have to touch
those pixels unless you want you to do
some special effects the 3 meses methods
we'll talk about is first a simple way
OpenGL must have a power of two textures
currently so what we're gonna do the
first way we'll talk about scaling with
Quicktime
to a power of two texture onload and
then fixing it as we display it
next we'll talk about segmenting to
power of two textures a little more
complicated when you take a large image
that is of abstract size any size you
want and you segment it down so you get
the pixel perfect representation of that
image in multiple textures and lastly
we'll talk about we're not gonna really
talk about I'll talk about it here we'll
talk about texture rectangle texture
rectangles and extension coming down the
road that will allow you to take any
shape or size now any shape but any
rectangular texture that is not
necessarily a power of two and texture
with that all the vendors are working
hard to support that we're working hard
to support that so look for that in the
future when you're talking about image
loading an image handling with OpenGL
it'll allow you to handle textures of
arbitrary size directly into the OpenGL
hardware accelerator I think I better
take a drink of water for this one the
QuickTime image compression API this is
let me actually back up a little bit
with with the code here the code I'm
putting up here will all be available on
sample code there's no code you'll see
on the screen that you'll not be able to
get a hold of so don't try and take
specific notes exactly how to do it on
the on this all most of this stuff will
be up before the end of the week some of
its up already and the rest of the stuff
will probably be up next week so you'll
definitely get up the code and look at
it yourself and look through it in
decipher and figure out what we're doing
with that so if you listen listen hard I
think I can go through and explain
exactly what's happening here first
thing to do is get graphics importer for
a file you get you have an FS spec from
your your nav services dialog or
something that you create yourself and
you get graphics importer for football
perfect for file it returns the GI comp
variable there what thing I do is get
the natural bounds of that basically
tells me how big the image is so I know
what I'm working with
second thing I do after that is decide
on what the stride of that image is too
crude to create a pointer for to hold
that image the reason I do this is
that's a step you may not normally see
that step creating the straw I tells you
exactly bit for a bit what's the width
of that image without padding and then
the new pointer I'm going to create a
buffer that's exactly the size of that
image without any padding without any
pad it's through a four byte or anything
that a gee world may normally do I then
call QT new Geo world from pointer it's
not a function
very often but it works very well
because it will use that pointer I've
created for that new G world and you'll
be able your supplying the memory you're
supplying the padding it'll say I know
the information there I have the stride
at the end so I'm gonna use that packed
image buffer for the G world and I'm not
going to allocate any memory myself that
is key to using PAC pixels and using
direct texture into OpenGL because that
allows you not to have to skip rows not
to have to do strange things with the
image before using it with OpenGL and
you see there's two different versions
here the reason there's two different
versions and you probably could could
make it one with assigning that constant
to a variable but in this case there's I
wanted to show you that there is 2k 32
AAA RGB pixel format and in K 16 16 B
e55 pixel format those are open I'm
sorry there's a QuickTime constants then
indicate the QuickTime what kind of
pixel format you're gonna have in that G
world there so now what are we talking
about remember I said the first
technique uses a scaled image into the
texture for Oprah texturing make it easy
and we're gonna scale it to whatever
power of two is closest in this case I'm
gonna use QuickTime to do that QuickTime
this scale image matrix works really
well the reason why it works well the
first time it has to set up the MIT has
to set up the scaling but after that for
example if you're using the same kind of
scaling with a movie it's set up a
lookup table and it does a lookup type
of scaling so it has accurate and quick
scaling and really I haven't seen any
performance penalty from using QuickTime
to scale images on load so you set the
identity matrix and then you're set your
scale matrix it's pretty simple
basically you're taking the difference
between the check texture whit your
desire and the image width you have and
you're scaling it down it's gonna fit
into that matrix remember to note that
you are at this point losing information
you've actually throwing away pixels so
then if that's important to you if an
important to get every pixel you may not
want to use this exact format you want
me want to use the next one and then you
set graphics graphics import set matrix
remember that GI comp we got before
we're gonna set the matrix there and
that'll set the scaling of anything we
draw into that so lastly we're gonna do
a graphics import set G world graphics
import set the quality for lossless
we're gonna get
gee world pixmap of course remember to
lock that and check the value of your
lock on the return because you don't
want to draw into a non lockpicks map
it's not good enough just to call that
lock pixels and then use graphics import
draw unlock pixels and close your
component now in that pics back that
you've created is the image off the disk
no problems so one of some other things
we want to talk about let's talk about
packed pixels how do we get that imaged
into OpenGL first real critical if
you're using a GL make sure you use a GL
no recovery a GL no recovery says I
don't want to create a software backup
renderer to backup if the system runs
out of resources for this what that does
in this case it says I don't want I want
to respect the hardware's pixel formats
only so if you're sending to a hardware
all you're going to do is create the
pixel format OpenGL is only going to use
a pixel format for that one specific
piece of hardware it's not gonna try and
use a software backup which happens to
not support pack pixels so may convert
everything to 32-bit so if you have a
16-bit format here you want to make sure
you do that to keep it 16-bit you need
OpenGL version 1 2 or the GL Apple pack
pixel extension 4 to have this to
support pack pixels and if you notice
one thing you'll notice it actually is
the extension string is packed pixel not
packed pixels which I think it's listed
as and this is just a little caveat so
if you search for pack pixel no matter
which way it which way we go with that
if we if it changes in the future you'll
always make sure you get it physical if
you search for packed pixels you may not
get it in the future or another way
around any case that this is the key
here is yield text in each 2d with a B
GRA extension in the unsigned in 8 8 8 8
reversed and then B GRA unsigned short 1
5 5 reverse the way you read that is the
reverse of the pixel format applies to
the B GRA so if a RGB 155 or a RGB 888
which we recognize as a standard Mac
pixel format
thus you can use these to directly
texture out of your image buffer so
scaling to power of 2 what do we do with
this if you notice I have the example
images here the first one is like a 256
by 256 compressed down to the exactly
what the text is going to look like what
we're gonna do with this is we're going
to restore the aspect ratio by storing
the image
took ratio that we read in and the
setting the polygons aspect ratio to the
same image texturing to fill the polygon
which would just stretch it back out
again we're losing information here but
the image appears correct on our screen
so scale the power of to this piece of
code here especially pretty simple and
maybe easier to look through it on your
own but really what I'm doing here is
fairly simple what I do is if the image
size is I step through I take my Marge's
largest power of true that I want I step
through and remove that from I'm sorry
that's incorrect
basically what I'm gonna do is I'm gonna
take I'm gonna use the fact that I can
shift things down a power of two at a
time by using the shift registers and
I'll shift it down until there's nothing
left when there's nothing left I'll put
a one in the first bit and shift it back
up and what that does it tells me the
largest power of two image that's
nearest and then at the end I basically
say if it's closer to one than the other
I checked above and below so basically I
got to the point where where I have a
power of two that's that it's gonna be
smaller and are larger than the image so
if I add a 249 pixel image I shift it
all the way down and I shift it back up
I got 256 because I have that one bit in
one of the one of the I have a one in
one bit of that of the image size and
then I compare that to that shifted to
128 I see which is closer so I'm getting
the nearest one so with this little
piece of code does it defines the
nearest power of two there's probably
different ways to slice it I thought
this was probably the easiest way to do
it and it's not real easy to explain but
I think if you step through it you can
see how it's working and that's all I
must bring up with the demo machine and
I'll show you this in action here what
we're gonna do is we're gonna choose a
file to open here and this is going to
open an image and it's just gonna
display it in OpenGL window if you
notice is stand a standard window
standard image remember this image does
not have the complete image quality that
we had originally if you read if you
look at the text which may be hard to
read up there but I'll tell you what it
says it's a 645
480 by 32-bit image that obviously isn't
a power of two we've put it into a 512
by 512 texture and then we've rescaled
it out so it looks like the exactly at
the right image if you want to see the
actual polygon I can zoom it out for you
that's actually a single polygon with a
single texture 512 by 512 stretched back
to the recurrent aspect ratio for the
image one of the nice things you see
here is high-performance 2d right that's
what we're talking about this is high
performance 2d using the OpenGL
transforms no problem rotating the image
and if you want to zoom in we can do
that too very easy we'll go to full
screen still get really nice transforms
real easy to manipulate images using
OpenGL using OpenGL as your 2d transform
but you say you know really I want all
that information that uh we can go back
to the presentation now I want all the
information that I had in the original
image I want to have all that I don't
want to lose information oh let me back
up if you notice feel carefully at the
screensaver naka was 10 OS 10 uses the
same technique it's it translates all
the images to 512 by 1024 textures and
uses that but you said man that
screensaver looks great great detail
you're losing detail that images is
77-68 tall it's square it's squished
down the 524 if it's 900 wide it's
stretched out to 1024 so it uses that to
get consistent images and have an easy
texture loading so that's an actual use
of this in a 2d environment and you see
then they can get the fades using the
OpenGL hardware acceleration instead of
having to do some kind of fade
themselves which is going to be a real
hard and real real taxing for the system
so the other option we have is
segmenting to power of 2 you don't want
to lose that and this is this is a an
interesting technique a little bit more
complicated but it uses the same basic
basis as the other ones in this case
what we have is you can see the image
then the green lines indicate the actual
polygons that produc comprise that image
I've segmented this image to about 4 by
4 as far as using it to display and so
I've kept all the pixels directly in one
texture and then I'm then I'm sliced out
of that texture for each image the way I
do that is when you call Jill text image
2d you provide it with a base address
that base address can be manipulated it
doesn't have to be the beginning of your
buffer so what you can do is provide it
with the beginning of every single
segment of that buffer very simply the
other thing you can provide is OpenGL
has a constant called unpacked Road
length and normally the width you pass
into GL text image to D is the width
that it uses obviously if you were
taking a piece out of this that weight
wouldn't make any sense and because you
would get to be all skewed it would try
and take the next pixel over and stick
it on the end of your image so you get
this kind of skewed image that came
across your screen well set unpack Road
lengths to actually the width of the
entire image vise so with that little
piece of texture we want so what it'll
do is for every row it'll skip that much
that's like it's image stride it'll skip
that much and come down to the next one
grab a piece of it skips the entire rest
of the row and grab the next piece so
it'll have that nice lined up pieces of
the polygons to win an Ashley tries to
draw it so we use the image width in the
unpack row length and well then the
texture width will be some smaller
subset of that ok Sigma into power of 2
what we're gonna do is this first first
black box statement up here basically
tells me how many textures I have in the
image dimension same technique we kind
of used before basically what I do is I
say okay I know that if I look at the
the bitwise representation of that
dimension I can just kind of count the
bits that are actually turned on so if I
look at the ones and zeros I can go how
many ones are there and that's the
number of textures I actually need to
use it's real simple works very well
when you realize that that that's a nice
little thing to have I then do a new
pointer clear for texture names and then
that's text your ex by text or Y so I'm
gonna create an array that I can store
my texture names in I'm gonna use the
pixel store I as we talked about the set
for the image width for the entire
texture within this case the entire
texture width and then I'm gonna do a
Gen textures to generate texture names
for each texture I need for that image
so in that case you saw it was a four by
four I'm going to generate 16 texture
names then when I draw the image I'll
use GL blind textures I'll bind each
texture in turn and I'll draw the image
that way so let's look at actually
getting that getting it into memory
another little black box function we
have here I'll show you in a few minutes
is get
to exercise it takes where you are
currently in the image and says what's
the next size what it does is I say I
have a max text as well as talking about
earlier like I confused I have a maxed
exercise and want to use 256 512 maybe I
don't want to go up to 2,000 for an
image mean I want to keep it down to 128
so I have 128 segments whatever you
prefer it steps through 128 removing
that from the length of the image and
then once it gets past that once there's
less than 128 left it takes the next
biggest power of 2 off the image and it
keeps doing that kind of down so it
gives you 128 64 okay there's no 32 so
I'll give you a 16 and then whatever
else you got left in this case we get
the next exercise in the loop we do go
for x and y for each of our textures we
off to our offset is offset is zero for
the y offset that means we're we're in
going in two columns we're going to
start the top of the image forever every
row we're also going to above this we're
gonna have the X offset is zero at the
very beginning and what we're gonna do
is we're gonna take the the pointer to
the buffer is the normal image buffer we
got before from quick-draw when we when
we use the the 2d get your new jeer
wheel from pointer we're gonna use the
offset Y by the texture width times the
image depth divided by eight in this
case it's shifted three so what
basically we're doing is saying that
this is going to offset into the image
for that corner of every texture so we
offset across four X offset down four Y
for every row we offset in based on what
the the width of each texture we need to
use is get the next exercise for the
height we've got it for the width so we
put in the previous exercise we call
bind texture we do that bind texture we
pick the next texture name in our list
of 16 and then we basically do the same
in the same GL text image 2d if you
notice I'm using the P buffer and I'll
go back a slide the P buffer is that P
buffer not the P image buffer the pH
buffer was the original pointer and the
P buffer is the pointer to whatever
specific texture and we've bound we've
told it the the unsigned row length
unpack row length is the width of the
whole image and so we use the GL text
image 2d at that point to grab the right
texture out of the image so we're kind
of doing the cookie cutter take a piece
take a piece take a piece move down take
some more pieces
we got we slice our image up that way
this is a little snippet of code from
the git texture numb from texture
dimension same kind of thing first thing
I'm going to do in this is I'm going to
remove what are my max size of textures
I have max text size I'm gonna divide by
max text and count how many max text or
images fit in there then I'm gonna take
whatever is left over the texture dim
reset there and I'm gonna is there
anything of 8,000 is there anything of
4,000 is there anything of 2,000 is
there anything of 1,000 I just go down
each one if you notice when you add all
that up it'll hang up back to back to
the complete size of the image so this
tells me how many textures across I have
or down and lastly here get the good
text and get next exercise this says hey
I got the size of my current one is it
one of the first max sized ones or am I
gonna step down into my list and
basically do the exact same thing
checking to see if I have an eight
thousand four thousand a two thousand
and 1000 like that so how do we display
these things first we're going to use an
orthographic display set up to the scale
of the image the scale of the screen so
the images you saw I scale your display
to be the pixel scale of the window so
when you're manipulating them if I
offset three pixels I'm often sending
three pixels in the window so this
actually is a good way to display it so
you're thinking like a quick like quick
drawl or normal what you used to
thinking as far as you invert the X and
you invert the y coordinate your X is
your origin is in the upper left corner
and at that point you're moving things
in a pixel centric manner I may ignore Z
I don't need Z so I don't need to
establish a depth buffer and I ignore
the Z for this one I use poly I'll use a
polygon to scale the image when you saw
me scaling and out that image before I
use the polygon to scale I use
manipulate the polygon positions I
didn't use any in texture coordinates I
didn't scale my my my window at all I
did was scale my polygon size my texture
stayed the same in OpenGL takes care of
the hardware accelerated transform to
get onto the polygon and you can use the
modelview matrix very simply in the in
this case to offset and rotate so as you
saw me dragged I used to a drag on the
mouse
I use a real real nice carbon events
told me when the mouse drags I figure
out what the star point in the end point
for the mouse drag is and I just drag
the image but that many pixels my offset
since
already in pixel space is exactly that
when I apply it into the modelview
matrix rotation I get the rotation in
degrees for the for the turn of the
mouse and I apply that again to the
rotation matrix of the around the z-axis
for the modelview matrix to manipulate
the rotation of it and now we're going
to go back to the demo and one thing
about the the multi-textured demo here
is that I can load the same image I did
before if you look notice this one it's
it styled in multiple textures let's
let's expand it out and I'll zoom in a
little bit to actually see what it looks
like what you've seen there is actual
pixel grid of the actual pixels on the
screen there are actual pixels of the
image and you see how the text is
actually lie right on the pixel grid so
you see that these come together right
on the pixel grid of the image and
OpenGL does a great job of stretching
that image I mean there's only a certain
amount of data there but it was a great
job of scaling the image and you still
got great performance on this 640 by 480
image but you say you know you know
maybe I want something bigger so I had
to dig around to find a bigger image but
I did find a bigger image and I haven't
tried this one for a while so I'll hope
that hopefully this one will be okay
this one takes a while to load and kind
of how big this image is I think it's
2800 by 2100 so it's a pretty big image
look at the polygons will zoom out a
little bit those are 5 the max texture
size 9 by 6 I think the max texture size
is 512 in this case so 2800 by 2100
pretty nice oh you want to rotate it we
can do the rotation too that's easy to
do you want to zoom in to see the actual
imprint nice detail there for the image
so you can see opengl works very well
for handling 2d images one thing it
might be interesting is see if I can
find one of these well we'll help
ourselves people have been working with
opengl sail but there's a little problem
with my technique and there actually is
they find the
the corner I'm gonna fail show it'll
show up right here probably Oh you kind
of can see it here not very well I
salute you better than I thought there
is a discontinuity along that line and
there's probably better examples of it
let me see you actually might find a
better one because it's kind of a good
thing the good thing to understand
probably on the edge here we'll be able
to see it better
back to the middle you say there's it I
can see it a bit down here
okay you ought to think oh there it is
actually you can see right there see
across there you can see right across
here there's a discontinuity
discontinuity is OpenGL uses filtering
OpenGL filters across the texture to
when it when it tries to man you mine
five minute five bytes by a texture by a
texture by texture I don't know every
best way but it it looks at each texture
separately remember you've moved this
image from image space that you had to
the textures and it looks at each
texture separately wants to filter it
when it when it tries to do the scaling
for form innovation and magnification
what that means is it looks at the next
pixel over and says what is the color of
the next pixel well in this case the it
says there's no next pixel at the edge
of the texture so there's a filtering
discontinuity between two edges so if
you really want the perfect
representation of this something I
didn't do in this demo what you can do
is take a texture border you can take a
border of one row of pixels from the
formula from your image so you make your
textures like one pixel smaller and take
a border around them so that
minification magnification filtering is
more accurate not something I did here
but you can do it if you if you want the
perfect example of this so again this is
a this is scaled up like two-and-a-half
times and you can see that it does a
pretty good job on this stuff so go back
to the slides
okay let's move on talking about sprites
sprites are actually pretty simple
sprites are just gonna be textures and
handle by OpenGL so we'll load the
sprite frames and Stormin textures well
what I'm going to do for sprites that
have multiple multiple images and most
people who want to use sprites want to
handle multiple images in one sprite
even if you want to animated cursor for
example how would you do that what you
can simply do is have one texture that
fits all of your sprites are some of
your sprites in there and you can set it
up so that your sprite frames are
offsetting you use the texture
coordinates to offset into the sprite
frame so you load the whole thing in one
texture and instead of like we did
before where you're loading a segment
that one image into a segmented texture
you're loading one texture and then
using it as a segment as it's in
segments by using texture coordinates
every time you go to a polygon for
example if you had a polygon if you had
ten sprites you would use a tenth of the
texture coordinates for that one sprite
and you can move maneuver across
depending on what sprite frame you're
talking about one thing you can also
most will want to do with sprites is
they want to use some kind of masking I
want to have just that cursor just my
animated cool animated cursor spinning
around or whatever they want to use that
some kind of masking so how do you do
that well you can easily do it by
passing in an alpha Channel or if you
read an image from QuickTime you may not
be sure of the alpha chain you may not
be sure that's the exact alpha channel
you want so you want to go in there
blast the Alpha bits of something you
know in this case I used a background
color of pure green and I looked for it
basically the green screen kind of thing
and I just remove the green screen by
setting alpha that's a zero I did it a
couple times and the first time I did it
I came out with what's on on the year on
your right hand side here with that
little green line around that it was
pretty cool effect but again texture
filtering that is it looks at the next
pixel and says huh I know this is green
there
you want me to filter from that green
onto your image and that's what you're
gonna give you so what you can do is the
other image what I did was I said I'll
put a kind of a neutral gray in there
and the neutral gray allowed me to
filter it to a reasonable color so
that's something just keep in mind when
you're doing it something the OpenGL
will want to filter that and it will use
both filtering on alpha and in filtering
in the image domain so you want to make
sure your filtering set up correctly
you're using an alpha mask so let's talk
about how to do alpha mask basically you
want to set the alpha alpha Channel
onload you want to look at the Alpha
Channel once you load it with QuickTime
and you want to blast the exact bits you
want into it in this case we're talking
about the top eight bits in 32-bit
format or the top 1 minute a 16-bit
format if you wanted to you can use
other formats by the way I'm not
limiting you to these they're just
something that I'm using for the talk
but you could use a four four four
format OpenGL would understand that
which would give me the four bits of
alpha in a 16-bit representation of
colors basically background color
pictures with pixels we're going to set
the Alpha to zero and we're gonna set
the pixel value of those of those pixels
to some neutral back background color
for all other pixels we're gonna set
that make sure the pixel is set to alpha
so in this case in the 16-bit case well
or that with 8,000 hex and in the 32-bit
case we'll set the two high bits to FF
to basically make sure they're all ones
in the two high bits you want to ensure
the alpha set because sometimes
depending on where the image comes from
and I've seen images that I thought out
of Photoshop that I had set the Alpha
correctly and it comes to a format and
when I look at it coming through into
the machine the Alpha is not set like I
would expect it to so just make sure you
you keep that in mind that you have a
known source of your alpha or your
setting into what you want to set them
to so how do we do this it's actually
pretty simple if it's 32-bit image what
we're gonna do is we're gonna get the
pixel from them from the image buffer we
saw before we're in a group we're gonna
step through for texture with and
texture I through the whole thing we're
all going to look at the the bits that
are our color masks in this case I'm
saying green FF and 32-bit format is our
color mask so I'm what I'm doing is I
want to ignore the Alpha in these bits I
don't want to compare to 0 0 0 0 FF I
don't want to compare to all 32 bits I
only want to compare to the 24 bits of
color so I'm not sure about that alpha
so I may have a difference in alpha and
they won't quite match my own mic my
source color key so I'm going to look at
the color bits so I'm gonna mash those
color bits out and if they equal that
pure green I'm gonna set the pixel to a
reasonable gray in this case with a 0 0
and the high bits for alpha so it'll be
blank and the filtering war will look
look good if not I'm gonna or that with
FF and the reason I want to or it with
that bath is to make sure that those two
alphabets are set at the top
or eight alphabets actually there's two
- and hex two digits are set for four
alpha and I'm going to loop through the
next pixel four sig the 16-bit case very
similar in this case I'm going to
compare it to the seven ffff to mask out
that one alphabet and make sure that
three zero which is the same five bits
of set for the green I set it to a
neutral gray or set the name to the top
alphabet so let's talk about alpha
blending there's another way you say now
you have the sprite mask you have your
cursor that you move around you've cut
out the background in we're talking
about alpha blending another thing you
may want to have some kind of
transparent text or some kind of feature
some kind of you know you have the
heads-up display in a game you want to
just use that with alpha mask real
simple to set with alpha mask in this
case what I'm going to do is I'm going
to use a luminance value of this texture
to set an alpha mask that's what this is
an example of here what I'm going to do
is I'm going to take the average which
is actually I'll correct myself before
anyone else does this is not actually
the luminance value this is this is my
poor man's luminance value if you want
really did you the y UV conversion and
just take the Y component you should do
a correct y UV conversion but in this
case I'm going to assume this is
grayscale all the bits of the same and
so that really the divided by three is
not necessary but it looked good for the
presentation so we take our poor man's a
poor man's luminance value we we divide
by three to get the average for the 255
we're going to shift it up into the high
bits and we're gonna or it into the
pixel so in this case what we're gonna
say is whatever the the kind of
grayscale value of that pixel is is
gonna be the Alpha so we'll get
transparency that varies with the
luminance this is good if you if you get
a some artist it has some kind of
background or something like that you
just want the luminance value we want to
set an alpha that equals it you can use
this kind of technique so how do you
draw with these things what are the key
OpenGL the unlock the OpenGL Kingdom
here we get the jail alpha func what I
do here this is the masking which talked
about that alpha we talked about the
mouth mask or the first thing we talked
about in this case we're gonna say
greater than 0.5 or less than 0.5 so if
it's greater than 0.5 we're gonna put
draw if it's less than 0.5 or not and so
in this case when we set everything to 0
we won't draw anything was set the one
we will draw and we said we our polygon
needs alpha test or if our our sprite
needs alpha test will turn off with
test on for blending for transparency we
use a blend func in this case we'll use
source alpha + 1 - source alpha so but
basically this say the contribution of
the front pixel is source alpha it's a
sources alpha value the contribution of
whatever is there already is 1 minus
that so if your alpha is 0.8 you can see
the source is 80% contributor the
background is about 20% contributor and
we enable blending if we have
transparency lastly depth test and then
when I do a depth test so if you have
multiple multiple pixels that are or
multiple sprites that are over top of
each other you can actually use Z even
in an orthographic context as a
delimiter of what stress were drawn in
front of what else otherwise we're going
to do doing the painters algorithm
what's ever drawn in the front last is
gonna be in the front so you may want to
use the blend function for that so let's
talk a little bit about how to do a
side-scroller the most people when we
when we think of side scrollers who
think of like you know 1400 by 480 kind
of backgrounds that if you play defender
on or run around and your your 2d
version of quake or whatever you're
doing and most people think of using a
2d raster engine to move these things
around to the screen because we can we
can compact it down we have a lot of
black spaces we can use the parallax
scrolling I'm saying you can use OpenGL
to get a much faster version of this
with a lot less imitations on you what
we're gonna do is we we use a tiled
images to form a large textured
background pretty unlimited you can even
load the tiles you know I mean if you
think of something like Diablo 2 it
actually was a similar method where it
loaded terrain loaded sprites as it
needed to it was a forced perspective
kind of two and a half D thing but a
reality it was still kind of the same
idea of using an accelerated engine to
drive something that could have been
done years ago or would be to attempt to
be done years ago with a 2d engine so we
used a grid of polygons for the
background we're gonna and there's a lot
of different ways to draw all the grid
of polygons we toyed around with him in
talking about it and there's some
different kind of techniques you can use
as far as deciding what's on the screen
and off the screen depending on your
limitation so I'm not gonna try and tell
you exactly how to do that because I
found out that depending on
what your limitations are of your
application that you can find some ways
are really easy and some ways they're
very difficult the idea here though is
that you keep a grid of polygons you're
gonna draw something slightly bigger
than the screen or make sure you draw
the polygons that are actually present
on the screen and you're gonna draw the
ones that you need to as the person
moves whether whether it's north south
east or west when they move that way
you're gonna draw some additional
polygons basically when you draw them
gridded polygons you're gonna find out
what your world center is in this case
we just attach it to a sprite and we
pick that sprites world center and we
offset everything else by that world
center so everything else just moves off
the screen out as it would I mean so you
don't have to worry about some kind of
strange transfer when you're writing
yourself just use OpenGL modelview
matrix and transform it as you would
with with the world center attached to
attach to one of your sprites we're
gonna rotate it by world rotation so if
your sprite rotates you can rotate the
entire world or you can rotate your
sprite whichever way you'd like to do it
so there's no limitation that everything
has to be kind of orthographically
displayed across your screen which i
mean if you're thinking about it from a
2-d standpoint I'm not thinking that
it's gonna be really easy to take a 1400
by 480 image and turn it slightly or
OpenGL can do that for you as we've seen
and your scale by world scale most of
time you can zoom in and zoom out this
case you can zoom in and zoom out or you
can draw your little radar map up in the
corner of your game or application or
showing the how you're manipulating a
large image by just showing it a scaled
version using MIT maps of that larger
texture you've already drawn so how do
we handle textures you load the
background as one texture or a set of
set of textures but basically you load a
large image as a single large texture
you're going to segment it with the
techniques we've shown before and you're
let OpenGL handle the text and do the
texture healing for you use generate a
gen textures to generate texture list as
we seen before and will let OpenGL
handle the memory memory management in
and out of VRAM or in a manner to a DP
space because the OpenGL implementation
on ten is a very good job of handling a
gpn it's gonna get better in the future
and that is a really should take
advantage of that fact instead of trying
to have your own what's in what's in
memory what's not in memory let's load
something let's remove something just
let it handle it for you it'll push
things out you're not using it'll bring
things in the that you need and I think
you'll find it it'll be very effective
in managing those textures for you and
we this is again the same as your
segmented images this is exactly the
same technique we've talked about before
so how do you draw how you draw this in
my case I set up an orthographic context
I thought about it later I thought they
were really cool that you could use a
perspective context actually for your
side scroller and when you zoomed in and
zoomed out it actually would have that
kind of parallax weird zooming but I
didn't have time to do that but that's
something to think about so you mean if
you really don't really have to set up
an orthographic context you can say you
think about it as your artwork bench
where you're looking straight down on
one of those you know camera mounted
above a table you actually can can
change or use the geo you look at and
you can change the field of view in the
aspect you want to zoom in you just
change your field of view to be wider
you want to zoom out you have changes to
be narrower you change your
your-your-your near and far planes and
that I mean that kind of thing you can
do and gives you a lot of the features
that you really would work hard for
doing it yourself in with your own 2d
engine you can give it to you with with
hardware acceleration in OpenGL in this
case we're gonna pin the world to a
sprite of interest here's another
another good thing you can do let's say
you're you have some kind of a strategy
game or some some something that you
have multiple places of interest you
have a very large document you're
handling you can have the the user can
jump to an area of interest very easily
by just manipulating the center of that
image by it'll do a lookup of your
stripe sprite information or do a lookup
information into your document jump to
this object on my document I want to see
where how it's positioned in my preview
and they can do that basically by you to
read out where that position is and
offset to that position without having
to do some kind of calculation off I
mean to move a lot of pixels around
yourself and what I did was to draw the
sprites I basically do it you kind of
grossly call the sprites to it to your
to your visible list you don't want to
send OpenGL every single sprite that's
off the screen in your your massive area
but you just grossly call the ones you
think are pretty or close to visible not
OpenGL do the fine work you may spend
you may find if you spend way too much
time trying to replicate the
transformations trying to figure out
exactly what's on the screen and off the
screen you're spending a lot of time
doing that one one cat one thing i'll
i'll tout here is the open g
optimization session tomorrow with John
Stauffer will look very closely at what
could open GL can do for you what I
can't do
as far as optimizations so come to that
session look at his example of
optimization apply it to your same
application make sure you know in real
time what percentage of time you're in
your application and what percentage of
time you're in OpenGL null out those
drawing routines and find out if you run
your application open-loop what actually
is usually if possible frame rate a lot
of times you'll find that you're not
really spending much time in OpenGL
you're spending a lot of time of your
application it doesn't matter how fast
the graphics engine is you're gonna have
to improve your application speed and
lastly we use the depth buffer for
layering I think we've seen that before
so we're gonna do a little demo here
I'll open my spread file it's a couple
sprites here I think the background
image is 2,000 by 1600 I have another
texture in there there's a thousand by
thousand both 32-bit and then I have my
sprite images which are also there's
actually 256 by 256 and I just use a
scaling of OpenGL to do it
so we're going to run around and see
what this thing can do so as you can see
I can move move the rocket back and
forth I can fly around very easily and
we can zoom in or we can zoom out real
nice that you get that you get a nice
feedback and if I can do this right I
probably can find Apple on here this is
that months to be a photograph of
Cupertino let me see you and do this
where am i oh I think we're up there
that's
no can't find out but I'll find it later
in any case you can see what I did here
for the poor man's version of tiling I
flipped the image and I guess use OpenGL
to do the same image you can zoom in you
can see that and we got the cloud layer
there so we have a nice layer of clouds
that's good flying over it one thing you
can do is this is your standard side
scroller you kind of fly around at
scroll side-to-side one thing you can do
also is you can do this very easily in
OpenGL which works the same it's doing
the same amount of work so I can fly
around in this case and my standard
method of you know up up you know the
top is up and have a pretty pretty
convincing game and make you all sick at
the same time so we'll we'll go back to
the presentation now and the last thing
we're to talk about it's QuickTime
integration how do you integrate
QuickTime and I have about 10 minutes I
think I can get through this a lot of
this we've talked about already the same
techniques this whole thing is using the
same technique it's all based on the
same stuff it's not it's not rocket
science it's not separate stuff you
learn some of this stuff once and it
apply to all these techniques QT new G
welcome pointer we've seen that at least
three times today that is how you can
allocate the texture to get a packed
texture format without having any any
extra space to do help to not allow your
texture we're gonna use pack pixels for
direct texture handling we're gonna
synchronize to QuickTime to minimize our
update what that means is we're gonna
sink we're going to give a callback
procedure to QuickTime to tell us when
it updated the frame if we haven't
updated the frame we're not going to
update the texture what this means is
the only textures uploaded to the card
are the ones that actually got updated
you can see obviously if you upload the
frame every single time it's like
playing an uncompressed movie you can
see how that can compare to a compressed
movie where you're only really updating
the frame every time it changes if you
have a frame if you have a OpenGL that
can do 100 frames per second in
QuickTime something in the movie at 24
frames per second there's no reason to
update that texture let the texture sit
in free Ram and then only update it when
you need to
drawling we're going to use sub in the
subtext image to that's wrong
may be wrong in any case we're going to
use a do a sub texture upload so roll in
fits the movies usually are not 256 by
256 or 512 by 512 they're usually a four
by three which is usually not the
texture size so we're gonna take the sub
texture out of that and we're going to
only update the part that's actually
updated again saving bandwidth to the
card so what we're gonna do here
new pointer clear we've seen that QT New
Year award from pointer we've seen that
set gee world set movie gee world we
haven't seen that so all you're gonna do
here is set the movie gee world same as
we did we think with the graphics
compressor and that'll tell the movie to
play into that G world texture formats
pretty simple gilt xmh 2d RGB that that
would be an uncompressed format and then
they have the reversed format showing
showing the actual pack pixel format
quick time synchronization here's
something new you look up there you see
draw movie draw upp that's gonna give
you a universal procedure pointer to a
movie to the movie completion procedure
would be new movie completion procedure
so you're actually creating it with my
movie movie draw proc and then you're
set the procedure very simply for the
movie G movie is actually the pointer to
your movie so that's something you get
from QuickTime and then in your
completion proc oh I'm sorry if you look
at the call it says movie drawing call
wind changed so it's going to call you
every time it updates that movie into
the G world so QuickTime will tell you
when it has something changed this is
the only code you have to do to like
QuickTime know when you've actually
updated it you can see this could
actually be very useful and some other
things that use QuickTime so you know
when there's actually something changed
in the movie you're not trying to update
something all the time and then a movie
draw proc really simply all I'm gonna do
is in the refcon I have a global
variable GF movie text update and I'm
gonna set that to true every time I get
to that procedure so when I go down to
draw my polygon
I'll get alright all ready to draw and
I'll say do I need to update the texture
if I do I'll react the texture from the
movie G world if not I'll just draw the
polygon with the texture you can use the
the idea of current textures because you
can bind it to any texture you want you
can have multiple movies plane you can
also just in this case among the example
you'll see I just have one movie
one texture so I just leave it leave a
current and how do we draw GL tech sub
image 2d I was right it was wrong it's
not subtext images text sub image 2d
geotextile image 2d you'll see here and
this basically is the exact same thing
you've seen before as far as using the
same kind of cord same kind of call can
you use the 32-bit version or the 16-bit
version then I'm going to move the frame
what that does is that's just my call to
move the frame around the screen however
I'm manipulating that quick time frame
that's the only the frame of the polygon
I'm using the modelview matrix to do
this I'm not doing this this is not
manipulating the movie pixels itself
just the frame and then I'm gonna do it
GL begin on coop with quads
and then you set the texture coordinates
for the corners I'm gonna set the color
to white for the whole for the whole
thing you can use cut you could colorize
your movie if you wanted to by changing
the polygon color and then I'm going to
very simple verdant vertices use the
user basically a square vertices and the
movies going to be inside that square so
let's go back to this demo
okay I'm gonna show you a couple
different things here first in this case
we're gonna bring our display size at
640 by 480 this demo does not use a
segmented texture but you could easily
integrate that into this demo this uses
a single texture that the movie is
compressed into 256 by 256 movie I'll
use a 16-bit off screen full screen I
will use pack pixels and I will sink to
the VBL which we'll see how that works
not on this actually let me turn off
sink to vbl water then we'll turn it
back on for the next run I'll pick a
movie we've seen this before and mr.
Schafer would you prefer window or aisle
so middle 620 friends per second playing
that movie through up with GL let's play
in the movie fullscreen and texturing as
fast as I can a single frame
some real real simple there hi
now we'll stay with this demo and we
will in this case well you would just
throw some effects in when use fog and
we'll sink the VBL and we just leave it
like that and we'll play it in a window
this time and mr. Schaefer would you
prefer window or aisle middle about 70
frames per second I'm not sure about the
sink on this because it seems to be very
more than I normally see because of the
the video simple fog turned on linear
fog for this real simple to show an
effect with the movie you get the full
complement of opengl cool stuff you can
do with it without having to do any
extra works it's a little movie I'm
working
let's go back to the presentation so
where do you get resources for this it's
pretty simple
the red book and blue book if you're
working with OpenGL make sure you got
the red book in the blue book the other
thing that I think that you should
everyone should have is in our OpenGL
SDK you'll see the spec look at the spec
I use that more than I use the red book
and blue book if you notice if you look
at the spec the red book and blue book
look very much like the spec but it has
a lot of it goes into more math and many
of us want to know sometimes but it
actually has the real what's behind the
engine how does it actually work which
is real important if you want if you're
trying to debug something like fog you
want to know exactly how the fog is
gonna work or you're trying to look at
how the texture that the unpacked Road
length you want a good explanation we'll
unpack roll length it goes through
exactly how it handles pixels so I think
the OpenGL one-to-one spec is a very
good thing to have and all programmers
working with OpenGL should have it it's
in our SDK it's in a PDF format you can
print it out and or look at it online
good places to know if you're working
with OpenGL wwl opengl org a lot of
resources tells you where the what the
current standard of OpenGL is he has a
complete list of the extensions for
OpenGL gives you a lot of programming
examples in starter code lists on Apple
comm search for OpenGL we have an OpenGL
Mac list I want to see you all on that
list it's a great place to get in for
information ask questions and we monitor
the list and try and help out when we
can and lastly developer.apple.com slash
OpenGL so that's another good place to
go all the developer.apple.com urls go
to the opengl page it's a pointer to the
sdk a pointer to the newest things we
got for it so what did we talk about
today we talked about using OpenGL
mainly for 2d and two and a half T and
we did some movie stuff too but how
OpenGL can be used to complications that
aren't interested in doing next the next
quake or the next Tomb Raider or the
next
nanus or or a kromagg rally we're
talking about games they want to use
just to display images want to use the
power of OpenGL to manipulate those
images and it's a really great thing
that you can do it easily you can set up
your transforms to let the transforms
handle the workload for you and this led
OpenGL do the hard work and
we will continue to improve OpenGL will
continue to work with with the hardware
hardware vendors hit the best hardware
acceleration we can and that will give
you they'll continue to improve your app
even without you doing work on it next
it does allow robust hardware
acceleration you're not going to get
that any other way you slice it you're
not gonna be able get to the hardware
acceleration unless you're gonna use
OpenGL or use one of our provided API
use don't have the resources and and the
ability to get at some of the hardware
acceleration unless you're writing the
drivers unless you're writing writing
the the OpenGL implementation and it
really shows you a new paradigm to solve
these real world problems you don't have
to say well hey I use copy bits I'm
gonna rotate that and I'm gonna gonna
use that and that's gonna be really
difficult maybe I'll go and Swizzle now
so OpenGL rotate your modelview matrix
throw it on the screen scale your
modelview matrix throw it on the screen
you can shift it over move it around
it's real easy it's just a couple
instructions sample code the samples
will be up very shortly the movie
samples up right now the image sample
should be up by the end of the week and
the the rocket samples should be up
probably next week let's talk about the
road map Sergio mekin mention there this
is the first of five sessions the
sessions you want that you want to go to
all right after this is geometry
modeling after that tomorrow morning
open G optimization John Stauffer the
head of our OpenGL team it's gonna be in
here and soon to take you through
optimizing an OpenGL app it's gonna be a
really great session he has some really
great tools to show you how to do how to
optimize that app and I'll show you how
to get the most performance out of an
OpenGL application advanced rendering
Troy Dawson one of our engineers in the
opengl team is gonna come come and show
some advanced rendering techniques and
you'll definitely want to see that if
she talks to talks about talks about
Stenson's using the stencil buffer to do
shadows talks about anisotropic
filtering and some other topics that'll
be really good talk and we have the
feedback form for open OpenGL as the end
of the day tomorrow please come talk to
us about what you want to see an OpenGL
where we're doing well where we're not
doing well we want to get your feedback
we want to take your your stuff in and
we want to make sure that we improve it
and make it the most world-class
implementation of OpenGL and 3d graphics
API anywhere and lastly there's a
feedback form for QuickTime for quick
the QuickTime API which I think you've
seen is a very
official to your work and OpenGL for
texture handling and for movie playback
this feedback for him for a quick time
also Thursday at 3:30 Sergio Mello is
the technology manager for 3d
technologies at Apple it's a great
contact if you have questions about
implementations questions about where
we're going questions about about what
we have today for to solve your problems
please contact him Sergio to Apple comm
with those kind of questions
you