Shadow Mapping on iPad

I’ve implemented shadow mapping on the MD2 Library using the Vortex Engine for iOS wrapper. Shadow mapping is a technique originally proposed in a paper called “Casting Curved Shadows on Curved Surfaces” [1], and it brought a whole new approach to implementing realtime shadows in 3D Apps.

Shadow mapping on iPad. The shadow is calculated in realtime and animates as the model moves.

Shadow mapping on iPad. The shadow is calculated in realtime and animates as the model moves.

Implementing shadow mapping on iOS is by nature a problem that spans several programming languages. Objective-C for creating the UI, C/C++ for interfacing with OpenGL and GLSL for implementing the technique in the GPU’s fragment shader.

The math involved in shadow mapping spans all of these languages, with different coordinate space transformations being implemented in the language appropriate to the pipeline stage we’re working on. This makes the technique a little tricky to implement the first time you attempt to.

Here is another screenshot of the technique running on an actual iPad. Notice how the shadow is cast on the floor as well as on top of the crate in the background.

Upfront capture of the shadow mapping technique running on the iPad simulator.

Upfront capture of the shadow mapping technique running on the iPad simulator.

Shadow mapping will be coming up in the next version of the MD2 Library app.

[1] – Lance Williams – Casting Curved Shadows on Curved Surfaces. http://artis.imag.fr/~Cyril.Soler/DEA/Ombres/Papers/William.Sig78.pdf

Light Scattering

When we introduced Programmable Pipeline support to Vortex Engine, we claimed that better visual effects could now be introduced into the renderer. With the introduction of Render-to-Texture functionality into Vortex 2.0, coupled with the power of Shaders, we can now make good on our promise.

Light Scattering (also known as “God Rays”) is a prime example of what can be achieved with Shaders and Render-to-Texture capabilities.

In the following image, a room consisting of an art gallery with tall pillars is depicted. We want to convey the effect of sun light coming from outside, illuminating the inner nave. Enter Light Scattering:

A Light Scattering algorithm is used for improving the visual experience. The scene is rendered using Vortex 2.0. Room courtesy of RealityFrontier. (Click to Enlarge)

 

It can be seen in the picture above how the the effect, although subtle, brings more life to the rendered scene. There is still much room to improve the visual results, though, particularily with the results of Kenny Mitchel’s article on GPU Gems 3.

There is also room for optimization. Currently, the scene is rendered in realtime at an average of 187 frames per second, producing 1024×1024 images on a NVIDIA GeForce GTX465. Moving the algorithm to mobile devices, although easy from a coding perspective, might require extra work to achieve a high frame rate on the embedded GPU.

Here are three more captures from different angles.

(Click to Enlarge)

 

(Click to Enlarge)

(Click to Enlarge)

 

A short detour along the way…

I wanted to improve the Stencil Shadow Volumes code a little bit and enable it for the Programmable Pipeline in Vortex, however, I had to take a small detour to fix an issue related to mobile device support.

It turns out that OpenGL ES, the 3D library that Vortex uses to render its graphics on mobile devices, does not support rendering indexed geometry for indices larger than 16 bits. Keep this in mind when developing for mobile devices such as the iPhone or iPad.

I can completely understand the reasoning behind this decision. 32-bit indices could be considered too much data to submit to the GPU in a mobile device. Furthermore, they are not strictly necessary, as they could be replaced (if needed) by splitting the geometry into two groups defined by 16-bit indices.

The solution I devised, which is now part of Vortex 2.0, is to allow the user to specify the data size for the indices when defining the geometry. This provides the flexibility to use 32, 16 or 8 bit indices. You can even have several geometric objects with different index sizes in a scene.

The advantage of leveraging this mechanism is that now it is very easy to fine-tune the number of bytes used for index representation for improving performance. For example, using 16-bit indices instead of 32-bit indices would make no difference for representing models composed of less than 65536 vertices, while requiring a copy of just half the number of bytes to the GPU.

In the extreme case of 8-bit indices we would be constrained to only 256 vertices, but we would be sending only one fourth of the data to the GPU.

This was mostly plumbing work, so no new picture this time. Stay tuned for more updates!

Stencil Shadow Volumes

I’ve built Stencil Shadow Volumes into the Vortex Engine.

Shadows are a very interesting feature to implement in an renderer, as they provide important visual cues that help depict the relationship between objects in a scene. Notice in the following image how the shadow tells our brains that the Knight is standing on the floor (as opposed to hovering over it).

A Knight, lit by a green light, casts a shadow on a tiled floor.

The implementation is at this point no more than a prototype and requires significant more testing, however, since the visual results are very appealing, I wanted to share some of the images.

I personally have a bias towards using Shadows Volumes instead of Shadow Maps; I think the former algoritm leaves out much of the guesswork that Shadow Maps require. Furthermore, Shadow Volumes are not subject to some of the limitations of Shadow Maps, such as the map’s resolution.

Another point for Shadow Volumes is the fact that they provide a natural way to implement “soft shadows”: shadows that are not completely black but rather transparent. The following image corresponds to the same scene but with two minor changes: the light has been changed to white and the floor texture is different. Notice how we can see the floor texture englobed in the Knight’s translucent shadow.

The same knight, lit by a white light this time, casts a soft shadow on an industrial floor. Notice the shadow "translucency".

I hope we can have Shadow Volumes available for both rendering pipelines as part of Vortex 2.0.

Edit: Thanks to Gabriel G. for noting the term “soft shadows” was being used incorrectly.

An Angel with Generated Normals

Sometimes the 3D models we have to display provide no information other than vertex data for the triangles they define.

This might be enough to approximate the shape of the object, but having no normal data, we are severely limited if we want to apply realistic lighting on the object’s surface. Without lighting and textures, it’s very hard to depict the 3D shape of objects.

Angel without Normals

An Angel Model lit but with no Normal Data.

To help solve this problem, Vortex now provides a simple Normal generation algorithm that “deduces” smooth per-vertex normals from the geometric configuration of the 3D model. The results are nothing short of astonishing.

Angel with Generated Normals

The same Angel Model lit after Normal data was generated automatically by Vortex.

Normal generation does come at a cost, however. The 3D model depicted in the figures above is composed of 237,018 vertices shared among 474,048 triangles.

In order to produce smooth normals, the generation algorithm must study the triangle adjacency for every vertex and produce exactly one normal for every one.

On the machine the algorithm was developed, a Core 2 Duo @ 2.66 GHz with 4GB of RAM and for the Angel model, this process takes about 841.334 seconds, that is about 14 minutes!

The good news is that given a model, its Normals only need to be generated once. Once the application has the normal data, it can cache it and reuse it every time the model is to be added to a scene, avoiding the computation time altogether.

This gives the developer the opportunity to generate all Normal data offline and then having it attached to the model at runtime.

The Angel model was downloaded from: http://www.cc.gatech.edu/projects/large_models/angel.html

Improving Vortex’s Smart Pointers

In out last post, we talked about the shift we are currently making in Vortex, moving towards reference-counted resource management.

In this context, one of the problems we mentioned was the fact that due to the fact that C++ does not provide out-of-the-box type covariance/contravariance like C#, we ended up with some rather uncomfortable constructs for dealing with polymorphic data types.

Now, one of the great features of C++ is that the language is super-extensible. You can literally change the way many things work in C++ and, it turns out that after some research and experimentation (and a little C++ magic too), we found a way to implement C# covariance/contravariance for our custom smart pointers!

The latest improvements enable the following features:

  1. Implicit casts from a smart pointer (to an instance) of a derived class to a smart pointer of a base class.
  2. Static casts from a smart pointer (to an instance) of a base class to a smart pointer of a derived class.
  3. Compile-time errors from attempting invalid casts.

The following code fragments illustrate these points. These are taken directly from out test suite, with minor adaptations to simplify readability.

Fragment #1: Implicit cast from derived instances to base class instances:

ref_ptr<TransformNode> xformNode(new TransformNode());
ref_ptr<Node> node = xformNode; // Valid: implicit cast

Fragment #2: Static cast from a base class instance to a derived class instance. Note we called our static cast operator “ptr_cast”.

ref_ptr<Node> node(new TransformNode()); // Storing as a "more general" pointer
ref_ptr<TransformNode> pxformNode = ptr_cast<TransformNode>(node); // Valid: explicit cast.

Fragment #3: Finally, the following code is invalid, and will trigger a compile-time error:

ref_ptr<SimpleSceneGraph> sg(new SimpleSceneGraph());
ref_ptr<Node> node = ptr_cast<Node>(sg); // Illegal conversion: types are not related

We are very happy with the result and are looking forward to continue migrating the Vortex codebase to our new smart pointers. Programming with Vortex just got a whole lot easier and safer.

Towards Reference-Counted Resource Management in Vortex

Now that we have advanced support for a dual rendering pipeline in Vortex, we are now shifting our attention into a completely different topic. This time we are looking into overhauling the engine’s inner resource management.

To this purpose, we are conducting experiments shifting resource and scene management from raw pointers to smart pointers. The idea of using smart pointers is to enable each resource to be automatically deleted once it is no longer referenced by any component or object. This is not to be confused with Garbage Collection. The main difference consists in that smart pointers allow us to simplify memory management without sacrificing deterministic object destruction and without incurring into the memory and CPU overhead of having the Garbage Collector kicking in to rearrange the application’s Heap at any time.

So far, this has proven a very interesting experience. We’ve bumped with some unexpected issues along the way, but we’ve also run into a couple of nice surprises. Among the problems encountered, there’s the fact that since C++ does not support out-of-the-box covariance/contravariance (in the sense C# does), exposing the polymorphic behavior of the scene objects became a little more involved that what it used to be when using raw pointers.

On the bright side, using smart pointers dramatically improves some of the roughest edges of the API. Just to pick a simple example, let’s consider image lifecycle management. Without smart pointers, Texture creation from images in Vortex might look like this:

void loadTexture()
{
  Vortex::Image* pimg = Vortex::Image::loadFromFile("./resources/texture.png");
  
  // Create Texture from image data...
  
  // loadImageFromFile transfers pointer ownership, 
  // we must therefore not forget to delete the image:
  delete pimg;
}

Using Vortex’s smart pointers, however, we can forget about the explicit delete operation:

void loadTexture()
{
  ref_ptr<Vortex::Image> pimg = Vortex::Image::loadFromFile("./resources/texture.png");
  
  // Create Texture from image data...
  
  // Image object ownership is now managed by the 
  // smart pointer. The pointer releases the data 
  // upon exiting this scope (unless the pointer is copied)
}

ref_ptr<T> is the typename we’ve given to our in-house smart pointer implementation. These allow us to easily cast inside type hierarchies as well as dropping into the raw pointer when needed, allowing us to dodge the performance tax in performance-critical code.

So far the results seem promising. We will continue experimenting moving the rest of the codebase to smart pointers and hopefully we will be able to include this along with the dual rendering pipeline in the very next version of Vortex: Vortex 2.0.

Dual Pipeline support in Vortex Engine

Continuing with the news related to the Vortex Engine, I’m glad to announce that the next version of Vortex will come with two separate rendering pipelines: a fixed rendering pipeline and a programmable rendering pipeline that supports custom shader programs.


Comparison of the Rendering Pipelines available in Vortex Engine. The image on the left represents the Fixed Pipeline. The image on the right represents the Programmable Pipeline (click to enlarge).


Supporting two pipelines enables the developer to use the most appropriate rendering pathway for the target device’s capabilities.

Newer video cards for Desktop computers and the latest mobile devices can be programmed for though the programmable pipeline, whereas older devices (such as the original iPhone and the iPhone 3G) can still be supported through the fixed pipeline.

Anyone in the field of Graphics will be able to tell that the fixed pipeline will not be able to provide the exact same visual capabilities as the programmable pipeline, however, it should be less resource intensive and thus more suitable for some older devices.

The programmable pipeline, on the other hand, will allow Vortex applications to leverage the hottest techniques in Computer Graphics such as Bump Mapping, Specular Mapping, Blur and Depth of Field (among others), dramatically improving the visual detail.

This list provides, at a glance, the criteria for selecting the appropriate Vortex rendering pipeline depending on the target device:

  • OpenGL 1.1 / OpenGL ES 1.1 devices -> Use Vortex’s Fixed Pipeline
  • OpenGL >= 2.0 / OpenGL ES 2.0 devices -> Choose between Vortex’s Fixed Pipeline and Vortex’s Programmable Pipeline (recommended).

Needless to say, we’re very excited about the possibilities that this brings to the table and are very eager to see to what extents can the Vortex Engine be now pushed.

In the picture above you can see a sample comparison between the lightning model of the fixed pipeline (hardware-accelerated Gouraud Shading) and the one that the programmable pipeline offers (hardware-accelerated Phong Shading with specular reflections in this case).

This is just the tip of the iceberg. Stay tuned for more updates on the Vortex Engine!

Md2 Models and Vortex Scene Graphs

Someone once said there is no building the engine without building a game. There is nothing like actually using your own tools in order to pinpoint the weak spots of the Engine.

Perilith Knight with the "knight skin".

Using Vortex to build different 3D applications, I’ve noticed that it may, at times, be a bit more verbose than one would expect from an Object Oriented C++ API.

One of the roughest edges in Vortex was Md2 model animation support. Until last week, integrating animated models into a Scene Graph was unnecessarily complicated. Fortunately, this is no longer the case. Adding an animated Md2 model to a Scene Graph is now as easy as adding any other node type.

Controlling animations and skins of Md2 Models programatically is now a breeze in Vortex.

Here are some captures from the famous Perilith Knight model. I may upload a YouTube video showing the smooth animation at some point.

 

Perilith Knight using the "evil skin".

Perilith Knight using the "ctf_r skin".

Stay tuned for more updates on the Vortex Engine!