WebVR: mixing HTML and WebGL

Intro: Why is this important?

Over the past few years, it has become increasingly obvious that virtual reality is going to be revolutionary. As we begin to outgrow many of the technical limitations that have held VR back in the past (the Oculus CV1 and HTC Vive as proof), I think the real challenges for the future of virtual reality are creative ones. In a world where we have become so accustomed to incremental changes in both hardware and software, the freshness of VR is both extremely rare and extremely exciting. I predict we will quite soon, find ourselves in a world in which our only limit will literally be our imagination. But in order to get to that point, we as a community of developers, have to advance the technologies of the web to enable that future.

Current State of WebVR

As of today, creating websites that seamlessly blend HTML and WebGL together is nearly impossible. The best way that I am aware of is using a technique that Jerome Etienne explained on his blog post, Mixing HTML Pages Inside Your WebGL.

Basically, you just render from both the three.js WebGL and CSS3D renderer.

function animate() {
  controls.update();
  glEffect.render( glScene, camera );
  cssEffect.render( cssScene, camera );
  requestAnimationFrame( animate );
}

However, there are many limitations to this which he explains in more detail. I stumbled across some really cool demos created by Harts Antler that expand upon this technique that are definitely worth checking out. Here is a live demo he recently uploaded.

I am aware that Josh Carpenter and the Mozilla team, along with Brandon Jones and the Chrome team, are working on improving WebVR support for both browsers. I also found it interesting that Ricardo Cabello (creator of three.js) mentioned a well-funded company is also trying to solve this problem. My guess is he was referring to famo.us and their mixed mode renderer, but perhaps I'm wrong. Regardless, this is clearly a hot area of research and much work is being done to improve WebVR support, which is great!

My experience creating a demo:

I recently ported over Chris Dolphin's daisywheeljs into a WebVR scene (source here) as an exploration in VR input. (see my blog post about input here) Luckily, because of how great three.js is, this was relatively straightforward. I only ran into one serious problem...

All I had to do was get Jerome's earlier example working in stereo. From the WebGL side of things, this has pretty much been solved. You just pull in a couple of scripts that are available from the MozVR github page.
But from the CSS3D side, it's much trickier. Ricardo has actually already created a three.js stereo renderer for CSS3D available here. However, the current implementation simply clones the HTML and renders it twice from both cameras. This isn't an issue if your HTML does not care about event listeners, like his periodic table demo, but in my demo it is not a viable solution. I am simply loading an iframe element into the scene. This means that I have to click on each of the iframes separately to get my gamepad demo working in stereo. (two cloned dom nodes, two separate event listeners)

As far as I am aware, there is currently no good solution to this problem. I tried playing around with rendering the HTML to a texture to map to the right scene using html2canvas, and I even tried alternate frame rendering. Which means I rendered the single iframe from the left camera on odd frames, and the right camera on even frames to give the illusion of stereo; but this cut the effective framerate in half, and caused undesirable flickering. Finally I considered managing the events for both the left and right nodes in the parent document, and manually synchronizing them. This was not only very difficult, but a total hack. If you can think of a clean and efficient way of rendering a single CSS3D element in stereo, please let me know. If not, feel free to share your thoughts as to how this could be solved in the future. I stumbled across this tweet from last summer in my searches:
As well as this w3 proposal, but I don't know if either are ongoing.

Conclusion:

It is an exciting time to be a developer! We have a tremendous opportunity in the coming years to shape the future of the web in a totally new way. We should embrace this new dimension offered two us by WebGL, but at the same time, keep intact all the things that make the 2D web awesome.