Manta Interactive Ray Tracer Development Mailing List

Text archives Help


Re: [MANTA] Running on a fat-node cluster


Chronological Thread 
  • From: Solomon Boulos <boulos@cs.utah.edu>
  • To: Abe Stephens <abe@sci.utah.edu>
  • Cc: Shreekanth Pavar <shreekanth.pavar@ucl.ac.uk>, MANTA <manta@sci.utah.edu>
  • Subject: Re: [MANTA] Running on a fat-node cluster
  • Date: Thu, 22 Mar 2007 22:29:02 -0400
I worked on dirt and even have the code still lying around (in fact, I bet I can still get it to run on our cluster here). For the most part, the ray tracing done in dirt wasn't the main difficulty but rather how to chop up the memory. Two different distributed shared memory (DSM) ideas were tried: object based (explicit request for memory) and page based (the user sees the standard "single flat memory" while some of these spots are located on other nodes). The object based DSM required modification to all the ray tracing code (based on the precursor to Manta) while the page based DSM (which I'm more fond of) basically doesn't. There's some setup work required in order to setup the distributed scene, etc, but for the most part the application is unchanged.

There were a number of problems with the page based DSM system outstanding though. One is that as described in the paper, it wasn't thread-safe because the mechanism used for automatically fetching memory from a remote node involved using page table valid bits to generate segfaults for addresses on remote nodes (these segfaults are then handled by our code which fetches the memory and then it looks like it just took a long time to access local memory) and had to change the state from invalid->valid to fill in the result (if another thread were to access the memory in the meantime it would get garbage data and crash).

I wrote a kernel module to get around this issue, but the next problem then crept up: we only had one network interface per machine. If you have 4 cores in a single box for rendering but only one network card, you quickly run into network latency and bandwidth as a bottleneck. Unfortunately, we only had access to gigabit ethernet and each node was a 2 CPU SMP machine, but we were able to saturate the network with this kernel module to basically match the performance of the thread-safe (by design) object based DSM -- anecdotally, this was about 1.5x over single thread.

Finally, the cost of a segfault is fairly large. It's a good trick to have a page based DSM, but for interactive applications this was still fairly costly.

There might have been a number of advances in the last year though in terms of operating system support for DSMs that I'm not aware of. In any case, if you'd like to see the underlying library we used for dirt, it's still available at Dave DeMarle's software page: http:// www.cs.utah.edu/~demarle/software/ (ice)

Solomon

On Mar 22, 2007, at 6:13 PM, Abe Stephens wrote:


My intention is to follow up on a suggestion made by Hansong Zhang at SGI to
port Manta to run on a fat-node (i.e. multicore) cluster. As far as I know,
this has not yet been tried. This would form the bulk of the implementation
work for my PhD.
No one has ported Manta to a cluster, however another IRT architecture, OpenRT has enjoyed great success on clusters. It is able to leverage multiple cores, I think in a more sophisticated manner than just starting up more renderer processes per processor. OpenRT doesn't (or didn't) support massive data set distribution, so for things like the 777 data set it is run on shared memory systems.

 I eventually plan to also introduce a data decomposition scheme.
With your permission, this could be the one which you used in the above paper.
I'm not sure if that dirt code is around still, as I believe it predates our current repository. The code certainly predates modern multi-core/simd CPU architectures, and uses an older software architecture than Manta. Still it might be a useful reference.

Steve might have a better idea of what its status is.

Abe






Archive powered by MHonArc 2.6.16.

Top of page