I was getting over 200 bytes of spills in a critical kernel that had no spills at all on Fermi.

Not good!  So what could I do?

In my case, the answer was to force use of the LLVM compiler with the --nvvm switch. This produced kernels with either zero or at most 8 bytes of locals.

My understanding is that this switch is unsupported for pre-Fermi devices but it worked very well for me and all of the kernels passed their verification tests.

On an old GT215 @ 550 MHz:

opencc:   29.14 MKeys/sec
4.1-nvvm: 42.56 MKeys/sec
5.0-nvvm: 43.30 MKeys/sec

Almost a 50% improvement... I'll take it!

When there are shared memory write conflicts within the warp, the write from a thread on a higher lane, therefore containing a later triangle, will override a write from a thread on a lower lane, containing an earlier triangle. The CUDA programming guide explicitly leaves it undefined which thread will succeed in the write, but at least on GF100 the behavior is consistent and can be exploited.

Good to know!

Meanwhile, here is the view from my desk:

First, some relevant specifications on this machine:

• Dual-core Atom D525 @ 1.8GHz
• Intel NM10 chipset
• NVIDIA ION2 GPU (GT218) with 512MB of DDR3
• DDR2 667/800 SO-DIMMs on a 64-bit bus
• A RealTek Gigabit PCIe Ethernet port

There are a number of netbooks and nettops that have nearly identical specifications.

The NM10 has 4 PCIe 1.0a root ports but is connected to the ION2 using a single lane (x1) PCIe link.  The link has a theoretical 250 MB/s data rate.

The result is that, according to the "bandwidthTest.exe" benchmark, the ION2 is able to copy from host-to-device at 165 MB/sec and from device-to-host at 204 MB/sec.  Onboard the GPU, device-to-device copies reach a much higher ~8GB/sec with the default GPU clock setting.

The GT218 is a standard Compute Capability 1.2 device and has 2 multiprocessors each with 8 cores.

See below for the following results using the 260.63 driver and 3.2 RC CUDA Toolkit:

• GPU-Z
• CUDA "deviceQuery.exe"
• CUDA "bandwidthTest.exe --memory=pinned --wc"
• CUDA "bandwidthTest.exe --memory=pinned --wc --mode=shmoo"
• CUDA "nbody.exe"

The GPU is running at its default speed with 4GB of DDR2-667 overclocked to 800.

Here is a GPU-Z screen capture:

GPU-Z incorrectly reports the ION2 is running on a PCIe 2.0 x1 lane.

The CUDA 3.2 Toolkit's "deviceQuery" output is here.

And the results of the "bandwidthTest" with write-combined and pinned memory are here and "shmoo" results here.

And, finally, a CUDA "N-Body" screen capture:

It's also worth noting that this machine is running Windows 7 Ultimate x64 and, in its off hours, operating as a Media Center serving two XBOX 360's.  The CPU speed is acceptable but on the video side the ION2 has played everything I've tried with excellent results.

Feel free to leave a comment below if you want to see additional CUDA benchmarks.

While I was on vacation, I read Shasha and Lazere's new book Natural Computing and was surprised to see a small section dedicated to Iverson's APL, J and K languages.

The last and only time I used J was in 1993 for a grad school class on queueing networks.  A half-page of terse code was dropped into my paper and I was done.  I would not have wanted or been able to do this in C or Fortran.

At some point, I'd like to investigate porting a J-like language to CUDA or OpenCL.  The obvious array processing parts of the interactive language would map well onto a GPU but the language might need a scheme to ensure that enough work could be issued at once so that the GPU was fully utilized.  It would be a fun project if there was interest.

Below is a brilliant roundtable discussion on APL featuring Ken Iverson and others.  I saw it a year ago, but it's worth watching again if you've had any exposure (or mental damage from) APL/J/K.  Pour yourself some scotch, sit back and enjoy:

There is also a detailed description on this video's original page and a discussion at LtU.

Also check out Catherine Lathwell's cleverly named blog Chasing Men Who Stare at Arrays.  Catherine is working on a documentary about the history of APL language and its founders.

When I got back to my desk I recalled that the mid-1990's also had a number of companies building media processors to offload MPEG-1/2/4 video, audio, 2D graphics and even modem codec processing.

Googling reveals a great example.  The 1996 Hot Chips 8 presentation by Paul Kalapathy of Chromatic Research which covers the Mpact multimedia processor.  A quick skim reveals an architecture that should be familiar to any GPU developer: a sea of general-purpose registers, huge on/off-chip bandwidth and SIMD vector opcodes driving integer ALUs and SFUs.  The part reportedly achieved 1 BOPS at 120MHz.

Very impressive and another great spoke on the wheel of reincarnation!

If you dig around a little you'll find a great video by the chassis architect that starts by describing the impetus for the product.

I thought it was really interesting that when an oil and gas customer came to Dell and asked for a chassis solution for GPUs, their "GPU-to-server" ratio requirement went from 2:1 in the beginning all the way up to 4:1 (4 GPUs per server). 

Presumably this ratio was determined by testing and maybe tuning their GPGPU application.  Or it simply might've been because the chassis made it practical to access 4 GPUs.

A ratio of 4:1 sounds high to many developers because it's a challenge to install, power and cool that many GPUs in a standard chassis.

If the GPU:CPU limit is going to be loosened then it raises several questions:

• Which applications scale to high GPU-to-CPU ratios?
• How can developers practically find this limit?
• What GPU coordination "patterns" are there for scaling up?
• What is the next bottleneck: PCIe transfer speed or the need for device-to-device transfers?

I particularly like this paragraph:

One of the biggest factors, though, is the degree of motivation. In the past, programmers could just wait for transistors to get smaller and faster, allowing microprocessors to become more powerful. So programs would run faster without any new programming effort, which was a big disincentive to anyone tempted to pioneer ways to write parallel code. The La-Z-Boy era of program performance is now officially over, so programmers who care about performance must get up off their recliners and start making their programs parallel.

He finishes up the article by predicting three possible outcomes: multicore performance improvements stall out, the industry moves to new architectures like GPUs and the third, most optimistic, option, magic parallelizing compilers and software finally arrive.

The second option is almost where we're at right now.  There is plenty of experimentation and some notable wins.  There is also plenty of core infrastructure software out there that can benefit from being ported to multicore or hybrid hardware.  If a "La-Z-Boy era" system can get a 10x or more improvement in performance and power we'll see some motivated development.

The article primarily focuses on FPGAs.  Nvidia is mentioned as well as OpenCL.

Another company that uses FPGA's to process market data is Exegy.

I suspect that GPUs are being looked at very closely by Wall Street but one challenge that early adopters need to overcome is architecting a solution that gets away from the simple but latent "kernel launch and wait" approach:

1. copy data to device
2. launch kernel
3. wait for kernel completion
4. copy results from device

Now that CUDA supports page-locked/write-combining/mapped/overlapped memory and 2.0 supports concurrent data transfers you can quickly imagine some architectures and long-running kernels that might help you avoid the approach listed above and drive compute latencies downward.

Having worked with both high-speed market data feeds and the entire trading workflow, I can think of GPU applications ranging from mundane processing tasks all the way up to HFT.

I'm sure that there are plenty of people who have already done this and aren't talking about it!

Update 5/18/2010:  There sure are.  Take a look at Hanweck & Associates' Volera product.  They're repricing the entire options market with 10 milliseconds of latency.  Note that the OPRA feed is expected to produce 14B messages/day by the end of 2010. 

Anyone that has managed a team or product that balances deep tech and high creativity should be familiar with what Ed is talking about but you'll probably learn something new.

Scott's summary is here.