The merge kernels in HotSort minimize global loads and stores by maximizing the number of element comparisons performed per thread.

Up until now, the same merging algorithm and register configurations were being used across all CUDA architectures and the resulting merge kernels were approaching the Fermi and GK104 63 register-per-thread limit.

Since GK110 and GT200 devices support high register-count kernels, two new merge kernels have been implemented to exploit this capability.

These new kernels further reduce the total number of global memory transactions resulting in an ~8-12% performance increase in sorting large arrays of 32-bit and 64-bit elements.

Not a bad result for simply adjusting a few configuration files and rebuilding!

You can see a comparison between the old and new Tesla K20c kernels here.

The updated HotSort Benchmarks doc for all architectures including GT200 is here.

Update:

One last note on performance, I can actually achieve an extra ~2% improvement on large arrays as well as on large numbers of small arrays if the new high register-count merge kernels are used as early as possible in the sorting process. However, this results in the very small single array benchmarks being ~1% below their peak. Right now I'm mostly interested in small array performance, so I chose not to disturb the small array sorting kernel launch logic. The assumption is that this is entirely due to SMX under-utilization. The fix is straightforward: launch the smaller merging kernels when performing small sorts. I'll save that work for later.

For this reason, you might think that exploiting the __constant__ qualifier and its associated cudaXXXSymbol() routines is a good way of reducing the number of kernel arguments and generally cleaning up your code.

CUDA __constant__ variables can definitely simplify your kernel code however you're not only shifting some complexity toward module symbol initialization but you're potentially closing off the opportunity to launch back-to-back grids with different __constant__ variable values.

Let's look at what is actually being generated by the compiler and see if there is any major difference between kernel args and __constant__ variables at the SASS level. The expectation is that there is not (see Section D.2.5.2 in the CUDA C Programming Guide).

These two kernels are functionally the same but one uses parameters and the other constants:

They produce identical SASS code:

Since kernel parameters are constants on sm_20+ I expected both kernels to produce similar SASS. The only difference is that it appears constants are being pulled from different constant banks.

If you look at pre-Fermi output you'll see the SASS is not the same because kernel parameters are passed via shared memory.

So back to the original question. Are module-scoped __constant__ variables useful given that strictly using kernel parameters produces similar code? Sure, you can write simpler code as long as you're sure you'll never need to update the __constant__ variables in a context. This very issue and a workaround was recently discussed in the CUDA Developer Forums.

But what I really want is the ability to optionally declare that a const kernel parameter has been raised to kernel scope (file scope?) as if it had been declared as a CUDA __constant__ variable.

It would be convenient and powerful to be able to write something like:

    __global__ foo(__constant__ int* bar, int* baz)
    {
       ...
    }

This would indicate that the parameter bar should be raised to kernel scope. The benefit being that the developer wouldn't have to invoke cudaXXXSymbol() initialization routines after loading the kernel module and before kernel launch.

An earlier file scope declaration that matches the parameter name and type might also be appropriate.

This syntax won't (ever) be appearing in CUDA C99/C++ but I think it's conceptually interesting for anyone writing a performance-focused DSL for GPUs to understand where there are mismatches between C99/C++ and CUDA.

Update: A compromise might be to create a structure that contains all of the constants you wish to raise to kernel scope and pass it (by value) as a parameter to the kernel. This way only a single reference needs to be passed throughout the kernel to access a number of launch-time CTA-specific constants.

The idiom would look something like this:

    typedef struct
    {
       int* bar;
       int* baz;
    } KernelEnv;

    __global__ foo(const KernelEnv kenv)
    {
       myxl(kenv,...);
       plyx(kenv,...);
    }

The code snippet and PTX/SASS dumps are here.

An open question was whether designing a transpose that performed 64-byte stores would achieve better performance at the cost of more instructions.

The answer is yes. Performing 64-byte transactions improves throughput by ~8%. The new 64-byte transpose kernel reaches ~148 GB/sec. on a 1024x1024 matrix of 32-bit elements on a K20c (758MHz). This is getting very close to the ~153 GB/sec. demonstrated by the shared transpose example.

    Tesla K20c : sm_35 * 13
    transposeKernel<<<1024,64>>>(1024 x 1024)
    .... Validated!
    loops (100)  : avg   0.05274 ms. =  148.12 GB/sec.

The interesting difference between this approach and last time is that of the different approaches I tried the highest performing kernel dispensed with SHFL's and implicitly performed rotations by manipulating the lower bits of the load and store pointers.

Similar to last time, neither __syncthreads() calls or shared memory are required.

Here is an illustration of how I chose to rearrange a tile of 4 lanes each holding 16 32-bit values:

A quick explanation:

Each lane loads 16 values. A logical tile contains 16x4 elements and occupies 4 lanes. I explicitly show the rotations that are performed on load but do not show the counter-rotations that must be performed on store. More on that later.

The rotations can be performed by either SHFL's or implicitly through pointer manipulation. Implicit "rotation pointers" are created by add-masking or using a BFI instruction. On load, each 4x4 block of registers is rotated one lane to the right of the block directly above it. With either approach, the warp performs 16 standard coalesced load transactions.

The exchange phase simply rearranges values in lanes 2 and 3 of the tile so that the next phase can use the SLCT opcode to choose which value to store back to device memory. More simply, this is every warp lane id that has bit 2 enabled.

Finally, the select phase takes advantage of the symmetries created by the rotation and exchange phases to enable all 4 lanes to coordinate a 64-byte (16-element) write. If you inspect each set of same-colored blocks you will see that they can be selected and written as vec4 instances simply by checking whether the lane is odd or even.

The 16x4 registers are written by the 4 coordinating lanes as 4 vec4's in the order: red, orange, green, blue. They're each separated by a matrix width of elements.

As noted above, the 4x4 blocks of registers across the 4 lanes can either be counter-rotated using SHFL or the store pointer can be manipulated to perform the same operation implicitly.

That's it and the performance is pretty good. I found that the pointer manipulation approach was slightly faster than using SHFL's. Conversely, in the previous blog post the SHFL implementation was slightly faster than the pointer approach when performing 32-byte stores.

There are quite a few ways to avoid using shared memory but... it's probably not possible to get that last 10 GB/sec. of throughput achieved by NVIDIA's shared transpose example on 1024x1024x32-bit matrices. The number of instructions in that kernel is tiny.

But if you're interested in creating your own specialized transpose or unique data-rearrangement kernel, here is a list of low-level GPU features that you can mix-and-match:

• writing 16-byte v2 or v4 types to simplify inter-lane word movement
• explicit shuffling
• rotations via shuffling
• implicit shuffling on store
• implicit shuffling on load
• maximizing use of the SELP opcode: d = pred ? a : b;
• thinking in terms of the non-existent opcode MOVP: if (pred) a = c; else b = c;
• exchange two registers with an XCHG: t = a; a = b; b = t;

Please let me know if you have any questions or feedback.

I needed a fast and minimal routine that could transpose a warp with 32 elements per lane and store the result to device memory as the final step in a complex kernel.

However, I had a hypothesis that Kepler SHFL instructions and 16-byte stores by pairs of lanes would be functionally equivalent to the standard matrix transpose approach but not require any shared memory or thread block synchronization yet still achieve 100% memory store efficiency.

I banged out the code over the weekend to see what the PTX/SASS would look like and after getting encouraging performance results I used the hypothesized warp-centric "pseudo-transpose" primitive to implement a full matrix transpose.

Results

A 1024x1024 matrix of 32-bit elements transposes on a K20c at ~137 GB/sec.  This is ~22% better than the "optimized outer" variant example in the CUDA Toolkit which averages 112 GB/sec.

Here's the output from the micro benchmark:

    Tesla K20c : sm_35 * 13
    transposeKernel<<<2048,64>>>(1024 x 1024)
    .... Validated!
    loops (100)  : avg   0.05677 ms. =  137.62 GB/sec.

This is pretty good and beats the default NVIDIA example.  

But if the "optimized outer" transpose example in the CUDA Toolkit is modified to use a 32x8 block size (32x32 tile size) it jumps from 112 GB/sec to 153.5 GB/sec. on a K20c (758 MHz).  Kudos!

If you have a reason to avoid using shared memory then the approach I describe below remains a good option.

Implementation

The excellent Matrix Transpose Example in the CUDA Toolkit uses a square tile of padded shared memory to switch the row-column ordering. 256 elements are loaded from device memory, stored to the shared tile, synchronized, loaded from the shared tile in transposed order and stored back out to device memory.  It's simple and fast.  Note that very few registers are actually put to use with this approach. This isn't a problem since the kernel has a 1:1 thread to element ratio -- i.e. many threads are being used.

My transpose routine takes a different approach. Instead of striving to construct wide 128-byte aligned stores, each warp in this routine performs a trivial rearrangement of each lane pair's elements followed by every lane storing half of an aligned 32-byte transaction.

It's easier to show how this works with a few illustrations.

We start with a conceptual warp of 8 lanes with 8 32-bit elements per lane: 

Next, exchange the odd 16 bytes in even lanes with the even 16 bytes in odd lanes:

Once the shuffle is complete, 32 bytes of elements from a lane column have been transformed into two lanes of 16 bytes in the same register row. The 32-byte groupings are highlighted in different colors:

At this point, each lane pair can perform an efficient 32-byte st.global.vec4.u32 (or equivalent).  The output address calculation is very simple and dependent on the size of the tile and matrix.  

Each lane stores twice in this example.  The conceptual 8-lane warp will perform two 128-bit stores per lane.  Each 128-bit store results in four 32-byte aligned global store transactions.

Finally, if the transposed warp were reloaded from device memory here is where each color-coded 32-byte store transaction occurred:

Extensions

Pre-Kepler devices require minimal use of shared memory to simulate a SHFL operation but synchronization is still not required as all shared stores and loads are warp-centric.  

Transposing 64-bit elements is nearly identical except that only two elements are exchanged per lane pair.

Conclusion

It's not only possible to transpose a matrix without using shared memory or synchronization but it's also efficient!

The summary results are: 

• shfl.idx handles negative indices without any problem which means "shuffle rotations" are feasible.
• shfl.up has no chance to mask the "lane - bval" value so it sets the in-range predicate to false when negative indices are produced and the lane's current value is assigned.
• shfl.down has similar behavior.
• negative shfl.[up|down] offsets are treated as unsigned 5-bit values.  e.g. -5 = 27.

It's also important to note that invoking shfl.idx with a signed offset subtracted from the laneId results in no surprises and is a two or three-instruction SASS sequence:

I like to think of this operation as a "shfl.rot".

Source code can be found here.

Here's a screenshot of the output.  An 'x' indicates a false predicate.

The results are very good.

When sorting 64-bit keys, Kepler achieves ~49% of the throughput of the 32-bit key benchmarks. The wider comparison sort performs twice the number of SASS comparisons and triple the number of calls to __syncthreads() on types that are twice as wide so getting half the throughput is excellent.

Additional optimizations were made in the past few weeks and there is now a general performance improvement across all architectures: approximately 12% on GT200 and almost 5% on Kepler. Fermi's improvement was the smallest at ~1%.

HotSort dominates Thrust Radix Sort until ~8m keys but, as before, the most important performance numbers to observe are HotSort's "binned" sorting rates. On a GTX 680, HotSort posts some pretty incredible numbers:

• 1.1 to 10.3 billion keys/sec. for subarrays containing 1K to 1M 32-bit keys.
• 500 million to 5.3 billion keys/sec. for subarrays containing 512 to 512K 64-bit keys.

Some pretty plots of the 64-bit results are below starting on page 8. The 64-bit binned sorting results for Kepler are on page 9.

I needed a sorter that was portable, in-place, supported key-vals wider than 32-bits and could sort binned (tiled) independent data sets output by other GPU kernels.

But most of all, the sorting algorithm had to be uber fast on small GPUs.

A number of months later HotSort was completed. I've achieved most of my design goals and exceeded a few of them.

The HotSort algorithm outperforms the current GPU champ — Thrust Radix Sort — by a large margin up until ~8m elements on an NVIDIA Kepler GTX 680 GPU. You can see some performance plots here. The implementation shows similar advantages on both small and large Fermi and GT200 devices.

When simultaneously sorting subarrays containing 1k to 1m elements then HotSort can achieve sustained sorting rates from 1 billion to over 10 billion keys/sec. on a Kepler GTX 680. This is exactly what I wanted.

I'm currently testing support for wider key-val sizes — u32b32, u64 and u64b32.

Also, for those of you with eagle eyes, you can see that the algorithm shows significant roll-off after its peak throughput. Clearly that implies the algorithm is not exhibiting O(nlgn) complexity past this point. Don't fret, the roll-off will be fixed and HotSort will become competitive on very large arrays.

I'm expecting the sorting gurus from NVIDIA — Merrill, Baxter, Harris, Garland, et al. — to improve their own truly awesome implementations now that they have a target. Always good to have a competitor!

Watch here for more updates on HotSort!

I was getting over 200 bytes of spills in a critical kernel that had no spills at all on Fermi.

Not good!  So what could I do?

In my case, the answer was to force use of the LLVM compiler with the --nvvm switch. This produced kernels with either zero or at most 8 bytes of locals.

My understanding is that this switch is unsupported for pre-Fermi devices but it worked very well for me and all of the kernels passed their verification tests.

On an old GT215 @ 550 MHz:

opencc:   29.14 MKeys/sec
4.1-nvvm: 42.56 MKeys/sec
5.0-nvvm: 43.30 MKeys/sec

Almost a 50% improvement... I'll take it!

Update Aug. 17, 2012:

CUDA 5.0 RC finally removed the --nvvm compiler switch.

The workaround is to generate sm_1x PTX with a 4.x compiler and generate the cubin with the bug-fixed 5.0 ptxas.

It wasn't mentioned above, but 4.x and 5.0 Preview had a bug where sm_12 devices were treated as if they were sm_11 devices with only 8192 registers.  That bug is fixed in 5.0 RC thus the baroque workaround I'm suggesting. 

  const unsigned int w99 = warpSize * 99; 

  mov.u32    %r5, WARP_SZ;
  mul.lo.s32 %r6, %r5, 99;

Yes, the stage after PTX will do a great job folding/propagating away WARP_SZ but sometimes you need to resolve logic in the preprocessor.

Including a #define WARP_SIZE 32 in your kernel is just fine until NVIDIA tells us otherwise.

When there are shared memory write conflicts within the warp, the write from a thread on a higher lane, therefore containing a later triangle, will override a write from a thread on a lower lane, containing an earlier triangle. The CUDA programming guide explicitly leaves it undefined which thread will succeed in the write, but at least on GF100 the behavior is consistent and can be exploited.

Good to know!