Introductory CUDA Programming Seminar
Lecture Notes

Prerequisites and Outcomes

Prerequisites -- The student must be familiar with:

• Computer programming in C or C++

• Program development using Unix, ssh, shell, and text editor

Outcomes -- The student will:

• Understand the hardware and software architecture of NVIDIA's Compute Unified Device Architecture (CUDA)

• Understand how to implement parallel programming patterns on a graphics processing unit (GPU) using CUDA
  • Result parallelism
  • Agenda parallelism
  • Parallel reduction

• Observe the relative performance of sequential CPU programs and parallel GPU programs

• Code and execute several simple GPU parallel programs

• Learn where to find documentation on CUDA

Agenda

• 10:00 -- Introductory CUDA programming; result parallelism; Programming Exercise 1

• 12:00 -- Lunch

• 1:00 -- Agenda parallelism; parallel reduction; Programming Exercise 2

• 2:00 -- CUDA memory regions; more parallel reduction; Programming Exercise 3

• 3:00 -- Research examples; Parallel Crypto

• 4:00 -- Finished

The Problem of Speed

• Processors are fast, main memory is slow

• Conventional solution: Caching
  • Put a fast cache on the processor chip
  • Large fraction of the chip area devoted to the cache
  • Rest of the chip area devoted to just a few processor cores
  • Program runs at full speed accessing the cache rather than stalling when accessing main memory

• Alternative solution: Massive multithreading
  • Omit the cache; access main memory directly
  • Devote most of the chip area to many processor cores, e.g. 32
  • Run many threads on the chip, e.g. 1024
  • When one thread stalls accessing main memory, another thread can run, keeping the cores 100% busy (latency hiding)

• The conventional solution has been and still is used for regular CPUs

• The alternative solution is used by graphics processing units (GPUs)

Acronyms

• GPGPU = General-Purpose Computation on Graphics Processing Units

• CUDA = Compute Unified Device Architecture
  • NVIDIA Corporation's proprietary hardware architecture and software API for GPGPU on their graphics cards

• OpenCL = Open Computing Language
  • Emerging open standard for general-purpose computing on CPUs and GPUs
  • Analogous to OpenGL for graphics rendering on CPUs and GPUs

clairaut.cs.rit.edu

• Main CPU
  • Sun Microsystems Ultra 40 workstation
  • 1 GHz dual-core AMD Opteron 2218 CPU
  • 8 GB main memory
  • 250 GB hard disk
  • 750 GB second hard disk
  • Ubuntu Linux 10.04

• GPU coprocessor
  • NVIDIA Tesla C2050
  • 448 cores (14 processors, 32 cores per processor)
  • 1.15 GHz clock
  • 3 GB onboard memory
  • 515 gigaflops peak double precision floating point performance
  • 1.03 teraflops peak single precision floating point performance
  • Latest-generation "Fermi" architecture

• Tesla card versus regular graphics cards
  • Aimed at scientific computing rather than gaming
  • More expensive
  • 4 times faster floating point performance
  • Tested and burned-in for long-running calculations

CUDA Programming Model -- Threading

• Thread
  • Similar to, but more limited than, POSIX threads or Java threads

• Kernel
  • The code executed by a thread
  • Maximum size is 2 million ptx (assembly language) instructions

• Information marked [C2050] below pertains specifically to the Tesla C2050 card
  • Other GPU cards are generally less capable

• Thread block
  • A group of threads that execute simultaneously on the same processor
  • Maximum 1024 threads per block [C2050]
  • Each thread within the block uniquely identified by one, two, or three indexes (1-D, 2-D, 3-D block)
  • Maximum thread block dimensions 1024 × 1024 × 64 [C2050]
  • Fast shared memory and lightweight barrier synchronization among threads in the same block

• Grid
  • A group of thread blocks, each block executing on a different processor
  • Each block within the grid uniquely identified by one or two indexes (1-D, 2-D grid)
  • Maximum 65,535 thread blocks along each dimension of the grid [C2050]
  • Maximum 65,535 thread blocks in a 1-D grid [C2050]
  • Maximum 65,535 × 65,535 = 4,294,836,225 thread blocks in a 2-D grid [C2050]

• Stream
  • A thread block grid in the process of executing a kernel
  • The system schedules each thread block to execute on an available processor
  • Each thread in each block executes the same code (kernel)
  • The thread blocks "stream" through the compute engine -- "stream programming"

CUDA Programming Model -- Memory Hierarchy

• Processor registers
  • 32768 registers maximum for all threads in the same block [C2050]
    • This may limit the number of threads per block to less than 1024
  • Local variables in the kernel code typically go in processor registers
  • Fastest speed

• Local memory
  • Read-write
  • Not shared (per-thread)
  • Off the processor chip
  • Slowest speed

• Shared memory
  • Read-write
  • Shared by all threads in the same block only
  • On the processor chip
  • 48 kilobytes maximum, organized into 16 banks [C2050]
  • Fastest speed (unless bank conflicts occur)

• Global memory
  • Read-write
  • Shared by all threads in all blocks
  • Off the processor chip
  • Slowest speed

• Caches [C2050; other GPU cards may not have caches]
  • L1 cache on each processor chip: configurable
    • Either 48 kilobytes shared memory and 16 kilobytes L1 cache (default)
    • Or 16 kilobytes shared memory and 48 kilobytes L1 cache
  • L2 cache shared by all processor chips: 768 kilobytes
  • Local memory accesses cached in L1 and L2
  • Global memory accesses: configurable
    • Either cached in L1 and L2 (default)
    • Or cached in L2 only

• Constant memory
  • Read-only
  • Shared by all threads in all blocks
  • Off the processor chip
  • 64 kilobytes maximum [C2050]
  • Cached on each processor chip, 8 kilobytes [C2050]
  • In-between speed

• Texture memory
  • Read-only
  • Shared by all threads in all blocks
  • Off the processor chip
  • Cached on each processor chip, 6-8 KB
  • In-between speed
  • Hardware can do specialized lookups and interpolations on the data in texture memory

Stream Programming

• Stream programming puts more responsibility on the programmer than "regular" programming

• The programmer must decide and explicitly code:
  • How to partition the computation into a grid, blocks, and threads
  • Where (in what kind of memory) to put each piece of data for best performance
  • When to move data from one kind of memory to another
    • Caching (if present) helps, but to a more limited extent than a regular CPU

• SIMD = Single Instruction stream, Multiple Data stream
  • Each processor executes the same sequence of instructions in lockstep with all the other processors
  • Each processor performs the instructions on different data items

• CUDA uses SIMT = Single Instruction stream, Multiple Threads
  • Same as SIMD, with multiple threads in each processor
  • SIMT is most efficient when all threads execute exactly the same sequence of instructions
    • All threads branch the same way on an if statement
    • All threads do the same number of iterations in a loop
  • The threads are allowed to execute different instruction sequences, but then the program takes longer

Introductory Example -- Vector Outer Product

• Compute an N×N-element matrix C that is the outer product of two N-element vectors A and B
  C_ij = A_i B_j, 0 ≤ i ≤ N−1, 0 ≤ j ≤ N−1

• Sequential main program: OuterProductCPU.cu (download)

• GPU parallel main program: OuterProductGPU.cu (download)

  Things to notice:

  • Storage allocation in CPU main memory and GPU global memory
  • Data copying between CPU main memory and GPU global memory
  • How the grid and block dimensions are determined
  • How each thread determines its position in the overall computation
  • Explicit copying of vector elements from (slow) global memory to (fast) shared memory
  • Subsequent vector element accessing in shared memory

• Pseudorandom number generator: Random.cu (download)

• Utility functions: Util.cu (download)

• Makefile (download)

  all : OuterProductCPU \
  	OuterProductGPU
  
  OuterProductCPU : OuterProductCPU.cu Util.cu Random.cu
  	nvcc -o OuterProductCPU OuterProductCPU.cu
  
  OuterProductGPU : OuterProductGPU.cu Util.cu Random.cu
  	nvcc -arch compute_20 -code compute_20,sm_20 -o OuterProductGPU OuterProductGPU.cu
  
  clean :
  	rm -f OuterProductCPU
  	rm -f OuterProductGPU
  

• Example run on the GPU on clairaut.cs.rit.edu

  $ ./OuterProductGPU 142857 16384
  CUDA device 0: Tesla C2050, compute capability 2.0
  A[0] = 0.856121
  A[16383] = 0.074512
  B[0] = 0.927307
  B[16383] = 0.063003
  C[0][0] = 0.793887
  C[0][16383] = 0.053938
  C[16383][0] = 0.069096
  C[16383][16383] = 0.004694
  23 msec computation
  

• CPU vs. GPU computation time and total time comparison (msec) -- three runs

         ----CPU----  ----GPU----
      N  Comp  Total  Comp  Total
   1024    26     32     0    156
   1024    25     32     1    192
   1024    25     32     0    244
   2048   101    112     0    284
   2048   102    112     0    284
   2048    74     88     1    284
   4096   250    268     1    352
   4096   229    244     1    284
   4096   209    224     1    224
   8192   711    748     5    664
   8192   734    768     5    668
   8192   737    772     5    564
  16384  2625   2728    23   1752
  16384  2549   2656    22   1784
  16384  2600   2704    22   1776
  

• CPU vs. GPU computation time and total time comparison (msec) -- minimum of three runs

         ----CPU----  ----GPU----
      N  Comp  Total  Comp  Total
   1024    25     32     0    156
   2048    74     88     0    284
   4096   209    224     1    224
   8192   711    748     5    564
  16384  2549   2656    22   1752
  

Programming Exercise 1

1. Log into the guest account on clairaut.

   $ ssh guest@clairaut.cs.rit.edu
   

2. Important: Create your own subdirectory in which to do your work. Name it using your initials plus four random digits; e.g. mine might be called "ark0114".

   $ mkdir ark0114
   

3. Important: Change into your own subdirectory.

   $ cd ark0114
   

4. Make copies of the example source files and makefile. (Note the period at the end of the command; in Unix, this refers to the current directory.)

   $ cp ../examples/* .
   

5. Compile the example programs.

   $ make
   

6. Try running the vector outer product programs with various arguments.

   $ ./OuterProductCPU 142857 1024
   $ ./OuterProductGPU 142857 1024
   $ ./OuterProductCPU 142857 2048
   $ ./OuterProductGPU 142857 2048
   

7. Modify the example programs so that each element of the outer product matrix is calculated using the formula below. Note: sqrtf(x) returns the square root of float variable x. Change the names of the modified programs; add the modified programs to the Makefile.
   C_ij = (A_i² + B_j²)^1/2, 0 ≤ i ≤ N−1, 0 ≤ j ≤ N−1

8. Compile and run the modified programs with various arguments. Are they getting the right answers? How do the running times of the CPU and GPU versions compare?

Parallel Reduction

• Each thread has computed some result

• The per-thread results must be combined ("reduced") into one overall result

• The per-thread results are combined using a reduction operation

• Shared memory reduction

  Initial state: Threads 0-7 have data that needs to be added together

      Thread:     0     1     2     3     4     5     6     7
      Data:     927   340    37   972   544   514   596   519
  

  Step 1: Threads 0-3 add threads 4-7 data to their own data; synchronize

      Thread:     0     1     2     3     4     5     6     7
      Data:    1471   854   633  1491   544   514   596   519
  

  Step 2: Threads 0-1 add threads 2-3 data to their own data; synchronize

      Thread:     0     1     2     3     4     5     6     7
      Data:    2104  2345   633  1491   544   514   596   519
  

  Step 3: Thread 0 adds thread 1 data to its own data; synchronize

      Thread:     0     1     2     3     4     5     6     7
      Data:    4449  2345   633  1491   544   514   596   519
  

  Final state: Thread 0's data is the sum of all the threads' original data

• Shared memory reduction requires O(log₂ n) time to reduce n data items

• Global memory reduction uses atomic operations
  • Multiple threads operating on the same memory location are automatically synchronized
  • Atomic add, subtract, exchange, minimum, maximum, increment, decrement, and, or, exclusive-or, compare-and-set (CAS)
  • Not available in all CUDA cards

Compute Capabilities

• Some hardware and software features do not exist in some CUDA cards

  Feature	Compute Capability
  	1.0	1.1	1.2	1.3	2.0	2.1
  CUDA cores per multiprocessor	8	8	8	8	32	48
  Single-precision floating point	Yes	Yes	Yes	Yes	Yes	Yes
  Double-precision floating point	No	No	No	Yes	Yes	Yes
  32-bit integer atomic functions in global memory	No	Yes	Yes	Yes	Yes	Yes
  64-bit integer atomic functions in global memory	No	No	Yes	Yes	Yes	Yes
  32-bit integer atomic functions in shared memory	No	No	Yes	Yes	Yes	Yes
  Single-precision floating point atomic addition in global and shared memory	No	No	No	No	Yes	Yes
  Maximum threads per block	512	512	512	512	1024	1024
  Maximum x- or y-dimension of a block	512	512	512	512	1024	1024
  Maximum z-dimension of a block	64	64	64	64	64	64
  Maximum x- or y-dimension of a grid	65535	65535	65535	65535	65535	65535
  Number of 32-bit registers per multiprocessor	8 K	8 K	16 K	16 K	32 K	32 K
  Maximum amount of shared memory per multiprocessor	16 KB	16 KB	16 KB	16 KB	48 KB	48 KB

Reduction Example -- Vector Angle

• Compute the angle θ between two N-element vectors A and B

  θ = cos−1	A ⋅ B
  	|A| |B|

  	N−1
  A ⋅ B =	Σ	Ai Bi
  	i=0

  	N−1
  |A| = (	Σ	Ai2)1/2
  	i=0

  	N−1
  |B| = (	Σ	Bi2)1/2
  	i=0

• Sequential main program: VectorAngleCPU.cu (download)

• GPU parallel main program: VectorAngleGPU.cu (download)

  Things to notice:

  • No need to store the vector elements (they are chosen at random on the fly)
  • GPU's role versus CPU's role in the computation
  • How the grid and block dimensions are determined
  • How each thread determines its position in the overall computation
  • How each thread generates the correct vector elements
  • How the "extra" threads are omitted from the computation
  • Shared memory reduction of per-thread results
  • Global memory reduction of shared memory results

• Pseudorandom number generator: Random.cu (download)

• Utility functions: Util.cu (download)

• Makefile (download)

• Example run on the GPU on clairaut.cs.rit.edu

  $ ./VectorAngleGPU 142857 1000000
  NT = 1024, NBX = 977, NBY = 1, threads = 1000448
  theta = 0.722877
  154 msec initialization + computation
  

• CPU vs. GPU time comparison (msec) -- three runs

           N     CPU   GPU
        1000       1   152
        1000       1   147
        1000       1   147
       10000       4   153
       10000       4   146
       10000       4   143
      100000      33   152
      100000      33   147
      100000      33   147
     1000000     202   121
     1000000     199   148
     1000000     226   148
    10000000    1374   164
    10000000    1353   158
    10000000    1382   158
   100000000   12693   279
   100000000   12716   286
   100000000   12749   287
  1000000000  127170  1258
  1000000000  127027  1255
  1000000000  126595  1255
  

• CPU vs. GPU time comparison (msec) -- minimum of three runs

           N     CPU   GPU
        1000       1   147
       10000       4   143
      100000      33   147
     1000000     199   121
    10000000    1353   158
   100000000   12693   279
  1000000000  126595  1255
  

Programming Exercise 2

1. If you haven't done Steps 1–5 of Programming Exercise 1, do them now.

2. Try running the vector angle programs with various arguments.

   $ ./VectorAngleCPU 142857 1000000
   $ ./VectorAngleGPU 142857 1000000
   $ ./VectorAngleCPU 142857 10000000
   $ ./VectorAngleGPU 142857 10000000
   

3. Modify the original vector outer product programs to calculate the norm of the matrix C, where C is the outer product of the vectors A and B. The norm of the matrix, |C|, is calculated with the formula below. Note: sqrtf(x) returns the square root of float variable x. Change the names of the modified programs; add the modified programs to the Makefile.

   	N−1		N−1
   |C| = (	Σ		Σ	Cij2)1/2
   	i=0		j=0

4. Compile and run the modified programs with various arguments. Are they getting the right answers? How do the running times of the CPU and GPU versions compare?

Another Reduction Example -- Statistical Test on a Block Cipher

• The PRESENT-80 block cipher
  • Input: 80-bit secret key
  • Input: 64-bit plaintext
  • Output: 64-bit ciphertext (encrypted plaintext)

• A block cipher's ciphertext should be random
  • Each bit of the ciphertext should look like an independent coin toss

• Statistical test on PRESENT-80
  • Encrypt a series of plaintexts to get a series of ciphertexts
  • Count the number of 1 bits in each ciphertext (the "population count" or "popcount")
  • The popcounts should obey a binomial(n,p) distribution with n = 64 and p = 0.5

    Prob [popcount = k] =	(	n	)	pk (1 − p)n−k
    		k

    (	n	)	=	n!
    	k			k! (n − k)!

  • Specifically, with n = 64 and p = 0.5,

    Prob [popcount = k] =	(	64	)	2−64
    		k

  • Do a chi-square test of the hypothesis that the popcounts obey a binomial(64,0.5) distribution:
  • Let T be the number of trials (plaintexts encrypted)
  • Let A_k be the actual number of occurrences of popcount k, 0 ≤ k ≤ 64
  • Let E_k be the expected number of occurrences of popcount k if the hypothesis is true, 0 ≤ k ≤ 64

    Ek =	(	64	)	2−64 T
    		k

  • Compute the chi-square statistic

    χ2 =

    64 Σ k=0

    (Ak − Ek)2 Ek

  • Compute the p-value of the chi-square statistic
  • This is the probability that a value of χ² or higher would be observed by chance, even if the hypothesis is true
  • If the p-value falls below a significance threshold, e.g. 0.01, conclude that the hypothesis is false and PRESENT-80 is not random

• GPU parallel main program: PresentStatTest.cu (download)

  Things to notice:

  • GPU's role versus CPU's role in the computation
  • Data stored in device constant memory
  • How the grid and block dimensions are determined
  • How each thread determines its position in the overall computation
  • Shared memory reduction of per-thread popcounts (histogram)
  • Global memory reduction of shared memory popcounts (histogram)

• Gamma function and related functions: Gamma.cu (download)

• Utility functions: Util.cu (download)

• Makefile (download)

• Example run on the GPU on clairaut.cs.rit.edu

  $ ./PresentStatTest 0000000000000000 0123456789abcdef 0123 200 200 ./PresentStatTest 0000000000000000 0123456789abcdef 0123 200 200 CUDA device 0: Tesla C2050, compute capability 2.0 Bin Actual Expected 0 0 2.22045e-12 1 0 1.42109e-10 2 0 4.47642e-09 3 0 9.25127e-08 4 0 1.41082e-06 5 0 1.69298e-05 6 0 1.66477e-04 7 0 1.37938e-03 8 0 9.82806e-03 9 0 6.11524e-02 10 1 3.36338e-01 11 1 1.65111e+00 12 8 7.29242e+00 13 24 2.91697e+01 14 111 1.06261e+02 15 349 3.54203e+02 16 1110 1.08475e+03 17 2994 3.06282e+03 18 7966 7.99736e+03 19 19319 1.93620e+04 20 44000 4.35645e+04 21 91539 9.12781e+04 22 178819 1.78407e+05 23 326411 3.25787e+05 24 556210 5.56553e+05 25 892012 8.90485e+05 26 1334546 1.33573e+06 27 1876935 1.87991e+06 28 2486335 2.48417e+06 29 3083521 3.08380e+06 30 3597558 3.59776e+06 31 3945805 3.94593e+06 32 4071770 4.06924e+06 33 3947480 3.94593e+06 34 3594596 3.59776e+06 35 3085671 3.08380e+06 36 2483511 2.48417e+06 37 1876226 1.87991e+06 38 1336657 1.33573e+06 39 890746 8.90485e+05 40 556169 5.56553e+05 41 325881 3.25787e+05 42 178601 1.78407e+05 43 91425 9.12781e+04 44 43422 4.35645e+04 45 19604 1.93620e+04 46 8024 7.99736e+03 47 3102 3.06282e+03 48 1061 1.08475e+03 49 328 3.54203e+02 50 112 1.06261e+02 51 31 2.91697e+01 52 7 7.29242e+00 53 1 1.65111e+00 54 0 3.36338e-01 55 1 6.11524e-02 56 0 9.82806e-03 57 0 1.37938e-03 58 0 1.66477e-04 59 0 1.69298e-05 60 0 1.41082e-06 61 0 9.25127e-08 62 0 4.47642e-09 63 0 1.42109e-10 64 0 2.22045e-12 chi^2 = 59.985633 p-value = 0.619148 4893 msec computation

• Running time (msec)

   NBX   NBY  Encryptions    Time  Encr/sec  Bits/sec
   100   100     10240000    1223    8.37e6    536.e6
   200   200     40960000    4893    8.37e6    536.e6
   400   400    163840000   19578    8.37e6    536.e6
   800   800    655360000   78290    8.37e6    536.e6
  1600  1600   2621440000  313201    8.37e6    536.e6
  

Programming Exercise 3

1. If you haven't done Steps 1–5 of Programming Exercise 1, do them now.

2. Try running the statistical test program with various arguments.

   $ ./PresentStatTest 0000000000000000 0123456789abcdef 0123 10 10
   $ ./PresentStatTest 0000000000000000 0123456789abcdef 0123 20 20
   

3. Does PRESENT-80 still look random with different plaintexts and different keys?

4. Modify the PRESENT-80 statistical test program to introduce a fault in the cipher algorithm by inverting one or more bits in the ciphertext. Does the statistical test program detect the fault? Why or why not?

5. Modify the PRESENT-80 statistical test program to introduce a fault in the cipher algorithm by setting one or more more ciphertext bits to 0 or 1 ("stuck-at-zero" or "stuck-at-one" fault). Does the statistical test program detect the fault? Why or why not?

6. (Advanced) Modify the PRESENT-80 statistical test program to count, for each bit position in the ciphertext, the number of 1s that occur in that bit position in the series of ciphertexts. Do a chi-square test on the data for each bit position. Does each bit position of PRESENT-80 look random?
   • For each bit position:
   • χ² = 2(A − E)²/E
   • A = actual number of 1 bits observed
   • E = expected number of 1 bits observed = T/2
   • p-value = chiSquarePvalue (1, chisqr)

Parallel Crypto Research

• Prof. Kaminsky's Parallel Crypto web site: http://www.cs.rit.edu/~ark/parallelcrypto/

• GPU Parallel Statistical and Cube Test Analysis of the SHA-3 Finalist Candidate Hash Functions:
  http://www.cs.rit.edu/~ark/parallelcrypto/sha3test01/

For Further Information

• D. Kirk and W. Hwu. Programming Massively Parallel Processors: A Hands-on Approach. Elsevier Morgan Kaufmann, 2010. ISBN 9780123814722.

• NVIDIA CUDA Zone: http://developer.nvidia.com/category/zone/cuda-zone

• NVIDIA CUDA Toolkit: http://developer.nvidia.com/cuda-toolkit

• NVIDIA CUDA Library documentation online: http://developer.download.nvidia.com/compute/DevZone/docs/html/C/doc/html/index.html

• NVIDIA CUDA C Programming Guide: http://developer.download.nvidia.com/compute/DevZone/docs/html/C/doc/CUDA_C_Programming_Guide.pdf

• GPGPU: http://www.gpgpu.org/

• OpenCL: http://www.khronos.org/opencl/

• A. Bogdanov, L. Knudsen, G. Leander, C. Paar, A. Poschmann, M. Robshaw, Y. Seurin, and C. Vikkelsoe. PRESENT: an ultra-lightweight block cipher. In P. Paillier and I. Verbauwhede, eds. 9th International Workshop on Cryptographic Hardware and Embedded Systems (CHES 2007), Springer LNCS 4727, 2007.

Alan Kaminsky • Department of Computer Science • Rochester Institute of Technology • 4486 + 2220 = 6706

Home Page

Copyright © 2012 Alan Kaminsky. All rights reserved. Last updated 09-May-2012. Please send comments to ark­@­cs.rit.edu.