Choosing the Right GPU for Deep Learning on AWS (2023)

How to choose the right Amazon EC2 GPU instance for deep learning training and inference—from the best performing to the most cost-effective and everything in between

Choosing the Right GPU for Deep Learning on AWS (1)

published in

Towards Data Science


25 minutes to read


July 25, 2020


(Video) AWS re:Invent 2021 - How to select Amazon EC2 GPU instances for deep learning (sponsored by NVIDIA)
Choosing the Right GPU for Deep Learning on AWS (3)

Just a decade ago, if you wanted to use a GPU to accelerate your data processing or scientific simulation code, you either had to contact a PC gamer or a friendly supercomputing center near you. You can now log into your AWS console and choose from a range of GPU-based Amazon EC2 instances.

You ask which GPUs can you access on AWS? You can boot GPUs with different GPU memory sizes (8 GB, 16 GB, 24 GB, 32 GB, 40 GB), NVIDIA GPU generations (Ampere, Turing, Volta, Maxwell, Kepler) with different capabilities (FP64, FP32, FP16, INT8, Sparsity, TensorCores, NVLink), each instance has a different number of GPUs (1, 2, 4, 8, 16), paired with a different CPU (Intel, AMD, Graviton2). You can also choose instances with different vCPUs (core threads), system memory, and network bandwidth, and add a range of storage options (object storage, network file system, block storage, etc.) - in short, you have options.

My goal with this blog post is to guide you on how to choose the right GPU instance for your deep learning project on AWS. I'll discuss the key features and benefits of the various EC2 GPU instances, and the workloads that are best suited for each instance type and size. If you're new to AWS, new to GPUs, or new to deep learning, I hope you found the information you need to make the right choice for your project.

Topics covered in this blog post:

  1. Top Tips for Busy Data Scientists/Machine Learning Practitioners
  2. why you should choose the rightGPU instancenot just rightsgraphics card
  3. Learn more about GPU instance types: P4, P3, G5 (G5g), G4, P2, and G3
  4. Other Machine Learning Accelerators and Instances on AWS
  5. Cost optimization tips when using GPU instances for ML
  6. What software and frameworks are used on AWS?
  7. Which GPUs should be considered for HPC use cases?
  8. A complete and exhaustive spreadsheet of all AWS GPU instances and their capabilities

hurry? Just want the final recommendation without digging in? I got you covered. Here are 5 GPU instance recommendations that should work for most deep learning use cases. However, I do recommend that you come back and review the rest of this article so you can make a more informed decision.

1. Highest performing multi-GPU instance on AWS

Choosing the Right GPU for Deep Learning on AWS (4)

Example: p4d.24xlarge
When to use it:When you need all the performance you can get. Use it for distributed training of large models and datasets.
what you get:NVIDIA A100GPU with 40 GB of GPU memory per GPU. Based on the latest NVIDIA Ampere architecture. Includes third-generation NVLink for fast multi-GPU training.

2. Highest performing single GPU instance on AWS:

Choosing the Right GPU for Deep Learning on AWS (5)

Example: p3. 2xlarge
When to use it:When you want the highest performance single GPU and 16 GB of GPU memory is fine.
what you get:
1NVIDIA V100GPU with 16 GB of GPU memory. Based on the old NVIDIA Volta architecture. The best performing single GPU is still the NVIDIA A100 on the P4 instance, but you only get 8 NVIDIA A100 GPUs on the P4. This GPU has a slight performance advantage over the NVIDIA A10G on the G5 instance discussed next, but the G5 is more cost-effective and has more GPU memory.

3.Best performance/cost, single GPU instance on AWS

Choosing the Right GPU for Deep Learning on AWS (6)

Example: g5.xlarge
When to use it:When you want high performance, more GPU memory at a lower cost than P3 instances
what you get:
1NVIDIA A10GGPU with 24 GB of GPU memory, based on the latest Ampere architecture. NVIDIA A10G could be seen as A100's low-power cousinp4d.24xlargeSo when you need more compute, you can easily migrate and scale. consider a larger sizeg5.(2/4/8/16)xlargeFor the same single GPU with more vCPUs and higher system memory, if you have more preprocessing or postprocessing steps.

4. Best performance/cost, multi-GPU instances on AWS:

Choosing the Right GPU for Deep Learning on AWS (7)

Example: p3.(8/16) extra large
When to use it:Cost-effective multi-GPU model development and training.
what you get:
p3. 8xlargeThere are 4NVIDIA V100graphics processor andp3. 16xlargeThere are 8 NVIDIA V100 GPUs with 16 GB of GPU memory on each GPU, based on the older NVIDIA Volta architecture. For larger models, datasets, and faster performance, consider P4 instances.

5. High Performance GPU Instances on AWS

Choosing the Right GPU for Deep Learning on AWS (8)

Example: g4dn.xlarge
When to use it:The performance of model development and training is lower than other options, and the cost is lower. Cost-effective model inference deployment.
what you get:
1Nvidia T4GPU with 16 GB of GPU memory. Based on the previous generation NVIDIA Turing architecture. considerg4dn.(2/4/8/16)xlargeIf you have more pre- or post-processing, you can get more vCPUs and higher system memory.

With it, you should have enough information to start your project. If you're still eager to learn more, let's dive into each instance type, GPU type, what they can do with AWS, and discuss when and why you should consider each of them.

Or why you should look at the whole system and not just the type of GPU

GPUs are the workhorse of deep learning systems, but the best deep learning systems are not just GPUs. You must choose the right computing power (CPU, GPU), storage, network bandwidth, and optimized software to maximize the use of all available resources.

Some deep learning models require higher system memory or a more powerful CPU for data preprocessing, while other models may run well with fewer CPU cores and lower system memory. This is why you see many Amazon EC2 GPU instance options, some with the same GPU type but different CPU, storage, and networking options. If you are new to AWS or new to deep learning on AWS, making this choice can be overwhelming.

Choosing the Right GPU for Deep Learning on AWS (9)

Let's start with advanced EC2 GPU instance nomenclature on AWS. There are two GPU instance families - the P-series and G-series of EC2 instances, and the table below shows the different instance generations and instance sizes.

Choosing the Right GPU for Deep Learning on AWS (10)

in historyPInstance types represent GPUs better suited for High Performance Computing (HPC) workloads, featuring higher performance (higher wattage, more cuda cores) and support for double precision (FP64) used in scientific computing.GInstance type GPUs are better suited for graphics and rendering, characterized by a lack of double precision and lower price/performance (lower wattage, lower number of cuda cores).

All of this is starting to change as the volume of machine learning workloads on GPUs has grown rapidly in recent years. Today, a new generationPandGInstance types are suitable for machine learning.PThe instance type is still recommended for HPC workloads and demanding machine learning training workloads, I recommendGAn instance type for machine learning inference deployment and less compute-intensive training. All of this will become clearer in the next section when we discuss specific GPU instance types.

Each instance size has a specific number of vCPUs, GPU memory, system memory, number of GPUs per instance, and network bandwidth. Numbers next to letters (P3, G5)Generated on behalf of the instance. The higher the number, the newer the instance type. Each instance generation can have GPUs of different architectures, the timeline image below shows NVIDIA GPU architecture generations, GPU types and corresponding EC2 instance generations.

Let us now look at these examples one by one by family, generation and size in the order listed below.

Choosing the Right GPU for Deep Learning on AWS (11)
Choosing the Right GPU for Deep Learning on AWS (12)

P4 instanceprovide accessNVIDIA A100 GPUBased on the NVIDIA Ampere architecture. it only comes inone size— Multi-GPU per instance with 8 A100 GPUs, 40 GB of GPU memory each, 96 vCPUs, and 400 Gbps of network bandwidthrecord training performance.

Choosing the Right GPU for Deep Learning on AWS (13)

Overview of P4 instance features:

  • GPU generation: NVIDIA Ampere
  • Supported precision types: FP64, FP32, FP16, INT8, BF16, TF32, Tensor Cores third generation (mixed precision)
  • video memory: 40 GB per GPU
  • GPU interconnect: NVLink High Bandwidth Interconnect, Generation 3

What's new with the NVIDIA Ampere-based NVIDIA A100 GPU on P4 instances?

Each new generation of GPUs is faster than the last, and this is no exception. NVIDIA A100 is much faster than NVIDIA V100 (found on the P3 instance discussed later), but also includes newer precision types suitable for deep learning, notably BF16 and TF32.

Deep learning training is usually done in single precision or FP32. The choice of the FP32 IEEE standard format predates deep learning, so hardware and chip manufacturers have started supporting newer precision types that are more suitable for deep learning. This is a perfect example of hardware evolving to meet application needs and developers having to change applications to work on existing hardware.

The NVIDIA A100 includes special cores for deep learning called Tensor Cores to run mixed-precision training, which were first introduced in the Volta architecture. Instead of training the model in single precision (FP32), your deep learning framework can use Tensor Cores to perform matrix multiplication in half precision (FP16) and accumulation in single precision (FP32). This usually requires updating your training scripts, but can lead to higher training performance. Each framework handles this differently, so refer to your framework's official guide (tensor flow,torchandMXNet) for use with mixed precision.

NVIDIA A100 GPUs support two new precision formats - BF16 and TensorFloat-32 (TF32). The advantage of TF32 is that TF32 Tensor Cores on NVIDIA A100 can read and consume FP32 data from deep learning frameworks and produce standard FP32 output, but internally it uses reduced internal precision. This means that frameworks such as TensorFlow and PyTorch can support TF32 out of the box, unlike mixed precision training, which typically requires code changes to the training script. BF16 is an alternative to the IEEE FP16 standard with a higher dynamic range and better suited for handling gradients without loss of accuracy. TensorFlow hasSupport BF16For a while, you can now take advantage of BF16 precision when using an NVIDIA A100 GPUp4d.24xlargeinstance.

P4 instances have only one size:p4d.24xlarge. Let's take a closer look.

p4d.24xlarge: The fastest GPU instance in the cloud

If you need the absolute fastest training GPU instance in the cloud, look no furtherp4d.24xlarge。This title was previously given byp3dn. 24xlarge, which has 8 NVIDIA V100 GPUs based on the Volta architecture.

You can use 8 NVIDIA A100 GPUs with 40 GB of GPU memory, withThe third generation of NVLinkThis theoretically doubles the inter-GPU bandwidth compared to the second-generation NVLink on NVIDIA V100 available on the P3 instance type we discuss in the next section. which makesp4d.24xlargeInstance types are well suited for distributed data-parallel training and model-parallel training of large models that do not fit on a single GPU. The instance also gives you access to 96 vCPUs, 1152 GB of system memory (up to the maximum on an EC2 GPU instance), and 400 Gbps of network bandwidth (up to the maximum on an EC2 GPU instance), which is important for near-linear scaling of massively distributed training jobs.

runningnvidia-smiIn this instance, you can see that the GPU memory is 40 GB. This is the maximum GPU memory per GPU you can find on AWS today. This is an instance to consider if your models are large or if you are processing 3D images or other large data batches. runningnvidia-smi topology matrixYou will see that NVLink is used for communication between GPUs. Compared to PCIe, NVlink provides higher inter-GPU bandwidth, which means multi-GPU and distributed training jobs will run faster.

Choosing the Right GPU for Deep Learning on AWS (14)
Choosing the Right GPU for Deep Learning on AWS (15)

G5 instances are interesting because there are two types of NVIDIA GPUs under this instance type. This differs from all other instance types which have a 1:1 relationship between the EC2 instance type and the GPU architecture type.

Choosing the Right GPU for Deep Learning on AWS (16)

Each of them has different instance sizes, including single-GPU instances and multi-GPU instances.

First let's look at the G5 instance types, specifically theg5.xlargeThe instance sizes I discussed in the key takeaway/recommendation list at the beginning.

List of G5 instance features:

  • GPU generation: NVIDIA Ampere
  • Supported precision types:Supported precision types: FP64, FP32, FP16, INT8, BF16, TF32, Tensor Cores third generation (mixed precision)
  • video memory: 24 GB
  • GPU interconnect: PCIe

What can G5 bring you?

Choosing the Right GPU for Deep Learning on AWS (17)

GPU instance:g5.(2/4/8/16)xlargeBest price/performance for single GPU instances on AWS. start fromg5.xlargeServes as your single-GPU model development, prototyping, and training instance. You can increase the size tog5.(2/4/8/16).xlargeGet more vCPUs and system memory to better handle CPU-dependent data pre- and post-processing. You can access single-GPU and multi-GPU instance sizes (4 GPU, 8 GPU). Single GPU optiong5.(2/4/8/16)xlargeNVIDIA A10G provides the best performance/cost profile for training and inference deployment.

If you look at the outputnvidia-smiforg5.xlargeFor example, you'll see that the thermal design power (TDP), the maximum power the GPU can draw, is 300W. Compare this with the output ofnvidia-smiAs shown in the P4 part of the figure above, the TDP is 400W. This makes the NVIDIA A10G in the G5 instance a low-power cousin of the NVIDIA A100 on the P4 instance type. Since it's also based on the same NVIDIA Ampere architecture, that means it includes all the features supported by the P4 instance type.

This makes G5 instances ideal for single-GPU training, if your model and data volume grows and you need to do distributed training, or if you want to run multiple parallel training experiments on faster GPUs, you can split your training workload The load is migrated to P4.

(Video) AWS re:Invent 2020: How to choose the right instance type for ML inference

Choosing the Right GPU for Deep Learning on AWS (18)

Although you have access to multi-GPU instance sizes, I would not recommend using them for multi-GPU distributed training as there is no NVIDIA high-bandwidth NVLink GPU interconnect and communication will fall back to the significantly slower PCIe. The multi-GPU option on the G5 is designed to host multiple models per GPU for inference deployment use cases.

List of G5g instance features:

  • GPU generation: NVIDIA Turing
  • Supported precision types:Supported precision types: FP32, FP16, Tensor Cores (mixed-precision), INT8
  • video memory: 16 GB
  • GPU interconnect: PCIe
Choosing the Right GPU for Deep Learning on AWS (19)

What can G5g bring to you?

Unlike G5 instances, G5g instances offer NVIDIA T4G GPUs based on the older NVIDIA Turing architecture. A close cousin of the NVIDIA T4G GPU is the NVIDIA T4 GPU available on Amazon EC2 G4 instances, which I discuss in the next section. Interestingly, the main difference between the G5g instances and the G4 instances is the choice of CPU.

Choosing the Right GPU for Deep Learning on AWS (20)

G5g instances offer ARM-based AWS Graviton2 CPUs, and
G4 instances offer x86-based Intel Xeon Scalable CPUs.
The performance profiles of the GPUs (T4 and T4g) are very similar

Our choice between the two should come down to your preferred CPU architecture. My personal preference for machine learning today is G4 instances over G5g instances because more open source frameworks are designed to run on Intel CPUs than ARM-based CPUs.

Choosing the Right GPU for Deep Learning on AWS (21)

P3 instanceprovide accessNVIDIA V100 GPUBased on the NVIDIA Volta architecture, you can launch a single GPU per instance or multiple GPUs per instance (4 GPUs, 8 GPUs). Single GPU instancep3. 2xlargeCan be the daily driver of your deep learning training. and the most capable instancep3dn. 24xlargeGives you access to 8 x V100 with 32 GB GPU memory, 96 vCPUs, 100 Gbps network throughput, ideal for distributed training.

Choosing the Right GPU for Deep Learning on AWS (22)

Overview of P3 instance features:

  • GPU generation: Nvidia Walter
  • Supported precision types: FP64, FP32, FP16, Tensor Cores (mixed-precision)
  • video memory:16 GBp3.2xlarge、p3.8xlarge、p3.16xlarge, on 32 GBp3dn. 24xlarge
  • GPU interconnect: Second-generation NVLink high-bandwidth interconnect

NVIDIA V100 also includes Tensor Cores to run mixed-precision training, but not the TF32 and BF16 precision types introduced in NVIDIA A100 available on P4 instances. However, P3 instances are available in 4 different sizes, from a single GPU instance size up to 8 GPU instance sizes, making them ideal for flexible training workloads. Let's look at each instance size belowp3. 2xlarge,p3. 8xlarge,p3. 16xlargeandp3dn.24xlarge。

p3.2xlarge: best GPU instance for single GPU training

If you need a single GPU and performance is a top priority, this should be the instance of choice for most of your deep learning training work. G5 instances are more cost-effective with slightly lower performance than P3. andp3. 2xlargeYou get access to an NVIDIA V100 GPU with 16 GB of GPU memory, 8 vCPUs, 61 GB of system memory, and up to 10 Gbps of network bandwidth. At the time of writing, the V100 is the fastest GPU in the cloud and supports Tensor Cores, which can further improve performance if your scripts can take advantage of mixed-precision training.

If you start an Amazon EC2p3. 2xlargeinstance and runnvidia-smiThe command can see that the GPU on the instance is V100-SXM2 version, which supports NVLink (we will discuss it in the next section). Under Memory-Usage, you'll see that it has 16 GB of GPU memory. If you need more than 16 GB of GPU memory for large models or large data, then you should considerp3dn. 24xlarge(see below for more details).

Choosing the Right GPU for Deep Learning on AWS (23)

p3.8xlarge and p3.16xlarge: Ideal GPU instances for small-scale multi-GPU training and running parallel experiments

If you need more GPUs for experimentation, more vCPUs for data preprocessing and data augmentation, or higher network bandwidth, considerp3. 8xlarge(with 4 GPUs) andp3. 16xlarge(with 8 GPUs). Each GPU is an NVIDIA V100 with 16 GB of memory. They also include an NVLink interconnect for high-bandwidth communication between GPUs, which will come in handy during multi-GPU training. andp3. 8xlargeYou have access to 32 vCPUs and 244 GB of system memory, andp3. 16xlargeYou have access to 64 vCPUs and 488 GB of system memory. This instance is ideal for several use cases:

Multi-GPU training job: If you are new to multi-GPU training, 4 GPUsp3. 8xlargeor 8 GPUsp3. 16xlargeCan give you a nice speedup. You can also use this instance to prepare training scripts for larger multi-node training jobs, which usually requires you to modify the training scripts withLibraries like Horovod,tf.distribute.Strategyortorch.distributed.See my step-by-step guide for distributed training with Horovod:

blog post:A Quick Guide to Distributed Training with TensorFlow and Horovod on Amazon SageMaker

parallel experiment: Multi-GPU instances also come in handy when you have to run variations of your model architecture and hyperparameters in parallel for faster experimentation. andp3. 16xlargeYou can train up to 8 variants of the model. Unlike multi-GPU training jobs, you can increase productivity during the model exploration phase because each GPU runs training independently and does not block the use of other GPUs.

p3dn.24xlarge:High performance and cost-effective training

This instance previously held the title of fastest GPU instance in the cloud and now belongs top4d.24xlarge.that won'tp3dn. 24xlargeIs a listless person. It's still one of the fastest instance types you'll find on the cloud today, and it's more cost-effective than P4 instances. You can use 8 NVIDIA V100 GPUs, but withp3. 16xlargeWith 16 GB of GPU memory, on the GPUp3dn. 24xlargeHas 32 GB of GPU memory. This means you can fit larger models and train with larger batches. This instance gives you access to 96 vCPUs, 768 GB of system memory, and 100 Gbps of network bandwidth, which is important for near-linear scaling of massively distributed training jobs.

runningnvidia-smiIn this instance, you can see that the GPU memory is 32 GB. The only instances with more GPU memory than this arep4d.24xlargeSame as A100 GPU with 40 GB GPU memory. This is an instance to consider if your models are large or if you are processing 3D images or other large data batches. runningnvidia-smi topo — matrixYou will see that NVLink is used for communication between GPUs. Compared to PCIe, NVlink provides higher inter-GPU bandwidth, which means multi-GPU and distributed training jobs will run faster.

Choosing the Right GPU for Deep Learning on AWS (24)
Choosing the Right GPU for Deep Learning on AWS (25)

G4 instanceProvides access to NVIDIA T4 GPUs based on the NVIDIA Turing architecture. You can launch one GPU per instance, or multiple GPUs per instance (4 GPUs, 8 GPUs). In the timeline diagram below, you can see that directly below the G4 instances are the G5g instances, both based on GPUs with NVIDIA Turing architecture. We have discussed the G5g instance type in the previous section, the GPUs in the G4 (NVIDIA T4) and G5g (NVIDIA T4G) are very similar in performance. Your choice will come down to the choice of CPU type on these instances.

G5g instances offer ARM-based AWS Graviton2 CPUs, and
G4 instances offer x86-based Intel Xeon Scalable CPUs.
The performance profiles of the GPUs (T4 and T4g) are very similar

In the GPU timeline graph, you can see that the NVIDIA Turing architecture follows the NVIDIA Volta architecture and introduces several new machine learning features such as next-generation Tensor Cores and integer precision support, which make it cost-effective for inference Ideal for deployment and graphics.

List of G4 instance features:

  • GPU generation: NVIDIA Turing
  • Supported precision types: FP64, FP32, FP16, Tensor Cores (mixed-precision), INT8, INT4, INT1
  • video memory: 16 GB
  • GPU interconnect: PCIe

What's new with NVIDIA T4 GPUs on G4 instances?

NVIDIA Turing was the first to introduce support for the integer precision (INT8) data type, which can significantly accelerate inference throughput. During training, model weights and gradients are usually stored in single precision (FP32). It turns out that to run predictions on a trained model, you don't actually need full precision, you can get away with reduced precision calculations at half precision (FP16) or 8-bit integer precision (INT8). Doing so improves throughput without sacrificing too much accuracy. Accuracy will drop somewhat, depending on various factors specific to your model and training. Overall, you get the best inference performance/cost with G4 instances compared to other GPU instances.NVIDIA's Support MatrixShows which neural network layers and GPU types support INT8 and other inference precisions.

The NVIDIA T4 (and NVIDIA T4G) are the lowest powered GPUs on any EC2 instance on AWS. runningnvidia-smiIn this example, you can seeg4dn.xlargeFeatures NVIDIA T4 GPU with 16 GB GPU memory. You'll also notice that the power cap is 70W compared to 300W on the NVIDIA A10G.

Choosing the Right GPU for Deep Learning on AWS (26)

The following instance sizes all give you access to a single NVIDIA T4 GPU and an increasing number of vCPUs, system memory, storage, and network bandwidth:g4dn.xlarge(4 vCPUs, 16 GB system memory),g4dn.2xlarge(8 vCPUs, 32 GB system memory),g4dn.4xlarge(16 vCPUs, 64 GB system memory),g4dn.8xlarge(32 vCPUs, 128 GB system memory),g4dn.16xlarge(64 vCPUs, 256 GB system memory). You can findProduct G4 example pageUnder the "Product Details" section.

The G4 instance size also includes two multi-GPU configurations:g4dn.12xlargeThere are 4 GPUs and g4dn.metal has 8 GPUs. However, if your use case is multi-GPU or multi-node/distributed training, you should consider using P3 instances. runningnvidia-smi topology --matrixon multiple GPUsg4dn.12xlargeFor example, you can see that the GPUs are not connected via the high-bandwidth NVLink GPU interconnect. P3 multi-GPU instances include a high-bandwidth NVLink interconnect that accelerates multi-GPU training.

Choosing the Right GPU for Deep Learning on AWS (27)
Choosing the Right GPU for Deep Learning on AWS (28)
(Video) Deep Learning on the Cloud - GPU TO LEARN FASTER

P2 instanceAllows you to use NVIDIA K80 GPUs based on the NVIDIA Kepler architecture. The Kepler architecture is several generations old (Kepler -> Maxwell -> Pascal -> Volta -> Turing), so they are not the fastest GPUs. They do have some specific features, such as full-precision (FP64) support, which make them attractive and cost-effective for high-performance computing (HPC) workloads that rely on extra precision. P2 instances come in 3 different sizes: p2.xlarge (1 GPU), p2.8xlarge (8 GPUs), p2.16xlarge (16 GPUs).

Choosing the Right GPU for Deep Learning on AWS (29)

The NVIDIA K80 is an interesting GPU. A single NVIDIA K80 is actually two GPUs on one physical board, which NVIDIA calls a dual-GPU design. This means that when you start an instancep2.xlarge, you only get one of these two GPUs on the physical K80 board. Likewise, when you start ap2.8xlargeYou can access eight GPUs on four K80 GPUs, andp2.16xlargeYou can access sixteen GPUs on eight K80 GPUs. runningnvidia-smiin ap2.xlargeWhat you see is one of the two GPUs on an NVIDIA K80 board with 12 GB of GPU memory

Choosing the Right GPU for Deep Learning on AWS (30)

Overview of P2 instance features:

  • GPU generation: NVIDIA Capper
  • Supported precision types: FP64, FP32
  • Video memory:12GB
  • GPU interconnect: PCIe

So, should I even be using P2 instances for deep learning?

No, there are better options discussed above. Prior to the introduction of Amazon EC2 G4 and G5 instances, P2 instances were the recommended instance type for cost-effective deep learning training. Since the introduction of G4 instances, I recommend G4 as the preferred cost-effective training and prototyping GPU instance for deep learning training. P2 is still cost-effective for HPC workloads in scientific computing, but you miss out on some new features, such as support for mixed-precision training (Tensor Cores) and reduced-precision inference, which have become the standard for new generations.

if you runnvidia-smiexistp2.16xlargeGPU instance, since the NVIDIA K80 has a dual GPU design, you will see 16 GPUs that are part of 8 NVIDIA K80 GPUs. This is the maximum number of GPUs you can get on a single instance on AWS. if you runnvidia-smi topology --matrix, you'll see that all inter-GPU communication happens over PCIe, unlike P3 multi-GPU instances which use faster NVLink.

Choosing the Right GPU for Deep Learning on AWS (31)
Choosing the Right GPU for Deep Learning on AWS (32)

G3 instanceAllows you to use NVIDIA M60 GPUs based on the NVIDIA Maxwell architecture. NVIDIA calls the M60 GPU a virtual workstation and positions it as professional graphics. However, for more powerful and cost-effective options for deep learning using P3, G4, G5, G5g instances, G3 is not a recommended option for deep learning. I'm including it here just for some history and completeness.

Choosing the Right GPU for Deep Learning on AWS (33)

G3 instance features at a glance:

  • GPU generation: NVIDIA Maxwell
  • Supported precision types:FP32
  • video memory:8GB
  • GPU interconnect: PCIe

Should you consider G3 instances for deep learning?

Single GPU G3 instances are cost-effective for development, testing, and prototyping prior to the availability of Amazon EC2 G4 instances. Although the Maxwell architecture is newer than the NVIDIA K80 Kepler architecture on P2 instances, you should still consider P2 instances for deep learning ahead of G3. Your selection order should be P3 > G4 > P2 > G3.

G3 instances come in 4 sizes,g3s.xlargeandg3.4xlarge(2 GPUs, different system configurations)g3.8xlarge(2 GPUs) andg3.16xlarge(4 GPUs). runningnvidia-smiin ag3s.xlargeYou can see that this instance gives you access to an NVIDIA M60 GPU with 8 GB of GPU memory.

Choosing the Right GPU for Deep Learning on AWS (34)
Choosing the Right GPU for Deep Learning on AWS (35)

NVIDIA GPUs are undoubtedly a staple of deep learning, but there are other instance options and accelerators on AWS that may be better choices for your training and inference workloads.

  • CPU: for training traditional ML models, prototyping, and inference deployment
  • DL1 instance based on Intel Habana-Gaudi: Uses 8 x Gaudi accelerators, which you can use as an alternative to P3dn and P4d GPU instances for training
  • Amazon EC2 Trn1 instance: Use up to 16 AWS Trainium chips, which you can use as an alternative to P3dn, P4d, and DL1 instances for training
  • AWS Elastic Inference: Save money on inference workloads by leveraging EI to add just the right amount of GPU acceleration to your CPU instancesdiscussed in this blog post
  • Amazon EC2 Inf1 instances: Up to 16 AWS Inferentia chips with 4 Neuron cores on each chip, a powerful and cost-effective inference deployment option atMore details on this blog post.

For a detailed discussion of deployment options for inference, see this blog post on choosing the right AI accelerator for inference:

Choosing the Right GPU for Deep Learning on AWS (36)

Blog post:
A Complete Guide to AI Accelerators for Deep Learning Inference—GPUs, AWS Inferentia, and Amazon Elastic Inference

You have several different options for optimizing the cost of training and inference workloads.

Spot instance

Spot Instance pricing makes high-performance GPUs more affordable and allows you to access spare Amazon EC2 compute capacity at a significant discount compared to On-Demand rates. For an up-to-date list of prices by instance and region, visitSpot Instance Advisor.In some cases you can save over 90% on training costs, but your instances can be preempted and terminated with 2 minutes notice. Your training script must implement frequent checkpoints and be able to resume training when Spot capacity is restored.

Amazon SageMaker Managed Training

During the development phase, most of your time is spent prototyping, tweaking code, and trying out different options in your favorite editor or IDE (VIM, obviously) - all without requiring a GPU. You can save costs by simply separating your development and training resources, andAmazon SageMakerwill let you do this easily. Using the Amazon SageMaker Python SDK, you can test scripts locally on your laptop, desktop, EC2 instance, or SageMaker notebook instance.

When you're ready to train, specify the type of GPU instance you want to train on, and SageMaker will provision the instance, copy the dataset to the instance, train your model, copy the results back to Amazon S3, and tear down the instance. You only pay for the exact duration of your training. Amazon SageMaker also supports managed Spot training for added convenience and cost savings.

Here is my guide:A Quick Guide to Using Spot Instances with Amazon SageMaker

Use only as many GPUs as you need for Amazon Elastic Inference

Save money on inference workloads by leveraging EI to add just the right amount of GPU acceleration to the CPU instances discussed in this blog post:A Complete Guide to AI Accelerators for Deep Learning Inference—GPUs, AWS Inferentia, and Amazon Elastic Inference

Optimize costs through increased utilization

  1. Optimize your training code to take full advantage of P3, G4, and G5 instances by using Tensor CoresEnable mixed precision training.Each deep learning framework does this differently, and you must refer to the documentation for your specific framework.
  2. Use reduced precision (INT8) inference on G4 and G5 instance types to improve performance. NVIDIA's TensorRT library provides an API to convert single-precision models to INT8, andprovide examplesin their documentation.

Downloading your favorite deep learning framework is easy, right? onlypip install XXXorconda install XXXordocker pull XXXAre you all ready? Not really, the frameworks you install from upstream repositories are usually not optimized for the target hardware they will run on. These frameworks are designed to support a wide variety of different CPU and GPU types, so they will support the lowest common denominator of feature and performance optimizations, which can result in significant performance degradation for AWS GPU instances.

For this reason, I strongly recommend usingAWS Deep Learning AMIsorAWS Deep Learning Containers (DLC)instead. They are qualified and tested by AWS on all Amazon EC2 GPU instances, and they include AWS optimizations for networking, storage access, and the latest NVIDIA and Intel drivers and libraries. Deep learning frameworks have upstream and downstream dependencies on higher-level schedulers and coordinators, and lower-level infrastructure services. By using an AWS AMI and AWS DLC, you know it has been tested end-to-end and guaranteed to give you the best possible performance.

High-performance computing (HPC) is another scientific field that relies on GPUs to accelerate computing for simulations, data processing, and visualization. While deep learning training can be performed on lower precision algorithms ranging from FP32 (single precision) to FP16 (half precision) and variants like Bfloat16 and TF32, HPC applications require high precision algorithms up to FP64 (double precision). NVIDIA A100, V100, and K80 GPUs support FP64 precision and are available on P4, P3, and P2 instances respectively.

In today's "I put it together because I can't find it anymore"Contribute and I introduce you to the list of GPU features on AWS. Before launching a GPU instance, I often wonder how much memory is on a particular GPU, or whether the GPU supports a particular precision type, or whether the instance has an Intel, AMD, or Graviton CPU, etc. To avoid wading through various web pages and NVIDIA white papers, I've painstakingly pulled all the information into one table. You can use the image below or go directly to the markdown table embedded at the end of the post and hosted on GitHub, your choice. enjoy!

Choosing the Right GPU for Deep Learning on AWS (37)

Do you prefer to consume content in graphic format, I have you covered too! The image below shows all GPU instance types and sizes on AWS. There isn't enough room for all the features, and I still recommend spreadsheets for that.

Choosing the Right GPU for Deep Learning on AWS (38)
(Video) How to Choose an NVIDIA GPU for Deep Learning in 2021: Quadro, Ampere, GeForce Compared

Thanks for reading. If you found this article interesting, please consider giving it a round of applause and following me on the media. Also check out my other blog postsModerateOr follow me on Twitter (@shshnkp),LinkedInor leave a comment below. Want me to write about a specific machine learning topic? I would love to hear from you!

(Video) Deep Learning on Cloud using Amazon AWS | EC2 GPU Instance


How do I choose a deep learning GPU? ›

The Most Important GPU Specs for Deep Learning Processing Speed
  1. Tensor Cores.
  2. Memory Bandwidth.
  3. L2 Cache / Shared Memory / L1 Cache / Registers.
  4. Practical Ada / Hopper Speed Estimates.
  5. Possible Biases in Estimates.
  6. Sparse Network Training.
  7. Low-precision Computation.
  8. Fan Designs and GPUs Temperature Issues.
Jan 30, 2023

What is the best consumer GPU for deep learning? ›

Using Consumer GPUs for Deep Learning
  • NVIDIA Titan V. The Titan V is a PC GPU that was designed for use by scientists and researchers. ...
  • NVIDIA Titan RTX. The Titan RTX is a PC GPU based on NVIDIA's Turing GPU architecture that is designed for creative and machine learning workloads. ...
  • NVIDIA GeForce RTX 2080 Ti.

Can you use GPU in AWS? ›

AWS and NVIDIA provide solutions across a broad range of areas ranging from delivering maintenance solutions to providing GPU processing power at the Edge with AWS Outposts and AWS Wavelength.

How much GPU is enough for deep learning? ›

Also keep in mind that a single GPU like the NVIDIA RTX 3090 or A5000 can provide significant performance and may be enough for your application. Having 2, 3, or even 4 GPUs in a workstation can provide a surprising amount of compute capability and may be sufficient for even many large problems.

How do I choose a preferred GPU? ›

How To Set the Default GPU for Applications and Games
  1. From Start Icon, type "Graphics Settings". Click the results from System Settings.
  2. Click Desktop App.
  3. Click your application. ...
  4. Once identified, click Options.
  5. Set the application to your preferred GPU.
  6. Click save.

Is RTX 3060 good for deep learning? ›

The RTX 3060 is an excellent graphics card for deep learning. Its efficient core architecture, CUDA cores and impressive memory bandwidth make it well-suited to running complex algorithms and models. Its GDDR6 RAM can easily handle larger datasets, making deep learning applications run smoother and faster.

Which GPU is best for deep learning Google cloud? ›

You can see that Nvidia A100 is the fastest. Nvidia Tesla P4 is the slowest. Nvidia A100 is the most expensive while Tesla P4 is the cheapest. While Tesla P4 has the highest operations per dollar, A100 has the lowest operations per dollar.

Is it worth buying a GPU for deep learning? ›

Dataset Size

Training a model in deep learning requires a large dataset, hence the large computational operations in terms of memory. To compute the data efficiently, a GPU is an optimum choice. The larger the computations, the more the advantage of a GPU over a CPU.

Do you really need a GPU for deep learning? ›

GPU is fit for training the deep learning systems in a long run for very large datasets. CPU can train a deep learning model quite slowly. GPU accelerates the training of the model. Hence, GPU is a better choice to train the Deep Learning Model efficiently and effectively.

Is AWS GPU free? ›

Key benefits are: Completely free, you only need a valid email - no credit card or AWS account required. No Set Up required - enabling you to focus on the data science lesson, not the configuration headaches.

Can we use GPU in AWS free tier? ›

The micro instances do not support GPUs. Thus, you cannot stay within the Free Tier while using GPUs. This user is first on the weekly AWS leaderboard.

How do I choose a GPU for Azure? ›

Go to the virtual machine you want to add the GPU extension to. In Details, select + Add extension. Then select a GPU extension to install. GPU extensions are only available for a virtual machine with a VM size from N-series.

Is RTX 3090 good for deep learning? ›

But also the RTX 3090 can more than double its performance in comparison to float 32 bit calculations. The GPU speed-up compared to a CPU rises here to 167x the speed of a 32 core CPU, making GPU computing not only feasible but mandatory for high performance deep learning tasks.

Why GPU is preferred over CPU for deep learning? ›

Compared to CPUs, GPUs have a far higher number of cores, allowing for more simultaneous computations. Deep neural network training involves millions of calculations; therefore, this parallelism is crucial for speeding up the process.

Do you need a powerful GPU for machine learning? ›

When dealing with machine learning, and especially when dealing with deep learning and neural networks, it is preferable to use a graphics card to handle the processing, rather than the CPU. Even a very basic GPU is going to outperform a CPU when it comes to neural networks.

Does it matter what kind of GPU you get? ›

The more powerful the GPU (sometimes referred to as a graphics card) the more information can be calculated and displayed in a shorter time, and the better your gameplay experience will be overall. In the early days of PCs, the CPU was responsible for translating information into images.

How do I prioritize my GPU over integrated graphics? ›

Right-click on the desktop and select Nvidia Control Panel.
  1. Switch to Manage 3D settings in the left-hand pane.
  2. Switch to the Program Settings tab.
  3. Under Select a program to customize, choose the relevant app.
  4. Under Select the preferred graphics processor for this program, choose the GPU you prefer.
May 6, 2021

Does it matter which brand GPU you get? ›

Regardless of the brand, you buy it from, the actual GPU chip that does all the work is not going to change. An RX 6600 XT is a 6600 XT whether you buy it from Sapphire or Gigabyte. Buying the more expensive version won't magically make it perform better, since it's still the same underlying chip.

What is the GPU benchmark for deep learning 2023? ›

NVIDIA's RTX 4090 is the best GPU for deep learning and AI in 2022 and 2023. It has exceptional performance and features that make it perfect for powering the latest generation of neural networks.

What is the most powerful GPU for AI? ›


In 2022 and 2023, NVIDIA's RTX 4090 will be the finest GPU for deep learning and AI. It powers the latest neural networks due to their greater functionality and performance.

Is 2060 better than 3060 for deep learning? ›

RTX 2060 has the first generation of Tensor Cores where as the RTX 3060 has second generation of tensor cores which are comparatively more powerful. The RTX 2060 is the weakest RTX card till date. Clearly, the RTX 3060 is a lot better than any RTX 2000 series graphics cards in terms of deep learning.

Is CPU faster than GPU for deep learning? ›

CPUs are less efficient than GPUs for deep learning because they process tasks in order one at a time. As more data points are used for input and forecasting, it becomes more difficult for a CPU to manage all of the associated tasks.

How faster is GPU compared to CPU for deep learning? ›

GPUs Are Better Suited for Complex Tasks

This is because GPUs can process multiple tasks simultaneously, allowing them to train deep learning algorithms much faster than CPUs. GPUs are also more energy-efficient than CPUs, making them a great choice for training deep learning algorithms.

Which GPU has the highest memory in the cloud? ›

A100 80GB has the largest GPU memory on the current market, while A6000 (48GB) and 3090 (24GB) match their Turing generation predecessor RTX 8000 and Titan RTX .

Is One GPU enough for deep learning? ›

It's common for a deep learning workstation to include a single GPU for training and a huge amount of GPU power for inference. However, training models for image recognition is impractical with a single GPU. Unless you have a GPU with a high amount of memory, it's simply too slow.

Do I need GPU for TensorFlow? ›

TensorFlow supports running computations on a variety of types of devices, including CPU and GPU.

How much faster is TensorFlow on GPU? ›

GPU-Accelerated TensorFlow

TensorFlow runs up to 50% faster on the latest Pascal GPUs and scales well across GPUs. Now you can train the models in hours instead of days.

What happens after 12 months of AWS free tier? ›

When your 12-month free usage term expires, or if your application use exceeds the tiers, you simply pay standard, pay-as-you-go service rates. Always Free – These free tier offers do not expire and are available to all AWS customers.

How do I avoid paying AWS? ›

Delete, stop, or terminate any resources that you don't want to be billed for. After identifying the AWS resources that are incurring charges, you can stop the billing by deleting, stopping, or terminating the resources.

Is Amazon AWS free forever? ›

Always Free: These free tier offers do not automatically expire at the end of your 12 month AWS Free Tier term, but are available to both existing and new AWS customers indefinitely.

How much does a GPU cost on AWS? ›

Product Details
Instance SizeOn-Demand Price/hr*
Single GPU VMsg4dn.2xlarge$0.752
12 more rows

What is the difference between CPU and GPU in AWS? ›

The main difference between a CPU and GPU lies in their functions. A server cannot run without a CPU. The CPU handles all the tasks required for all software on the server to run correctly. A GPU, on the other hand, supports the CPU to perform concurrent calculations.

How do I know if my AWS instance has GPU? ›

2 Answers
  1. Example: nvidia-smi --id=0 --loop=5 --query --display=UTILIZATION. --id=0 the number of the GPU. Use nvidia-smi --list-gpus to get the list of GPUs. --query display GPU or unit information. --loop=5 repeat the query every 5 seconds. -display=UTILIZATION display only utilization.
  2. The output is something like:

How do I choose which GPU to use TensorFlow? ›

TensorFlow GPU strings have an index starting from zero. Therefore, to specify the first GPU, you should write “/device:GPU:0”. Similarly, the second GPU is “/device:GPU:1”.

What is the cheapest GPU in Azure? ›

The cheapest GPU configuration is a Standard_NC6 instance in Azure which has one NVidia Tesla K80 GPU and six CPU cores.

How to choose between AMD and Nvidia? ›

The most basic difference between AMD GPUs and Nvidia GPUs is that Nvidia chips tend to be more powerful, especially at the high-end, while AMD cards offer better value at lower price points and a more friendly user interface.

Which is better deep learning 3090ti or 4090? ›

RT and Tensor Cores

One of the major differences between the RTX 40-series and RTX 3090 Ti are the RT cores and the Tensor cores. The RTX 4090 comes with 128 RT cores as opposed to the 84 RT cores in the RTX 3090 Ti and has 512 Tensor Cores as opposed to the 336 Tensor Cores in the RTX 3090 Ti.

Is RTX 4090 good for AI? ›

The 24 GB memory capacity allows you to process large amounts of data. The GeForce RTX 4090 is primarily designed for gaming and is great for various types of computing solutions: AI, data analysis, and machine learning. The new architecture greatly outperforms the previous generation of NVIDIA GPUs.

Is Intel GPU good for deep learning? ›

GPUs are important for machine learning and deep learning because they are able to simultaneously process multiple pieces of data required for training the models. This makes the process easier and less time-consuming.

How many GPUs needed for AI? ›

Hundreds of GPUs are required to train artificial intelligence models, like large language models. The chips need to be powerful enough to crunch terabytes of data quickly to recognize patterns.

What are the disadvantages of GPU for machine learning? ›

Disadvantages of GPUs compared to CPUs include: Multitasking—GPUs can perform one task at massive scale, but cannot perform general purpose computing tasks. Cost—Individual GPUs are currently much more expensive than CPUs. Specialized large-scale GPU systems can reach costs of hundreds of thousands of dollars.

What is the best GPU for deep neural networks? ›

NVIDIA GeForce RTX 3090 Ti is one of the best GPU for deep learning if you are a data scientist that performs deep learning tasks on your machine. Its incredible performance and features make it ideal for powering the most advanced neural networks than other GPUs.

Which CPU GPU is best for machine learning? ›

Intel Core i9-13900KS

In conclusion, there are several great options when it comes to choosing the best CPU for machine learning. The Intel Core i9-13900KS stands out as the best consumer-grade CPU for deep learning, offering 24 cores, 32 threads, and 20 PCIe express lanes.

What is the difference between deep learning RTX 3060 and 3080? ›

Two 3060s have vastly more memory than a single 3080, but fewer CUDA cores (even combined), and slightly slower memory speed. They also take up an extra slot in case you ever wanted to expand.

Why Nvidia is better than AMD for deep learning? ›

Nvidia vs AMD

You can use AMD GPUs for machine/deep learning, but at the time of writing Nvidia's GPUs have much higher compatibility, and are just generally better integrated into tools like TensorFlow and PyTorch.

How much faster is GPU vs CPU for deep learning? ›

GPU vs CPU Performance in Deep Learning Models

Generally speaking, GPUs are 3X faster than CPUs.

Is RTX 3050 enough for deep learning? ›

The RTX 3050 is a powerful Nvidia GPU that has many applications. Its high performance make it well-suited for deep learning tasks, including training and inference of large machine learning models.

Which GPU is best for deep learning 4090? ›

For most users, NVIDIA RTX 4090, RTX 3090 or NVIDIA A5000 will provide the best bang for their buck. Working with a large batch size allows models to train faster and more accurately, saving time.

Is 3090 better than 4090 stable diffusion? ›

The RTX 4090 is now 72% faster than the 3090 Ti without xformers, and a whopping 134% faster with xformers. The 4080 also beats the 3090 Ti by 55%/18% with/without xformers. The 4070 Ti interestingly was 22% slower than the 3090 Ti without xformers, but 20% faster with xformers.

Is it worth upgrading from RTX 3060 to RTX 3080? ›

At 1080p, both the RTX 3060 Ti and RTX 3080 should serve you very well. However, in most cases, it's unlikely that the 3080 will be worth its extra cost for 1080p gaming despite its better performance. This is especially true if you don't have a high-end, current-gen CPU.

Is a 3080 better then a 3060 TI? ›

The RTX 3080 is definitely a better GPU compared to the RTX 3060Ti in terms of performance. However, the 3060Ti is more power-efficient than the RTX 3080. This means that for notebooks, the 3060Ti is a better choice for extending battery life.

Should I upgrade from RTX 3060 to RTX 3080? ›

More powerful GPU: The RTX 3080 has a more powerful GPU than the RTX 3060 Ti, with 8704 CUDA cores compared to the 3060 Ti's 4864 CUDA cores. This translates to significantly better performance in demanding games and applications.


1. [#15] Cómo elegir la GPU correcta para Deep Learning en AWS
2. AWS re:Invent 2022 - Deep learning on AWS with NVIDIA: From training to deployment (PRT219)
(AWS Events)
3. Deep Learning on AWS made simple - AWS Online Tech Talks
(AWS Online Tech Talks)
4. Build your own Deep learning Machine - What you need to know
(The A.I. Hacker - Michael Phi)
5. L3/1 Setting up a GPU instance on AWS
(Alex Smola)
6. NVIDIA GPU Cloud with AWS, Step by Step
(NVIDIA Developer)


Top Articles
Latest Posts
Article information

Author: Jeremiah Abshire

Last Updated: 18/08/2023

Views: 5573

Rating: 4.3 / 5 (54 voted)

Reviews: 93% of readers found this page helpful

Author information

Name: Jeremiah Abshire

Birthday: 1993-09-14

Address: Apt. 425 92748 Jannie Centers, Port Nikitaville, VT 82110

Phone: +8096210939894

Job: Lead Healthcare Manager

Hobby: Watching movies, Watching movies, Knapping, LARPing, Coffee roasting, Lacemaking, Gaming

Introduction: My name is Jeremiah Abshire, I am a outstanding, kind, clever, hilarious, curious, hilarious, outstanding person who loves writing and wants to share my knowledge and understanding with you.