Choosing the Right GPU for Deep Learning on AWS (2023)

Just a decade ago, if you wanted to use a GPU to accelerate your data processing or scientific simulation code, you either had to contact a PC gamer or a friendly supercomputing center near you. You can now log into your AWS console and choose from a range of GPU-based Amazon EC2 instances.

You ask which GPUs can you access on AWS? You can boot GPUs with different GPU memory sizes (8 GB, 16 GB, 24 GB, 32 GB, 40 GB), NVIDIA GPU generations (Ampere, Turing, Volta, Maxwell, Kepler) with different capabilities (FP64, FP32, FP16, INT8, Sparsity, TensorCores, NVLink), each instance has a different number of GPUs (1, 2, 4, 8, 16), paired with a different CPU (Intel, AMD, Graviton2). You can also choose instances with different vCPUs (core threads), system memory, and network bandwidth, and add a range of storage options (object storage, network file system, block storage, etc.) - in short, you have options.

My goal with this blog post is to guide you on how to choose the right GPU instance for your deep learning project on AWS. I'll discuss the key features and benefits of the various EC2 GPU instances, and the workloads that are best suited for each instance type and size. If you're new to AWS, new to GPUs, or new to deep learning, I hope you found the information you need to make the right choice for your project.

Topics covered in this blog post:

1. Top Tips for Busy Data Scientists/Machine Learning Practitioners
2. why you should choose the rightGPU instancenot just rightsgraphics card
3. Learn more about GPU instance types: P4, P3, G5 (G5g), G4, P2, and G3
4. Other Machine Learning Accelerators and Instances on AWS
5. Cost optimization tips when using GPU instances for ML
6. What software and frameworks are used on AWS?
7. Which GPUs should be considered for HPC use cases?
8. A complete and exhaustive spreadsheet of all AWS GPU instances and their capabilities

hurry? Just want the final recommendation without digging in? I got you covered. Here are 5 GPU instance recommendations that should work for most deep learning use cases. However, I do recommend that you come back and review the rest of this article so you can make a more informed decision.

1. Highest performing multi-GPU instance on AWS

Example: p4d.24xlarge
When to use it:When you need all the performance you can get. Use it for distributed training of large models and datasets.
what you get:8×NVIDIA A100GPU with 40 GB of GPU memory per GPU. Based on the latest NVIDIA Ampere architecture. Includes third-generation NVLink for fast multi-GPU training.

2. Highest performing single GPU instance on AWS:

Example: p3. 2xlarge
When to use it:When you want the highest performance single GPU and 16 GB of GPU memory is fine.
what you get:1NVIDIA V100GPU with 16 GB of GPU memory. Based on the old NVIDIA Volta architecture. The best performing single GPU is still the NVIDIA A100 on the P4 instance, but you only get 8 NVIDIA A100 GPUs on the P4. This GPU has a slight performance advantage over the NVIDIA A10G on the G5 instance discussed next, but the G5 is more cost-effective and has more GPU memory.

3.Best performance/cost, single GPU instance on AWS

Example: g5.xlarge
When to use it:When you want high performance, more GPU memory at a lower cost than P3 instances
what you get:1NVIDIA A10GGPU with 24 GB of GPU memory, based on the latest Ampere architecture. NVIDIA A10G could be seen as A100's low-power cousinp4d.24xlargeSo when you need more compute, you can easily migrate and scale. consider a larger sizeg5.(2/4/8/16)xlargeFor the same single GPU with more vCPUs and higher system memory, if you have more preprocessing or postprocessing steps.

4. Best performance/cost, multi-GPU instances on AWS:

Example: p3.(8/16) extra large
When to use it:Cost-effective multi-GPU model development and training.
what you get:p3. 8xlargeThere are 4NVIDIA V100graphics processor andp3. 16xlargeThere are 8 NVIDIA V100 GPUs with 16 GB of GPU memory on each GPU, based on the older NVIDIA Volta architecture. For larger models, datasets, and faster performance, consider P4 instances.

5. High Performance GPU Instances on AWS

Example: g4dn.xlarge
When to use it:The performance of model development and training is lower than other options, and the cost is lower. Cost-effective model inference deployment.
what you get:1Nvidia T4GPU with 16 GB of GPU memory. Based on the previous generation NVIDIA Turing architecture. considerg4dn.(2/4/8/16)xlargeIf you have more pre- or post-processing, you can get more vCPUs and higher system memory.

With it, you should have enough information to start your project. If you're still eager to learn more, let's dive into each instance type, GPU type, what they can do with AWS, and discuss when and why you should consider each of them.

Or why you should look at the whole system and not just the type of GPU

GPUs are the workhorse of deep learning systems, but the best deep learning systems are not just GPUs. You must choose the right computing power (CPU, GPU), storage, network bandwidth, and optimized software to maximize the use of all available resources.

Some deep learning models require higher system memory or a more powerful CPU for data preprocessing, while other models may run well with fewer CPU cores and lower system memory. This is why you see many Amazon EC2 GPU instance options, some with the same GPU type but different CPU, storage, and networking options. If you are new to AWS or new to deep learning on AWS, making this choice can be overwhelming.

Let's start with advanced EC2 GPU instance nomenclature on AWS. There are two GPU instance families - the P-series and G-series of EC2 instances, and the table below shows the different instance generations and instance sizes.

in historyPInstance types represent GPUs better suited for High Performance Computing (HPC) workloads, featuring higher performance (higher wattage, more cuda cores) and support for double precision (FP64) used in scientific computing.GInstance type GPUs are better suited for graphics and rendering, characterized by a lack of double precision and lower price/performance (lower wattage, lower number of cuda cores).

All of this is starting to change as the volume of machine learning workloads on GPUs has grown rapidly in recent years. Today, a new generationPandGInstance types are suitable for machine learning.PThe instance type is still recommended for HPC workloads and demanding machine learning training workloads, I recommendGAn instance type for machine learning inference deployment and less compute-intensive training. All of this will become clearer in the next section when we discuss specific GPU instance types.

Each instance size has a specific number of vCPUs, GPU memory, system memory, number of GPUs per instance, and network bandwidth. Numbers next to letters (P3， G5)Generated on behalf of the instance. The higher the number, the newer the instance type. Each instance generation can have GPUs of different architectures, the timeline image below shows NVIDIA GPU architecture generations, GPU types and corresponding EC2 instance generations.

Let us now look at these examples one by one by family, generation and size in the order listed below.

P4 instanceprovide accessNVIDIA A100 GPUBased on the NVIDIA Ampere architecture. it only comes inone size— Multi-GPU per instance with 8 A100 GPUs, 40 GB of GPU memory each, 96 vCPUs, and 400 Gbps of network bandwidthrecord training performance.

Overview of P4 instance features:

• GPU generation: NVIDIA Ampere
• Supported precision types: FP64, FP32, FP16, INT8, BF16, TF32, Tensor Cores third generation (mixed precision)
• video memory: 40 GB per GPU
• GPU interconnect: NVLink High Bandwidth Interconnect, Generation 3

What's new with the NVIDIA Ampere-based NVIDIA A100 GPU on P4 instances?

Each new generation of GPUs is faster than the last, and this is no exception. NVIDIA A100 is much faster than NVIDIA V100 (found on the P3 instance discussed later), but also includes newer precision types suitable for deep learning, notably BF16 and TF32.

Deep learning training is usually done in single precision or FP32. The choice of the FP32 IEEE standard format predates deep learning, so hardware and chip manufacturers have started supporting newer precision types that are more suitable for deep learning. This is a perfect example of hardware evolving to meet application needs and developers having to change applications to work on existing hardware.

The NVIDIA A100 includes special cores for deep learning called Tensor Cores to run mixed-precision training, which were first introduced in the Volta architecture. Instead of training the model in single precision (FP32), your deep learning framework can use Tensor Cores to perform matrix multiplication in half precision (FP16) and accumulation in single precision (FP32). This usually requires updating your training scripts, but can lead to higher training performance. Each framework handles this differently, so refer to your framework's official guide (tensor flow,torchandMXNet) for use with mixed precision.

NVIDIA A100 GPUs support two new precision formats - BF16 and TensorFloat-32 (TF32). The advantage of TF32 is that TF32 Tensor Cores on NVIDIA A100 can read and consume FP32 data from deep learning frameworks and produce standard FP32 output, but internally it uses reduced internal precision. This means that frameworks such as TensorFlow and PyTorch can support TF32 out of the box, unlike mixed precision training, which typically requires code changes to the training script. BF16 is an alternative to the IEEE FP16 standard with a higher dynamic range and better suited for handling gradients without loss of accuracy. TensorFlow hasSupport BF16For a while, you can now take advantage of BF16 precision when using an NVIDIA A100 GPUp4d.24xlargeinstance.

P4 instances have only one size:p4d.24xlarge. Let's take a closer look.

p4d.24xlarge: The fastest GPU instance in the cloud

If you need the absolute fastest training GPU instance in the cloud, look no furtherp4d.24xlarge。This title was previously given byp3dn. 24xlarge, which has 8 NVIDIA V100 GPUs based on the Volta architecture.

You can use 8 NVIDIA A100 GPUs with 40 GB of GPU memory, withThe third generation of NVLinkThis theoretically doubles the inter-GPU bandwidth compared to the second-generation NVLink on NVIDIA V100 available on the P3 instance type we discuss in the next section. which makesp4d.24xlargeInstance types are well suited for distributed data-parallel training and model-parallel training of large models that do not fit on a single GPU. The instance also gives you access to 96 vCPUs, 1152 GB of system memory (up to the maximum on an EC2 GPU instance), and 400 Gbps of network bandwidth (up to the maximum on an EC2 GPU instance), which is important for near-linear scaling of massively distributed training jobs.

runningnvidia-smiIn this instance, you can see that the GPU memory is 40 GB. This is the maximum GPU memory per GPU you can find on AWS today. This is an instance to consider if your models are large or if you are processing 3D images or other large data batches. runningnvidia-smi topology matrixYou will see that NVLink is used for communication between GPUs. Compared to PCIe, NVlink provides higher inter-GPU bandwidth, which means multi-GPU and distributed training jobs will run faster.

G5 instances are interesting because there are two types of NVIDIA GPUs under this instance type. This differs from all other instance types which have a 1:1 relationship between the EC2 instance type and the GPU architecture type.

Each of them has different instance sizes, including single-GPU instances and multi-GPU instances.

First let's look at the G5 instance types, specifically theg5.xlargeThe instance sizes I discussed in the key takeaway/recommendation list at the beginning.

List of G5 instance features:

• GPU generation: NVIDIA Ampere
• Supported precision types:Supported precision types: FP64, FP32, FP16, INT8, BF16, TF32, Tensor Cores third generation (mixed precision)
• video memory: 24 GB
• GPU interconnect: PCIe

What can G5 bring you?

GPU instance:g5.(2/4/8/16)xlargeBest price/performance for single GPU instances on AWS. start fromg5.xlargeServes as your single-GPU model development, prototyping, and training instance. You can increase the size tog5.(2/4/8/16).xlargeGet more vCPUs and system memory to better handle CPU-dependent data pre- and post-processing. You can access single-GPU and multi-GPU instance sizes (4 GPU, 8 GPU). Single GPU optiong5.(2/4/8/16)xlargeNVIDIA A10G provides the best performance/cost profile for training and inference deployment.

If you look at the outputnvidia-smiforg5.xlargeFor example, you'll see that the thermal design power (TDP), the maximum power the GPU can draw, is 300W. Compare this with the output ofnvidia-smiAs shown in the P4 part of the figure above, the TDP is 400W. This makes the NVIDIA A10G in the G5 instance a low-power cousin of the NVIDIA A100 on the P4 instance type. Since it's also based on the same NVIDIA Ampere architecture, that means it includes all the features supported by the P4 instance type.

This makes G5 instances ideal for single-GPU training, if your model and data volume grows and you need to do distributed training, or if you want to run multiple parallel training experiments on faster GPUs, you can split your training workload The load is migrated to P4.

Although you have access to multi-GPU instance sizes, I would not recommend using them for multi-GPU distributed training as there is no NVIDIA high-bandwidth NVLink GPU interconnect and communication will fall back to the significantly slower PCIe. The multi-GPU option on the G5 is designed to host multiple models per GPU for inference deployment use cases.

List of G5g instance features:

• GPU generation: NVIDIA Turing
• Supported precision types:Supported precision types: FP32, FP16, Tensor Cores (mixed-precision), INT8
• video memory： 16 GB
• GPU interconnect: PCIe

What can G5g bring to you?

Unlike G5 instances, G5g instances offer NVIDIA T4G GPUs based on the older NVIDIA Turing architecture. A close cousin of the NVIDIA T4G GPU is the NVIDIA T4 GPU available on Amazon EC2 G4 instances, which I discuss in the next section. Interestingly, the main difference between the G5g instances and the G4 instances is the choice of CPU.

G5g instances offer ARM-based AWS Graviton2 CPUs, and
G4 instances offer x86-based Intel Xeon Scalable CPUs.
The performance profiles of the GPUs (T4 and T4g) are very similar

Our choice between the two should come down to your preferred CPU architecture. My personal preference for machine learning today is G4 instances over G5g instances because more open source frameworks are designed to run on Intel CPUs than ARM-based CPUs.

P3 instanceprovide accessNVIDIA V100 GPUBased on the NVIDIA Volta architecture, you can launch a single GPU per instance or multiple GPUs per instance (4 GPUs, 8 GPUs). Single GPU instancep3. 2xlargeCan be the daily driver of your deep learning training. and the most capable instancep3dn. 24xlargeGives you access to 8 x V100 with 32 GB GPU memory, 96 vCPUs, 100 Gbps network throughput, ideal for distributed training.

Overview of P3 instance features:

• GPU generation: Nvidia Walter
• Supported precision types: FP64, FP32, FP16, Tensor Cores (mixed-precision)
• video memory：16 GBp3.2xlarge、p3.8xlarge、p3.16xlarge, on 32 GBp3dn. 24xlarge
• GPU interconnect: Second-generation NVLink high-bandwidth interconnect

NVIDIA V100 also includes Tensor Cores to run mixed-precision training, but not the TF32 and BF16 precision types introduced in NVIDIA A100 available on P4 instances. However, P3 instances are available in 4 different sizes, from a single GPU instance size up to 8 GPU instance sizes, making them ideal for flexible training workloads. Let's look at each instance size belowp3. 2xlarge,p3. 8xlarge,p3. 16xlargeandp3dn.24xlarge。

p3.2xlarge: best GPU instance for single GPU training

If you need a single GPU and performance is a top priority, this should be the instance of choice for most of your deep learning training work. G5 instances are more cost-effective with slightly lower performance than P3. andp3. 2xlargeYou get access to an NVIDIA V100 GPU with 16 GB of GPU memory, 8 vCPUs, 61 GB of system memory, and up to 10 Gbps of network bandwidth. At the time of writing, the V100 is the fastest GPU in the cloud and supports Tensor Cores, which can further improve performance if your scripts can take advantage of mixed-precision training.

If you start an Amazon EC2p3. 2xlargeinstance and runnvidia-smiThe command can see that the GPU on the instance is V100-SXM2 version, which supports NVLink (we will discuss it in the next section). Under Memory-Usage, you'll see that it has 16 GB of GPU memory. If you need more than 16 GB of GPU memory for large models or large data, then you should considerp3dn. 24xlarge(see below for more details).

p3.8xlarge and p3.16xlarge: Ideal GPU instances for small-scale multi-GPU training and running parallel experiments

If you need more GPUs for experimentation, more vCPUs for data preprocessing and data augmentation, or higher network bandwidth, considerp3. 8xlarge(with 4 GPUs) andp3. 16xlarge(with 8 GPUs). Each GPU is an NVIDIA V100 with 16 GB of memory. They also include an NVLink interconnect for high-bandwidth communication between GPUs, which will come in handy during multi-GPU training. andp3. 8xlargeYou have access to 32 vCPUs and 244 GB of system memory, andp3. 16xlargeYou have access to 64 vCPUs and 488 GB of system memory. This instance is ideal for several use cases:

Multi-GPU training job: If you are new to multi-GPU training, 4 GPUsp3. 8xlargeor 8 GPUsp3. 16xlargeCan give you a nice speedup. You can also use this instance to prepare training scripts for larger multi-node training jobs, which usually requires you to modify the training scripts withLibraries like Horovod,tf.distribute.Strategyortorch.distributed.See my step-by-step guide for distributed training with Horovod:

blog post:A Quick Guide to Distributed Training with TensorFlow and Horovod on Amazon SageMaker

parallel experiment: Multi-GPU instances also come in handy when you have to run variations of your model architecture and hyperparameters in parallel for faster experimentation. andp3. 16xlargeYou can train up to 8 variants of the model. Unlike multi-GPU training jobs, you can increase productivity during the model exploration phase because each GPU runs training independently and does not block the use of other GPUs.

p3dn.24xlarge：High performance and cost-effective training

This instance previously held the title of fastest GPU instance in the cloud and now belongs top4d.24xlarge.that won'tp3dn. 24xlargeIs a listless person. It's still one of the fastest instance types you'll find on the cloud today, and it's more cost-effective than P4 instances. You can use 8 NVIDIA V100 GPUs, but withp3. 16xlargeWith 16 GB of GPU memory, on the GPUp3dn. 24xlargeHas 32 GB of GPU memory. This means you can fit larger models and train with larger batches. This instance gives you access to 96 vCPUs, 768 GB of system memory, and 100 Gbps of network bandwidth, which is important for near-linear scaling of massively distributed training jobs.

runningnvidia-smiIn this instance, you can see that the GPU memory is 32 GB. The only instances with more GPU memory than this arep4d.24xlargeSame as A100 GPU with 40 GB GPU memory. This is an instance to consider if your models are large or if you are processing 3D images or other large data batches. runningnvidia-smi topo — matrixYou will see that NVLink is used for communication between GPUs. Compared to PCIe, NVlink provides higher inter-GPU bandwidth, which means multi-GPU and distributed training jobs will run faster.

G4 instanceProvides access to NVIDIA T4 GPUs based on the NVIDIA Turing architecture. You can launch one GPU per instance, or multiple GPUs per instance (4 GPUs, 8 GPUs). In the timeline diagram below, you can see that directly below the G4 instances are the G5g instances, both based on GPUs with NVIDIA Turing architecture. We have discussed the G5g instance type in the previous section, the GPUs in the G4 (NVIDIA T4) and G5g (NVIDIA T4G) are very similar in performance. Your choice will come down to the choice of CPU type on these instances.

G5g instances offer ARM-based AWS Graviton2 CPUs, and
G4 instances offer x86-based Intel Xeon Scalable CPUs.
The performance profiles of the GPUs (T4 and T4g) are very similar

In the GPU timeline graph, you can see that the NVIDIA Turing architecture follows the NVIDIA Volta architecture and introduces several new machine learning features such as next-generation Tensor Cores and integer precision support, which make it cost-effective for inference Ideal for deployment and graphics.

List of G4 instance features:

• GPU generation: NVIDIA Turing
• Supported precision types: FP64, FP32, FP16, Tensor Cores (mixed-precision), INT8, INT4, INT1
• video memory： 16 GB
• GPU interconnect: PCIe

What's new with NVIDIA T4 GPUs on G4 instances?

NVIDIA Turing was the first to introduce support for the integer precision (INT8) data type, which can significantly accelerate inference throughput. During training, model weights and gradients are usually stored in single precision (FP32). It turns out that to run predictions on a trained model, you don't actually need full precision, you can get away with reduced precision calculations at half precision (FP16) or 8-bit integer precision (INT8). Doing so improves throughput without sacrificing too much accuracy. Accuracy will drop somewhat, depending on various factors specific to your model and training. Overall, you get the best inference performance/cost with G4 instances compared to other GPU instances.NVIDIA's Support MatrixShows which neural network layers and GPU types support INT8 and other inference precisions.

The NVIDIA T4 (and NVIDIA T4G) are the lowest powered GPUs on any EC2 instance on AWS. runningnvidia-smiIn this example, you can seeg4dn.xlargeFeatures NVIDIA T4 GPU with 16 GB GPU memory. You'll also notice that the power cap is 70W compared to 300W on the NVIDIA A10G.

The following instance sizes all give you access to a single NVIDIA T4 GPU and an increasing number of vCPUs, system memory, storage, and network bandwidth:g4dn.xlarge(4 vCPUs, 16 GB system memory),g4dn.2xlarge(8 vCPUs, 32 GB system memory),g4dn.4xlarge(16 vCPUs, 64 GB system memory),g4dn.8xlarge(32 vCPUs, 128 GB system memory),g4dn.16xlarge(64 vCPUs, 256 GB system memory). You can findProduct G4 example pageUnder the "Product Details" section.

The G4 instance size also includes two multi-GPU configurations:g4dn.12xlargeThere are 4 GPUs and g4dn.metal has 8 GPUs. However, if your use case is multi-GPU or multi-node/distributed training, you should consider using P3 instances. runningnvidia-smi topology --matrixon multiple GPUsg4dn.12xlargeFor example, you can see that the GPUs are not connected via the high-bandwidth NVLink GPU interconnect. P3 multi-GPU instances include a high-bandwidth NVLink interconnect that accelerates multi-GPU training.

(Video) Deep Learning on the Cloud - GPU TO LEARN FASTER

P2 instanceAllows you to use NVIDIA K80 GPUs based on the NVIDIA Kepler architecture. The Kepler architecture is several generations old (Kepler -> Maxwell -> Pascal -> Volta -> Turing), so they are not the fastest GPUs. They do have some specific features, such as full-precision (FP64) support, which make them attractive and cost-effective for high-performance computing (HPC) workloads that rely on extra precision. P2 instances come in 3 different sizes: p2.xlarge (1 GPU), p2.8xlarge (8 GPUs), p2.16xlarge (16 GPUs).

The NVIDIA K80 is an interesting GPU. A single NVIDIA K80 is actually two GPUs on one physical board, which NVIDIA calls a dual-GPU design. This means that when you start an instancep2.xlarge, you only get one of these two GPUs on the physical K80 board. Likewise, when you start ap2.8xlargeYou can access eight GPUs on four K80 GPUs, andp2.16xlargeYou can access sixteen GPUs on eight K80 GPUs. runningnvidia-smiin ap2.xlargeWhat you see is one of the two GPUs on an NVIDIA K80 board with 12 GB of GPU memory

Overview of P2 instance features:

• GPU generation: NVIDIA Capper
• Supported precision types: FP64, FP32
• Video memory:12GB
• GPU interconnect: PCIe

So, should I even be using P2 instances for deep learning?

No, there are better options discussed above. Prior to the introduction of Amazon EC2 G4 and G5 instances, P2 instances were the recommended instance type for cost-effective deep learning training. Since the introduction of G4 instances, I recommend G4 as the preferred cost-effective training and prototyping GPU instance for deep learning training. P2 is still cost-effective for HPC workloads in scientific computing, but you miss out on some new features, such as support for mixed-precision training (Tensor Cores) and reduced-precision inference, which have become the standard for new generations.

if you runnvidia-smiexistp2.16xlargeGPU instance, since the NVIDIA K80 has a dual GPU design, you will see 16 GPUs that are part of 8 NVIDIA K80 GPUs. This is the maximum number of GPUs you can get on a single instance on AWS. if you runnvidia-smi topology --matrix, you'll see that all inter-GPU communication happens over PCIe, unlike P3 multi-GPU instances which use faster NVLink.

G3 instanceAllows you to use NVIDIA M60 GPUs based on the NVIDIA Maxwell architecture. NVIDIA calls the M60 GPU a virtual workstation and positions it as professional graphics. However, for more powerful and cost-effective options for deep learning using P3, G4, G5, G5g instances, G3 is not a recommended option for deep learning. I'm including it here just for some history and completeness.

G3 instance features at a glance:

• GPU generation: NVIDIA Maxwell
• Supported precision types:FP32
• video memory：8GB
• GPU interconnect: PCIe

Should you consider G3 instances for deep learning?

Single GPU G3 instances are cost-effective for development, testing, and prototyping prior to the availability of Amazon EC2 G4 instances. Although the Maxwell architecture is newer than the NVIDIA K80 Kepler architecture on P2 instances, you should still consider P2 instances for deep learning ahead of G3. Your selection order should be P3 > G4 > P2 > G3.

G3 instances come in 4 sizes,g3s.xlargeandg3.4xlarge(2 GPUs, different system configurations)g3.8xlarge(2 GPUs) andg3.16xlarge(4 GPUs). runningnvidia-smiin ag3s.xlargeYou can see that this instance gives you access to an NVIDIA M60 GPU with 8 GB of GPU memory.

NVIDIA GPUs are undoubtedly a staple of deep learning, but there are other instance options and accelerators on AWS that may be better choices for your training and inference workloads.

• CPU: for training traditional ML models, prototyping, and inference deployment
• DL1 instance based on Intel Habana-Gaudi: Uses 8 x Gaudi accelerators, which you can use as an alternative to P3dn and P4d GPU instances for training
• Amazon EC2 Trn1 instance: Use up to 16 AWS Trainium chips, which you can use as an alternative to P3dn, P4d, and DL1 instances for training
• AWS Elastic Inference: Save money on inference workloads by leveraging EI to add just the right amount of GPU acceleration to your CPU instancesdiscussed in this blog post
• Amazon EC2 Inf1 instances: Up to 16 AWS Inferentia chips with 4 Neuron cores on each chip, a powerful and cost-effective inference deployment option atMore details on this blog post.

For a detailed discussion of deployment options for inference, see this blog post on choosing the right AI accelerator for inference:

Blog post:
A Complete Guide to AI Accelerators for Deep Learning Inference—GPUs, AWS Inferentia, and Amazon Elastic Inference

You have several different options for optimizing the cost of training and inference workloads.

Spot instance

Spot Instance pricing makes high-performance GPUs more affordable and allows you to access spare Amazon EC2 compute capacity at a significant discount compared to On-Demand rates. For an up-to-date list of prices by instance and region, visitSpot Instance Advisor.In some cases you can save over 90% on training costs, but your instances can be preempted and terminated with 2 minutes notice. Your training script must implement frequent checkpoints and be able to resume training when Spot capacity is restored.

Amazon SageMaker Managed Training

During the development phase, most of your time is spent prototyping, tweaking code, and trying out different options in your favorite editor or IDE (VIM, obviously) - all without requiring a GPU. You can save costs by simply separating your development and training resources, andAmazon SageMakerwill let you do this easily. Using the Amazon SageMaker Python SDK, you can test scripts locally on your laptop, desktop, EC2 instance, or SageMaker notebook instance.

When you're ready to train, specify the type of GPU instance you want to train on, and SageMaker will provision the instance, copy the dataset to the instance, train your model, copy the results back to Amazon S3, and tear down the instance. You only pay for the exact duration of your training. Amazon SageMaker also supports managed Spot training for added convenience and cost savings.

Here is my guide:A Quick Guide to Using Spot Instances with Amazon SageMaker

Use only as many GPUs as you need for Amazon Elastic Inference

Save money on inference workloads by leveraging EI to add just the right amount of GPU acceleration to the CPU instances discussed in this blog post:A Complete Guide to AI Accelerators for Deep Learning Inference—GPUs, AWS Inferentia, and Amazon Elastic Inference

Optimize costs through increased utilization

1. Optimize your training code to take full advantage of P3, G4, and G5 instances by using Tensor CoresEnable mixed precision training.Each deep learning framework does this differently, and you must refer to the documentation for your specific framework.
2. Use reduced precision (INT8) inference on G4 and G5 instance types to improve performance. NVIDIA's TensorRT library provides an API to convert single-precision models to INT8, andprovide examplesin their documentation.

Downloading your favorite deep learning framework is easy, right? onlypip install XXXorconda install XXXordocker pull XXXAre you all ready? Not really, the frameworks you install from upstream repositories are usually not optimized for the target hardware they will run on. These frameworks are designed to support a wide variety of different CPU and GPU types, so they will support the lowest common denominator of feature and performance optimizations, which can result in significant performance degradation for AWS GPU instances.

For this reason, I strongly recommend usingAWS Deep Learning AMIsorAWS Deep Learning Containers (DLC)instead. They are qualified and tested by AWS on all Amazon EC2 GPU instances, and they include AWS optimizations for networking, storage access, and the latest NVIDIA and Intel drivers and libraries. Deep learning frameworks have upstream and downstream dependencies on higher-level schedulers and coordinators, and lower-level infrastructure services. By using an AWS AMI and AWS DLC, you know it has been tested end-to-end and guaranteed to give you the best possible performance.

High-performance computing (HPC) is another scientific field that relies on GPUs to accelerate computing for simulations, data processing, and visualization. While deep learning training can be performed on lower precision algorithms ranging from FP32 (single precision) to FP16 (half precision) and variants like Bfloat16 and TF32, HPC applications require high precision algorithms up to FP64 (double precision). NVIDIA A100, V100, and K80 GPUs support FP64 precision and are available on P4, P3, and P2 instances respectively.

In today's "I put it together because I can't find it anymore"Contribute and I introduce you to the list of GPU features on AWS. Before launching a GPU instance, I often wonder how much memory is on a particular GPU, or whether the GPU supports a particular precision type, or whether the instance has an Intel, AMD, or Graviton CPU, etc. To avoid wading through various web pages and NVIDIA white papers, I've painstakingly pulled all the information into one table. You can use the image below or go directly to the markdown table embedded at the end of the post and hosted on GitHub, your choice. enjoy!

Do you prefer to consume content in graphic format, I have you covered too! The image below shows all GPU instance types and sizes on AWS. There isn't enough room for all the features, and I still recommend spreadsheets for that.

(Video) How to Choose an NVIDIA GPU for Deep Learning in 2021: Quadro, Ampere, GeForce Compared

Thanks for reading. If you found this article interesting, please consider giving it a round of applause and following me on the media. Also check out my other blog postsModerateOr follow me on Twitter (@shshnkp),LinkedInor leave a comment below. Want me to write about a specific machine learning topic? I would love to hear from you!