Login to Course Computing (2023)

We offer several options for completing the hands-on component of the course, including Amazon Cloud (AWS), Google Cloud (GCP), and Docker images.

Completing this course using cloud computing (AWS or GCP) involves two main parts:(1)launching an instance on the cloud (essentially renting a virtual computer by the hour and turning it on) and(2)Log in to the instance.

Using Docker involves running precompiled images locally on your computer. For specific instructions on how to use each option, choose one of the following options:

calculation options

1. use amazon
2. use GCP
3. use docker

use amazon

In this section: Log into the AWS EC2 console, launch an instance from the course AMI and configure it in the console (choose the instance AMI, instance type, instance details, storage volumes, tags, security groups, and key pairs).

Launch an AWS instance

in the previous sectionIntroduction to AWSWe reviewed the basic concepts of cloud computing as well as some jargon and features unique to AWS. In this section we will learn how to start an instance specifically for this class.

In order to launch your own instance, you will need to use your own personal AWS account, one provided through your institution, or if you are taking the live version of this course, you will be assigned an AWS account using the IAM system. The instructions below are for the live version of this course, but the process is basically the same when using a personal or institutional account.

• Once your EC2 instance is up and running, make a note of its IP address.
• Instructions for logging into this cloud instance (including instructions for Windows systems) can be found below.
• This only happens once we're in the classroom because it costs money to run these servers.
• Each student will launch their own instance from a preconfigured AMI.

In a nutshell, the process of launching an EC2 instance for this course involves the following steps:

1. Log in to AWS using the specific URL and credentials (user/password) provided to you.
2. Search for and select EC2 to access the EC2 console.
3. Make sure you are in the Northern Virginia area
4. Choose Launch Instance.
5. Name and Label: Give your instance a name (for example, FirstName_LastName).
6. Amazon Machine Image: Search My AMIs for "cshl-seqtec-2022" and select.
7. Instance Type: Select "m5.2xlarge", and then select Next.
8. Key Pair (Login): Select an existing key pair (for example, cshl_2022_student).
9. Network Settings: Select an existing security group named "SSH/HTTP/Jupyter".
10. Configure Storage: Make sure you see both volumes.
11. Advanced Details: Select Termination Protection: Enabled.
12. Summary: View and start an instance.
13. Review all instances and wait for your instance to finish launching.
14. Find and select your instance in the console, then click connect to get your public IP address (EC2 instance connection).
15. Log in to the instance as described below.

Log in to your own AWS EC2 instance

• In order to log in to your instance, you will need a security certificate or "key file".
• You will get a key file called "cshl_2022_student.pem" (for Mac/Linux users) or "cshl_2022_student.ppk" (for Windows/PuTTy users).
• NOTE: It is very important to only use your own instance (IP address or dns name) when logging in! If two people are logged into the same Amazon machine, they may conflict trying to write files to the same location, which can lead to errors and confusion.
• On the AWS cloud, we will use the default username: "ubuntu"

Log in to AWS using a terminal (Mac/Linux)

• First open a terminal session (Applications -> Utilities -> Terminal))
• Make sure the permissions on the certificate are secure. Use chmod on the downloaded key file:

change mode400 cshl_2022_student.pem

• To log in to a node, specify your credentials with the -i command-line argument:

ssh-Icshl_2022_student.pem ubuntu@[your IP address]

-ISelect a file from which to read public key certificates.Ubuntuis the username on the system you are logging into (the default user for the Ubuntu operating system).[your IP address]is the address of the linux system on Amazon that you are logging into. Instead of ip addresses, you can also use public dns names.

Logging into AWS with PuTTy (Windows)

To configure PuTTy, start PuTTy and do the following:

• Fill in the "Hostname" field with your IP address.

• In the categories on the left, select Data under the Connections category. In the autologin username field writeUbuntu.

• In the categories on the left, click + in the Connections category next to SSH. Click Authorize. In the Private key file for authentication field, click Browse and locatecshl_2022_student.ppkThe certificate you downloaded above.

• In the categories on the left, click Sessions. Write the Amazon node in the Saved Sessions field and click Save.

• Now that Putty is configured, all you have to do is start Putty and double-click on the "Amazon Node" to log in.

Copy files from AWS to your computer

• To copy a file from an instance, use scp in a similar fashion (in this case to copy a file called nice_alignments.bam):

scp-Icshl_2022_student.pem ubuntu@[your ip address]:nice_alignments.bam.

• All content created in your cloud workspace is also available through the web server on your cloud instance. Just go to the following in your browser:

http://[your ip address]/ or http://[your dns name]

File system layout on AWS instances

When you log in, you'll notice that you already have a directory: "workspace".

• The "workspace" directory is where we will keep all the files and analysis results for this course.

Upload your data to an AWS instance

If you want to upload data to an AWS instance, use the example scp command below. Be sure to replace the following variables with the local path to your data,my data, and the Amazon instance IP,YOUR_IP_ADDRESS.

scp-Icshl_2022_student.pem __MY_DATA__ ubuntu@[your ip address]:/

Use AWS outside of the workshop to complete this course

If you are attempting to complete this course on your own using only online materials, of course no AWS EC2 instance is set up for you. If you have access to an AWS account, you can start with the same Amazon AMI that we use to create instances for each student. Currently this is:

Name:cshl-seqtec-2022（ID：ami-09e0764f72d9687e0) is available in the US East, N. Virginia region (us-east-1).

We usually use the instance typem5.2x large.For detailed instructions on how we create the AMI and configure each instance, seeAWS settingsPage.

Basic introduction to AWS EC2 instances

After logging into the EC2 instance with the commandssh, you can use the following command to explore the basic information of the machine

#How is the storage volume mounted?lsblk#How much storage space is used for various mount points?whereabouts -H#Hardware detailsETC# Immediately use "top" to view active resource usage. Use "1" to view all CPUs individually. Use "q" to quittop

use GCP

coming soon

use docker

We have built a docker image with all the software tools needed to complete this lesson installed. This allows hands-on exercises to be performed on your own desktop or laptop, or wherever docker is installed. The following commands can be used to download this image and run it as an interactive docker container.

First you need to install docker. Usually this involves installingdocker desktop.Once installed, start docker, and open a terminal application. How to do this is detailed aboveuse amazonpart. In terminal you can test if your docker installation is working/working withdocker --helpWith .docker working, you can download, setup and run the rnabio image with the following commands.

1. Pull the image rnabio from the griffithlab repository with tag 0.0.1 to your local Docker client:

   docker pull griffithlab/rnabio:0.0.1

   (Video) Introduction to Programming and Computer Science - Full Course
2. Set up a local workspace directory for the RNAseq class. If you changed the path or command used here in step 2, make sure to update the path to the workspace directory accordingly in step 3.

   Table of contents -p~/rnabio-workspace

3. Initialize a Docker container using the image we pulled above. -v tells Docker to mount our workspace directory as /workspace inside the Docker container with read and write permissions. You will see in the RNAseq lessons that /workspace is the base directory for almost all commands and steps. Note: We run this docker container interactively and enter it at the bash shell prompt. We also switched the default user to "ubuntu" to match the RNAseq course tutorial.

   docker run-v~/rnabio-workspace:/workspace:rw- userfree: free-itgriffithlab/rnabio:0.0.1/bin/bash

The docker session should now be ready for the Unix tutorial and hands-on exercises.