How to Create an Amazon EC2 Instance Running Centos

in

Using a combination of Amazon EC2 and S3 is a great way to setup a redundant and fault tolerant website. It is easy to deploy and reasonable priced. Below I attempted to outline the process I went through to get a simple knowledgebase website onto EC2. This is by no means a thorough tutorial, there are plenty of better ones available just by googling them. This is mostly just to document the process just in case I forget certain components.

The server is a Centos 5 box running the latest and greatest LAMP stack. I'm basically trying to create a backup instance of a server we have hosted on media temple. I installed it as a virtual machine on VMWare Fusion with 512mb RAM and 10G HD space for the demo (to keep costs and upload times down)

Before proceeding further be sure to download your Private Key and Certificate. These are available by logging into your Amazon Web Services account and clicking on the "Your Web Services Account" and then "AWS Access Identifiers". You'll probably need to create the initial cert. Make sure to download the private key and STORE IN A SAFE LOCATION! You are not able to retrieve it and must create a new one. Also jot down your Amazon Access ID and your Secret Access Key available in the same account access area as the cert generation.

The first step to get EC2 running is to create an AMI machine image of your machine. This creates a snapshot of your server and creates an AMI image file. In order to use the EC2/S3 tool kits supplied by amazon you'll need to have Ruby 1.8.4+ and Java installed. I used YUM to install Ruby and downloaded Java right from sun.

One thing to keep into consideration is your ami instance is an exact clone of your machine at a particular state. Make sure all of your daemons are configured to run, all the software you need is installed, etc. EC2 uses run level 4 so make sure you changes are made in rc4.d. I kept things pretty minimal and only enabled the following services:

  • iptables
  • network
  • syslog
  • sshd
  • sendmail
  • mysqld
  • httpd

You'll want to adjust the services based on your server needs. I tend to go with the "less is more" mantra when it comes to services (primarily due to my ignorance on what many of them do).

One area where I encountered numerous problems with was the EC2 file system being different than the one I had set in the machine I was attempting to clone. After many (many, many) failed attempts at booting the EC2 image i discovered a few work-arounds. The most critical one has to do with the fstab. EC2 did not like the default Centos fstab. You'll need to fire up your host then modify the fstab, THEN make sure to restore the original back to the Centos version otherwise you'll need to use a util like GPARTD to boot the machine and fix the fstab. The procedure is something like the following:

UPDATE: With the lastest release of the EC2 AMI toolkit there is now an option for it to create an EC2 fstab. This is awesome as it makes the preceding paragraph a non-issue!

Another area that cause a bit of confustion for me was the kernel. In the Amazon docs they specify you must use their kernel. However, I was sucessfully able to get the Centos kernel to run under EC2 with no changes. I did, however need to alter the modules a little bit. I looked through the 64bit Centos creation script on www.rightscale.com and noticed they downloaded their own xenU kernel modules. After some trial and error something similar to the following should work:

  cd /usr/local/src 
  wget http://s3.amazonaws.com/rightscale_software/kernel-modules-2.6.16.33-xenU.tgz
  tar -xvzf kernel-modules-2.6.16.33-xenU.tgz
  mv kernel-modules-2.6.16.33-xenU.tgz /lib/modules
  ln -s /lib/modules/2.6.16.33-xenU /lib/modules/2.6.16-xenU

The above modules resolved any warnings in the EC2 console.

You may find there are other files you need to alter for the EC2 instance. One I had changed was resolv.conf

Once all of the dependencies are met and your machine is in a satisfactory state download the tools from Amazon:

 

 

I like to extract all files and extract the applications in /opt. Once the applications are all downloaded and extracted I found it easy to script several of the actions to streamline the process. First thing I scripted was ALL the environmental variables needed by the various utils. Below is a simple bash script for setting the vars:

	#!/bin/bash
	#Path Variables for ec2
	export EC2_HOME=/opt/ec2-api-tools-1.3-19403/
	export EC2_AMITOOL_HOME=/opt/ec2-ami-tools-1.3-21885/
	export EC2_USER=777777777
	export PATH=$PATH:/opt/ec2-api-tools-1.3-19403/bin/
	export PATH=$PATH:/opt/ec2-ami-tools-1.3-21885/bin/ 
	export EC2_PRIVATE_KEY=~/ec2/pk-MyPrivateKey.pem
	export EC2_CERT=~/ec2/cert-MyCert.pem
	#S3 Variables
	export AWS_ACCESS_KEY_ID=12345678
	export AWS_SECRET_ACCESS_KEY=ABC123XYZDDD
	export PATH=$PATH:/opt/s3sync/
	export SSL_CERT_DIR=/etc/ssl/certs
	#Java Path Variables
	export JAVA_HOME=/usr/java/default/

Execute the previous script to set all the environmental variables. This needs to be done every time you wish to use the ec2 tools or you can edit your .bashrc file.

	[root@localhost ec2]# . export_vars.sh

When environmental variables are all set you are now ready to create and test your instances. Verify that you can connect to S3 by listing your buckets:

	[root@localhost ec2]# s3cmd.rb listbuckets

The previous command should list your S3 buckets, assuming you have some. You should now be able to create an AMI. It is EXTREMELY helpful to review the EC2 docs on amazons site. I created the following shell script for automatically creating images. It is crude and there is plenty of room for refinement.

	#!/bin/bash
	#Script for creating an AMI image
	ec2_bundle_image="$(which ec2-bundle-vol)"
	ec2_key=$EC2_PRIVATE_KEY
	ec2_cert=$EC2_CERT
	ec2_user=$EC2_USER
	echo "Creating EC2 Image. This will take a while! ..."
	$ec2_bundle_image -d /mnt -k $ec2_key -c $ec2_cert -u $ec2_user -r i386 --no-inherit --generate-fstab

That script should take about 10 minutes to run. After that you are ready to upload to S3. Amazon really did a nice job with these tools by making a simple util for uploading to the service. Again I created a simple bash script for uploading the img.

  #!/bin/bash
  ec2-upload-bundle -b bucket_name -m /mnt/image.manifest.xml --access-key $AWS_ACCESS_KEY_ID --secret-key $AWS_SECRET_ACCESS_KEY

It should take another 10 minutes to upload (on a fast connection). When that is complete you are done!. I suggest using the most excellent "ElasticFox" plugin for actual managing of the instances. I also need to further document logging in via ssh. That will be posted soon.

The next guide I'm working on is how to push data from another server to EC2 so that it can be used as a backup server.

Potential Problems

Problem: You don't have any certs in /etc/ssl/certs.

Solution: This is only an example. These certs cannot be guaranteed.

	mkdir /home/your-user/s3sync/certs
	cd /home/your-user/s3sync/certs
	# Use John Eberly's example to get the certificates
	wget http://mirbsd.mirsolutions.de/cvs.cgi/~checkout~/src/etc/ssl.certs.shar
	# Run the script
	sh ssl.certs.shar

Problem: You are getting 403 errors when running any S3/EC2 commands.

Solution CHECK ALL your environmental variables (AWS_ACCESS_KEY, AWS_SECRET_ACCESS_KEY). Also check the system date. A quick and dirty hack is to use ntpdate to set it.

	ntpdate 2.pool.ntp.org

Problem:EC2 Won't Boot

Solution: Looks like a path issue when creating the AMI (http://solutions.amazonwebservices.com/connect/thread.jspa?threadID=11885&tstart=0&messageID=94954)

	ln -s /lib/modules/2.6.18-92.1.10.el5/ /lib/modules/2.6.16-xenU

Also make sure the file system is setup according to spec based on machine image size. See the Amazon Docs.

Problem:EC2 Returns a fsck.ext3 Label=/ error

Solution:This one was real frustrating to solve but *think* i found it. After you boot Centos alter your fstab to look like:

	/dev/sda1 / ext3 defaults 1 1
	/dev/sda2 /mnt ext3 defaults 1 1
	tmpfs /dev/shm tmpfs defaults 0 0
	devpts /dev/pts devpts gid=5,mode=620 0 0
	sysfs /sys sysfs defaults 0 0
	proc /proc proc defaults 0 0

On 8/21/2008 Amazon released a new tool set that resolves all the fstab issues.