Select Page

EC2 persistent boots with pivot root

S Rich | | March 2, 2010

Amazon recently allowed Elastic Block Store to boot persistent images. However, there are two concerns I have with the method. * The EBS boot volumes must be EBS Snapshots, which cost more than regular EBS volumes. ( * The EBS boot volumes currently do not work within the Virtual Private Cloud (VPC) infrastructure. ( To work around these two issues, it is possible to “pivot” from a normal AMI’s local root volume (/dev/sda1) to an arbitrary EBS volume that contains a full OS installation. An additional benefit is that a very small AMI may be used to launch the EBS backed instance, so that the instance launches much faster than a large AMI. The goal of this post is to provide instructions on how to build a small AMI that is capable of launching a full-blown persistent Linux installation that is backed on EBS storage. BusyBox, “The Swiss Army Knife of Embedded Systems”, provides a solid foundation for our small AMI. To build a BusyBox AMI and an instance backed by EBS, simply follow these directions: 1. Start an instance from an AMI that is running the desired kernel and software configuration. (You can use an existing public AMI or a custom AMI that you created. Personally, I prefer to create my own AMI’s, so I know what is in them.) 2. Log into the instance. 3. Download the latest version of BusyBox. wget “” 4. Create a busyroot directory. mkdir busyroot 5. Extract BusyBox. bunzip2 busybox-1.15.3.tar.bz2 tar xvf busybox-1.15.3.tar 6. Configure BusyBox. (You can experiment here to reduce the image size, but the configuration listed below works.) cd busybox-1.15.3 make config Select “y” for the STATIC option and the default values for everything else. Build BusyBox as a static binary (no shared libs) (STATIC) [N/y/?] y 7. Make and install BusyBox. make CONFIG_PREFIX=$HOME/busyroot install chmod 4755 $HOME/busyroot/bin/busybox 8. Create required directories. cd $HOME/busyroot mkdir dev sys etc proc mnt mnt/new-root 9. Create the necessary devices. (We will use /dev/sdj for the EBS volume, but this could be any block device not used by the normal AMI.) MAKEDEV -d $HOME/busyroot/dev -x sdj MAKEDEV -d $HOME/busyroot/dev -x console MAKEDEV -d $HOME/busyroot/dev -x null MAKEDEV -d $HOME/busyroot/dev -x zero 10. Create the init file. mv $HOME/busyroot/sbin/init $HOME/busyroot/sbin/init.orig cat <<'EOL' > $HOME/busyroot/sbin/init #!/bin/busybox sh PATH=/bin:/usr/bin:/sbin:/usr/sbin NEWDEV=”/dev/sdj” NEWTYP=”ext3″ NEWMNT=”/mnt/new-root” OLDMNT=”/mnt/old-root” OPTIONS=”noatime,ro” SLEEP=10 echo “Remounting writable.” mount -o remount,rw / [ ! -d $NEWMNT ] && echo “Creating directory $NEWMNT.” && mkdir -p $NEWMNT while true ; do echo “sleeping…” sleep $SLEEP echo “Trying to mount $NEWDEV writable.” mount -t $NEWTYP -o rw $NEWDEV $NEWMNT || continue echo “Mounted.” break; done [ ! -d $NEWMNT/$OLDMNT ] && echo “Creating directory $NEWMNT/$OLDMNT.” && mkdir -p $NEWMNT/$OLDMNT echo “Remounting $NEWMNT $OPTIONS.” mount -o remount,$OPTIONS $NEWMNT echo “Trying to pivot.” cd $NEWMNT pivot_root . ./$OLDMNT for dir in /dev /proc /sys; do echo “Moving mounted file system ${OLDMNT}${dir} to $dir.” mount –move ./${OLDMNT}${dir} ${dir} done echo “Trying to chroot.” exec chroot . /bin/sh -c “umount ./$OLDMNT; exec /sbin/init $*” < /dev/console > /dev/console 2>&1 EOL chmod 755 $HOME/busyroot/sbin/init 11. Create the fstab file. cat <<'EOL' > $HOME/busyroot/etc/fstab /dev/sda1 / ext3 defaults 1 1 none /dev/pts devpts gid=5,mode=620 0 0 none /proc proc defaults 0 0 none /sys sysfs defaults 0 0 EOL 12. Create a 4MB loopback file. cd dd if=/dev/zero of=busybox.fs bs=1M count=4 mkfs.ext3 busybox.fs 13. Mount the loopback file. mkdir $HOME/busyimg mount -o loop $HOME/busybox.fs $HOME/busyimg 14. Copy the staged files and directories to the image. (Technically, the BusyBox image could have been built directly in $HOME/busyimg, but we were not sure how big the image was going to be.) cp -rp $HOME/busyroot/* $HOME/busyimg 15. Un-mount the image. sync umount -d $HOME/busyimg 16. Set environment variables. export EC2_HOME=/opt/ec2-api-tools export EC2_CERT=/path/to/your/cert.pem export EC2_PRIVATE_KEY=/path/to/your/pk.pem export AWS_ACCOUNT_NUMBER=”NNNN-NNNN-NNNN” export AWS_ACCESS_KEY_ID=your_key export AWS_SECRET_ACCESS_KEY=your_secret_key export EC2_BUCKET=”your_bucket” export JAVA_HOME=/usr/java/default export ARCH=`uname -i` export AKI=`curl -s` export ARI=`curl -s` export INSTANCE_ID=`curl -s` export AVAIL_ZONE=`curl -s` export SEC_GROUP=`curl -s` export PUB_KEY=`wget -q -O – “” | awk -F= ‘{print $2}’` 17. Bundle the image. ec2-bundle-image -i $HOME/busybox.fs -d /tmp -k $EC2_PRIVATE_KEY -c $EC2_CERT -u $AWS_ACCOUNT_NUMBER -r $ARCH –kernel $AKI –ramdisk $ARI 18. Upload the image. ec2-upload-bundle -b $EC2_BUCKET -m /tmp/busybox.fs.manifest.xml -a $AWS_ACCESS_KEY_ID -s $AWS_SECRET_ACCESS_KEY 19. Register the AMI. BUSYBOX_AMI=`ec2-register “$EC2_BUCKET/busybox.fs.manifest.xml” | awk ‘{print $2}’` echo “BUSYBOX_AMI: $BUSYBOX_AMI” 20. Create an EBS volume of the desired size (10G or more) in the desired availability zone. VOLUME_ID=`ec2-create-volume -s 10 -z $AVAIL_ZONE | awk ‘{print $2}’` echo “VOLUME_ID: $VOLUME_ID” 21. Attach the volume to the current instance as /dev/sdj. ec2-attach-volume $VOLUME_ID -i $INSTANCE_ID -d /dev/sdj 22. Create an EXT3 file system on /dev/sdj. mkfs.ext3 /dev/sdj 22. Mount the EBS volume. mkdir /mnt/ebs_boot mount /dev/sdj /mnt/ebs_boot 23. Copy the current AMI to the EBS volume. rsync -avHx / /mnt/ebs_boot 24. Fix the /etc/fstab file. vi /mnt/ebs_boot/etc/fstab Remove the local file systems. /dev/sda1 / ext3 defaults 1 1 /dev/sdb /mnt ext3 defaults 1 2 /dev/sda3 swap swap defaults 0 0 Add the /dev/sdj file system. /dev/sdj / ext3 defaults 1 1 25. Fix the /etc/inittab file. The cloud AMI’s are normally configured for runlevel 4. vi /mnt/ebs_boot/etc/inittab Edit the following line if necessary: id:4:initdefault: 26. Un-mount the EBS volume. sync umount /mnt/ebs_boot 27. Detach the volume. ec2-detach-volume $VOLUME_ID -i $INSTANCE_ID -d /dev/sdj 28. Create a new instance running the BusyBox AMI. BUSYBOX_ID=`ec2-run-instances $BUSYBOX_AMI -z $AVAIL_ZONE -k $PUB_KEY -g $SEC_GROUP | awk ‘{print $6}’` 29. Wait until the instance is running… ec2-describe-instances $BUSYBOX_ID 30. Attach the EBS volume to the BusyBox instance as /dev/sdj. ec2-attach-volume $VOLUME_ID -i $BUSYBOX_ID -d /dev/sdj 31. Reboot the BusyBox instance to make sure it picks up the new device. ec2-reboot-instances $BUSYBOX_ID 32. Check the BusyBox instance’s console output to make sure it came up as expected. ec2-get-console-output $BUSYBOX_ID 33. Log into the new EBS backed instance. That should be it. You now have a persistent instance that is backed by EBS storage!

Subscribe to Our Blog

Never miss a post! Stay up to date with the latest database, application and analytics tips and news. Delivered in a handy bi-weekly update straight to your inbox. You can unsubscribe at any time.

ORA-12154: TNS:could not resolve the connect identifier specified

Most people will encounter this error when their application tries to connect to an Oracle database service, but it can also be raised by one database instance trying to connect to another database service via a database link.

Jeremiah Wilton | March 4, 2009

12c Upgrade Bug with SQL Tuning Advisor

Learn the steps to take on your Oracle upgrade 11.2 to 12.1 if you’re having performance problems. Oracle offers a patch and work around to BUG 20540751.

Megan Elphingstone | March 22, 2017

Oracle EPM Cloud Vs. On-Premises: What’s the Difference?

EPM applications help measure the business performance. This post will help you choose the best EPM solutions for your organization’s needs and objectives.

Bobby Ellis | April 10, 2018

Work with Us

Let’s have a conversation about what you need to succeed and how we can help get you there.


Work for Us

Where do you want to take your career? Explore exciting opportunities to join our team.