Saturday, April 23, 2005

Cloning XP with Linux and ntfsclone

I recently had two different occasions where I wanted to clone Windows XP on a few identical machines. The first time, I cloned the entire disk, bit by bit, using the dd and netcat method described in a previous post. That worked very well, but was very slow: several hours for a little 40 GB disk over a Gigabit link.

This time, I had 80 GB disks, and I not only wanted to clone the machine, but also to keep the image as a backup. So I wanted a method which would not blindly copy the whole disk, but only the parts of it that are really in use.

The partitions being NTFS, I tried ntfsclone. After falling into a few traps (completely unrelated to ntfsclone itself), it worked perfectly, and was pretty fast (using Gigabit Ethernet). So here is the recipe, for myself when I will need it again, and for whoever may find it useful:

Ingredients:

  • The machine to clone which I will boringly call Master.
  • The new clone, unsurprisingly called Slave here, and which is assumed to be made of identical hardware, particularly the hard disk.
  • Gigabit Ethernet.
  • A server for the Master disk image
  • A Linux "Live CD" (I used Knoppix 3.8.1, which had ntfsclone version 1.9.4).

The Backup:

Boot Master from the Linux live CD

Open a root shell

Set my swiss keyboard layout if needed
# setxkbmap fr_CH
or in newer versions:
# setxkbmap ch fr or setxkbmap ch de


Check if the network is up
# ifconfig eth0

It wasn't for me, and DHCP tended to fail for some reason, so I configured it manually:

# ifconfig eth0 192.168.1.27
# echo nameserver 192.168.1.4 > /etc/resolv.conf
# echo search example.lan >> /etc/resolv.conf
# route add -net 0.0.0.0 gw 192.168.1.100

The machine displaying a stupid time and time zone, I also did
# tzselect
and pasted the string it suggested on the command line,
# TZ='Europe/Zurich'; export TZ
and then set the clock:
# ntpdate pool.ntp.org
# hwclock --systohc

This was not really necessary, but I noticed that the file times on the server would be wrong if the client had a wrong time and/or time zone.

And now the real stuff:

Create a mount point
# mkdir /tmp/server

Mount the server's share. I used a share called diskimages on a Samba server, but it could have been Windows, an NFS server, or whatever.
# mount -t smbfs -o username=my_user_name   //server_name/diskimages /tmp/server

Check how your live CD called the partitions you want to save
# cat /proc/partitions
major minor #blocks name

8 0 78150744 sda
8 1 20482843 sda1
8 2 1 sda2
8 5 20482843 sda5
8 6 37182411 sda6
180 0 253952 uba
180 1 253936 uba1
240 0 1939136 cloop0

I want to save that 80 GB disk sda, which has a primary partition sda1,
and an extended partition sda2 containing logical partitions sda5 and
sda6. So what I want to save is sda1, sda5 and sda6.
First I saved the partition table and the Master Boot Record
# sfdisk -d /dev/sda >/tmp/server/master-sfdisk-sda.dump
# dd if=/dev/sda bs=512 count=1 of=/tmp/server/master-sda.mbr

and then the partitions:
ntfsclone -s -o - /dev/sda1   | gzip | split -b 1000m - /tmp/server/master-sda1.img.gz_
ntfsclone -s -o - /dev/sda5 | gzip | split -b 1000m - /tmp/server/master-sda5.img.gz_
ntfsclone -s -o - /dev/sda6 | gzip | split -b 1000m - /tmp/server/master-sda6.img.gz_
This is where I fell into the first trap. My Samba server doesn't seem to accept files larger than 2 GBytes! That is why the output is piped through split. I still don't know why I cannot write files larger than 2 GB, and if you do, please let me know. This is a Samba 3.x server running on Debian with a 2.6.x kernel, and the share is on a 36GB ext3 partition.
(update: this comment suggests to add the lfs option to smbfs mount. This allowed me to write more than 2 GB, but not more than 4GB. Probably because it's a FAT32 partition)

Anyway, split solved that problem nicely, chopping the output into 1 GB files, but I had added gzip in the hope of making things faster, and that gzip and split combination bit me later. And I'm not even sure that the gzip overhead is worth the bandwidth saving. Gigabit Ethernet can be really fast. In fact, it can be faster than the hard disks. That may be worth benchmarking some time. (I also tried bzip2, which has better compression, but that was excruciatingly slow).

That's it for the backup. Now, to the next part:

The Restore:

Boot Master from the Linux live CD, and proceed as for the backup:

setxkbmap fr_CH
ifconfig eth0 192.168.1.27
echo nameserver 192.168.1.4 > /etc/resolv.conf
echo search example.lan >> /etc/resolv.conf
route add -net 0.0.0.0 gw 192.168.1.100
TZ='Europe/Zurich'; export TZ


ntpdate pool.ntp.org
hwclock --systohc

mkdir /tmp/server
mount -t smbfs -o username=my_user_name //server_name/diskimages /tmp/server
(I just copied/pasted this whole block into the shell)

Check your partitions again, and make sure you will not overwrite some other disk!

# cat /proc/partitions

Now I first restored the partition table and the master boot record

# sfdisk /dev/sda < /tmp/server/master-sfdisk-sda.dump
# dd if=/tmp/server/master-sda.mbr of=/dev/sda

And then the partitions. Since I had several files produced by split for my primary partition, I needed to take them all, in the right order of course. split adds "aa", "ab", "ac", etc. to the end of the file name.

# ls -l /tmp/server

will help you check which files you need

This is where the second trap got me. gunzip's documentation led to believe that I could do something like gunzip -c file1 file2 file3 | ntfsclone ... which would be the same as cat file1 file2 file3 | gunzip -c | ntfsclone ...

Well, it is not the same, and my first tries would result in the process aborting after a (long) while, with the error "gunzip: unexpected end of file".

Eventually, it worked:
cd /tmp/server
cat master-sda1.img.gz_aa master-sda1.img.gz_ab master-sda1.img.gz_ac | gunzip -c | ntfsclone --restore-image --overwrite /dev/sda1 -
cat master-sda5.img.gz_aa | gunzip -c | ntfsclone --restore-image --overwrite /dev/sda5 -
cat master-sda6.img.gz_aa | gunzip -c | ntfsclone --restore-image --overwrite /dev/sda6 -
Reboot into your new Windows XP clone.

Now I wonder if there is anything I overlooked with machine IDs (SID?) and such, but I haven't seen a problem so far.
Do I need to do something, to change the SID of the clone?

If you don't need to save the image and want to be faster, you could of course combine this method with netcat and skip the server.

Labels: , , , ,

23 Comments:

Anonymous Anonymous said...

mount with the lfs option to get samba to work with files larger than 2GB.

mount -t smbfs -o lfs,username=blahblah....

27 May, 2005 01:07  
Anonymous Anonymous said...

SysInternals makes a SID changer utility; they publish good stuff. SID changing is routine with products like Ghost, etc.

http://www.sysinternals.com/ntw2k/source/newsid.shtml

27 May, 2005 04:14  
Anonymous Anonymous said...

I had exprienced the same 2 GB limit problems. mount -cifs with new 2.6 kernel works well. Also you can mount as nfs.

Regards
-Ron

14 July, 2005 17:54  
Anonymous Anonymous said...

I used ntfsclone on a 40GB partition that has 50% free space; I saved the image to an external usb drive. It took over 8 hours to have 25% completed! before I interrupted it. My questions: Is this normal? What time does it take normally? Can I speed this up?

Output drive partition is FAT32. I booted Knoppix 3.8 with usb2 option. My command line was:
ntfsclone -s -o - /dev/hda1 | gzip | split -b 600m - /mnt/uba6/hda1.img.gz.

Thanks for your help.
Kai-Uwe

24 July, 2005 11:43  
Anonymous Milivoj said...

The lfs option to mount -t smbfs worked indeed. Thanks.

However, it didn't work for files larger than 4 GB. Probably because it was on a vfat partition. So using split still seems like the best way which will always work.

26 July, 2005 19:24  
Anonymous Milivoj said...

Kai-Uwe, no such times are not normal. You probably have a problem with USB. I was not able to use USB 2.0 speeds with one Knoppix CD (maybe 3.8, but I'm not sure).

On slower machines, using gzip may be counter-productive, but nothing like what you cite.

I recently did a 6.8 GB restore from a USB 2 disk through a Gigabit network to a SATA disk, and it took 40 minutes.

26 July, 2005 19:35  
Anonymous Anonymous said...

Thank you, Milivoj. I tried an older Knoppix and it worked perfekt. Now it took about 3 hours. I experienced the same with Partimage before, so this must be a normal duration. I did not investigate my problem any further but it was most likely a problem with Knoppix 3.8 and usb2.

Again, Thank you for your help and your article about ntfsclone.
Kai-Uwe

14 August, 2005 13:07  
Anonymous Thomas said...

Instead of using cat master-sda1.img.gz_aa master-sda1.img.gz_ab master-sda1.img.gz_ac to recombine the individual files for feeding to gzip and ntfsclone, you may be able to use:

cat master-sda1.img.gz_a[a-c]

I forget where I saw this, it does seem to work on the Knoppix 4.0.2 CD. I'm not 100% sure that you'll always get back the files in alphabetical order, although my testing did.

To test this, I created half a dozen text files named 'test1.txt' through 'test6.txt' with single lines of text in each one ('alpha', 'bravo', 'charlie'...). I created these files out of order (#6 first, then #1, then #3, then #2, etc.). Typing the following gave me:

# cat test[1-6].txt
alpha
bravo
charlie
delta
echo
foxtrot
#

So even though I created files out of order, I did get them back in sorted order during the test.

18 October, 2005 03:10  
Anonymous Anonymous said...

SID problem? sysprep is your friend

25 February, 2006 19:35  
Anonymous k404 said...

the NFS solution is the easy one ...
but if you want 2 get REAL speed out of your gear you shoud use netcat & DD

the NFS trick got me to max 17MB/sec under gigabit network

netcat does it @ 34MB/sec , over the same gear ... (do the math its 1GB @30 SEC only !!!)

stop using NFS - netbios/SMB are bad 4 you

05 March, 2006 20:26  
Anonymous Anonymous said...

We have several identical PCs which need coning so these are the steps I took to get it working.

hda - Gentoo
hdc - source
hdc - destination

[Backup]
# sfdisk -d /dev/hdc > sfdisk-hdc.dump
# dd if=/dev/hdc bs=512 count=1 of=master-hdc.mbr
# ntfsclone -s -o S56G_test.clone /dev/hdc1

[Restore]
# sfdisk -d /dev/hdd < sfdisk-hdc.dump

this is to sync the disk before writing to it. I needed to do that.
# echo wq | fdisk /dev/hdd

# dd if=master-hdc.mbr of=/dev/hdd
# ntfsclone --restore-image --overwrite /dev/hdd1 S56G_test.clone

I still need to work on the SID but good advice found above :)

Hope this can help someone.

21 March, 2006 13:32  
Anonymous Jon said...

I may be off track, but I ran into a similar problem, I found that the norton ghostwalk program will change the sid, so I investigated further and I think this following one is one of the few freeware programs, check the license though please ..... dont just take my word for it.

http://www.sysinternals.com/Utilities/NewSid.html

13 June, 2006 20:52  
Anonymous Anonymous said...

Great blog!
Um i made bla.image without the -s option by acident only the -o option was used.

Can i still use ntfsclone --restore-image --overwrite /dev/hda1 bla.img

or is there a big diferents with the -s option left out?

all i could see is it might produce the .img faster and it might come out a little smaller with -s.
But other than that isn't it just the same quality and will it restore just the same with: ntfsclone --restore-image --overwrite /dev/hda1 backup.img ?

03 December, 2006 02:23  
Anonymous Milivoj said...

Cloned without the "-s" option? I have never tried it, but looking at http://forum.linux-ntfs.org/viewtopic.php?t=549 and the manual, it seems it should be no problem. Just don't use --restore-image when restoring.

03 December, 2006 18:56  
Anonymous Anonymous said...

During restore, ntfsclone 1.9.4 said "You must specify a device file". Then I realized I forgot the "-" at the end of the line. Finally it runs now..

08 December, 2006 18:32  
Anonymous flabdablet said...

If the server where your images live is running Windows Server 2003, use mount -t cifs instead of mount -t smbfs to mount the share.

Also, beware of ntfsclone image-file format incompatibilities. I have a bunch of image files I made using ntfsclone 1.12.1, and 1.13.1 won't restore them (I get "no space left on device"). Make sure you restore your images using the same version of ntfsclone that made them.

More about how I use this stuff here.

03 January, 2007 15:15  
Anonymous Anonymous said...

In case anyone is wondering this method also works for Windows XP virtual machines. I needed to upgrade to a larger drive so after running through all your steps I used Qparted to resize the partition and after the first false start boot (disk error ctrl+alt+del to reboot) the second boot worked like a charm! Thanks for this how-to!

05 June, 2007 17:46  
Anonymous Anonymous said...

The 2 GB problem is a gzip problem. If you get the latest CVS you'll be good ;)

11 June, 2007 18:24  
Anonymous namit said...

Deadly job on the tutorial it works deadly, now just got to get it working with pxe bootup and script it all thanksq

06 August, 2007 19:50  
Anonymous namit said...

Just one problem if your hard disk are different sizes this on restore moans about sizes.
sfdisk /dev/sda < /tmp/server/master-sfdisk-sda.dump

Anyone know a way around this? I am in middle of shrinking my computers hard then doing a copy then restoring it.

12 August, 2007 02:22  
Anonymous Conor said...

Just one problem if your hard disk are different sizes this on restore moans about sizes

There is no requirement for ntfsclone to restore to the same size partition. Other than having enough space of course! Create the partitions manually if you wish and then restore to the one you want.

Note that you may have issues if the first block of the partition is in a different place - ie: if you generated the image from /dev/hda1 and restored to /dev/hda2... Not sure, but I think windows is a bit dim...

13 August, 2007 21:16  
Anonymous Conor said...

As this is the one of the first hits from google when ntfsclone is mentioned I'll add some more info!

Firstly, use "mount -t cifs". Smbfs is deprecated. And a hack.

Second, you can restore an ntfs image to a larger partition - create it in fdisk and make sure it has type id of 7 - and then simply use "ntfsresize /dev/hda1" or whatever. That's magic that is!

Third, restoring a bootable windows to a different partition is slightly painful. Having a bootable XP/W2K cd will help a lot - fixboot and bootcfg but you can help yourself by pulling extra lines for partitions 1,2,3,4 in the boot.ini before starting!

I'm not sure if fixboot will fix up the "start offset" in the ntfs partition boot sector but I did it by following this.

Essentially you change the dword at offset 1c to give the start sector number (get it from "fdisk -ul"). Mine was /dev/hda1 and start sector was 63, or 0x3F. So I put the numbers 3f,00,00,00 (little endian order) in from offset 0x1c. I did it without a hex editor but you don't want to know how! (dd and vi -b!!!)

Hope this helps someone!

17 August, 2007 19:22  
Anonymous Anonymous said...

There's a good post giving step-by-step instructions on how to do this with Knoppix here.

19 August, 2007 16:26  

Post a Comment

Links to this post:

Create a Link

<< Home