Remaster From Hd Install HowTo


From Knoppix Documentation Wiki
Revision as of 17:05, 27 December 2004 by Markus (Talk | contribs)

Jump to: navigation, search

(posted by Crashed Again (sorry no web address); comments & additions welcome)


How to remaster Knoppix directly from an operating 'Knoppix type" HD install instead of from a special system accessed by chrooting into it. This allows a much easier setup and you can boot directly into the remaster and operate from it, checking and improving the system as the customization progresses. It is also less confusing, you don't have to keep track of whether or not you are in the chrooted environment.

This method also includes a second installation of a 'toHD' or 'poor mans install' as there are some operations which must be done from outside the system being remastered. This second installation provides this capability, serves as a backup system and also serves as a test bed for the final remastered compressed file system.

The toHD installation is not absolutely necessary as the Knoppix CD could be used instead but operating from the HD is faster and much more convenient. When combined with a 'persistent home' and configuration setup saved to HD, the toHD install makes a very efficient & compact operating system.

The persistent home is another knoppix feature which can be very useful in remastering. Since remastering often becomes an ongoing process, persistent home can be used to maintain consistency and retain system configuration settings throughout the process. You can even use the same persistent home for the HDinstall remaster source and the backup toHD system.


System Requirements & Setup

This method is applicable to Knoppix versions later than 3.3. Earlier versions do not have the provision for the required 'Knoppix style' HD installation.

Any computer capable of running Knoppix can be used for remastering provided it has enough free space on the hard drive. Approximately 6 Gig of free space is required:

  • HD installed Knoppix......................................................min 2.3 Gig
  • assembled compressed file system...............................700 Meg
  • final iso file system ready to burn to CD........................700 Meg
  • combined RAM & swap space totalling 1 Gig..................500-700Meg
  • toHD Knoppix operating system.....................................700 Meg
  • Total...............................................................................6.1 Gig


For this example we are going to assume we are starting from a HD with some form of Windows on it which will be set up for dual boot as this is probably more typical than a Linux only system. Functionally, the only difference having the Windows system actually makes is to change the partition numbers.

The recommended HD partition is as follows:

  • hda1...Windows O/S....................................size as appropriate
  • hda5...Knoppix HD install.............................min 2.3 Gig, 3-3.5 preferred
  • hda6...new remastered isos........................min 1.5 Gig
  • hda7...shared Windows/Linux data, VFAT format, all remaining space
  • hda8...Linux swap......................................approx 500-700 Meg
  • (swap size as needed to total 1Gig RAM + swap)

NOTE: Some Linux users add a separate partition for /home to keep their data on a separate partition from the O/S; instead of using a separate partition, put your data on hda7 & add symlinks in your /home directory to access the directories on hda7. The command for making symlinks is

ln -s /mnt/hda7/yourdirectory /home/knoppix/directoryname

The HD install being remastered must be on a separate partition by itself (hda5) as the whole partition will be used as the source for creating the compressed file system.

The create compressed file system command must be run from a different operating system than the source for the compressed file system. We are going to use the toHD install for this operation.

The toHD system should preferably be installed on hda1 with a persistent home and configuration files on hda7. The reason is that Knoppix will not give write access to the partition on which the toHD system is installed. Since we must have write access to hda6 to create the compressed file system and should also have write access to the hda7 as it is our main data drive, the toHD install should go on hda1. Not having write access to hda1 is of no consequence as it contains only the Windows operating system.

If it is not possible to put the toHD system on hda1, it should go on hda7 either with no persistent home or with persistent home on its own separate partition.


Step 1: Partition the HD and install Knoppix

Partition the HD as above and install Knoppix as a 'Knoppix style' HD install. With Knoppix 3.3 & 3.4 the install command is:

sudo knoppix-installer

This may change for later versions of Knoppix. Put the bootloader lilo on the MBR, it will be used to boot the toHD install as well.

Step 2: Install the toHD system & set up to boot from lilo

To install, just boot Knoppix with the boot code 'knoppix tohd=/dev/hda1' at the boot prompt.

In order to boot the toHD install without requiring a boot disk, a directory /boot containing at least the minimum required files to boot Knoppix must be installed on hda1. Using the HD install on hda5, first create the directory then as root do the following:

For Knoppix 3.3:

The minimum files are vmlinuz and miniroot.gz which can not be directly copied from the CD, they are buried in /boot.img. To extract the files, the boot.img must first be mounted on a loop device then on an intermediate mount point (use the existing directory /mnt/floppy) from which they can be copied to hda1. For simplicity we are just going to copy all the files in the boot.img.

/sbin/losetup /dev/loop0 /mnt/cdrom/KNOPPIX/boot.img
mount /dev/loop0 /mnt/floppy/
cp /mnt/floppy/* /mnt/hda1/boot

Then edit /etc/lilo.conf adding the following

image=/mnt/hda1/boot/vmlinuz
label="toHDKnoppix"
append="lang=us fromhd=/dev/hda1 home=scan myconfig=scan apm=power-off"
initrd=/mnt/hda1/boot/miniroot.gz
read-only

(Change "lang=" as appropriate, only include the home=scan & myconfig=scan if you are using 'persistent home')

For Knoppix 3.4:

The required files are linux24, linux26, minirt24.gz & minirt26.gz. These are located in the directory /boot/isolinux on the CD and can be directly copied to hda1 from there. The easiest way is to just use konqueror and drag-drop the entire directory '/mnt/cdrom/isolinux' to /mnt/hda1/boot/.

Then add the following to lilo.conf:

image=/mnt/hda1/boot/isolinux/linux24
label="Knoppix3.4-24"
append="lang=us fromhd=/dev/hda1 home=scan myconfig=scan apm=power-off"
initrd=/mnt/hda1/boot/isolinux/minirt24.gz
read-only image=/mnt/hda1/boot/isolinux/linux26
label="Knoppix3.4-26"
append="lang=us fromhd=/dev/hda1 home=scan myconfig=scan apm=power-off"
initrd=/mnt/hda1/boot/isolinux/minirt26.gz
read-only

Don't forget to run lilo to update the settings before exiting.

Now you can reboot into the toHD install, change the settings to your preferences, set up your persistent home and save your configuration settings to hda7. Persistent home doesn't have to be very large as the main data will be on hda7, /home contains primarily setup files & symlinks to hda7.

Step 3: Prepare hda6 to receive the remaster:

First, check your file /etc/fstab. There should be a line like this 
/dev/hda6 /mnt/hda6 vfat noauto,users,exec,umask=000,uid=knoppix,gid=knoppix 0 0
as root, change it to add 'shortname=mixed' 
/dev/hda6 /mnt/hda6 vfat shortname=mixed,auto,users,exec,umask=000,uid=knoppix,gid=knoppix 0 0

This is to insure that the file KNOPPIX will be named KNOPPIX, not knoppix. Since linux is case sensitive, lower case 'knoppix' will cause a 'cannot find filesystem' error on the remaster.

Make a directory which will contain the assembled remastered file system prior to creating the final iso.

mkdir /mnt/hda6/knx-remaster

Copy everything from the CD except the big KNOPPIX filesystem to this directory

Step 4: Remastering Tips:

You are now ready to tweak the remaster into your own personal system. There is lots of information on customizing and remastering available, I am not going to go into any detail except to add the following tips.

Tip: Updating sources

Use 'dselect update' instead of 'apt-get update'
dselect update' will keep the apt & deselect package lists in sync.

Tip: Removing applications

dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -n
will list packages sorted by size.
deborphan | xargs apt-get -y remove
will remove orphaned files not used by any package.
apt-get clean
will remove archived .deb packages. These are no longer needed once the package has been installed.
Use a purgelist to remove multiple packages at once
make up a 'purgelist' of the applications to remove then apt-get remove them all at once.
example purgelist 
3270 ace-of amanda anacron artwiz-cursor atitvout bonobo brltty chromium emacs enigma euro falconseye fetchmail fluxbox fortunes freeciv
Note that this is a space-separated list, all on one line and that wildcards are permitted. If you use a CR separated list and apt-get comes to an item it cannot find, it will not continue. If you use dpkg instead of apt-get you cannot use wildcards. Open Office can generate a space separated list from a CR separated by saving as filetype .csv
The command for using the purgelist is
apt-get remove `cat purgelist `
(The ` is the character under the tilde (~) key.)
When making up your purgelist , TEST BEFORE USING. Interwoven dependencies can create some unexpected results...for instance removing aalib* will remove the Gimp and removing libarts1 will nuke all of KDE. For all but the simplest operations, apt-get will ask for confirmation before it does anything but be safe & use 'apt-get -s remove...' first. Read through the lists and make sure of what is going to happen before actually doing it.

Clean up configuration files

Apt-get remove <package> will remove the package but does not remove the configuration files. You can clean up the configuration files with

COLUMNS=200 dpkg -l |grep ^rc |awk '{print $2} ' | xargs dpkg -P

Or you can do it all at once with

apt-get --purge remove <package>

Tip: Changing the default background

cp your-favorite-image.jpg /usr/local/lib/knoppix.jpg

Make it the same as your WM background and you have a nice transition from text mode to desktop.

Tip: Language support

Internationalization takes up over 5% of the Knoppix filesystem; removing uneeded language support will free up considerable filespace. Install the package 'localepurge' to remove all the language support you don't need.

apt-get remove kde-i18n*

will clear over 100 Meg by removing all alternate kde language support files except english. Of course if you need support for another language you will only be able to remove the ones you don't need.

Tip: Accidentally removing knoppix-installer

The hard drive installer package needs the lib file kdelibs4. Removing KDE by the common method of 'apt-get remove libarts1' (which will remove all of KDE) will also remove the knx-installer, so you must either reinstall it after nuking KDE or use a different method of removing KDE packages. Personally I am a KDE user so don't nuke KDE but it may still become an issue when upgrading KDE.

The package knx-installer is available from: http://debian.tu-bs.de/knoppix/installer/

Tip: Removing Window managers:

Boot 'Knoppix 2' if you are planning to remove your window manager; the system gets pretty confused if you remove KDE while running KDE.

Step 4: Cleaning Up

Remove the file /etc/X11/XF86Config-4. The /etc/X11/xsession.d/45xsession script will generate the correct configuration for the hardware found only if there is no existing XF86Config-4 file.

Remove any unneeded configuration files for applications you have removed from /home/knoppix. These are hidden files in your /home/knoppix directory. Check with your favorite file manager with 'show hidden files' enabled and remove any you don't need.

Transfer your /home/knoppix directory to /etc/skel if you wish to save your setup. Knoppix CD loads default settings from /etc/skel unless a persistent home is used.

(as root)
cp /home/knoppix/* /etc/skel
cp /home/knoppix/.* /etc/skel
cp /home/knoppix/Desktop/* /etc/Desktop

It should not be necessary to remove your existing /home/knoppix prior to making your compressed file system except to save space; Knoppix will automatically overwrite /home/knoppix with /etc/skel during boot. If you do remove /home/knoppix, save it somewhere first so you can restore it if need be. Your HDinstall will resort to default settings unless your existing /home/knoppix is restored.

Update your system databases

updatedb
ldconfig
update-modules

You should also run apt-get clean, purge old configuration files and remove deborphans as above before making the compressed file system.

You should also create default /etc/resolv.conf & /etc/dhcpc/resolv.conf files 
cat >/etc/dhcpc/resolv.conf <<EOT
 # insert nameservers here
# nameserver 127.0.0.1
EOT
chmod 644 /etc/dhcpc/resolv.conf
ln -s /etc/dhcpc/resolv.conf /etc/
cat >/etc/resolv.conf <<EOT
# insert nameservers here
# nameserver 127.0.0.1
EOT
chmod 644 /etc/resolv.conf

You can remove other system config files such as /etc/sysconfig/ and /etc/sysconfig/network-scripts/. This will result in your remaster starting up with all default Knoppix settings including language settings.

You may not want to do this. Since you are making a custom remaster rather than a general purpose universal operating system like the Knoppix original, you may wish to retain your personal settings instead.

Cleanup Script

A 'clean-up' script called 'remaster-clean' will be included with this how-to. It is based on the original script by Klaus Knopper but modifications by several users have been incorporated as well as comments to help the neophyte understand what it is doing. It does not restore all config files to default values nor does it remove /home/knoppix. I have set it up this way because it leaves the HD install which I am using as a source intact after making the remaster.

WARNING: This script has been successfully tested but as usual there are no guarantees it will work on your system. It should give slightly smaller file size than the basic cleanup outlined above since it removed expanded manpage files, etc. but be prepared for possible problems.

It must be run as root. Copy the script to the /user/sbin directory then 'sudo remaster.clean' to run.

Copy by using cut & paste but check the script before using with a text editor such as KDE's Kate; the most likely source of copy problems is unwanted spaces at the beginning of the line. This may cause problems, in particular with the line

EOT

which is found in two places in the script. This 'EOT' must be at the beginning of the line

<space>EOT will not work.

Step 5: Making the compressed file system

This must be run from our 'alternate' operating system (the toHD install) 
mkisofs -R -U -V "KNOPPIX.net filesystem" -P "KNOPPIX www.knoppix.net" -hide-rr-moved -cache-inodes -no-bak -pad /mnt/hda5 | nice -5 /usr/bin/createcompressedfs - 65536 > /mnt/hda6/knx-remaster/KNOPPIX/KNOPPIX

(this should be all on one line)

Using the -b option for maximum compression will result in slightly better compression but will take much longer, sometimes as much as 12 hours.

mkisofs -R -U -V "KNOPPIX.net filesystem" -P "KNOPPIX www.knoppix.net" -hide-rr-moved -cache-inodes -no-bak -pad /mnt/hda5 | /usr/bin/createcompressedfs -b - 65536 > /mnt/hda6/knx-remaster/KNOPPIX/KNOPPIX

FYI: What this means

mkisofs:           the application to make the iso fs
-R: generate SUSP & RR records
-U: allow untranslated filenames
(this is not iso9660 compliant & generates an error message which is of no consequence)
-V "KNOPPIX.net filesystem": Volume ID is "Knoppix.net filesystem"
(change this to whatever name you wish)
-P "KNOPPIX www.knoppix.net": Publisher ID is "KNOPPIX www.knoppix.net"
(change this to whatever name you wish)
-hide-rr-moved: renames the directory RR_MOVED to .rr-moved making it a hidden file
-cache-inodes: this is a default setting
-no-bak: do not include backup files
-pad: pad the whole image by 300 sectors
/mnt/hda5: location of the source directory to be compressed
nice -5: sets an operating system priority of -5 on the compression job
| /usr/bin/createcompressedfs: pipe output through the createcompressedfs application
-b: use maximum compression (very slow output)
- : use std input (ie accept input from the pipe)
65536 : block size
> /mnt/hda6/knx-remaster/KNOPPIX/KNOPPIX: direct output to the file KNOPPIX

Step 6: Testing the compressed filesystem

Simply replace the compressed file system 'KNOPPIX' on our toHD install with the remaster.

cp  /mnt/hda6/knx-remaster/KNOPPIX/KNOPPIX /mnt/hda1/KNOPPIX/KNOPPIX

Save the original first if you want to but it is almost as easy to just reinstall from the CD if you need to restore it.

After copying, boot into your toHD install. You should now see the results of your remastering efforts.

I also use this method instead a CD to transfer the remaster to my laptop which is currently running a toHD system due to lack of HD space. Since I can transfer directly from the desktop to the laptop using the 'fish' protocol, I am not limited to the 700 Mb size limit imposed by the CD for my remaster.

Step 7: Make the final iso ready for burning to CD

Update the md5 sums

cd /mnt/hda6/knx-remaster/
rm -f KNOPPIX/md5sums
find -type f -not -name md5sums -not -name boot.cat -exec md5sum {} \; >> KNOPPIX/md5sums

Make the iso file to put on CD

For knoppix 3.3:

mkisofs -pad -l -r -J -v -V "KNOPPIX" -b KNOPPIX/boot.img   -c KNOPPIX/boot.cat -hide-rr-moved -o /mnt/hda6/knx-remaster/knoppix.iso /mnt/hda6/knx-remaster/

For knoppix 3.4:

mkisofs -pad -l -r -J -v -V "KNOPPIX" -no-emul-boot -boot-load-size 4 -boot-info-table -b boot/isolinux/isolinux.bin -c boot/isolinux/boot.cat -hide-rr-moved -o /mnt/hda6/knx-remaster/knoppix.iso /mnt/hda6/knx-remaster/

Step 8: Burn your iso CD:

I just use k3b to do this.

Or you can use Qemu ( see [1] ) with the ISO image like this: qemu -m 128 -cdrom /temp/knoppix-custom.iso -boot d -user-net

Other:

There are other tweaks which can be done to improve performance such as sorting the files prior to burning the CD to optomize CD access time but the above covers the basics. There are also many other howtos & references available, in particular check out

[[Knoppix RemasteringHowto]]

Klaus Knopper's original scripts on which the remaster.clean script is based are posted here

but there are also newer versions posted here

which also includes scripting for remastering using the chroot method.


Cleanup Script:

#!/bin/bash
#This is the minimum cleanup script
#does not edit out all config files, remove /home/knoppix or restore system to default state
#Original: Knoppix.checklibs (C) 2003 Klaus Knopper
#modified by ![[Crashed Again]] for personal use
umask 022 /etc/init.d/autofs stop
/etc/init.d/apmd stop
[[-x /etc/init.d/knoppix-terminalserver ]] && /etc/init.d/knoppix-terminalserver deconfigure #remove only "temporary" or saved files in the given directories
#(rm -f is force, ignore non-existant files, never prompt)
nuke(){
for i in `find "$@" -name \*.gz -o -name \*.bz2 -o -name \*.0 -o -name \ \*.0.log -o -name browse.dat -o -name \*.\*pk 2>/dev/null`; do
rm -f "$i"
done
}
#set all files in the list to a length of zero zero(){
for i in `find "$@" -type f -size +0 -not -name \*.ini 2>/dev/null`; do :> "$i"
done
}
#remove mount points
rmdir /mnt/cdrom?* /mnt/hd?* /mnt/sd?* 2>/dev/null
# these config files must be removed for the Knoppix to setup for new hardware
rm -f /etc/X11/XF86Config /etc/X11/XF86Config-4 /etc/XF86Config /etc/XF86Config-4 # optional: remove these config files also
rm -rf /tmp/* /var/tmp/* /var/backups/* \
/home/*/.ssh /home/*/.bash_history \
/var/run/screen/* \
nuke /var/log /var/cache
zero /var/local /var/log /var/spool /var/mail \
/var/lib/games /var/cache/man /var/lib/wine \
/var/lib/nfs /var/lib/xkb #remove expanded manpage files if a compressed version exists
for i in `find /usr/*/man -name \*.\[[0-9ln\]]` ; do
[[ -f "$i".gz -o -f "$i".bz2 ]] && rm -f "$i"
done
# delete old dowloaded packages
echo "Deleting old downloaded packages....."
apt-get clean
echo " Done"
#remove 'orphaned files' not listed as used by any package
#note: may be problems if you have added applications from source instead of .deb pkgs
echo -n "Cleaning up orphans..."
deborphan | xargs apt-get -y remove
echo " Done"
echo -n "Cleaning up config files..."
COLUMNS=200 dpkg -l |grep ^rc |awk '{print $2} ' | xargs dpkg -P
echo " Done"
# Strip X11 Libs strip-unneeded removes all symbols not needed for relocation processing
strip --strip-unneeded `find /usr/X11R6/lib/modules -name \*.so -o -name \*.o` # Recreate empty utmp and wtmp
:>/var/run/utmp
:>/var/run/wtmp
# regenerate module dependencies and ls.so.cache
echo -n "Updating ld.so.cache..."
ldconfig
echo " Done." echo -n "Updating modules.dep..."
depmod -a 2>/dev/null
echo " Done." echo -n "Updating mandb..."
mandb -c
man doesnotexist >/dev/null 2>&1
echo " Done." echo -n "Updating locate-database..."
# slocate is a security enhanced 'locate' file for finding files on the system
# dlocate is a prgm to view pkg information
updatedb --prunepaths="/mnt/hd /mnt/cdrom /tmp /usr/tmp /var/tmp"
update-dlocatedb
echo " Done." echo -n "Fixing permissions in /dev/..." chown root.root /dev/ttyp* chmod 666 /dev/ttyp* /dev/sg* /dev/audio* /dev/dsp* /dev/mixer* /dev/sequencer* echo " Done" echo -n "Fixing permissions in /etc/ppp/..." chown root.dip /etc/ppp/*secrets /etc/ppp/ppp_* /etc/ppp/peers/*provider /etc/ppp/peers /etc/ppp \ /etc/chatscripts /etc/chatscripts/* chmod g+rwXš š /etc/ppp/*secrets /etc/ppp/ppp_* /etc/ppp/peers/*provider /etc/ppp/peers /etc/ppp \ /etc/chatscripts /etc/chatscripts/* echo " Done" echo -n "Removing unused architecture Kernel sources: " for i in `ls -1 /usr/src/linux/arch/ | grep -v i386`; do echo -n "$i "; rm -rf /usr/src/linux/arch/"$i" /usr/src/linux/include/asm-"$i" done echo " Done."