Convert any ISO into a hybrid & use dd to place any

[slavvo67]

Hi GCMartin:

I don't believe that 's 100% accurate. I think the ISOHybrid format with the dd instruction on page one creates a once only write (unless you use dd back over it) to the USB drive; not allowing for a save file on the same USB. ETP advised the .img format seems to allow for different partitions and, therefore a safe file on the same stick.

Not sure if there's a way to use the "dd" script while carving a second partition for save. That would be the most sensible, as it seems wasteful to use one USB for booting the puppy and another USB just for the save file. I'm not sure if it's possible, however, based on ETP's direction.

For me, the "dd" scripts seem to be the most simplistic way to get your iso on a USB. I've confirmed this process works with Fluxpup, Tahrpup and 3 Headed-Dog so as I believe ETP has stated, it seems to work with all puppies.

Note: I wouldn't use a big USB when using dd, as you'll waste a bunch of space. I would suggest an 8gb stick, maybe even smaller would work.

[goingnuts]

I've confirmed this process works with Fluxpup, Tahrpup and 3 Headed-Dog so as I believe ETP has stated, it seems to work with all puppies.

I do have the earlier quoted problems with isohybrid and fx. puppy-4.1.2-k2.6.25.16-seamonkey.iso
Mounting the iso and copying isolinux.bin out of it - and running

Code: Select all

#strings isolinux.bin | grep ISOLINUX
ISOLINUX 3.63 2008-04-10

indicate that older puppies might not be converted without a remaster of the iso after updating isolinux.bin to a newer version.

[greengeek]

Not sure if there's a way to use the "dd" script while carving a second partition for save. .

I would be interested to know if a second partition is tolerated. I would like to use this method to boot a standard generic puppy (without savefile) then pull a "personalisations" .pet from a second partition and install it to apply necessary personalisations in an effectively readonly manner. (ie the session would have personalisations but not remember todays activities or corruptions...)

[amigo]

OK, let me explain a bit. What the hyrbrid tool does is create a partition table at the beginning of the iso image. CD images are blank for the first 32K (IIRC), so the tool has plenty of room to insert a boot sector. Once done, if the image is burned to a CD, then when the bootloader looks at the drive it sees a normal bootable CD image -since it is a CD drive, the bootloader starts looking at the 32K boundary. However, if the same image is burned to a mass-storage device, then the bootloader will first see the boot sector and partition table.

To repeat, the bootloader will use different criteria to decide if a device is bootable, depending on what type of device it is. The process of creating the hybrid image entails making the image appear to be several things at once. The makebootfat tool does something similar by making an image or RW device appear to be two or three things at once -namely a normal FD floppy drive, a bootable ZIP device, or HD hard drive -and if you don't know it already, a CD image presents itself as a large floppy (2.88K).

Now, since using dd writes directly to the device or image, including writing the boot sector if included in the image. mkisohybrid will create a partition table -it doesn't know the details of the actual drive- so it creates a single partition of a size which will encompass the image.

**After that** you can run fdisk on the image in order to add new partitions or change the size of the first one. You can also do this after dd-ing the image to the drive(USB or whatever). Of course, you'll need to leave the first partition big enough to hold the entire original image, but otherwise you are then free to add as many partitions you like, of whatever size.

As always, the difficulty with implementing a tool which automates this process is in being able to do it safely. What I mean is that you may need/want user input before arbitrarily overwriting a drive -you surely don't want to destroy user data without asking first, right?

[ETP]

Code: Select all

Example 1 - Place 4MB Hybird ISO onto 16GB SD card
(Neither the drive or the iso are mounted)

# dd if=/root/Downloads/sda6-hybrid-boot-rapid-pup-703.iso of=/dev/sde
8192+0 records in
8192+0 records out
4194304 bytes (4.2 MB) copied, 1.12173 s, 3.7 MB/s
#

Remove stick then re-insert.
Hover over sde1 drive icon & now shows as "Filesystem: iso 9660 Size 4MB"
GParted does not recognise it as it is now an ISO and gives a recursive partition error.
I am not aware of any way of making use of the unused space on the card. It is wasted.

Example 2 - Place 245 MB Hybrid ISO using same card still with 4MB iso on it.

# dd if=/root/Downloads/chromebook-pupv1.iso of=/dev/sde
501760+0 records in
501760+0 records out
256901120 bytes (257 MB) copied, 74.5973 s, 3.4 MB/s
#

Remove stick then re-insert.
Hover over sde1 drive icon & now shows as "Filesystem: iso 9660 Size 245MB"
Partview correctly shows size as 245M with 0M free.
Tested bootability on another PC then shut down without saving.

Isohybrid is if you like, a trick that fools a PCs bios into booting a treated bootable CD image (An iso) into
into booting as a CD-Rom when placed on a card or stick. To all intents and purposes it is then behaves in the same
manner as a CD. You cannot create extra partitions on a CD and currently AFAIK there is no way of utilising any
free space on a card or stick as you will destroy the bootability trick in so doing.

A hybrid ISO is very convenient for both users of burners and those people without who can quickly place such an
image on a card or stick.
The ISOHYBRID command has been around for several years & many mainstream distros such as Debian, Ubuntu Arch & Mint in recent
years haved issued their ISOs as hybrids.

The ideal time to create a hybid iso is at the point of production rather than later when problems can arise due to a mismatch
between the versions of syslinux used to create isolinux.bin (the bootloader) and isohybrid which should contain the same MBR code.
goingnuts wrote:

older puppies might not be converted without a remaster of the iso after updating isolinux.bin to a newer version.

I agree with his comment and that may be needed with some later ones post syslinux 3.72

@ amigo
Having written the above I saw your post about using fdisk to make use of any free space.
Could you detail that using chromebook-pupv1.iso as a real world example?

[amigo]

Simply run fdisk on the iso image to view or create a partition table. How big is the image? Round that figure up to a convenient partition size and make the first partition big enough to hold the complete image. Leave it unformatted as it is already 'formatted' as an iso. Create any other partitions you want in the remaining space on the drive. Then format them however you want.

A more geeky solution would have you use dd to write directly to the drive *after* the space occupied by the iso image (using dd's 'seek' option). And you could then associate the data at the point where you dd'ed it to the loop-back device using losetup's 'offset' option. -Better use option above.

[L18L]

Code: Select all

-
 am not aware of any way of making use of the unused space on the card. It is wasted.
-

New UEFI flash drive installetion method

[slavvo67]

Amigo,

It is possible to run fdisk on an uninstalled .iso file? That seems odd. I tried fdisk after dd installing the iso and it didn't work.

There was a better way but I lost the instructions to do it. I think it was creating a small fat32 with a larger f2fs and using grub4dos to boot the fat32 partition. I think it's in this darn forum somewhere but easier to find a needle in a haystack.....

[gcmartin]

Code: Select all

root# fdisk -l /mnt/PC-on-204/PUPs/TahrPUP/tahr64-6.0.3.8.iso

Disk /mnt/PC-on-204/PUPs/TahrPUP/tahr64-6.0.3.8.iso: 234 MB, 234881024 bytes
64 heads, 32 sectors/track, 224 cylinders, total 458752 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x3839bfd0

                                         Device Boot      Start         End      Blocks   Id  System
/mnt/PC-on-204/PUPs/TahrPUP/tahr64-6.0.3.8.iso1   *           0      458751      229376   83  Linux

Not sure if this is what you ask.

[slavvo67]

@GCMartin:

Kind of but adding a partition to an .iso?

[jamesbond]

Please read this then get the script for your review. It's for fatdog so it's not directly applicable (fatdog isohybrid has 2 partitions instead of one), but the idea is the same.

[Puppus Dogfellow]

is it possible to add files--whether from a decompressed quirky.xz or puppy.iso--to a drive that's already been formatted as two partitions, one a 16mb vfat and the the other the balance of the drive as f2fs? (basically what Barry Kauler's quirky installers do to flashdrives, unless that's changed for the most recent quirky). it'd be nice to be able to keep the data on the large partition and alter files on it and the boot partition as necessary to accommodate different or additional puppies or quirkies (multiboot would be nice, but i'd be happy to just reuse a drive without having to entirely wipe it first). instead of expanding out to an entire partition, can this method be used to force a save folder on that partition? could the conversion to isohybrid be done in a folder and then the contents moved?

thanks for the guidance/patience (i'm not exactly sure how idiotic these questions are, but i feel they may be significantly so).

[amigo]

Of course you can add, change or delete files from the partitions, they just neeed to be mounted first.

You cant 'hybridize' a folder. The process writes data to the first sector of an iso image(a file).

[Puppus Dogfellow]

Of course you can add, change or delete files from the partitions, they just neeed to be mounted first.

You cant 'hybridize' a folder. The process writes data to the first sector of an iso image(a file).

thanks for the reply, amigo.

just to be clear, how accurate is the following?

starting with two drives formatted ETP's way or by way of 01micko's f2fs installer or BarrkyK's quirky installer (i apologize if i'm mistakenly clumping these tools together--superficially at least, they seem to do more or less the same thing to the target flashdrive), i use one for a period and decide i want to swap in a different OS. i move all the original 2nd partition files to and archive folder, delete the contents of the boot partition, copy over the files from the second, untouched drive, and drive 1 will boot with the new OS and the archive folder (remnants of personal and system files of the previous installation on the drive) accessible?

if so, that seems extraordinarily convenient.

_____


here's how quirky installs to two partitions:

Code: Select all

16MB partition 1:
boot.msg  ldlinux.sys  vmlinuz
help.msg  logo.16     syslinux.cfg

Code: Select all

27 GB partition 2:
audit  dev   lib       mnt   root   sys  var
bin    etc   lib64     opt   run    tmp
boot   file  man2html  proc  sbin    usr

[this second batch plus any personal files/folders the user creates on the partition would be the ones i'd want to archive.]

how hard-wired (for lack of a more accurate term--accuracy about to plummet further) or unforgiving is the code identifying the drive and tying it to the OS it's running? if this won't work as conjectured, how involved would a manual fix be? this method doesn't require any changes to the menu.list, so it seems like it could be potentially a very newbie friendly installation method. there'd be one fewer thing to mess with, and gparted could allow for very specific formatting of the target drives.

[amigo]

Whoa, partner. "delete the contents of the boot partition, copy over the files" This doesn't restore the boot sector. The boot sector is not a part of the file system -it precedes the first sectors used by any filesystem. You still need to use 'dd' on the first sector of the drive or image to write any new bootsector.

To use the proper terms and get the whole picture, the first
446 bytes is the executable boot code -the boot sector. The next
64 bytes is the partition table. The next
2 bytes is the 'magic' which signals the end of the MBR. The MBR
is the 512 first bytes of the disc.

A bootable CD or iso image do not have an MBR. There is a significant empty region at the start of the disk -32K IIRC. bootloader code 'knows' to look at this location when booting from a CD device. mkisohybrid takes advantage of the fact that both bootable CD code and MBR can exist on the same image/device without 'colliding'.

filesystems do not usually start in the first (0 zero) sector. They start in the second (1 one) sector.

If you create extended partitions, then the last bits in the partition table will point to the spot where the extended partition data is located -this is also somewhere in the first sector.

bootloaders also can use the space left over in the zero sector.

copy/move operations never touch the first (zero) sector as 'cp' and 'mv' work with filesystems. 'dd' or 'cat' can write to the zero sector. The zero sector refers to whole device -not to a partition ( /dev/hda and not /dev/hda1 ). Technically, it shouldn't matter if any of the partitions are mounted during writing to the zero sector -but good-practise would encourage you to not do so.

Using fdisk also writes to the zero sector, but only to the partition table areas -it does not write to the real bootsector. Hence, any partitioning can be done after dd'ing. Usually, one might dd a complete 512 byte MBR which already has a partition table, then use fdisk to alter the partition table.

[Puppus Dogfellow]

[...]

To use the proper terms and get the whole picture, the first
446 bytes is the executable boot code -the boot sector. The next
64 bytes is the partition table. The next
2 bytes is the 'magic' which signals the end of the MBR. The MBR
is the 512 first bytes of the disc.

is there any way to copy over the boot sector, partition table, and the 2 bytes of magic from one flash drive to the other without destroying the rest of the data already on the second drive? even if it means the machine misidentifies the drive (is one of these areas where the manufacturers store device-specific info like make, model, etc? how does my machine know a kingston drive from a corsair?), will it boot if you could somehow just transfer over the MBR? is there info in the MBR that's looking for a specific OS to boot? is it tied to some sort of identifying code on the drive housing it? and am i correct in thinking the small 16mb partition just points to needed files on the larger one? assuming you could get the drive to boot and read the first partition, would it then read partition 2 or would it balk because what it finds there wasn't the OS originally burned or dd'ed to the drive?

[...]
copy/move operations never touch the first (zero) sector as 'cp' and 'mv' work with filesystems. 'dd' or 'cat' can write to the zero sector. The zero sector refers to whole device -not to a partition ( /dev/hda and not /dev/hda1 ). Technically, it shouldn't matter if any of the partitions are mounted during writing to the zero sector -but good-practise would encourage you to not do so.

Using fdisk also writes to the zero sector, but only to the partition table areas -it does not write to the real bootsector. Hence, any partitioning can be done after dd'ing. Usually, one might dd a complete 512 byte MBR which already has a partition table, then use fdisk to alter the partition table.

perhaps i'm getting ahead of myself, but what programs or commands would i need to copy over the mbr from one drive to the other without touching anything else? if that part works, would a simple transfer of the isohybridized system (boot partition 1, system files in partition 2) face any other obstacles preventing the setup from working as well as if one of the methods described earlier had been used directly on the drive?

thanks again, amigo.

[gcmartin]

Hello

is there any way to copy over the boot sector, partition table, and the 2 bytes of magic from one flash drive to the other without destroying the rest of the data already on the second drive? even if it means the machine misidentifies the drive (is one of these areas where the manufacturers store device-specific info like make, model, etc?

This is an almost impossible dilemma that would result. Physically, YES, you can copy to a 2nd. Couple reasons why it may not give the results you would want:

• Drive geometry(s) are different. Even sometimes where drives/unit have the same model numbers. And, the PAT tables which are valid on the device you copy from, would be invalid for the drive you are copying to.
  And, other info that would reside identifying useful information, for say, fdisk (and others) would be using information that would not be reliable for the unit.

I am sure that there are some programs, out there, which can interrogate the invalid drive and correct that area; where you would need to use one of them, immediately after a copy operation, to "fix" the information for proper use.

Hope this helps, a little.

[Puppus Dogfellow]

Hello

is there any way to copy over the boot sector, partition table, and the 2 bytes of magic from one flash drive to the other without destroying the rest of the data already on the second drive? even if it means the machine misidentifies the drive (is one of these areas where the manufacturers store device-specific info like make, model, etc?

This is an almost impossible dilemma that would result. Physically, YES, you can copy to a 2nd. Couple reasons why it may not give the results you would want:

• Drive geometry(s) are different. Even sometimes where drives/unit have the same model numbers. And, the PAT tables which are valid on the device you copy from, would be invalid for the drive you are copying to.
  And, other info that would reside identifying useful information, for say, fdisk (and others) would be using information that would not be reliable for the unit.

I am sure that there are some programs, out there, which can interrogate the invalid drive and correct that area; where you would need to use one of them, immediately after a copy operation, to "fix" the information for proper use.

Hope this helps, a little.

thanks, gcmartin, it does help. it also anticipates the follow up, which was going to be along the lines of, if the move is possible, what then may need to be edited or corrected and how involved would that be.

[amigo]

I usually don't say 'copy' when referring to the MBR and rest of sector zero. I say 'write'. Again, 'cp' and 'mv' only work within filesystems. 'writing' to a full device (not a partition) is done directly by using 'dd' or 'cat', and indirectly by using fdisk or other such tools.

Yes, you can use 'dd' to 'write' any segment of the disk. Using the --skip and --seek options you can tell it to start and stop on any byte you want. Other tools like grub or grub-install also write to the MBR area, but only to the first 446 bytes -the real 'bootsector'. fdisk, parted and such onyl write to the partition table -the 64 bytes after skipping the first 446 bytes. The two bytes of 'magic' are really a part of the partiton table -without them the partition table is not complete.

One thing to realize about disk operations using *filesystem* tools like cp, mv and rm, is that when you 'rm' a file, the data does not get removed at all -only the *entry* for the item gets removed from the filesystem 'journal'. The data from a file remains in place.

Remember that the whole filesystem resides on a partition which is defined in the partition table, which is *not* part of the filesystem. The 'structure' of a filesystem begins with certain characteristic bits -which must be placed at the point where the partition table says the partition begins. In other words, if you change your partition table, the data which resides in the old partitions is not touched. If you change the partition table exactly like it was before, the filesystem will still be intact.

However, if after partitioning you *format* the partition, you destroy the old filesystem journal(or allocation table), but note that the original data contained in your files will *still* be untouched. You could still use a rescue tool to 'read' the files -possibly. But, if you actually start creating or copying new files into the newly-formatted filesystem, you will be overwriting the old data areas, at which point the old data is definitely not recoverable.

If you are hoping to automate something here, then you need to read the info in the original partition table, and then reproduce it so that the filesystem(s) can be found. But that makes no sense -simply use dd to selectively write only what is needed.

vis-a-vis mkisohybrid, I am not exactly sure what all it writes -whether it is a complete MBR, or with an empty bootsector (446bytes) with *only* a partition table. If it is complete, then it will have a standard DOS-type MBR which contains the boot code which does the standard MS-ey thing -look for the first bootable partition and start execution there. The thing which makes the iso *hybrid* is that it has both a partition table and an 'El Toro' boot sector(the boot code for an ISO/CD device beginning after the 32K boundary).

The main utility of the above is that you 'recover' any space left over on your flash drive(or whatever), after writing (using dd) a hybridized ISO image to the drive -which contains a partition table entry for the first partition which skips over the content of the ISO. If the ISO is 50MB, then the first partition must be set to at least that size. Any second (or more) partitions have to be placed after that.

fdisk -l will report the real raw disk size and the size of any partitions. mkisohybrid is used for altering *ISO images*, so it has no way of knowing anything about the size of any drive or partition info. It arbitrarily creates a first partition the same size as the ISO image it is being used on -it doesn't know any more than that, by default. It may have options which let you specify partition size or even multiple ones.

[Puppus Dogfellow]

If you are hoping to automate something here, then you need to read the info in the original partition table, and then reproduce it so that the filesystem(s) can be found. But that makes no sense -simply use dd to selectively write only what is needed.

i'm hoping to automate or simplify the swapping of puppies or quirkies onto pre-formatted and partitioned flash drives. i'm primarily interested in using f2fs for the main system, but it would be nice to be able to perhaps use some of the drive for other formats. all the tools seem to wipe the drives clean before installing the the OS, so i guess what i'm after is something a little more precise and a little less destructive. here's a scenario. the user has or formats an sd card, flash drive, or ssd in a way that allows the use of not only a primary, f2fs partition (and it's requisite mini-partition, since as it stands, booting from f2fs appears not yet doable), but also (the possibility of) others--ext2/3/4 to be able to boot multiple pups, and/or maybe a swap partition. now, if done correctly in gparted, the drive should have all the necessary mbr bits still in tact after the formatting, but nothing has yet been done to make it into a bootable drive (though the copied directories from the new OS (partition 2 in example above) may be on a/the f2fs partition). how involved would it be to create a correct mbr or change the existing mbr (or parts thereof) on the target drive without touching anything else except perhaps that one mini-partition (if those files also haven't been transferred to the new drive)? even if the user has to manually transfer files over to the f2fs partition and/or the vfat partition, this type of set up would allow fairly easy swaps to other quirkies or puppies--you'd only have to hybridize an iso or img file (is this what quirkies are? can one merely decompress a quirky.xz for the needed files?) somewhere and then move the files over.

how completely off is all this/thanks for your patience.