UDF, as a read-write filesystem for USB flash drive

Miscellaneous tools
Post Reply
Message
Author
labbe5
Posts: 2159
Joined: Wed 13 Nov 2013, 14:26
Location: Canada

UDF, as a read-write filesystem for USB flash drive

#1 Post by labbe5 »

http://tanguy.ortolo.eu/blog/article93/usb-udf

Fat32 is your regular filesystem for USB flash drive. Did you know you can switch to UDF instead of FAT32, or exfat, or ext 2,3,4? It has a lot for it.

Below, there is a post about a script called format-udf.sh, which i have tested. It works, runs fast, and the resulting UDF format is stable, enabling read-write.

Note :
You need to be careful with this script, it is so fast you can wipe and format your internal drive in a few seconds. It happened to a friend of mine, using sda, instead of sdb.

USB sticks are traditionally formatted with FAT 32, because this file system is implemented by almost every operating system and device. Unfortunately, it sucks, as it cannot use more than 2 TiB, store files larger than 2 GiB or store symbolic links for instance. In a word, it is an obsolete and deficient file system.

exFATis a stinking proprietary, secret and patented file system. There are free implementations of that shit, but it is safer to stay away from it.

UDF
Good news: there is one file system that is implemented almost everywhere as well, and which does not suffer from such limitations. UDF, the Universal Disk Format, is an ISO standard originally designed for DVDs, but it is perfectly usable for USB sticks. It also supports POSIX permissions, with one killer feature for removable media: a file can belong to no specific person or group.

So, to use it, assuming you USB stick is /dev/sdc:

$ dd if=/dev/zero of=/dev/sdc bs=1M count=1
$ mkudffs -b 512 --media-type=hd /dev/sdc

Conclusion :
After that, your USB stick will be usable for reading and writing with GNU/Linux and the other free operating systems of course, but also with current versions of Windows (read-only with the outdated version XP) and with MacOS.

Tested and working, showing on Trinitydog as LinuxUDF (but unstable, using a SD card, maybe caused by wrong block size).

Note :
you need to have udftools installed.
If you run gparted, don't expect it to show an UDF partition.
https://www.quora.com/Why-doesnt-GParte ... UDF-format
Patch available for UDF support :
https://bugzilla.gnome.org/show_bug.cgi?id=784533
comparison chart between FAT32 and UDF :
https://j0nam1el.wordpress.com/2015/02/ ... -with-udf/
Format-UDF script :
https://github.com/JElchison/format-udf
Last edited by labbe5 on Wed 16 Aug 2017, 19:53, edited 2 times in total.

User avatar
bigpup
Posts: 13886
Joined: Sun 11 Oct 2009, 18:15
Location: S.C. USA

#2 Post by bigpup »

What about this statement from the link you posted?
UDF is, in principle, a write-once-and-read-only filesystem. It would never be suitable for use on a hard drive.
In Puppy, it does everything on a USB stick that you would do on a hard drive.
The things they do not tell you, are usually the clue to solving the problem.
When I was a kid I wanted to be older.... This is not what I expected :shock:
YaPI(any iso installer)

User avatar
bigpup
Posts: 13886
Joined: Sun 11 Oct 2009, 18:15
Location: S.C. USA

#3 Post by bigpup »

I was under the impression that this was the best file system for USB sticks.

F2FS
https://en.wikipedia.org/wiki/F2FS
The things they do not tell you, are usually the clue to solving the problem.
When I was a kid I wanted to be older.... This is not what I expected :shock:
YaPI(any iso installer)

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#4 Post by musher0 »

Hello all.

A couple of interesting articles:

This one has a table of everything you want in a Linux file system, and
UDF comes out on top:
http://duncanlock.net/blog/2013/05/13/u ... ash-drives

This one is interesting for the benchmarks:
http://florian-berger.de/en/articles/fi ... nilfs-ext4
Ext4 is the overall winner here, but UDF is "okay". On a flash drive you'd
want to cancel ext4's history capacity to avoid too much wear and tear.

I can't find a technical URL validating this ATM, but from experience, I
noticed that UDF takes +/- 200 Mg's less overhead on the partition than
ext? filesystems. Meaning you get +/- 200 Mg's more as storage space
on your partition. This by itself may make UDF worth using on a smaller-
sized flash-drive.

The only problem with UDF is that you can't boot from it. (Need a URL for that, too...)

BFN.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

User avatar
perdido
Posts: 1528
Joined: Mon 09 Dec 2013, 16:29
Location: ¿Altair IV , Just north of Eeyore Junction.?

#5 Post by perdido »

labbe5 wrote:
Tested and working as intended, showing on Trinitydog as LinuxUDF.
From the article labbe5 linked to

Code: Select all

It also supports POSIX permissions, with one killer feature for removable media: a file can belong to no specific person or group.
Hey labbe5, have you checked if file permissions for user and group are available on files saved on your LinuxUDF partition?

.

belham2
Posts: 1715
Joined: Mon 15 Aug 2016, 22:47

Re: UDF, as a read-write filesystem for USB flash drive

#6 Post by belham2 »

labbe5 wrote:....
you need to have udftools installed.
If you run gparted, don't expect it to show an UDF partition.
https://www.quora.com/Why-doesnt-GParte ... UDF-format
Patch available for UDF support :
https://bugzilla.gnome.org/show_bug.cgi?id=784533

Straight from the latest developers & release of Gparted 0.29:

........ 8 August 2017: GParted Live 0.29.0-1 Stable Release

The GParted team is pleased to announce a new stable release of GParted Live.

This release includes GParted 0.29.0, patches for libparted for FAT file system operations, and other improvements.

Items of note include:

Includes GParted 0.29.0 which adds the following enhancements since the last stable release:
Add support for UDF file system
Fix segmentation fault on disk with corrupted FAT file system
Fix snap-to-alignment of operations creating partitions
Based on the Debian Sid repository (as of 2017/Aug/08)
Linux kernel updated to 4.11.11-1
Package udftools added
Includes patched version of libparted which fixes:
check FAT32 crashes (bug 762448)
resized FAT32 not recognized by Windows (bug 759916)

This release of GParted Live has been successfully tested on VirtualBox, VMware, BIOS, UEFI, and physical computers with AMD/ATI, NVidia, and Intel graphics.

User avatar
perdido
Posts: 1528
Joined: Mon 09 Dec 2013, 16:29
Location: ¿Altair IV , Just north of Eeyore Junction.?

Re: UDF, as a read-write filesystem for USB flash drive

#7 Post by perdido »

belham2 wrote:
labbe5 wrote:....
you need to have udftools installed.
If you run gparted, don't expect it to show an UDF partition.
https://www.quora.com/Why-doesnt-GParte ... UDF-format
Patch available for UDF support :
https://bugzilla.gnome.org/show_bug.cgi?id=784533

Straight from the latest developers & release of Gparted 0.29:

........ 8 August 2017: GParted Live 0.29.0-1 Stable Release

The GParted team is pleased to announce a new stable release of GParted Live.

This release includes GParted 0.29.0, patches for libparted for FAT file system operations, and other improvements.

Items of note include:

Includes GParted 0.29.0 which adds the following enhancements since the last stable release:
Add support for UDF file system
Fix segmentation fault on disk with corrupted FAT file system
Fix snap-to-alignment of operations creating partitions
Based on the Debian Sid repository (as of 2017/Aug/08)
Linux kernel updated to 4.11.11-1
Package udftools added
Includes patched version of libparted which fixes:
check FAT32 crashes (bug 762448)
resized FAT32 not recognized by Windows (bug 759916)

This release of GParted Live has been successfully tested on VirtualBox, VMware, BIOS, UEFI, and physical computers with AMD/ATI, NVidia, and Intel graphics.
That is interesting and coincides nicely with this thread. I can now burn the ISO to CD and have the tool available in GParted if'n I wants to try a UDF partition using my favorite partition manager, :)
Link to downloads for various media -->http://gparted.org/livecd.php

That would ~probably mean that the GParted 0.29.0 is going to be updated with the patch GParted Live 0.29.0.1 has received maybe?

.

sheldonisaac
Posts: 902
Joined: Mon 22 Jun 2009, 01:36
Location: Philadelphia, PA

Re: UDF, as a read-write filesystem for USB flash drive

#8 Post by sheldonisaac »

labbe5 wrote:you need to have udftools installed.
Thanks.

Micro Center 16GB microSD card
Slacko64 Puppy 6.9.9.9

Code: Select all

# blkid
/dev/mmcblk0p1: SEC_TYPE="msdos" UUID="239E-D6FA" TYPE="vfat" PARTUUID="2652418e-01"
/dev/mmcblk0p2: UUID="598e4371LinuxUDF" LABEL="LinuxUDF" TYPE="udf" PARTUUID="2652418e-02"
I don't understand some of that blkid info.
Dell E6410: BusterPup, BionicPup64, Xenial, etc
Intel DQ35JOE, Dell Vostro 430
Dell Inspiron, Acer Aspire One, EeePC 1018P

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#9 Post by musher0 »

Hi Sheldon.

Me neither ! :lol: Don't worry about it! ;)

BFN.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

sheldonisaac
Posts: 902
Joined: Mon 22 Jun 2009, 01:36
Location: Philadelphia, PA

Re: UDF, as a read-write filesystem for USB flash drive

#10 Post by sheldonisaac »

labbe5 wrote:Fat32 is your regular filesystem for USB flash drive. Did you know you can switch to UDF instead of FAT32, or exfat, or ext 2,3,4? It has a lot for it.
I'd formatted the 16GB microSD card to UDF, and indeed MS-Windows and Puppy Linux can read and write it.
But my Sansa Clip+ media player doesn't recognize it.
Nor does the LG Rebel phone.
At http://tanguy.ortolo.eu/blog/article93/usb-udf
the author had said, in part:
USB sticks are traditionally formatted with FAT 32, because this file system is implemented by almost every operating system and device. Unfortunately, it sucks, as it cannot use more than 2 TiB, store files larger than 2 GiB or store symbolic links for instance. In a word, it is an obsolete and deficient file system.
Well, since those limitations haven't caused me any problems, I guess I'll use FAT32.
Dell E6410: BusterPup, BionicPup64, Xenial, etc
Intel DQ35JOE, Dell Vostro 430
Dell Inspiron, Acer Aspire One, EeePC 1018P

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#11 Post by musher0 »

@Sheldon:

One can count on popular apps to use ancient technology! :twisted:
It's beyond logic, but the boss is a conservative! ;)

BFN.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#12 Post by musher0 »

Hello all.

Five-six posts above I talked about udf's overhead on a thumb drive. The
attached pic describes the overhead taken by the udf format on an 8Mb
SD card: not quite 2 Mb's. Linux' ext2 would occupy 130 Mb's on a drive of
the same size. A difference of 128 Mb's.

I formatted the card with the mkudffs utility. Please notice the drive name:
sdd. I think udf cannot do partitions like sd1, sd2, etc.

I still haven't found a serious article comparing file systems overhead , but
there you go: empirical data.

BFN.
Attachments
Less_than_2Mb_overhead_udf.jpg
(87.35 KiB) Downloaded 1247 times
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#13 Post by musher0 »

Hi all.

I found a previous forum thread on UDF here.

BFN.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

sheldonisaac
Posts: 902
Joined: Mon 22 Jun 2009, 01:36
Location: Philadelphia, PA

#14 Post by sheldonisaac »

musher0 (in part) wrote:The attached pic describes the overhead taken by the udf format on an 8Mb SD card: not quite 2 Mb's. Linux' ext2 would occupy 130 Mb's on a drive of
the same size. A difference of 128 Mb's
I formatted my 8GB SD card to ext4 (had read somewhere that works well).
It does seem to use up a few hundred MB, even though there were no files on it.
With no journal.
(Oh, I will go try some other tune2fs options)
Anyhow, the cell phone can't mount it.
Dell E6410: BusterPup, BionicPup64, Xenial, etc
Intel DQ35JOE, Dell Vostro 430
Dell Inspiron, Acer Aspire One, EeePC 1018P

labbe5
Posts: 2159
Joined: Wed 13 Nov 2013, 14:26
Location: Canada

format-UDF

#15 Post by labbe5 »

Bash script to format a block device (hard drive or Flash drive) in UDF. The output is a drive that can be used for reading/writing across multiple operating system families: Windows, macOS, and Linux. This script should be capable of running in macOS or in Linux.

Installation :

format-udf is a self-contained script. Simply copy format-udf.sh to a directory of your choosing. Don't forget to make it executable:

chmod +x format-udf.sh

Prerequesites :
sudo apt-get install udftools coreutils vim-common

Note :
instead of Vim, use Geany, already installed on all Dogs.

Link here for complete information about format-udf (tested, working and stable) :
https://github.com/JElchison/format-udf

Block Size

If's extremely important that format-udf use the correct block size when formatting your drive. format-udf will attempt to detect and use the correct (logical) block size. If you know what you're doing, the format-udf -b BLOCK_SIZE option can be used to explicitly override the detected block size value.

If the wrong block size is used while formatting (i.e. one that doesn't match the logical block size of your drive), the resultant drive will likely have OS compatibility issues and suffer from non-optimal performance issues.


Maximum UDF File System Capacity

The UDF format has a maximum of 2^32 blocks. With format-udf, these blocks equate to logical blocks.

If your drive's logical block size is 512 bytes, then your maximum UDF file system capacity will be 2 TiB
If your drive's logical block size is 4096 bytes, then your maximum UDF file system capacity will be 16 TiB

Post Reply