pUPnGO - 6Mb ISO - Basic Building Block Puplet

[goingnuts]

I have used foremost to scan my /dev/sdc4 partition which is partly a former XP ntfs partition. The partition is ext3 now. The partition is filled 90% with new files added after the partition was made. Still foremost found over 5000 (!) jpg-files where some of them are (I am almost sure...) from the previous ntfs partition.

Various ways to wipe "empty space" are described here and there. One is to use "sfill" ...did not work...foremost find all files after "sfill"...

Another way is to use dd to fill the partition:

Code: Select all

 dd if=/dev/zero of=zero.small.file bs=1024 count=102400
 dd if=/dev/zero of=zero.file bs=1024

(check with df that it is actually filled 100% - otherwise repeat second command changing filename - until df reports 100%)

Now partition should be fully rewritten upon and the generated files can be deleted.

BUT: Still foremost find all the files...

How can one get rid of them and having foremost find nothing?

I would like to include a privacy checking app in pupngo to verify that empty disk space is...empty. And at the same time have an undelete app as well...for the last purpose foremost seems quite effective.

[technosaurus]

Code: Select all

cat /dev/zero >junkfile_on_partition
rm junkfile_on_partition

[goingnuts]

Unfortunately same effect as using dd - files still found.

[PANZERKOPF]

Unfortunately same effect as using dd - files still found.

You can erase whole drive using an internal function of ATA HDD.

[technosaurus]

Unfortunately same effect as using dd - files still found.

you likely have a fragmented drive... if your filesystem has a defragment utility, do that first.

of course if you want to zero the whole drive you can just use /dev/sda (or b,c,.. or sda1 for just the partition) as the output file. ... and to meet certain standards, this should be alternated with /dev/random 7+ times.

[PANZERKOPF]

of course if you want to zero the whole drive you can just use /dev/sda (or b,c,.. or sda1 for just the partition) as the output file. ... and to meet certain standards, this should be alternated with /dev/random 7+ times.

If you want to erase the whole drive, mentioned above internal security function is fastest, because no I/O between drive and host. A drive erases itself using functions of internal firmware. In Linux world this possibly can be done with hdparm, although there are other special programs for working with HDDs.

[amigo]

You need to use dban or something of that caliber. The data has to be overwritten multiple times to make it really disappear.

[goingnuts]

Thanks technosaurus & PANZERKOPF! I don't want to wipe entire drive. I want to be sure no traces left of deleted files in free space on the partition keeping what is there. Foremost finds a lot.

As for fragmentation fsck -nvf /dev/sdc4 reports 103454 non-contiguous inodes (6.4%) so fragmentation is there. Partition is 90% filled. So how do I defrag?
Found e2defrag (view image below of it running) and tried to run it with switch -r (read only) - I am a chicken. But would hate to loose over 30 Gb sources, build scripts, patches and various tool chains. Which remind me that I should backup some of those things soon...
But after crunching for some time (10 min?) e2defrag actually simulate a relocation which goes quite fast (1 min?). Might try to clone the drive and do a test on the clone with e2defrag.

For the adventurous reader - NO CLAIMS ACCEPTED! - I have attached a static build of e2defrag. And you can get foremost here

Amigo: I will try that as well - thanks.

Never thought I should on this journey just to wipe free space on a partition...

[Ibidem]

I have used foremost to scan my /dev/sdc4 partition which is partly a former XP ntfs partition. The partition is ext3 now. The partition is filled 90% with new files added after the partition was made. Still foremost found over 5000 (!) jpg-files where some of them are (I am almost sure...) from the previous ntfs partition.

1. What mount options? (Is it full-journalling?)
2. What's on the drive? sources?

If it's mounted on /mnt/sdc4, try this:

Code: Select all

find -iname '*.jpg' -o -iname '*.jpeg' /mnt/sdc4|wc -l

[goingnuts]

From dmesg:

Code: Select all

kjournald starting.  Commit interval 5 seconds
EXT3 FS on sdc4, internal journal
EXT3-fs: mounted filesystem with ordered data mode.

Code: Select all

# find /mnt/sdc4 -iname *.jpg -o -iname *.jpeg |wc -l
1204

and foremost running on unmounted sdc4:

Code: Select all

Foremost started at Thu Dec 12 05:35:32 2013
Invocation: foremost -v -T -t jpg /dev/sdc4 
...
1512 FILES EXTRACTED
	
jpg:= 1512

Drive holds mostly source packages, unpacked/packed. Quite a lot of deletion and unpacking/compiling/packaging is done on an everyday basis...

The jpg´s found - are lots of small icon-images, background-images where some seems to come from webpages or manpages...

[amigo]

Yes, be very careful with e2defrag as it is very old. Don't use it on anything you don't have a copy of elsewhere.

Somewhere here I have a gtk1 app which shows the fragmentation status of a drive or file -but I don't find it right now -something with 'dav' in the name IIRC. Ahh, here it is, but it's not on my site davl it's called:
http://davl.sourceforge.net/

[goingnuts]

e2defrag is maintained - cant say its safe - but seems quite up to date.
Thanks for the gdavl-link - cool!

[amigo]

He, He, that davl really reminds of the windows tool for that. Ummm, are you gonna patch it so that it uses e2defrag to actually do a defrag instead of just showing fragmentation??? Yeah, Yeah, Yeah?? Just kidding, but I think I read you okay -most of the time.

Nice find there about e2defrag. from the website:

This poor ancient package used to be known as the defrag packge but was removed from Debian and hence Ubuntu due to it not having had a maintainer in many years and suffering from bit rot. I am rescuing it from the bit bucket.

Very nice indeed.
There is also e4defrag included with e2fsprogs, but nothing for ext3. I still use ext3 for my daily use as ext4 still 'hits a bump' every now and then.

[Ibidem]

From dmesg:

Code: Select all

kjournald starting.  Commit interval 5 seconds
EXT3 FS on sdc4, internal journal
EXT3-fs: mounted filesystem with ordered data mode.

Code: Select all

# find /mnt/sdc4 -iname *.jpg -o -iname *.jpeg |wc -l
1204

and foremost running on unmounted sdc4:

Code: Select all

Foremost started at Thu Dec 12 05:35:32 2013
Invocation: foremost -v -T -t jpg /dev/sdc4 
...
1512 FILES EXTRACTED
	
jpg:= 1512

Drive holds mostly source packages, unpacked/packed. Quite a lot of deletion and unpacking/compiling/packaging is done on an everyday basis...

The jpg´s found - are lots of small icon-images, background-images where some seems to come from webpages or manpages...

That's about 300 (or 20%) that aren't from jpg files.
data=ordered appears to not result in the file contents getting saved in the journal...as far as I can tell.
I'd say that's reasonable....

[goingnuts]

amigo:
Ibidem: I don't catch your point: "find" finds files not deleted, foremost finds deleted files...

To speed up testing and avoid using drive with precious content I created a smaller (6Gb) partition by resizing 2 ntfs - and then create the new in between with gparted.

To start out its ext2.

Code: Select all

# find /mnt/sdc9 -iname *.jpg -o -iname *.jpeg | wc -l
0

Then I run foremost on unmounted partition

Code: Select all

# foremost -v -T -w -t jpg /dev/sdc9
...
526 FILES EXTRACTED
        
jpg:= 526

So jpg-left overs from the ntfs can be found...
Now I try to wipe with

Code: Select all

dd if=/dev/zero of=zero.small.file bs=1024 count=102400
dd if=/dev/zero of=zero.file bs=1024

df reports

Code: Select all

/dev/sdc9              6048132   6048132         0 100% /mnt/sdc9

I let the files stay and umount partition. So now I expect to find nothing there with foremost (drive is full):

Code: Select all

0 FILES EXTRACTED

Then I delete the two files created with dd and run foremost again and

Code: Select all

0 FILES EXTRACTED

Good! This is the expected behavior - now I need to verify that if the drive is fragmented the above wont wipe free space...later today...

[Ibidem]

amigo:
Ibidem: I don't catch your point: "find" finds files not deleted, foremost finds deleted files...

From what I understand, foremost finds all files having that signature, whether deleted or not.

BTW, there are a few files that contain embedded jpegs...some mp3 files, for example. But if it's mainly source code, that's irrelevant.

To speed up testing and avoid using drive with precious content I created a smaller (6Gb) partition by resizing 2 ntfs - and then create the new in between with gparted.

To start out its ext2.

Code: Select all

# find /mnt/sdc9 -iname *.jpg -o -iname *.jpeg | wc -l
0

Then I run foremost on unmounted partition

Code: Select all

# foremost -v -T -w -t jpg /dev/sdc9
...
526 FILES EXTRACTED
        
jpg:= 526

So jpg-left overs from the ntfs can be found...
Now I try to wipe with

Code: Select all

dd if=/dev/zero of=zero.small.file bs=1024 count=102400
dd if=/dev/zero of=zero.file bs=1024

df reports

Code: Select all

/dev/sdc9              6048132   6048132         0 100% /mnt/sdc9

I let the files stay and umount partition. So now I expect to find nothing there with foremost (drive is full):

Code: Select all

0 FILES EXTRACTED

Then I delete the two files created with dd and run foremost again and

Code: Select all

0 FILES EXTRACTED

Good! This is the expected behavior - now I need to verify that if the drive is fragmented the above wont wipe free space...later today...

My suspicion had been that it was something to do with the journalling. But it looks like it probably isn't, so I don't have any ideas.

[goingnuts]

Ibidem: thanks for the explanation. I might have had a few mp3-files there as well - but the found images did not look like mp3-stuff.

The continuation of the journey comes here:

Now I fill partition with unpacked/extracted source files and large amounts of videos.

fsck reports 145 non-contiguous inodes (0.2%) - and foremost says:

Code: Select all

19 FILES EXTRACTED
        
jpg:= 19

??? I recognize a scrambled video-cover between them - so best guess is that they have entered via the copy of files to the partition...

Now I delete something until approx. 90% of drive still filled.
fsck reports same 0.2% fragmentation. - and foremost find same things as before...
I fill partition with dd, delete the created dd files - and foremost finds - same as before.

OK - time to exercise e2defrag. It goes without problems. fsck reports 18 non-contiguous inodes (0.0%) and gdavl reports fragmented files: 23.
Foremost finds - same 19 jpg files. I do the dd filling again - hoping - but no luck. Those files that foremost finds are resistant.

One last trial: I delete things down to 55% filled partition, run e2defrag, fill partition with dd and run foremost - now only 18 files are found - only 1 has gone.

Well - one more: delete everything - fill with dd - foremost finds nothing now. So whenever creating a new partition it might me good practice to do the dd-thing before starting to use the partition - just to get rid of all old stuff.

Now that was at lot of testing and unfortunately with a poor outcome concerning a simple privacy app. But it seems that e2defrag works and gdavl is a nice tool too. And foremost finds things quite well - so thats a good tool for undelete files...

[technosaurus]

In case anyone is building jwm with translucency support...
I tracked down steam's patched xcompmgr:
http://repo.steampowered.com/steamos/po ... .14.tar.gz

it may also need:
http://repo.steampowered.com/steamos/po ... 1.8.tar.gz

NOTE: They left a bunch of debugging code lying around.

[goingnuts]

technosaurus: Thanks for the links!

I realize that no pUPnGO2013 is going to be published - maybe a 2014...working on it.

Having fun with the basic core at the moment - converting various original puppys to squash-3.1 formate and loading them after boot - thats easy kernel switch - if you like 2.6.25.16.

[technosaurus]

I've been messing with reimplementing hotplug here if anyone is interested in playing with it. Currently it does about the same that mdev does, but since it is written in shell, it can easily be modified.

Re: kernel... It would be nice to have some of the new syscalls (rfkill, finit_module) backported to 2.6.32 (oldest maintained LTS kernel) and use that for a 586+mmx kernel. I suggest this because anything less does not run many things efficiently (486 only got to ~133Mhz with a few exceptions) and there are still mainstream CPUs that are not 686 (technically they are but they are missing CMOV) but AFAIK they all have mmx (but not necessarily 3dnow and others)
We should use 3.10 (the newest LTS) for other architectures (basically what musl-libc and aboriginal linux support) ... for non-x86 architectures it is essential to use a newer kernel since much work on these has been a result of android and has accelerated over the last few years.