What flash drive speeds do you get?

What works, and doesn't, for you. Be specific, and please include Puppy version.
Message
Author
stevielee
Posts: 36
Joined: Thu 26 Jul 2012, 02:51

What flash drive speeds do you get?

#1 Post by stevielee »

OK, everyone -

I am wondering about your real-life experiences with USB flash drives.


What write speeds do you get?

How do you optimize those speeds?

Do you see disparities between Linux and Windows?


I have looked all over and seen so much FUD on this - it is terribly confusing, and I thought that this forum is the place to ask as you people are so knowledgeable and helpful. I can't see why this is not a bigger subject given the use of Puppy Linux as a highly portable OS.

I have tried to find reliable info on how to measure speeds and how to improve file transfers, but the ideas I have encountered are too numerous to list here.


Thanks,

Steven

User avatar
neerajkolte
Posts: 516
Joined: Mon 10 Feb 2014, 07:05
Location: Pune, India.

#2 Post by neerajkolte »

Hi,
See this few months old discussion. I still use f3 for testing my flash drive and usb ports. Same drive sometimes give slightly different result on different port.
Next week I am gonna add USB 3.0 PCI card to my machine. Then I will test more.

Thanks.

- Neeraj.
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson

“We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.â€￾
- Amara’s Law.

stevielee
Posts: 36
Joined: Thu 26 Jul 2012, 02:51

#3 Post by stevielee »

Yes, I saw that discussion when I did a lot of looking on the subject. I usually try to do a decent amount of research before asking a question here. This problem seems to elude significant progress, much less a resolution. The speeds of 6-7 megs/sec is not too far from my own values which I arrived at using the dd if=/dev/urandom of=/mnt/sdX/tempfile conv=fdatasync,notrunc given at the Arch Linux wiki on SSD benchmarking.

Riight now, I am gravitating more to looking at deficiencies with the kernel/driver. I fear such a thing can't be resolved. The other good avenue of inquiry involves filesystem type, partition alignment, block alignment, and transfer block size.

With the ATTO benchmarking program, I get around 15 megs/sec in WIndows using the same computer, flash drive, and filesystem.



Steven

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#4 Post by mikeb »

bash-3.1# hdparm -tT /dev/sdb1

/dev/sdb1:
Timing cached reads: 210 MB in 2.00 seconds = 104.93 MB/sec
Timing buffered disk reads: 44 MB in 3.09 seconds = 14.24 MB/sec
bash-3.1#

testing a kingston 8gb flash stick.

I have had instances of badly configured or buggy kernels give poor drive throughput.

mike

stevielee
Posts: 36
Joined: Thu 26 Jul 2012, 02:51

#5 Post by stevielee »

Oh dear, I should have made it clear that I am looking for write speeds in particular. Naturally, I am more demanding and petulant when it comes to the harder of the two directions for data transfer...we don't do Linux because we're casual users :-) I can live with 20-25 megs/sec for reads, but the 6 megs/sec for my flash drive writes is driving me crazy. The measurements I have given in the previous messages apply to write speeds.

I have had values similar to the above for the three flash drives I have checked in both Puppy and Lubuntu.

Thanks for replies!

Steven

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#6 Post by mikeb »

ouch yes.... erm 1.1MB/s is the miserable figure with the same setup.... I will test other combinations.


mike

Keisha
Posts: 469
Joined: Tue 18 Nov 2014, 05:43

Read and write speeds on SanDisk Extreme usb3 stick

#7 Post by Keisha »

SanDisk Extreme (usb3) 32 GB usb stick on a usb3 port, formatted as ext4, tested under upup-3.9.9.1:

root# f3write /mnt/sdc1
Free space: 5.06 GB
Creating file 1.h2w ... OK!
Creating file 2.h2w ... OK!
Creating file 3.h2w ... OK!
Creating file 4.h2w ... OK!
Creating file 5.h2w ... OK!
Creating file 6.h2w ... OK!
Free space: 0.00 Byte
Average writing speed: 33.29 MB/s

root# f3read /mnt/sdc1
SECTORS ok/corrupted/changed/overwritten
Validating file 1.h2w ... 2097152/ 0/ 0/ 0
Validating file 2.h2w ... 2097152/ 0/ 0/ 0
Validating file 3.h2w ... 2097152/ 0/ 0/ 0
Validating file 4.h2w ... 2097152/ 0/ 0/ 0
Validating file 5.h2w ... 2097152/ 0/ 0/ 0
Validating file 6.h2w ... 115336/ 0/ 0/ 0

Data OK: 5.05 GB (10601096 sectors)
Data LOST: 0.00 Byte (0 sectors)
Corrupted: 0.00 Byte (0 sectors)
Slightly changed: 0.00 Byte (0 sectors)
Overwritten: 0.00 Byte (0 sectors)
Average reading speed: 197.30 MB/s

In the real world, write speed is faster than the 33.29 MB/s shown by f3write, and read speed is slower than the 197.30 MB/s shown by f3read.

I just copied a 1401 MB movie .avi from the (sata 6GB/sec) hard disk to the usb stick in 18 seconds. I issued the sync command in a terminal window as soon as the copy was finished and sync completed instantly. 1401 MB in 18 seconds = 77 MB/second, measured write speed.

Copy the other way, from the usb stick to the hard disk, happens in 13 seconds. 1401 MB in 13 seconds = 107.7 MB/second, measured read speed.

wboz
Posts: 233
Joined: Wed 20 Nov 2013, 21:07

#8 Post by wboz »

I can't speak to the differences created by different OS or file systems (like, at all), so I'll leave that to others. But I do have something to contribute despite that.

There are major differences in speed between flash drives. I only became aware of this a few months ago, when I started looking for a flash drive to back up photos (as an alternative to more-expensive backup hard drives).

I could go into why, but, I want to admit that I don't fully understand the function of SSD memory controllers. That notwithstanding, what I learned is, there are a very few USB flash drives which incorporate something like a SSD memory controller and operate at speeds similar to a low end SSD. There are flash drives with a simpler controller and lower quality memory which operate a quick-but-slower speeds. And then there's the generic, price-before-all flash drives that provide no technical details beyond the Amazon price of $8.99. Yes they will take your data and disgorge it later, but not quickly and not with a long life expectancy.

No-name drives have their place, absolutely. I run Puppy off of many of them in fact! (Unless you care a ton about boot speed, it doesn't really matter ...) For transferring files here and there, who cares, right? But the 2GB freebie drive you got from a networking event isn't going to be good at absorbing a month's worth of photos at a time. You don't need to go overboard, but you do need to spend a bit more than the commodity-drive-prices to get a flash drive that won't drag. I got a Mushkin Ventura Plus. And if you DO care about speed, there are some options above that too :). They're marketing 450MB/s for really high-end stuff, and last time I checked my admittedly low-end SSD can't hit that because Corsair swapped in substandard DRAM halfway through the production run (after the positive reviews were in from tech blogs!) and didn't change the part number ... :P

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#9 Post by mikeb »

Ok I did the dd test on various drives and noticed I was getting 1.1 to 1.4MB/s on anything including tmpfs ! I tried a real file and voila speeds jumped to expected..I guess /dev/urandom is not a fast source in this case....

retested with 20MB file ...
bash-3.1# dd if=/mnt/hda3/Moooovies/blender of=/mnt/sdb1/tempfile conv=fdatasync,notrunc
39746+1 records in
39746+1 records out
20350372 bytes (20 MB) copied, 3.23219 s, 6.3 MB/s

on the 8GB kingston

As for manufacturers/devices the sandisc 2GB one gave 2.5MB/s
both are fat32 format

For comparison the same test on tmpfs gave 54MB/s and IDE hard drive was 20MB/s ...this is a pentium 3 and a usb 2 card feeding a usb hub...so not a setup I expect to fly either but in this case the speed limit is still probably that of the usb device regardless. (and anything that may also limit speed in the kernel??)

And yes it appears kingston do the 'get the review then change the device but not the name' on their cheaper ssd boogie too.

hope that info is helpful

mike

User avatar
Fossil
Posts: 1157
Joined: Tue 13 Dec 2005, 21:36
Location: Gloucestershire, UK.

#10 Post by Fossil »

What of processor speed and that of RAM: does this in any way intrude into the copying speed?

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#11 Post by mikeb »

I would have thought that would affect maximum possible speed...eg my tmpfs test giving 54MB/s which means its unlikely to affect a device showing only a few MB/s significantly. That 54MB/s would also be affected by the source file read speed...an IDE drive..indeed probably the main bottleneck. /dev/urandom was definitely a major slowdown for me.

Actually repeated the IDE to tmpfs and got ~70MB/s....after the first time...drive spin up/loaded to cache perhaps.
I also get pretty mush the same figure doing a tmpfs to tmpfs copy....

RAM and CPU are way ahead of the speed of physical and flash drives.... a newer system with faster multi core/ram and sata/usb3 would still have a similar speed ratio.

back to main topic... seem to have a method of determining true usb flash write speed on linux... need something comparable for windows since I have not tried that yet....


Mike

User avatar
ETP
Posts: 1193
Joined: Tue 19 Oct 2010, 19:55
Location: UK

#12 Post by ETP »

May as well pull my own stats into this thread.
They compare smartmedia/HD/USB2/USB3/SATA2/SATA3/SSD
Please see end of this post:
http://www.murga-linux.com/puppy/viewto ... 145#768942
Regards ETP
[url=http://tinyurl.com/pxzq8o9][img]https://s17.postimg.cc/tl19y14y7/You_Tube_signature80px.png[/img][/url]
[url=http://tinyurl.com/kennels2/]Kennels[/url]

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#13 Post by mikeb »

Hmm defining a test is fun...I transferred a 700MB file to the kingston and was 30 seconds to finish and sync...thats hitting the 12MB/s limit of usb 2...

mike

gcmartin

#14 Post by gcmartin »

I had run some numbers on a laptop that I have running. There are conflicting numbers as, depending on which distro/OS I run, the numbers swing differently. Thus I cannot compute a confidence factor for the 3 USB devices just tested.

As such, we need an agreed to utility distro that is to be used to generate a set of values that aims to be a measurer of USB devices, HDD devices on bus-channels, where the distro remains constant and the devices are targeted to gather behavior. In some cases, specific scenarios need to be prescribed so that the results collected can have a degree of significance that applies to only the device behavior derived consistently to be a good valid measure.

Summary, something necessary for community agreement
  • We need a model to use as a test bed with a measurement utility for I/O storage devices. That model remains the same for all storage device results reporting.
    Or
  • We need a measurement utility that can be used to measure distro I/O behavior keeping the storage device the same while interchanging distros.
    Note: Recognize that these are measuring different things; namely the storage device(s), or, the system(s) I/O activity to a designated device.
Otherwise, the numbers as we are reporting are merely for a single specific case.

Hope this is helpful.

User avatar
LazY Puppy
Posts: 1934
Joined: Fri 21 Nov 2014, 18:14
Location: Germany

#15 Post by LazY Puppy »

mikeb wrote:Hmm defining a test is fun...I transferred a 700MB file to the kingston and was 30 seconds to finish and sync...thats hitting the 12MB/s limit of usb 2...

mike
If you really really want to know, how fast the your USB Flash drive is writing data, try copying 700 MB (or even more) of data in .doc Files (KB Sized) and report.

Jasper

#16 Post by Jasper »

Hi,

700MB/30s = 23.33MB/s which is not close to 12MB/s.

The effective use of a splash of logic with simple arithmetic might be of help to gcmartin, but "confidence levels" are for statisticians.

I find a usb 2 flash stick with a claimed read speed of 22MB/s and write speed of 6MB/s yields excellent results with apps and data that were moved to it from my running Multi-session-live-CD and then copied back with links.

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#17 Post by mikeb »

yep a quick piece of failed metal arithmetic.... I was 30s for 700 = 1400 for a minute then divided by 100 .....erm there are 60 seconds in a minute :D

Point was the figure is way higher then my original 6MB/s result.

This kingston 8GB is a recent item so I expect it will be a faster bunny.

700MB of tiny files... ok that would also test the file table speed as well as raw data writing.

So again, what is a suitable way to benchmark ? lots of small files, several large ones, what about the file system used, what about the host system?

A can of worms opened.

mike

Jasper

#18 Post by Jasper »

Another practical example: I have Quirky unicorn on a usb 2 stick that is reckoned to have respective read/write speeds of 15/6 GB.

It works super-fast and it shuts down quickly. However, a much faster stick would yield a big improvement in back-up time.

Once I've purchased/have any particular stick (and makers rarely publish speed data) it's only my cans/usage that matter to me.

PS In my first example (in my previous post) I back-up that stick partition using Snap2 in mirror mode so that speed is always excellent and a faster stick would be of negligible help.

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#19 Post by mikeb »

I believe the point of the thread is to determine if there is a problem with linux/kernels in relation to usb write times.

It also could benefit generally by collecting information so that potential users can make informed choices of what devices to use before buying based on actual results rather than sales hype....perhaps a fresh thread may make sense for the latter.

Mike

stevielee
Posts: 36
Joined: Thu 26 Jul 2012, 02:51

A response...

#20 Post by stevielee »

I appreciate the enthusiasm for the subject! I have been away and have not been able to respond. Thanksgiving is coming up, and I may not get to respond for a few days after this. So, I will try to get my ideas in this installment.

Also, soon after I began this thread, I was looking ever more deeply into the matter and ran across the flashbench program - a command line utility written by (I think) a Linux kernel developer. I then tried it on a couple of my flash drives......

I failed to appreciate that it appears to write directly to the device rather than via the intermediary of the filesystem on said device. The program cut into the FAT on the drives. Now I have a damaged 32 GB drive and am rather burned on all things consumer electronics. Before I get look into the topic of data recovery, I would like to make some progress on the subject of the thread here....but for any amateurs out there, be careful with the flashbench program.



My main irritation with the flash drives in Linux is that they seemed to be a good deal slower than in Windows. I don't know for sure if that is true, but the little bit of benchmarking I have done in Windows suggests that it is so. Nir Sofer's 'Nirsoft' site usbspeed.nirsoft.net has a page for reporting speeds, but is rather vague. Phoronix has done a more reliable test of several drives, but they results suggest a wide range of speeds. I was hoping to see if there is general agreement about Windows performing better in this regard. If there is a difference, then it would be nice to see if we can find out the cause so as to get the most out of our hardware.

My best guess is that the big influences - on the software side - on write speed are device drivers, filesystem drivers, and parameters such as device block size, block size of the writes, and alignment of the files, partitions, and other elements of filesystem structure.

Perhaps it is the case that WIndows has optimized these software elements such that the same flash drive performs better there than in Linux. Maybe Windows does this, as is often the case, in a way that gives no hint to the user as to what is really happening. This assumes that performance actually is different with respect to choice in the OS.

This brings me to the need to reliably measure write performance. Gcmartin's interest in a standard method makes sense, but how does one choose? I am still looking into that question as well. Using the simple SSD benchmarking method I mentioned above gives answers that vary significantly from one run to the next. The nature of flash memory seems to be rather volatile. A write to the drive may need to erase some parts of the memory and not others. The write may reveal conflicts with the erase block sizes and boundaries of such blocks. This could be convolved with the size of the blocks of the write.

I am a beginner in the subject of personal computing, but my impression is that the concept of a flash drive's speed is very much a moving target. Characterizing the write speed may require a description more sophisticated than a single number. Perhaps it will also require a procedure to let the user get the most of the device.

As to hard numbers, I was more interested in USB 2.0 as that is what I am forced to use. I may try to go with an external HDD as I have seen a lot of convergence around 20MB/s as the typical write speed. Using the /dev/urandom device was not a problem for me. I wrote the random file to the drive and there was no problem. I then put the file in the home or root directory ( that is, /~ ), which is essentially putting it into RAM (I was running Puppy off a flash drive), and got about the same write speed.

Mike's reports of problems with the /dev/urandom device may be based in his ancient Pentium III processor. My reason for suspecting such is because I use one as well! It creaks badly - especially with its accompanying USB 1.1 ports. The write speed tests I conducted were done with a Core2Duo.

But I have tired of the pokey hardware....I am committed to getting an i5 Haswell soon...or whatever the upcoming Intel generation is.

Post Reply