Understanding SSDs vs HDDs vs USB Flash storage

[gcmartin]

HDD storage (and spinning disks, in general) have been used for centuries, per se. But, the landscape for data storage of all types, sequential/transactional, has seen the recent decade's entry of SSDs and USB Flash storage (this I group into a category I call solid state storage). These are devices which uses non-volatile RAM as a storage media to PCs. Even as this is new to the PC world, this type of solid state I/O storage is not new in the IT world as mainframes have had these available for over 35 years with dramatic system workload impacts where needed. The same kind of workload benefits is seen in the PC world with use of the SSDs and USB flash units.

SSDs, today, intend to be installed permanently, while USB Flash intend to be hot pluggable giving them portability. Yet both of these use non-volatile RAM with their differences, chiefly, is the technology to connect to our PCs. RAM storage will be referenced as solid state storage in this post. S
Edited: Some late news on SSDs are discussed/shared here in this Puppy forum.

Using this kind of storage devices, just as was the case 3 decades ago, the same device data storage problems exist in today's PC arena. Device utilization and I/O channel utilization have an increasing impact, over time, as individual SSD/USB flash device use increases in the PC's operations. To address device problems, the units have intelligence on the units to manage these traditional problems. To do so, units require space on the unit to be free. This available space's use is called "over-provisioning". In some early PC drives this is made available by the administrator at installation-setup time, manually. And, in the past, many users have unwittingly allocated ALL solid state space to writable partitions without understanding their requirement to leave free unallocated space for over-provisioning. Over-provisioning is NOT required for HDDs. Yet,the over-provisioning aspect of solid state storage is an important function in data management and for maintaining data integrity. Over-provisioning is a reserve of free AND unallocated AND unpartitioned space which is available so that there are replacement storage blocks available when actual data storage blocks on SSD/USB become defective. And, some solid state storage units use this space for "TRIM and Garbage Collection" which is done in a different manner than is seen on a HDD. In a HDD, when a data block is freed, it is marked and can directly be reused. NOT SO in the SSD world, as, the data block must be emptied clean when it becomes available BEFORE any new data can be written to the free block. This is NOT a problem as is done as a background operation on the solid state storage when no I/O operations are active.

So question: "What do you think happens when you have lots and lots of R/W operations occurring at a busy time where the SSD has run short on actual clean blocks to write knowing that you have blocks on the device which are free but have not been cleaned? Yes, you guess correct; you must, either, stop all I/O activity to clean blocks before continuing or you will employ some kind of method to clean then write, then clean more then write, then ... one block at a time. As you can see, THIS is detrimental to the solid state devices performance and your system is sitting having to wait for this.

Now that we understand why this free unallocated space is needed, how much should be free for its use? I have seen recommendations over the 3+ decades from 15% to 40% of free space, when manually done. These recommendations vary. In the case of newer modern SSDs/USBs, the manufacturer does this for you by leaving as much as 10% untouchable or unseen by the OS for the device's management needs.

For optimum reliability (and to a lesser extent, performance) SSDs require some free space, which, without going into too much detail, they use a writing method for spreading data around the drive to avoid constantly writing to the same place (which wears them out). Most everyone has heard that you do NOT want to write (or read either) too many times to one place on these devices. Just as was the case decades ago, the same is true with these types of storage systems in today's world. The solid state storage industry builds in a feature called "load-leveling" which works as described. That is, it spread data all over the device so as not to prematurely wear out individual blocks by attempting to even disperse data blocks use. When the devices are new, no blocks were ever written to, so data file(s) is usually written contiguously to SSD/USB blocks. As time goes on and every block has been written to, at least once, load-leveling is at work to spread new data around with fragmenting of files being a normal behavior of the device as load leveling operates. Normally, for most every workload using solid state devices, this does NOT cause system/applications problems. But, for some workloads, this is detrimental even though these devices are "direct access" devices going directly to its RAM for the actual block of data needed when reading or writing (this can be seen where some system elements may require data to be contiguous during boot use, etc).

HDDs have traditionally had this problem of fragmentation, too. But unlike solid state storage, they do NOT have a load-leveler at work and in many, many more cases, data will NOT be fragmented excepting due to some PC I/O subsystem decisions. Fragmentation of files on a HDD means that the HDD must use its mechanical means to spin and move arms to find data locations for files which became fragmented across the drive. This can and does have significant impact on workloads accessing HDD data as timings are affected waiting for data to be plucked and presented from the spinning platters. SSDs are memory devices, in normal use (not the special cases mentioned earlier) is NOT impacted as it finds its data using a direct location means where the time is always the same no matter if its the 1st memory block on the unit or the last. By comparison, a single data block's access via SSD might be 0.1ms while the same data reference on HDD may be 15ms; an ENORMOUS difference.

Thus, these are items you MUST be marginally aware of as you expect your solid state storage to be used in a fashion where you understand why your system may/may-not slow at times in the use due to the unit's internal management; and, how the device actually operates with your data usage. Solid state storage devices have brought us much beneficial use. Along with it, we have a responsibility to understand its behavior.

Most everyone of us have been around HDDs for so long that we understand HDDs.

Hope this helps the few in better understanding this PC storage.

Contribute to this thread as you see appropriate. All contributions for use and understanding is welcomed.

[gcmartin]

Intel's SSF vs Samsung's (and others) approach.

FYI

[8Geee]

I would also like to add a few paragraphs about SSD's.

In the beginning, SSD's were Single Level Cell (SLC), a fast and very stable (reliable) device with Mean Time Between Failure (MTBF) in the 1 million re-write range. These were expensive (The eee-pc 8G has an 8Gb SLC that reportedly was worth 25% of the final cost). Price considerations meant new Multi-Layered Cell (MLC) came to market. These, though cheaper had re-write limits of only 100,000 cycles. Further, 4 levels were employed, that slows these devices per clock. Recently, these have been 'upgraded to 2-level, without much cost , a nice speed gain, and with some re-write gains. Naturally, the data-pipline has markedly improved in terms of speed from 50Mb/sec in 2008 to 300+Mb/sec today.

Speed is usually the limiting factor of life, since the faster a drive can be re-written, the less time the drive will live. Consider an SLC SSD clocked at 50Mb/sec and a new MLC that clocks at 200 Mb/s, and also include the number of re-writes before failure. sec/50Mb * 1million and sec/200Mb * 0.2million are compared using 1000Mb drive-size (so-called per gigabit life). The SLC is 20 seconds to write 1Gb, and can do this 1million times. The MLC is 5 seconds to write 1Gb and can do this 200,000 times. The life of the SLC is 20million seconds per Gb, and the MLC is 1million seconds per Gb. Even if the write speeds are equal, the SLC has a 200Mb/ sec rating five times longer than an MLC device.

We can consider USB-flash to be MLC speed/10 and MLC re-write/10. That paints a rather cheap picture of a 32Gb USB flash-stick as compared to a 32Gb MLC SSD. Just remember 20,000 cycles!

[wboz]

Are you certain that the faster the solid state storage can be written, the fewer times it can be written?

I understood that the limitation on lifespan was a function of how many times each location is written to.

In testing, SSDs always end up having a lifespan that exceeds any reasonable use case. Yet I hear all the time about phone flash storage (invariably eMMC) or SSDs that have corrupted data ... or problems with deleting files that doesn't actually free up space. I think the core problem is that manufacturers view solid state as a commodity, and they tolerate flaws that the true "storage" customer wouldn't put up with.

I absolutely buy the use of SSD as boot and medium-term storage. But I'm about to add a HDD for my desktop because I think it's more likely to preserve my stuff long term. I used to say that "I have never had a HDD actually fail" but it actually isn't true ... I have. They've just given warning because of the mechanical noise! Maybe that's why SSD upset people. You never know when it's going to go.

[bark_bark_bark]

Hard Disk Drive Master Race!

[gcmartin]

Opening post is updated as there have been 2 important announcements of technology moves for our awareness. It is expressly remarked in this post.

[corvus]

I believe that the following article don't leaves doubts on the reliability of SSDs.

https://blog.korelogic.com/blog/2015/03 ... age-issues

Thanks to all who participated in this discussion it has avoided me from making a mistake.

Regards

[natgab]

I know that Puppy runs in RAM, but we usually have it installed on a USB.

But how much read / write does the USB actually get? I dual boot with Ubuntu on my HD and a mini USB always plugged in with LxPup. I was just wondering if I could install Ubuntu on the USB, would I damage the USB from Ubuntu running from it? I have an 8GB USB, would a bigger one help?

I know a live disc can run off a USB, but this is usually for a short time and then it is installed to a HD ( mechanical or SSD ). What makes the memory in the SSD better than the one in the USB.

Sorry if it is in the explanation but I could not figure it out.

[musher0]

@gcmartin: Interesting and informative article, gc. Have you thought of
publishing it somewhere else as well?

@all: SSD's speed makes them tempting, but can you trust them for
long-term storage? I mean as compared to HDD's and DVD's, for example.

If your ultra-fast SSD gives up the ghost after six months, that's a lot of
money -- and data -- down the drain. Just a thought.

Also, how well do SSD's handle disk-intensive jobs, like archiving? Should we
not stick with HDD's a little longer? Tried and true, you know!

BFN.

musher0

[p310don]

For what it's worth, my real world experience....

I have a PC with 7 hard drives in it. Puppy is on the SSD, the rest is old fashioned spinning disks.

I run a frugal install, done to the SSD, so I get great boot up speeds. My save file is 6gig in size, which gets a FSCK at boot. Previously, on the rotating drive, a FSCK on the save file took nearly a minute, now on the SSD it takes a few seconds.

Puppy, due to it's differences is ideal to use with an SSD. I have the OS and all associated SFSes and save file on the SSD. I also make a backup on a regular spinning drive. The SSD gives me the speed benefits, but I have the save file there on the traditional drive just in case of failure. (one of the many benefits of frugal!)

Having said all that, I have owned an SSD for about two years. I bought it at the same time as buying 2x 2 Terabyte spinning disc drives. One of those has failed. The SSD is as good as new (I hope).

[musher0]

Hi, p310don.

Thanks for your input. It's always good to know an individual experience.
Main system on SSD with back-up on HDD. Sounds logical!

But I was wondering if there is more general comparison data for life-spans
of SSD vs HDD. SSD's probably too "young" for that, eh?

BFN.

musher0

[Pelo]

I have been using pendrives since 4 years now. Some dead, mainly because of bad quality (Platinum) or mechanical breakdown (pins).
I no longer buy retractable USBS.
None is dead because the number of writings. It's my personal experience.
No conclusion for other users and what they do with their own pendrives.
The smaller the storage USB is, the more i Prefer. Unfortunately it's hard to find nowadays 2GB models.
Nothing against HDD storage, but you need to keep it well organized to find what you are looking for quickly.
In my opinion too, but not only, access to the pendrive is faster than to the HDD. That true when you start your session, and false after. To be discussed. The bigger size the storage is, the longer the computer searches.
I backup my HDD on a 16GB USB (Windows mainly)
I keep Puppy with the policy nothing on HDD (there are exceptions) . No boot or frugall install on HDD (there is no exception). All my distros are booting from the pendrives.

[Pelo]

SSD is nice see last information from Musher0 le forum Puppy de France

[gcmartin]

There are some VERY important aspects of SSD that YOU must be aware of as it affects whether it will "connect" in your system and the performance that you will get.

Connections to PCs of the last decade:

• SATA
• M.2
• U.2
• NVMe
• AHCI

With Intel's processor dominance, cores and lanes are a factor in the benefit you will get when using any SSDs with Intel motherboards. There are similar concerns for use of the "connects" on AMD motherboards.

Here is a performance report and a forum thread here targeting 2 of the connects pitting Intel vs Samsung SSD drives. There are other articles too, attempting to bring to light those particulars for helping sort thru important aspects that affect your decision to add this to your PC(s).

When planning on a purchase decision, "Its a bit more than just a simple purchase", as was the case with older HDDs.

And if you merely want to reduce your decision to just AHCI or NVMe, this document might provide help as it can simplify difference understanding.

Hope this is helpful as all of this closes in on Intel's NAND change in SSDs, which will significantly improve the life expectancy (the write expectancy) on data on drives, rumored to be coming this year.

Hope this helps