NAND Flash 101: Enterprise SSD Form Factors Simplified

By Louis Liao | Aug 23, 2021 | All, NAND Flash 101, Technology

Today’s enterprise data centers are constantly faced with the unique challenge of balancing power, speed and cooling efficiency with limited space in the server rack. When it comes to storage, the need to find the right balance is even more important. One of the key obstacles to this balance however is form factors. Many traditional form factors were designed for Hard Disk Drives (HDDs) not Solid State Drives (SSDs). As such, they lack the cooling efficiency and storage capacity required for large data centers. Below is a NAND Flash overview that discusses these challenges and offers solutions to help with decision-making in your data center.

NAND Flash

NAND Flash is a type of non-volatile memory. As such, it does not require constant power to retain stored data. Just like HDD, the data will not be lost when power is removed. Additionally, NAND flash has faster read, write and erase speeds than typical HDDs. They also have greater areal density and they consume less power than HDDs.

Current challenges with enterprise storage

With the emergence of Solid State Drives (SSDs) for consumer technology, enterprises saw the benefit of switching from traditional Hard Disk Drives (HDDs) to SSDs. The benefits to SSDs include:

- - More durable than HDDs
  - Faster transfer speed
  - More powerful and energy-efficient
  - More practical in size

Despite these benefits, switching to these more efficient drives does not come without challenges. The form factors for these drives present unique challenges for enterprise data centers. SSD enclosures come in 2.5-inch and 3.5-inch form factors which are the same size as HDDs. This makes it easy for companies to transition to SSDs without significant modification to the infrastructure.

However, the mechanical design of 2.5-inch & 3.5-inch enclosures was originally designed for rotational media such as HDD. As such, they block the airflow and constrain the scalability of SSD. These challenges fostered new technology to better meet the needs of the enterprise data center.

NAND Flash: The M.2 form factor and its considerations

The M.2 is a NAND flash form factor (formerly known as the Next-generation Form Factor) and is an internally mounted expansion card that plugs directly into the motherboard using a M.2 slot. M.2 was initially designed for mobile platforms and consumer devices, where space was limited. Thus, the small size of the M.2 makes them ideal for enterprise servers.

Challenges with M.2

One of the key requirements of a data center is the ability to hyper-scale. Thus, storage capacity and thermal optimization are critical. Unfortunately the size of M.2 limits capacity scalability and cooling efficiency. In addition, M.2 is not hot-pluggable so it is hard to maintain the system during runtime.

NAND Flash: The evolution of add-in-cards (AICs) as PCIe SSD form factor

With the advancement of technology, the need for speed and power in computing has risen over the years. Add-in Cards (AICs) also known as PCIe cards (Peripheral Component Interconnect Express) accommodate these advancements by providing an interface to plug components directly to the motherboard via a PCIe slot.

PCIe cards increase bandwidth

A PCIe card can offer higher performance than M.2. However, M.2 utilizes x4 slots, whereas the PCIe card can use x8 or even x16 slots to increase bandwidth.

Optimized cooling efficiency

SSD with an AIC form factor is more suitable for high-performance use cases. (Figure 3) PCIe cards also offer a larger area to place more NAND Flash packages while optimizing cooling efficiency. Many vendors provide PCIe Card SSD solutions with heat sink and blower fans to keep the temperature low and make sure SSD running at full speed.

Downsides to PCIe cards

PCIe cards are typically installed inside the server chassis, which makes them inconvenient to service. Servicing this type of card would require you to remove the cover of the server to access the device.

Although these cards offer higher performance, you’ll have to give up one PCIe slot. Given the limited availability of slots in the rack, they are typically reserved for devices with higher bandwidth requirements than SSD (graphics for example). As such, SSD with AIC form factor would be installed only when the workload requires such high performance.

NAND Flash: EDSFF as the next generation form factor

Although the above NAND flash form factors have their advantages, they are not optimized for enterprise data centers. Hence, a new family of form factors for SSD was born called EDSFF. EDSFF stands for Enterprise and Data Center SSD Form Factor. This form factor was specifically designed for enterprise and data center applications. This family of NAND flash form factors consists of several variants: (Figure 4)

- - E1.S
  - E1.L
  - E3

The goals of EDSFF include:

- - Delivering more power
  - Allowing hot-swapping
  - Increasing storage capacity and density per rack unit
  - Reducing cooling cost

What problems does EDSFF solve?

EDSFF solves the problem of maximizing space in the server rack while allowing for cooling efficiency. It also supports scalability for future expansion. With SFF-TA-1002 connectors, EDSFF can support PCIe x4, x8, x16 lanes and up to 112Gbps signaling.

EDSFF is ready for the future

Given these features, EDSFF can support PCIe 4.0 & PCIe 5.0 and we can expect EDSFF to be compatible with PCIe 6.0. In addition, EDSFF became a NAND flash form factor standard in SNIA and was adopted by OCP. Every major SSD vendor supports EDSFF and launches SSD products with EDSFF. Phison is also developing enterprise SSD products with EDSFF that will be launched soon.

EDSFF E1.S

E1.S is the most popular among the three variants in the EDSFF family. Its mechanical design is to replace the data center use of M.2 drives. (Figure 5) It offers higher density and higher power while retaining a small size.

The gum stick-shaped E1.S can fit vertically in 1U space. A 1U storage server can support up to 32 E1.S storage drives.[1] This allows the data center to scale storage capacity and support data and application growth.

Owing to the 12V supply, E1.S can accommodate a higher power level than M.2 and save the voltage regulator from 12V to 3.3V. However, a higher power level may induce SSD to get hot especially when SSD is running at the highest performance. Thus, E1.S reserves mounting holes for five different thickness options of heat sinks to improve cooling efficiency and decrease the required airflow. (Figure 6) 5.9mm thickness with bare PCBA and 8.01mm thickness with heat spreader is suitable for use cases with lower power consumption.

For power consumption up to 20W, a 9.5mm thick symmetric enclosure can be used and which makes the device look like a shortened version of E1.L. For an even higher power use case, an asymmetric enclosure like 15mm or 25mm thickness can be applied to dissipate heat more efficiently. Density and cooling efficiency always contradict each other. Microsoft Azure believes a 15mm asymmetric enclosure is a good balance between 9.5mm and 25mm thickness. 9.5mm thickness does not offer enough cooling whereas a 25mm enclosure sacrifices too much space.

EDSFF E1.L

Formerly known as ruler, E1.L is designed as the shape of a long ruler and is optimized for 1U server. The maximized board space can support more NAND Flash packages and enhance cooling efficiency.

Typically, we use U.2 form factor for large capacity drives but sometimes the large capacity U.2 drives are built with two PCBs folded over. The components on the inside are like in the middle of a sandwich and the heat from those components is hard to be cooled down.

E1.L solves the thermal issue by spreading all the area to the surface along with two thickness options of the metal enclosure. (Figure 7) It only requires 55% less airflow in comparison to the U.2 form factor and the cost of power is reduced significantly as well. [2] E1.L is the densest storage NAND flash form factor. A single 1U storage server full of E1.L drives can reach 1PB capacity. Enterprise data centers can use E1.L to enhance density while lowering the Total Cost of Ownership (TCO).

EDSFF E3

The biggest difference between E3 and the other NAND flash form factors in the EDSFF family is shape. It looks similar to the traditional 2.5-inch form factor but allows up to x16 PCIe lanes and 70W power.

E3 replaces U.2 form factor

The E3 was created to replace the 2.5-inch U.2 form factor with the advantages U.2 doesn’t have. The similarity of shape enables an easy transition from U.2 to E3. Some server vendors provide chassis solutions, which can support both U.2 and E3, and it allows the data center to transfer the storage devices from U.2 to E3 gradually rather than replacing all U.2 devices at one stroke.

E3 sizes

Optimized for both 1U and 2U servers, E3 comes in four different types of form factors. E3.L, E3.S and either in single or double widths. (Figure 8) Industry calls single-width as ‘1T’ and double widths as ‘2T.’ T stands for thickness. 2T is usually used for a high-power device with more heat generated such as computational storage and high-capacity storage due to better heat dissipation.

Flexibility with the E3

E3 is also flexible from the inside. A small-sized E1.S PCBA can fit in an E3 enclosure, thus low-capacity storage with E3 form factor could leverage E1.S PCBA to lower the cost. (Figure 9) Moreover, E3 supports not just SSD; it can support multiple device types like persistent memory, GPU, NIC, etc.

The future or EDSFF

EDSFF is quickly gaining more adopters. This versatile form factor meets the data center’s need for density, power, capacity, performance and cooling. As such, SSD vendors alongside server providers are building newer infrastructures to accommodate EDSFF.

Although there are some challenges for transition to EDSFF; for example, the server chassis should be redesigned to fit in with EDSFF. It is estimated that EDSFF will be deployed widely in the future and take over obsolete 2.5-inch and M.2 form factors. (Figure 10)

Optimizing storage space while balancing power, speed and cooling efficiency requires enterprises to give more attention to storage drive form factors. While there are many available, only a certain few are ideal for use in the data center. The EDSFF is the next generation in storage form factors that take up minimal space while still providing high density and high-power storage options.