The Internet age has exponentially increased the volume of digital media that is being shared and distributed. Broadband Internet has made technologies
such as high quality streaming video on demand possible. Large scale supercomputers
also consume and create huge quantities of data. This media and data must
be stored, cataloged and retrieved with high-performance. Researching high-performance
storage subsystems to meet the I/O demands of applications in modern scenarios
is crucial.
Advances in microprocessor technology have given rise to relatively cheap off-the-shelf
hardware that may be put together as personal computers as well as servers.
The servers may be connected together by networking technology
to create farms or clusters of workstations (COW). The evolution
of COWs has significantly reduced the cost of ownership of high-performance clusters
and has allowed users to build fairly large scale machines based on commodity
server hardware.
As COWs have evolved, networking technologies like InfiniBand
and 10 Gigabit Ethernet have also evolved. These networking technologies
not only give lower end-to-end latencies, but also allow for better
messaging throughput between the nodes. This allows us to connect
the clusters with high-performance interconnects at a
relatively lower cost.
With the deployment of low-cost, high-performance hardware and networking
technology, it is increasingly becoming important to design a storage
system that can be shared across all the nodes in the cluster.
Traditionally, the different components of the file system have
been stringed together using network connections.
The protocol generally used over the network is TCP/IP.
The TCP/IP protocol stack in general has been shown to have poor performance
especially for high-performance networks.
In this dissertation, we research the problem of designing
high-performance communication subsystems for network attached storage (NAS)
systems. Specifically, we delve into the issues
and potential solutions with designing communication protocols
for high-end single-server and clustered server NAS systems. Orthogonally,
we also investigate how a caching architecture may potentially
enhance the performance of a NAS system. Finally, we
look at the potential performance implications of using some of these designs
in two scenarios; over a long haul network and when used as a basis for
checkpointing parallel applications.