RelevantIT

IT As A Service - ITAAS

Technology continues to evolve and both technology professionals and luddites continue to evolve with it.  This means that they way we use technology changes.  Changes cause us to either adopt new technologies into our existing operations to take advantage of capabilities that were not there before, or restructure operations to take advantage of new opportunities and new efficiencies that were not possible before.  Adopting iPhones from flip phones was an integration of new technology into their existing process. Adoption of Apple TV and Netflix caused changes in behavior.  Most obviously, that there are no more trips to the video store. More subtly the way content is consumed has changed, several episodes of a series are consumed consecutively, rather than piecemeal spread out over time.  Interruptions invoke the pause button rather than waiting for a commercial breaks.  The previous evolution of TiVo, quickly meant that shows that couldn't be time shifted suddenly became inconvenient, commercial breaks that couldn't be fast forwarded were a nuisance.  In short live TV has become a pain to watch.

The introduction of cloud has introduced a new variable into the technology equation.  Businesses have a choice on where they procure services.  IT infrastructure can be built out to provide services to support the needs of the business or services can be purchased for a cost from a cloud provider.    The appropriate solution depends on the needs, of the business.  How quickly does the business need it? How long will it be needed for?  What are the costs?  What information will be involved? What are the security concerns? 

Cloud has the advantage of being extremely elastic.  Services can be rapidly allocated and deployed, and when they are no longer useful they can jsut as quickly be disbanded. To the end user the cloud hides all of the underlying layers of complexity and replaces them with a shopping cart.  Ordering IT services is now as easy as buying a book online.

The business demands to be able to roll out applications and services much more quickly.  For comparison look at how software updates cycles have shrunk, to where apps on your phone are now updated weekly.  IT needs to be able to quickly provision services to meet the needs of the business in real time.  The model for IT infrastructure is now changing to one of pool based resources where slices of these resources can be allocated on demand, and applications can be spun up in minutes.  Virtualization is a key enabler of this evolution, but better management and operations tools are critical to removing the underlying layers of complexity.  With these tools it is possible to provide the business with a shopping portal for IT services, where they can immediately access these resources and spin up applications.  In the date center there are pools of compute, storage and networking resources being provisioned to enable this process, but it is entirely automated.  Charge back tools charge the business units for the resources they use.  Fee for service discourages IT sprawl, services that are no longer being used and legacy applications are disbanded.  Compared to physical systems, which remain in place consuming space and resources until they are fully depreciated and can be removed.  In this model we treat IT resources as we do the gas tank in the car, we fill it up with what we need and when it starts to run low we add more.  We don't stop at the gas station for every trip to the store, or commute to work. 

What does this mean for IT professionals?  Our own internal customers will source their needs from the provider that makes the most sense for the business.  With the introduction of Cloud, IT customers now have a choice. It is critical for internal IT to be able to meet those needs of their internal customers.  Growth in scale of IT is leading us to manage resources as commodity services to the business based on the SLA they provide.  Stovepipes of infrastructure are being replaced with compute, storage, and networking as a service.  Changes in the density of the technology are making this necessary. As we approach multiple petabytes of data we can no longer manage individual gigabytes of data. Upfront allocation methods that pre-allocate based on anticipated need are inefficient and lead to wasted resources.   Implementing pools of storage that can dynamically grow or shrink depending on the needs allow for higher utilization.  Hardware is evolving toward denser footprints, which require less management.  Resources are being offered in pre configured blocks, or a datacenter in a box.  Fueling this trend Intel is developing servers on a chip, and HP is developing a server down to the size of a disk drive and can mount them in trays similar to the ones used for disk drives.  At this scale the need for IT expertise in storage, networking, and servers is as critical as ever, but we can no longer effectively micromanage each individual component. That granularity of management requires the use of tools and automation for the infrastructure to grow out cost effectively at scale, and for internal IT to be able to offer the same services and response time of the cloud. 

 

StorageGuru.org is now RelevantIT.org

I have re-branded StorageGuru.org to RelevantIT.org.  This change reflects a broader change within the industry  as we start to look at technology not as a collection of individual components, but as pools or resources from which we can offer services.  While storage is no less relevant during this transformation, I believe that it is important to broaden the topic to better understand how this component fits into the ecosystem as a whole, and to explain how the ecosystem is evolving.  Advancements in technology continue to evolve IT. I look forward to exploring changes that are relevant to the ongoing Evolution of IT, and how IT will stay relevant to the needs of business. 

Big Data: How Target Knows You're Teenage Daughter is Pregnant Before You Do

Most of us are aware that as we move around the web we leave our footprints behind in the grains of silicon sand.  Web sites leave tracking cookies in our browser, Gmail indexes our email, Google search results feed the ads we see.  Twitter, Facebook, and Linked In are public and directly linked to our user ID.  We are accepting of this in return for free email, better search results, stores that remember what we had in our shopping cart, and for having interactive social media.  

 

When we move about IRL (In real Life) we assume that we are one of a faceless mass assuming that our identity is lost or obscured in the volume of the crowd.  In reality even our offline activities leave digital crumbs behind. Individually these crumbs are meaningless like an individual piece in a puzzle.  As more of the pieces are collected, we are able to generate a more complete picture.  The challenge is amasing all of these individual pieces finding a way to store and then process them into meaningful information.  

 

Recently we have seen huge advances in the ability of analytic systems such as Greenplum to process massive quantities of seemingly unrelated information to determine trends an correlation in what was previously assumed to be unrelated data.  Big Data offers us unique opportunities for us to process enormous quantities of data for Genomics, oil and gas exploration, and also for us to understand people.  Companies are realizing tremendous opportunities to better understand their customer.  Political campaigns are using this data to better understand their constituents zip code by zip code and modeling the impact of embracing one demographic of voters while alienating another demographic.  

 

The New York Times has a excellent article detailing how Target Corporation leverages Big Data analysis of it's customers. They have focused their marketing around potential customers that are experiencing major life changes such as having a baby.  Based on credit card data and purchase history they are able to build highly detailed profiles of their customers that can indicate when a baby is on the way and even the expected due date.  This worked so well their model predicted a teenage girl was pregnant before her father knew.  This article is so well written that to fill you in on all the relevant details would we outright plagiarism, so I'll just provide the link 

http://www.nytimes.com/2012/02/19/magazine/shopping-habits.html?_r=1&partner=rss&emc=rss&pagewanted=all  (Febuary 15th, 2012)

This topic has since been picked up by Time magazine and Forbes.

 

Big Data has enter politics as well.  Demographic information on voters enables political campaigns to figure out which voters will respond to specific messages and the impact of those messages on election results.  Information is used to weigh the impact of alienating a certain demographic of voters in a zip code vs. attracting a different demographic of voters with a different viewpoint in multiple districts.  For example by aligning himself with voters making less then $50,000 a year might pick up and extra 10% of the votes in that demographic, but loose 10% of voters making over $100,000 and loose 15% of the voters making over $200,000.  Big data allows the campaign to calculate the impact of loosing or gaining these voters in key swing geographies by analyzing income by zip code, the number of voters in the income brackets in those zip codes, and their party affiliation.  This allows the campaign to target winning the electoral college votes required, even if the popular vote margin is razor thin. (NY Times Febuary 15th, 2012   http://www.nytimes.com/2012/02/19/magazine/nate-silver-obama-reelection-chances.html?_r=1&ref=politics&pagewanted=all  )  

The Shifting Economics of End User Computing

n the mid 80's when we started to get the first computers at school and early adopters bought them for their home or small business they were machines costing thousands of dollars and something to be revered and respected.  The first PC we had in our home was a WSYE x80-386 With 2MB of Ram and a 40MB hard drive and a monochrome display circa about 1985.  The PC cost $3,000 or adjusting for inflation $6,000 today.  This was back when you washed your hands before you used the computer and didn't dare eat or drink coffee while using it for fear of damaging the equipment.  A plastic dust cover was often placed over the equipment when not in use.

 

Fast forward to 2012 and consumer expectations of compute technology has changed dramatically.  Devices are expected to be scratch resistant and water resistance is the biggest new selling feature.  We want to be able to sweat all over the device at the gym, steam it in the bathroom while we take a shower, and survive the drop in the toilet that has ended so many smart phones.  At the current pace we should expect a mocha latte resistant device with an iCoaster App to appear at the local Starbucks sometime soon. 

 

End users have a cost expectation for a new consumer device that the hardware should cost between $0-$500 and up to a maximum $1,000 on the ultra high end.  there is increasing resistance toward any device that cost more then a toaster.  The expectations for software have changed radically as well.  Software has shifted from expensive application suites with an exhaustive features to single purpose Apps.  the price expectation for an App is $0-$1 with some very expensive applications costing $5-$15.  

 

In the two decades since the 80386 made the consumer PC popular we have seen exponential growth in processing power and memory growth.  Moore's Law dictates that the capabilities of Processors and memory double every two years.  Today mainstream processor and memory chips have more capacity then many end users require.  Instead of introducing the latest and fastest chip many consumer devices focus on smaller low performance chips that consume less power.  It's worth noting that even these low power chips continue to evolve and double every two years.  Low power hardware has ushered in an era of much less expensive devices.

 

Many of these low power devices work on a Just Enough Compute model, single tasking an active application such as email or web browsing and putting the inactive application to sleep in the background. Due to their small screens and simple interfaced many of these low power devices are ideally suited for single tasking.  The end user perception of these devices is enhanced by their ability to leverage cloud based computing services. This allows the device to complete resource intensive tasks by offloading the processing to the cloud.  For example voice activated SIRI listens to the user on the device and then uploads a recording of the request to the SIRI servers where the voice recognition takes place and then returns the response from the server.  I use an application to scan business cards.  My phone takes a picture of the card and uploads it to the Apps cloud where the text recognition takes place and my phone receives the persons name, email address, and phone number ready to be imported into my phone book.  

 

Cloud offload is aggressively used by the amazon Kindle Fire. the fire offloads much of the processing for simple tasks such as we browsing and video to the Amazon S3 cloud.  The result is a very inexpensive piece of hardware, that is light weigh and gets good battery life.  The true cost of running the device is shielded from the end user who never sees the warehouse of servers and hardware that makes all of this possible.

 

Many cloud applications such as Google, Gmail, and Facebook offer their cloud services to the end user for free, in return for inputting their information and allowing the cloud provider to analyze their data and provide metadata to 3rd parties.  In this business model the cost of development, support, hardware and the true economic value for these applications is completely hidden from the end user.  Its almost as if they are available by magic. To the end user they appear to be "free".  

 

Hardware costs for devices have also been obscured.  The most common example are smart phones which are often bundled with a cell phone and data plan for a fixed period.  A portion of the data service covers the cost of the hardware and the price at the time of purchase can be reduced or even "free".  this allows a user to buy a device like the iPhone for $200 when they sign up for a contract. However anyone who has ever lost, broken, or had their phone stolen can tell you a replacement before the contract if up for renewal will cost $600 or more. Over the life of a 2 year contract a $25 a month data plan will cost you $600.  However the perception of the cost of the device to the consumer has been reduced. The price the consumer pays to purchase the device is actually less the the cost to produce the device.  

 

The economic value Economic Value = Profit margin*(Parts+Labor) of computing has diverged significantly from the perceived cost of proving these services Consumer Cost = $ Paid.  Indirect cost structures of low cost devices coupled with cloud based services generate a high perceived value in the mind of the consumer. Due to the low direct price paid directly by the consumer Cloud Service Value = Actual Operating expenses / Cost of services directly paid .

 

A cloud based service Facebook provides messaging, games, photo, and video sharing for "free" to the end user.  According to Facebook's SEC S-1 filing  it cost $1.955 Billion to provide this free service in 2011. By providing this "free" service they were able to generate $3.711 Billion in revenue and 41 Billion in net Income.  Google which provides free search, email, calendar, electronic books, phone calls, videos, and other services spent $26.273 Billion on operating expenses in 2011.  Google generated $37.905 Billion in revenue and Net Income of $9.737 Billion. 

 

The economics of computing have changed and the cost burden has been shifted away from direct billing of the end user. As a result the end user perception has changed to view information services and hardware as an inexpensive commodity that should be provided at very little if any direct cost.  In return the end user is willing to accept good enough up-time and availability, self management through an easy to use interface and self support through knowledge bases or forums.  More expensive paid services are expected to provide a better Service Level Agreement and support.  However, these low cost services have increased consumer resistant to more expensive options and demand for more expensive services and hardware is now highly elastic.  

 

The prevalence of these inexpensive cloud services has been enabled by development of simplified user interfaces which are intuitive to the end user and require no training to operate.  This removes Information Technology profesionals from having to manage end user activities or provide training which are time consuming and costly.  However, the simplification of the end user product has led to a devaluation of the Information Technology that creates this experience.  This has led to a devaluation of Information Technology skills in the eyes of the end user.  Inversely the systems that support these services have become increasingly complex and require specialized skills to develop and maintain. 

 

Epilogue

Someone might have a friend over for dinner who technology skills and think nothing of asking them to fix their phone, tablet, PC, or other device.  However, if their dinner guest was a mechanic they would never toss them the keys to their car and ask them to change the oil before dinner.  

RAID for Data Protection and Performance

Previously we discussed the performance and reliability of disk drives. The eventual failure of disk drives is an absolute certainty, but the question is when.  Various environmental and usage factors affect their life expectancy.  Singular disk drives are also bound by performance limitations.  To overcome the reliability and performance limitations of individual disk drives, groups of drives are grouped together using RAID (Redundant Array of Independent Drives). This technology is usually the first line of defense in protecting critical data from hardware failure.  RAID technologies allow for a single or multiple drive failures to occur in a group of disks without causing data loss.  Further the data can be kept online during a failure. RAID performance considerations will be discussed in the next post.  

 

There are two variations of RAID technology, Mirrored RAID and Parity RAID.  These variations are further delineated by RAID type, which describes the data protection mechanism being used.  There are 5 RAID types in use at this time.  There are others which have since become defunct or are no longer in use. Drives are configured in groups with the same RAID type. A server can have multiple RAID groups with the same or different RAID groups.  Each RAID group is an independent entity and a separate failure domain.  

 

RAID 0,  JBOD - Just a Bunch Of Disk (Do not use). RAID 0 takes the data written to the disk and spreads it across all of the available drives in the RAID group. Breaking the data up into smaller chunks and spreading it across the drives is called striping or data stripes. Spreading the data across multiple drives gives us performance benefits, because every transaction grabs a piece from each of the drives and spreads out the workload.  RAID 0 has No Data Protection.  The probability of a hardware failure causing data loss is actually increased, because a single failure of any of one of the drives will cause the data to be unreadable on all of the drives. Unless the data is of very little value another RAID type is strongly recommended.   It is worth pointing out that some small NAS appliances and workstation PC's allow RAID  0 in their configurations.  This configuration should be avoided.  

 

 

Mirrored RAID is a group of drives which for every data drive, there is second drive with and exact copy of the data on it.  This results in a 1:2 ratio of data protection or 50% usable capacity.  

 

RAID 1 Is an exact mirror between two drives. This offers us reliability benefits, because if we have a hardware failure on one of the drives we can still access the data on the surviving drive.  Read operations also see a performance benefit because we can read from either of the two drives.  Write operations will not see a performance benefit because we will need to write to both drives simultaneously.  Usable capacity will be 50% because the drives are exact copies of each other. 

 

RAID 10 or RAID 1/0 combine the performance benefits of a striped RAID with the data protection of a Mirrored RAID. This RAID type will give you the highest level of performance for Reads or Writes as well as the highest level of data protection. However, it requires 50% overhead.  In this configuration the data is striped across 1/2 the drives.  The second 1/2 of the drives contain an exact copy of the 1st half.  RAID 1/0 is a great choice for Logs, swap files, and other write intensive workloads.  

 

There are exactly 10 kinds of people in this world, those that understand binary and those that do not.  

 

Parity RAID breaks the data up into chunks and stripes them across the drives.  In order to protect the data, bit's or the individual 1's and 0's of each byte are added together to produce a check-sum of all the bytes.  This check-sum is refereed to as parity.  The parity byte is stored on one of the drives or If we have a failure and loose the bytes on one of the drives, we can recalculate the data using the parity byte and the bytes on the surviving drives.   

 

RAID 5 stripes the data across all of the available drives except one and places the parity data on the remaining drive.  In Raid 5 the drive that hold the parity data is alternated with each strip to increase the performance of the RAID group.  Read operations are spread across all of the available drives offering good read performance. In order to write to the bytes on one of the drives RAID 5 must read the data from the drives, calculate parity and re-write the data and parity.  In the case of large sequential writes RAID 5 is optimized to do a full stripe write, where the entire stripe is overwritten and the RAID controller calculates parity in memory prior to writing to the drives.  the data and the parity are then written in a single operation.  This makes RAID 5 very efficient for large block sequential I/O.  RAID 5 offers n+1 protection and can withstand a single drive failure in the RAID group before the entire group becomes unreadable.  It is important to replace faulted disk quickly to minimize the chance of a second simultaneous failure, which would double fault the RAID group and cause data loss. 

 

RAID 6 is implemented by several vendors under different trademarked names, such as double parity or RAID DP.  RAID 6 used Parity protection similar to the way RAID 5 works.  However, as individual disk drives have increased in capacity, and slower larger capacity drives have become popular the amount of time required to rebuild a RAID from parity to replace the faulty drive has grown.  This longer window of time increases the risk that a second drive could fail in the RAID group while they are rebuilding.  Some implementations of RAID such as on board RAID controllers in servers do not continuously check the drives for errors. Without periodically checking each of the sectors of the drive it is possible to have a drive sector go bad without noticing if we are not trying to access the data.  This is known as silent data corruption.  In order to rebuild a failed drive the RAID controller has to read all of the sectors on the surviving drives. If there are bad sectors on the surviving drives, the RAID controller will be unable to rebuild.  To prevent this problem RAID 6 was introduced which keeps two parity bytes for each stripe.  The first is similar to a RAID 5 parity calculation and the second is more sophisticated erasure coding algorithm.  Using these 2 parity bits it is possible for a RAID 6 to survive up to 2 simultaneous failures without data loss (n+2).  However, the additional parity calculations mean there is an impact to write performance as well as the overhead equivalent to two drives to store the parity data.  

When choosing a storage device or RAID type for an application it is important o consider the performance and reliability of that configuration.  Using a sub optimal RAID configuration or too few drives can result in poor performance that will make the application slow or unresponsive.  We also want to ensure that we have our data protected by RAID. This allows the system to keep running during a hardware failure eliminating downtime all together. Further it eliminates the lengthy process of having to restore from back up or the loss of data that that changed since the last backup.  Please read the next post on how to determine the right number of drives for a RAID group. 

Understanding Disk Drive Performance and Reliability

Disk drives are mechanical devices with components engineered to specs measured in fractions of a human hair.  Unlike computer chips which are also built to extremely tight tolerances these devices also contain many moving parts moving at extremely high speed.  There are few things in life that are as certain as death, taxes, and disk drive failure.  All disk drives will fail eventually and it's a matter of when it will happen, how much of an impact it will have, and what we will do to recover.  Solutions such as backup are excellent for recovering data after a drive failure.  However, restoring from a backup means that any data changed since the last backup will be lost and they system will be unavailable until the data is restored, which can be time consuming. 

RAID (to be discussed in the next entry) allows us to protect against single drive failure as well as provide better performance and capacity then we can get from a single disk drive. RAID is a key component of all data storage specific devices. Almost all server class hardware has the ability to implement RAID. Even some desktops and laptops can use this.  RAID spreads data across multiple disk drives to distribute the workload and aggregate the capacity of the individual drives. Additionally RAID offers data protection using parity (checksum) or a mirror (copy) of the data.  Using RAID allows a single disk drive to fail without any loss of data or downtime.  In the event of a failure the faulted drive is removed and replaced. The data is then rebuilt from the surviving disk drives making the device self-healing.  In many cases the drives are hot swappable and can be changed in minutes while the system stays online and users can access their data.  This is a significant advantage over single disk drives even for the storage admin. 

 

Performance

Disk drives are mechanical devices that store data on a magnetic medium.  If you were to envision how a disk drive works it looks much like a record player.  There is a round spinning platter with a magnetic coating that contains the data. A mechanically actuated arm holds a head hovering over the spinning disk without actually touching it.  As the platter spins around the head mounted on the arm can read or write data to the platter underneath it. There are actually several of these platter stacked on top of each other.  In order to access different sections of the drive the arm moves the head back and forth.  In order to access the data the drive must move the arm and wait for the section of the drive with the information to rotate around and pass under the head.  The time required to complete this process is called seek time. 

 

The largest determination of how quickly a drive can access the data is a function of the drives rotational speed.  The specs of drives vary by model and manufacturer. Your mileage will vary from these specs depending on variables such as the size of the I/O request, whether they are sequential or random, and a number of other factors.  A simple rule of thumb for determining disk I/O throughput is about 180 IOPS (Input Output Per Second) for a 15,000 RPM drive, 120 IOPS for a 10,000 RPM Drive, 80 IOPS for a 7,500 RPM disk drive, and 40 IOPS for a 5,400 RPM disk drive .  In order to achieve better performance then what a single disk drive can handle we need to spread the work load over several individual disks using RAID.

Many server admins don't often work in IOPS numbers.  Below are the drive bandwidth numbers 64KB block transactions.  The larger the block size the higher the bandwidth, but most open systems including windows use a 64KB block size.  The limitation of using an individual drive becomes rapidly apparent in a server environment where you have multiple users competing for resources. Any user that has watched their hard drive light blink away while they wait knows the effect of having a disk drive that can't keep up.  Having many users sharing the same physical drive in a server significantly opens up the possibility for disk saturation.  Further the additional workload on a non enterprise class drive can lead to premature drive failure and data loss.

15k RPM FC or SAS	8MB/s
10K RPM FC or SAS	6.0 MB/s
7.5K SATA  or NL-SAS	4.0 MB/S
5.4K SATA or NL-SAS	2.5 MB/s
Enterprise Flash Drive	100 MB/s

 

Drive Sizes 2.5" vs. 3.5"

Recently 2.5" disk drives have been introduced for storage systems and servers.  Noticeably smaller than the standard 3.5" drives they are replacing they are bout the same physical dimensions as a laptop hard drive.  However, these drives are available in 10k RPM and in some cases 15K RPM and meant to withstand 24x7x365 use.  Due to the increase in Arial density (the ability to put the same amount of data on less space) these drives have some interesting performance characteristics.  When we look at the Seek and access times of the 2.5' drives they are several milliseconds faster than their larger 3.5" counterparts.  The 2.5" 10K rpm drives are within ~1ms of their 3.5" 15K counterparts.  Their IOPS throughput numbers are fairly close, however their bandwidth (MB/s) are lower than their 3.5" counterparts. 

The reason for this is that the 2.5" drive head has less distance to travel to access different areas of the drive because the patter is smaller.  Additionally since the circumference of the platter is smaller it takes less time for the platter to rotate the section of the drive with the data under the drive head.  The tradeoff is that the smaller diameter platter is able to store less sequential data per track and reading additional sequential data requires repositioning the drive head.  This results in lower overall bandwidth. 

 

Reliability

Disk Drive manufactures measure reliability as the Mean Time between Failure (MBTF) measured in hours.  The MBTF for a drive is based on a testing sample of drives and calculating the frequency of how often failures occur.  This is measurement is useful for understanding the probability of a failure and not for estimating when any one particular drive will fail.  Many drives will fail before the drives spec MBTF is reached and many other will exceed the anticipated life span. 

MBTF = (Test Time * Number of Drives Tested) / Number of Failures

Some Drive manufacturers have developed alternative ways of measuring anticipated drive reliability.  One method is Seagate's Annualized Return Rate (ARR).  This is a measure of drive reliability based on the number of drives returned as defective during a time period.  Another variation on this method is the Annualized Failure Rate (AFR).

ARR = Number of Units Returned for a Year / Total Units Shipped for the Year

Not all disk drives are created equal.  MBTF varies by the type of drive and its intended use case.  Consumer drives for laptops and desktops are rated at around 500,000 hours MBTF.  External USB drives would also fall into the types of drives found in this category.  Enterprise class drives found in server and storage hardware are rated at 1.2 - 1.6 million hours MBTF. With the 15K and 10K RPM SAS and Fibre Channel drives being closer to 1.6M  and the near line SAS or SATA drives being closer to 1.2M hours.  Consumer desktop grade hard drives are also only intended for 9x5 usage where enterprise drives are rated for 24x7x365 usage. 

The number of start and stop cycles of a drive also contribute to the life expectancy of a disk drive. When a disk drive spins up the rotation of the platter forces a cushion of air between the disk surface and the read/write head.  When the drive spins down the actuator arm moves the head to a location that doesn't contain any data to prevent the head from causing damage to the disk.  Still each time the drive spins up there is additional wear on the drive head as well as to the motors that spin the drive and other components.  New features such as disk spin down have increased the significance of this wear indicator and many drive manufacturers provide data and how many start/stop cycles a drive can endure.  Again this spec is an indicator of overall reliability and not when a particular drive will fail. 

 

Google Study

To better understand disk drive life expectancy Google did a study on 100,000 ATA disk drives in individual servers in their data center between 2005 and 2006.  Using the data collected Google was able to determine which factors were most influential in premature disk failure.  In many of the studies done there was an initial high mortality rate in the 3-6 months' time period that was usually attributed to the infant mortality of defective drives.  For example after the initial 3 months disk drive failure rates drop off substantially and are highest in years 2 and 3.  Year 3 offers the highest drive failure rate at 8%.  

Heavy utilized disk drives fail by about 10% in the first 3 months, where low use disk drives only fail about 4% of the time, and medium use drives fail about 2% of the time.   In drives 2 years and older this utilization made very little difference in the failure rate of the drives, accounting for a <0.05% difference in drive failures.  However, it is interesting to see that low utilization drives fail most often followed by high utilization drives. 

During the testing period temperature data was collected from SMART.  The temperature data did not show any significant increase in drive failure as temperature increased until extreme temperatures were reached.  Most of the drive failures were at lower temperatures.  Perhaps this was due to intermittent use or extra start / stop cycles of those drives. 

SMART error data was also collected during the test and analyzed to see if it could accurately detect failures.  The Google study found that SMART could indicate disk failure but that SMART was not reliable in detecting all disk failures.  Of the drives that failed 36% of the drives had 0 errors and 56% of the drives had 0 errors that would have indicated a strong signal the drive might fail.  Google considers strong signals; scan errors, reallocation count, offline reallocation, and probational count to be good indicators for determining a drive may fail.  Other error codes such as seek errors and CRC errors did not have a statistical correlation to drive failures.

Drives showing SMART Scan Errors are 10x more likely to experience drive failure.  Scan errors have a higher effect on the mortality of younger drives then on older ones.  Multiple errors increase the probability that a drive will fail and indicates that the drive will fail more quickly. 

Reallocation Errors indicate that there was a problem writing to or reading from a section of a drive and that data was moved to a different section of the drive.  Offline reallocations occur when an error is found by background tasks checking un-accessed sections of the drive.  SMART Reallocation Errors indicate a drive that is 3-6x more likely to fail.  Drives are 14x more likely to fail within 60 days of their first reallocation error then drives without any errors.  The probability of failure within 60 days increases to 21x more likely if the reallocation is done offline. 

 

Further Reading:

http://www.google.com/url?sa=t&source=web&cd=4&sqi=2&ved=0CD4QFjAD&url=http%3A%2F%2Flabs.google.com%2Fpapers%2Fdisk_failures.pdf&rct=j&q=hard%20disk%20reliability&ei=0CjyTcqjD4LZgAeUgKHPCw&usg=AFQjCNGQnQZT4n9wJDyYMmOnmVqfnx1udw&cad=rja

http://www.samsung.com/global/business/hdd/learningresource/whitepapers/LearningResource_OverallReliability.html

http://www.dell.com/downloads/global/products/pvaul/en/enterprise-hdd-sdd-specification.pdf

Hands on With Isilon OneFs

Yesterday I had the chance to get hands on with the Isilon NAS storage array and OneFS.  Including running through the installation and configuration process and adding a node to the cluster.  Initial setup of the first node using a serial cable to configure these options.   Installation is incredibly simple with a text based wizard for configuring basic settings such as IP addresses, default gateway, subnet mask, and user name and password.  Afterwards you can manage the array through the web browser interface.  The GUI was very clean and easy to use. 

When configuring the array you will be asked to provide a range of IP addresses for installation.  This allows the cluster to easily add new nodes using IP addresses as needed from the range provided.  You can specify multiple discontinuous ranges of IP addresses if need be.  Each inifiniband backend switch will need its own range of IP addresses as part of the configuration.

When adding a new node to the cluster the node is automatically reimaged and either upgraded or downgraded to match the version of software running on the cluster. 

After establishing the cluster a default file share called isi-status is provided for testing purposes. It's worth noting that you can point to any node in the cluster to access a particular share and the cluster will retrieve it from the appropriate node for you. This process is completely transparent to the end user. 

Overall the installation process as well as the management GIU is very user friendly and incredibly easy to manage. 

One errata worth noting is that there is an interruption of service when you add the second node to the cluster.  The reason this is not widely published or advertised is because all Isilon clusters are sold with a minimum of 3 nodes.  The interruption of service only happens when you are adding nodes and you have less than 3 in your cluster.  This is due to an election process that must take place between the nodes as they establish quorum.  Since this would only happen when you are initially setting up your cluster and you would not yet have any hosts attached it is a Non-issue.  Also at the moment there is no De-duplication support. 

Building Your Private Cloud in Your Data Center

Cloud has become the buzz word of the day along with all the ensuing hype that has come along with it.  Every technology company tries to use the word at least once a sentence.  It almost every aspect "Cloud Computing" is to this decade what the "information superhighway" was to the 90's.  It was a heavily used buzzword by everyone in all walks of life from AOL to school teachers.  Despite all of the hype, I believe that the internet phenomenon has full caught on.  Interestingly enough it is this phenomenon that is driving cloud computing today. 

 

When we hear about cloud computing we often think about cloud as an abstract vision of something "out there" in the big white puffy blob.  When we ask what it means to our it infrastructure today or how we leverage it to improve our business we often see more slides with poofy white clouds on them and more buzzwords.  I'm going to discuss the implementation of cloud and it's benefits on both your IT and business operations. Most importantly I'm going to discuss how you implement a private cloud in your datacenter today.

 

Private cloud centers around the virtualized datacenter. Meaning that you are running software (aka a Hypervisor) that allows you to run multiple systems on the same physical hardware. This is the 1^st and key step in the private cloud because it allows for higher utilization and sharing of resources.  CPU, memory, network connectivity, and storage are now pools of resources that you can pull from to run your business applications.  This allows you to quickly provision resources for applications as well as maintain higher levels of availability and reduce operational cost.  Virtualizing applications also makes them portable and able to be moved between datacenters r even between the public and private cloud.

 

The second step in building the Private cloud is securing the pools of resources that you have allocated to your applications. By protecting and isolating your applications we allow applications from multiple business units or even multiple companies to share resources in a concept we call Multi-Tenancy. Securing a multi-tenant environment ensures that one application cannot negatively impact another in the case of a misconfiguration or a security breach.  In VMware this is managed through vShield.  vShield can isolate devices and virtual networks and even allows for hardened virtual appliance to manage network traffic at layer 2 of the OSI model.

 

The third step which brings us further from just having a virtualized datacenter to having or own private cloud involves the tools for administration and management of the Private Cloud.  Providing a portal for business units to provide self-service reduces the involvement of IT.  Self-service also allows the business unit to bring applications online much quicker. Having the proper tools in place allows IT to manage this change efficiently and effectively is also important.  Configuration manager from VMware allows the business unit to provision an application from a pre-defined template. The template ensures a standardized and approved configuration. Configuration manager also provides auditing to ensure that the applications are compliant with IT policies, and manage resources such as software license keys.   IT can allocate resource pools to the business unit to use for provisioning or provide a shopping cart for the business to purchase the resources it needs.  This encourages the business unit to make intelligent business decision about which applications they decide to run and if the value to the business is worth the cost.  Configuration manager also handles the reporting on utilization of resources and billing business units.

 

This offers a dynamic change from traditional IT funded as a cost center to the business overall or as an internal budget "tax".  Directly relating the billing of IT costs to the resources used encourages managers to prioritize the resources that are used and determine which applications really add value to the business, reducing waste.  This also shifts IT from being a cost center to a revenue producing organization.  IT can also offer solutions or resources to other business partners.  For Example a hospital implementing a private cloud starts off by providing resources for it' own internal needs.  However, the same systems would also benefit their other affiliate hospitals.  The hospital can then offer their solution as a service for a fee to their business partners and bring in additional revenue for the business.  Similar situations could be done for school districts which have similar needs for their students such as Blackboard software and other course ware.  Rather than each school district building out their own infrastructure on school district could offer these applications as a service for a fee to their neighboring school district, or they could be offered by the state to all of the school districts.   This tremendously reduces the cost of implementing such systems