Storage Guru

Most of us are aware that as we move around the web we leave our footprints behind in the grains of silicon sand.  Web sites leave tracking cookies in our browser, Gmail indexes our email, Google search results feed the ads we see.  Twitter, Facebook, and Linked In are public and directly linked to our user ID.  We are accepting of this in return for free email, better search results, stores that remember what we had in our shopping cart, and for having interactive social media. 

When we move about IRL (In real Life) we assume that we are one of a faceless mass assuming that our identity is lost or obscured in the volume of the crowd.  In reality even our offline activities leave digital crumbs behind. Individually these crumbs are meaningless like an individual piece in a puzzle.  As more of the pieces are collected, we are able to generate a more complete picture.  The challenge is amasing all of these individual pieces finding a way to store and then process them into meaningful information. 

Recently we have seen huge advances in the ability of analytic systems such as Greenplum to process massive quantities of seemingly unrelated information to determine trends an correlation in what was previously assumed to be unrelated data.  Big Data offers us unique opportunities for us to process enormous quantities of data for Genomics, oil and gas exploration, and also for us to understand people.  Companies are realizing tremendous opportunities to better understand their customer.  Political campaigns are using this data to better understand their constituents zip code by zip code and modeling the impact of embracing one demographic of voters while alienating another demographic. 

The New York Times has a excellent article detailing how Target Corporation leverages Big Data analysis of it's customers. They have focused their marketing around potential customers that are experiencing major life changes such as having a baby.  Based on credit card data and purchase history they are able to build highly detailed profiles of their customers that can indicate when a baby is on the way and even the expected due date.  This worked so well their model predicted a teenage girl was pregnant before her father knew.  This article is so well written that to fill you in on all the relevant details would we outright plagiarism, so I'll just provide the link
http://www.nytimes.com/2012/02/19/magazine/shopping-habits.html?_r=1&partner=rss&emc=rss&pagewanted=all  (Febuary 15th, 2012)
This topic has since been picked up by Time magazine and Forbes.

Big Data has enter politics as well.  Demographic information on voters enables political campaigns to figure out which voters will respond to specific messages and the impact of those messages on election results.  Information is used to weigh the impact of alienating a certain demographic of voters in a zip code vs. attracting a different demographic of voters with a different viewpoint in multiple districts.  For example by aligning himself with voters making less then $50,000 a year might pick up and extra 10% of the votes in that demographic, but loose 10% of voters making over $100,000 and loose 15% of the voters making over $200,000.  Big data allows the campaign to calculate the impact of loosing or gaining these voters in key swing geographies by analyzing income by zip code, the number of voters in the income brackets in those zip codes, and their party affiliation.  This allows the campaign to target winning the electoral college votes required, even if the popular vote margin is razor thin. (NY Times Febuary 15th, 2012   http://www.nytimes.com/2012/02/19/magazine/nate-silver-obama-reelection-chances.html?_r=1&ref=politics&pagewanted=all  )  

In the mid 80's when we started to get the first computers at school and early adopters bought them for their home or small business they were machines costing thousands of dollars and something to be revered and respected.  The first PC we had in our home was a WSYE x80-386 With 2MB of Ram and a 40MB hard drive and a monochrome display circa about 1985.  The PC cost $3,000 or adjusting for inflation $6,000 today.  This was back when you washed your hands before you used the computer and didn't dare eat or drink coffee while using it for fear of damaging the equipment.  A plastic dust cover was often placed over the equipment when not in use.

Fast forward to 2012 and consumer expectations of compute technology has changed dramatically.  Devices are expected to be scratch resistant and water resistance is the biggest new selling feature.  We want to be able to sweat all over the device at the gym, steam it in the bathroom while we take a shower, and survive the drop in the toilet that has ended so many smart phones.  At the current pace we should expect a mocha latte resistant device with an iCoaster App to appear at the local Starbucks sometime soon.

End users have a cost expectation for a new consumer device that the hardware should cost between $0-$500 and up to a maximum $1,000 on the ultra high end.  there is increasing resistance toward any device that cost more then a toaster.  The expectations for software have changed radically as well.  Software has shifted from expensive application suites with an exhaustive features to single purpose Apps.  the price expectation for an App is $0-$1 with some very expensive applications costing $5-$15. 

In the two decades since the 80386 made the consumer PC popular we have seen exponential growth in processing power and memory growth.  Moore's Law dictates that the capabilities of Processors and memory double every two years.  Today mainstream processor and memory chips have more capacity then many end users require.  Instead of introducing the latest and fastest chip many consumer devices focus on smaller low performance chips that consume less power.  It's worth noting that even these low power chips continue to evolve and double every two years.  Low power hardware has ushered in an era of much less expensive devices.

Many of these low power devices work on a Just Enough Compute model, single tasking an active application such as email or web browsing and putting the inactive application to sleep in the background. Due to their small screens and simple interfaced many of these low power devices are ideally suited for single tasking.  The end user perception of these devices is enhanced by their ability to leverage cloud based computing services. This allows the device to complete resource intensive tasks by offloading the processing to the cloud.  For example voice activated SIRI listens to the user on the device and then uploads a recording of the request to the SIRI servers where the voice recognition takes place and then returns the response from the server.  I use an application to scan business cards.  My phone takes a picture of the card and uploads it to the Apps cloud where the text recognition takes place and my phone receives the persons name, email address, and phone number ready to be imported into my phone book. 

Cloud offload is aggressively used by the amazon Kindle Fire. the fire offloads much of the processing for simple tasks such as we browsing and video to the Amazon S3 cloud.  The result is a very inexpensive piece of hardware, that is light weigh and gets good battery life.  The true cost of running the device is shielded from the end user who never sees the warehouse of servers and hardware that makes all of this possible.

Many cloud applications such as Google, Gmail, and Facebook offer their cloud services to the end user for free, in return for inputting their information and allowing the cloud provider to analyze their data and provide metadata to 3rd parties.  In this business model the cost of development, support, hardware and the true economic value for these applications is completely hidden from the end user.  Its almost as if they are available by magic. To the end user they appear to be "free". 

Hardware costs for devices have also been obscured.  The most common example are smart phones which are often bundled with a cell phone and data plan for a fixed period.  A portion of the data service covers the cost of the hardware and the price at the time of purchase can be reduced or even "free".  this allows a user to buy a device like the iPhone for $200 when they sign up for a contract. However anyone who has ever lost, broken, or had their phone stolen can tell you a replacement before the contract if up for renewal will cost $600 or more. Over the life of a 2 year contract a $25 a month data plan will cost you $600.  However the perception of the cost of the device to the consumer has been reduced. The price the consumer pays to purchase the device is actually less the the cost to produce the device. 

The economic value Economic Value = Profit margin*(Parts+Labor) of computing has diverged significantly from the perceived cost of proving these services Consumer Cost = $ Paid.  Indirect cost structures of low cost devices coupled with cloud based services generate a high perceived value in the mind of the consumer. Due to the low direct price paid directly by the consumer Cloud Service Value = Actual Operating expenses / Cost of services directly paid .

A cloud based service Facebook provides messaging, games, photo, and video sharing for "free" to the end user.  According to Facebook's SEC S-1 filing  it cost $1.955 Billion to provide this free service in 2011. By providing this "free" service they were able to generate $3.711 Billion in revenue and 41 Billion in net Income.  Google which provides free search, email, calendar, electronic books, phone calls, videos, and other services spent $26.273 Billion on operating expenses in 2011.  Google generated $37.905 Billion in revenue and Net Income of $9.737 Billion.

The economics of computing have changed and the cost burden has been shifted away from direct billing of the end user. As a result the end user perception has changed to view information services and hardware as an inexpensive commodity that should be provided at very little if any direct cost.  In return the end user is willing to accept good enough up-time and availability, self management through an easy to use interface and self support through knowledge bases or forums.  More expensive paid services are expected to provide a better Service Level Agreement and support.  However, these low cost services have increased consumer resistant to more expensive options and demand for more expensive services and hardware is now highly elastic. 

The prevalence of these inexpensive cloud services has been enabled by development of simplified user interfaces which are intuitive to the end user and require no training to operate.  This removes Information Technology profesionals from having to manage end user activities or provide training which are time consuming and costly.  However, the simplification of the end user product has led to a devaluation of the Information Technology that creates this experience.  This has led to a devaluation of Information Technology skills in the eyes of the end user.  Inversely the systems that support these services have become increasingly complex and require specialized skills to develop and maintain.

Epilogue
Someone might have a friend over for dinner who technology skills and think nothing of asking them to fix their phone, tablet, PC, or other device.  However, if their dinner guest was a mechanic they would never toss them the keys to their car and ask them to change the oil before dinner. 
 

Previously we discussed the performance and reliability of disk drives. The eventual failure of disk drives is an absolute certainty, but the question is when.  Various environmental and usage factors affect their life expectancy.  Singular disk drives are also bound by performance limitations.  To overcome the reliability and performance limitations of individual disk drives, groups of drives are grouped together using RAID (Redundant Array of Independent Drives). This technology is usually the first line of defense in protecting critical data from hardware failure.  RAID technologies allow for a single or multiple drive failures to occur in a group of disks without causing data loss.  Further the data can be kept online during a failure. RAID performance considerations will be discussed in the next post.  

Mark Zuckerberg and his look alike Andy Samberg recently took the stage at the f8 tech industry conference and encouraged us to share more.  Facebook has been retooled to actively push updates from friends.  As Facebook is shifting focus from being the site that allows us to be our own and our friend's personal Paparazzi to a marketing machine.  The more user data passing through Facebook, the more relevant and effective it becomes.  Metcalfe's law states: that the value of a network is proportional to the square of the number of connected users.   Previously this would have been applied to the number of Facebook users, but now we can apply it to the number of businesses sharing information, and pushing marketing through Facebook.

Traditional Advertising such as TV, Radio, print, and display based advertising such as billboards are hampered by comparatively low hit ratio with their target audience. Despite having troves of market research, Neilson ratings, and targeting mediums with the most appropriate demographics advertisers in traditional media have to spend large amounts of money to reach a minority of interested consumers.   Online advertising allows advertisers to more efficiently target very specific niche audiences. 

Online advertising varies in how effectively the target audience can be profiled.  Web site banner ads are limited to being relevant to the content of the page on which their displayed or the browsing history of the user.  Google, Yahoo, and Microsoft's Bing display ads based on your recent search results, but know very little about you.  Gmail displays ads based on key words gleaned from your email. Social media such as Facebook, Linked in and Google+ opens up tremendous opportunity for very specific highly targeted advertisements.  Because we provide social media with additional information about us the ads can be much more specific.  For example if a retailer wants advertise a sweater they can target gender, age, geographic location, income level, level of education, what kind of music you listen to and what movies you watch. For example if J Crew wanted to post an ad for a sweater, you might see the add if you live in the northern part of the country, are a female, 40-60 years old,  with an income of over $50,000 per year.  If you make less than $50k you might see an ad from Wal-Mart instead.  This ad could be further customized to show you a blue sweater if you your favorite color is blue, or if you happen to like the Punk Band Godsmack, you might not see a J Crew ad, and instead see something from Pac Sun.

Facebook provides the option for you to authenticate to other affiliate sites using your Facebook user name and password.  For the end user this allows a simple single login without needing to generate an account for every web site they visit, or memorize passwords for infrequently visited sites.  In return these sites are granted permission to access information in the users Facebook profile which can be used to deliver customized content for the end user.   Facebook is taking this a step further and soon your friends will automatically be able to see what you've been reading, what you've been buying, and this will be able to be followed on your profile page.  When you listen to music or watch video on the Facebook site your friends will see it and be offered a link to consume the same media.  If you buy a new pair of Nike sneakers, Nike will slip a post into your news stream showing off your new kicks, with a link for all of your friends to buy their own pair if they like them.

Recently I helped a small nice construction firm specializing in Data Centers with an online advertising campaign.  By selecting social media sites we were able very specifically target the audience we thought would be most receptive.  For example we could focus on building managers, facilities, and IT professionals in a very narrow geography. They avoided wasting advertising dollars on viewers that weren't interested in their services.  The more precise you can target the advertisement the greater the value of the ad, and the higher the premium it can command.  Currently ad revenue for search is declining. Google's revenue per search has declined from $23.50 per 1,000 searches to $13.90 according to Forbes. By comparison Facebook's  advertising revenue is expected to grow 67% from previous year to 2.01 billion in 2010 according to the Wall Street Journal and forecasts for 2011 are around 110%.

Online advertising revenue is generated two ways. Advertisements are billed based on the number per thousand impressions or times that the ad is viewed.  Alternatively advertisements can be billed based on their click through rate or the number of times a user actually clicks on an advertisement.   The rate at which ads are billed by either method is determined by the advertisement provider and the content publisher on the site displaying the advertisements.  Google, Facebook and Linked in allow you to bid a certain rate for your ads to be displayed and the ad provider will broker displaying your advertisement with one of its content providers. 

For Example StorageGuru.org has an advertisement banner at the top to defray the cost of hosting this blog.   If you click on the Ads I could receive anywhere up to a dollar.  Go ahead try it! Better try a couple more just to be sure.  These ads are placed based on the content of my blog and change to be relevant to the topic discussed on the page. 

So we ask ourselves why the push to share everything? The more we "share" the more data points are collected and the more precise the advertisements can be.  We are also more likely to share the interests of our peer group and therefore our social network friends are more likely to interested as well.  Individuals also respond more enthusiastically to suggestions from someone they know.  Mark Zuckerberg recently announced at the f8 conference, he would like us to share everything, even what we ate for breakfast. In turn we may get an ad for cheerios. The more time we spend on Facebook messaging, reading, and consuming media, the more opportunity Facebook has to display ad content and generate revenue.  When you need to venture off into other parts of the web Facebook can still leverage their knowledge of you and open up opportunities for additional revenue by allowing you to login with Facebook with its affiliate sites.

Social media is unique in that most of the content is user generated, but the content that generates the revenue is all commercially driven.  It is a bit ironic that Mark Zuckerberg's Facebook page is an online press release.  There are no cell phone photos from a last call at a bar, no wall posts from friend's only comments.   The best perk of running Facebook, appears to be the ability to keep yourself off Facebook, or at least remove that photo you don't like.  Increasingly ironic has been the lack of privacy controls available to regular users until Google launched their own social network.  I think that mark makes a good example even if in a "do as I do, not as I say" kind of way, that we should all share carefully.  After all you're not getting a "share" of the ad revenue from the content you provide.  

There has been some buzz circulating the blogosphere regarding EMC's Thin Provisioning features and Storage Pool architectures on their VNX storage array.  I read two post that discussed the architectural considerations and use cases for this technology in depth.  In general these were great reading and very insightful, however I found two points that could cause confusion or be misleading. 

Vijay's post on Virtualeverything @veverything wrote a very in depth analysis on how thin provisioning works.  However, there is some confusion by the reader regarding the allocation of space using thin pools.  In the paragraph titled "Caveat/Design consideration #1" he mentions that you may end up with more or less usable space depending on how many drives you allocate.  The EMC VNX array allocates the storage for the pool in a way that optimizes efficiency, performance, and data protection.  Following EMC best practices you would allocate disks in increments of 5.  Groups of 5 drives allow for a data to parity ratio of 4-1 protecting against drive failures in the pool.  The RAID 5 algorithms on the array also able to produce slightly better performance with disk groups of 5 or 9 drives.  If the end user decides to add an irregular number of drives to the pool the array will determine the best way to group the storage.  When adding drives in increments of , the drives are configured as 4+1 groups. If 12 drives are added the array will configure the group as 2(5+1), if 14 drives are added you will get 2(4+1) and 1(3+1).  When carving up the disks in groups of 5 if there is a remainder of 1 or 2 then the other groups will be enlarged to accommodate the odd drives (i.e 5+1 or 6+1). If the remainder is 3 or 4 then these drives will be allocated into an additional group of less then 5 drives (i.e 2+1 or 3+1).   The smaller RAID group will be less efficient in terms of usable capacity.

The purpose of storage pools is ease of use and simplicity.  Adding additional storage maintains this simplicity and ease of use from both an array perspective and an end user perspective.  If your trying to allocate pool storage with odd sizes and custom disk configurations your are reintroducing much of the complexity that storage pools were designed to eliminate.


The post on Just be Better  @virtualtacit discusses the allocation owner of a storage pool on an EMC VNX array.   The allocation owner tracks the configuration meta data of the storage pool.  this data is used to create and expand the storage pool.  Normal best practices would dictate that you divide LUN ownership across both storage controllers.  When you initially assign LUN ownership the allocation will be assigned with it.  In the event you need to re-balance the workload the LUN owner can be trespassed to the other storage controller and the allocation owner will remain the same.  This is perfectly fine since the allocation owner is only responsible for managing meta data that doesn't represent any significant workload.  In the event of an storage controller failure the array will be opperating in a degraded mode with the performance of only a single SP. In this case again the allocation owner is pretty trivial and will not be a critical factor.  The larger concern would be swapping out the faulted storage controller and restoring the array to full operating capacity.  This particular blog posts exaggerates this concern heavily. 

Its worth noting that all thin or pool based multi controller architectures store and use meta data about their configuration on their controlers.  The non EMC mid-range arrays that I have encountered store and manage their metadata in much less distributed fashion then the manufacturer would like you to believe. 

Disk drives are mechanical devices with components engineered to specs measured in fractions of a human hair.  Unlike computer chips which are also built to extremely tight tolerances these devices also contain many moving parts moving at extremely high speed.  There are few things in life that are as certain as death, taxes, and disk drive failure.  All disk drives will fail eventually and it's a matter of when it will happen, how much of an impact it will have, and what we will do to recover.  Solutions such as backup are excellent for recovering data after a drive failure.  However, restoring from a backup means that any data changed since the last backup will be lost and they system will be unavailable until the data is restored, which can be time consuming. 

RAID (to be discussed in the next entry) allows us to protect against single drive failure as well as provide better performance and capacity then we can get from a single disk drive. RAID is a key component of all data storage specific devices. Almost all server class hardware has the ability to implement RAID. Even some desktops and laptops can use this.  RAID spreads data across multiple disk drives to distribute the workload and aggregate the capacity of the individual drives. Additionally RAID offers data protection using parity (checksum) or a mirror (copy) of the data.  Using RAID allows a single disk drive to fail without any loss of data or downtime.  In the event of a failure the faulted drive is removed and replaced. The data is then rebuilt from the surviving disk drives making the device self-healing.  In many cases the drives are hot swappable and can be changed in minutes while the system stays online and users can access their data.  This is a significant advantage over single disk drives even for the storage admin. 

 

Performance

Disk drives are mechanical devices that store data on a magnetic medium.  If you were to envision how a disk drive works it looks much like a record player.  There is a round spinning platter with a magnetic coating that contains the data. A mechanically actuated arm holds a head hovering over the spinning disk without actually touching it.  As the platter spins around the head mounted on the arm can read or write data to the platter underneath it. There are actually several of these platter stacked on top of each other.  In order to access different sections of the drive the arm moves the head back and forth.  In order to access the data the drive must move the arm and wait for the section of the drive with the information to rotate around and pass under the head.  The time required to complete this process is called seek time. 

The largest determination of how quickly a drive can access the data is a function of the drives rotational speed.  The specs of drives vary by model and manufacturer. Your mileage will vary from these specs depending on variables such as the size of the I/O request, whether they are sequential or random, and a number of other factors.  A simple rule of thumb for determining disk I/O throughput is about 180 IOPS (Input Output Per Second) for a 15,000 RPM drive, 120 IOPS for a 10,000 RPM Drive, 80 IOPS for a 7,500 RPM disk drive, and 40 IOPS for a 5,400 RPM disk drive .  In order to achieve better performance then what a single disk drive can handle we need to spread the work load over several individual disks using RAID.

Many server admins don't often work in IOPS numbers.  Below are the drive bandwidth numbers 64KB block transactions.  The larger the block size the higher the bandwidth, but most open systems including windows use a 64KB block size.  The limitation of using an individual drive becomes rapidly apparent in a server environment where you have multiple users competing for resources. Any user that has watched their hard drive light blink away while they wait knows the effect of having a disk drive that can't keep up.  Having many users sharing the same physical drive in a server significantly opens up the possibility for disk saturation.  Further the additional workload on a non enterprise class drive can lead to premature drive failure and data loss.

15k RPM FC or SAS	8MB/s
10K RPM FC or SAS	6.0 MB/s
7.5K SATA  or NL-SAS	4.0 MB/S
5.4K SATA or NL-SAS	2.5 MB/s
Enterprise Flash Drive	100 MB/s

 

Drive Sizes 2.5" vs. 3.5"

Recently 2.5" disk drives have been introduced for storage systems and servers.  Noticeably smaller than the standard 3.5" drives they are replacing they are bout the same physical dimensions as a laptop hard drive.  However, these drives are available in 10k RPM and in some cases 15K RPM and meant to withstand 24x7x365 use.  Due to the increase in Arial density (the ability to put the same amount of data on less space) these drives have some interesting performance characteristics.  When we look at the Seek and access times of the 2.5' drives they are several milliseconds faster than their larger 3.5" counterparts.  The 2.5" 10K rpm drives are within ~1ms of their 3.5" 15K counterparts.  Their IOPS throughput numbers are fairly close, however their bandwidth (MB/s) are lower than their 3.5" counterparts. 

The reason for this is that the 2.5" drive head has less distance to travel to access different areas of the drive because the patter is smaller.  Additionally since the circumference of the platter is smaller it takes less time for the platter to rotate the section of the drive with the data under the drive head.  The tradeoff is that the smaller diameter platter is able to store less sequential data per track and reading additional sequential data requires repositioning the drive head.  This results in lower overall bandwidth. 

 

Reliability

Disk Drive manufactures measure reliability as the Mean Time between Failure (MBTF) measured in hours.  The MBTF for a drive is based on a testing sample of drives and calculating the frequency of how often failures occur.  This is measurement is useful for understanding the probability of a failure and not for estimating when any one particular drive will fail.  Many drives will fail before the drives spec MBTF is reached and many other will exceed the anticipated life span. 

MBTF = (Test Time * Number of Drives Tested) / Number of Failures

Some Drive manufacturers have developed alternative ways of measuring anticipated drive reliability.  One method is Seagate's Annualized Return Rate (ARR).  This is a measure of drive reliability based on the number of drives returned as defective during a time period.  Another variation on this method is the Annualized Failure Rate (AFR).

ARR = Number of Units Returned for a Year / Total Units Shipped for the Year

Not all disk drives are created equal.  MBTF varies by the type of drive and its intended use case.  Consumer drives for laptops and desktops are rated at around 500,000 hours MBTF.  External USB drives would also fall into the types of drives found in this category.  Enterprise class drives found in server and storage hardware are rated at 1.2 - 1.6 million hours MBTF. With the 15K and 10K RPM SAS and Fibre Channel drives being closer to 1.6M  and the near line SAS or SATA drives being closer to 1.2M hours.  Consumer desktop grade hard drives are also only intended for 9x5 usage where enterprise drives are rated for 24x7x365 usage. 

The number of start and stop cycles of a drive also contribute to the life expectancy of a disk drive. When a disk drive spins up the rotation of the platter forces a cushion of air between the disk surface and the read/write head.  When the drive spins down the actuator arm moves the head to a location that doesn't contain any data to prevent the head from causing damage to the disk.  Still each time the drive spins up there is additional wear on the drive head as well as to the motors that spin the drive and other components.  New features such as disk spin down have increased the significance of this wear indicator and many drive manufacturers provide data and how many start/stop cycles a drive can endure.  Again this spec is an indicator of overall reliability and not when a particular drive will fail. 

 

Google Study

To better understand disk drive life expectancy Google did a study on 100,000 ATA disk drives in individual servers in their data center between 2005 and 2006.  Using the data collected Google was able to determine which factors were most influential in premature disk failure.  In many of the studies done there was an initial high mortality rate in the 3-6 months' time period that was usually attributed to the infant mortality of defective drives.  For example after the initial 3 months disk drive failure rates drop off substantially and are highest in years 2 and 3.  Year 3 offers the highest drive failure rate at 8%.  

Heavy utilized disk drives fail by about 10% in the first 3 months, where low use disk drives only fail about 4% of the time, and medium use drives fail about 2% of the time.   In drives 2 years and older this utilization made very little difference in the failure rate of the drives, accounting for a <0.05% difference in drive failures.  However, it is interesting to see that low utilization drives fail most often followed by high utilization drives. 

During the testing period temperature data was collected from SMART.  The temperature data did not show any significant increase in drive failure as temperature increased until extreme temperatures were reached.  Most of the drive failures were at lower temperatures.  Perhaps this was due to intermittent use or extra start / stop cycles of those drives. 

SMART error data was also collected during the test and analyzed to see if it could accurately detect failures.  The Google study found that SMART could indicate disk failure but that SMART was not reliable in detecting all disk failures.  Of the drives that failed 36% of the drives had 0 errors and 56% of the drives had 0 errors that would have indicated a strong signal the drive might fail.  Google considers strong signals; scan errors, reallocation count, offline reallocation, and probational count to be good indicators for determining a drive may fail.  Other error codes such as seek errors and CRC errors did not have a statistical correlation to drive failures.

Drives showing SMART Scan Errors are 10x more likely to experience drive failure.  Scan errors have a higher effect on the mortality of younger drives then on older ones.  Multiple errors increase the probability that a drive will fail and indicates that the drive will fail more quickly. 

Reallocation Errors indicate that there was a problem writing to or reading from a section of a drive and that data was moved to a different section of the drive.  Offline reallocations occur when an error is found by background tasks checking un-accessed sections of the drive.  SMART Reallocation Errors indicate a drive that is 3-6x more likely to fail.  Drives are 14x more likely to fail within 60 days of their first reallocation error then drives without any errors.  The probability of failure within 60 days increases to 21x more likely if the reallocation is done offline. 

 

Further Reading:

http://www.google.com/url?sa=t&source=web&cd=4&sqi=2&ved=0CD4QFjAD&url=http%3A%2F%2Flabs.google.com%2Fpapers%2Fdisk_failures.pdf&rct=j&q=hard%20disk%20reliability&ei=0CjyTcqjD4LZgAeUgKHPCw&usg=AFQjCNGQnQZT4n9wJDyYMmOnmVqfnx1udw&cad=rja

http://www.samsung.com/global/business/hdd/learningresource/whitepapers/LearningResource_OverallReliability.html

http://www.dell.com/downloads/global/products/pvaul/en/enterprise-hdd-sdd-specification.pdf

I previously wrote about my experience with the EMC Atmos cloud storage platform at EMC World.  While I was there I also got some stick time with Oxygen's Cloud.  Oxygen offers an Atmos based Cloud as a subscription based solution or access to your own internal private cloud.  Taking it a bit further Oxygen offers their own desktop integration solution as well as apps for mobile platforms such as the iPad.  A tool bar in the system tray offers immediate access to your Oxygen cloud storage. The tool bar allows you to access file, configure your setting and perform administrative tasks depending on your access level.

Integration into the desktop includes an O drive integrated right into Windows explorer or OS X finder.  You can drag and drop files onto the O drive like any other drive in windows explorer.  Files can also be opened for editing or access.  A change in the icon informs the user if the file has been fully synced with the Atmos cloud or is still pending. 

Right click menu integration includes the ability to flag or un-flag files for Synchronization.  Unflagging the file allows you to work locally without having to worry about corrupting or unintentionally changing your remote copy. 

Oxygen also has some really cool tools for integrating directly with IPhone and iPad. The iOS app allows you to access your files on the device and to share them via email.  Another really cool feature is to save your incoming email attachments directly to the cloud.  Unfortunately they didn't have iPads for us to try this out on and I don't have any screen shots.

Atmos has a rich feature set that leaps ahead of traditional windows or UNIX file sharing mechanisms.  Allowing greater information protection and control over content distribution.  Traditional file servers are severely limited in the number of files they can store as well as the directory and file structure.  Traditional CIFS (windows file share) and NFS (UNIX/LINUX file share) were both created when most of organizations critical information was stored in large databases and mainframes (i.e. airline reservation systems).  Since then we now have a shift toward unstructured content (individual files), where most of the content is created by the end user.  Imagine every digital picture, movie clip, PowerPoint, Excel, and Word document ever created and where we would store it.  CIFS and NFS only allow a single point of access to maintain file locking, and replication solutions must often replicate the whole file and not just the part that changed. CIFS assumes that all of the clients are within your organization or on the same network. Sharing of CIFS files with external users is difficult if not impossible.  Further challenges are introduced when you start to try to back up a traditional file server.  The more files you have and the smaller the files the more challenging the backup becomes.  Atmos moves forward with an object based model that makes CIFS obsolete.  Solving many of the limitations and challenges of the file server and NAS share.

The first part of my test drive was using a web browser interface to access my Atmos cloud.  The experience was anything but what you'd expect accessing files through a web browser.  For someone used to using Gmail accessing files through a web browser requires tedious upload and save ass windows and only allows uploads or downloads of one file at a time.  With the Atmos GUI the process is completely transparent.  You drag and drop just like you do for every other file on a desktop. If you want to move 20 files you just drag and drop them to or from the cloud GUI.  A standard web browser (we used Firefox) allows you to login through a secure login that uses RSA encryption and authentication.  You can also click on a file to open it right from within the web browser window. 

More traditional file access controls are included for sharing files with other Atmos users. Through a very simple pop up box you are able to set Read, Write, and Full control permissions for users or groups of users. 

When dealing with unstructured data, Meta data becomes equally important. Meta data allows you to note what the file contains, and more importantly to enable search for specific files that are relevant.  In the case of Atmos the usage of Meta data allows the classification of unstructured data.  Based on the data the file contains Atmos can apply data placement policies to determine the level of protection, the number of copies, and the distribution of the copies of a file. 

Through the GUI the user can also view the properties of a file including the object ID.  In order to overcome the limitations of other file and directory structures, Atmos stores files as object's.  The object maintains the same object ID and allows the file to be identified no matter what device or location the file is stored on.   This methodology of storing files based on an ID is similar to the Google Haystack implementation and other systems designed to allow massive scalability.  The file properties also include the policy assigned to the object.  Atmos distributes the data and determines how many copies and in which location they should be stored based on the policy assigned. 

Polices allow Atmos to store data based on business need.  For example a free user of a platform could be assigned a policy which maintains the data with Geo parity protection, maintaining 1 copy in a remote location in a spun down near line state.  While a paid user or revenue generating application might maintain local data protection using parity and maintain a synchronous copy and several asynchronous copies in different geographic locations.  

Geo parity recently released in Atmos uses erasure coding to allow separate a file into data fragments and parity fragments.  Erasure coding is one of the key differentiators that separates Atmos from other file or object storage devices.  It allows you to specify the level of data protection for each individual object.  For example you can specify for a given file that you want to have 9 data fragments and 3 parity fragments. This allows you to tolerate up to 3 failures because you have decided to store 3 parity fragments.  Storing additional parity fragments increases redundancy but uses additional space.  Geo parity allows these fragments to be spread out over Atmos systems in different locations to protect against array, site, or even region level failures.  This is key to providing up time and availability to user data.  Recently we've seen incidents in New Orleans, New Zealand, Australia, Japan, Egypt, and Libya where being able to protect as well as deliver data from multiple geographic locations is key to avoiding local instability.  Geo mirror keeps a full copy of the data at a remote site, but requires more space then Geo parity.  Atmos allows read and write requests to come into an array at any site.  This allows content to be delivered from the closest source saving on WAN links and lowering latency.  Policies protecting data as well as retention are set at the object level and maintained globally across each site.  Retention policies can enforce data retention for a specified period of time as well as the automatic deletion of data when the time period expires. 

For the desktop the Open Hanna client allows easy uploading and downloading of files on the desktop.  Open Hanna will also allow you to see how your data is being replicated. Clicking on the replication information, you can quickly see how many fragments of data and parity you have for each object and how their distributed within the Atmos system.

According to the slide ware that accompanied my hands on test drive Atmos 2.0 introduces a few new features.  Geo drive speeds the publishing of content to the cloud.  Ina incorporates caching, compression, and encryption making it LAN and WAN friendly.  Atmos hardware can now be used as a target for Content Addressable Storage.  The use case for content addressable storage includes the ability to act as a target for EMC Centera based client API's.   For the iOS based iPad the EMC folio app in the app store allows access to Atmos content directly on the iPad. You can sync and share files directly from your tablet. 

Atmos offers much deeper integration into applications then the file share use case that I experienced using APIs for SOAP, REST, HTML and other protocols.  I looked at the features that I was able to benefit from as an end user accessing a simple file share.  Web based multi-user applications can benefit from many of the same features even if those features will be masked from the end user.  The ability to geographically distribute, store and access information is unmatched.  The ability to access the content as read and writable in multiple locations at the same time is unparalleled.  There are a few niche based proprietary applications out there developed by providers for their own use such as the Amazon cloud and Google for their own use. However, this is the first purpose built cloud storage platform available to end users and content providers that offers a turnkey cloud storage solution.

I will be following up with a post on a test drive of Oxygen a public cloud provider's Implementation of Atmos.

Yesterday I had the chance to get hands on with the Isilon NAS storage array and OneFS.  Including running through the installation and configuration process and adding a node to the cluster.  Initial setup of the first node using a serial cable to configure these options.   Installation is incredibly simple with a text based wizard for configuring basic settings such as IP addresses, default gateway, subnet mask, and user name and password.  Afterwards you can manage the array through the web browser interface.  The GUI was very clean and easy to use. 

When configuring the array you will be asked to provide a range of IP addresses for installation.  This allows the cluster to easily add new nodes using IP addresses as needed from the range provided.  You can specify multiple discontinuous ranges of IP addresses if need be.  Each inifiniband backend switch will need its own range of IP addresses as part of the configuration.

When adding a new node to the cluster the node is automatically reimaged and either upgraded or downgraded to match the version of software running on the cluster. 

After establishing the cluster a default file share called isi-status is provided for testing purposes. It's worth noting that you can point to any node in the cluster to access a particular share and the cluster will retrieve it from the appropriate node for you. This process is completely transparent to the end user. 

Overall the installation process as well as the management GIU is very user friendly and incredibly easy to manage. 

One errata worth noting is that there is an interruption of service when you add the second node to the cluster.  The reason this is not widely published or advertised is because all Isilon clusters are sold with a minimum of 3 nodes.  The interruption of service only happens when you are adding nodes and you have less than 3 in your cluster.  This is due to an election process that must take place between the nodes as they establish quorum.  Since this would only happen when you are initially setting up your cluster and you would not yet have any hosts attached it is a Non-issue.  Also at the moment there is no De-duplication support.  

Cloud has become the buzz word of the day along with all the ensuing hype that has come along with it.  Every technology company tries to use the word at least once a sentence.  It almost every aspect "Cloud Computing" is to this decade what the "information superhighway" was to the 90's.  It was a heavily used buzzword by everyone in all walks of life from AOL to school teachers.  Despite all of the hype, I believe that the internet phenomenon has full caught on.  Interestingly enough it is this phenomenon that is driving cloud computing today. 

 

When we hear about cloud computing we often think about cloud as an abstract vision of something "out there" in the big white puffy blob.  When we ask what it means to our it infrastructure today or how we leverage it to improve our business we often see more slides with poofy white clouds on them and more buzzwords.  I'm going to discuss the implementation of cloud and it's benefits on both your IT and business operations. Most importantly I'm going to discuss how you implement a private cloud in your datacenter today.

 

Private cloud centers around the virtualized datacenter. Meaning that you are running software (aka a Hypervisor) that allows you to run multiple systems on the same physical hardware. This is the 1^st and key step in the private cloud because it allows for higher utilization and sharing of resources.  CPU, memory, network connectivity, and storage are now pools of resources that you can pull from to run your business applications.  This allows you to quickly provision resources for applications as well as maintain higher levels of availability and reduce operational cost.  Virtualizing applications also makes them portable and able to be moved between datacenters r even between the public and private cloud.

 

The second step in building the Private cloud is securing the pools of resources that you have allocated to your applications. By protecting and isolating your applications we allow applications from multiple business units or even multiple companies to share resources in a concept we call Multi-Tenancy. Securing a multi-tenant environment ensures that one application cannot negatively impact another in the case of a misconfiguration or a security breach.  In VMware this is managed through vShield.  vShield can isolate devices and virtual networks and even allows for hardened virtual appliance to manage network traffic at layer 2 of the OSI model.

 

The third step which brings us further from just having a virtualized datacenter to having or own private cloud involves the tools for administration and management of the Private Cloud.  Providing a portal for business units to provide self-service reduces the involvement of IT.  Self-service also allows the business unit to bring applications online much quicker. Having the proper tools in place allows IT to manage this change efficiently and effectively is also important.  Configuration manager from VMware allows the business unit to provision an application from a pre-defined template. The template ensures a standardized and approved configuration. Configuration manager also provides auditing to ensure that the applications are compliant with IT policies, and manage resources such as software license keys.   IT can allocate resource pools to the business unit to use for provisioning or provide a shopping cart for the business to purchase the resources it needs.  This encourages the business unit to make intelligent business decision about which applications they decide to run and if the value to the business is worth the cost.  Configuration manager also handles the reporting on utilization of resources and billing business units.

 

This offers a dynamic change from traditional IT funded as a cost center to the business overall or as an internal budget "tax".  Directly relating the billing of IT costs to the resources used encourages managers to prioritize the resources that are used and determine which applications really add value to the business, reducing waste.  This also shifts IT from being a cost center to a revenue producing organization.  IT can also offer solutions or resources to other business partners.  For Example a hospital implementing a private cloud starts off by providing resources for it' own internal needs.  However, the same systems would also benefit their other affiliate hospitals.  The hospital can then offer their solution as a service for a fee to their business partners and bring in additional revenue for the business.  Similar situations could be done for school districts which have similar needs for their students such as Blackboard software and other course ware.  Rather than each school district building out their own infrastructure on school district could offer these applications as a service for a fee to their neighboring school district, or they could be offered by the state to all of the school districts.   This tremendously reduces the cost of implementing such systems.