Do yourself a favour...

Discussion in 'Virtual and Cloud Computing' started by zebulebu, Dec 17, 2008.

  1. zebulebu

    zebulebu Terabyte Poster

    3,748
    330
    187
    If you sit down in your lab one day and start to hear a 'click' sound every five seconds or so coming from, well, pretty much ANY of your boxes, it might be time for you to check that box out. Not, as I did, leave it for five days then come in one day to find your ESX hosts spitting out lovely warnings about not being able to locate paths to iSCSI targets, and your VI client complaining about datastore unavailability...

    Yep, you guessed it - hard disk failure on my Openfiler box. The disks aren't even three months old and one has already gone 'pop' - taking eight of my VMs with it :cry:

    Luckily I'd backed them up a week ago, but its taught me two salutary lessons:

    Lesson one - Don't ignore clicking sounds from your hard disks (duh!)

    Lesson two - Don't scrimp on your filer - get a third disk and RAID 5 it instead of being greedy and trying to eke out as much storage as you can (ummm... RAID 0 isn't particularly noted for its fault tolerance :oops:)

    Arse!
     
    Certifications: A few
    WIP: None - f*** 'em
  2. DrkSnpr4

    DrkSnpr4 Bit Poster

    10
    0
    2
    Oh, don't worry. It is good that you've learnt something from this. What were the hard disks that you are using?
     
  3. zebulebu

    zebulebu Terabyte Poster

    3,748
    330
    187
    Seagate Barracuda 7.2k 250Gb SATAs.

    Warranty or no warranty I'm not putting any of the f*****s back in there - I'm going back to Western Digitals. I've always used them and never had a problem before (other than EOL issues which you can't really moan about) but this is the second Seagate drive that's **** the bed on me in the last six months. That's not a failure rate I'm comfortable with - irrespective of whether its just bad luck.
     
    Certifications: A few
    WIP: None - f*** 'em
  4. Kitkatninja
    Highly Decorated Member Award 500 Likes Award

    Kitkatninja aka me, myself & I Moderator

    11,139
    555
    383
    Wow, Seagate? I've never had any problems with them? Maybe a duff batch... The only HDD's I've ever had poblems with are Fujitsu's, I will never touch them again if I can help it.

    -ken
     
    Certifications: MSc, PGDip, PGCert, BSc, HNC, LCGI, MBCS CITP, MCP, MCSA, MCSE, MCE, A+, N+, S+, Server+
    WIP: MSc Cyber Security
  5. Phoenix
    Honorary Member

    Phoenix 53656e696f7220 4d6f64

    5,749
    200
    246
    I loves me some WD, got WDs that are over 5 years old (the ones that had 1yr warranty) still kicking good with close to 24/7 uptime

    love those buggers

    and yeah for the price of an extra drive and even a hw raid card, go RAID! (Raid 10 is what im thinking next, that way i get performance and redundancy, and **** a 1TB disk is < $100 these days)
     
    Certifications: MCSE, MCITP, VCP
    WIP: > 0
  6. zebulebu

    zebulebu Terabyte Poster

    3,748
    330
    187
    Ryan - stupid thing is, I bought a damn e200i card to use but never got round to putting it in the filer. Rest assured that I've done that now, and it's currently sitting there waiting for 2Tb of Western Digital Goodness to arrive from Dabs tomorrow :biggrin

    I've never had a problem with a WD drive yet - I had one run for nearly eight years in my old smoothwall box (finally gave up the ghost six months ago, after somewhere close to 99.9% uptime since February 2000!)

    How's the job going? May be over in the US next year around March for a month as we're moving our colo from Yaaarkshire to our parent company's data centre in Virginia (nowhere near Idaho, sadly!)
     
    Certifications: A few
    WIP: None - f*** 'em
  7. Phoenix
    Honorary Member

    Phoenix 53656e696f7220 4d6f64

    5,749
    200
    246
    Hey I may be in Boston at EMC for training or something, you never know!

    Been employed 3 weeks and I'm down in Orange County for a weeks training, then Jan 4th flying out to Montana for an install (it's already touching -5c in Billings Montana, and I'm heading so some tiny place called Miles City)

    Apparently I will need more than my fleece (the warmest outer layer I own) for the trip, think I can probably expense that ;)

    They are a good bunch here, technically minded, I dont need to put my laptop on the domain, can put whatever OS I want on it, did not even have to ASK for the 4GB memory upgrade (the upgrade to 8gb would sadly cost 800 bucks, think I may need to ask for that)
    Got a shiney new HP Elitebook, 1650x1050 screen, decent hdd, 4GB ram, 64bit OS, eSATA, HDMI, ExpressCard/54, all the bells and whistles!

    Spent most of my lunches recently at 'lunch meetings' getting to meet new clients, and can expense mileage that costs me $15 for about $100, not a bad start to employed life, so different from my last cowboy outfit

    the irony is my last outfit was a multi national and I'm now working for a 100 man outfit that spans the whole north west (9 folks in our office)

    Anyway, going well! :)
    Got my new ride the other week (need two now as the missis and I both work seperate places/hours)
    2008 Ford Edge, can tow my ATVs and Snowmobiles up to the hills nicely (one I have them, first ATV should be arriving in Feb!)
     
    Certifications: MCSE, MCITP, VCP
    WIP: > 0
  8. mattstevenson

    mattstevenson Byte Poster

    214
    6
    44
    We generally find that when we send a drive back to Seagate and they 'repair' it, it's pretty much certain that it's going to fail again. I don't think we've ever had a fully working drive returned to us. I wish they'd just replace the damn thing, particularly since we're going to be wasting time with warranty repairs over, and over, and over. *Sigh.*
     
    Certifications: Triple A+. Network+, CCENT
    WIP: MCP, ICND2, Sec+
  9. ThomasMc

    ThomasMc Gigabyte Poster

    1,507
    49
    111
    WD VelociRaptors are what am looking at right now and so far i can't fault them, well worth a shoofty
     
    Certifications: MCDST|FtOCC
    WIP: MCSA(70-270|70-290|70-291)
  10. zebulebu

    zebulebu Terabyte Poster

    3,748
    330
    187
    Yeeesh - what a friggin nightmare this turned out to be!

    Just spent close to six hours restoring everything - including farting around with ESX to get rid of some damn VMs that kept haning around in a failed state. MAJOR pain in the arse. In the end I managed to salvage five of the seventeen VMs I had in my cluster by bringing up local VMFS on one of the hosts, pulling the VMs across to that, vaping the SAN volumes then rebuilding my Openfiler box from scratch - this time making sure I RAID5'd the sucka. Re-upped the iSCSI targets and did a global rescan, then Pulled all my VMs from backup - which took about four hours.

    Net result - two new volumes at 310Gb a piece, with 310Gb left over for snapshots on the filer. RAID5 performance will be poor with only three disks, but I haven't got a molex splitter and HP - in their infinite wisdom - only put three spare molex adapters on the PSU for the ML110 - meaning I've got a 500Gb WD HDD sat on the bench doing nowt. Trip to maplins later this week to get a splitter and I'll put it in and expand the array. Ho hum.

    Once more - NEVER ignore a clicking sound from a HDD in a non-redundant array - even if the disk is only two months old!

    Nevereverevereverevereverevereverevereverever!
     
    Certifications: A few
    WIP: None - f*** 'em
  11. Arroryn

    Arroryn we're all dooooooomed Moderator

    4,015
    193
    209
    Thanks for posting your experience, Zeb - it's been quite an interesting read!

    It's reassuring for a relative noob like me that the mighty also go 'whoops' as well. (for whoops, read expletives, but I've been watching Blackadder Goes Forth far too much recently).
     
    Certifications: A+, N+, MCDST, 70-410, 70-411
    WIP: Modern Languages BA
  12. zebulebu

    zebulebu Terabyte Poster

    3,748
    330
    187
    LOL - I wouldn't call myself 'mighty' after this! 'Dopey' might be a more accurate assessment! :oops:
     
    Certifications: A few
    WIP: None - f*** 'em
  13. Raffaz

    Raffaz Kebab Lover Gold Member

    2,976
    56
    184
    Its stories like this that make me happy that ive got a RAID 5 setup with 4 x 1tb samsung drives:)
     
    Certifications: A+, MCP, MCDST, AutoCAD
    WIP: Rennovating my house
  14. Arroryn

    Arroryn we're all dooooooomed Moderator

    4,015
    193
    209
    After the fun I had with the Filers, I'm not in a position to call anybody dopey, lol :wink:
     
    Certifications: A+, N+, MCDST, 70-410, 70-411
    WIP: Modern Languages BA
  15. zebulebu

    zebulebu Terabyte Poster

    3,748
    330
    187
    Ya - did what I suggested work for you in the end?
     
    Certifications: A few
    WIP: None - f*** 'em
  16. Arroryn

    Arroryn we're all dooooooomed Moderator

    4,015
    193
    209
    It did, thank you. It took f**king ages, but it worked :biggrin
     
    Certifications: A+, N+, MCDST, 70-410, 70-411
    WIP: Modern Languages BA

Share This Page

Loading...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.