Hi all,

The main server that hosts the Saturation Hall and Langstonedale websites has suffered a disk failure, meaning both sites are off-line for the time being. A replacement disk has been sourced and I should be rebuilding the machine tomorrow (Friday), once the new disk is in and backups have been restored, normal service will resume, we'll update here when it's done.

Anyone who's membership was due to expire within the next week will have their end date extended for a week from the point we resume service, so no-one will miss out on any updates.

Note our UMD download stores are unaffected by this, it only affects the membership websites.

Download stores (operating as normal):

https://saturationhall.umd.net/ - Saturation Hall and Langstonedale content
https://imperatrix.umd.net/ - Kinkier stuff

Membership websites (down until the server is restored):

http://saturationhall.com
http://langstonedale.com

Apologies for the outage.

RAID and hot-swap bays are your friends

I was about to ask the same thing, RAID 1+6 is a glorious thing! So much parity.

I'm sure its more complicated than just "a disk had failed" DM1?

Normal service resumed as of UK lunchtime today (Friday) however see below for more details.


Much geekery follows below, but for customers the main point is: The replacement drive turned out to be faulty so has been returned to the dealer for replacement, however we've managed to resurrect the original drive for the time being, we plan that there will now be another brief outage, probably next week, when we get the replacement new disk. In the event of any extended outage (if the recovered original disk fails again meanwhile), membership expirys will be extended to ensure no-one misses any updates.

Detail: I'm well familiar with RAID in my IT day work, RAID50 is my preferred config for both speed and resilience while not wasting too much of the disk capacity, but the Saturation Hall server doesn't have RAID, I built it six or seven years ago when we were only just starting to turn a profit, and it has a 1TB disk for the OS, a 4TB drive for /home, and another 4TB for /var. / and /home are WD Black drives, /var is a Seagate Barracuda. When I built it I'd not yet discovered the joys of the BackBlaze hard drive reliability stats, which nowadays are my bible for all things HDD.

It's the /home disk that's failing, and today's plan was to swap it out for a brand new 10TB WD Ultrastar. When the machine was de-racked and powered back up on the bench, fdisk managed to restore /home (I'd previously tried that remotely but it failed claiming the superblock was toast). So I thought to add the new disk as /home2, rsync all the data over, shut down, remove old disk, remount new one as /home, and off we go.

Except that when I powered the system up with the new disk connected, it started making noises like a demented dentist drill (so much so that staff from the datacentre office next door popped in to see what the noise was), and was clearly beyond use. It sounded like it had a bent platter scraping the interior of the disk space, as we'd had it delivereed, I suspect it had been damaged in transit. So the machine is back in service with the original disk for now, hopefullty it will remain operational long enough for our suppliers to provide a replacement disk - and this time we'll collect from the warehouse instead of having it delivered.

For the long term future, I'm tempted to build a ZFS array, in a hot-swap casing, been playing with that on the replacement for the main file server here at the Hall, with great results so far.

Thanks for the explanation, I knew it would be more complicated, why drive iv with all that off the bat if you don't need too.

Have you considered mdadm, it's mainly a matter of preference between the two, I just find it more flexible than zfs. You'll might even be able to do your rebuild with the one disk you are planning but put it in the array and promote disks later when you are ready?

One server I have at home is mdadm, raid 1 with a hot spare, anything fails the spare takes over and I can switch the broken one and promote new disks into the array whilst it's running. Way too overthinking for home but you have to have a hobby right, lol.

Any further geeking we sould take into private, i sense we'll get a complaint soon! Lol quick talk wam!

Loving your new rainwear shoots you've done recently, any chance you've done any more? About a year ago you shared a picture of someone in a blue rainsuit shopping and then covered in custard, when is thay coming out?

No problem re explanation, happy to provide details for people who are interested, plus I know customers can tell when you try and pull the wool, I'd rather be up front and honest with people.

That scene you mention featured Maude in the rainwear, we went shopping for snacks with her in costume for the next scene send I took a photo of her outside an Asda in the rainsuit. No planned release date, when it comes time for each next release I look at what models, outfit styles, and substances we have recently released, and pick one from the queue to keep it varied. With the district now under enhanced lockdown I've just cancelled all shoots and visitors to the Hall from now to at least Midwinter and most likely all the way to the spring equinox, but as we've years worth of scenes already in the can our updates and releases will just keep rolling.

I realised the other day we've not released many gunge (as opposed to food) scenes recently, so will probably start putting those out on a regular basis, perhaps alternating food / gunge each release through the winter.

Update: I've now received and installed a working new disk, and copying of the website data off the old one has commenced. This is going to take some time due to the state of the original disk - it'll read a few dozen files then pause for several seconds before reading any more, plus we're seeing data errors and read failures from time to time. Both Saturaton Hall and Langstonedale are down for the time being, I've just extended the memberships of everyone who was due to expire between 20th September and 10th October to 12th October, though we don't expect it to take quite that long. This includes members who had expired, who will be re-acivated at midnight UK time tonight. We expect Langstonedale to be restored first with Saturation Hall to follow.

We won't be skipping any updates because of this, when full service is resumed we'll release everything that should have been released during the downtime. We've never skipped a scheduled update in fifteen years, so we don't plan to start now.

Further announcements will follow as we get the sites back into full service.