Cloud

[quixoticgeek]

I fucking hate the cloud.

This rant is brought to you by having to deal with a sysadmin that thinks the cloud is a good idea.

I really fucking hate the cloud.

J

[Afasoas]

I fucking hate the cloud.

This rant is brought to you by having to deal with a sysadmin that thinks the cloud is a good idea.

I really fucking hate the cloud.

J

Am I allowed to laugh at that?
Only because I think I'd be ranting much akin to you in the same position^*.

*_{I don't hate the cloud, but I'm tired of people who pretend it is a panacea and a solution to all problems. It's very good for solving some problems and it is very good for cost effectively dealing with situations where you've got very elastic work loads or you need certain levels of cheap geo-redundancy but it's not like you migrate everything to the road and suddenly we all start farting rainbows.}

[Afasoas]

Actually, I think 5-10 years from now, cloud is going to have far reaching and chilling effects on freedom and choice as far computing goes. The continuing decline in on-prem and self-hosting could see servers become very expensive as they become a niche product. Truly niche open source Linux distros will see their users and support dwindle, because taking a virtual machine or managed service relying on Amazon or Azure Linux will be the path of least resistance. The recently announced change to RHEL and the ongoing flurry of Open Core companies changing their licenses to try and stop AWS, Azure and GCP beating them out of existence by competing with their own codebase is just the beginning.

Dark times ahead.

[Ben T]

I quite like the cloud. Well, sort of. In a similar way to how miners like coal.

[pcolbeck]

The Cloud is just marketing speak for renting compute and networking in someone else's datacentre it isn't really anything new.
Whats made it new and shiny is how easy it is to consume theses days which is the result of the development of APIs for all this stuff that have allowed the creation of front ends to deploy tenants in the network and compute.

It's certainly not a panacea but like most things it has its uses.

I do love management types who think they can move their entire enterprise into the cloud and check the pricing every six months and migrate in the blink of an eye to which ever other cloud platform is cheaper at the time. They have definitely read too much marketing copy.

One of the best things about it is that most people retaining their own datacentres are now following the model used by cloud providers and not building point solutions for each service but just building fast leaf and spine network fabrics with blocks of identical compute attached. Just spin up a VM for whatever you need to do and assign it as much CPU and storage as required. Much cleaner than the old way of doing things.

[ian]

Hmm, looks at spreadsheet, I can safely say what these subdecks of the mothership's greatest expenditure is.

It works though, anything requires more storage, processing, or memory, it's just a bigger bill.

[Greenbank]

It can be awesome when you've got those kind of resources available internally (and don't have to worry about billing).

We have a test suite that runs on a build of our product. A build takes 30 minutes on a fast x86 box (it could probably go faster if the underlying Makefiles were written in a way that supported 'make -j' properly but that's a separate story).

The full test suite takes close to 12 hours to run on each build (some tests are designed to run for minutes rather than seconds). (It takes close to 24 hours to run on the slowest platform, which is becoming a problem for our nightly builds as we add more tests). We have dedicated build machines for each platform but only a few devtest machines for each platform.

It means that if you make a change you generally have to wait until the next day for the full test suite to be run (although you can run individual tests or suites of tests yourself).

With the cloud we plan on spinning up as many servers as we want for a short amount of time, tell them to grab the latest build and then get them to pull individual tests to be run from a central server. So if we tell it to spin up 20 machines then it can all be done in about an hour. All of this done through simple API calls.

Done properly we'll be able to make it do the entire test suite in 10 minutes by simply changing a single value from 20 to 100. (Obviously not every company has that level of resources available, and 100 servers is a tiny fraction of what is available internally.)

[Greenbank]

We also have a cloud based system for dealing with customer issues.

I can get fully GDPR compliant clean box(es) of the spec of my choosing for looking at customer data within ~60 seconds.

Access is via a gateway Windows VM that is accessed through a browser page. It works surprisingly well as it just goes full screen and you forget you're accessing it remotely. The remote desktop access prevents most copy-and-paste (we're trusted to copy out certain things like error messages/etc and small log snippets as long as we know they shouldn't include customer data). Network access to the machine(s) is cut off (except for the remote desktop obviously) as soon as you've finished setting it up and copied all of the information/binaries/code/etc onto the machine(s) that you need, once it's locked down the customer data files become available and there's a way to push more files onto the machine (for debug builds, testfixes, scripts, etc) but no way to get data off the machine. You can create any number of Windows/Linux VMs on the private side too. The VMs for each customer case are isolated so you can't leak info between cases either.

So I can think "I need to look at case AB012345678 and I'll need a RHEL 7.9 VM with 8GB RAM and 4 cores and an RHEL 8.3VM with 4GB RAM and 2 cores" and that's a few mouse clicks and it'll be ready and running in under 60 seconds. Connecting to the Windows VM gateway machine is just a mouse click and a login, then I scp in my setup script (which downloads and installs a version of our product), that's another 60 seconds, click the button to lock the environment and make the customer files available, that completes (in under a minute, even with GBs of logs) and I'm ready to go.

The old-fashioned pre-GDPR method of running things locally took longer.

I also have access to similar system for system that's HIPAA compliant too.

[ian]

All the mothership's platforms run on the cloud now (ironically it's cheaper, we used to be part of a bigger company and were 'obliged' to the use the datacentre of another part of the company at rates that were, erm, not commerically viable*). I've been working on a project most of this year that needed a non-negotiable excess of 300 GB RAM and a lot of cores if you wanted a runtime that wasn't measured in weeks. If we had to spend capital on that sort of hardware, well, we wouldn't.

*I remember the old school though, I used to work for a US startup who rented rackspace, datacentre relocation involved us loading all the servers and hardware into the back of the CTO's pickup and driving as quickly as possible across Virginia in the dead of night to minimize downtime.

[Greenbank]

One of the internal cloud environments has 783,124 GB of memory allocated to VMs at the moment.

Can't even begin to imagine what AWS/Google/Azure have between them.

[Afasoas]

We spun out a couple of cloud solutions for WFH.

The first thing was capacity in the regions we are allowed to use. Restrictions applied on the types of compute instances for instance, because of lack of capacity in those regions.

The second element was performance. You really do have to pay to get it. Or you Netflix it up. Spin up an instance, performance test it, if it doesn't come up to spec, kill the instance and spin up another one.

Then there's orchestration. We are getting that down to a fine art for our on-prem infrastructure. Even installing servers is trivial. Stack, rack, push configuration. We automated the cloud configuration and the sheer number of API calls we finished up with was surprising. And we found a lot of inconsistencies in the APIs, adding to the complexity and the weight of it. We evaluated a few tools to help, but they fell into two camps. In one camp they are all singing and all dancing, but carry some fairly stifling assumptions/constraints. In the other, they are much more flexible but you have to do a lot more work yourself. So we wrote our own, perhaps niaively underestimating the number of moving parts we would end up with.

I think the third surprise was how long provisioning of some services can take. Many of the APIs are asynchronous and if you have dependencies, your orchestration has to sit their polling endpoints until provisioning tasks are complete.

I think the final niggle is that many of the services and appliances available, I feel, have yet to reach maturity. You kind of end up having to choose in many cases between a service, an appliance or just getting a compute instance and provisioning the service yourself. So that leads to a lot of evaluation/testing/trial/error.

In terms of cost, we regularly compare our on-prem costs to equivalent cloud hosting costs. Colo fees for on-prem are not cheap. Hardware is not cheap. There are costs to in testing hardware, driving to data centres and support contracts. But cloud hosting always comes out more expensive. Way way more expensive.

We have fairly comprehensive monitoring of our cloud infrastructure, from our data centres and other locations. The connectivity into the data centres is much more reliable. The number of timeouts in simple checks to cloud hosted services I see on a day to day basis vastly out number those I see for our on-prem end points. And less than 5% of our infrastructure is cloud hosted.

Cloud is very useful for some things. CDNs. Messaging. Queues. Things you have reasons for not hosting on your own infrastructure. Things on which latency and performance don't matter. Geo-redundant storage of backups. Spiking out ideas. Start-ups with little capital, etc..

[Afasoas]

Seems apt:

https://www.theregister.com/2020/12/10/google_cloud_over_run/

[Morat]

I host an MS SQL Server on-prem for our EPOS and Time and Attendance systems. It's a VM on VMWare. I thought the licensing for the OS and SQL itself was exorbitant and it IS compared to the old days when you put a single license on a single box. Thing is, if you compare with SQL on Azure it's a snip.

[Chris S]

I'm currently looking at ways to utilise Kubernetes for our live deployments, that doesn't involve spending the company's meager profits on Azure or AWS, and yet minimises the pain of on-prem setup. I don't want to spend the Chrimbo hols going blind reading manuals and watching HowTo videos on You Tube.

Sucks to be the only tech guy in the company that knows even the first thing about Cloud solutions. And all I know is, it's really really easy to rack up a vast bill on one of the favoured platforms, without even trying.

[Morat]

I'm currently looking at ways to utilise Kubernetes for our live deployments, that doesn't involve spending the company's meager profits on Azure or AWS, and yet minimises the pain of on-prem setup. I don't want to spend the Chrimbo hols going blind reading manuals and watching HowTo videos on You Tube.

Sucks to be the only tech guy in the company that knows even the first thing about Cloud solutions. And all I know is, it's really really easy to rack up a vast bill on one of the favoured platforms, without even trying.

I suspect the only way to make Cloud pay is to sack all the sysadmin/network/infra guys and just hire devs.
Ulp!
It probably makes sense if you're looking down from on a single cost centre called "IT" from a great height* and think that all the little people below are completely interchangeable.

*Like the Ferris wheel scene in "The Third Man"

[ian]

Hey, we have one left. He presses the big button labelled CLOUD.