3 ms·

> hiring in some infrastructure people Part of the problem is that these, at least the ones making the fundamental decisions, are likely to be software people

by mmt 8y ago

> hiring in some infrastructure people Part of the problem is that these, at least the ones making the fundamental decisions, are likely to be software people, with not enough ops/sysadmin experience. > who actually know how to make a stable platform using _proven "boring"_ stuff, not sexy The ones who do may have gotten bored of it, themselves, and no longer admit it, instead leaning toward as much of the "Dev" part of DevOps (as a title) as possible, since that's where the excitement (and decision making power) tends to be. The rest of us are borderline unemployable.

KaiserPro 8y ago
I don't quite understand that statement. With the exception of my time at a financial news paper, virtually all professional life has been trying to make highly scalable pipelines that run as smoothly as possible. In any project, you have about three innovation tokens to spend. Each token will cost you time and momentum, however they may give you lots of reward. Personally I like spending them on the bits of the software that give you the USP, Not something thats already solved. To illustrate the point: using loads of d blades with internal storage to create a Ceph storage mesh == expensive nightmare and slow, especially as you are going to be serving it over NFS. Storage bandwidth is contending serving bandwidth. Not only that you now have three SPOFs the NFS servers, the ceph cluster (no, its really not HA) and the blade fabric. three or four real file servers with proper jbods/raid arrays if you want to go down the NFS route. the smaller the better, spread the load amongst them evenly, meaning that if one goes down it degrades but not totally wipes out access. Better yet to use SAS/FC to route block access, and either pay for hardware duplication, or write the state to two different LUNs (its git after all, its fairly simple to duplicate.) failing that, just us VMware and some decent block storage, and let it figure out how to do it. It has rather reasonable HA clustering, which don't need software changes. or just host it in the cloud. Which is what they did in the end.
- mmt 8y ago
  > I don't quite understand that statement. I didn't make a single statement, but I also made what is more of an accusation (i.e. stating the opinion of others I disagree with) than my own assertion. > Personally I like spending them on the bits of the software that give you the USP, Not something thats already solved. I'm suggesting that most people (or at least in my own "bubble", as it were) don't differentiate. Re-inventing the wheel is just fine to them, so long as it's fun. That said, I'm not convinced large (enough) "global" filesystems are a solved problem. Of course, GitLab doesn't necessarily need such a thing, but implementing that kind of storage "sharding" (for lack of a better word) may well be spending an innovation token, too. > or just host it in the cloud. Except that doesn't magically solve any of the architectural/operational problems. For all we know they're still doing Ceph-with-NFS, just paying much more to do so, without the ability to optimize the hardware for cost and/or performance.
  - KaiserPro 8y ago
    Gotcha. I do agree that, a lot of my time can be taken up with explaining why I don't want to replace a file server with a Cassandra cluster(yes really). I was at one point all for single name space global filesystems. GPFS makes data affinity, HSM and provide rich event hooks. However, its global, which means global outages. It has its user cases though. Depending on your use case sharding a file system can be pretty simple. The root folder can be a directory of dynamic symlinks controlled by config management. However its never as simple as that, as you know.
    - mmt 8y ago
      > I do agree that, a lot of my time can be taken up with explaining why I don't want to replace a file server with a Cassandra cluster(yes really). Adding the complexity of a DBMS (and a non-standard one at that) on top of a distributed system seems excessive, but I'd believe it. > GPFS I've never been a strong advocate for proprietary/enterprise solutions, as that's an easy way to spend more than a cloud provider would cost. Since storage (hardware) is the worst offender for markups, I'm extra-sensitive in that area. > However its never as simple as that, as you know. Indeed, which is why I suggested it would require innovation. For any non-trivial case, the app would need to be aware, and it may as well do all that work.

by mmt 8y ago

KaiserPro 8y ago

I don't quite understand that statement. With the exception of my time at a financial news paper, virtually all professional life has been trying to make highly scalable pipelines that run as smoothly as possible. In any project, you have about three innovation tokens to spend. Each token will cost you time and momentum, however they may give you lots of reward. Personally I like spending them on the bits of the software that give you the USP, Not something thats already solved. To illustrate the point: using loads of d blades with internal storage to create a Ceph storage mesh == expensive nightmare and slow, especially as you are going to be serving it over NFS. Storage bandwidth is contending serving bandwidth. Not only that you now have three SPOFs the NFS servers, the ceph cluster (no, its really not HA) and the blade fabric. three or four real file servers with proper jbods/raid arrays if you want to go down the NFS route. the smaller the better, spread the load amongst them evenly, meaning that if one goes down it degrades but not totally wipes out access. Better yet to use SAS/FC to route block access, and either pay for hardware duplication, or write the state to two different LUNs (its git after all, its fairly simple to duplicate.) failing that, just us VMware and some decent block storage, and let it figure out how to do it. It has rather reasonable HA clustering, which don't need software changes. or just host it in the cloud. Which is what they did in the end.

mmt 8y ago
> I don't quite understand that statement. I didn't make a single statement, but I also made what is more of an accusation (i.e. stating the opinion of others I disagree with) than my own assertion. > Personally I like spending them on the bits of the software that give you the USP, Not something thats already solved. I'm suggesting that most people (or at least in my own "bubble", as it were) don't differentiate. Re-inventing the wheel is just fine to them, so long as it's fun. That said, I'm not convinced large (enough) "global" filesystems are a solved problem. Of course, GitLab doesn't necessarily need such a thing, but implementing that kind of storage "sharding" (for lack of a better word) may well be spending an innovation token, too. > or just host it in the cloud. Except that doesn't magically solve any of the architectural/operational problems. For all we know they're still doing Ceph-with-NFS, just paying much more to do so, without the ability to optimize the hardware for cost and/or performance.
- KaiserPro 8y ago
  Gotcha. I do agree that, a lot of my time can be taken up with explaining why I don't want to replace a file server with a Cassandra cluster(yes really). I was at one point all for single name space global filesystems. GPFS makes data affinity, HSM and provide rich event hooks. However, its global, which means global outages. It has its user cases though. Depending on your use case sharding a file system can be pretty simple. The root folder can be a directory of dynamic symlinks controlled by config management. However its never as simple as that, as you know.
  - mmt 8y ago
    > I do agree that, a lot of my time can be taken up with explaining why I don't want to replace a file server with a Cassandra cluster(yes really). Adding the complexity of a DBMS (and a non-standard one at that) on top of a distributed system seems excessive, but I'd believe it. > GPFS I've never been a strong advocate for proprietary/enterprise solutions, as that's an easy way to spend more than a cloud provider would cost. Since storage (hardware) is the worst offender for markups, I'm extra-sensitive in that area. > However its never as simple as that, as you know. Indeed, which is why I suggested it would require innovation. For any non-trivial case, the app would need to be aware, and it may as well do all that work.