7 ms·
Walmart Will Manage Distribution Centers with OneOps, Jenkins, and Kubernetes
- philip1209 10y agoNFS in a modern system seems like a huge red flag.
- existencebox 10y agoSpeaking as someone who has maintained some rather large NFS utilizing systems/datacenters in the last decade, and found it perfectly suitable to my needs, I'd be curious to why you'd say that?
- brobinson 10y agoNot a block device, so not having cgroups (read/write bps limiting + iops limiting) is one annoyance of NFS in large deployments
- Karunamon 10y agoI'd also add extreme fault intolerance. Even when you're mounting soft, or hard+intr, NFS failures have a way of requiring reboots to fully fix.
- Scaevolus 10y agoThe intr mount option has been a no-op for 8 years. /bin/umount does an fstat before the umount syscall, so if an NFS mount is broken, it will probably hang forever instead of unmounting it! You can invoke the syscall directly to get the behavior you need.
- acdha 10y agoThat hasn't been dependable in my experience - the best workaround I've found is either a lazy unmount + remount if you can tolerate the hung process taking the full NFS retry timeout to unblock or downing the network interface to accelerate the process.
- xir78 10y agoThis is fair to a degree, but vendors have VIPs that make failures fairly transparent. But if you need low latency on a large data set is there something better? Where low latency means <100ma reads
- X86BSD 10y agoSo use Jails, ZFS and rctl. Problem solved.
- xorgar831 10y agoI'd agree, it's hard to beat if you need high bandwidth and low latency access to large data sets.
- beachstartup 10y agothrowing away 30+ years of protocol development for the latest and greatest containerization paradigm is what the cool kids south of market street talk about at happy hour.
- cosmie 10y agoWhat's wrong with NFS? If you need a shared, POSIX-compliant drive between multiple hosts, there's not really an alternative. Even the fairly new AWS Elastic File System service is just a NFS4.1-based managed service
- jacques_chester 10y agoWhile the grandparent comment might be a little too emphatic, surrendering the flexibility of file stores in favour of object stores does lend itself to permitting nice non-functional properties.
- falsedan 10y agoYou can pay money for a SAN, which is $$$$ & enterprise but is a viable alternative…
- jo909 10y agoA SAN gives you block storage - virtual hard disks. You will need a cluster filesystem to be able to use the same virtual disks on multiple machines concurrently, which are not easy to operate and have their own bunch of performance problems.
- fh973 10y agoSome of the file systems come with native clients. I'd say for container environments that's a viable alternative to NFS. When done properly the native client can do low latency as it usually has the minimum number of network hops and do failover without requiring things like virtual IPs.
- superuser2 10y agoIf you want volumes that can be dynamically attached to any machine, you're not buying a SAN, and you're not in a cloud provider with a portable block storage abstraction, what do you propose? About the only thing I can think of is Ceph, and it stands to reason there is a lot more NFS than Ceph expertise on the job market.
- deleted 10y ago[deleted]
- shiftoutbox 10y agoNope nope nope nope
- jacques_chester 10y agoThis is one of the better bespoke cloud efforts I've seen, but in their position -- given the decision to bet on Kubernetes -- I might've chosen OpenShift instead of rolling my own PaaS. I've also seen the Jenkins-to-Nexus thing a few times, never particularly happily. That said, I don't have particularly deep experience in Java shops, so it's possible that it works really well in some places. Disclosure: I work for Pivotal, we're the majority donors of engineering to Cloud Foundry, a PaaS competing with OpenShift.
- empath75 10y agoI messed around with oneops a bit and it's basically a bunch of tooling built around their own fork of chef. It's almost impossible to just drop into an existing workflow without a whole team of people managing it.
- avitzurel 10y agoI had a very interesting discussion about this subject a couple of weeks ago with a CTO of a big company. I've commented before that Kube absolutely takes over the bigger more complex cloud installations out there, you can see how many companies are betting their infrastructure on it. The only thing that I don't see is standardization of the cloud, just like what Amazon did. You see too many companies doing too many of the same things and reinventing the wheel. Personally, I would love to see smaller installation as a standard of how to take things into the cloud as a cluster. Imagine what Heroku did to deployments. You can't beat this ease of use. Deis and Convox are both trying but not really "hitting it". As for Walmart, absolutely stoked seeing it from them. This move and what the white-house did with the digital shows a lot of promise and hope. I wonder how much of this is on top of "older" management and how much is just complete restructuring.
- jacques_chester 10y ago> The only thing that I don't see is standardization of the cloud, just like what Amazon did ... Deis and Convox are trying but not really "hitting it" My colleagues at Red Hat might disagree, they're working on OpenShift. My more immediate colleagues at Pivotal, IBM, SAP, Microsoft, Google, Cisco, Dell-EMC, VMWare et al might disagree too. We're working on Cloud Foundry and BOSH. Disclosure: I work for Pivotal.
- avitzurel 10y agoOpenShift origin looks very promising for sure. That's the thing though, you still see too many people not using it. Here are things that should be standard (IMHO) and you shouldn't reinvent 1. Underlying infrastructure auto scaling (Google, AWS) 2. Service Discovery 3. DNS (internal and external for multiple sources) 4. Networking The real issue that I feel no-one really answered is who's this for. If a SRE is the target audience than there's a lot more we're missing as a community.
- jacques_chester 10y agoI think Red Hat's problem with OpenShift is getting out from under the brand-masking power of tech they chose. Too many engineers want to roll their own Docker+Kubernetes platform and underestimate the difficulty of doing so. The thing is, everyone has a different don't-reinvent list. And they want to be not reinvented in different ways. Then they discover all the things that they didn't realise they'll need to reinvent. > The real issue that I feel no-one really answered is who's this for. I see PaaSes as serving three constituencies. Operators, who wish to christ that Developers would stop making their lives hell by breaking stuff. Developers, who wish to christ that Operators would stop making their lives hell by blocking stuff. Business, who wonder why everything takes so long, costs so much and breaks so often.
- falsedan 10y ago> As shown in the above diagram, Jenkins pipelines deliver application updates directly to the enterprise Nexus instance Well, at least they're not using Jenkins to run their deploys.
- bluedino 10y agoWal-Mart is a very odd to work with from the vendor side. They have a purchase order tracking system named PULSE that is completely independent from their EDI system and their supplier Retail Link program (which is the slowest, most horribly designed website you could imagine). You're required to SFTP up a CSV file of what you've shipped to them that day, along with information about. They also have a creaky acknowledgement/acceptance procedure and the technical folks (seems outsourced) aren't very impressive. You have to pass a 'visual inspection' with your test files and it's the most ridiculous process you can imagine with box-checkers flagging you for the dumbest reasons. So from this side of the transaction it always makes me wonder about articles like this, all their good tech must be internal only. To be fair to Wal-Mart the situation with other big store chains usually isn't any better.
- user5994461 10y agoSFTP and CSV are robust and simple. That's solid choices.