5 ms·

Play stupid games, win stupid prizes. If you need 40,000 servers to keep your business running (which you don't, your ~3-8 million weekly transactions can be p

Play stupid games, win stupid prizes. If you need 40,000 servers to keep your business running (which you don't, your ~3-8 million weekly transactions can be processed on 1 computer, but whatever), hire people that will work on you, and whose paycheck depends on keeping those computers working, to keep those computers working. Game theory arguments like "they wouldn't screw me over because other people won't want to do business with them" don't work when the other party is trying to maximize quarterly earnings, and their long-term thinking is in the order of ~2 years.

amluto 1y ago
To be fair, and I know nothing about Tesco’s actual stack, a large grocery chain needs to track their contracts with suppliers, track their inventory in each location and in transit, track what goods they want in which locations, understand which larger pallets and big boxes contain which goods, track things prepped in house, and also optimize what to move from where, to where, and when and how. The latter part probably uses some spiffy stack involving something like CPLEX or Gurobi, and it’s not running on their “1 computer” OLTP stack. That being said, I don’t see what 40k servers is for unless the POS machines are thin clients that use a substantial fraction of a server each.
- regularfry 1y ago
  A long and storied history of nasty surprises has taught me to never underestimate the complexity of an unfamiliar domain. In that sense I'm surprised it's only 40,000.
  - Terr_ 1y ago
    Yeah, I'm pessimistically sure that there is other stuff, like: * Checking whether each item scanned has satisfied a logical contract for a discount, some of which may be per-region, per-store, or even per-customer. * If multiple exclusive coupons or deals are available, resolve the contradiction, preferably in favor of the customer. * Check if any items or quantities of items require an ID to be shown before proceeding, and record information about the employee authorizing it. * Update customer "rewards" data and generate any special offers so that you can put it onto their receipt. And that's not even starting to get into all the other less-customer-synchronous stuff that you still need CPU power somewhere to do. Managing stock levels, orders, deliveries, price changes, anti-"shrinkage", employee shifts, market-research, status and repairs of freezer-units, operational logging and telemetry, every form of reporting/dashboard "strategic insight" stuff beloved by upper management...
- alexvitkov 1y ago
  If you're doing 10 million transactions per week (which is likely way more than what they're pulling) that's about 16 transactions processed per second. You can add inventory management, payroll management, you can run the company's email server, write all that in JavaScript, and you'll still have room to run a Minecraft server on the same laptop. My point was not that running all that on one computer is a great idea, just that 40,000 servers for a CRUD application is way past what should be considered reasonable. But even that's fine. I like computers, you can have 40,000 of them if you want, even if the only reason they exist is some guy's job security. However, you're insane if the guy keeping them running doesn't work for you.
  - Anonbrit 1y ago
    You're suggesting that all Tesco branches should do all the PoS transactions online to a single server? Tell me you've never designed a system at this scale without telling me you've never designed something at this scale...
    - IsTom 1y ago
      Running it on a single laptop might be an exaggeration, but I can't imagine there's any essential complexity that requires more than a few dozen servers.
    - alexvitkov 1y ago
      No, I did not suggest that, in fact in the very comment you're replying to I said: My point was not that running all that on one computer is a great idea... Regardless, if you want to strawman my passing remark, I'm happy to defend it. Let's even say my numbers are wildly wrong, and they're processing 100x more transactions than what I claimed (which was already an overestimate). Tell me why you can't process 1600 transactions per second on one computer, especially for a country the size of the UK, where you would expect a ~15ms ping when talking to a server on the other side of the country.
      - lillecarl 1y ago
skydhash 1y ago
Most admins are more keen to shift blame than to keep things running. Having another company to point fingers at is more attractive than a proper functioning team.
jansper39 1y ago
Numbers I found vary but Tesco has around 3500 stores in the UK alone alongside other chains they have a hand in. They also have a large online presence, click and collect operations, estates, data collection schemes and a whole logistics network to operate. I'd have actually thought it would be higher than ~11 VMs per store.
- datadrivenangel 1y ago
  That works out to ~11 computers/licenses per store, which sounds a tad high but also very easy to do if you let new system accrete over time and factor in the need for offline operations and redundancy across regions.
  - csomar 1y ago
    The cashier register is probably running on a vmware.
jimt1234 1y ago
I'd hate to be the lowly, underpaid sysadmin who responded "40,000 servers" when asked the current number of servers, but he meant to respond "4,000 servers". LOL
ajcp 1y ago
You are severely underappreciating how complex a retail organization of Tesco's size is. I work in this space for a retailer almost the same size as Tesco and when factoring in all the attendant organizations, businesses, and functions it requires, 40k servers does not surprise me at all. Outside of just the brick-and-mortar stores you have Marketing, Retail, eComm, Merchandising, Strategic Sourcing, FP&A, Finance & Accounting, Asset Protection, Corporate Real Estate, Retail Real Estate, Internal Audit, Supply Chain, Transportation, Business Services, Data Science, etc etc. and IT at every level of those. Each one of these components is large enough to be a medium-to-large sized company in its own right.
lemmsjid 1y ago
You’re ignoring general heavy workloads such as observability. How much telemetry do they gather and analyze for tracking and fraud detection. A quick google on Tesco engineering shows that they process 35k qps against couchbase, and 35 terrabytes of telemetry data per day. They track 150k devices in their ecosystem, which, reading between the lines, would produce the telemetry and require observability, state management, anomaly detection, etc. They have hundreds of thousands of employees using these devices for varying purposes. We’re talking quite a bit of compute, which also requires high availability. I know nothing about Tesco beyond that quick google search, but I’ve been at several companies where I would read online comments claiming we could reduce our workload to a few servers, and I would think of our tens of thousands of fully loaded machines and roll my eyes.

amluto 1y ago

To be fair, and I know nothing about Tesco’s actual stack, a large grocery chain needs to track their contracts with suppliers, track their inventory in each location and in transit, track what goods they want in which locations, understand which larger pallets and big boxes contain which goods, track things prepped in house, and also optimize what to move from where, to where, and when and how. The latter part probably uses some spiffy stack involving something like CPLEX or Gurobi, and it’s not running on their “1 computer” OLTP stack. That being said, I don’t see what 40k servers is for unless the POS machines are thin clients that use a substantial fraction of a server each.

regularfry 1y ago
A long and storied history of nasty surprises has taught me to never underestimate the complexity of an unfamiliar domain. In that sense I'm surprised it's only 40,000.
- Terr_ 1y ago
  Yeah, I'm pessimistically sure that there is other stuff, like: * Checking whether each item scanned has satisfied a logical contract for a discount, some of which may be per-region, per-store, or even per-customer. * If multiple exclusive coupons or deals are available, resolve the contradiction, preferably in favor of the customer. * Check if any items or quantities of items require an ID to be shown before proceeding, and record information about the employee authorizing it. * Update customer "rewards" data and generate any special offers so that you can put it onto their receipt. And that's not even starting to get into all the other less-customer-synchronous stuff that you still need CPU power somewhere to do. Managing stock levels, orders, deliveries, price changes, anti-"shrinkage", employee shifts, market-research, status and repairs of freezer-units, operational logging and telemetry, every form of reporting/dashboard "strategic insight" stuff beloved by upper management...
alexvitkov 1y ago
If you're doing 10 million transactions per week (which is likely way more than what they're pulling) that's about 16 transactions processed per second. You can add inventory management, payroll management, you can run the company's email server, write all that in JavaScript, and you'll still have room to run a Minecraft server on the same laptop. My point was not that running all that on one computer is a great idea, just that 40,000 servers for a CRUD application is way past what should be considered reasonable. But even that's fine. I like computers, you can have 40,000 of them if you want, even if the only reason they exist is some guy's job security. However, you're insane if the guy keeping them running doesn't work for you.
- Anonbrit 1y ago
  You're suggesting that all Tesco branches should do all the PoS transactions online to a single server? Tell me you've never designed a system at this scale without telling me you've never designed something at this scale...
  - IsTom 1y ago
    Running it on a single laptop might be an exaggeration, but I can't imagine there's any essential complexity that requires more than a few dozen servers.
  - alexvitkov 1y ago
    No, I did not suggest that, in fact in the very comment you're replying to I said: My point was not that running all that on one computer is a great idea... Regardless, if you want to strawman my passing remark, I'm happy to defend it. Let's even say my numbers are wildly wrong, and they're processing 100x more transactions than what I claimed (which was already an overestimate). Tell me why you can't process 1600 transactions per second on one computer, especially for a country the size of the UK, where you would expect a ~15ms ping when talking to a server on the other side of the country.
    - lillecarl 1y ago

skydhash 1y ago

Most admins are more keen to shift blame than to keep things running. Having another company to point fingers at is more attractive than a proper functioning team.

jansper39 1y ago

Numbers I found vary but Tesco has around 3500 stores in the UK alone alongside other chains they have a hand in. They also have a large online presence, click and collect operations, estates, data collection schemes and a whole logistics network to operate. I'd have actually thought it would be higher than ~11 VMs per store.

datadrivenangel 1y ago
That works out to ~11 computers/licenses per store, which sounds a tad high but also very easy to do if you let new system accrete over time and factor in the need for offline operations and redundancy across regions.
- csomar 1y ago
  The cashier register is probably running on a vmware.

jimt1234 1y ago

I'd hate to be the lowly, underpaid sysadmin who responded "40,000 servers" when asked the current number of servers, but he meant to respond "4,000 servers". LOL

ajcp 1y ago

You are severely underappreciating how complex a retail organization of Tesco's size is. I work in this space for a retailer almost the same size as Tesco and when factoring in all the attendant organizations, businesses, and functions it requires, 40k servers does not surprise me at all. Outside of just the brick-and-mortar stores you have Marketing, Retail, eComm, Merchandising, Strategic Sourcing, FP&A, Finance & Accounting, Asset Protection, Corporate Real Estate, Retail Real Estate, Internal Audit, Supply Chain, Transportation, Business Services, Data Science, etc etc. and IT at every level of those. Each one of these components is large enough to be a medium-to-large sized company in its own right.

lemmsjid 1y ago

You’re ignoring general heavy workloads such as observability. How much telemetry do they gather and analyze for tracking and fraud detection. A quick google on Tesco engineering shows that they process 35k qps against couchbase, and 35 terrabytes of telemetry data per day. They track 150k devices in their ecosystem, which, reading between the lines, would produce the telemetry and require observability, state management, anomaly detection, etc. They have hundreds of thousands of employees using these devices for varying purposes. We’re talking quite a bit of compute, which also requires high availability. I know nothing about Tesco beyond that quick google search, but I’ve been at several companies where I would read online comments claiming we could reduce our workload to a few servers, and I would think of our tens of thousands of fully loaded machines and roll my eyes.