6 ms·
Years and years ago I saw an advertisement by a SAN array vendor where their gimmick was that they supported very cheap read/write snapshots of very large datas
by jiggawatts 6y ago
Years and years ago I saw an advertisement by a SAN array vendor where their gimmick was that they supported very cheap read/write snapshots of very large datasets, in combination with a set of transforms such as data masking or anonymisation.
Their target market was same as the OP: developers that need rapid like-for-like clones of production. The SAN could create a full VM server pool with a logical clone of hundreds of terabytes of data in seconds, spin it up, and blow it away.
The idea was that instead of the typical "DEV/TST/PRD" environments, you'd have potentially dozens of numbered test environments. It was cheap enough to deploy a full cluster of something as complex as SAP or a multi-tier Oracle application for a CI/CD integration test!
Meanwhile, in the Azure cloud: Practically nothing has a "copy" option. Deploying basic resources such as SQL Virtual Machines or App Service environments can take hours. Snapshotting a VM takes a full copy. Etc...
It's like the public cloud is in a weird way ahead of the curve, yet living with 1990s limitations.
Speaking of limitations: One reason cheap cloning is "hard" is because of IPv4 addressing. There are few enough addresses (even in the 10.0.0.0/8 private range) that subnets have to be manually preallocated to avoid conflicts, addresses have to be "managed" and carefully "doled out". This makes it completely impossible to deploy many copies of complex multi-tier applications.
The public cloud vendors had their opportunity to use a flat IPv6 address space for everything. It would have made hundreds points of complexity simply vanish. The programming analogy is like going from 16-bit addressing with the complexity of near & far pointers to a flat 64-bit virtual address space. It's not just about more memory! The programming paradigms are different. Same thing with networking. IPv6 simply eliminates entire categories of networking configuration and complexity.
PS: Azure's network team decided to unnecessarily use NAT for IPv6, making it exactly (100%!) as complex as IPv4...
- znpy 6y ago> There are few enough addresses (even in the 10.0.0.0/8 private range) that subnets have to be manually preallocated to avoid conflicts I hate to be that guy, but it's 2021 and we should start deploying stuff on ivp6 networks to avoid this kind of problems (beside other kind of problems)... edit: ok I just saw the PS about ipv6 and Azure... sorry.
- dilyevsky 6y agogcp doesn’t even give you ipv6 at all. Talk about limitations...
- pgwhalen 6y agoYour comment makes me realize how lucky I am at my job: we do this production environment replication via “SAN magic” and it is so insanely useful.
- rurounijones 6y agoHow do you deal with replicating customer data / data the developer / tester should not have access to? Do you copy that into the ephemeral test environment as well? How are permissions managd for accessing that data? Does it have the same restrictions as in prod? (i.e. even in this env the developer / tester has no access to systems / data they would not have in prod).
- tilolebo 6y agoThis is an interesting question. I've seen 1:1 copy of prod databases running in test environments. Very convenient from a dev perspective but a no-go for many customers. I've also seen test envs using fixtures only. More privacy-compliant, but it makes it harder for in-depth testing or to uncover weird edge cases.
- Archelaos 6y agoUsing real person-related data for testing was made illegal in Europe by GDPR. Even with explicit consent of each affected person, the tester needs to make sure that the test really requires real person-related data, otherwise it would violate the principle of "privacy by design" from GDPR. Without consent it is entirely forbidden, as of my understanding of the GDPR. Due to GDPR I had to implement an anonymization feature for the production database anyway. So I am able to clone the production database for testing and run the anonymization function on every entry in the test database before using it.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- otabdeveloper4 6y ago> Azure's network team decided to unnecessarily use NAT for IPv6, making it exactly (100%!) as complex as IPv4... People like the NAT. It's a feature, not a bug. If your selling point for IPv6 is "no more NAT" then no wonder it never went anywhere! P.S. No, "you're doing it wrong" and "you're not allowed to like the NAT" are not valid responses to user needs.
- funcDropShadow 6y agoWhat do people like about NAT? I am guessing that the perceived increase in security. But perhaps there are more real or perceived advantages.
- w7 6y agoWhat I find common is people conflate NAT with stateful firewalling, and believe that if you lose NAT you lose all forms of edge/perimeter network security. They don't understand that you can still filter and prevent unwanted packets from reaching hosts without NAT.
- otabdeveloper4 6y agoOf course you can. But why would you? You're replacing something that is simple, easy to understand and works perfectly well with a nebulous something that invites user error and security nightmares. For example, my (modest) home LAN is five routers, a NAS/media server, a media player, two "smart TVs" and dozens of notebooks and phones connected via Wi-Fi. What do you propose? Manage a firewall on each of those devices? I suppose you mean setting up a firewall on the WAN link to block all incoming traffic? How is that different from a NAT? Merely a lack of 'masquerade' setting on the firewall rule? What's the benefit to me and why should I care? Or do you propose some sort of hybrid scheme to intelligently block traffic while making all my countless devices pingable from the Internet? Not in this timeline, sorry.
- femiagbabiaka 6y agoYour home network and a cloud datacenter aren’t comparable. Many clouds have host level firewall policies as a core feature, and anyone competent is managing them profile-style using Terraform or an equivalent. It’s really quite easy from that perspective.
- whoisburbansky 6y agoDo you remember the name of the original SAN array provider?
- znpy 6y agoIt could be NetApp, they had a huge marketing wave 2-3 years ago. In fairness, their appliances are very good, I've seen them in action both on-prem and in the cloud (on AWS). The pricing however... They're very expensive, particularly so in the cloud (since you're paying licensing on top of cloud resources).
- rad_gruchalski 6y agoOnTAP Cloud? That's what their system was called.
- znpy 6y agoYup, we were beta testers or something the time
- rad_gruchalski 6y agoI still have the 8.1 manuals laying around here. Nifty technology but pricey, indeed.
- mcny 6y ago> Meanwhile, in the Azure cloud: Practically nothing has a "copy" option. Deploying basic resources such as SQL Virtual Machines or App Service environments can take hours. Snapshotting a VM takes a full copy. Etc... To add to the list of etc, I can't query across two databases in azure SQL.
- jiggawatts 6y agoYou can't even copy an Azure SQL Database across subscriptions! This is something that comes up regularly at my day job: there's an Enterprise Agreement production subscription, and an Enterprise Dev/Test subscription for non-production that's much cheaper. This would be great, except for stupid limitations like this.
- steverb 6y agoI believe you can now query across DBs in Azure via a Linked Server. It still feels a little clunky to me though. https://docs.microsoft.com/en-us/sql/relational-databases/linked-servers/linked-servers-database-engine?view=sql-server-ver15 https://docs.microsoft.com/en-us/sql/relational-databases/li...
- olavgg 6y agoI want to a copy of our Azure SQL database dumped to my local dev MS SQL instance very often. At the moment I do this by using SSMS, where I generate a script of the Azure SQL database and use sqlcmd to import that script. It is a really painful process, and sqlcmd sometimes just hang and becomes unresponsive. Why is there no simple export/import functionality?
- jiggawatts 6y agoYou can export Azure SQL Databases to a "BACPAC" file, which is much faster. This can then be restored onto on-prem SQL equally fast. It's about 20-30 lines of PowerShell to do this, and then you can just double-click the script and drink a coffee. Alternatively, for data-only changes you can use Azure SQL Sync, but I've found that it's primitive and a bit useless for anything complicated...