5 ms·

This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is

by beachy 5y ago

This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative that you have the ability to selectively restore individual customers. The usual approach is to do a full database restore into a separate instance, then run your pre-prepared "restore customer" scripts to extract a single customer's data from there and pump it across your prod instance. In Oracle for example you might use database links to give your restore code access to prod and also the restore instance at the same time. Atlassian - MUST DO BETTER.

drsim 5y ago
Which SaaS platforms provide account-level restores? If you contact them and say "please restore our data to as it was last week" those I know do not offer this.
- hotpotamus 5y ago
  I actually did this once with Dropbox, though it wasn't a feature they actually published. I clobbered my Dropbox directory accidentally, but I was able to find a script someone wrote to roll it back to a previous point in time and it worked quite well. After that I also took my own snapshots just in case.
  - MapleWalnut 5y ago
    Dropbox support can rollback your Dropbox account to a previous point in time too.
- boardwaalk 5y ago
  I wouldn’t expect them to advertise such a thing, but the question is “can they recover from their own mistakes” not “can they recover from mine.” I don’t care if this is with an “account-level restore” or whatever; it shouldn’t be my concern.
  - brazzledazzle 5y ago
    I’ve seen customer and resource level restores deprioritized more than once and the only hypothetical given serious thought is avoiding helping customers who accidentally deleted something because of the support burden/cost. No one seems to have much concern for what happens when they’re the ones that screwed up.
- fknorangesite 5y ago
  I wouldn't expect it if I just asked. I think it's reasonable as part of their disaster recovery though.
- Tobani 5y ago
  I accidentally built out this feature at a company once and it totally saved our asses a week later.
- ibejoeb 5y ago
  I did. It was an first-principles architectural decision. A client could request any point-in-time within the contracted period, and it could be either a restoration or a fully operational, parallel instance of the account. It was initially a cover-my-own-ass design, but it turned out to be an extremely popular feature that was never even used for disaster recovery. Instead, it was used for audit support, trial scenarios, projections, and all kinds of other stuff.
  - jimbokun 5y ago
    What kind of data volumes were you dealing with?
- kgeist 5y ago
  >Which SaaS platforms provide account-level restores? We restore deleted accounts on request sometimes. There was a client, for example, who forgot to renew the subscription and did nothing for 30 days, so their account was automatically deleted. We restored it from backups. It helps that every tenant has their own isolated database, so it's mostly a matter of restoring that one single DB. Some microservices store data without DB-level sharding, so we have a script which is able to make a partial dump for a specific account. There's also a popular option to restore deleted data - nothing is ever hard-deleted (it's marked deleted but stays in the DB) and we have a script which can restore individual records (and related records). There's maybe 5 such requests per month. We don't offer rolling everything back to a specific point in time, though. Technically it's possible by undoing the event queue but it's untested. We also have a script to migrate customers from cloud to on-premises and back.
- twistedpair 5y ago
  I know plenty of places (small/med startups) with "undelete" and "restore" account/data options built into their admin panels. Engineers shouldn't be flipping bits by hand, under duress. I really wonder what these Altassian restore tools look like it if takes "hundreds of engineers across the company" to restore 400 accounts. Are backups siloed across many teams?
Animats 5y ago
Rather, customers must stop using Atlassian cloud services.
- imroot 5y ago
  Which is becoming more and more difficult due to them focusing on Cloud Products (my on-prem renewal jumped almost 8x this year). I’d rather use request tracker or bugzilla over Atlassian these days
  - randmeerkat 5y ago
    Try Kitemaker, it’s a YComb backed JIRA alternative. https://kitemaker.co/ https://kitemaker.co/
    - Aeolun 5y ago
      This landing page is exceedingly light on explaining what problem they solve for me.
- Karrot_Kream 5y ago
  This is a great opportunity to be an Atlassian competitor. I'm sure more than one business added Atlassian cloud services as a business risk in the meantime, even if they weren't affected.
  - galangalalgol 5y ago
    gitlab seems to be eating their lunch already. Its bit bucket plus Jenkins plus the best bits of jira and it's starting price is free... I like confluence a lot, but separate design/documentation tools with their files pushed up to the repo are good enough. Latex, doxygen, or PowerPoint, whatever you want.
scottlamb 5y ago
Is it standard for a RFP to have a long list of questions like this? I've never been involved in an RFP from either side. Is it standard to (in addition or instead) to have something more general/forward-looking like: how do you watch other providers' postmortems and apply the lessons to your own system? > - Can you restore data for a single customer, and if so, what is the RTO for that operation? If I were to aim something at this specifically, it'd be: can you restore data for N customers or N% of customers, and if so, what is the RTO for that operation? I mentioned in another comment that Gmail had a similar outage in which they had to restore from tape. https://news.ycombinator.com/item?id=31017160 https://news.ycombinator.com/item?id=31017160 They had a tool for restoring a single account but not for restoring N accounts in bulk, which would be significantly more efficient than doing the one-account process N times. (E.g., in the case of tape backups, imagine the difference between pulling data from the tape library sequentially for each user vs all N at once, particularly when one tape may hold data for many of these customers.)
- taude 5y ago
  Yes, pages of them. Multiple pages of security questions, ciphers used, how data is stored, when is it encrypted, etc. I filled out a 20 pager once. As the company got better and more mature, we had a bunch of canned answers to make it easier and faster....
  - fphhotchips 5y ago
    Entire (excellent) start-ups exist to fill the role of 'RFP library' so that you don't have the whole sales team rewriting the same answers 100 times a year. Loopio saved me hours in the last role I was in that had them - even if you do have to edit some of the responses from colleagues you're not sure passed 9th grade English.
    - porker 5y ago
      Any other startups you can recommend? I'm filling in my first RFP in a decade and answering what they mean to ask with the questions rather than answering questions literally is not something that comes easily to me. Plus coming up with an answer to the vague question on "describe your project methodology" (I build what you want, it works - nope, they expect half a page). Or the 3 questions on project management systems and communication software choices that to my reading should have the same answer.
shp0ngle 5y ago
> Atlassian - MUST DO BETTER. it’s not like people will stop using jira and confluence, lol they basically have a monopoly there
- bastardoperator 5y ago
  Nobody likes using this stuff though. I think the new github boards might give jira a run for their money given the cost... free.
  - pjmlp 5y ago
    Here is someone that likes it, and is usually put down by lack of features in competing products.
  - eddy_chan 5y ago
    My engineering team runs off Github Issues > Projects > Project Boards but it's been hard to get anyone outside of product managers and devs themselves to be fully immersed in it. Sales and management throw their hands up in the air and say 'Github Issues is too hard'. I'm a PM so I've fought and won this battle to stay on Github Issues because I could quantify that a 5-10% loss in dev productivity in moving to a 'management friendly' tool would equal so many thousands of dollars of developer hours per month.
  - shp0ngle 5y ago
    Once a project gets to feature parity with atlassian, it will become atlassian. Same with SAP. Once your ERP has same feature parity with SAP, it will become SAP. JIRA is SAP of engineering... yeah why I haven't thought of that before.
dx034 5y ago
I still don't get why they didn't separate clients on a database level. Sure, put many clients on one database server to save resources. But why not use different databases? They cost nothing and provide perfect separation. It also drastically lowers the attack surface as you can set all permissions via database software. And if they had done that, this would've never been a multi-day outage. If Jira was a product used by individuals I'd get it. Maybe a database is overkill for a sole developer. But pretty much all users of Jira are companies with tens or hundreds of users on average. I don't see how separating on a db level is overkill in that situation.
- beachy 5y ago
  There are lots of downsides to doing absolute partitioning of tenants (along with lots of upsides as you point out). Really annoying things that slash your velocity. You can't easily run pan-customer queries, can't aggregate data. Ironically too using a single database makes full backup and restore much easier.
  - dx034 5y ago
    Wouldn't you extract data anyway to another system for analytics? Running analytics queries on production databases seems a bit risky in any setting?
    - treis 5y ago
      You can have a read-only replica if you're concerned about performance.
- donavanm 5y ago
  > why not use different databases? They cost nothing and provide perfect separation. I understand the sentiment, but This is a pretty simplistic take that I very much doubt will hold true for meaningful traffic. Many databases have licensing considerations that arent amenable. Beyond that you get in to density and resource problems as simple as IO, processes, threads etc. But most of all theres the time and effort burden in supporting migrations, schema updates, etc. Yes layered logical separation is a really good idea. Its also really expensive once you start dealing with organic growth and a meaningful number of discrete customers. Disclaimer: Principal at AWS who was helped build and run services with both multi tenant and single tenant architectures.

by beachy 5y ago

drsim 5y ago

Which SaaS platforms provide account-level restores? If you contact them and say "please restore our data to as it was last week" those I know do not offer this.

hotpotamus 5y ago
I actually did this once with Dropbox, though it wasn't a feature they actually published. I clobbered my Dropbox directory accidentally, but I was able to find a script someone wrote to roll it back to a previous point in time and it worked quite well. After that I also took my own snapshots just in case.
- MapleWalnut 5y ago
  Dropbox support can rollback your Dropbox account to a previous point in time too.
boardwaalk 5y ago
I wouldn’t expect them to advertise such a thing, but the question is “can they recover from their own mistakes” not “can they recover from mine.” I don’t care if this is with an “account-level restore” or whatever; it shouldn’t be my concern.
- brazzledazzle 5y ago
  I’ve seen customer and resource level restores deprioritized more than once and the only hypothetical given serious thought is avoiding helping customers who accidentally deleted something because of the support burden/cost. No one seems to have much concern for what happens when they’re the ones that screwed up.
fknorangesite 5y ago
I wouldn't expect it if I just asked. I think it's reasonable as part of their disaster recovery though.
Tobani 5y ago
I accidentally built out this feature at a company once and it totally saved our asses a week later.
ibejoeb 5y ago
I did. It was an first-principles architectural decision. A client could request any point-in-time within the contracted period, and it could be either a restoration or a fully operational, parallel instance of the account. It was initially a cover-my-own-ass design, but it turned out to be an extremely popular feature that was never even used for disaster recovery. Instead, it was used for audit support, trial scenarios, projections, and all kinds of other stuff.
- jimbokun 5y ago
  What kind of data volumes were you dealing with?
kgeist 5y ago
>Which SaaS platforms provide account-level restores? We restore deleted accounts on request sometimes. There was a client, for example, who forgot to renew the subscription and did nothing for 30 days, so their account was automatically deleted. We restored it from backups. It helps that every tenant has their own isolated database, so it's mostly a matter of restoring that one single DB. Some microservices store data without DB-level sharding, so we have a script which is able to make a partial dump for a specific account. There's also a popular option to restore deleted data - nothing is ever hard-deleted (it's marked deleted but stays in the DB) and we have a script which can restore individual records (and related records). There's maybe 5 such requests per month. We don't offer rolling everything back to a specific point in time, though. Technically it's possible by undoing the event queue but it's untested. We also have a script to migrate customers from cloud to on-premises and back.
twistedpair 5y ago
I know plenty of places (small/med startups) with "undelete" and "restore" account/data options built into their admin panels. Engineers shouldn't be flipping bits by hand, under duress. I really wonder what these Altassian restore tools look like it if takes "hundreds of engineers across the company" to restore 400 accounts. Are backups siloed across many teams?

Animats 5y ago

Rather, customers must stop using Atlassian cloud services.

imroot 5y ago
Which is becoming more and more difficult due to them focusing on Cloud Products (my on-prem renewal jumped almost 8x this year). I’d rather use request tracker or bugzilla over Atlassian these days
- randmeerkat 5y ago
  Try Kitemaker, it’s a YComb backed JIRA alternative. https://kitemaker.co/ https://kitemaker.co/
  - Aeolun 5y ago
    This landing page is exceedingly light on explaining what problem they solve for me.
Karrot_Kream 5y ago
This is a great opportunity to be an Atlassian competitor. I'm sure more than one business added Atlassian cloud services as a business risk in the meantime, even if they weren't affected.
- galangalalgol 5y ago
  gitlab seems to be eating their lunch already. Its bit bucket plus Jenkins plus the best bits of jira and it's starting price is free... I like confluence a lot, but separate design/documentation tools with their files pushed up to the repo are good enough. Latex, doxygen, or PowerPoint, whatever you want.

scottlamb 5y ago

Is it standard for a RFP to have a long list of questions like this? I've never been involved in an RFP from either side. Is it standard to (in addition or instead) to have something more general/forward-looking like: how do you watch other providers' postmortems and apply the lessons to your own system? > - Can you restore data for a single customer, and if so, what is the RTO for that operation? If I were to aim something at this specifically, it'd be: can you restore data for N customers or N% of customers, and if so, what is the RTO for that operation? I mentioned in another comment that Gmail had a similar outage in which they had to restore from tape. https://news.ycombinator.com/item?id=31017160 https://news.ycombinator.com/item?id=31017160 They had a tool for restoring a single account but not for restoring N accounts in bulk, which would be significantly more efficient than doing the one-account process N times. (E.g., in the case of tape backups, imagine the difference between pulling data from the tape library sequentially for each user vs all N at once, particularly when one tape may hold data for many of these customers.)

taude 5y ago
Yes, pages of them. Multiple pages of security questions, ciphers used, how data is stored, when is it encrypted, etc. I filled out a 20 pager once. As the company got better and more mature, we had a bunch of canned answers to make it easier and faster....
- fphhotchips 5y ago
  Entire (excellent) start-ups exist to fill the role of 'RFP library' so that you don't have the whole sales team rewriting the same answers 100 times a year. Loopio saved me hours in the last role I was in that had them - even if you do have to edit some of the responses from colleagues you're not sure passed 9th grade English.
  - porker 5y ago
    Any other startups you can recommend? I'm filling in my first RFP in a decade and answering what they mean to ask with the questions rather than answering questions literally is not something that comes easily to me. Plus coming up with an answer to the vague question on "describe your project methodology" (I build what you want, it works - nope, they expect half a page). Or the 3 questions on project management systems and communication software choices that to my reading should have the same answer.

shp0ngle 5y ago

> Atlassian - MUST DO BETTER. it’s not like people will stop using jira and confluence, lol they basically have a monopoly there

bastardoperator 5y ago
Nobody likes using this stuff though. I think the new github boards might give jira a run for their money given the cost... free.
- pjmlp 5y ago
  Here is someone that likes it, and is usually put down by lack of features in competing products.
- eddy_chan 5y ago
  My engineering team runs off Github Issues > Projects > Project Boards but it's been hard to get anyone outside of product managers and devs themselves to be fully immersed in it. Sales and management throw their hands up in the air and say 'Github Issues is too hard'. I'm a PM so I've fought and won this battle to stay on Github Issues because I could quantify that a 5-10% loss in dev productivity in moving to a 'management friendly' tool would equal so many thousands of dollars of developer hours per month.
- shp0ngle 5y ago
  Once a project gets to feature parity with atlassian, it will become atlassian. Same with SAP. Once your ERP has same feature parity with SAP, it will become SAP. JIRA is SAP of engineering... yeah why I haven't thought of that before.

dx034 5y ago

I still don't get why they didn't separate clients on a database level. Sure, put many clients on one database server to save resources. But why not use different databases? They cost nothing and provide perfect separation. It also drastically lowers the attack surface as you can set all permissions via database software. And if they had done that, this would've never been a multi-day outage. If Jira was a product used by individuals I'd get it. Maybe a database is overkill for a sole developer. But pretty much all users of Jira are companies with tens or hundreds of users on average. I don't see how separating on a db level is overkill in that situation.

beachy 5y ago
There are lots of downsides to doing absolute partitioning of tenants (along with lots of upsides as you point out). Really annoying things that slash your velocity. You can't easily run pan-customer queries, can't aggregate data. Ironically too using a single database makes full backup and restore much easier.
- dx034 5y ago
  Wouldn't you extract data anyway to another system for analytics? Running analytics queries on production databases seems a bit risky in any setting?
  - treis 5y ago
    You can have a read-only replica if you're concerned about performance.
donavanm 5y ago
> why not use different databases? They cost nothing and provide perfect separation. I understand the sentiment, but This is a pretty simplistic take that I very much doubt will hold true for meaningful traffic. Many databases have licensing considerations that arent amenable. Beyond that you get in to density and resource problems as simple as IO, processes, threads etc. But most of all theres the time and effort burden in supporting migrations, schema updates, etc. Yes layered logical separation is a really good idea. Its also really expensive once you start dealing with organic growth and a meaningful number of discrete customers. Disclaimer: Principal at AWS who was helped build and run services with both multi tenant and single tenant architectures.