4 ms·

Gitlab Epic Issue Relationships Deleted

62 points by sverhagen 5y ago

m4r35n357 5y ago
???????????????
vesinisa 5y ago
Seems like they have it mostly resolved, the releationships were restored from a backup 3 hours prior to the incident, and they are working to restore the data for the 313 (sic.) items still affected: https://gitlab.com/gitlab-org/gitlab/-/issues/348547 https://gitlab.com/gitlab-org/gitlab/-/issues/348547
- ta20200710 5y ago
  What is "(sic)" about "they are working to restore the data for the 313 items still affected"?
  - vesinisa 5y ago
    Sorry - I meant "as cited", i.e. there were exactly 313 items still requiring attention (presumably number of links that were created between the last backup and the incident.) This is a very low number, bordering on something you could fix by hand or at least a small script. This is actually what they did, apparently: https://gitlab.com/gitlab-org/gitlab/-/issues/348547#note_777669404 https://gitlab.com/gitlab-org/gitlab/-/issues/348547#note_77...
- slimsag 5y ago
  I am curious, how frequently does GitLab backup their DB? 3 hours seems pretty recent
  - dikei 5y ago
    I would expected them to have point-in-time recovery, especially from the last time when they accidentally dropped the production database.
  - raffraffraff 5y ago
    Worst case in 2021 should be a daily full backup + transaction logs. That's enough to get you point in time recovery. On AWS, EBS snapshots are incremental (I didn't know that until studying for AWS certification), so you could schedule a snapshot every 5 minutes if you wanted. Looking at the timeline, I'm just gonna repeat something I said in another thread this week: it's easy to underestimate how long it takes to do basically anything with a few hundred GB of important data.
    - Aeolun 5y ago
      Found this out recently when my 250GB database had to roll back a transaction and was stuck for hours :/ Restoring a new database from the AWS snapshot and transaction logs was literally faster.
  - dspillett 5y ago
    For a full backup of a huge DB a three-hour window is pretty small, but I doubt that is what is happening here. A far more common setup is regular full backups, often daily though sometimes more/less frequent, with much smaller transaction log backups at high frequency, perhaps every 15 minutes, between. That way you lose at most what has happened since the last log backup, and you can restore to any point in time between the full backup and the last log backup. It takes more effort to restore as you need to first restore the full then restore (replay) the log backups in sequence until the point you care about. Sometimes there is a third layer between: differential backups. These are usually much smaller than a full backup, while also smaller (and less faf/time to restore) than the log backups for the period they cover, but don't offer point-in-time recovery.
    - arcatek 5y ago
      Out of curiosity, how do transaction logs handle things like created_at fields, or randomly generated UUID, which rely on contextual data? Is the server time/rng seed faked for each replayed transaction?
      - dikei 5y ago
        In most cases, transaction logs/write-ahead logs will contain the return value of non-determistic functions like created_at or random uuid, instead of the function call.
      - dspillett 5y ago
        The transaction log is everything that gets done, not how it is done, so it can be replayed reliably. How values that are random, time sensitive, or otherwise arbitrary, are derived during a transaction is not important. What is logged is the fact that the values x/y/z were recorded in row 123,456 which is in page 987,654,321¹ which means that when the log is replayed you end up with exactly the same state the original database was in at the point the log is run to. [1] in fact it could just be logged at the page level, the granularity of the log structure will vary between systems, if logged at the row level it may be the case that the physical datafile after restore is not exactly the same but the data will still be “random” values & all.
nefitty 5y ago
I was confused by the title. Apparently epics are a way to organize issues by themes across projects. GitHub doesn't have this concept, in case anyone else was wondering. https://docs.gitlab.com/ee/user/group/epics/ https://docs.gitlab.com/ee/user/group/epics/ https://github.community/t/epics-github-issues/1430 https://github.community/t/epics-github-issues/1430
- orangejewce 5y ago
  That's because github isn't trying to be a replacement for JIRA and source control for most teams.
  - pilif 5y ago
    their latest updates to their project management features[1] launched as part of the "new" issues are totally going in that direction though. I'm unsure as to whether that's a good thing or not. 1) https://github.blog/2021-06-23-introducing-new-github-issues/ https://github.blog/2021-06-23-introducing-new-github-issues...
  - tyingq 5y ago
    https://github.com/features/issues https://github.com/features/issues Scroll down to "break issues into actionable tasks", and they have an example that's even labeled as an "Epic". They are definitely headed in the same direction as Gitlab.
    - nefitty 5y ago
      Ah, thanks for pointing that out. I missed it on my quick check.
- Karellen 5y ago
  Huh. While trying to figure out the reasoning behind the name used, TIL that "epic" has acquired a new meaning of "(computing) In software development, a large or extended user story." https://en.wiktionary.org/wiki/epic https://en.wiktionary.org/wiki/epic
nix23 5y ago
I was hoping that this has something todo that i CANNOT delete my UnrealEngine repo/project in gitlab...but no, well it's not my storage ;)
- deleted 5y ago
  [deleted]
sarki_247 5y ago
GitLab team member here, most of the affected relationships have been restored. If you are still affected, please let us know in the incident issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6072 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6...
m4tthumphrey 5y ago
Getting an eye into how the Gitlab team are working to resolve this is mind boggling (in a good way!) Just look at everything going on in this issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6072 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6...

62 points by sverhagen 5y ago

???????????????

Seems like they have it mostly resolved, the releationships were restored from a backup 3 hours prior to the incident, and they are working to restore the data for the 313 (sic.) items still affected: https://gitlab.com/gitlab-org/gitlab/-/issues/348547 https://gitlab.com/gitlab-org/gitlab/-/issues/348547

ta20200710 5y ago
What is "(sic)" about "they are working to restore the data for the 313 items still affected"?
- vesinisa 5y ago
  Sorry - I meant "as cited", i.e. there were exactly 313 items still requiring attention (presumably number of links that were created between the last backup and the incident.) This is a very low number, bordering on something you could fix by hand or at least a small script. This is actually what they did, apparently: https://gitlab.com/gitlab-org/gitlab/-/issues/348547#note_777669404 https://gitlab.com/gitlab-org/gitlab/-/issues/348547#note_77...
slimsag 5y ago
I am curious, how frequently does GitLab backup their DB? 3 hours seems pretty recent
- dikei 5y ago
  I would expected them to have point-in-time recovery, especially from the last time when they accidentally dropped the production database.
- raffraffraff 5y ago
  Worst case in 2021 should be a daily full backup + transaction logs. That's enough to get you point in time recovery. On AWS, EBS snapshots are incremental (I didn't know that until studying for AWS certification), so you could schedule a snapshot every 5 minutes if you wanted. Looking at the timeline, I'm just gonna repeat something I said in another thread this week: it's easy to underestimate how long it takes to do basically anything with a few hundred GB of important data.
  - Aeolun 5y ago
    Found this out recently when my 250GB database had to roll back a transaction and was stuck for hours :/ Restoring a new database from the AWS snapshot and transaction logs was literally faster.
- dspillett 5y ago
  For a full backup of a huge DB a three-hour window is pretty small, but I doubt that is what is happening here. A far more common setup is regular full backups, often daily though sometimes more/less frequent, with much smaller transaction log backups at high frequency, perhaps every 15 minutes, between. That way you lose at most what has happened since the last log backup, and you can restore to any point in time between the full backup and the last log backup. It takes more effort to restore as you need to first restore the full then restore (replay) the log backups in sequence until the point you care about. Sometimes there is a third layer between: differential backups. These are usually much smaller than a full backup, while also smaller (and less faf/time to restore) than the log backups for the period they cover, but don't offer point-in-time recovery.
  - arcatek 5y ago
    Out of curiosity, how do transaction logs handle things like created_at fields, or randomly generated UUID, which rely on contextual data? Is the server time/rng seed faked for each replayed transaction?
    - dikei 5y ago
      In most cases, transaction logs/write-ahead logs will contain the return value of non-determistic functions like created_at or random uuid, instead of the function call.
    - dspillett 5y ago
      The transaction log is everything that gets done, not how it is done, so it can be replayed reliably. How values that are random, time sensitive, or otherwise arbitrary, are derived during a transaction is not important. What is logged is the fact that the values x/y/z were recorded in row 123,456 which is in page 987,654,321¹ which means that when the log is replayed you end up with exactly the same state the original database was in at the point the log is run to. [1] in fact it could just be logged at the page level, the granularity of the log structure will vary between systems, if logged at the row level it may be the case that the physical datafile after restore is not exactly the same but the data will still be “random” values & all.

nefitty 5y ago

I was confused by the title. Apparently epics are a way to organize issues by themes across projects. GitHub doesn't have this concept, in case anyone else was wondering. https://docs.gitlab.com/ee/user/group/epics/ https://docs.gitlab.com/ee/user/group/epics/ https://github.community/t/epics-github-issues/1430 https://github.community/t/epics-github-issues/1430

orangejewce 5y ago
That's because github isn't trying to be a replacement for JIRA and source control for most teams.
- pilif 5y ago
  their latest updates to their project management features[1] launched as part of the "new" issues are totally going in that direction though. I'm unsure as to whether that's a good thing or not. 1) https://github.blog/2021-06-23-introducing-new-github-issues/ https://github.blog/2021-06-23-introducing-new-github-issues...
- tyingq 5y ago
  https://github.com/features/issues https://github.com/features/issues Scroll down to "break issues into actionable tasks", and they have an example that's even labeled as an "Epic". They are definitely headed in the same direction as Gitlab.
  - nefitty 5y ago
    Ah, thanks for pointing that out. I missed it on my quick check.
Karellen 5y ago
Huh. While trying to figure out the reasoning behind the name used, TIL that "epic" has acquired a new meaning of "(computing) In software development, a large or extended user story." https://en.wiktionary.org/wiki/epic https://en.wiktionary.org/wiki/epic

nix23 5y ago

I was hoping that this has something todo that i CANNOT delete my UnrealEngine repo/project in gitlab...but no, well it's not my storage ;)

deleted 5y ago
[deleted]

sarki_247 5y ago

GitLab team member here, most of the affected relationships have been restored. If you are still affected, please let us know in the incident issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6072 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6...

m4tthumphrey 5y ago

Getting an eye into how the Gitlab team are working to resolve this is mind boggling (in a good way!) Just look at everything going on in this issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6072 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6...