3 ms·

I like how frank the report is, no sugarcoating. "We relied on manual error prone verification and made a mistake. We have to automate the process." As ARIN bl

by progbits 10mo ago

I like how frank the report is, no sugarcoating. "We relied on manual error prone verification and made a mistake. We have to automate the process." As ARIN block owner this situation is kinda scary but reading this actually makes me think it's less likely to happen again .

_zagj 10mo ago
You don't find this part > We have to automate the process. to be ominous?
- Aurornis 10mo ago
  I don’t. The report says part of this process relied on flat files and spreadsheets. Automating that with software is a good idea. “Automate the process” doesn’t mean feeding everything to an LLM.
- aaomidi 10mo ago
  Certificate issuance was once only possible manually.
  - qingcharles 10mo ago
    Domains too, well into the 90s.
netfortius 10mo ago
The road to automation is always full of outages.
stefan_ 10mo ago
I'm curious how these fellas took something like IP block allocation and turned it into an Excel based workflow.
- jonathanlydall 10mo ago
  “Workflow” is probably a bit generous to describe how they probably use Excel. Having worked at a mom and pop ISP a couple of decades ago where we used Excel to track a lot of things, I can see how this might have happened. To actually know who is allocated what is ultimately just a list. And when there are only a few people who edit the list (and probably no more than 1 person at a time) you can get by with even a plain text file, but Excel is quite a bit nicer as you can do things like filtering and sorting easily, maybe even some formulas to help with things. Building a program backed by a database might be nice, but hard to justify when the manual system has never been a problem before. They’ve probably been thinking for a while they should, but it’s just never been enough of a pain point for them to invest the effort. Looks like they see this incident as justification that they need a system with hard coded rules and constraints, no more manual checking.
  - stefan_ 10mo ago
    It's ARIN, this is essentially their only job
- mmooss 10mo ago
  The world's financial systems run on Excel, to a great extent. I'm more surprised that a single person, apparently without seniority, could delete a block. IME deleting user data is usually a significant event; an IP block would especially be a big deal, especially for the IP block issuers. From the OP: > RSD has implemented additional process controls that require a dual review for all ticketing type workflows that include a network delete. > Only a limited set of experienced analysts are permitted to perform this function. Great that they didn't blame the person who deleted it. ARIN seems to have put them in position where a failure was likely, eventually. Without any inside knowledge, I'd hope the culture would have any engineer leary about pressing that button without a second set of eyes reviewing it carefully and without clear authorization; I don't imagine they delete many blocks each day so it shouldn't interfere with productivity.
- bigbuppo 10mo ago
  They've improved over the decades. At one point the authoritative database was a physical paper notebook.

_zagj 10mo ago

You don't find this part > We have to automate the process. to be ominous?

Aurornis 10mo ago
I don’t. The report says part of this process relied on flat files and spreadsheets. Automating that with software is a good idea. “Automate the process” doesn’t mean feeding everything to an LLM.
aaomidi 10mo ago
Certificate issuance was once only possible manually.
- qingcharles 10mo ago
  Domains too, well into the 90s.

netfortius 10mo ago

The road to automation is always full of outages.

stefan_ 10mo ago

I'm curious how these fellas took something like IP block allocation and turned it into an Excel based workflow.

jonathanlydall 10mo ago
“Workflow” is probably a bit generous to describe how they probably use Excel. Having worked at a mom and pop ISP a couple of decades ago where we used Excel to track a lot of things, I can see how this might have happened. To actually know who is allocated what is ultimately just a list. And when there are only a few people who edit the list (and probably no more than 1 person at a time) you can get by with even a plain text file, but Excel is quite a bit nicer as you can do things like filtering and sorting easily, maybe even some formulas to help with things. Building a program backed by a database might be nice, but hard to justify when the manual system has never been a problem before. They’ve probably been thinking for a while they should, but it’s just never been enough of a pain point for them to invest the effort. Looks like they see this incident as justification that they need a system with hard coded rules and constraints, no more manual checking.
- stefan_ 10mo ago
  It's ARIN, this is essentially their only job
mmooss 10mo ago
The world's financial systems run on Excel, to a great extent. I'm more surprised that a single person, apparently without seniority, could delete a block. IME deleting user data is usually a significant event; an IP block would especially be a big deal, especially for the IP block issuers. From the OP: > RSD has implemented additional process controls that require a dual review for all ticketing type workflows that include a network delete. > Only a limited set of experienced analysts are permitted to perform this function. Great that they didn't blame the person who deleted it. ARIN seems to have put them in position where a failure was likely, eventually. Without any inside knowledge, I'd hope the culture would have any engineer leary about pressing that button without a second set of eyes reviewing it carefully and without clear authorization; I don't imagine they delete many blocks each day so it shouldn't interfere with productivity.
bigbuppo 10mo ago
They've improved over the decades. At one point the authoritative database was a physical paper notebook.