5 ms·

ARIN Public Incident Report – 4.10 Misissuance Error

146 points by immibis 10mo ago

gbil 10mo ago
A couple of years ago ARIN increased their fees considerably - way higher than fees paid to RIPE for way less resources - and had a call with their management to express my frustration, not because I was paying from my pocket but because of the high discrepancy of the what they wanted to get and the quantity/quality of their services. Now I can see that their backbone services haven't really improved while their income for sure has. On a sidenote, what I appreciate in both RIPE and ARIN is that you can have at least a proper discussion when you have valid arguments with their support teams.
- rmoriz 10mo ago
  Now ARIN is much cheaper than RIPE for small entities.
  - rmoriz 10mo ago
    fee schedules FYI - ARIN 2026 PDF: https://www.arin.net/resources/fees/images/2026feeschedule.pdf https://www.arin.net/resources/fees/images/2026feeschedule.p... - RIPE 2026 : https://www.ripe.net/membership/payment/ https://www.ripe.net/membership/payment/ Enthusiasts, trainees and small orgs are paying a lot more with RIPE.
    - icedchai 10mo ago
      Not necessarily. Many have their RIPE registrations through an existing, “sponsoring” LIR. They’re not paying that 1800 Euro, the LIR is.
      - rmoriz 10mo ago
        A single AS resource and a single PI assignment cost more than the ARIN fee.
        icedchai 10mo ago
        Are you sure? For RIPE I see a 50 ASN plus 75 euro PI fee. ARIN is $275. Maybe I’m looking at it wrong. It’s cheaper as a hobbyist to use a RIPE LIR. Even in the US. That’s what I’ve been doing for years.
        rmoriz 10mo ago
        afaik that's +VAT and also for LIRs only. LIRs apply markup, see https://www.lir.services/lir-sponsoring/ https://www.lir.services/lir-sponsoring/ they charge 200€ per resource, so ASN + PI would be at last 400€/year that's way above the price of ARIN and you have a middleman. You must have a sponsoring LIR for your resources or become a LIR yourself. The only exception is LEGACY resources (IPv4, no ASN) but that's a different story.
        icedchai 10mo ago
        There are more competitive LIRs out there. Example: https://lagrange.cloud/products/lir https://lagrange.cloud/products/lir It’s also cheaper for me because I have legacy ARIN space. All I really needed was an ASN. The LIR gives me some PA v6 space for cheap, too.
        rmoriz 10mo ago
        Okay, but that is not enough to operate independently. PA v6 is another dependency. With ARIN you get your personal IPv6 assignment.
        icedchai 10mo ago
        For a hobbyist, the difference is academic. You can announce PA space with your own ASN, which is what I do. If I change LIRs I’ll have to renumber my IPv6 space.
        rmoriz 10mo ago
progbits 10mo ago
I like how frank the report is, no sugarcoating. "We relied on manual error prone verification and made a mistake. We have to automate the process." As ARIN block owner this situation is kinda scary but reading this actually makes me think it's less likely to happen again .
- _zagj 10mo ago
  You don't find this part > We have to automate the process. to be ominous?
  - Aurornis 10mo ago
    I don’t. The report says part of this process relied on flat files and spreadsheets. Automating that with software is a good idea. “Automate the process” doesn’t mean feeding everything to an LLM.
  - aaomidi 10mo ago
    Certificate issuance was once only possible manually.
    - qingcharles 10mo ago
      Domains too, well into the 90s.
- netfortius 10mo ago
  The road to automation is always full of outages.
- stefan_ 10mo ago
  I'm curious how these fellas took something like IP block allocation and turned it into an Excel based workflow.
  - jonathanlydall 10mo ago
    “Workflow” is probably a bit generous to describe how they probably use Excel. Having worked at a mom and pop ISP a couple of decades ago where we used Excel to track a lot of things, I can see how this might have happened. To actually know who is allocated what is ultimately just a list. And when there are only a few people who edit the list (and probably no more than 1 person at a time) you can get by with even a plain text file, but Excel is quite a bit nicer as you can do things like filtering and sorting easily, maybe even some formulas to help with things. Building a program backed by a database might be nice, but hard to justify when the manual system has never been a problem before. They’ve probably been thinking for a while they should, but it’s just never been enough of a pain point for them to invest the effort. Looks like they see this incident as justification that they need a system with hard coded rules and constraints, no more manual checking.
autoexec 10mo ago
I can't remember a screw up by ARIN this bad before. I'm not too concerned about it. I understand that mistakes can happen. That said, I'm a little surprised at how easy it was to make this one. I'm entirely unsurprised that this mistake involved an excel spreadsheet. Out of all the databases and IP management software they could be using which would have prevented this the first thing the employee reached for was excel. Almost every company I've worked for has employees using excel for data that would be better managed/stored/presented outside of an office document.
- patmorgan23 10mo ago
  From the nanog thread it seemed like the IP allocations for the IPv6 transition space (4.10) was the only space using this manual Excel process. That's probably how they initially started managing these allocations with the intention to build it into their automated systems but hadn't gotten around to it. And it sounds like they're prioritizing that work now, and have implemented an additional lay of checks in the mean time. This is a really big egg on face moment for ARIN, but it sounds like they are responding appropriately.
simonjgreen 10mo ago
All the RIRs are, in my experience, a very consistent and safe set of hands. This sort of things is vanishing rare to the point of borderline inconsequence by many providers of major internet infrastructure. The fact they care enough to take it seriously and publish shows how much they care about getting it right. I just completed a fairly major reorganisation of resources with RIPE, and I’ve interacted with them for two decades, and my experience is they remain as steady and consistent as ever. Sure, you may not like a particular policy at some moment, or may not agree with the charging structure at some point in time when it’s not advantageous to you, but they do at least do what they say and say what they do.
mlhpdx 10mo ago
So at least a good chunk of the Internet does indeed operate on a spreadsheet. Good to know.
- 12_throw_away 10mo ago
  All data begins life in a spreadsheet and dies in a spreadsheet. Automation is an illusion; databases are illusions. Only Excel is real.
  - ang_cire 10mo ago
    This reads like a joke, but I've known two DBAs who don't use database management tools beyond exporting whole tables to excel, making manual changes, and importing to update the tables. Scary stuff.
aftbit 10mo ago
I've considered setting up an ASN and grabbing an IPv6 block for myself for a while now, but have never had the gumption, time, and funds at the same time.
galaxygate 10mo ago
Affected customer here, if you're curious on our original NANOG post on the whole situation: Hey NANOG, After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely, and had been reallocated to another organization and announced from their network. I created a ticket in ARIN and called immediately. They confirmed that our subnet had been accidentally reallocated to another customer, and that they are currently working on returning it to us. After a couple hours, they told us the other organization will stop announcing the prefix, and WHOIS will be returned shortly. I’m guessing there’s no way to prevent this kind of thing on our side if the RPKI ROA itself is removed along with the allocation? I’m planning on adding checks to look for missing ROAs (in addition to invalid/expiring ones), which I'm guessing would've caught this earlier. Have any of you had anything like this happen with ARIN or another RIR? I’m especially curious what might have happened if we’d only noticed and reached out a few weeks later instead of within a few minutes.
- thaumaturgy 10mo ago
  [flagged]
  - AndroTux 10mo ago
    Maybe some college of theirs on HN recognized the story and shared it with them.
  - nateb2022 10mo ago
    > Unless you've got a regular HN account and just set up a new business-facing one for this? This is likely; I can't imagine a regular HN user would appreciate having their subnet publicly available in their comment history.
  - galaxygate 10mo ago
    Yup, another engineer that works on our team mentioned seeing the report here, I figured I'd make an account to add some further context
yoan9224 10mo ago
The transparency in this incident report is refreshing. "We relied on manual Excel-based verification and screwed up" - no corporate speak, just honest assessment. What's scary is that IPv4 allocations are literally internet infrastructure. Having your /24 suddenly reassigned to someone else could be catastrophic for a business. The fact that RPKI didn't catch this is interesting. The ROA was deleted along with the allocation, so from RPKI's perspective everything was valid. This is a good reminder that RPKI protects against hijacking but not against the RIR itself making mistakes. Glad they're automating this. Anything involving copy-pasting IP ranges in Excel is an accident waiting to happen.
squigz 10mo ago
This is a bit beyond my paygrade, but... this is as serious as it sounds, right? I'm just a bit surprised/confused by the response in these comments, especially compared to outages like when CF goes down. It's like that Gordon Ramsay meme. Is ARIN the 8 year old in this situation?

146 points by immibis 10mo ago

gbil 10mo ago

A couple of years ago ARIN increased their fees considerably - way higher than fees paid to RIPE for way less resources - and had a call with their management to express my frustration, not because I was paying from my pocket but because of the high discrepancy of the what they wanted to get and the quantity/quality of their services. Now I can see that their backbone services haven't really improved while their income for sure has. On a sidenote, what I appreciate in both RIPE and ARIN is that you can have at least a proper discussion when you have valid arguments with their support teams.

rmoriz 10mo ago
Now ARIN is much cheaper than RIPE for small entities.
- rmoriz 10mo ago
  fee schedules FYI - ARIN 2026 PDF: https://www.arin.net/resources/fees/images/2026feeschedule.pdf https://www.arin.net/resources/fees/images/2026feeschedule.p... - RIPE 2026 : https://www.ripe.net/membership/payment/ https://www.ripe.net/membership/payment/ Enthusiasts, trainees and small orgs are paying a lot more with RIPE.
  - icedchai 10mo ago
    Not necessarily. Many have their RIPE registrations through an existing, “sponsoring” LIR. They’re not paying that 1800 Euro, the LIR is.
    - rmoriz 10mo ago
      A single AS resource and a single PI assignment cost more than the ARIN fee.
      - icedchai 10mo ago
        Are you sure? For RIPE I see a 50 ASN plus 75 euro PI fee. ARIN is $275. Maybe I’m looking at it wrong. It’s cheaper as a hobbyist to use a RIPE LIR. Even in the US. That’s what I’ve been doing for years.
        rmoriz 10mo ago
        afaik that's +VAT and also for LIRs only. LIRs apply markup, see https://www.lir.services/lir-sponsoring/ https://www.lir.services/lir-sponsoring/ they charge 200€ per resource, so ASN + PI would be at last 400€/year that's way above the price of ARIN and you have a middleman. You must have a sponsoring LIR for your resources or become a LIR yourself. The only exception is LEGACY resources (IPv4, no ASN) but that's a different story.
        icedchai 10mo ago
        There are more competitive LIRs out there. Example: https://lagrange.cloud/products/lir https://lagrange.cloud/products/lir It’s also cheaper for me because I have legacy ARIN space. All I really needed was an ASN. The LIR gives me some PA v6 space for cheap, too.
        rmoriz 10mo ago
        Okay, but that is not enough to operate independently. PA v6 is another dependency. With ARIN you get your personal IPv6 assignment.
        icedchai 10mo ago
        For a hobbyist, the difference is academic. You can announce PA space with your own ASN, which is what I do. If I change LIRs I’ll have to renumber my IPv6 space.
        rmoriz 10mo ago

progbits 10mo ago

I like how frank the report is, no sugarcoating. "We relied on manual error prone verification and made a mistake. We have to automate the process." As ARIN block owner this situation is kinda scary but reading this actually makes me think it's less likely to happen again .

_zagj 10mo ago
You don't find this part > We have to automate the process. to be ominous?
- Aurornis 10mo ago
  I don’t. The report says part of this process relied on flat files and spreadsheets. Automating that with software is a good idea. “Automate the process” doesn’t mean feeding everything to an LLM.
- aaomidi 10mo ago
  Certificate issuance was once only possible manually.
  - qingcharles 10mo ago
    Domains too, well into the 90s.
netfortius 10mo ago
The road to automation is always full of outages.
stefan_ 10mo ago
I'm curious how these fellas took something like IP block allocation and turned it into an Excel based workflow.
- jonathanlydall 10mo ago
  “Workflow” is probably a bit generous to describe how they probably use Excel. Having worked at a mom and pop ISP a couple of decades ago where we used Excel to track a lot of things, I can see how this might have happened. To actually know who is allocated what is ultimately just a list. And when there are only a few people who edit the list (and probably no more than 1 person at a time) you can get by with even a plain text file, but Excel is quite a bit nicer as you can do things like filtering and sorting easily, maybe even some formulas to help with things. Building a program backed by a database might be nice, but hard to justify when the manual system has never been a problem before. They’ve probably been thinking for a while they should, but it’s just never been enough of a pain point for them to invest the effort. Looks like they see this incident as justification that they need a system with hard coded rules and constraints, no more manual checking.

autoexec 10mo ago

I can't remember a screw up by ARIN this bad before. I'm not too concerned about it. I understand that mistakes can happen. That said, I'm a little surprised at how easy it was to make this one. I'm entirely unsurprised that this mistake involved an excel spreadsheet. Out of all the databases and IP management software they could be using which would have prevented this the first thing the employee reached for was excel. Almost every company I've worked for has employees using excel for data that would be better managed/stored/presented outside of an office document.

patmorgan23 10mo ago
From the nanog thread it seemed like the IP allocations for the IPv6 transition space (4.10) was the only space using this manual Excel process. That's probably how they initially started managing these allocations with the intention to build it into their automated systems but hadn't gotten around to it. And it sounds like they're prioritizing that work now, and have implemented an additional lay of checks in the mean time. This is a really big egg on face moment for ARIN, but it sounds like they are responding appropriately.

simonjgreen 10mo ago

All the RIRs are, in my experience, a very consistent and safe set of hands. This sort of things is vanishing rare to the point of borderline inconsequence by many providers of major internet infrastructure. The fact they care enough to take it seriously and publish shows how much they care about getting it right. I just completed a fairly major reorganisation of resources with RIPE, and I’ve interacted with them for two decades, and my experience is they remain as steady and consistent as ever. Sure, you may not like a particular policy at some moment, or may not agree with the charging structure at some point in time when it’s not advantageous to you, but they do at least do what they say and say what they do.

mlhpdx 10mo ago

So at least a good chunk of the Internet does indeed operate on a spreadsheet. Good to know.

12_throw_away 10mo ago
All data begins life in a spreadsheet and dies in a spreadsheet. Automation is an illusion; databases are illusions. Only Excel is real.
- ang_cire 10mo ago
  This reads like a joke, but I've known two DBAs who don't use database management tools beyond exporting whole tables to excel, making manual changes, and importing to update the tables. Scary stuff.

aftbit 10mo ago

I've considered setting up an ASN and grabbing an IPv6 block for myself for a while now, but have never had the gumption, time, and funds at the same time.

galaxygate 10mo ago

Affected customer here, if you're curious on our original NANOG post on the whole situation: Hey NANOG, After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely, and had been reallocated to another organization and announced from their network. I created a ticket in ARIN and called immediately. They confirmed that our subnet had been accidentally reallocated to another customer, and that they are currently working on returning it to us. After a couple hours, they told us the other organization will stop announcing the prefix, and WHOIS will be returned shortly. I’m guessing there’s no way to prevent this kind of thing on our side if the RPKI ROA itself is removed along with the allocation? I’m planning on adding checks to look for missing ROAs (in addition to invalid/expiring ones), which I'm guessing would've caught this earlier. Have any of you had anything like this happen with ARIN or another RIR? I’m especially curious what might have happened if we’d only noticed and reached out a few weeks later instead of within a few minutes.

thaumaturgy 10mo ago
[flagged]
- AndroTux 10mo ago
  Maybe some college of theirs on HN recognized the story and shared it with them.
- nateb2022 10mo ago
  > Unless you've got a regular HN account and just set up a new business-facing one for this? This is likely; I can't imagine a regular HN user would appreciate having their subnet publicly available in their comment history.
- galaxygate 10mo ago
  Yup, another engineer that works on our team mentioned seeing the report here, I figured I'd make an account to add some further context

yoan9224 10mo ago

The transparency in this incident report is refreshing. "We relied on manual Excel-based verification and screwed up" - no corporate speak, just honest assessment. What's scary is that IPv4 allocations are literally internet infrastructure. Having your /24 suddenly reassigned to someone else could be catastrophic for a business. The fact that RPKI didn't catch this is interesting. The ROA was deleted along with the allocation, so from RPKI's perspective everything was valid. This is a good reminder that RPKI protects against hijacking but not against the RIR itself making mistakes. Glad they're automating this. Anything involving copy-pasting IP ranges in Excel is an accident waiting to happen.

squigz 10mo ago

This is a bit beyond my paygrade, but... this is as serious as it sounds, right? I'm just a bit surprised/confused by the response in these comments, especially compared to outages like when CF goes down. It's like that Gordon Ramsay meme. Is ARIN the 8 year old in this situation?