6 ms·

For anyone who is interested, after about [edited] 40k requests, this has cost me $10 in OpenAI charges and another estimated ~$100 in domain verification.

digitalsin 4y ago
To be clear, it was 40k requests and not $40k in requests?
- kirillzubovsky 4y ago
  oh yeah, I hope...!
janalsncm 4y ago
Another option might be to download the entire .com registry database from ICANN and query it live. I say “might” because I am not sure if your use case would be permitted under their TOS.
- kirillzubovsky 4y ago
  Yup, someone's suggested that. I might do it going forward, if it's allowed, but for the first day I tried to prioritize getting it out there. Also, I have another project for which I store a huge amount of data, and as it turns out, data costs a lot of money to store! Depends on the size of the list, I might be better off paying for the Domainr API.
  - samwillis 4y ago
    According to a quick google search there are about 367million domains names registered. The max length is 253. So: 367000000 * 253 bytes = 92.8GB As you only need to know the domain exists. In reality it would be smaller than that as the average length will be much shorted. I expect 10-20GB You could get clever and use a bloom filter to save space, falling back to the api.
    - kirillzubovsky 4y ago
      I will try it in the next few days and report back. Most likely a combination approach would be the best - check against the known data, then double-check with API on case-by-case basis. I might require login to get extra data, so it would be just a subset of users doing a deep-dive. Thank you!
      - _pn3l 4y ago
        I’ve downloaded a copy in the past. It’s a large zone file full of NS records (obviously?) and compressible to a high degree. https://czds.icann.org/home https://czds.icann.org/home
    - askiiart 4y ago
      Also, this is only .com domains being suggested, right? So you wouldn't need the entire dataset, just .com domains.
    - Atlas22 4y ago
      Just checked, latest zone file for .com is 4.9GB compressed, .net is only 456 MB compressed (2.2GB decompressed). Not sure how large .com is decompressed as CZDS is giving me download issues for .com today but some rough extrapolation from .net would land around 24GB.
mike_d 4y ago
Shoot me an email (in my profile) and I'll get you a free API that will check domain availability for you.
cobertos 4y ago
FYI, it seems to have generated a domain with an apostrophe in it. Screenshot, https://i.imgur.com/eGIOu0t.png https://i.imgur.com/eGIOu0t.png
mikkom 4y ago
I created a similar service (migtyname.com and separate personal toolkit) in the past for checking name combinations and checked the domain name availability using custom code that costs nothing. If this saves you money then I'm happy. so drumroll..... here you are (whois.py is my project name for the file) import socket tld = 'whois.verisign-grs.com' def whois(domain): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect((tld, 43)) str = domain+'\n' s.send(str.encode()) buffer = '' while True: data = s.recv(1024).decode() if data: buffer += data else: break return buffer def available(domain): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect((tld, 43)) str = domain+'\n' s.send(str.encode()) buffer = '' data = s.recv(1024).decode() if "No match for" in data: return True return False
- kirillzubovsky 4y ago
  Thanks, I will give it a try. I am going to build a cascading approach where I will use inhouse tools to check first, and then when necessary or unclear, you other tools to verify. This would come in handy!
- swyx 4y ago
  dont you need to set a timeout on this in case it doesnt connect in tim?
danskeren 4y ago
> after about 40k requests, this has cost me $10 in OpenAI charges That's amazing. Hope they can continue to keep the pricing that low.
lamsal 4y ago
Hi OP, I liked your project! Doing something similar and I'm wondering what engines you are using? Because mine is already charged 0.18$ for 200 requests. Here is my current setup: "max_tokens" => 150, "temperature" => 0.6, "top_p" => 1, "frequency_penalty" => 1, "presence_penalty" => 1, model is text-davinci-003. Can you let me know how you are only charged $10 for 40K requests?

digitalsin 4y ago

To be clear, it was 40k requests and not $40k in requests?

kirillzubovsky 4y ago
oh yeah, I hope...!

janalsncm 4y ago

Another option might be to download the entire .com registry database from ICANN and query it live. I say “might” because I am not sure if your use case would be permitted under their TOS.

kirillzubovsky 4y ago
Yup, someone's suggested that. I might do it going forward, if it's allowed, but for the first day I tried to prioritize getting it out there. Also, I have another project for which I store a huge amount of data, and as it turns out, data costs a lot of money to store! Depends on the size of the list, I might be better off paying for the Domainr API.
- samwillis 4y ago
  According to a quick google search there are about 367million domains names registered. The max length is 253. So: 367000000 * 253 bytes = 92.8GB As you only need to know the domain exists. In reality it would be smaller than that as the average length will be much shorted. I expect 10-20GB You could get clever and use a bloom filter to save space, falling back to the api.
  - kirillzubovsky 4y ago
    I will try it in the next few days and report back. Most likely a combination approach would be the best - check against the known data, then double-check with API on case-by-case basis. I might require login to get extra data, so it would be just a subset of users doing a deep-dive. Thank you!
    - _pn3l 4y ago
      I’ve downloaded a copy in the past. It’s a large zone file full of NS records (obviously?) and compressible to a high degree. https://czds.icann.org/home https://czds.icann.org/home
  - askiiart 4y ago
    Also, this is only .com domains being suggested, right? So you wouldn't need the entire dataset, just .com domains.
  - Atlas22 4y ago
    Just checked, latest zone file for .com is 4.9GB compressed, .net is only 456 MB compressed (2.2GB decompressed). Not sure how large .com is decompressed as CZDS is giving me download issues for .com today but some rough extrapolation from .net would land around 24GB.

mike_d 4y ago

Shoot me an email (in my profile) and I'll get you a free API that will check domain availability for you.

cobertos 4y ago

FYI, it seems to have generated a domain with an apostrophe in it. Screenshot, https://i.imgur.com/eGIOu0t.png https://i.imgur.com/eGIOu0t.png

mikkom 4y ago

I created a similar service (migtyname.com and separate personal toolkit) in the past for checking name combinations and checked the domain name availability using custom code that costs nothing. If this saves you money then I'm happy. so drumroll..... here you are (whois.py is my project name for the file) import socket tld = 'whois.verisign-grs.com' def whois(domain): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect((tld, 43)) str = domain+'\n' s.send(str.encode()) buffer = '' while True: data = s.recv(1024).decode() if data: buffer += data else: break return buffer def available(domain): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect((tld, 43)) str = domain+'\n' s.send(str.encode()) buffer = '' data = s.recv(1024).decode() if "No match for" in data: return True return False

kirillzubovsky 4y ago
Thanks, I will give it a try. I am going to build a cascading approach where I will use inhouse tools to check first, and then when necessary or unclear, you other tools to verify. This would come in handy!
swyx 4y ago
dont you need to set a timeout on this in case it doesnt connect in tim?

danskeren 4y ago

> after about 40k requests, this has cost me $10 in OpenAI charges That's amazing. Hope they can continue to keep the pricing that low.

lamsal 4y ago

Hi OP, I liked your project! Doing something similar and I'm wondering what engines you are using? Because mine is already charged 0.18$ for 200 requests. Here is my current setup: "max_tokens" => 150, "temperature" => 0.6, "top_p" => 1, "frequency_penalty" => 1, "presence_penalty" => 1, model is text-davinci-003. Can you let me know how you are only charged $10 for 40K requests?