3 ms·
I built Sitetruth.com to try to solve that problem. I'm going to shut it down soon. The goal of SiteTruth was to try to find the real-world business behind a w
by Animats 5y ago
I built Sitetruth.com to try to solve that problem. I'm going to shut it down soon.
The goal of SiteTruth was to try to find the real-world business behind a web site, and look up information about the business, such as how long it has been in business and its annual revenue. That's become harder and harder over the last decade.
First, it's now acceptable to have an online business with no real-world address and no visible legal existence. This is illegal in the European Union and illegal in California if the business accepts payments, but enforcement is nonexistent.
Second, more sites have become inaccessible to scraping by servers. I have a system which looks for a human-readable business address on a site. It looks in the obvious places (front page, "about", "legal", "terms", "contact", etc.) and quits after trying the 20 most likely pages. It uses a honest agent ID ("Sitetruth.com site verification system", registered with the now meaningless "bots vs browsers" list) and obeys robots.txt. A sizeable fraction of the time, it can't read the site at all.
Third, the data sources for company information have been becoming less accessible. There used to be two reliable data sources: Hoovers, and Dun and Bradstreet. They merged. Dun and Bradstreet for a while became rather corrupt. They licensed a company in Santa Monica, CA to use their name, and sent the small-business part of the business to them. This unit's marketing approach was "Nice credit rating. Be a shame if it something happened to it". After much litigation, DnB HQ bought the Santa Monica company, but the reputational damage was done and DnB is no longer the gold standard of company information. There are lower tier data sources (look up "US Business List"), but the data quality is poor. Anything based on user recommendations, like Yelp, gets spammed, so that's out. Yahoo Directory, which was reasonably spam free, is gone.
Fourth, the SSL cert industry became corrupt. OV standards were never very high, and EV standards started slipping. Then there was the Cloudflare problem. Cloudflare is a certificate authority, and they issue certs to themself for domains which run through Cloudflare. So looking up a cert just gets Cloudflare's info.
Fifth, Google is making it harder and harder to have Chrome plug-ins that critique their ads. I dropped Chrome support recently, and only have a Firefox add-on at this point.
So, after fifteen years, Sitetruth is coming to an end.
- ttyp3 5y agoYes, it's difficult to even determine in which _state_ a (US) business is operating now.
- Animats 5y agoIn NFT land, "which country" is seldom specified.
- crooked-v 5y ago> Fourth, the SSL cert industry became corrupt. OV standards were never very high, and EV standards started slipping. Then there was the Cloudflare problem. Cloudflare is a certificate authority, and they issue certs to themself for domains which run through Cloudflare. So looking up a cert just gets Cloudflare's info. Tying SSL certs to identity (rather than just to verification that someone has the authority over a given domain) was a dumb idea in the first place, so I hardly see a problem with workarounds in that regard.
- bryanrasmussen 5y agowell not for the problem of security, but for their problem of trying to identify who is actually behind a site it of course is a problem that the ties to identity became weaker. on edit: yes, they want to verify identity because their purpose is really to provide a form of security as well, so it is sort of removed by a step / nothing works all the way down, getting confusing at this point.
- Animats 5y agoMy metric is, "if you needed to sue the website operator, could you find them?"
- Melatonic 5y agoSounds like a problem of policy really at the core - maybe when we get some more nerds in Congress this will be finally addressed? Seems like this could just be a government run database (assuming these businesses are paying taxes and not completely shady) with some of it publicly accessible. Have you tried also looking at US shipment logs? I know there is that guy who a few years ago made a site that scraped and indexed all of the public shipping logs (anything crossing US borders by land or sea goes into huge public databases - for some reason air is excluded). Could be a way to cross reference this data to fill in the blanks for some of the things you are talking about?