deusu

8 ms·

31.
by deusu 11y ago
Non-ASCII characters are indeed messed up. I wrote the HTML parser about 15 years ago. Back then my programming style was so bad, that I'm afraid nothing short of a complete rewrite of the parser will be able to fix it. The Archive.org
32.
by deusu 11y ago
Did some baby steps in Modula-2 out of curiosity about 20 years back. The IDE and compiler were horrible compared to Turbo Pascal, so I went right back to that.
33.
by deusu 11y ago
Pascal - still use it today. I started with Turbo Pascal 3.0, went through several versions of Borland Pascal (DOS and Windows), several versions of Delphi, and am now using FreePascal. I do most of my work on Linux nowadays. Pascal still g
34.
by deusu 11y ago
I don't know what technology these blogs are using, but on the occasions where I looked at the HTML-source, the content was there. It just didn't show with Javascript disabled. Yes, you can use Javascript to make things prettier.
35.
by deusu 11y ago
I agree. But I would go even farther. Block every site where you won't be able to read the content without having Javascript enabled. I have experienced several blogs where you won't see anything without Javascript.
36.
Improving Search-Result Ranking on deusu.org(deusu.org)
5 points by deusu 12y ago|0 comments
37.
by deusu 12y ago
Yes. That's what I intend to do.
38.
by deusu 12y ago
The completed index takes up 337gb. The incoming data to the crawler... not so sure... something on the order of 10-20tb. I didn't really measure this. And I don't keep all the data. There is no "cache" function. On a 1g
39.
by deusu 12y ago
I usually start a crawl with a list of the top one million websites according to Alexa.com. Unfortunately that list doesn't seem to be available for download anymore, so I have to make due with the latest one I have which is from April
40.
by deusu 12y ago
Making the software scale to N servers is one of the reasons behind the planned rewrite. I'm already considering making "special" indexes for blogs, news etc. like you mentioned. In fact that same suggestion came up in a conv
41.
by deusu 12y ago
Many of the design choices that I made many, many years ago aren't valid anymore. So it won't just be a port to a different language, but a complete redesign and rewrite. Which means that regarding the amount of work, it doesn&#x2
42.
by deusu 12y ago
I do the crawling myself. There are currently about 320 million pages in the search-index.
43.
by deusu 12y ago
No pagerank. But I use backlink-counts and Alexa.com data. Also position of keywords in url, title and snippet. Length of url and number of elements in the url are also a part.
44.
by deusu 12y ago
And why should it? You don't need DeuSu to find itself. You are already on the site. :)
45.
by deusu 12y ago
I started writing the software a LONG time ago. As far as I know Solr/Elastic Search didn't exist back then. Query logs are preserved, but they do NOT contain the IP-adress of the user. So all I know is that someone of the billi
46.
by deusu 12y ago
The main crawl is several months old. There is a separate news-crawler which checks a few dozen news sites every 10 minutes or so.
47.
by deusu 12y ago
That is probably your browser keeping track of what you entered into input-fields that are named "q" for "query". And no, this data does NOT get transmitted. So DeuSu does not know about what you searched-for on Google.
48.
by deusu 12y ago
The rewrite in JavaScript will be mixture of porting and redesign. Some parts of the Pascal-sources are relatively new and can almost be translated as is. But the older the Pascal-source is, the more a redesign is a better idea. The old par
49.
by deusu 12y ago
I know of YaCy. It's written in Java which I'm almost completely unfamiliar with. But I know and follow the YaCy-community. Like myself the lead person of YaCy is located in Germany. We even share the same first name. :)
50.
by deusu 12y ago
Given that I'm in the process of rewriting it in JavaScript I'm not sure about that... On the other hand it's probably worth a try. Thanks for the suggestion.
51.
by deusu 12y ago
It's open-source: https://github.com/MichaelSchoebel/OpenAcoon This is currently written in Pascal. I'm in the process of rewriting it JavaScript/Node.js. First part for the rewrite is the ranking. That
52.
by deusu 12y ago
Please keep in mind that this is running on one server. Google has what? Several hundred thousand servers? Yes, it needs to get better. But it's a good start, I think.
53.
Show HN: Advertising-Free Search-Engine(deusu.org)
55 points by deusu 12y ago|59 comments