3 ms·
Part II For now, sure, I know some math and can stir up some more, and I know some computing and can learn more. So, I've got a 1.8 GHz AMD single core proces
by graycat 12y ago
Part II
For now, sure, I know some math and can
stir up some more, and I know some
computing and can learn more.
So, I've got a 1.8 GHz AMD single core
processor, Windows XP Professional SP3 (I
have an official copy of Windows 7
Professional on DVD but have seen no great
reason to go to all the trouble to install
it and rebuild all my software
environment yet), three hard disks, 100
million files, .NET Framework 4.0, a good
text editor (KEdit), Visual Basic .NET
(comes with the .NET Framework), ASP.NET,
ADO.NET, IIS (low level Web server), SQL
Server Express, etc.
I thought, why not go after about 2/3rds
of Internet search, the safe for work
part served at best poorly by looking for
keywords/phrases?
So, how to do that? Not with just routine
software! And not with anything commonly
talked about for search!
I stirred up some math, typed the theorems
and proofs into Knuth's TeX, worked up a
scalable architecture, and typed in the
software.
Maybe I should have used Redis, but
instead I thought that writing my own
session state server would be faster than
even understanding Redis. So, my session
state server is single threaded ("Look,
Ma, no concurrency problems"), for faster
lookup rates is trivial to run as several
instances as in sharding, and is just
some TCP/IP sockets, some de/serialization
of instances of my session state class,
and uses two instances of a .NET
collection class. Simple.
It's fast! A server for less than $1500
should be able to keep session state for
an hour of inactivity for each user and do
the session state work for sending 5000+
Web pages a second. Two standard racks
should be able to handle session state for
the world.
The rest of the software is also readily
scalable also from just simple sharding.
Currently I have one bug in one Web page
-- it's not handling session state just
right! But I have the fix in code in
another Web page and should copy it over
today!
My interactive development environment
is just KEdit with about 100 macros and
some careful use of file system
directories.
I'm using SQL Server only to record the
data from the users and for the results of
some batch computations; at one point I
actually do make use of a transaction;
the data for the searches is drawn from
SQL Server with a batch program (run it
maybe once a day); some solid state drives
(write rarely, read thousands of times a
second) should do wonders for the data for
the searches.
I was about to fix the bug in the Web page
but took some opportunities to gather some
good initial data. The site will start
focused and only slowly grow to be
comprehensive -- right, at first do some
things that don't scale and please some
niche group of users a lot instead of
trying to please 2+ billion users a
little.
Currently the database has only some
meaningless data I put in for first
testing of the software. It's about time
to load in some of the good initial data I
have. Then give a critical review, go
live, etc.
It's getting there.
All the work uniquely mine has been fast,
fun, and easy, but the whole project has
taken far, far, far too long. Why? I
worked through about a cubic foot of books
and 6000+ Web pages of documentation of
Windows, .NET, Visual Basic .NET, ASP.NET,
ADO.NET, SQL Server, etc.
The main problems: (1) Badly written,
obscure documentation (worst bottleneck in
the future of computing); (2) computer
viruses; (3) SQL Server installation bugs
destroying my boot partition requiring
rebuilding starting with the XP DVD
(barbed wire enema with an unanesthetized
upper molar root canal procedure); (4) SQL
Server management and administration
(e.g., a week of throwing stuff against a
wall to see what sticks just to get a SQL
Server connection string that will let
code for a server side Web page connect
with SQL Server); (5) clean, smooth means
of system backup and recovery (including
for both user data and bootable
partitions); (6) Sony DVD drives that quit
for no good reason (and inability to buy
more IDE DVD drives).
Good stuff: (1) KEdit and its macro
language; (2) the scripting language Rexx
(Microsoft's PowerShell may also be
terrific but have yet to move to it); (3)
NTFS (fantastic); (4) Visual Basic .NET
design, functionality, speed of
compilation, compiler error messages,
minimal bugs (sweetheart language); (5)
what ASP.NET does when it compiles a
Visual Basic program (enough for a really
nice IDE); (6) NTBACKUP (once understand
how to use it, e.g., do have to ask to
save "system state", whatever the heck
that is, or the saved copy, restored,
won't boot -- learn this and how to get
around it the hard way, weeks of work);
(7) XCOPY; (8) the tools to have server
side Web page code write to a log file;
(9) Firefox (except for virus
vulnerabilities); (10) the classes in the
.NET Framework (once learn how to learn
about them and use them); (11) Adobe
Acrobat (except for virus
vulnerabilities); (12) the ability of XP
to find device drivers and recognize new
devices; (13) Microsoft's anti-virus
Safety Scanner (if only from CP67/CMS and
Multics, there should be no virus
vulnerabilities, but since there are the
safety scanner is terrific to have); (14)
Knuth's TeX; (15) the Western Digital
Passport Ultra 2 TB USB drive!
I'm not retired or retiring! Likely I'm
still 100% unemployable at anything that
would pay enough to let me keep a car
going to commute to the job. Y Combinator
and VCs want nothing to do with me.
But if I can get my Web site up to a
search a second, on a server for about
$2000, I will be in decent shape
financially and on the way to organic
growth for my business and much more.
Then I'll get a nice house, a building for
some cars, at least one Corvette, visit
the rest of the family still alive, take
off two weeks to pig out on lobster in
Maine, get some good grape juice from
between Beaune and Dijon, do some cooking,
give some dinner parties, go to concerts
and operas, continue with the business, go
to seminars on mathematical physics, etc.
I'll implement and deploy my server farm
monitoring techniques and maybe spin it
off as a separate business.
I have some guesses for some approaches to
real AI and might try to implement
those.
And I will get a kitty cat! We'll see!
Good luck on your work with math. Maybe
what I typed in here will help; I wish I'd
known all that when I started.
- lightcatcher 12y agoWriting this as a reply rather than an email because there's no contact info on graycat's profile. Hi graycat, I'm an applied math & CS undergrad at Caltech and I love all of your posts, especially the ones about how data center scale computing really needs to embrace statistics. The FedEx stories are also excellent. I'm personally interested in high performance computing and machine learning, but I'm also interested in solving "real problems" and like how you seem to focus on the actual value of the applications. I love the feeling of engineering solutions to mathematical problems, and this seems to be something that you also enjoy. I'd really like to hear more about your career, research, and also the startup that you're working on. I'd be very happy if you shoot me an email at eric@ericmart.in Best, Eric
- somberi 12y agoDear Graycat, I really enjoy your posts and your writing style. Thanks much. I am currently working on some math problems as a part of a business that makes accessible to Indians who do not have access to structured banking services. Would love to discuss it with you, if you are inclined. My email is takenottie at google's mail. Thanks.