12 ms·

a16z Programming Test

92 points by ghosh 13y ago

nilsbunger 13y ago
Cool, but... Why is A16z building an engineering team?
- grosen 13y ago
  Probably want to internally build some products. Makes for much better returns when you own 50+% of a company vs. ~20% via traditional funding.
- Oculus 13y ago
  My guess is to have an in-house team that can help out the companies they fund. I know that Netflix's cloud architect, Adrian Cockcroft, left to join a venture capital firm to advise on scaling issues for companies that firm funded[0]. 0 - http://gigaom.com/2014/01/07/netflixs-cloud-architect-adrian-cockcroft-is-leaving-to-join-battery-ventures/ http://gigaom.com/2014/01/07/netflixs-cloud-architect-adrian...
  - pmarca 13y ago
    "My guess is to have an in-house team that can help out the companies they fund." Nope.
    - seizethecheese 13y ago
      I'm really curious... a hint?
  - rythie 13y ago
    It's a big assumption though, and you need to spend 3 hours to find out what you are actually applying for.
    - fsk 13y ago
      You spend 3 hours to MAYBE find out what you're applying for. If you fail, presumably that's the end of the process.
- cnaut 13y ago
  Probably to build internal tools. A lot of VCs firms are trying to apply more data to their investment decisions and that requires internal tools to handle it all. Also makes sense given the qualifications they are asking for.
  - graycat 13y ago
    A better way: Actually read and understand the submissions from the entrepreneurs!
    - ironchef 13y ago
      Are you suggesting that a16z doesn't do this? Or that their firm is not doing well? It would seem to me that their limited track record is ... more than adequate. Similarly, I would infer that your suggestion is ignorant that they have interest in the companies post-funding (as cnaut suggests "A lot of VCs firms are trying to apply more data to their investment decisions and that requires internal tools to handle it all"... which I think would suggest that they would want to examine the results of said decisions as well (which goes beyond simply the submission process)).
      - graycat 13y ago
        > Are you suggesting that a16z doesn't do this? Yes I am. (1) It appears, really is quite clear, that they have high contempt for initial contacts from entrepreneurs via e-mail. Instead they want to believe that the 'good' entrepreneurs will have enough 'exposure', enough in 'networking', and enough in 'hustle', e.g., on LinkedIn, GetHub, etc., to come to the attention of A16Z by means other than an e-mail message "over the transom". E.g., they want an entrepreneur to get introductions from bankers, lawyers, accountants, or CEO's of companies in the A16Z portfolio. So, A16Z (A) wants others to 'filter' their entrepreneurs for them and/or (B) believes that lawyers, etc. are good sources of good recommendations. Nonsense. (2) A16Z will just refuse to consider any very technical support for a project. E.g., if a project has some powerful, valuable 'secret sauce', intellectual property, and high technological barrier to entry, A16Z can easily just dismiss the technical issues, the project, and the entrepreneur. A16Z wants to believe that they have 'deep domain knowledge' where they invest and, thus, wants to believe that anything new that A16Z has not heard of must be nonsense. If some supporting technical material is not easy reading for them, then they believe that they are justified in just regarding it as nonsense. But, looking at the partners of A16Z, it's tough to find one with enough technical qualifications and/or accomplishments to (A) get a tenure track position in the engineering school of a leading research university, (B) be an editor of a leading technical journal of original research, (C) publish a peer-reviewed paper of original research in a technical field in a good or better journal of original research, or (D) get an NSF grant for original research. Net, A16Z apparently is not competent to review or even to direct a competent review of technical material in information technology. What the A16Z financial performance really is is likely not easy to discover. You seem to suggest that the project for which the OP is some recruiting is for an internal information system to do causal modeling and make valuable predictions for venture investing. It would appear that they want to do cross tabulation and curve fitting; sorry, there is nearly no chance those techniques will work. Why? In simple terms, they are looking for new, exceptional cases, and looking at 'averages' from the past, even of successes in the past, is just intellectual self-abuse. If they are making money, then, fine. But from all I can see, in information technology their competence is way below a level that would be useful for actual technical work or doing a technical review of such work. I've been around technical work in US national security, US business and start ups, including one of the most successful venture funded companies ever, and high end academic research, hold a Ph.D. in engineering from one of the world's best research universities, have published peer-reviewed original research in computer science, applied mathematics, and artificial intelligence, and in comparison the technical qualifications of A16Z look like a bad joke. Yes, most good quantitative analysis needs some computing and programming. So, in their recruiting A16Z seems to believe that the computing, which is necessary, is also sufficient. It's not. By an analogy, A16Z is looking for nurse practitioners instead of MD physicians. Both are necessary, but the nurse practitioners are not sufficient. There is a difference! But, why help A16Z? They have money enough now!
- gautamsivakumar 13y ago
  They have $4.2bn under management, and a very smart team. I am sure there is a lot that can be done...
- neil_s 13y ago
  My guess would be a team to provide a data-centric approach to investment. Crunch a whole lot of numbers to identify industry trends and specific firms that are promising, and see if those can replicate human performance. Or they're simply handling recruitment for a stealth portfolio startup.
  - graycat 13y ago
    > industry trends Why? The VC firms never invest in an "industry", i.e., there are no such 'index funds'. Instead, VC firms invest in individual companies, and there what is just crucial is what the company does. If the company looks good, then mostly to heck with the darned industry. > Or they're simply handling recruitment for a stealth portfolio startup. In this case, poor startup: A16Z is doing a "piss poor" job 'helping' the startup!
tpae 13y ago
I seriously hope this doesn't become a norm. I hate timed tests.
- rdl 13y ago
  Seems like the obvious (if dishonest) thing would be to use a throwaway phone # to get the test, then spend a day or so solving it, then type it in (with plausible delays) using your real number a day later. Assuming the test is roughly the same each time (even if details are different, it's probably of the same general class). In grading, the results of 6-8h of work entered in 2h would look more impressive than 3h of work in 3h, I suspect. (Sometimes I hate that whenever someone gives me a situation, my first thought is "how can this be cheated or gamed", followed by "what controls can I put in place to prevent that, and and what costs"; it's basically automatic.)
  - rmmw 13y ago
    Credentials through phone # is more of a deterrent. We're also going to have an in-person coding exercise as a next step, which would be much harder to game :)
    - deleted 13y ago
      [deleted]
    - flylib 13y ago
      are you saying you go straight to a in-person coding exercise as the next step? seems odd, there is no phone interview between then?
  - golergka 13y ago
    Why would you want to appear better then you are at the interview? You'll get found out eventually later, and this will end up just spending your time.
    - sireat 13y ago
      There are plenty of developers/programmers who do not ace tests with rigid constraints, but are quite competent in a more relaxed setting. That is they are not the 1% that the all hirers, but they are maybe top 20% and that is actually what most jobs require no matter the delusions of hiring managers. Granted this particular job might be an exception but I doubt it.
elnate 13y ago
Doesn't accept my (NZ) mobile number as valid :(
- balajis 13y ago
  Did you include the country code prefix? Otherwise please try emailing vivek at hackerrank dot com with the bug to see if he can reproduce.
- rmmw 13y ago
  You can also email test-help AT a16z DOT com and we can send you a direct link (where you won't need to receive credentials by phone)
  - pjreddie 13y ago
    I love that people do this. Do you really thing someone is going to write a web scraper and NOT include obvious things like converting AT -> @ and DOT com -> .com? Also, who even worries about spam any more? I haven't seen actual spam in my inbox in years.
    - cflee 13y ago
      Should we just go "AT AT AT" or something instead? Or perhaps adapt from Mandarin speakers, who literally call the @ sign "the little mouse" ;) Spam remains an issue if for whatever reason you're still using rulesets/blacklists/greylisting and the like, instead of Bayesian filtering.
Oculus 13y ago
1:30 am on the night before a physics test - why must this challenge be so tempting! There goes that studying :)
abalone 13y ago
I wonder if this could act as a negative filter for good programmers. Who worth their salt is going to invest 3 hours of their time just to submit a resume? Wouldn't you want to talk to the employer first to see if they're a good fit? In a tight labor market, people willing to do this are more desperate than others. So it's going to filter for desperate developers who passed a 3 hour basic skills test. Old fashioned prior experience and education is probably still a better first filter. This would probably make more sense specifically for junior hires right out of school or with no experience.
- balajis 13y ago
  Well, empirically we're getting quite a lot of people attempting it! :) Three hours is pretty lenient (you could blaze through it in less time), though the last question is challenging. We think the questions are kind of fun as well. Without something like this we'd just have 100s-1000s of unstructured resumes in different formats to rank against each other. This process is more fair and (IMO) allows someone without a college degree, or from outside the US, to compete on a level playing field.
  - jay_kyburz 13y ago
    Do you mind if people do the test with no interest in the job?
    - balajis 13y ago
      Sure, go ahead! We just ask that you not share the questions with others after completing it.
  - 7Figures2Commas 13y ago
    > Well, empirically we're getting quite a lot of people attempting it! > Without something like this we'd just have 100s-1000s of unstructured resumes in different formats to rank against each other. So instead of hundreds or thousands of unstructured resumes in different formats, you have hundreds or thousands of completed programming tests. Even assuming that you automatically filter out submissions that fail testing, given that there are a lot of ways to solve a problem, you'd still be stuck evaluating potentially hundreds of correct submissions. More importantly, a test doesn't seem to be aligned with your top priority: > ...but the most important requirement is significant independent programming experience as demonstrated by your GitHub account, personal projects, academic publications, or startup success. Are you assuming that all of the folks most accomplished in the real world are going to complete your programming test successfully? Since you ultimately seem to be interested in folks who can build dashboards, I would think you'd be far better off asking candidates to build a dashboard prototype using sample data you provide. There are lot of really smart developers out there with advanced degrees who could solve programming quizzes all day long but couldn't design and implement a useful dashboard application.
deleted 13y ago
[deleted]
peapod91 13y ago
Will this be an ongoing opportunity or is there a deadline to submit this by?
- balajis 13y ago
  It's an ongoing opportunity, but we will be hiring the first set of people soon. So it will be to your advantage to apply sooner rather than later.
_pmf_ 13y ago
> machine learning (scatterplots & histograms, logistic/linear regression, R/Matlab) So, someone taking Statistics 101 is considered to know machine learning in the startup circus? (Not that this would really surprise me, given the tendency to appropriate terms that had a certain meaning for purposes that have only a passing similarity to the original meaning.)
- balajis 13y ago
  We could certainly rattle off more terms :) Kernel density estimation? Deep learning? Semi-supervised learning? But it's amazing how far you can get with some simple scatterplots, histograms and linear/logistic regressions. a) Most of the time, more data beats better algorithms: http://anand.typepad.com/datawocky/2008/03/more-data-usual.html http://anand.typepad.com/datawocky/2008/03/more-data-usual.h... b) Even those seemingly simple things get complicated when you have a lot of data: http://komarix.org/ac/lr http://komarix.org/ac/lr
buro9 13y ago
Does this mean Mattermark https://mattermark.com/ https://mattermark.com/ have started a tech arms war amongst the larger VC firms to create their own toolsets to differentiate their deal flow from others? Mattermark has gone some way to provide proof that data can make a difference, at least at highlighting opportunities that might not have been visible before. I wondered whether this would result in herding investors towards some companies, which means differentation and opportunity comes from spotting things the others miss... finding the outliers that Mattermark miss. Interesting that the test is "make a dashboard", well into the standard skillset for most analyst programmers who work for financial institutions (I built a lot of this stuff for the finance sector in London).
dang 13y ago
HN policy for years has been to allow two kinds of job ad: (1) job posts by YC startups, and (2) the Who's Hiring threads. We kill job ads that are neither of these. (Incidentally, PG designed the /jobs posts to be the one privilege that YC startups get on HN. Other than that, the site policy is to treat them the same way as everybody else.) It's not uncommon for posts to be in a grey area between job ad and story. A programming test for an interview process doesn't seem all that grey, though, so I've been killing these. Sorry to be a hard-ass about this, but I don't want to be accused of enforcing the policy erratically. Edit: I'll unkill the post (but otherwise leave it demoted) so the discussion can remain open. Edit 2: The argument has been put to me that this is newsworthy because A16Z are doing it. Since that's corroborated by external sources (e.g. Twitter), and the post has gotten a lot of upvotes even while it was buried, that's reasonable—so we'll restore it as news rather than as an ad. Besides which, HN is one of the few places where a programming test counts as interesting.
- deleted 13y ago
  [deleted]
- rvivek 13y ago
  Thanks dang
- ig1 13y ago
  Can an actual list of policies be posted somewhere ? - I don't ever recall there ever being a discussion of such a policy before. Certainly there have been highly voted non-HN job posts in the past (e.g. Khan Academy)
  - dang 13y ago
    I'm reluctant to mess with the informal way that HN has been run over the years, so there isn't likely to be a complete list of rules. We do intend to augment the HN guidelines, though—just not any time soon. The new team has to get settled in first. In the meantime, I've been posting feedback comments most places that issues like this come up. You may need to look to the bottom of the threads to see them, because we usually mark them off-topic. And you can always get answers to questions by emailing hn@ycombinator.com.
argonaut 13y ago
Is this just for full-timers, or will there ever be an option for internships?
numair 13y ago
This doesn't quite make sense to me. Or, it makes sense, but not for the reasons presented on the surface. If you're Andreesen Horowitz, you have access to a nearly-unlimited supply of capital. You also have incredible access to information and dealflow throughout the industry. In situations in which you are not capital constrained, the most efficient way to establish a workforce is to acquire it from somewhere else; that's why acqui-hires make so much sense, because you've basically had someone else do the hard (and often completely unpredictable) work of figuring out if people can work together in a successful manner (which, I should note, is different from determination of whether the thing they're working on is successful). If I were A16Z, I would arbitrage my information advantage to secure a high quality engineering team by acquiring a company that had smart people but a lame product; going through a traditional hiring process would be a waste of my time (and since everyone is throwing money at my firm, it would also be a waste of time that could be better spent on figuring out where to deploy my capital). A16Z is rich enough to buy people, if it really wants people, for internal use (whether it is for their own company needs, or for some idea that they want to incubate inside of the firm). And even if they don't want to buy a whole set of people, they have enough wisdom and experience to know that you want to hire within a cohort of successful people rather than randoms (to see this in practice, take a look at Don Melton's post about Eazel alumni ending up at Apple on the front page right now). This tells me that their programming test has more to do with some sort of evaluation of this Hacker Rank service ("tell you what, we will run our own test on the service to see what happens!") or some other sort of analytical adventure. Or, you know, maybe it is really a programming test after all. Maybe they are hoping to find some sort of diamond-in-the-rough engineers who are socially and career-wise disconnected from an existing network. Some sort of "black swan programmer" or something. But I highly doubt it. So basically, if you go through this test, you're helping A16Z evaluate something; that something, however, probably isn't you.
- gcb0 13y ago
  Rich people are cheap. Why acquihire someone who knows how much they can ask if you can pay close to nothing to that test website (it's a genetic service) and get a clueless buy competent nerd that will be glad to work alone on a project for peanuts? That's why we're poor and they have "unlimited capital"
xedarius 13y ago
An entire page detailing what you want from me, coupled with a 180 minute test. Yet in all of this there are only 4 bullet point sentences describing what you're offering me. That deal is not for me.
- eternalban 13y ago
  They are establishing the tone of the relationship .. [& edit: it is a great filter for the personality type they want.]
lifeisstillgood 13y ago
My problem is I like taking these tests to find out what I don't know (quite a lot usually). To be fair though code reviews usually tell me more than tests these days. The constraint that gets me is the honor code - I cannot use an IDE with code completion?! but that leaves nano or notepad ...
- nostromo 13y ago
  I think you misread that. "You may use reference language documentation or use an IDE that has code completion features. "
  - lifeisstillgood 13y ago
    oooooooohhhhhhhhh what's the first thing your teachers said ? Read the instructions carefully :-)
jheriko 13y ago
aside from the fact that tests generally don't work i find this hilarious. "the most important requirement is significant independent programming experience as demonstrated by your GitHub account" a great way to find newbie hipsters and not experienced and seasoned devs. or at least limiting yourself to one community
- hesselink 13y ago
  Who says they're not recruiting other kinds of people through other channels? This kind of test is appealing to some people, and I'd guess those people have a github account.
- balajis 13y ago
  Hmm. In all fairness, the full quote is: A BS/MS/PhD in Computer Science or the equivalent is nice to have, but the most important requirement is significant independent programming experience as demonstrated by your GitHub account, personal projects, academic publications, or startup success. Your accomplishments are much more important than your paper credentials.
riffraff 13y ago
I'd take the test to see what's inside but... 3 hours? Can I get a t-shirt if I complete it at least? :)
ojbyrne 13y ago
Random question - how are "scatterplots and histograms" machine learning?
- graycat 13y ago
  This is a very difficult question and, thus, illustrates the transcendent brilliance and astoundingly "deep domain knowledge" of A16Z. An answer is available only on the bleeding edge of the most brilliant insights of research in machine learning. Ordinary people can't expect to understand such depth! All bow down to the incandescent brilliance of A16Z -- after all, one of their founders used C++ to write a GUI application to read a text markup language and display the formatted results! Maybe this was the first time, I mean after Word, TeX, and some others?
6thSigma 13y ago
Is anyone in their programming job given several distinct questions and only 3 hours to solve all of them? A more realistic programming test would be a single, complicated problem with an "end of day" time restraint.
graycat 13y ago
Looks to me like A16Z is blowing it. If Ben Horowitz prides himself on good management of technical work as suggested by his blog posts, then he didn't work very hard on the project of the OP. Issues: (1) Engineering School. Uh, guys, here in the US we have universities with schools of engineering -- MIT, Johns Hopkins, CMU, Georgia Tech, etc. Running a good engineering school is a lot of hard work. But this A16Z project has A16Z being their own engineering school, that is, specifying 'course material', testing for it, and deciding on pass/fail. Somehow it seems a bit arrogant for A16Z to believe that they can do the work of an engineering school. A16Z gets really off the track with their, > A BS/MS/PhD in Computer Science or the equivalent is nice to have, but the most important requirement is significant independent programming experience as demonstrated by your GitHub account, personal projects, academic publications, or startup success. Wow! "PhD in Computer Science" ... "is nice to have, but"...! Gee, guys, just shut down MIT, Johns Hopkins, CMU, Georgia Tech, etc. because now A16Z is here with "the right stuff"! Next, for their "as demonstrated by ... academic publications", sorry, A16Z, "academic publications" are not at all about "programming experience". Instead, the usual criteria are "new, correct, and significant" where "programming" really does not count more than, say, just the word processing for the paper. (2) Machine Learning. Good grief! That "machine learning" is about "scatterplots & histograms, logistic/linear regression, R/Matlab" fills a "much needed gap in the literature" and "would be illuminating if ignited". Their definition is for a 'statistical data assistant' mostly just for 'descriptive statistics'. Ever hear of J. Tukey? Ah, I'll be nice to you guys; not fun to be too harsh on small children! Then there is their unforgettable, > If you also know enough data science/machine learning to design useful metrics for dashboards and workflows, that’s a major plus. Uh, ever hear of reliability and validity? You have three hours to respond here with a relevant description! You are not permitted to use Google or Wikipedia! (3) Ego. It appears that the A16Z specification was written by someone who has enough ego to believe that their own 'skill set' is just "the right stuff". (4) Linux. People programming on Microsoft's platform can have done well but encountered only a small fraction of the topics in the A16Z document. The document is mostly about Linux. Sorry, guys, I'm writing in Visual Basic .NET; no, C# has too much of the deliberately 'idiosyncratic' syntax of C which I find too error prone and hate. (5) Three Letter Acronyms (TLAs). It appears that A16Z is using obscure TLAs as code word filters. Hmm. So, they have ORMs. Gee, what the heck is that? So, Google a little and get a Wikipedia entry and discover ORM abbreviates 'object-relational mapping'. Okay. Sure, if write the software with objects and keep the data in a relational database, then need move data between objects and tables. Given the huge variety of objects and tables, tough to believe that there is a significant 'subject' here. Then there is their ETL. Google again and discover that that TLA abbreviates "extract, transform, and load". So that is essentially 'extracting' data from whatever original sources, 'transforming' the data as needed by the work to be done, and 'loading' the data into a data base. Okay. Sounds entirely routine. I'm surprised they didn't also list (A) LRT, look at a screen, read it, and type in what need, (B) PSS for put on shoes and socks, (C) SDP for start car, drive to the office, and park the car, and especially the crucially important (D) GSL for Google a question, select the promising results, and learn what needed. One more: WTH, waste time on Hacker News! Okay, A16Z, two can play this game: Think of a large server farm or internet protocol network. Since this 'facility' is subject to errors of wide variety, it is important to monitor the facility for unexpected problems of wide variety, say, from software errors, hardware failures, system administration errors, security breaches, etc. To do this collect some data, say, from Microsoft's instrumentation means, HP's Mercury Interactive, what a relational database management system or Cisco box can report, etc. So, maybe 100 times a second, collect numerical data on each of 25 variables. Now we want a detector of 'anomalies', including for problems never seen before, that is, 'zero-day' problems. Such monitoring is essentially hypothesis testing. So, we want a suitable hypothesis test. Due to the fact that the multi-dimensional probability distribution of that 25 dimensional data is essentially impossible to describe at all accurately, we need an hypothesis test that is 'distribution-free'. Since the data is 25 dimensional, our test must also be 'multi-dimensional'. We want false alarm rate known in advance and adjustable in small steps over a wide range. And we want a good shot at doing nearly as well as the Neyman-Pearson result but with much less data than that result requires. Okay, A16Z, your mission, and I'd like to see you accept it, is to say how to do that. Of course, since you guys are so good and brilliant, you already know my published paper for the answer. But there is a typo in the paper. Where is it? You have three hours to post your response here. Also needed, find a fast way to do the specified computations. I found a way, but that is not in the paper. You have 24 hours to post the answer to this part! Don't report your answer in some silly way such a running code. Instead, report your answer in clear English with any appropriate mathematics written as high quality mathematical writing, e.g., W. Rudin. Go for it, guys! Uh, you won't find the answer on Google or GitHub! And asking around Silicon Valley won't help, either. You might want to rush off to the RAD lab project of Patterson and Fox funded by Google, Microsoft, and Sun for when they worked on such problems, but that won't help, either. My view: Here A16Z showed that their understanding of computing is still down at the level of a junior high self taught programmer.
minikomi 13y ago
Is that last question typical of interviews in the States? What is it selecting for? Aptitude in Maths?
thomaslangston 13y ago
The test's restriction on consulting material from websites and books, excluding the official reference documentation boggles my mind. Unofficial 3rd party documentation is standard kit for a developer, especially when using community driven programming languages.
jqm 13y ago
I didn't upload a resume (just a note saying I didn't have one) but did take the test. Hopefully they send some feedback... I'm not looking for a job, just wanted to know how I would do if I were:) Besides, I love programming challenges.
- jqm 13y ago
  I was actually disappointed when I didn't get feedback right away after doing the test. Something to be aware of if you are considering doing this... it looks like it really is for employment and not a challenge. I guess one of the first question "Are you ready to go to work in Menlo Park CA" (No) should have probably clued me in:) Oh well... they will "be in contact". Right.
  - jqm 13y ago
    And... my suspicions were confirmed. I still have no idea how I did on the test and they have provided no feedback. So, unless something changes, which I don't predict, it was a waste of time I wouldn't recommend doing the test nor applying to work at a company like this.
joncooper 13y ago
Programmer == Labor; don't ever forget it. The overwhelmingly largest share of returns accrue to capital. It's a good idea to resist being cast in the role of simply "programmer" over time, for this reason.
dpweb 13y ago
Have you guys thought about possibly employing smarterer.com rankings? I took your test, I think smarterer may yield a better evaluation, although you can't write code on a clock.

92 points by ghosh 13y ago

nilsbunger 13y ago

Cool, but... Why is A16z building an engineering team?

grosen 13y ago
Probably want to internally build some products. Makes for much better returns when you own 50+% of a company vs. ~20% via traditional funding.
Oculus 13y ago
My guess is to have an in-house team that can help out the companies they fund. I know that Netflix's cloud architect, Adrian Cockcroft, left to join a venture capital firm to advise on scaling issues for companies that firm funded[0]. 0 - http://gigaom.com/2014/01/07/netflixs-cloud-architect-adrian-cockcroft-is-leaving-to-join-battery-ventures/ http://gigaom.com/2014/01/07/netflixs-cloud-architect-adrian...
- pmarca 13y ago
  "My guess is to have an in-house team that can help out the companies they fund." Nope.
  - seizethecheese 13y ago
    I'm really curious... a hint?
- rythie 13y ago
  It's a big assumption though, and you need to spend 3 hours to find out what you are actually applying for.
  - fsk 13y ago
    You spend 3 hours to MAYBE find out what you're applying for. If you fail, presumably that's the end of the process.
cnaut 13y ago
Probably to build internal tools. A lot of VCs firms are trying to apply more data to their investment decisions and that requires internal tools to handle it all. Also makes sense given the qualifications they are asking for.
- graycat 13y ago
  A better way: Actually read and understand the submissions from the entrepreneurs!
  - ironchef 13y ago
    Are you suggesting that a16z doesn't do this? Or that their firm is not doing well? It would seem to me that their limited track record is ... more than adequate. Similarly, I would infer that your suggestion is ignorant that they have interest in the companies post-funding (as cnaut suggests "A lot of VCs firms are trying to apply more data to their investment decisions and that requires internal tools to handle it all"... which I think would suggest that they would want to examine the results of said decisions as well (which goes beyond simply the submission process)).
    - graycat 13y ago
      > Are you suggesting that a16z doesn't do this? Yes I am. (1) It appears, really is quite clear, that they have high contempt for initial contacts from entrepreneurs via e-mail. Instead they want to believe that the 'good' entrepreneurs will have enough 'exposure', enough in 'networking', and enough in 'hustle', e.g., on LinkedIn, GetHub, etc., to come to the attention of A16Z by means other than an e-mail message "over the transom". E.g., they want an entrepreneur to get introductions from bankers, lawyers, accountants, or CEO's of companies in the A16Z portfolio. So, A16Z (A) wants others to 'filter' their entrepreneurs for them and/or (B) believes that lawyers, etc. are good sources of good recommendations. Nonsense. (2) A16Z will just refuse to consider any very technical support for a project. E.g., if a project has some powerful, valuable 'secret sauce', intellectual property, and high technological barrier to entry, A16Z can easily just dismiss the technical issues, the project, and the entrepreneur. A16Z wants to believe that they have 'deep domain knowledge' where they invest and, thus, wants to believe that anything new that A16Z has not heard of must be nonsense. If some supporting technical material is not easy reading for them, then they believe that they are justified in just regarding it as nonsense. But, looking at the partners of A16Z, it's tough to find one with enough technical qualifications and/or accomplishments to (A) get a tenure track position in the engineering school of a leading research university, (B) be an editor of a leading technical journal of original research, (C) publish a peer-reviewed paper of original research in a technical field in a good or better journal of original research, or (D) get an NSF grant for original research. Net, A16Z apparently is not competent to review or even to direct a competent review of technical material in information technology. What the A16Z financial performance really is is likely not easy to discover. You seem to suggest that the project for which the OP is some recruiting is for an internal information system to do causal modeling and make valuable predictions for venture investing. It would appear that they want to do cross tabulation and curve fitting; sorry, there is nearly no chance those techniques will work. Why? In simple terms, they are looking for new, exceptional cases, and looking at 'averages' from the past, even of successes in the past, is just intellectual self-abuse. If they are making money, then, fine. But from all I can see, in information technology their competence is way below a level that would be useful for actual technical work or doing a technical review of such work. I've been around technical work in US national security, US business and start ups, including one of the most successful venture funded companies ever, and high end academic research, hold a Ph.D. in engineering from one of the world's best research universities, have published peer-reviewed original research in computer science, applied mathematics, and artificial intelligence, and in comparison the technical qualifications of A16Z look like a bad joke. Yes, most good quantitative analysis needs some computing and programming. So, in their recruiting A16Z seems to believe that the computing, which is necessary, is also sufficient. It's not. By an analogy, A16Z is looking for nurse practitioners instead of MD physicians. Both are necessary, but the nurse practitioners are not sufficient. There is a difference! But, why help A16Z? They have money enough now!
gautamsivakumar 13y ago
They have $4.2bn under management, and a very smart team. I am sure there is a lot that can be done...
neil_s 13y ago
My guess would be a team to provide a data-centric approach to investment. Crunch a whole lot of numbers to identify industry trends and specific firms that are promising, and see if those can replicate human performance. Or they're simply handling recruitment for a stealth portfolio startup.
- graycat 13y ago
  > industry trends Why? The VC firms never invest in an "industry", i.e., there are no such 'index funds'. Instead, VC firms invest in individual companies, and there what is just crucial is what the company does. If the company looks good, then mostly to heck with the darned industry. > Or they're simply handling recruitment for a stealth portfolio startup. In this case, poor startup: A16Z is doing a "piss poor" job 'helping' the startup!

tpae 13y ago

I seriously hope this doesn't become a norm. I hate timed tests.

rdl 13y ago
Seems like the obvious (if dishonest) thing would be to use a throwaway phone # to get the test, then spend a day or so solving it, then type it in (with plausible delays) using your real number a day later. Assuming the test is roughly the same each time (even if details are different, it's probably of the same general class). In grading, the results of 6-8h of work entered in 2h would look more impressive than 3h of work in 3h, I suspect. (Sometimes I hate that whenever someone gives me a situation, my first thought is "how can this be cheated or gamed", followed by "what controls can I put in place to prevent that, and and what costs"; it's basically automatic.)
- rmmw 13y ago
  Credentials through phone # is more of a deterrent. We're also going to have an in-person coding exercise as a next step, which would be much harder to game :)
  - deleted 13y ago
    [deleted]
  - flylib 13y ago
    are you saying you go straight to a in-person coding exercise as the next step? seems odd, there is no phone interview between then?
- golergka 13y ago
  Why would you want to appear better then you are at the interview? You'll get found out eventually later, and this will end up just spending your time.
  - sireat 13y ago
    There are plenty of developers/programmers who do not ace tests with rigid constraints, but are quite competent in a more relaxed setting. That is they are not the 1% that the all hirers, but they are maybe top 20% and that is actually what most jobs require no matter the delusions of hiring managers. Granted this particular job might be an exception but I doubt it.

elnate 13y ago

Doesn't accept my (NZ) mobile number as valid :(

balajis 13y ago
Did you include the country code prefix? Otherwise please try emailing vivek at hackerrank dot com with the bug to see if he can reproduce.
rmmw 13y ago
You can also email test-help AT a16z DOT com and we can send you a direct link (where you won't need to receive credentials by phone)
- pjreddie 13y ago
  I love that people do this. Do you really thing someone is going to write a web scraper and NOT include obvious things like converting AT -> @ and DOT com -> .com? Also, who even worries about spam any more? I haven't seen actual spam in my inbox in years.
  - cflee 13y ago
    Should we just go "AT AT AT" or something instead? Or perhaps adapt from Mandarin speakers, who literally call the @ sign "the little mouse" ;) Spam remains an issue if for whatever reason you're still using rulesets/blacklists/greylisting and the like, instead of Bayesian filtering.

Oculus 13y ago

1:30 am on the night before a physics test - why must this challenge be so tempting! There goes that studying :)

abalone 13y ago

I wonder if this could act as a negative filter for good programmers. Who worth their salt is going to invest 3 hours of their time just to submit a resume? Wouldn't you want to talk to the employer first to see if they're a good fit? In a tight labor market, people willing to do this are more desperate than others. So it's going to filter for desperate developers who passed a 3 hour basic skills test. Old fashioned prior experience and education is probably still a better first filter. This would probably make more sense specifically for junior hires right out of school or with no experience.

balajis 13y ago
Well, empirically we're getting quite a lot of people attempting it! :) Three hours is pretty lenient (you could blaze through it in less time), though the last question is challenging. We think the questions are kind of fun as well. Without something like this we'd just have 100s-1000s of unstructured resumes in different formats to rank against each other. This process is more fair and (IMO) allows someone without a college degree, or from outside the US, to compete on a level playing field.
- jay_kyburz 13y ago
  Do you mind if people do the test with no interest in the job?
  - balajis 13y ago
    Sure, go ahead! We just ask that you not share the questions with others after completing it.
- 7Figures2Commas 13y ago
  > Well, empirically we're getting quite a lot of people attempting it! > Without something like this we'd just have 100s-1000s of unstructured resumes in different formats to rank against each other. So instead of hundreds or thousands of unstructured resumes in different formats, you have hundreds or thousands of completed programming tests. Even assuming that you automatically filter out submissions that fail testing, given that there are a lot of ways to solve a problem, you'd still be stuck evaluating potentially hundreds of correct submissions. More importantly, a test doesn't seem to be aligned with your top priority: > ...but the most important requirement is significant independent programming experience as demonstrated by your GitHub account, personal projects, academic publications, or startup success. Are you assuming that all of the folks most accomplished in the real world are going to complete your programming test successfully? Since you ultimately seem to be interested in folks who can build dashboards, I would think you'd be far better off asking candidates to build a dashboard prototype using sample data you provide. There are lot of really smart developers out there with advanced degrees who could solve programming quizzes all day long but couldn't design and implement a useful dashboard application.

deleted 13y ago

[deleted]

peapod91 13y ago

Will this be an ongoing opportunity or is there a deadline to submit this by?

balajis 13y ago
It's an ongoing opportunity, but we will be hiring the first set of people soon. So it will be to your advantage to apply sooner rather than later.

_pmf_ 13y ago

> machine learning (scatterplots & histograms, logistic/linear regression, R/Matlab) So, someone taking Statistics 101 is considered to know machine learning in the startup circus? (Not that this would really surprise me, given the tendency to appropriate terms that had a certain meaning for purposes that have only a passing similarity to the original meaning.)

balajis 13y ago
We could certainly rattle off more terms :) Kernel density estimation? Deep learning? Semi-supervised learning? But it's amazing how far you can get with some simple scatterplots, histograms and linear/logistic regressions. a) Most of the time, more data beats better algorithms: http://anand.typepad.com/datawocky/2008/03/more-data-usual.html http://anand.typepad.com/datawocky/2008/03/more-data-usual.h... b) Even those seemingly simple things get complicated when you have a lot of data: http://komarix.org/ac/lr http://komarix.org/ac/lr

buro9 13y ago

Does this mean Mattermark https://mattermark.com/ https://mattermark.com/ have started a tech arms war amongst the larger VC firms to create their own toolsets to differentiate their deal flow from others? Mattermark has gone some way to provide proof that data can make a difference, at least at highlighting opportunities that might not have been visible before. I wondered whether this would result in herding investors towards some companies, which means differentation and opportunity comes from spotting things the others miss... finding the outliers that Mattermark miss. Interesting that the test is "make a dashboard", well into the standard skillset for most analyst programmers who work for financial institutions (I built a lot of this stuff for the finance sector in London).

dang 13y ago

HN policy for years has been to allow two kinds of job ad: (1) job posts by YC startups, and (2) the Who's Hiring threads. We kill job ads that are neither of these. (Incidentally, PG designed the /jobs posts to be the one privilege that YC startups get on HN. Other than that, the site policy is to treat them the same way as everybody else.) It's not uncommon for posts to be in a grey area between job ad and story. A programming test for an interview process doesn't seem all that grey, though, so I've been killing these. Sorry to be a hard-ass about this, but I don't want to be accused of enforcing the policy erratically. Edit: I'll unkill the post (but otherwise leave it demoted) so the discussion can remain open. Edit 2: The argument has been put to me that this is newsworthy because A16Z are doing it. Since that's corroborated by external sources (e.g. Twitter), and the post has gotten a lot of upvotes even while it was buried, that's reasonable—so we'll restore it as news rather than as an ad. Besides which, HN is one of the few places where a programming test counts as interesting.

deleted 13y ago
[deleted]
rvivek 13y ago
Thanks dang
ig1 13y ago
Can an actual list of policies be posted somewhere ? - I don't ever recall there ever being a discussion of such a policy before. Certainly there have been highly voted non-HN job posts in the past (e.g. Khan Academy)
- dang 13y ago
  I'm reluctant to mess with the informal way that HN has been run over the years, so there isn't likely to be a complete list of rules. We do intend to augment the HN guidelines, though—just not any time soon. The new team has to get settled in first. In the meantime, I've been posting feedback comments most places that issues like this come up. You may need to look to the bottom of the threads to see them, because we usually mark them off-topic. And you can always get answers to questions by emailing hn@ycombinator.com.

argonaut 13y ago

Is this just for full-timers, or will there ever be an option for internships?

numair 13y ago

This doesn't quite make sense to me. Or, it makes sense, but not for the reasons presented on the surface. If you're Andreesen Horowitz, you have access to a nearly-unlimited supply of capital. You also have incredible access to information and dealflow throughout the industry. In situations in which you are not capital constrained, the most efficient way to establish a workforce is to acquire it from somewhere else; that's why acqui-hires make so much sense, because you've basically had someone else do the hard (and often completely unpredictable) work of figuring out if people can work together in a successful manner (which, I should note, is different from determination of whether the thing they're working on is successful). If I were A16Z, I would arbitrage my information advantage to secure a high quality engineering team by acquiring a company that had smart people but a lame product; going through a traditional hiring process would be a waste of my time (and since everyone is throwing money at my firm, it would also be a waste of time that could be better spent on figuring out where to deploy my capital). A16Z is rich enough to buy people, if it really wants people, for internal use (whether it is for their own company needs, or for some idea that they want to incubate inside of the firm). And even if they don't want to buy a whole set of people, they have enough wisdom and experience to know that you want to hire within a cohort of successful people rather than randoms (to see this in practice, take a look at Don Melton's post about Eazel alumni ending up at Apple on the front page right now). This tells me that their programming test has more to do with some sort of evaluation of this Hacker Rank service ("tell you what, we will run our own test on the service to see what happens!") or some other sort of analytical adventure. Or, you know, maybe it is really a programming test after all. Maybe they are hoping to find some sort of diamond-in-the-rough engineers who are socially and career-wise disconnected from an existing network. Some sort of "black swan programmer" or something. But I highly doubt it. So basically, if you go through this test, you're helping A16Z evaluate something; that something, however, probably isn't you.

gcb0 13y ago
Rich people are cheap. Why acquihire someone who knows how much they can ask if you can pay close to nothing to that test website (it's a genetic service) and get a clueless buy competent nerd that will be glad to work alone on a project for peanuts? That's why we're poor and they have "unlimited capital"

xedarius 13y ago

An entire page detailing what you want from me, coupled with a 180 minute test. Yet in all of this there are only 4 bullet point sentences describing what you're offering me. That deal is not for me.

eternalban 13y ago
They are establishing the tone of the relationship .. [& edit: it is a great filter for the personality type they want.]

lifeisstillgood 13y ago

My problem is I like taking these tests to find out what I don't know (quite a lot usually). To be fair though code reviews usually tell me more than tests these days. The constraint that gets me is the honor code - I cannot use an IDE with code completion?! but that leaves nano or notepad ...

nostromo 13y ago
I think you misread that. "You may use reference language documentation or use an IDE that has code completion features. "
- lifeisstillgood 13y ago
  oooooooohhhhhhhhh what's the first thing your teachers said ? Read the instructions carefully :-)

jheriko 13y ago

aside from the fact that tests generally don't work i find this hilarious. "the most important requirement is significant independent programming experience as demonstrated by your GitHub account" a great way to find newbie hipsters and not experienced and seasoned devs. or at least limiting yourself to one community

hesselink 13y ago
Who says they're not recruiting other kinds of people through other channels? This kind of test is appealing to some people, and I'd guess those people have a github account.
balajis 13y ago
Hmm. In all fairness, the full quote is: A BS/MS/PhD in Computer Science or the equivalent is nice to have, but the most important requirement is significant independent programming experience as demonstrated by your GitHub account, personal projects, academic publications, or startup success. Your accomplishments are much more important than your paper credentials.

riffraff 13y ago

I'd take the test to see what's inside but... 3 hours? Can I get a t-shirt if I complete it at least? :)

ojbyrne 13y ago

Random question - how are "scatterplots and histograms" machine learning?

graycat 13y ago
This is a very difficult question and, thus, illustrates the transcendent brilliance and astoundingly "deep domain knowledge" of A16Z. An answer is available only on the bleeding edge of the most brilliant insights of research in machine learning. Ordinary people can't expect to understand such depth! All bow down to the incandescent brilliance of A16Z -- after all, one of their founders used C++ to write a GUI application to read a text markup language and display the formatted results! Maybe this was the first time, I mean after Word, TeX, and some others?

6thSigma 13y ago

Is anyone in their programming job given several distinct questions and only 3 hours to solve all of them? A more realistic programming test would be a single, complicated problem with an "end of day" time restraint.

graycat 13y ago

Looks to me like A16Z is blowing it. If Ben Horowitz prides himself on good management of technical work as suggested by his blog posts, then he didn't work very hard on the project of the OP. Issues: (1) Engineering School. Uh, guys, here in the US we have universities with schools of engineering -- MIT, Johns Hopkins, CMU, Georgia Tech, etc. Running a good engineering school is a lot of hard work. But this A16Z project has A16Z being their own engineering school, that is, specifying 'course material', testing for it, and deciding on pass/fail. Somehow it seems a bit arrogant for A16Z to believe that they can do the work of an engineering school. A16Z gets really off the track with their, > A BS/MS/PhD in Computer Science or the equivalent is nice to have, but the most important requirement is significant independent programming experience as demonstrated by your GitHub account, personal projects, academic publications, or startup success. Wow! "PhD in Computer Science" ... "is nice to have, but"...! Gee, guys, just shut down MIT, Johns Hopkins, CMU, Georgia Tech, etc. because now A16Z is here with "the right stuff"! Next, for their "as demonstrated by ... academic publications", sorry, A16Z, "academic publications" are not at all about "programming experience". Instead, the usual criteria are "new, correct, and significant" where "programming" really does not count more than, say, just the word processing for the paper. (2) Machine Learning. Good grief! That "machine learning" is about "scatterplots & histograms, logistic/linear regression, R/Matlab" fills a "much needed gap in the literature" and "would be illuminating if ignited". Their definition is for a 'statistical data assistant' mostly just for 'descriptive statistics'. Ever hear of J. Tukey? Ah, I'll be nice to you guys; not fun to be too harsh on small children! Then there is their unforgettable, > If you also know enough data science/machine learning to design useful metrics for dashboards and workflows, that’s a major plus. Uh, ever hear of reliability and validity? You have three hours to respond here with a relevant description! You are not permitted to use Google or Wikipedia! (3) Ego. It appears that the A16Z specification was written by someone who has enough ego to believe that their own 'skill set' is just "the right stuff". (4) Linux. People programming on Microsoft's platform can have done well but encountered only a small fraction of the topics in the A16Z document. The document is mostly about Linux. Sorry, guys, I'm writing in Visual Basic .NET; no, C# has too much of the deliberately 'idiosyncratic' syntax of C which I find too error prone and hate. (5) Three Letter Acronyms (TLAs). It appears that A16Z is using obscure TLAs as code word filters. Hmm. So, they have ORMs. Gee, what the heck is that? So, Google a little and get a Wikipedia entry and discover ORM abbreviates 'object-relational mapping'. Okay. Sure, if write the software with objects and keep the data in a relational database, then need move data between objects and tables. Given the huge variety of objects and tables, tough to believe that there is a significant 'subject' here. Then there is their ETL. Google again and discover that that TLA abbreviates "extract, transform, and load". So that is essentially 'extracting' data from whatever original sources, 'transforming' the data as needed by the work to be done, and 'loading' the data into a data base. Okay. Sounds entirely routine. I'm surprised they didn't also list (A) LRT, look at a screen, read it, and type in what need, (B) PSS for put on shoes and socks, (C) SDP for start car, drive to the office, and park the car, and especially the crucially important (D) GSL for Google a question, select the promising results, and learn what needed. One more: WTH, waste time on Hacker News! Okay, A16Z, two can play this game: Think of a large server farm or internet protocol network. Since this 'facility' is subject to errors of wide variety, it is important to monitor the facility for unexpected problems of wide variety, say, from software errors, hardware failures, system administration errors, security breaches, etc. To do this collect some data, say, from Microsoft's instrumentation means, HP's Mercury Interactive, what a relational database management system or Cisco box can report, etc. So, maybe 100 times a second, collect numerical data on each of 25 variables. Now we want a detector of 'anomalies', including for problems never seen before, that is, 'zero-day' problems. Such monitoring is essentially hypothesis testing. So, we want a suitable hypothesis test. Due to the fact that the multi-dimensional probability distribution of that 25 dimensional data is essentially impossible to describe at all accurately, we need an hypothesis test that is 'distribution-free'. Since the data is 25 dimensional, our test must also be 'multi-dimensional'. We want false alarm rate known in advance and adjustable in small steps over a wide range. And we want a good shot at doing nearly as well as the Neyman-Pearson result but with much less data than that result requires. Okay, A16Z, your mission, and I'd like to see you accept it, is to say how to do that. Of course, since you guys are so good and brilliant, you already know my published paper for the answer. But there is a typo in the paper. Where is it? You have three hours to post your response here. Also needed, find a fast way to do the specified computations. I found a way, but that is not in the paper. You have 24 hours to post the answer to this part! Don't report your answer in some silly way such a running code. Instead, report your answer in clear English with any appropriate mathematics written as high quality mathematical writing, e.g., W. Rudin. Go for it, guys! Uh, you won't find the answer on Google or GitHub! And asking around Silicon Valley won't help, either. You might want to rush off to the RAD lab project of Patterson and Fox funded by Google, Microsoft, and Sun for when they worked on such problems, but that won't help, either. My view: Here A16Z showed that their understanding of computing is still down at the level of a junior high self taught programmer.

minikomi 13y ago

Is that last question typical of interviews in the States? What is it selecting for? Aptitude in Maths?

thomaslangston 13y ago

The test's restriction on consulting material from websites and books, excluding the official reference documentation boggles my mind. Unofficial 3rd party documentation is standard kit for a developer, especially when using community driven programming languages.

jqm 13y ago

I didn't upload a resume (just a note saying I didn't have one) but did take the test. Hopefully they send some feedback... I'm not looking for a job, just wanted to know how I would do if I were:) Besides, I love programming challenges.

jqm 13y ago
I was actually disappointed when I didn't get feedback right away after doing the test. Something to be aware of if you are considering doing this... it looks like it really is for employment and not a challenge. I guess one of the first question "Are you ready to go to work in Menlo Park CA" (No) should have probably clued me in:) Oh well... they will "be in contact". Right.
- jqm 13y ago
  And... my suspicions were confirmed. I still have no idea how I did on the test and they have provided no feedback. So, unless something changes, which I don't predict, it was a waste of time I wouldn't recommend doing the test nor applying to work at a company like this.

joncooper 13y ago

Programmer == Labor; don't ever forget it. The overwhelmingly largest share of returns accrue to capital. It's a good idea to resist being cast in the role of simply "programmer" over time, for this reason.

dpweb 13y ago

Have you guys thought about possibly employing smarterer.com rankings? I took your test, I think smarterer may yield a better evaluation, although you can't write code on a clock.