5 ms·

A Modern App Developer and an Old-Timer System Developer Walk into a Bar

148 points by zhenjl 11y ago

cellis 11y ago
I'm betting on the modern app developers system being much easier to maintain and understand ( and pass off to junior programmers), at the cost of speed.
- err4nt 11y ago
  I don't know about that. I'm a front-end developer, so I know none of the above. I'd rather decipher some complex bit-counting scheme one person made (and documented) in a language I don't write, than to try to piece together a whole system of tools written by various people I can't talk to, and tied together by one person for this specific task. Which one is going to require more reading? Which one is going to be easier to refactor? Although neither develop sounds like a teammate I'd hope to collaborate with, I'd much rather inherit Mr Old-Timer's code to maintain than Mr Modern Developer
- merpnderp 11y ago
  I was going to say no way, basic boolean logic is doable by every programmer. But then I saw the "old-timer" had some boolean errors themself. So......Maybe you're right.
nobullet 11y ago
Is this post Go biased? Why not C++ or Erlang? :) You can do the same with Python
- bottled_poe 11y ago
  Agreed, I think very few "Old-timers" would be using go.
- x1024 11y ago
  Looking at the other posts in the blog, it's definitely Go biased. Not that it's necessarily a bad thing. But it's making some seriously tenuous assumptions about "modern app developers". It's almost "Goofus & Gallant" levels of hyperbole.
hellcow 11y ago
As VCs often look for keywords, I suppose the trick is to build the old-timer system but describe it as the modern web developer.
- brianwawok 11y ago
  Lol store the bits in a file called hadoop.txt. if anyone asks how you did a query say you looked in hadoop.
kogepathic 11y ago
Why wouldn't you want to use an SQL database with an index to store this data? Just have a column for the ports and a boolean flag for the status of the port? Honestly, I'd probably use bash/nmap/ping and psql to insert data. Want to query? psql and grep. These examples seem like a great way to re-invent the wheel with modern buzzwords.
- Spoom 11y ago
  I was thinking this as well. This is a problem with a well-defined schema. Why would you use a schemaless NoSQL database? Additionally, many of the bitwise operations used by the old school dev have analogous abstractions in SQL, which are probably easier to understand.
- hackercomplex 11y ago
  Why not SQLite ?
  - hendler 11y ago
    or redis
  - Sanddancer 11y ago
    I'd just take that as favorite tool for the job, SQLite's pretty interchangeable. Also, if you've already gotten a database engine running, it's probably best to just keep everything there. Otherwise, you get into the situation I was in at a previous job in which certain page loads from an app written before I started there required a sun jsp app for the main part of the page, which made requests to pull some data from an Oracle database, some data from an apache mod_perl app that queried from a mysql database, some data from an apache mod_jk app that pulled from a c++ app with its own database format, and some data from an apache mod_jk app that stored and retrieved from a postgresql instance. Everything was nice and well-documented, but mein gott was that a lot of moving parts.
  - skrause 11y ago
    In PostgreSQL you could use a native type to store your IP addresses and subnets (even for IPv6): http://www.postgresql.org/docs/9.5/static/datatype-net-types.html http://www.postgresql.org/docs/9.5/static/datatype-net-types...
bottled_poe 11y ago
Maybe this is a stupid question, but why not just use a relational database?
- x1024 11y ago
  Because then it would seem that modern app developers are actually competent, and that's not the author's intent.
opticalfiber 11y ago
I'm not really sure what the point of this article is. If you're clever, you can represent any data as a bit array. And once you're there, counting bits or XORing them together is easy. But is that system easy to understand? Is the code easy to read for the rest of the developers who work on the project? Or for future hires? What happens if the data set becomes too large to fit in memory on a single machine? Storage is cheap and getting cheaper every day. The whole data set described in this problem (done the "inefficient" way) fits on a flash drive at today's capacities. There are always tradeoffs when it comes to architecture design. Speed and storage space are only two of the many factors that require consideration.
- amorphic 11y ago
  +1 Ill add that having your data in Elastic means that a (probably non-technical) user still can use it as part of their analysis without having to get dev involved.
  - opticalfiber 11y ago
    Yes! ES is great for a lot of reasons, not the least of which is the one you point out.
- jdmoreira 11y ago
  "if the data set becomes too large to fit in memory on a single machine?" You just use mmap. If it become bigger than the disk capacity of single machine, you can still mount a distributed filesystem and use mmap.
  - voltagex_ 11y ago
    Slightly off topic: http://yourdatafitsinram.com/ http://yourdatafitsinram.com/ and more seriously https://www.youtube.com/watch?v=SiSY1b0am5w https://www.youtube.com/watch?v=SiSY1b0am5w which is a really good talk about the challenges of sysadmining for scientists. What do you do when 2TB of RAM isn't enough?
the_cat_kittles 11y ago
basically just an example of how choosing a bad data structure can make your life a headache
olalonde 11y ago
Business Guy shakes his head and gets a subscription on https://www.shodan.io/ https://www.shodan.io/
- chrismorgan 11y ago
  And that is so often the best kind of response.
- merpnderp 11y ago
  I was afraid I was going to miss the joke until I clicked the link and got a real chuckle.
smoyer 11y ago
"I can perform a simple AND operation on the 3 monthly bit arrays, and then count the number of “1” bits." I think an old-timer would tell you that he'd be using an "OR" operation to count the number of hosts seen in the last three months. As a long-time embedded systems engineer, you learn to guard every CPU cycle and memory location jealously. That behavior, along with avoiding needless abstractions that complicate the system AND cost CPU and memory, have served me well as I've switched to higher-level software. As an aside, memory resident databases are now practical even for problems of this size. I've used the CERN Colt library to provide sparse arrays stored in memory in place of databases several times now. My current pet project is processing a mouse genome for my daughter's lab. With the amount of data being read, I'd always be I/O bound if I wasn't packing a lot of working data into memory. This is the programmers equivalent of the CPU's cache, but one we can easily control. Give it a try!
- Liru 11y ago
  It may be because I haven't had my coffee yet, but how would using OR help here? 1 OR 0 OR 0 returns 1, but the host was offline in 2 of the past 3 months, which means that it should be 0...
  - smoyer 11y ago
    You might be right ... but it's entirely dependent upon what the author meant. The problem statement was "How Many Total Hosts Were Seen as “Up” in the Past 3 Months" which I read as "How Many Total Hosts Were Seen as “Up” in any of the Past 3 Months". You've correctly pointed out that the author might have meant "How Many Total Hosts Were Seen as “Up” in all of the Past 3 Months". This sort of ambiguity is why BDUF is so hard to pull off. Only lawyers are trained to write completely unambiguous prose (and they fall back to having very precise meanings for words to help them succeed).
    - sokoloff 11y ago
      Given that the modern app developer did a de-dupe operation for his solution, and that the most plain reading of the English text is "in any of the scans", I think the OR is correct.
ComSubVie 11y ago
That's actually a good comparison between different mindsets. Even if it reads a bit biased pro-go (or pro-old-time-system-developer) it doesn't tell what's the best way to implement this - and in my opinion it's neither one. So what's actually the best way to implement this? What are the motivations for choosing one way? "Modern App Developer Way": + adoptable, scalable, readable data formats - storage, computation "Old-Time System Developer Way": + storage, computation - "locked-in" data formats "Database Developer Way": + ease of implementation ? storage, computation Any other options/ideas?
- sklivvz1971 11y ago
  Use Redis?
VLM 11y ago
Funny quote from the article: "This is a big data problem." Hmm maybe on a raspberry pi, or some kind of retrocomputing challenge (Do this on a simulated IBM 1130!). I don't know how an app developer thinks, but I know the article characterization of the old developer is wrong, we'd chuck it into the existing relational DB, maybe spin up a new cloud instance and reserve some NAS space if necessary but this is a pretty small data set by modern standards so probably nothing special is required. All the reporting devolves into silly SQL onliner competition. The first question is a COUNT(*) and GROUP BY. The second question is a ridiculously simple SELECT. The third question is another simple select if you stored your ip addr bytes in separate columns, even if you allow non /24 addrs. The fourth is another GROUP BY. The article does fit the stereotype I've seen that new programmers prioritize ease of storing data over ease of reporting, and old timers vice versa. Like the difference between coming up with the fastest next move in checkers vs "solving" checkers in the game theory sense.
smarx007 11y ago
Old-time system developers use C.
- dd9990 11y ago
  And likely C89... certainly not Go.
- pskocik 11y ago
  Wouldn't you classify golang authors as old-time system developers?
- Silhouette 11y ago
  Old-time system developers wrote Go.
dzhiurgis 11y ago
Old timer spends rest of 2016 to implement his solution that saves $100 in computing resources.
- Silhouette 11y ago
  Reality: OId timer spends about 10 minutes to implement his solution. If the older timer's solution does save $100 in computing resources, then at that rate the the old timer is saving around $1M per year compared to the "modern" developer.
  - dzhiurgis 11y ago
    Given both are experts, point-and-click will always be faster to implement.
szc 11y ago
A couple of errors near the end for "How Many Hosts Were “Up” Last Month But Now It’s “Down”" and vice versa. The error is that using xor on the same item twice is a no-op... (A xor B) xor A == B. The correct equations would be, !this_month and last_month = "up" last month, down now. and this_month and !last_month = "down" last month, up now.
raesene2 11y ago
Pentester: I'll use mass scan, someone else already solved this problem. https://github.com/robertdavidgraham/masscan https://github.com/robertdavidgraham/masscan
tyre 11y ago
I suppose I'm a modern app developer, so here's what I would actually do: Postgres. 1) I'm not going to invent my own system. I am not the first person who wants to store addresses and boolean values. If I'm looking at this occasionally over many months, I don't want to keep relearning my elegant bit-structure. 2) SQL is great, Postgres is even better with built-in network operators[1] for the more specific queries (e.g. subnets.) 3) Bitfields are all fun and games until you want to change things. I'm already having flashbacks to rails `has_bitfield` columns. Also, fewer people can confidently twiddle bits than can understand boolean SQL columns. 4) The author mentions possible compression methods. I'd rather a tested project like Postgres think about everything to do with storing my data. I understand the criticism, but with cloud storage so cheap it feels like optimization just to show off. I'd rather save time than bits. If there is a satire that involves me using Postgres when I shouldn't, that would be welcome. [1] http://www.postgresql.org/docs/current/static/functions-net.html http://www.postgresql.org/docs/current/static/functions-net....
- krick 11y ago
  Exactly. Change "json" to "relational DB" and it's not close as ridiculous as author implies. All questions mentioned could be answered with a single SQL: although it will take quite a while. And it is much more extendable than this "huge array" approach. The only thing is that by saying "Postgres" you say pretty much nothing: the real question here is how'd you structure the db. 300M a month is pretty huge, so it's really worth to think if we should just pile everything in 1 table (unique key ip+month, all ports are columns) or if we can denormalize something.
  - GFK_of_xmaspast 11y ago
    300M per month doesn't sound like all that much to me?
    - krick 11y ago
      Well, it will be quite a table to do your selects against. Assuming that your hardware for such a project is fairly modest, after a couple of months index won't really fit into the memory, and even selecting a record by the key would take 30 jumps over that table just after 4 months. And most of the tasks in question require running through the whole thing. Not a tragedy, but well worth considering aggregating some data as it comes, especially as it costs pretty much nothing at this point (nmap takes longer anyway). Like counting number of IPs with the open port 22 would be just incrementing some number at the dedicated table by simple rule in the code. Maybe it's worth to invent some more interesting denormalization in order to be able to run more complex queries easily. Maybe I'm wrong: I'd have to try out that to see how fast it actually computes, but I remember having problems with much smaller tables: like these 300M in total, maybe. Although, it was much "wider", with varchars and stuff.
swinglock 11y ago
Unfortunately our hero the array developer got a bit too clever, pun intended, and produced an incorrect program. The last two of the XOR tricks won't work the way he thought.
andybak 11y ago
Ah. Thank god the imaginary old school guy is better than the imaginary straw man modern guy! I'd better go learn Go immediately. Personally I'd start off by looking into using Cython or Numpy and maybe pickle to disk for storage. Does that make me the old guy or the new guy or is it a false dichotomy?
tyingq 11y ago
Lost me at "Old-Timer Developer: I will use Go." The old-timers in your vicinity must be different than the ones in mine.
- Namrog84 11y ago
  I think we might come from a similar vicinities. Here I am thinking go was modern. I expected old timer to be using c. As that's probably what I would have done
  - tyingq 11y ago
    "What's this Go? C for people that are afraid of malloc() ?"
- emmelaich 11y ago
  I think Go is an acceptable modern language for lovers of C. Especially those who are allergic to C++.
  - snaky 11y ago
    Go is an acceptable modern language for modern haters of C
asragab 11y ago
As a relatively junior developer, I have been told my some of my senior colleagues that I sometimes miss the forest for the trees, and so this may be another instance of that. But it doesn't seem "trivial" to me how one gets the output of nmap into whatever database, data structure one chooses here. I know nmap can produce XML, presumably there is a csv format, but it would seem like the XML/CSV -> JSON conversion (following our intrepid Modern App Developer) would be an easier more maintainable way to go, versus XML -> to bit array (memory map file). Also, is managing the nmap or masscan and whatever other ancillary processes required to execute this plan equally as onerous in either paradigm? Finally, and this is likely controversial, this particular problem "feels" like its stacked against the Modern App Developer, given that it isn't trying to solve a problem most Modern Apps try to solve (or try to solve as an end rather than a means to an end)
- deleted 11y ago
  [deleted]
- int_handler 11y ago
  Or, if you want to use a binary format without having to deal with all of the complexities of inventing your own, you can also use Protocol Buffers. You can even use Protocol Buffers as a schema for JSON too as of 3.0.
slightlycuban 11y ago
Point of this article is not the point of this article. The clever bit was: > ...the IPv4 address will convert into a number... Many fancy, post-modern app developers might insist, "You won't do math with an address; it's not a number." But some things are numbers, with a scheme and pattern you can exploit.
Silhouette 11y ago
It's fascinating how many people here are defending the "modern app developer" approach, mostly with arguments about flexibility, maintainability, ability to pass on the code to junior developers, and the like. If you think about what the "old-timer's" code here would actually be, these kinds of objections make no sense at all. Assuming the code to run nmap itself would be equivalent either way and we're interested in the data storage and analysis functions here, the old-timer would write that entire part of the program in about one screen of any decent programming language in perhaps 10 minutes. The functions would be short and simple, needing only basic iteration and bitwise arithmetic. Any junior programmer who's going to get anywhere in this industry would be able to understand that code in moments with no specialist knowledge or additional training. Nothing about the code would get in the way of any reasonable commenting or testing policy either. If in the future someone didn't find the compact data structure appropriate for some new application, they could easily convert the data to a more suitable alternative format, because the current format would be well-specified, simple, efficient, and without external dependencies. Hypothetical arguments about scalability are silly. The problem is fundamentally built around IPv4 addresses, which have been 32 bits wide since they were devised and will still be 32 bits wide tomorrow and next year. Designing for something more scalable is some horrible combination of scope creep, YAGNI violation, and worst of all, not giving even cursory thought to what the requirements actually mean. (I await the seemingly inevitable unintentionally amusing response about IPv6...) The only thing I really quibble with here is the characterisation of the two types of developer. I don't think this is really about modern vs. old-timer. It's just about a good programmer -- who looks at each problem on its merits, chooses suitable tools for the job, and leaves their options open -- and the bad programmer, who does not. Well, that and the fact that no self-respecting old-timer would misuse the word "performant" so heinously, but I digress. :-)
- GFK_of_xmaspast 11y ago
  There's absolutely a scalability problem lurking there regarding ports, and what happens when you want to scan more than 21 of them.
  - Silhouette 11y ago
doomrobo 11y ago
If anybody is curious, the zmap project from the University of Michigan does this and more. https://zmap.io https://zmap.io
return0 11y ago
Did this all happen while they were in the bar? Seriously though, both stereotypes are off-base (Go? really?)
hosay123 11y ago
> Old-Timer Developer: > I will use [a language with mandatory garbage collection] I lol'd
hendler 11y ago
There's no simple solution without a lot of context. http://mockingeye.com/a-classic-of-soviet-engineering/ http://mockingeye.com/a-classic-of-soviet-engineering/
GFK_of_xmaspast 11y ago
My background is much closer to the "old timer" but I think I'm definitely on the side of modernity here, along with everybody shouting "sql". Also I don't see why the old timer needs both a bit array and a uint64 port array, they can put the up/down bit in the high bit of each uint64.
yyin 11y ago
One could differentiate the two proposed solutions by the level of involvement of third parties (e.g., cloud hosting providers, authors of garbage collected or scripting languages, etc.). Some of these third parties have been involved since "old times", others have not.
Manishearth 11y ago
This just compares a decent programmer with a terrible one. Not a modern app developer with an old timer. This is especially evident where the systems programmer is aware enough to use Go (which is a debatable choice, but certainly displays good awareness), but the modern app dev doesn't know when (not) to use Big Data and when to try optimizing things (e.g. the JSON choice). The only "mistake" I see a modern app developer making might be using Python, but hey, computers are fast (so python should still be fine). Most would probably come up with a similar bit-twiddly solution or a simple DB-based solution. No biggie. There's no dearth of horrible systems code out there either. It might be less likely for a systems programmer (even a horrible one) to mess up this exercise, but you could probably choose an exercise which would have the reverse effect. In the end, it's a toy exercise, not one where you actually design a significant piece of software. There is a point buried in all of this; which is that learning systems programming will probably make you a better modern app developer since you get the correct mindset to tackle problems like this (also, vice versa?). But there's too much hyperbole obscuring it. There's also the other point about how systems programming is done vs modern app dev. It's a valid one, but there are benefits to both approaches, and it boils down to each being useful in its own domain.
iofj 11y ago
Am I the only one who thinks that there are far more efficient data structures than a bit array to represent this data ? How about an RLE encoded list of hosts, for instance ? (since they're consecutive 32-bit integers). There'd be way less data than in a bitfield which will make most of these queries far faster than iterative bitfield lookups. Also, much more of the data would fit and stay in memory, which means that all queries that iterate over it will be 10x faster or maybe more. Of course experimenting with data structures is not something you can do very efficiently in Go, as it'll be painfully verbose code. C++ would be far more useful. But this is basically the old argument for/against optimizing code. The real problem with the "old timer" programmer is that there are quick ways to break his program. When new queries present themselves, the "old timer" will quickly find his datastructures not optimized for the queries, or that they require complex calculations, which means that for data analysis the modern app developer will probably "win". When it comes to putting a product in production, it needs to be fast and cheap. Any company that doesn't hire an old-timer developer for that will quickly find their costs exploding. This may be acceptable for a few weeks when trying to find product/market fit but it won't last long.
na85 11y ago
Based on my recent experiences, the modern app developer would write a slow, bloated web app that takes ages to produce a result but has a slick UI with awesome animations.
ryandrake 11y ago
Couple of defensive, bruised egos here in the responses. Lighten up, it's funny! We all know that one guy for whom the answer to every question is HADOOP MAP REDUCE!! I thought the writing was fun.
z3t4 11y ago
I think this is unfair because it's basically in the old-school systems domain of problems.

148 points by zhenjl 11y ago

I'm betting on the modern app developers system being much easier to maintain and understand ( and pass off to junior programmers), at the cost of speed.

err4nt 11y ago
I don't know about that. I'm a front-end developer, so I know none of the above. I'd rather decipher some complex bit-counting scheme one person made (and documented) in a language I don't write, than to try to piece together a whole system of tools written by various people I can't talk to, and tied together by one person for this specific task. Which one is going to require more reading? Which one is going to be easier to refactor? Although neither develop sounds like a teammate I'd hope to collaborate with, I'd much rather inherit Mr Old-Timer's code to maintain than Mr Modern Developer
merpnderp 11y ago
I was going to say no way, basic boolean logic is doable by every programmer. But then I saw the "old-timer" had some boolean errors themself. So......Maybe you're right.

nobullet 11y ago

Is this post Go biased? Why not C++ or Erlang? :) You can do the same with Python

bottled_poe 11y ago
Agreed, I think very few "Old-timers" would be using go.
x1024 11y ago
Looking at the other posts in the blog, it's definitely Go biased. Not that it's necessarily a bad thing. But it's making some seriously tenuous assumptions about "modern app developers". It's almost "Goofus & Gallant" levels of hyperbole.

hellcow 11y ago

As VCs often look for keywords, I suppose the trick is to build the old-timer system but describe it as the modern web developer.

brianwawok 11y ago
Lol store the bits in a file called hadoop.txt. if anyone asks how you did a query say you looked in hadoop.

kogepathic 11y ago

Why wouldn't you want to use an SQL database with an index to store this data? Just have a column for the ports and a boolean flag for the status of the port? Honestly, I'd probably use bash/nmap/ping and psql to insert data. Want to query? psql and grep. These examples seem like a great way to re-invent the wheel with modern buzzwords.

Spoom 11y ago
I was thinking this as well. This is a problem with a well-defined schema. Why would you use a schemaless NoSQL database? Additionally, many of the bitwise operations used by the old school dev have analogous abstractions in SQL, which are probably easier to understand.
hackercomplex 11y ago
Why not SQLite ?
- hendler 11y ago
  or redis
- Sanddancer 11y ago
  I'd just take that as favorite tool for the job, SQLite's pretty interchangeable. Also, if you've already gotten a database engine running, it's probably best to just keep everything there. Otherwise, you get into the situation I was in at a previous job in which certain page loads from an app written before I started there required a sun jsp app for the main part of the page, which made requests to pull some data from an Oracle database, some data from an apache mod_perl app that queried from a mysql database, some data from an apache mod_jk app that pulled from a c++ app with its own database format, and some data from an apache mod_jk app that stored and retrieved from a postgresql instance. Everything was nice and well-documented, but mein gott was that a lot of moving parts.
- skrause 11y ago
  In PostgreSQL you could use a native type to store your IP addresses and subnets (even for IPv6): http://www.postgresql.org/docs/9.5/static/datatype-net-types.html http://www.postgresql.org/docs/9.5/static/datatype-net-types...

bottled_poe 11y ago

Maybe this is a stupid question, but why not just use a relational database?

x1024 11y ago
Because then it would seem that modern app developers are actually competent, and that's not the author's intent.

opticalfiber 11y ago

I'm not really sure what the point of this article is. If you're clever, you can represent any data as a bit array. And once you're there, counting bits or XORing them together is easy. But is that system easy to understand? Is the code easy to read for the rest of the developers who work on the project? Or for future hires? What happens if the data set becomes too large to fit in memory on a single machine? Storage is cheap and getting cheaper every day. The whole data set described in this problem (done the "inefficient" way) fits on a flash drive at today's capacities. There are always tradeoffs when it comes to architecture design. Speed and storage space are only two of the many factors that require consideration.

amorphic 11y ago
+1 Ill add that having your data in Elastic means that a (probably non-technical) user still can use it as part of their analysis without having to get dev involved.
- opticalfiber 11y ago
  Yes! ES is great for a lot of reasons, not the least of which is the one you point out.
jdmoreira 11y ago
"if the data set becomes too large to fit in memory on a single machine?" You just use mmap. If it become bigger than the disk capacity of single machine, you can still mount a distributed filesystem and use mmap.
- voltagex_ 11y ago
  Slightly off topic: http://yourdatafitsinram.com/ http://yourdatafitsinram.com/ and more seriously https://www.youtube.com/watch?v=SiSY1b0am5w https://www.youtube.com/watch?v=SiSY1b0am5w which is a really good talk about the challenges of sysadmining for scientists. What do you do when 2TB of RAM isn't enough?

the_cat_kittles 11y ago

basically just an example of how choosing a bad data structure can make your life a headache

olalonde 11y ago

Business Guy shakes his head and gets a subscription on https://www.shodan.io/ https://www.shodan.io/

chrismorgan 11y ago
And that is so often the best kind of response.
merpnderp 11y ago
I was afraid I was going to miss the joke until I clicked the link and got a real chuckle.

smoyer 11y ago

"I can perform a simple AND operation on the 3 monthly bit arrays, and then count the number of “1” bits." I think an old-timer would tell you that he'd be using an "OR" operation to count the number of hosts seen in the last three months. As a long-time embedded systems engineer, you learn to guard every CPU cycle and memory location jealously. That behavior, along with avoiding needless abstractions that complicate the system AND cost CPU and memory, have served me well as I've switched to higher-level software. As an aside, memory resident databases are now practical even for problems of this size. I've used the CERN Colt library to provide sparse arrays stored in memory in place of databases several times now. My current pet project is processing a mouse genome for my daughter's lab. With the amount of data being read, I'd always be I/O bound if I wasn't packing a lot of working data into memory. This is the programmers equivalent of the CPU's cache, but one we can easily control. Give it a try!

Liru 11y ago
It may be because I haven't had my coffee yet, but how would using OR help here? 1 OR 0 OR 0 returns 1, but the host was offline in 2 of the past 3 months, which means that it should be 0...
- smoyer 11y ago
  You might be right ... but it's entirely dependent upon what the author meant. The problem statement was "How Many Total Hosts Were Seen as “Up” in the Past 3 Months" which I read as "How Many Total Hosts Were Seen as “Up” in any of the Past 3 Months". You've correctly pointed out that the author might have meant "How Many Total Hosts Were Seen as “Up” in all of the Past 3 Months". This sort of ambiguity is why BDUF is so hard to pull off. Only lawyers are trained to write completely unambiguous prose (and they fall back to having very precise meanings for words to help them succeed).
  - sokoloff 11y ago
    Given that the modern app developer did a de-dupe operation for his solution, and that the most plain reading of the English text is "in any of the scans", I think the OR is correct.

ComSubVie 11y ago

That's actually a good comparison between different mindsets. Even if it reads a bit biased pro-go (or pro-old-time-system-developer) it doesn't tell what's the best way to implement this - and in my opinion it's neither one. So what's actually the best way to implement this? What are the motivations for choosing one way? "Modern App Developer Way": + adoptable, scalable, readable data formats - storage, computation "Old-Time System Developer Way": + storage, computation - "locked-in" data formats "Database Developer Way": + ease of implementation ? storage, computation Any other options/ideas?

sklivvz1971 11y ago
Use Redis?

VLM 11y ago

Funny quote from the article: "This is a big data problem." Hmm maybe on a raspberry pi, or some kind of retrocomputing challenge (Do this on a simulated IBM 1130!). I don't know how an app developer thinks, but I know the article characterization of the old developer is wrong, we'd chuck it into the existing relational DB, maybe spin up a new cloud instance and reserve some NAS space if necessary but this is a pretty small data set by modern standards so probably nothing special is required. All the reporting devolves into silly SQL onliner competition. The first question is a COUNT(*) and GROUP BY. The second question is a ridiculously simple SELECT. The third question is another simple select if you stored your ip addr bytes in separate columns, even if you allow non /24 addrs. The fourth is another GROUP BY. The article does fit the stereotype I've seen that new programmers prioritize ease of storing data over ease of reporting, and old timers vice versa. Like the difference between coming up with the fastest next move in checkers vs "solving" checkers in the game theory sense.

smarx007 11y ago

Old-time system developers use C.

dd9990 11y ago
And likely C89... certainly not Go.
pskocik 11y ago
Wouldn't you classify golang authors as old-time system developers?
Silhouette 11y ago
Old-time system developers wrote Go.

dzhiurgis 11y ago

Old timer spends rest of 2016 to implement his solution that saves $100 in computing resources.

Silhouette 11y ago
Reality: OId timer spends about 10 minutes to implement his solution. If the older timer's solution does save $100 in computing resources, then at that rate the the old timer is saving around $1M per year compared to the "modern" developer.
- dzhiurgis 11y ago
  Given both are experts, point-and-click will always be faster to implement.

szc 11y ago

A couple of errors near the end for "How Many Hosts Were “Up” Last Month But Now It’s “Down”" and vice versa. The error is that using xor on the same item twice is a no-op... (A xor B) xor A == B. The correct equations would be, !this_month and last_month = "up" last month, down now. and this_month and !last_month = "down" last month, up now.

raesene2 11y ago

Pentester: I'll use mass scan, someone else already solved this problem. https://github.com/robertdavidgraham/masscan https://github.com/robertdavidgraham/masscan

tyre 11y ago

I suppose I'm a modern app developer, so here's what I would actually do: Postgres. 1) I'm not going to invent my own system. I am not the first person who wants to store addresses and boolean values. If I'm looking at this occasionally over many months, I don't want to keep relearning my elegant bit-structure. 2) SQL is great, Postgres is even better with built-in network operators[1] for the more specific queries (e.g. subnets.) 3) Bitfields are all fun and games until you want to change things. I'm already having flashbacks to rails `has_bitfield` columns. Also, fewer people can confidently twiddle bits than can understand boolean SQL columns. 4) The author mentions possible compression methods. I'd rather a tested project like Postgres think about everything to do with storing my data. I understand the criticism, but with cloud storage so cheap it feels like optimization just to show off. I'd rather save time than bits. If there is a satire that involves me using Postgres when I shouldn't, that would be welcome. [1] http://www.postgresql.org/docs/current/static/functions-net.html http://www.postgresql.org/docs/current/static/functions-net....

krick 11y ago
Exactly. Change "json" to "relational DB" and it's not close as ridiculous as author implies. All questions mentioned could be answered with a single SQL: although it will take quite a while. And it is much more extendable than this "huge array" approach. The only thing is that by saying "Postgres" you say pretty much nothing: the real question here is how'd you structure the db. 300M a month is pretty huge, so it's really worth to think if we should just pile everything in 1 table (unique key ip+month, all ports are columns) or if we can denormalize something.
- GFK_of_xmaspast 11y ago
  300M per month doesn't sound like all that much to me?
  - krick 11y ago
    Well, it will be quite a table to do your selects against. Assuming that your hardware for such a project is fairly modest, after a couple of months index won't really fit into the memory, and even selecting a record by the key would take 30 jumps over that table just after 4 months. And most of the tasks in question require running through the whole thing. Not a tragedy, but well worth considering aggregating some data as it comes, especially as it costs pretty much nothing at this point (nmap takes longer anyway). Like counting number of IPs with the open port 22 would be just incrementing some number at the dedicated table by simple rule in the code. Maybe it's worth to invent some more interesting denormalization in order to be able to run more complex queries easily. Maybe I'm wrong: I'd have to try out that to see how fast it actually computes, but I remember having problems with much smaller tables: like these 300M in total, maybe. Although, it was much "wider", with varchars and stuff.

swinglock 11y ago

Unfortunately our hero the array developer got a bit too clever, pun intended, and produced an incorrect program. The last two of the XOR tricks won't work the way he thought.

andybak 11y ago

Ah. Thank god the imaginary old school guy is better than the imaginary straw man modern guy! I'd better go learn Go immediately. Personally I'd start off by looking into using Cython or Numpy and maybe pickle to disk for storage. Does that make me the old guy or the new guy or is it a false dichotomy?

tyingq 11y ago

Lost me at "Old-Timer Developer: I will use Go." The old-timers in your vicinity must be different than the ones in mine.

Namrog84 11y ago
I think we might come from a similar vicinities. Here I am thinking go was modern. I expected old timer to be using c. As that's probably what I would have done
- tyingq 11y ago
  "What's this Go? C for people that are afraid of malloc() ?"
emmelaich 11y ago
I think Go is an acceptable modern language for lovers of C. Especially those who are allergic to C++.
- snaky 11y ago
  Go is an acceptable modern language for modern haters of C

asragab 11y ago

As a relatively junior developer, I have been told my some of my senior colleagues that I sometimes miss the forest for the trees, and so this may be another instance of that. But it doesn't seem "trivial" to me how one gets the output of nmap into whatever database, data structure one chooses here. I know nmap can produce XML, presumably there is a csv format, but it would seem like the XML/CSV -> JSON conversion (following our intrepid Modern App Developer) would be an easier more maintainable way to go, versus XML -> to bit array (memory map file). Also, is managing the nmap or masscan and whatever other ancillary processes required to execute this plan equally as onerous in either paradigm? Finally, and this is likely controversial, this particular problem "feels" like its stacked against the Modern App Developer, given that it isn't trying to solve a problem most Modern Apps try to solve (or try to solve as an end rather than a means to an end)

deleted 11y ago
[deleted]
int_handler 11y ago
Or, if you want to use a binary format without having to deal with all of the complexities of inventing your own, you can also use Protocol Buffers. You can even use Protocol Buffers as a schema for JSON too as of 3.0.

slightlycuban 11y ago

Point of this article is not the point of this article. The clever bit was: > ...the IPv4 address will convert into a number... Many fancy, post-modern app developers might insist, "You won't do math with an address; it's not a number." But some things are numbers, with a scheme and pattern you can exploit.

Silhouette 11y ago

It's fascinating how many people here are defending the "modern app developer" approach, mostly with arguments about flexibility, maintainability, ability to pass on the code to junior developers, and the like. If you think about what the "old-timer's" code here would actually be, these kinds of objections make no sense at all. Assuming the code to run nmap itself would be equivalent either way and we're interested in the data storage and analysis functions here, the old-timer would write that entire part of the program in about one screen of any decent programming language in perhaps 10 minutes. The functions would be short and simple, needing only basic iteration and bitwise arithmetic. Any junior programmer who's going to get anywhere in this industry would be able to understand that code in moments with no specialist knowledge or additional training. Nothing about the code would get in the way of any reasonable commenting or testing policy either. If in the future someone didn't find the compact data structure appropriate for some new application, they could easily convert the data to a more suitable alternative format, because the current format would be well-specified, simple, efficient, and without external dependencies. Hypothetical arguments about scalability are silly. The problem is fundamentally built around IPv4 addresses, which have been 32 bits wide since they were devised and will still be 32 bits wide tomorrow and next year. Designing for something more scalable is some horrible combination of scope creep, YAGNI violation, and worst of all, not giving even cursory thought to what the requirements actually mean. (I await the seemingly inevitable unintentionally amusing response about IPv6...) The only thing I really quibble with here is the characterisation of the two types of developer. I don't think this is really about modern vs. old-timer. It's just about a good programmer -- who looks at each problem on its merits, chooses suitable tools for the job, and leaves their options open -- and the bad programmer, who does not. Well, that and the fact that no self-respecting old-timer would misuse the word "performant" so heinously, but I digress. :-)

GFK_of_xmaspast 11y ago
There's absolutely a scalability problem lurking there regarding ports, and what happens when you want to scan more than 21 of them.
- Silhouette 11y ago

doomrobo 11y ago

If anybody is curious, the zmap project from the University of Michigan does this and more. https://zmap.io https://zmap.io

return0 11y ago

Did this all happen while they were in the bar? Seriously though, both stereotypes are off-base (Go? really?)

hosay123 11y ago

> Old-Timer Developer: > I will use [a language with mandatory garbage collection] I lol'd

hendler 11y ago

There's no simple solution without a lot of context. http://mockingeye.com/a-classic-of-soviet-engineering/ http://mockingeye.com/a-classic-of-soviet-engineering/

GFK_of_xmaspast 11y ago

My background is much closer to the "old timer" but I think I'm definitely on the side of modernity here, along with everybody shouting "sql". Also I don't see why the old timer needs both a bit array and a uint64 port array, they can put the up/down bit in the high bit of each uint64.

yyin 11y ago

One could differentiate the two proposed solutions by the level of involvement of third parties (e.g., cloud hosting providers, authors of garbage collected or scripting languages, etc.). Some of these third parties have been involved since "old times", others have not.

Manishearth 11y ago

This just compares a decent programmer with a terrible one. Not a modern app developer with an old timer. This is especially evident where the systems programmer is aware enough to use Go (which is a debatable choice, but certainly displays good awareness), but the modern app dev doesn't know when (not) to use Big Data and when to try optimizing things (e.g. the JSON choice). The only "mistake" I see a modern app developer making might be using Python, but hey, computers are fast (so python should still be fine). Most would probably come up with a similar bit-twiddly solution or a simple DB-based solution. No biggie. There's no dearth of horrible systems code out there either. It might be less likely for a systems programmer (even a horrible one) to mess up this exercise, but you could probably choose an exercise which would have the reverse effect. In the end, it's a toy exercise, not one where you actually design a significant piece of software. There is a point buried in all of this; which is that learning systems programming will probably make you a better modern app developer since you get the correct mindset to tackle problems like this (also, vice versa?). But there's too much hyperbole obscuring it. There's also the other point about how systems programming is done vs modern app dev. It's a valid one, but there are benefits to both approaches, and it boils down to each being useful in its own domain.

iofj 11y ago

Am I the only one who thinks that there are far more efficient data structures than a bit array to represent this data ? How about an RLE encoded list of hosts, for instance ? (since they're consecutive 32-bit integers). There'd be way less data than in a bitfield which will make most of these queries far faster than iterative bitfield lookups. Also, much more of the data would fit and stay in memory, which means that all queries that iterate over it will be 10x faster or maybe more. Of course experimenting with data structures is not something you can do very efficiently in Go, as it'll be painfully verbose code. C++ would be far more useful. But this is basically the old argument for/against optimizing code. The real problem with the "old timer" programmer is that there are quick ways to break his program. When new queries present themselves, the "old timer" will quickly find his datastructures not optimized for the queries, or that they require complex calculations, which means that for data analysis the modern app developer will probably "win". When it comes to putting a product in production, it needs to be fast and cheap. Any company that doesn't hire an old-timer developer for that will quickly find their costs exploding. This may be acceptable for a few weeks when trying to find product/market fit but it won't last long.

na85 11y ago

Based on my recent experiences, the modern app developer would write a slow, bloated web app that takes ages to produce a result but has a slick UI with awesome animations.

ryandrake 11y ago

Couple of defensive, bruised egos here in the responses. Lighten up, it's funny! We all know that one guy for whom the answer to every question is HADOOP MAP REDUCE!! I thought the writing was fun.

z3t4 11y ago

I think this is unfair because it's basically in the old-school systems domain of problems.