6 ms·

"I am frustrated by the industry as a whole" Unfortunately I have to agree as a developer. My job is to make a fast, reliable, stable product but at the same t

• collyw 10y ago
  As long as the database isn't relational, I guess.
  • kasey_junk 10y ago
    10M rows in a relational database is a very low number (depending on the size of the row of course).
    • collyw 10y ago
      I know, it was a sarcastic follow up to the "now they have big data" part of the original comment. "SQL doesn't scale". It needs to be in Mongo or whatever NoSQl database is in right now. I have heard all sorts of nonsense regarding "big data" in the last few years.
      • chewbacha 10y ago
        ahaha, i didn't read the sarcasm that time, sorry for replying with tmi
  • chewbacha 10y ago
    10 million rows in a relational database doesn't need to be bad nor is it big data. Rows is a bad measure of "big" when it comes to data. A measurement of bytes and probably more specifically bytes per field and how many fields the records have, as this gives a better indication into the way this will be written and potentially searched. 10 million rows of 5 integer values is pittance for any relational database worth using in production. 10 million rows of 250 text columns would be horrendous for a relational database.
    • mseebach 10y ago
      Someone once suggested to me that 'big data' begins when it doesn't fit in RAM in a single rack any more.
      • jules 10y ago
        That's soon going to be on the order of 100 terabytes, so there will be only a handful of companies doing big data ;-)
        • imtringued 10y ago
          I'm only aware of servers up to 12TB. Care to elaborate?
          • jules 10y ago
            He/she said a whole rack of servers. I actually took 30 servers of 2TB each and rounded up to 100. With 12TB per server it will already be over that.
      • chewbacha 10y ago
        Yup, that's essentially what looking that byte-size means. However, just because it doesn't fit in memory might not make it big data if it's just poorly engineered. But many times this happens because of wasted or bloated indexes that aren't useful. Or it happens when data types are picked incorrectly. For example, I once worked on a database where the original developer used Decimal(23, 0) as a primary key. This was on MySql and that ended up taking up 11 bytes per row, versus a Long which would have just been 8. In one table, maybe not so bad but when you start putting those primary keys into foreign key relationships... we ended up with a 1 billion row table in MySql that had 4 of these columns in it. That might make it "big data" by that definition but it's also just bad design. Another example in that same database was using text fields in mysql for storing JSON. Since text fields in mysql are stored as separate files, this meant that every table that had one (and we had several tables that housed multiple) ran in large IO and disk access issues. "big" data is probably a bad term to use these days because of easy it is to accidentally create a large volume of data but not need a big data solution outside of the fact that it's not the business that needs it, it's the poorly implemented system that does. But the real reason we talk about fitting in memory comes from the core of the issue: IO. Even a super large memory set could end up being slow if it's postgres and single threaded reader that's scanning a 500 GB index. AWS offers up to 60 GB/s memory bandwidth and we'd need it for this index, since that would still take almost 10 seconds to warm up the indexes in the first place.
• koolba 10y ago
  > But sometimes it's also very easy to please people. Big data: just insert 10M records in a database and suddenly everyone is happy because they now have big data :| Since when is 10M records is considered big data? My goto gauge for big data is that it can't fit in memory on a single machine. And since that means multiple TB[1] these days, most people don't really have big data. [1]: *Heck you can even rent ~2TB for $14/hour! https://aws.amazon.com/ec2/instance-types/x1/ https://aws.amazon.com/ec2/instance-types/x1/
  • sigzero 10y ago
    I think that is why he had the :| face at the end.
    • koolba 10y ago
      Oh haha. I thought that was a typo!
  • heartbreak 10y ago
    Yeah that's everybody's gauge if they actually work with it, which was the point.
  • FreeFull 10y ago
    I'd say if it can't fit in RAM, but still can fit on a single SSD it doesn't count as big data either
    • TylerE 10y ago
      Not sure how accurate that is, since you can buy 60TB SSDs these days.
      • tedunangst 10y ago
        ergo, not big data.
  • eitally 10y ago
• 6DM 10y ago
  On a positive note, it sounds like proprosals to use newer technology are welcome. I keep seeing the opposite, "No this is too different, could break stuff."
  • collyw 10y ago
    If it ain't broken, why fix it?
    • 6DM 10y ago
      In my case, usually something is broken or breaking in production frequently enough to warrent some changes. Plus, there are other reasons you can make a change even though it's not broken. Sometimes it can make you more productive. Or though your site is still responding to current customer demands in a timely fashion, you know that the mobile experience could be significantly improved now that browsing via cell phone is on the rise. Another thing to consider is employability both from a company and individual perspective. If you can keep up with moderately current (not the latest and greatest) trends, you'll attract people who want to grow in their careers. I wouldn't want to work on C# 2.0 using Visual Source Safe. It's hard to convince a company that you can learn git on the job. In general I like to move without introducing breaking changes. I'm not a cowboy coder, it's really exhausting working with one. I do think there's merit in realizing when it's time to change though.
  • CaptSpify 10y ago
    IME that comes with: Sure, the new tool looks cool, but is it battle-tested? How many tools end up being relied on heavily while they're still in beta? And does it solve any of our current problems? or is it just neat? As a grumpy SA, I see way too many people try to push for new tools because they "seem cool", instead of "Do they solve a problem we have?"
    • lj3 10y ago
      It took me years to realize the reason programmers do this is because the tools that "seem cool" make their lives easier at the expense of everything else. This is where the popular traits of "laziness and hubris" become a liability instead of an asset. More programmers need to embrace the suck.
• Waterluvian 10y ago
  These discussions are perfect examples of why building good social skills can be more important than learning the next greatest programming language 5.0.
  • fl0wenol 10y ago
    I love you for saying this because it needs to be said. Thank you.