3 ms·

I know splunk is expensive, especially if you don't have somebody who is actively monitoring your spend, but sumo is a pretty good alternative. It's actually so

I know splunk is expensive, especially if you don't have somebody who is actively monitoring your spend, but sumo is a pretty good alternative. It's actually something I look for in the interview. Regardless, many companies underspend by an order of magnitude on dev tools. If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it (multiply this by engineers at the company who need the tool). Soon you'll see that a tool costing hundreds of thousands to license a year is often a steal. So you may say "oh splunk is overkill and unfair expectation" but in their CI/CD system, if your job failed (frequent occurrence) and you wanted to search for something in there you'd manually DOWNLOAD a 300 meg file to your machine and grep it for errors. Basically a fang engineer's time costs $3 a minute minimum, so if a query could save 1 minute then the break-even cost is paying up to $3 to run that query.

taeric 3y ago
Even in a fang, most money spent on log storage is fully wasted. The article yesterday about monitoring being a pain was spot on. More amusing is when you see someone build a giant Elastic Search pipeline to a Kibana dashboard so that they can get what would have been table stakes metrics if they had used the normal service templates. Without a ridiculously large bill. Or folks that think they can warp around the high cardinality traps of making a metric out of everything. Assuming if you can make a system that works for the testing environment, of course it will work when you open the floodgates. Seen people argue that sampling shouldn't be used, "because you could miss data?" Reservoir sampling is a thing, for very real reasons. At any rate, I don't mean to just yell about splunk, from all I've heard it is nice. I am annoyed that folks seem to ignore the OLAP and OLTP divide, such that they think your metrics system should somehow be optimized for both. At the same time. Pulling back to builds, though. What, exactly, is the gold standard in industry? I have yet to see it. Python builds, in particular, strike me as not good.
Hermitian909 3y ago
> If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it Hear me, nameless internet stranger, that you might learn from my mistakes: This is not true. Efficiency is only worth money if it increases profits - concretely this means your efficiency gain must result in the following: 1. Delivering features faster 2. Delivering features with meaningfully higher quality 3. Delivering the same features with lower headcount costs (2) is hard to measure, so you can generally only sell on (1) and (3). During the 80s the business world learned the hard way you can throw away a lot of money on useless efficiency. If that interests you I recommend "The Goal" by Goldratt for a fictionalized account of those learnings. * There's also a subtler point that corporate finances may mean that even if efficiency is perfectly captured, a 1% efficiency increase may only be worth 0.1% of your salary.
- rsdbdr203 3y ago
  100% this... profit, or it's not worth it
Spooky23 3y ago
Splunk can save money, as long as you’re intelligent about processing data. I can think of dozens of security and operational incidents where splunk saved the day. I’ve been through a few savings exercises where everything is gonna go open source dashboards etc. Once the guy who understands how it all works disappears, it migrates back to splunk. One security incident response will pay for the entire splunk infrastructure.
- rsdbdr203 3y ago
  This isn't a great argument... sure, Splunk "saved the day" because it caught that really bad security thing... but what other tools, at half the cost, could have also "saved the day"? If most of them could have, then it's not really worth the price... right?
  - Spooky23 3y ago
    Sure. I can save a ton of money by fixing my own car too - if I have the ability to execute. If you can, that seriously awesome. The point is, many can’t.

taeric 3y ago

Even in a fang, most money spent on log storage is fully wasted. The article yesterday about monitoring being a pain was spot on. More amusing is when you see someone build a giant Elastic Search pipeline to a Kibana dashboard so that they can get what would have been table stakes metrics if they had used the normal service templates. Without a ridiculously large bill. Or folks that think they can warp around the high cardinality traps of making a metric out of everything. Assuming if you can make a system that works for the testing environment, of course it will work when you open the floodgates. Seen people argue that sampling shouldn't be used, "because you could miss data?" Reservoir sampling is a thing, for very real reasons. At any rate, I don't mean to just yell about splunk, from all I've heard it is nice. I am annoyed that folks seem to ignore the OLAP and OLTP divide, such that they think your metrics system should somehow be optimized for both. At the same time. Pulling back to builds, though. What, exactly, is the gold standard in industry? I have yet to see it. Python builds, in particular, strike me as not good.

Hermitian909 3y ago

> If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it Hear me, nameless internet stranger, that you might learn from my mistakes: This is not true. Efficiency is only worth money if it increases profits - concretely this means your efficiency gain must result in the following: 1. Delivering features faster 2. Delivering features with meaningfully higher quality 3. Delivering the same features with lower headcount costs (2) is hard to measure, so you can generally only sell on (1) and (3). During the 80s the business world learned the hard way you can throw away a lot of money on useless efficiency. If that interests you I recommend "The Goal" by Goldratt for a fictionalized account of those learnings. * There's also a subtler point that corporate finances may mean that even if efficiency is perfectly captured, a 1% efficiency increase may only be worth 0.1% of your salary.

rsdbdr203 3y ago
100% this... profit, or it's not worth it

Spooky23 3y ago

Splunk can save money, as long as you’re intelligent about processing data. I can think of dozens of security and operational incidents where splunk saved the day. I’ve been through a few savings exercises where everything is gonna go open source dashboards etc. Once the guy who understands how it all works disappears, it migrates back to splunk. One security incident response will pay for the entire splunk infrastructure.

rsdbdr203 3y ago
This isn't a great argument... sure, Splunk "saved the day" because it caught that really bad security thing... but what other tools, at half the cost, could have also "saved the day"? If most of them could have, then it's not really worth the price... right?
- Spooky23 3y ago
  Sure. I can save a ton of money by fixing my own car too - if I have the ability to execute. If you can, that seriously awesome. The point is, many can’t.