15 ms·

Can I have a smaller Prometheus

142 points by wejick 5y ago

deleted 5y ago
[deleted]
whateveracct 5y ago
Clutching pearls about binary size is and always will be hilarious to me.
- yodon 5y ago
  And oh what a beautiful bike shed it will be...
- jeffbee 5y ago
  Author doesn't even say why they object to the size. Are they aware that file-backed executables are paged on demand and only the active parts of the program will be resident?
  - wejick 5y ago
    sorry not to make it obvious in the article, I'm planning to run it in small iot pi based device locally. So having something small and fast is preferable, however the runtime performance is a more important thing I haven't touch.
    - ts4z 5y ago
      I'm curious -- is it the binary size that's a problem, or the resident size in memory? Demand paging should help, although you'd be stuck with carrying the enlarged binary.
      - wejick 5y ago
        my gut feeling tell it will be both memory and cpu utilization. cant be sure, until I can find good way to measure it.
        SuperQue 5y ago
        If only there was a monitoring system you could install to measure such a thing.
        ts4z 5y ago
        ps RSS measurement is pretty good for a start (although note in a forked process, shared copy-on-write pages are both reported as RSS for both the parent and child process). top reports this as RES. IIRC, debugging information is in a separate part of the process, so it's not loaded until it's used. Does that make it free? Probably not quite, but the kernel can ignore it until the process (presumably via its debugger) looks at it.
    - jeppesen-io 5y ago
      Prometheus is rather efficient, but it's focus is a little different than yours. Its designed to for large scale collection of metrics, scraped from many remote endpoints You can run it locally but the "prometheus" way for iot env would be a central prometheus server that scrapes the iot devices running a prometheus exporter, which tend to be very light weight
      - wejick 5y ago
        Totally agree. Another part of it is just feeding curiosities.
    - deleted 5y ago
      [deleted]
    - SuperQue 5y ago
      Prometheus works fine as-is on Pi devices. You'll spend most of your memory on ingestion buffering. I did the same tests as you did a while back, it only saves like 25-50MiB of memory IIRC. The only thing you really need to worry about on a Pi is that the default kernels are still 32-bit, and are set to 2GiB kernel boundary. So you'll be limited to how much TSDB storage can be mmap'd unless you switch to a 64-bit kernel. You may want to consider agent mode on your IoT device, and stream the data to an external server/service. https://prometheus.io/docs/prometheus/latest/feature_flags/#prometheus-agent https://prometheus.io/docs/prometheus/latest/feature_flags/#...
      - wejick 5y ago
        That's a good insight. Thanks
  - Topgamer7 5y ago
    Granted these days everyone is used to applications consuming massive amounts of drive space. But perhaps they're using legacy hardware for a home lab, or a IoT device with limited disk space. From a security stand point, reduced application code decreases risk. It was service discovery code he removed, what if it reached out to discover services on application start up, that's a potential attack vector.
    - 8note 5y ago
      Does it actually reduce the risk? Sure if you audit, its easier to identify the risks, but a windows 98 program is going to be full of vulnerabilities while being small. Being small doesn't remove the vulnerabilities
    - shoo 5y ago
      > From a security stand point, reduced application code decreases risk. It was service discovery code he removed, what if it reached out to discover services on application start up, that's a potential attack vector. Agreed. I've see a similar pattern with certain open source libraries. The first example I think of is the spf13/viper [1] library, used to load configuration into go applications. Viper is equipped with code for reading config from various file formats, environment variables, as well as remote config sources such as etcd, consul. If you introduce the viper library as a dependency of your application to merely read config from environment variables and YAML files in the local filesystem, then your go application suddenly gains a bunch of transitive dependencies on modules related to remote config loading for various species of remote config provider. It's not uncommon for these kind of remote config loading dependencies to have security vulnerabilities. As well as the potential increased attack surface if a bunch of unnecessary code to load application configuration from all manner of remote config providers ends up in your application binary [2], if you work in an environment that monitors for vulnerabilities in open source dependencies, if you depend on an open source library that drags in dozens of transitive dependencies you don't really need, it adds a fair bit of additional overhead re: detecting, investigating and patching the potential vulnerabilities. I guess there's arguably a "Hickean" simple-vs-easy tradeoff in how such libraries are designed. The "easy" design, that makes it quick for developers to get started and achieve immediate success with a config loading library, is to include code to load config from all popular supported config sources into the default configuration of the library, reducing the amount of steps a new user has to do to get the library to work for their use case. A less easy but arguably "simpler" design might be to only include a common config-provider interface in the core module and push all config-provider-specific client/adaptor code into separate modules, and force the user to think about which config sources they want to read from and then manually add and integrate the dependencies for the corresponding modules that contain the additional code they want. edit: there has indeed been some discussion about the proliferation of dependencies, and what to do about them, in viper's issue tracker [3] [4] [1] https://github.com/spf13/viper https://github.com/spf13/viper [2] this may or may not actually happen, depending on which function calls you actually use and what the compiler figures out. If your application doesn't call any remote-config-provider library functions then you shouldn't expect to find any in your resulting application binary, even if the dependency is there at the coarser-grain module dependency level [3] https://github.com/spf13/viper/issues/887 https://github.com/spf13/viper/issues/887 [4] https://github.com/spf13/viper/issues/707 https://github.com/spf13/viper/issues/707
  - cosmotic 5y ago
    Image pull size for a container is likely the concern. It could shave a few seconds off a regularly-run integration test. If it's run via on-demand build agents, then there's no image cache.
    - jeffbee 5y ago
      If it takes multiple seconds to pull ~35MB of compressible text into your CI environment, there may be other, larger problems to solve.
      - cosmotic 5y ago
        I was estimating off the speed it takes to pull images to my local computer where the limiting factor appears to be something other than my internet connection so either the image extraction process or a docker hub throttle.
  - wahern 5y ago
    That only helps if the code is well segregated by usage. Looking at the ELF symbol table for prometheus-2.33.0-rc.1.linux-amd64, it's not clear to me this is the case. Not sure how it's ordered. Lexical import order? Anyhow, without profiling how could the compiler know how to order things optimally? I think this is one of those cases where, in the absence of profiling or some other hack (e.g. ensuring all routines within a library are cleanly segregated across page boundaries within the static binary and the I/O scheduler doesn't foil your intent), dynamic linking would prove superior, at least for such large amounts of code.
- 0xbadcafebee 5y ago
  It correlates to performance, speed to iterate, security, and design complexity, but ok
  - Zababa 5y ago
    Any hard data on any of that?
    - goodpoint 5y ago
      Unless a fat binary embeds pictures and some music, it's all CPU instructions. Tenths of megabytes of CPU instructions is complexity. This kind of bloat is the number one enemy of security, as any security engineer could confirm.
    - 0xbadcafebee 5y ago
      Sure, lots, go look for studies on estimation of defects based on LOC and project size/complexity (they go back to the 1970s). But you don't need to look, the principles are simple. Unless an application is filled with JPEGs or uncompressed arbitrary data files, its size reflects lines of code (machine code, interpreted code, etc). Bigger the app, the more lines of code. Every line of code has a non-zero bug probability. Every new line of code increases probability. More lines of code, higher probability. Bugs include security bugs; higher probability of bugs, higher probability of security bugs. CPU cache is finite. Only so many lines of code can be cached or optimized. Larger size takes up more room in memory, which when combined with lot of other gigantic apps, means less memory for heap space, disk cache, etc. Larger size also takes up more room on disk, which adds up when you don't delete old builds on disk and loop over a build process. Since larger size means more lines of code, that means longer compile times, which means longer wait every time you change a line and need to recompile, copy an artifact somewhere, retest. More lines of code means more code executed. If you have 10 lines of code in a function, and you add 100 lines to it, the compiler doesn't just optimize away all 100 new lines, it's going to add more machine code and code paths. Unless you only ever add new code paths, some of that new code will extend existing code paths or add instructions, and that means more CPU cycles to complete execution. (Same concept for interpreted code) More lines of code means more code paths. More code paths increases complexity. The more code paths, the longer and more difficult testing gets to the point you can't even develop enough tests to cover all the code paths, so it's impossible to even find all the bugs. More complexity leads to difficulty in humans understanding and working with the codebase, and difficulty in understanding leads to slower and more error-prone development. Larger means more network bandwidth, meaning file transfers take longer, increasing speed to iterate and producing worse UX. If people download your app every 10 minutes in their CI/CD pipeline, larger size means more network bandwidth used. "Free" CDNs have limits; the larger a project gets, the more file size affects network performance, reliability, and cost. If you pay for bandwidth, a 100MB file costs 100x more than a 1MB file. The more apps you use that are big, the more every one of these effects increase. One big app you might not notice. 100 big apps lead to noticeable slowness, bugs, less memory, less disk space.
- akireu 5y ago
  It's all fun and games until you're stuck for a hour downloading 600MB of updated packages over a metered LTE. The same is with RAM usage: 512MB was enough for a phone back in 2014, now a smart TV with 2GB is barely capable of multitasking. Sure, binary sizes don't matter in most contexts. But when they do, it's a PITA.
  - jayd16 5y ago
    Galaxy S5 from 2014 had 2GB and that was 1080p vs 4k texture sizes for today. Seems on par.
    - akireu 5y ago
      What you're kind of missing is that the S5 was a flagship phone. Generally, one has to save for more than a month to afford a purchase like that. The idea of working an extra month so that some FAANG prick meets their KPI by cutting corners on optimization doesn't even look like feudalism. It looks like idiocracy. Paying the lip service of fat shaming code bloat is the cost-effective option by comparison :)
      - asiachick 5y ago
        What does FAANG have to do with this? Don't FANNG people obsess over bloat because they're trying to reach billions of customers? It might not seem that way since their pages are bigger but I'd be surprised if they were happy to leave 10s of millions of customers on the table.
        ysleepy 5y ago
        FAANG are the worst offenders. Didn't facebook employ ungodly hacks to unload/load parts of the android app to navigate around the 65k method limit of dex? Have you looked at the js monstrosity of the Google hardware shop website?
        akireu 5y ago
        They're just poster children for the particular brand of disdain $100k+/year "tech workers" bear for their users: they make enough for the shiniest of toys, so they're too far above spending their valuable time to make their software run smooth on our $100 crap phones. Nevermind that each Fb client update likely produces hundreds of tons of toxic trash called gadgets. Sure, sometimes they do optimizations. Generally, though, both Fb and Google keep exploring the physical limits to code bloat. Remember that one time that Fb hit the JVM class count limit?
- xuhu 5y ago
  Auditd_2.8-amd64.deb is 194kb on debian, rsyslog_8.32-amd64 is 411kb, and they both support centralized auditing and log collection from multiple hosts.
  - djbusby 5y ago
    Do they do the metrics like Prometheus does? And include the central collector and basic graph builder?
  - dtech 5y ago
    This doesn't seem a fair comparison. Prometheus is statically linked like all Go applications, and those packages are not. You can debate the merits of that, but if you compare a "only rsyslog" server vs a "only prometheus" server the 2 will be much closer in size.
  - dralley 5y ago
    Even journalctl is only 90kb, and the entire combined systemd package is 4mb (but that's including all the documentation and a dozen other different binaries)
- sigmonsays 5y ago
  I also find this comical. I'd love to know why 100MB is that big of a deal. If network is slow, cache locally. Seems like nothing here to worry about.
  - deleted 5y ago
    [deleted]
- jeppesen-io 5y ago
  This one does not even make sense - 100 megs for a binary for centralized metrics? Who would even notice next to the OS and metrics storage. By design you should not install prometheus on every server you monitor - it's designed to scrape metrics Its a database, webui with support for email, webhooks, slack, pagerduty, aws api and many others. 100megs does not sound like a lot for all Pormetheus provides
- deleted 5y ago
  [deleted]
- deleted 5y ago
  [deleted]
- TillE 5y ago
  I work in a lot of situations with hard or soft resource limits where I actually do need to count bytes and/or CPU cycles, so it's bizarre to see anyone shrugging about distributing tens of megabytes of fat for literally no reason. One thing Microsoft got right a long time ago was separating out debug symbols into their own file by default. I think that's still awkward on Linux.
  - Arnavion 5y ago
    >I think that's still awkward on Linux. At least for .deb and .rpm packages, the default build process automatically extracts debug symbols into separate packages. Eg the process of building package `foo` also produces `foo-dbgsym` and `foo-debuginfo` packages respectively that contain debug symbols for every involved binary and library, while `foo` contains the stripped files. So anyone who wants to debug a coredump / live process just installs the corresponding -dbgsym / -debuginfo package and now gdb has all the debug info it needs. Distros have also started incorporating debuginfod into their repos so that gdb can download symbols automatically. So you don't even have to hunt for the right debuginfo package.
- foxfluff 5y ago
  It's kind of ironic reading this comment given that at the time you posted it, I was screaming at gcc's stupid code generator for wasting bytes recreating constants that were already there in that very register! That code needs to fit in a couple hundred bytes.. And half an hour ago I was (once again) checking out hosting providers and lamenting the fact that most don't seem to offer support for loading custom ISOs so I could install a 30 megabyte distro and make the most out of the cheap plans that only offer something like 10 gigabytes of storage. Half of it is wasted after you install one of the these obese mainstream distros.
  - mitjam 5y ago
    Hetzner Cloud can start instances from ISOs - here is an example for ipfire : https://wiki.ipfire.org/installation/hetzner-cloud https://wiki.ipfire.org/installation/hetzner-cloud
    - foxfluff 5y ago
      I got a VPS from Hetzner last year but they decided to block my home IP. After reading some anecdotes on the internet, I had to conclude they're exactly the kind of company I want to avoid (large, opaque, they employ weird algorithms/heuristics to flat out reject customers or suddenly take down their servers, no warning, you can't get an explanation, you're just fucked, just like when Google decides to arbitrarily block you; I've been there). IMO the point of a hosting provider is supposed to be that you can have some peace of mind and not worry about your shit breaking (that's still a worry as I continue to host everything at home). Instead with providers like this, you worry about them breaking your shit.
  - TheSmiddy 5y ago
    vultr lets you install custom ISOs, can also pxeboot.
- dang 5y ago
  Please keep snark, name-calling, shallow dismissal, and supercilious putdowns off this site. We're trying to avoid all of that here. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html Edit: we've had to ask you repeatedly to follow the site guidelines. Could you please review them and start following them now?
  - whateveracct 5y ago
    ACK :)
mpolun 5y ago
deleted 5y ago
[deleted]
tobyjsullivan 5y ago
Great work on the part of the author. Pareto principal holds. Often all it takes is one person motivated enough to look for efficiency opportunities. As for next steps, I can't imagine the Prometheus crew would object to a proposal + PR to make the Service Discovery an optional add-on in the next major version. It does open a can of worms around how such an add-on would be distributed if not built into the binary. (Caveat: I have no familiarity with this particular project or its unique constraints or goals.)
- rob74 5y ago
  Easy: they can provide an option to remove the SD functionality at compile time, and if you really care about the executable size, you can compile the code with this option (and `-ldflags="-s -w"`). The standard build would still be the "all batteries included" one to avoid support issues (people downloading the smaller binary and then asking why SD isn't working).
  - mongrelion 5y ago
    Caddy for example does this on their website. You can pick whatever addon you want and you'll be able to download a binary with exactly what you want.
- SuperQue 5y ago
  We've already talked about it at length, for many years. There's an approved development proposal to implement it. But, nobody has stepped up to contribute the code. The main problem is, Go doesn't have any kind of reasonable loadable library system. The current proposal is to make it easier to do compile-time plugins, similar to how Caddy and CoreDNS do things. We don't even need a major version to add such a thing. Just the time to write the feature. The thing is, it's just not that important an improvement. When you are running Prometheus in a production environment, it will end up using gigabytes of memory and disk space to operate. The savings of a few megabytes of binary and runtime memory are just not that important.
rektide 5y ago
i'd like to see memory usage differences, load time & runtime performance impacts. i expect most of these to be small but i expect some impact. also just worth oting that the memory impact of statically compiling in general is probably massive. most systems probably would have a good percent of these libraries in memory already if promtheus were using dynamic linking.
- SuperQue 5y ago
  I've done this test in the past. The typical footprint of default Prometheus is around 100MiB of RSS. Removing everything but flie and static configs reduces it to about 50MiB. Interesting for more embedded use cases, but not really a big deal when you're using a few GiB of memory for TSDB ingestion buffering.
akireu 5y ago
On a side note, Prometheus seems to be built for bloat. AFAIK, it isn't even designed to consume metrics other than from apps linked to its client library. It's like a microservice, but with the footprint of an operating system.
- momothereal 5y ago
  > AFAIK, it isn't even designed to consume metrics other than from apps linked to its client library. Could you elaborate? I use Prometheus to scrape from an HTTP endpoint in various Pods in Kubernetes, so the service discovery is pretty useful to me. I could see the Kubernetes & the other SDs split out of the core binary if default size is really an issue. Or are you talking about something else?
  - akireu 5y ago
    I'm talking about the way I'm expected to provide metrics for my apps. Rather than exporting free-form JSON and then scripting Prometheus to understand it, I'm expected to use a custom client library to export the metrics. As for Kubernetes, you can only use it with Prometheus because of not insignificant amount of work on both sides. Basically, the latter is designed for vendor lock-in.
    - gempir 5y ago
      Prometheus follows the OpenMetrics standard I'm not sure what you find propietary about that or specific to prometheus. https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md https://github.com/OpenObservability/OpenMetrics/blob/main/s...
      - momothereal 5y ago
        To be precise it was the other way around. OpenMetrics is a standardization effort for the format Prometheus made up. However Prometheus was designed before JSON was standardized itself, so I'm just glad they didn't choose XML!
etcet 5y ago
pphysch 5y ago
rob_c 5y ago
I suspect this is 70%+ of all features of all tools remain undiscovered by the users.
deleted 5y ago
[deleted]
jdalsgaard 5y ago
> Prometheus alternative Well... if size of the executable is really a concern, perhaps Victoria Metrics is worth considering; my amd64 executable is about 17MiB in size.
- fullsend 5y ago
  Everyone should consider Victoria Metrics anyway. It scales better performance wise, and they broke out components to improve scalability (vmagent, vmalert, etc) when Prometheus was just one huge process that did all the things. The two work closely together, and even did a good talk together about the differences.
  - pphysch 5y ago
    I love Prometheus because the (OpenMetrics) data protocol is so darn simple and easy to grok. You can do things like take an arbitrary data source, pipe it through awk and curl, and get it into prometheus metrics via remotewrite. You can also easily write your own /metrics endpoint in your favorite language. VictoriaMetrics sweetens the deal by offering a solution to long-term storage and more flexible service architecture without leaving the simple and highly interopable Prometheus ecosystem.
- madushan1000 5y ago
  Not only that, we were able to reduce the total virtual machine ram where our monitoring was hosted by half and storage is more efficient too I think when we switched to victoriametrics.
hughrr 5y ago
Yeah that’s only unused until you need it at which point it doesn’t involve futzing with anything. One of the things that kills me is running fluentd because you have to fuck around with ruby gems in containers every two minutes to get it to do something reasonable. This is pain. Prom is not.
NelsonMinar 5y ago
For comparison librrd is a 0.4MB .so; the entire rrdtool userspace is about 1MB. Yes, I realize librrd is much simpler and less featureful than Prometheus. But it's an interesting comparable for what a small, old utility that does something similar could be.
- wejick 5y ago
  I just knew about this, good stuff. Many projects I used are using this one.
- petre 5y ago
  Yet RRD databases are awfully big but often compress to single digit percentages. We use them to store one time graph data, so they do not ever get updated after the report job finishes. I'm looking for an alternative and it better not be a huge json array of arrays plus a header dictionary. It also should be an on disk something, not Influx or Prometheus.
  - simonmales 5y ago
    I only know RRD from Munin graphs, and now have a toy idea to used it directly to render a SVG. If disk space is an issue, do you or have you considered decompressing on the fly ?
    - chrismorgan 5y ago
      Or use a file system that supports transparent compression (e.g. btrfs).
  - SuperQue 5y ago
    You can use the Prometheus TSDB package, it's easy to import and use. Plus it's designed around mmap, so the kernel deals with the caching. You don't have to use a whole Prometheus server to make something like this work.
- SuperQue 5y ago
  Yup, it's tiny and simple. But the goals are wildly different. The same Prometheus binary you can run on a Pi scales to millions of series and millions of samples per second. The Prometheus TSDB itself is only one part of a the larger system. But, compared to librrd, it's vastly more functional. Besides the scaling I mentioned * It's ACID compliant. * It has WAL for reliability. * It has CPU and memory efficient compression. * It has an efficient mmap-based data loader. RRDtool, while efficient from a '90s perspective, is a toy by comparison. And yes, I've used rrdtool. Back in old-school days when Cacti was the new hot shit compared to MRTG.
dewey 5y ago
In which use case where Prometheus is used does it matter if the binary is 103MB or 2MB?
- ori_b 5y ago
  In the case where some rarely used feature is exploitable via some misconfiguration. See also: log4j
  - dylan604 5y ago
    This is the thing. Is all of that 105MB vs 2MB of stuff the devs actually know exactly what it is and why it's there? As others have stated, the dependancy fluff is just a multitude of footguns cocked, locked, and ready to rock your world
  - tgv 5y ago
    But how? I've got prometheus running in its own, non-root account. That makes exploiting vulnerabilities nearly useless, unless you can find a way to get it to spawn a process which can sudo. It runs behind nginx, which has a simple name/pwd protection on it. That makes it very hard to exploit. And prometheus only runs on one server; the others run node_exporter, or publish app data. What am I missing?
    - zufallsheld 5y ago
      You described multiple layers of security. Removing dependencies and stripping down code is the same: another layer of security.
    - ori_b 5y ago
      > What am I missing? If I knew, it would have a CVE number. However, the less code you have, the fewer the places there are where you need to ask that question.
- atmosx 5y ago
  I don't think anyone cares. They could make the binary 15kb, no one would notice. Most of the times runs inside a pod that will feature a 500+ MB operating system anyway...
mperham 5y ago
I maintain Faktory, a complex background job system written in Go. I keep the dependencies to a minimum and use `upx` to compress the built binary... ...for a grand total binary size of 5 MB. So many modern systems are huge because of the complex dependency tree they pull in. My entire binary likely fits within the L3 cache of the CPU you are using.
- 8note 5y ago
  Does that improve the usage of it? Have you experimented with larger sizes?
  - fjhfhgf 5y ago
- liftm 5y ago
  The first thing a 5 MB upx-compressed binary does when executed is uncompress itself to 15~30 MB of memory, right? So what does the on-disk binary size have to do with my L3 cache?
  - kimixa 5y ago
    Yup, upx is just a loading stub that decompresses the whole thing then gets out the way. If anything it'll use more memory from the remains of the decompression stub and not being able to be clever about only reading from the backing file needed memory pages.
  - mperham 5y ago
    Uncompressed is less than 10MB.
    - morelisp 5y ago
      So instead of 10MB you're paying 15MB? Seems like UPX is an unnecessary dependency.
- 5y ago
mappu 5y ago
If only service-discovery features are used from the entire SDK, then the real story here is that Go's DCE passes aren't strong enough - or the SDK is coupled in a way that makes DCE not work (overuse of reflection?).
- jrockway 5y ago
  Reflection in general isn't the problem, but if you specifically use reflection to look up a method in your program, then dead code elimination gives up. https://go.dev/src/cmd/link/internal/ld/deadcode.go https://go.dev/src/cmd/link/internal/ld/deadcode.go Taught to me on HN recently: https://news.ycombinator.com/item?id=30041763 https://news.ycombinator.com/item?id=30041763
  - nhoughto 5y ago
    Interesting detail, didn’t realize go dead code analysis was this easily made ineffective. 52MB for ec2 SDK is a big chunk of code that must be mostly dead/unused. Seems like eventually this should become a priority for library writers to support dead code analysis, otherwise going to get worse and worse..
- yencabulator 5y ago
  These SDKs typically hang everything off of the same Client type, and since methods can be reached via reflect I'd guess none of the exported API is safe to drop. The SDKs would need to be written to be much more modular. Those organizations also tend to spew a lot of code, most of it quite tangled together. They're definitely the largest dependencies I regularly see.
  - Too 5y ago
    Not only are they spewing out a lot of manual code. Additionally they include a lot of auto generated and versioned code. Azure python for example has a duplicate of every method for every version of the api.
- barsonme 5y ago
jrockway 5y ago
Now that Go has reflection in the upcoming 1.18 release, most HN comments about Go relate to binary size. Here we are again.
- esprehn 5y ago
  Do you mean generics? Go has had reflection for a long time.
  - jrockway 5y ago
    Yup! Long day. And I just wrote a more substantive comment about reflection...
PostThisTooFast 5y ago
physicles 5y ago
We were using Prometheus for metrics (for k8s services), but switched to influxdb because we were using it to store other non-metrics data, and it’s nice not to have two time series databases with two different query languages you need to learn. We went from using Prometheus itself to scrape metrics from services and export them to influxdb (fine, but very heavy), to using kapacitor (an utter nightmare for this particular use case), to just writing our own Prometheus metrics harvester from scratch (took about 3 days with service discovery). The current solution has been in place for more than a year and it’s perfect. I love influxdb’s on-disk compression, but its query planner and unpredictable ram usage leave a lot to be desired (at least the 1.x versions).
- dgnorton 5y ago
  Hi physicles, member of the InfluxDB team here. What version of 1.x are you running? Also curious to know if you've tried 2.x.
  - physicles 5y ago
    We're still on 1.7.x and using TSM (not TSI), because we've found that to work well enough for the time being. Haven't tried 2.x yet because the investment needed to properly benchmark and upgrade would probably be around a week at our scale, and there are just more pressing things to do. To be fair, we still haven't paid you guys a dime so I can't complain. But if you're interested in hearing my thoughts I can email you after next week's holiday.
    - dgnorton 5y ago
      I haven't looked but the latest 1.7.x is probably close to 2 years old. There have been quite a few improvements since 1.7 so you might see some improvements by upgrading to the latest 1.x. Minor nit on "TSM (not TSI)" - TSM (.tsm files) are the data files and that format hasn't changed. TSI is the newer, although mature at this point, indexing option that spills onto disk and can therefore be larger than the original in-mem index. You're probably using in-mem indexing. We're always interested in feedback. My name is david. You can email me at <name> at influxdata dot com.
dekhn 5y ago
Why optimize a binary that's 109MB? That's too small to matter.
- immibis 5y ago
  30 years later: " Why optimize a binary that's 109GB? That's too small to matter."
  - dekhn 5y ago
    I mean, for current computers (or even my 10 year old server), 128MB is so small that it's not worth optimizing. My $25 raspberry pis can run this without any problems while also running a bunch of other programs. my first linux computer had 4MB of RAM but that doesnt' mean I try to fit anything into that (once I upgraded to 32MB, I could run g++, emacs, X11 and xterm at the same time!)
    - rixed 5y ago
      Why did you upgrade back then? Because you wanted to be able to run more stuff, or because you wanted to be able to run the exact same executables, just bigger?
      - dekhn 5y ago
        Less paging. I could swap but it was slow. basically nobody is swapping because of a 128MB executable. if you are, get more ram or don't run prometheus.
        immibis 5y ago
        Less paging. What causes paging? Programs using lots of memory.
- dralley 5y ago
  A binary that doesn't fit in cache isn't too small to matter.
moondev 5y ago
How does the author determine how "most" people use prometheus?
- contravariant 5y ago
  Simple, you notice it's capable of communicating with lots of mutually exclusive cloud services and note that it could be smaller if you remove some of the relevant dependencies. Now whether that's a particularly useful observation I'm still not sure.
fnlurkr 5y ago
Maybe plugins are the way to go? https://medium.com/learning-the-go-programming-language/writing-modular-go-programs-with-plugins-ec46381ee1a9 https://medium.com/learning-the-go-programming-language/writ...
djmetzle 5y ago
Outstanding!
aliswe 5y ago
these software development kits are to a large extent a strongly typed representation of the REST API graph. Even though the application might only need two or three endpoints in kubernetes - which would be trivial to implement in go in just a couple of lines - they favor strong typing and include the SDK which is several megabytes. And the same for AWS, Azure, ... I'm not passing any judgment here by the way.
uaas 5y ago
Your use-case is not completely clear to me based on the article, but you might be better off with Prometheus’ agent approach, introduced recently: https://prometheus.io/docs/prometheus/latest/feature_flags/#prometheus-agent https://prometheus.io/docs/prometheus/latest/feature_flags/#...

142 points by wejick 5y ago

deleted 5y ago

[deleted]

whateveracct 5y ago

Clutching pearls about binary size is and always will be hilarious to me.

yodon 5y ago
And oh what a beautiful bike shed it will be...
jeffbee 5y ago
Author doesn't even say why they object to the size. Are they aware that file-backed executables are paged on demand and only the active parts of the program will be resident?
- wejick 5y ago
  sorry not to make it obvious in the article, I'm planning to run it in small iot pi based device locally. So having something small and fast is preferable, however the runtime performance is a more important thing I haven't touch.
  - ts4z 5y ago
    I'm curious -- is it the binary size that's a problem, or the resident size in memory? Demand paging should help, although you'd be stuck with carrying the enlarged binary.
    - wejick 5y ago
      my gut feeling tell it will be both memory and cpu utilization. cant be sure, until I can find good way to measure it.
      - SuperQue 5y ago
        If only there was a monitoring system you could install to measure such a thing.
      - ts4z 5y ago
        ps RSS measurement is pretty good for a start (although note in a forked process, shared copy-on-write pages are both reported as RSS for both the parent and child process). top reports this as RES. IIRC, debugging information is in a separate part of the process, so it's not loaded until it's used. Does that make it free? Probably not quite, but the kernel can ignore it until the process (presumably via its debugger) looks at it.
  - jeppesen-io 5y ago
    Prometheus is rather efficient, but it's focus is a little different than yours. Its designed to for large scale collection of metrics, scraped from many remote endpoints You can run it locally but the "prometheus" way for iot env would be a central prometheus server that scrapes the iot devices running a prometheus exporter, which tend to be very light weight
    - wejick 5y ago
      Totally agree. Another part of it is just feeding curiosities.
  - deleted 5y ago
    [deleted]
  - SuperQue 5y ago
    Prometheus works fine as-is on Pi devices. You'll spend most of your memory on ingestion buffering. I did the same tests as you did a while back, it only saves like 25-50MiB of memory IIRC. The only thing you really need to worry about on a Pi is that the default kernels are still 32-bit, and are set to 2GiB kernel boundary. So you'll be limited to how much TSDB storage can be mmap'd unless you switch to a 64-bit kernel. You may want to consider agent mode on your IoT device, and stream the data to an external server/service. https://prometheus.io/docs/prometheus/latest/feature_flags/#prometheus-agent https://prometheus.io/docs/prometheus/latest/feature_flags/#...
    - wejick 5y ago
      That's a good insight. Thanks
- Topgamer7 5y ago
  Granted these days everyone is used to applications consuming massive amounts of drive space. But perhaps they're using legacy hardware for a home lab, or a IoT device with limited disk space. From a security stand point, reduced application code decreases risk. It was service discovery code he removed, what if it reached out to discover services on application start up, that's a potential attack vector.
  - 8note 5y ago
    Does it actually reduce the risk? Sure if you audit, its easier to identify the risks, but a windows 98 program is going to be full of vulnerabilities while being small. Being small doesn't remove the vulnerabilities
  - shoo 5y ago
    > From a security stand point, reduced application code decreases risk. It was service discovery code he removed, what if it reached out to discover services on application start up, that's a potential attack vector. Agreed. I've see a similar pattern with certain open source libraries. The first example I think of is the spf13/viper [1] library, used to load configuration into go applications. Viper is equipped with code for reading config from various file formats, environment variables, as well as remote config sources such as etcd, consul. If you introduce the viper library as a dependency of your application to merely read config from environment variables and YAML files in the local filesystem, then your go application suddenly gains a bunch of transitive dependencies on modules related to remote config loading for various species of remote config provider. It's not uncommon for these kind of remote config loading dependencies to have security vulnerabilities. As well as the potential increased attack surface if a bunch of unnecessary code to load application configuration from all manner of remote config providers ends up in your application binary [2], if you work in an environment that monitors for vulnerabilities in open source dependencies, if you depend on an open source library that drags in dozens of transitive dependencies you don't really need, it adds a fair bit of additional overhead re: detecting, investigating and patching the potential vulnerabilities. I guess there's arguably a "Hickean" simple-vs-easy tradeoff in how such libraries are designed. The "easy" design, that makes it quick for developers to get started and achieve immediate success with a config loading library, is to include code to load config from all popular supported config sources into the default configuration of the library, reducing the amount of steps a new user has to do to get the library to work for their use case. A less easy but arguably "simpler" design might be to only include a common config-provider interface in the core module and push all config-provider-specific client/adaptor code into separate modules, and force the user to think about which config sources they want to read from and then manually add and integrate the dependencies for the corresponding modules that contain the additional code they want. edit: there has indeed been some discussion about the proliferation of dependencies, and what to do about them, in viper's issue tracker [3] [4] [1] https://github.com/spf13/viper https://github.com/spf13/viper [2] this may or may not actually happen, depending on which function calls you actually use and what the compiler figures out. If your application doesn't call any remote-config-provider library functions then you shouldn't expect to find any in your resulting application binary, even if the dependency is there at the coarser-grain module dependency level [3] https://github.com/spf13/viper/issues/887 https://github.com/spf13/viper/issues/887 [4] https://github.com/spf13/viper/issues/707 https://github.com/spf13/viper/issues/707
- cosmotic 5y ago
  Image pull size for a container is likely the concern. It could shave a few seconds off a regularly-run integration test. If it's run via on-demand build agents, then there's no image cache.
  - jeffbee 5y ago
    If it takes multiple seconds to pull ~35MB of compressible text into your CI environment, there may be other, larger problems to solve.
    - cosmotic 5y ago
      I was estimating off the speed it takes to pull images to my local computer where the limiting factor appears to be something other than my internet connection so either the image extraction process or a docker hub throttle.
- wahern 5y ago
  That only helps if the code is well segregated by usage. Looking at the ELF symbol table for prometheus-2.33.0-rc.1.linux-amd64, it's not clear to me this is the case. Not sure how it's ordered. Lexical import order? Anyhow, without profiling how could the compiler know how to order things optimally? I think this is one of those cases where, in the absence of profiling or some other hack (e.g. ensuring all routines within a library are cleanly segregated across page boundaries within the static binary and the I/O scheduler doesn't foil your intent), dynamic linking would prove superior, at least for such large amounts of code.
0xbadcafebee 5y ago
It correlates to performance, speed to iterate, security, and design complexity, but ok
- Zababa 5y ago
  Any hard data on any of that?
  - goodpoint 5y ago
    Unless a fat binary embeds pictures and some music, it's all CPU instructions. Tenths of megabytes of CPU instructions is complexity. This kind of bloat is the number one enemy of security, as any security engineer could confirm.
  - 0xbadcafebee 5y ago
    Sure, lots, go look for studies on estimation of defects based on LOC and project size/complexity (they go back to the 1970s). But you don't need to look, the principles are simple. Unless an application is filled with JPEGs or uncompressed arbitrary data files, its size reflects lines of code (machine code, interpreted code, etc). Bigger the app, the more lines of code. Every line of code has a non-zero bug probability. Every new line of code increases probability. More lines of code, higher probability. Bugs include security bugs; higher probability of bugs, higher probability of security bugs. CPU cache is finite. Only so many lines of code can be cached or optimized. Larger size takes up more room in memory, which when combined with lot of other gigantic apps, means less memory for heap space, disk cache, etc. Larger size also takes up more room on disk, which adds up when you don't delete old builds on disk and loop over a build process. Since larger size means more lines of code, that means longer compile times, which means longer wait every time you change a line and need to recompile, copy an artifact somewhere, retest. More lines of code means more code executed. If you have 10 lines of code in a function, and you add 100 lines to it, the compiler doesn't just optimize away all 100 new lines, it's going to add more machine code and code paths. Unless you only ever add new code paths, some of that new code will extend existing code paths or add instructions, and that means more CPU cycles to complete execution. (Same concept for interpreted code) More lines of code means more code paths. More code paths increases complexity. The more code paths, the longer and more difficult testing gets to the point you can't even develop enough tests to cover all the code paths, so it's impossible to even find all the bugs. More complexity leads to difficulty in humans understanding and working with the codebase, and difficulty in understanding leads to slower and more error-prone development. Larger means more network bandwidth, meaning file transfers take longer, increasing speed to iterate and producing worse UX. If people download your app every 10 minutes in their CI/CD pipeline, larger size means more network bandwidth used. "Free" CDNs have limits; the larger a project gets, the more file size affects network performance, reliability, and cost. If you pay for bandwidth, a 100MB file costs 100x more than a 1MB file. The more apps you use that are big, the more every one of these effects increase. One big app you might not notice. 100 big apps lead to noticeable slowness, bugs, less memory, less disk space.
akireu 5y ago
It's all fun and games until you're stuck for a hour downloading 600MB of updated packages over a metered LTE. The same is with RAM usage: 512MB was enough for a phone back in 2014, now a smart TV with 2GB is barely capable of multitasking. Sure, binary sizes don't matter in most contexts. But when they do, it's a PITA.
- jayd16 5y ago
  Galaxy S5 from 2014 had 2GB and that was 1080p vs 4k texture sizes for today. Seems on par.
  - akireu 5y ago
    What you're kind of missing is that the S5 was a flagship phone. Generally, one has to save for more than a month to afford a purchase like that. The idea of working an extra month so that some FAANG prick meets their KPI by cutting corners on optimization doesn't even look like feudalism. It looks like idiocracy. Paying the lip service of fat shaming code bloat is the cost-effective option by comparison :)
    - asiachick 5y ago
      What does FAANG have to do with this? Don't FANNG people obsess over bloat because they're trying to reach billions of customers? It might not seem that way since their pages are bigger but I'd be surprised if they were happy to leave 10s of millions of customers on the table.
      - ysleepy 5y ago
        FAANG are the worst offenders. Didn't facebook employ ungodly hacks to unload/load parts of the android app to navigate around the 65k method limit of dex? Have you looked at the js monstrosity of the Google hardware shop website?
      - akireu 5y ago
        They're just poster children for the particular brand of disdain $100k+/year "tech workers" bear for their users: they make enough for the shiniest of toys, so they're too far above spending their valuable time to make their software run smooth on our $100 crap phones. Nevermind that each Fb client update likely produces hundreds of tons of toxic trash called gadgets. Sure, sometimes they do optimizations. Generally, though, both Fb and Google keep exploring the physical limits to code bloat. Remember that one time that Fb hit the JVM class count limit?
xuhu 5y ago
Auditd_2.8-amd64.deb is 194kb on debian, rsyslog_8.32-amd64 is 411kb, and they both support centralized auditing and log collection from multiple hosts.
- djbusby 5y ago
  Do they do the metrics like Prometheus does? And include the central collector and basic graph builder?
- dtech 5y ago
  This doesn't seem a fair comparison. Prometheus is statically linked like all Go applications, and those packages are not. You can debate the merits of that, but if you compare a "only rsyslog" server vs a "only prometheus" server the 2 will be much closer in size.
- dralley 5y ago
  Even journalctl is only 90kb, and the entire combined systemd package is 4mb (but that's including all the documentation and a dozen other different binaries)
sigmonsays 5y ago
I also find this comical. I'd love to know why 100MB is that big of a deal. If network is slow, cache locally. Seems like nothing here to worry about.
- deleted 5y ago
  [deleted]
jeppesen-io 5y ago
This one does not even make sense - 100 megs for a binary for centralized metrics? Who would even notice next to the OS and metrics storage. By design you should not install prometheus on every server you monitor - it's designed to scrape metrics Its a database, webui with support for email, webhooks, slack, pagerduty, aws api and many others. 100megs does not sound like a lot for all Pormetheus provides
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
TillE 5y ago
I work in a lot of situations with hard or soft resource limits where I actually do need to count bytes and/or CPU cycles, so it's bizarre to see anyone shrugging about distributing tens of megabytes of fat for literally no reason. One thing Microsoft got right a long time ago was separating out debug symbols into their own file by default. I think that's still awkward on Linux.
- Arnavion 5y ago
  >I think that's still awkward on Linux. At least for .deb and .rpm packages, the default build process automatically extracts debug symbols into separate packages. Eg the process of building package `foo` also produces `foo-dbgsym` and `foo-debuginfo` packages respectively that contain debug symbols for every involved binary and library, while `foo` contains the stripped files. So anyone who wants to debug a coredump / live process just installs the corresponding -dbgsym / -debuginfo package and now gdb has all the debug info it needs. Distros have also started incorporating debuginfod into their repos so that gdb can download symbols automatically. So you don't even have to hunt for the right debuginfo package.
foxfluff 5y ago
It's kind of ironic reading this comment given that at the time you posted it, I was screaming at gcc's stupid code generator for wasting bytes recreating constants that were already there in that very register! That code needs to fit in a couple hundred bytes.. And half an hour ago I was (once again) checking out hosting providers and lamenting the fact that most don't seem to offer support for loading custom ISOs so I could install a 30 megabyte distro and make the most out of the cheap plans that only offer something like 10 gigabytes of storage. Half of it is wasted after you install one of the these obese mainstream distros.
- mitjam 5y ago
  Hetzner Cloud can start instances from ISOs - here is an example for ipfire : https://wiki.ipfire.org/installation/hetzner-cloud https://wiki.ipfire.org/installation/hetzner-cloud
  - foxfluff 5y ago
    I got a VPS from Hetzner last year but they decided to block my home IP. After reading some anecdotes on the internet, I had to conclude they're exactly the kind of company I want to avoid (large, opaque, they employ weird algorithms/heuristics to flat out reject customers or suddenly take down their servers, no warning, you can't get an explanation, you're just fucked, just like when Google decides to arbitrarily block you; I've been there). IMO the point of a hosting provider is supposed to be that you can have some peace of mind and not worry about your shit breaking (that's still a worry as I continue to host everything at home). Instead with providers like this, you worry about them breaking your shit.
- TheSmiddy 5y ago
  vultr lets you install custom ISOs, can also pxeboot.
dang 5y ago
Please keep snark, name-calling, shallow dismissal, and supercilious putdowns off this site. We're trying to avoid all of that here. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html Edit: we've had to ask you repeatedly to follow the site guidelines. Could you please review them and start following them now?
- whateveracct 5y ago
  ACK :)

mpolun 5y ago

deleted 5y ago

[deleted]

tobyjsullivan 5y ago

Great work on the part of the author. Pareto principal holds. Often all it takes is one person motivated enough to look for efficiency opportunities. As for next steps, I can't imagine the Prometheus crew would object to a proposal + PR to make the Service Discovery an optional add-on in the next major version. It does open a can of worms around how such an add-on would be distributed if not built into the binary. (Caveat: I have no familiarity with this particular project or its unique constraints or goals.)

rob74 5y ago
Easy: they can provide an option to remove the SD functionality at compile time, and if you really care about the executable size, you can compile the code with this option (and `-ldflags="-s -w"`). The standard build would still be the "all batteries included" one to avoid support issues (people downloading the smaller binary and then asking why SD isn't working).
- mongrelion 5y ago
  Caddy for example does this on their website. You can pick whatever addon you want and you'll be able to download a binary with exactly what you want.
SuperQue 5y ago
We've already talked about it at length, for many years. There's an approved development proposal to implement it. But, nobody has stepped up to contribute the code. The main problem is, Go doesn't have any kind of reasonable loadable library system. The current proposal is to make it easier to do compile-time plugins, similar to how Caddy and CoreDNS do things. We don't even need a major version to add such a thing. Just the time to write the feature. The thing is, it's just not that important an improvement. When you are running Prometheus in a production environment, it will end up using gigabytes of memory and disk space to operate. The savings of a few megabytes of binary and runtime memory are just not that important.

rektide 5y ago

i'd like to see memory usage differences, load time & runtime performance impacts. i expect most of these to be small but i expect some impact. also just worth oting that the memory impact of statically compiling in general is probably massive. most systems probably would have a good percent of these libraries in memory already if promtheus were using dynamic linking.

SuperQue 5y ago
I've done this test in the past. The typical footprint of default Prometheus is around 100MiB of RSS. Removing everything but flie and static configs reduces it to about 50MiB. Interesting for more embedded use cases, but not really a big deal when you're using a few GiB of memory for TSDB ingestion buffering.

akireu 5y ago

On a side note, Prometheus seems to be built for bloat. AFAIK, it isn't even designed to consume metrics other than from apps linked to its client library. It's like a microservice, but with the footprint of an operating system.

momothereal 5y ago
> AFAIK, it isn't even designed to consume metrics other than from apps linked to its client library. Could you elaborate? I use Prometheus to scrape from an HTTP endpoint in various Pods in Kubernetes, so the service discovery is pretty useful to me. I could see the Kubernetes & the other SDs split out of the core binary if default size is really an issue. Or are you talking about something else?
- akireu 5y ago
  I'm talking about the way I'm expected to provide metrics for my apps. Rather than exporting free-form JSON and then scripting Prometheus to understand it, I'm expected to use a custom client library to export the metrics. As for Kubernetes, you can only use it with Prometheus because of not insignificant amount of work on both sides. Basically, the latter is designed for vendor lock-in.
  - gempir 5y ago
    Prometheus follows the OpenMetrics standard I'm not sure what you find propietary about that or specific to prometheus. https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md https://github.com/OpenObservability/OpenMetrics/blob/main/s...
    - momothereal 5y ago
      To be precise it was the other way around. OpenMetrics is a standardization effort for the format Prometheus made up. However Prometheus was designed before JSON was standardized itself, so I'm just glad they didn't choose XML!

etcet 5y ago

pphysch 5y ago

rob_c 5y ago

I suspect this is 70%+ of all features of all tools remain undiscovered by the users.

deleted 5y ago

[deleted]

jdalsgaard 5y ago

> Prometheus alternative Well... if size of the executable is really a concern, perhaps Victoria Metrics is worth considering; my amd64 executable is about 17MiB in size.

fullsend 5y ago
Everyone should consider Victoria Metrics anyway. It scales better performance wise, and they broke out components to improve scalability (vmagent, vmalert, etc) when Prometheus was just one huge process that did all the things. The two work closely together, and even did a good talk together about the differences.
- pphysch 5y ago
  I love Prometheus because the (OpenMetrics) data protocol is so darn simple and easy to grok. You can do things like take an arbitrary data source, pipe it through awk and curl, and get it into prometheus metrics via remotewrite. You can also easily write your own /metrics endpoint in your favorite language. VictoriaMetrics sweetens the deal by offering a solution to long-term storage and more flexible service architecture without leaving the simple and highly interopable Prometheus ecosystem.
madushan1000 5y ago
Not only that, we were able to reduce the total virtual machine ram where our monitoring was hosted by half and storage is more efficient too I think when we switched to victoriametrics.

hughrr 5y ago

Yeah that’s only unused until you need it at which point it doesn’t involve futzing with anything. One of the things that kills me is running fluentd because you have to fuck around with ruby gems in containers every two minutes to get it to do something reasonable. This is pain. Prom is not.

NelsonMinar 5y ago

For comparison librrd is a 0.4MB .so; the entire rrdtool userspace is about 1MB. Yes, I realize librrd is much simpler and less featureful than Prometheus. But it's an interesting comparable for what a small, old utility that does something similar could be.

wejick 5y ago
I just knew about this, good stuff. Many projects I used are using this one.
petre 5y ago
Yet RRD databases are awfully big but often compress to single digit percentages. We use them to store one time graph data, so they do not ever get updated after the report job finishes. I'm looking for an alternative and it better not be a huge json array of arrays plus a header dictionary. It also should be an on disk something, not Influx or Prometheus.
- simonmales 5y ago
  I only know RRD from Munin graphs, and now have a toy idea to used it directly to render a SVG. If disk space is an issue, do you or have you considered decompressing on the fly ?
  - chrismorgan 5y ago
    Or use a file system that supports transparent compression (e.g. btrfs).
- SuperQue 5y ago
  You can use the Prometheus TSDB package, it's easy to import and use. Plus it's designed around mmap, so the kernel deals with the caching. You don't have to use a whole Prometheus server to make something like this work.
SuperQue 5y ago
Yup, it's tiny and simple. But the goals are wildly different. The same Prometheus binary you can run on a Pi scales to millions of series and millions of samples per second. The Prometheus TSDB itself is only one part of a the larger system. But, compared to librrd, it's vastly more functional. Besides the scaling I mentioned * It's ACID compliant. * It has WAL for reliability. * It has CPU and memory efficient compression. * It has an efficient mmap-based data loader. RRDtool, while efficient from a '90s perspective, is a toy by comparison. And yes, I've used rrdtool. Back in old-school days when Cacti was the new hot shit compared to MRTG.

dewey 5y ago

In which use case where Prometheus is used does it matter if the binary is 103MB or 2MB?

ori_b 5y ago
In the case where some rarely used feature is exploitable via some misconfiguration. See also: log4j
- dylan604 5y ago
  This is the thing. Is all of that 105MB vs 2MB of stuff the devs actually know exactly what it is and why it's there? As others have stated, the dependancy fluff is just a multitude of footguns cocked, locked, and ready to rock your world
- tgv 5y ago
  But how? I've got prometheus running in its own, non-root account. That makes exploiting vulnerabilities nearly useless, unless you can find a way to get it to spawn a process which can sudo. It runs behind nginx, which has a simple name/pwd protection on it. That makes it very hard to exploit. And prometheus only runs on one server; the others run node_exporter, or publish app data. What am I missing?
  - zufallsheld 5y ago
    You described multiple layers of security. Removing dependencies and stripping down code is the same: another layer of security.
  - ori_b 5y ago
    > What am I missing? If I knew, it would have a CVE number. However, the less code you have, the fewer the places there are where you need to ask that question.
atmosx 5y ago
I don't think anyone cares. They could make the binary 15kb, no one would notice. Most of the times runs inside a pod that will feature a 500+ MB operating system anyway...

mperham 5y ago

I maintain Faktory, a complex background job system written in Go. I keep the dependencies to a minimum and use `upx` to compress the built binary... ...for a grand total binary size of 5 MB. So many modern systems are huge because of the complex dependency tree they pull in. My entire binary likely fits within the L3 cache of the CPU you are using.

8note 5y ago
Does that improve the usage of it? Have you experimented with larger sizes?
- fjhfhgf 5y ago
liftm 5y ago
The first thing a 5 MB upx-compressed binary does when executed is uncompress itself to 15~30 MB of memory, right? So what does the on-disk binary size have to do with my L3 cache?
- kimixa 5y ago
  Yup, upx is just a loading stub that decompresses the whole thing then gets out the way. If anything it'll use more memory from the remains of the decompression stub and not being able to be clever about only reading from the backing file needed memory pages.
- mperham 5y ago
  Uncompressed is less than 10MB.
  - morelisp 5y ago
    So instead of 10MB you're paying 15MB? Seems like UPX is an unnecessary dependency.
5y ago

mappu 5y ago

If only service-discovery features are used from the entire SDK, then the real story here is that Go's DCE passes aren't strong enough - or the SDK is coupled in a way that makes DCE not work (overuse of reflection?).

jrockway 5y ago
Reflection in general isn't the problem, but if you specifically use reflection to look up a method in your program, then dead code elimination gives up. https://go.dev/src/cmd/link/internal/ld/deadcode.go https://go.dev/src/cmd/link/internal/ld/deadcode.go Taught to me on HN recently: https://news.ycombinator.com/item?id=30041763 https://news.ycombinator.com/item?id=30041763
- nhoughto 5y ago
  Interesting detail, didn’t realize go dead code analysis was this easily made ineffective. 52MB for ec2 SDK is a big chunk of code that must be mostly dead/unused. Seems like eventually this should become a priority for library writers to support dead code analysis, otherwise going to get worse and worse..
yencabulator 5y ago
These SDKs typically hang everything off of the same Client type, and since methods can be reached via reflect I'd guess none of the exported API is safe to drop. The SDKs would need to be written to be much more modular. Those organizations also tend to spew a lot of code, most of it quite tangled together. They're definitely the largest dependencies I regularly see.
- Too 5y ago
  Not only are they spewing out a lot of manual code. Additionally they include a lot of auto generated and versioned code. Azure python for example has a duplicate of every method for every version of the api.
barsonme 5y ago

jrockway 5y ago

Now that Go has reflection in the upcoming 1.18 release, most HN comments about Go relate to binary size. Here we are again.

esprehn 5y ago
Do you mean generics? Go has had reflection for a long time.
- jrockway 5y ago
  Yup! Long day. And I just wrote a more substantive comment about reflection...

PostThisTooFast 5y ago

physicles 5y ago

We were using Prometheus for metrics (for k8s services), but switched to influxdb because we were using it to store other non-metrics data, and it’s nice not to have two time series databases with two different query languages you need to learn. We went from using Prometheus itself to scrape metrics from services and export them to influxdb (fine, but very heavy), to using kapacitor (an utter nightmare for this particular use case), to just writing our own Prometheus metrics harvester from scratch (took about 3 days with service discovery). The current solution has been in place for more than a year and it’s perfect. I love influxdb’s on-disk compression, but its query planner and unpredictable ram usage leave a lot to be desired (at least the 1.x versions).

dgnorton 5y ago
Hi physicles, member of the InfluxDB team here. What version of 1.x are you running? Also curious to know if you've tried 2.x.
- physicles 5y ago
  We're still on 1.7.x and using TSM (not TSI), because we've found that to work well enough for the time being. Haven't tried 2.x yet because the investment needed to properly benchmark and upgrade would probably be around a week at our scale, and there are just more pressing things to do. To be fair, we still haven't paid you guys a dime so I can't complain. But if you're interested in hearing my thoughts I can email you after next week's holiday.
  - dgnorton 5y ago
    I haven't looked but the latest 1.7.x is probably close to 2 years old. There have been quite a few improvements since 1.7 so you might see some improvements by upgrading to the latest 1.x. Minor nit on "TSM (not TSI)" - TSM (.tsm files) are the data files and that format hasn't changed. TSI is the newer, although mature at this point, indexing option that spills onto disk and can therefore be larger than the original in-mem index. You're probably using in-mem indexing. We're always interested in feedback. My name is david. You can email me at <name> at influxdata dot com.

dekhn 5y ago

Why optimize a binary that's 109MB? That's too small to matter.

immibis 5y ago
30 years later: " Why optimize a binary that's 109GB? That's too small to matter."
- dekhn 5y ago
  I mean, for current computers (or even my 10 year old server), 128MB is so small that it's not worth optimizing. My $25 raspberry pis can run this without any problems while also running a bunch of other programs. my first linux computer had 4MB of RAM but that doesnt' mean I try to fit anything into that (once I upgraded to 32MB, I could run g++, emacs, X11 and xterm at the same time!)
  - rixed 5y ago
    Why did you upgrade back then? Because you wanted to be able to run more stuff, or because you wanted to be able to run the exact same executables, just bigger?
    - dekhn 5y ago
      Less paging. I could swap but it was slow. basically nobody is swapping because of a 128MB executable. if you are, get more ram or don't run prometheus.
      - immibis 5y ago
        Less paging. What causes paging? Programs using lots of memory.
dralley 5y ago
A binary that doesn't fit in cache isn't too small to matter.

moondev 5y ago

How does the author determine how "most" people use prometheus?

contravariant 5y ago
Simple, you notice it's capable of communicating with lots of mutually exclusive cloud services and note that it could be smaller if you remove some of the relevant dependencies. Now whether that's a particularly useful observation I'm still not sure.

fnlurkr 5y ago

Maybe plugins are the way to go? https://medium.com/learning-the-go-programming-language/writing-modular-go-programs-with-plugins-ec46381ee1a9 https://medium.com/learning-the-go-programming-language/writ...

djmetzle 5y ago

Outstanding!

aliswe 5y ago

these software development kits are to a large extent a strongly typed representation of the REST API graph. Even though the application might only need two or three endpoints in kubernetes - which would be trivial to implement in go in just a couple of lines - they favor strong typing and include the SDK which is several megabytes. And the same for AWS, Azure, ... I'm not passing any judgment here by the way.

uaas 5y ago

Your use-case is not completely clear to me based on the article, but you might be better off with Prometheus’ agent approach, introduced recently: https://prometheus.io/docs/prometheus/latest/feature_flags/#prometheus-agent https://prometheus.io/docs/prometheus/latest/feature_flags/#...