42 ms·
Hacker deleted all of NewsBlur’s Mongo data and is now holding the data hostage
- jepler 5y ago10:35p ET: Looks like the snapshot will take 10 hours to make. Ordinarily this wouldn't be a problem becuase the service is running and a snapshot would be made on a secondary DB. But all of the Mongo DBs faithfully deleted their data, so I'm taking a snapshot of a recent good backup. Once done, I can replicare the DB and we'll be back. 9:54p ET: Holy moly, when I switched to a new Mongo DB server, a hacker deleted all of NewsBlur’s mongo data and is now holding NewsBlur’s data hostage. I’m dipping into a backup from a few hours ago and will keep you all updated. They've also given some info on twitter https://twitter.com/NewsBlur https://twitter.com/NewsBlur that I haven't digested yet.
- deleted 5y ago[deleted]
- wolverine876 5y agoThese are crimes. Where is law enforcement? I am not sure why the public still gives them a pass and treats computer crimes differently. With the resources of the federal government, it shouldn't be hard to find and take down the criminals. Think of how easily the criminals exploit their victims - it is just as hard for the criminals to play defense as it is for everyone else.
- naikrovek 5y agodifficult or impossible to locate, then catch. don't know the first thing about how to even collect evidence to aid in prosecution. don't know what crime has happened. no one died. probably from another country and "that ain't my jurisdiction" lazy etc.
- elliekelly 5y agoThe Computer Fraud and Abuse Act. And I can assure you the DOJ & FBI are more than capable of collecting evidence to aid in prosecution. The problem is the criminals often located in jurisdictions that “look the other way” with respect to cyber crimes committed against companies and people in other countries. So prosecuting the criminals is a whole lot of paperwork for a whole lot of nothing to happen. It’s not as though Russia or North Korea are going to extradite their own citizens to the US to stand trial.
- naikrovek 5y agoyes they are, but they don't answer when you call 911, do they? local police fumble around with this kind of thing every day, and crimes go unpunished thousands of times a second, every second, of every day.
- sp332 5y agoWe know generally where they are, because there are only a few countries that turn a blind eye like this and totally refuse to cooperate internationally. Anything outside their home country's jurisdiction is fair game. Ransomware can be a real boost to a local economy. https://sec.okta.com/articles/2020/08/crimeops-operational-art-cyber-crime https://sec.okta.com/articles/2020/08/crimeops-operational-a...
- tyingq 5y agoIn case anyone else was curious what NewsBlur is supposed to look like: https://web.archive.org/web/20210601112225if_/https://www.newsblur.com/ https://web.archive.org/web/20210601112225if_/https://www.ne...
- qbasic_forever 5y agoNewblur was one of the big alternatives and go-to options when Google Reader shut down their RSS reader. It's a great service.
- elp 5y agoI've been a paying user since google shut down Reader. I really agree with your comment. On the other hand I'm expecting to be ridiculously productive at work until Newsblur is back up.
- asteroidbelt 5y agoOne possible solution to such problems is to prohibit paying ransom by law. Hackers can still be destructive, but at least they will have less incentive to participate in such activities.
- SamReidHughes 5y agoAnother option is to add a 900% tax on ransom payments, owed by the payer. It would either lower the price of ransoms or get less of them paid. I don't know why that would be better than prohibition, but it would be funnier.
- afterburner 5y agoIt wouldn't be funnier, it would have the exact same effect as a prohibition law.
- ipython 5y agoUnfortunately it looks like the exact opposite is occurring: https://apnews.com/article/technology-business-government-and-politics-d8c1e9958ad1e89eab83f44e6ca70a94 https://apnews.com/article/technology-business-government-an... The regularity of ransomware has apparently made the expense “ordinary” therefore now tax deductible. Shrug.
- SilverRed 5y agoWouldn't other types of crime like theft or damaged windows be the same?
- qbasic_forever 5y agoCompanies get around that by hiring a security 'consultant' or other firm to pay the ransom. Conveniently the cost of the consultant is exactly the ransom. It's enough legal cover for the company to release statements like "XYZ corp is not negotiating or paying the ransom."
- 5y ago
- conesus 5y agoNewsBlur's founder here. I'll attempt to explain what's happening. This situation is more of a script kiddie than a hacker. I'm in the process of moving everything on NewsBlur over to Docker containers in prep for the big redesign launching next week. It's been a great year of maintenance and I've enjoyed the fruits of Ansible + Docker for NewsBlur's 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models). About two hours before this happened, I switched the MongoDB cluster over to the new servers. When I did that, I shut down the original primary in order to delete it in a few days when all was well. (Thank goodness I did that! It'll come in handy a few hours from now). Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didn't work on a new server because of Docker. When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world. So while my firewall was "active", doing a `sudo iptables -L | grep 27017` showed that MongoDB was open the world. More info on SO[1]. To be honest, I'm a bit surprised it took over 3 hours from when I flipped the switch to when a script kiddie dropped NewsBlur's MongoDB collections, and ransomed about 250GB of data. I am now running a snapshot on that old primary, just in case it reconnects to a network and deletes everything. Once done, I'll boot it up, secondary it out, and be back in business. Let's hope my assumptions hold. [1]: https://stackoverflow.com/questions/30383845/what-is-the-best-practice-of-docker-ufw-under-ubuntu https://stackoverflow.com/questions/30383845/what-is-the-bes...
- dawnerd 5y agoThis actually got me a while ago but with redis and some script kiddy turning my dev server into a bitcoin miner. Anyone else running docker and using iptables really needs to read this https://docs.docker.com/network/iptables/ https://docs.docker.com/network/iptables/
- squeaky-clean 5y agoAlmost exactly the same thing has happened to me except Selenium and they were trying to log into Playstation Network accounts.
- deleted 5y ago
- haimez 5y agoIs it too soon to ask whether their mongo instance was publicly available without password protection?
- fiddlerwoaroof 5y agohttps://news.ycombinator.com/item?id=27613661 https://news.ycombinator.com/item?id=27613661
- haimez 5y agoRight. Maybe someday, with enough public shaming in situations like this, mongo will reconsider those defaults. Or not, maybe their best potential customers should continue to get burned publicly in incidents that have a direct line to their poor decisions.
- steffan 5y agoDefaults have long been changed to prevent listening on 0.0.0.0 by default; this has been the case since May of 2017, over 4 years ago. In order for this to occur, a user has to explicitly turn on listening on all interfaces. In conjunction with this, prudence would dictate that you enable authentication as well. In this case, it seems that reliance was placed on Docker to maintain iptables settings to disallow connections from untrusted IPs and that iptables setting was reset. As always, defense in depth is a good strategy; authentication in addition to firewall rules would have prevented this.
- haimez 5y agoThanks for the info, I haven’t payed attention to mongo since the last time I was personally burned by data corruption in the pre-2017 and web-scale (/dev/null db) era. Sounds there’s a mix of blame to go around- but it also sounds like exactly the MongoDB era I remember and hate. They know very well what their dockerhub image install looks like, and if they didn’t like it- they could request a change. Edit: they/(you || your employer). I know it gets tedious, but calling out your conflicts of interest can save everyone a lot of time.
- haimez 5y agoIs it too soon to ask whether the mongo instance in question was available publicly to connect to with perhaps the default installation password?
- fiddlerwoaroof 5y agohttps://news.ycombinator.com/item?id=27613661 https://news.ycombinator.com/item?id=27613661
- ncmncm 5y agoJoke's on them. People who use Mongo use it mostly because their data doesn't matter very much, even to them. Deleting it just saves them time cleaning it out. It's like stealing their garbage off the curb. The only valuable part is the can.
- deleted 5y ago[deleted]
- whalesalad 5y agoThis is one reason why most of my services are inaccessible from the public internet by design (on private subnets). Live and learn.
- zylent 5y agoZero reason to have database servers with an internet connection. These days you can ship in an entire environment with containers into borderline air gapped environments.
- lmm 5y agoI'd argue that that's a backwards approach and actually what lead to this hack - building this system around the database servers not being publicly exposed, thinking the database servers weren't publicly exposed, and then when you accidentally publicly expose them (and you will sooner or later, a network is too big a boundary to protect all of it) it's a disaster. It's better to build every server for public exposure from day 1 and treat all connections as potentially hostile, even if they're coming from the internal network.
- zylent 5y agoDefense in depth / zero trust is definitely the way to go, however it's trivial to prevent a system from having internet access - for this hack to occur, the system had to be deployed with a public IP address directly assigned. NAT based internet access (IGW in AWS) or a private VPC with no IGW and no public on the instance is borderline standard in production cloud deployments these days. Re: "You will sooner or later..." it's super easy to test for stuff like this with sentinel - I use this and scan dev / stage in my CI pipelines with rapid7 which will SCREAM about stuff like no DB password.
- lmm 5y agoEvery step is easy once you think about it; the hard part is spending any attention on it in the first place. I would definitely say it's more effective to test your existing layers before adding more layers, and I think the "defence in depth" concept leads people astray there. Having multiple porous layers works on a battlefield where attacks are costly; it doesn't work on the internet where if one attack gets through an outer layer then all the other attacks can immediately get through the same way and start hitting the inner layer.
- 0-1 5y agoFrom a quick skim through https://github.com/samuelclay/newsblur https://github.com/samuelclay/newsblur for models extending mongo.Document, it looks like the following private customer data has been breached: - all story content from all private feeds - any uploaded OPML files, including URLs for any private RSS feeds - User’s twitter/facebook account info and access tokens, if the user had linked those services with their newsblur account - all data that would be used to create a user profile page, including email address, whether the user had a public profile or not However most personal data, such as password hashes and billing info, was stored in postgres.
- amphorasource 5y agoSamuel's nonchalant reply to this is highly disturbing to me. I'm a Newsblur customer and as far as I can tell, my feed data is in the hands of some hacker and he doesn't care at all. I am much less concerned about the service being restored, which seems to be all that he's worried about, and more about knowing who has my data. On top of that, I used to use his "forward newsletters to Newsblur" feature for a long time. I've long stopped using it and deleted all the feeds with newsletters, partly because it never worked very well, but mostly because I more or less had an inkling that something like this would happen and it's just not worth it, too many email newsletters leak personal data all over the place. However, I have no clue if those were really deleted or if they stuck around in MongoDB. Clarification what exactly the ransom is (did he just dump it locally and encrypt it? or did the hacker download it and is threatening to leak it?) would be very welcome.
- canuckintime 5y ago> On top of that, I used to use his "forward newsletters to Newsblur" feature for a long time. I've long stopped using it and deleted all the feeds with newsletters, partly because it never worked very well, would you share your alternative?
- amphorasource 5y agoNewsblur mangled the formatting of a very large amount of newsletters I forwarded. The grouping per sender was great, but not really worth it if many newsletters end up unreadable. Considering Newsblur's solution relied on setting up (sender/subject) filters on your email provider, I just kept doing that, but instead of forwarding to Newsblur, I now direct them all to a separate folder. Lost the grouping per sender, but I honestly didn't explore an alternative too much. Even if Newsblur didn't mess with the newsletters' HTML and displayed them as GMail does, it was just too much of liability to blindly forward emails to a third party service like that: many companies do obnoxious things like send transactional emails from the same address as their newsletters, or blur the line between what is bulk and targeted mail, and I'd rather not have things like emails with flight information and other random tidbits of personal data floating around in someone's MongoDB.
- jacquesm 5y agoThe big takeaway here is that you simply should not use any technology on an internet facing server that you are not 100% committed to. There is no such thing as a 'casual' SaaS, you either dedicate the time and the effort required to intimately familiarize yourself with the crappy little details of all the tools you are using as well as their caveats or you will end up getting hacked. That said, software suppliers have a serious responsibility to choose sane defaults, especially for security related items. If that inconveniences the users to the point where they have to explicitly overrule the safe settings and that reduces adoption then so be it, that's a small price to pay. Failure to do so will make those suppliers accomplices in all future hacks due to their lack of respect for reality: the internet is a hostile place and anything that can end up facing the unfiltered net will eventually do just that. Finally, we will eventually end up with a regulated internet because of all these script kiddies and other wannabe hackers, where just like in the real world you'll need a permit to operate a server, mandatory pentests and so on. The likes of AWS already make it a bit harder to expose an insecure server to the net by checking for common configuration errors such as the one that caused this particular failure.
- cunthorpe 5y ago> The likes of AWS already make it a bit harder to expose an insecure server Has this changed recently? S3 was a huge part of data leaks a few years ago, and that's basically a managed server.
- jffry 5y agoIndeed, AWS has both made it more difficult to create a world-readable S3 bucket, and made it much more obvious in the UI. Some things I've noticed just clicking around the S3 console right now: - There is an account-level S3 setting to instruct S3 to ignore public access grants in all buckets in that account (i.e. no matter how bucket is configured, public access is impossible). - The list of S3 buckets in the S3 console homepage includes a prominent column saying if a bucket is private or not. Buckets that allow public access have a warning icon and red color in this column to make them stand out. - When creating a new bucket, the default is "Block Public Access settings for this bucket". If you change this, it gives you a warning and asks you to check a box saying "I acknowledge that the current settings might result in this bucket and the objects within becoming public" - When editing a bucket's access control list, if you grant public access, it asks you to acknowledge "When you grant access to the Everyone or Authenticated users group grantees, anyone in the world can access the objects in this bucket." - Whenever you are looking at file listings or settings of a bucket with a public access rule in its ACL, the S3 console includes a prominent red "Publicly accessible" next to the bucket name in the top nav on every page IMO, this is significantly better than how it used to be, and helps reduce or, with the account-level setting on, fully-eliminate accidentally public buckets. The reality is that proper cross-account IAM role based access is still a little tricky to set up and difficult to test without coordinating with the other party, which means that this won't stop people looking to transfer data to some other account from making a bucket public and assuming it's OK as long as there is an obscure name for the bucket.
- traskjd 5y agoI saw this mentioned on twitter yesterday, Microsoft Attack Surface Analyzer, an open source toolset for seeing what config changed when performing software installation. https://github.com/microsoft/AttackSurfaceAnalyzer https://github.com/microsoft/AttackSurfaceAnalyzer Interesting to see such a strong example of where tools like this could help the very next day. Note: I haven't used this yet, just saw it and made a note.
- cusack 5y agoPlz don’t be lame
- eecc 5y agoAccidentally (or because of some other tool’s poor design) open up a server to the WWW. But unauthenticated unfettered remote admin access by default?! Is it still a thing for Mongo? Oh my! I guess that’s what you get when the “conversion funnel” guys take over “engineering”
- threeseed 5y ago> Is it still a thing for Mongo? No. Hasn't been for years. > I guess that’s what you get when the “conversion funnel” guys take over “engineering” That's also what you get when you post rubbish without even bothering to check.
- eecc 5y agoIs it? https://docs.mongodb.com/manual/administration/security-checklist/ https://docs.mongodb.com/manual/administration/security-chec... No matter how you twist it, it doesn't have the same meaning as the common English phrase: "enabled by default". And mind your manners...
- pas 5y agoThe convenient "mongo" docker image is still default open.
- eyelovewe 5y agoI must be a bit out of the loop, what’s the usecase to run a database inside Docker? What is the general trend for Dockerizing everything based upon? Are we not already largely running in virtualized hypervisor instances on our clouds and do folks actually run multiple contained apps on one cloud instance? Not referring to using ones cloud providers scalable Kubernetes systems, of course, as I see where Docker comes in to play in that case.
- tomhoward 5y agoGah, I know just how easily this can happen. About a year ago, an app I run had grown to the point where a Linode setup wasn't adequate/cost-effective enough and I migrated it to a multi-server dedicated environment with a Redis Docker container handling queue processing and caching between the machines. I presumed the UFW rules I'd set would protect it from the outside world. It all seemed to be working fine when I went to bed, then when I woke up in the morning, someone had found the open Redis port and had set up a replication node and was streaming all the data to themselves. Super-luckily it wasn't handling anything sensitive (just weather data in a small farming region in Australia) but boy oh boy did it hit me just how bad it could have been if it was handling sensitive data for a lot of users. So I feel for these guys; I'm no security expert but I've been running web apps on Linux servers for nearly 20 years and have never had a breach before, so I feel like it's a pretty easy mistake to make.
- pingec 5y agoI am super scared of accidentally exposing a port to the internet. Is there a service / tool that I could provide with a list of all my public IP machines and it would keep port scanning them periodically, sending me a report of all open ports by email each month and sending me an email each time a new port becomes open to the public internet?
- schipplock 5y agoI use "nmap" for this. $ nmap example.com PORT STATE SERVICE 80/tcp open http 443/tcp open https 1119/tcp closed bnetgame 1935/tcp closed rtmp
- exciteabletom 5y agoA more complete answer using nmap and cron: MAILTO="youremail@yourdomain.com" */30 * * * * nmap yourdomain.com | grep open > nmap.log.tmp; diff nmap.log nmap.log.tmp; mv nmap.log.tmp nmap.log
- bauruine 5y agoShodan has a monitor feature. [0] I haven't used it myself but the description looks like it's about what you want. [0] https://monitor.shodan.io/ https://monitor.shodan.io/
- jam48 5y agoI think that shodan.io can do this, if you give it an IP it will monitor it and email you about services it finds.
- blackcat201 5y agoI am not familiar with MongoDB but is there a mechanism in which white/blacklist base on IP, just like pg_hba in Postgresql which blocks/allows only certain IP access? Few years ago when I am still using MongoDB there's only basic authentication method(user password) which blocks unwanted access. I wonder if there's anything new now
- jd_mongodb 5y agoMongoDB 3.6, which was released in November 2016 defaults to listening on localhost only. A user must explicitly configure listening on a public IP address. https://docs.mongodb.com/manual/release-notes/3.6-compatibility/#std-label-3.6-bind_ip-compatibility https://docs.mongodb.com/manual/release-notes/3.6-compatibil...
- robotmay 5y agoWhen running inside a docker container this won’t be much use though as the container handles the port forwarding. It would be a much better default to ensure authentication by default, considering how widespread exploiting of this has become with bots.
- kdmytro 5y agoYou don't have to bind to 0.0.0.0:[port]. If you want the server to remain accessible only locally, bind the container to 127.0.0.1:[port]. Docker is not preventing anyone from doing this.
- robotmay 5y agoYeah that's all fine and dandy, but the docker default is to bind to 0.0.0.0, so it really should be taken into account. I honestly would have to go and look up the flags needed to change the bind address, but I know the port ones (as I'm sure do many people who copy/paste docker lines from random repos), so it's still insecure for a common configuration/setup. I've never quite understood the opposition to just shipping mongodb with authentication on by default. What sort of use-case does it solve by not requiring it, and is it worth all the bad publicity every time this crops up in a new exploit report?
- devit 5y agoI guess targeting MongoDB is a very effective strategy for the blackhats since the fact itself that someone is running MongoDB is a reliable indicator that their overall technical ability, thus including their ability to design and implement a secure system, is significantly lower than average.
- deleted 5y ago[deleted]
- jbverschoor 5y agoAhh.. insecure defaults. Such a nice thing. I don't understand why this is still used. But I really can't understand the people who are advocating this, and claim that you're stupid for not hardening everything. Insecure software defaults = bad software.
- lbriner 5y agoBeing a big fan of containerisation for many reasons, for me, the default open-networking is the biggest footgun of the lot. Docker Swarm seems to be better for treating networks as principle objects whereas K8S requires that you install custom network drivers, otherwise you only seem to get namespaces for protection (i.e. no protection) within a cluster. For me also, the tooling to actually see what is happening at the network level, what DNS has been assigned, what can and can't route is not easy to identify even though I understand a reasonable amount about the theory. Even an obvious question like, "if we are sharing a registry between development and production clusters, does that introduce a vulnerability?" doesn't have an obvious answer. VLANs are great but again, they don't seem to exist in K8S by default and we already read that Docker was punching its own holes in firewalls anyway. Maybe the default for all of these orchestrators should be private networks unless you specifically open them up otherwise I can see why people might recommend running DB servers on VMs with more obvious attack surfaces.
- dddddddddddd 5y agoIn production never ever use ufw or firewalld. Learn a few rules in iptables and that all or create a simple script. The problem with ufw or firewalld(not only with this 2) they add so many rules in iptables, that a human can do very easy mistakes or the rules are not on your control.
- mercora 5y ago... even if you handcraft your rules, docker will still, by default, add its own rulesets... ive helped people bitten by this on occasions mostly because it changes the default forward policy to drop... its also really hard to manage this concurrently... i.e if you need to reload your own rulesets you need to tell docker to reinsert its own afterward or make sure you run after docker did its thing... oh and of course you should not just save your current rulesets either in order to leave the docker rulesets out of yours and still make sure you know how their rules will affect yours then... its horrible...
- couponplusdeals 5y agoShop for yourself or the women in your life at one of the world’s best clothing, lingerie, body are, and accessories retail brands. https://www.couponplusdeal.com/victoriassecret-ksa-coupons https://www.couponplusdeal.com/victoriassecret-ksa-coupons
- lawwantsin17 5y agoAll good advice, but Docker is a garbage fire if it's automagically opening up your system to abuse. How about Delete Docker?
- EricE 5y agoIf you have backups there is nothing to hold hostage. I wish people would refrain from bombastic titles like these.
- EMM_386 5y agoThat's not true, they have your private user data and you may not want them releasing that to the world.
- mackrevinack 5y agohmm reminds me i must switch to the self-hosted version of newblur soon. i only use it when im at home so i dont rely need anything to be stored in the cloud