9 ms·

I am a solo founder of a website monitoring SaaS [0]. Theoretically, my uptime should be higher than that of my customers'. Here are a few things that I found h

by jmstfv 7y ago

I am a solo founder of a website monitoring SaaS [0]. Theoretically, my uptime should be higher than that of my customers'. Here are a few things that I found helpful in the course of running my business: * Redundancy. If you process background jobs, have multiple workers listening on the same queues (preferably in different regions or availability zones). Run multiple web servers and put them behind a load balancer. If you use AWS RDS or Heroku Postgres, use Multi-AZ deployment. Be mindful of your costs though, because they can skyrocket fast. * Minimize moving parts (e.g. databases, servers, etc..). If possible, separate your marketing site from your web app. Prefer static sites over dynamic ones. * Don't deploy at least 2 hours before you go to sleep (or leave your desk). 2 hours is usually enough to spot botched deploys. * Try to use managed services as much as possible. As a solo founder, you probably have better things to focus on. As I mentioned before, keep an eye on your costs. * Write unit/integration/system tests. Have a good coverage, but don't beat yourself up for not having 100%. * Monitor your infrastructure and set up alerts. Whenever my logs match a predefined regex pattern (e.g "fatal" OR "exception" OR "error"), I get notified immediately. To be sure that alerts reach you, route them to multiple channels (e.g. email, SMS, Slack, etc..). Obviously, I'm biased here. I'm not gonna lie, these things make me anxious, even to this day (it used to be worse). I take my laptop everywhere I go and make sure that my phone is always charged. [0] https://tryhexadecimal.com https://tryhexadecimal.com

arunpn123 7y ago
Nice website jmstfv. I have a question about how customers trust your monitoring saas. Say I have a website/service, my deployment config would probably be similar to what you have. If AWS goes down, both mine and your monitoring saas will go down. Can you please share some details about your architecture that makes it more reliable? I am curious about what sort of customers are you targeting.
- jmstfv 7y ago
  Background workers are geographically spread out across multiple AWS regions: https://tryhexadecimal.com/docs/workers#locations https://tryhexadecimal.com/docs/workers#locations It is highly unlikely that all of them will be knocked down at the same time. If workers suspect that there is something wrong with your website, they will check it from all locations at once. Website is only marked as down if more than half of them can agree on that. That alone knocks down most of the false positives: https://tryhexadecimal.com/docs/uptime#sequence-and-frequency https://tryhexadecimal.com/docs/uptime#sequence-and-frequenc...
miemo 7y ago
Your website looks awesome, I really like the clean design. Have you considered a trial extending to their first outage notification, rather than 14 days? I would imagine if they don't see an outage in 14 days they're not seeing the value. However if their trial expired following their first 'site is down' reminder - that's where your value comes in. Just my $2c
- jmstfv 7y ago
  That's really interesting, it never occurred to me before (because I've never seen such thing), but it makes sense. A lot of work happens in the background, and folks don't see the value until their website is down or their certificate is about to expire. I'll investigate that. Appreciate the suggestion.
no_wizard 7y ago
Honestly I’ve worked on teams with over 30 engineers that I wished abided by half this (particularly about the prefer static over dynamic bit This is just straight up good advice
tnolet 7y ago
This is is scary. This is exactly how I run my (also monitoring) SaaS and it shows that the OP has learned how to minimize risk and prepare for the worst. You are married to your mobile & laptop. I also shows there is no free lunch: you need to invest in redundancy.
- jmstfv 7y ago
  From time to time I ponder about all the terrible things that could happen: what can go wrong will probably go wrong so it helps to be (kinda) prepared.
  - tnolet 7y ago
    Yes, however you are extremely and acutely aware of this. Your business depends on it. This is not a long term solution though. You need to add people and take a break at some stage.
  - o-__-o 7y ago
    As a single owner (of also a monitoring SaaS) I am currently putting out a fire where my primary datacenter died. I have suffered data loss even with precautions I had in place, one server lost all filesystems which took out my git repos. I have backups, I have clones. I've still been in partial outage for 4 days and will be fully up tomorrow when I literally drive my servers to a new DC. Surprisingly I have slept 8 hours every night and I'm not worried. I've been in contact with my customers and provided solutions to keep them alive. If they leave they're going to leave, nothing I can do. I am looking to make sure everything is built uniformly (the server that died was the last of an old build process) and invest in scaling to the cloud in a bit more efficient and orderly manner.
    - deleted 7y ago
      [deleted]
rurp 7y ago
This is a great post, thanks. On your last point, do you mind sharing what you use to monitor the logs and send out alerts?
- jmstfv 7y ago
  I send my logs to Papertrail and they have a feature that will notify you if your logs match a predefined pattern. Some of the patterns that I use: * Account deletion (DELETE FROM \"accounts\") * New successful sign up (INSERT INTO "accounts") * New signup attempt (Started POST "/signup") [0] * Fatal Exception Error ("fatal" OR "exception" OR "error") * Warning ("warn") I do share my costs and tools publicly, in case if you're interested: https://tryhexadecimal.com/costs https://tryhexadecimal.com/costs [0] I got hit by a botnet attack couple weeks ago, so I keep a tab on their activity (https://news.ycombinator.com/item?id=21327416 https://news.ycombinator.com/item?id=21327416)
krat0sprakhar 7y ago
Offtopic: Really love how clean and readable the website is! Great work!
- apankrat 7y ago
  Indeedly! Very clean and simple, but not trivial or bare. Really good job, jmstfv.
  - vmurthy 7y ago
    From this link [0], it appears that the site uses Netlify. I'd love for the OP to point us in the direction of the theme and/or setup details as I am fairly new to Netlify [0]https://tryhexadecimal.com/costs https://tryhexadecimal.com/costs
    - jmstfv 7y ago
      It is just static HTML and CSS files, with sprinkles of JS when necessary. I don't like using third-party frameworks or themes because they tend to get in my way. I designed it all from scratch. I've been experimenting with functional (atomic) CSS lately and built a small framework that ended up using in Hexadecimal (if you peek at the source code, you can see weird classes like m-0, p-0, etc..). I liked it, but one should be careful in not overusing it, otherwise, it is an unmaintainable mess, yet again. Below, I linked my Netlify config [0]. My static files live in the src/ folder, so you should probably change that if you have them in a different folder. If you don't know what Content-Security-Policy (CSP) or Strict-Transport-Security (HSTS) headers do, leave them out. It is easy to shoot yourself in the foot if you don't know what you're doing. [0] https://privatebin.net/?c794ba90771f6e10#GSnJJ3dnd5r3ZDGgi9RtEYegZwhFYchbPXsKJcQCrhwC https://privatebin.net/?c794ba90771f6e10#GSnJJ3dnd5r3ZDGgi9R...
randlet 7y ago
With regards to AWS RDS & EC2 multi-az deploys and cost...buy reserved instances! It is much cheaper if you can afford to pay up front for 1yr or 3 years at a time.
- jmstfv 7y ago
  I do consider that but I'm still early in this game. My AWS spending is $1.38/day on EC2 and $1.05/day on RDS, all covered by AWS credit: https://tryhexadecimal.com/costs https://tryhexadecimal.com/costs
- threeseed 7y ago
  For EC2 you really should be using spot fleets and making sure your architecture can support instances coming and going.
  - randlet 7y ago
    I need to keep exactly two instances running (in separate AZs) and will never need to scale up beyond that...is it still worth investigating spot fleets in this case?
daniel_iversen 7y ago
> Monitor your infrastructure and set up alerts [..] "fatal" OR "exception" OR "error" I almost have the regex "fatal|invalid|unknown|error|except|critical|cannot" in muscle memory many years after having last had to type it - must have typed it thousands of times tailing and grepping logs :- )
- dTal 7y ago
  That's a smell. If you have to type a mouthful like that more than twice, put it in a script.
- unoti 7y ago
  Instead of having a Regex that searches for error|critical|except it’s a good thing to have a log level in your logging infrastructure, so that you can query for example log level=2 and get all the bad things. It takes a bit to work this into the code and infrastructure everywhere but it’s worth it.
- cutler 7y ago
  In Ruby: %w[fatal invalid unknown error except critical cannot].any? {|word| 'My critical error message'.include? word} ... may be better than a regex if it's just a list of literal strings. Javascript also has something similar.
- xxpor 7y ago
  watch out for someone writing [FATAL] in a form somewhere and it getting logged :)
vmurthy 7y ago
I really wish you all the best jmstfv ! The website is obviously great but I love the fact that you have shared such great tips at https://tryhexadecimal.com/costs https://tryhexadecimal.com/costs for solo founders/ small business owners.
lettergram 7y ago
This covered most of my thoughts (also a solo founder). I also have a fallback server running that is an entire duplicate app running on my local machine (in house PC). When an error occurs it fails over to my local machine. At fail-over my local machine will also switch from the dev database to prod (connecting to the RDS instance). It'll only has read-only access to relevant data, but it'll keep the site 90% functional. Is it perfect? Hell no. Pretty hacky, but it's kept my application running twice after a failure in the past two years.
cagmz 7y ago
Just curious; what do you use Redis for? Thanks for your post!
- jmstfv 7y ago
  Jobs (e.g. uptime checks, payment processing, sending emails) are added to the job queue (Redis) and background workers pick it up from there, and churn through them asynchronously. Most of the work happens in the background, so in my case, Redis is a critical piece of infrastructure. https://devcenter.heroku.com/articles/background-jobs-queueing https://devcenter.heroku.com/articles/background-jobs-queuei...
  - cagmz 7y ago
    Ah, of course! I saw background jobs and Redis but didn't make the connection at the time.

by jmstfv 7y ago

arunpn123 7y ago

Nice website jmstfv. I have a question about how customers trust your monitoring saas. Say I have a website/service, my deployment config would probably be similar to what you have. If AWS goes down, both mine and your monitoring saas will go down. Can you please share some details about your architecture that makes it more reliable? I am curious about what sort of customers are you targeting.

jmstfv 7y ago
Background workers are geographically spread out across multiple AWS regions: https://tryhexadecimal.com/docs/workers#locations https://tryhexadecimal.com/docs/workers#locations It is highly unlikely that all of them will be knocked down at the same time. If workers suspect that there is something wrong with your website, they will check it from all locations at once. Website is only marked as down if more than half of them can agree on that. That alone knocks down most of the false positives: https://tryhexadecimal.com/docs/uptime#sequence-and-frequency https://tryhexadecimal.com/docs/uptime#sequence-and-frequenc...

miemo 7y ago

Your website looks awesome, I really like the clean design. Have you considered a trial extending to their first outage notification, rather than 14 days? I would imagine if they don't see an outage in 14 days they're not seeing the value. However if their trial expired following their first 'site is down' reminder - that's where your value comes in. Just my $2c

jmstfv 7y ago
That's really interesting, it never occurred to me before (because I've never seen such thing), but it makes sense. A lot of work happens in the background, and folks don't see the value until their website is down or their certificate is about to expire. I'll investigate that. Appreciate the suggestion.

no_wizard 7y ago

Honestly I’ve worked on teams with over 30 engineers that I wished abided by half this (particularly about the prefer static over dynamic bit This is just straight up good advice

tnolet 7y ago

This is is scary. This is exactly how I run my (also monitoring) SaaS and it shows that the OP has learned how to minimize risk and prepare for the worst. You are married to your mobile & laptop. I also shows there is no free lunch: you need to invest in redundancy.

jmstfv 7y ago
From time to time I ponder about all the terrible things that could happen: what can go wrong will probably go wrong so it helps to be (kinda) prepared.
- tnolet 7y ago
  Yes, however you are extremely and acutely aware of this. Your business depends on it. This is not a long term solution though. You need to add people and take a break at some stage.
- o-__-o 7y ago
  As a single owner (of also a monitoring SaaS) I am currently putting out a fire where my primary datacenter died. I have suffered data loss even with precautions I had in place, one server lost all filesystems which took out my git repos. I have backups, I have clones. I've still been in partial outage for 4 days and will be fully up tomorrow when I literally drive my servers to a new DC. Surprisingly I have slept 8 hours every night and I'm not worried. I've been in contact with my customers and provided solutions to keep them alive. If they leave they're going to leave, nothing I can do. I am looking to make sure everything is built uniformly (the server that died was the last of an old build process) and invest in scaling to the cloud in a bit more efficient and orderly manner.
  - deleted 7y ago
    [deleted]

rurp 7y ago

This is a great post, thanks. On your last point, do you mind sharing what you use to monitor the logs and send out alerts?

jmstfv 7y ago
I send my logs to Papertrail and they have a feature that will notify you if your logs match a predefined pattern. Some of the patterns that I use: * Account deletion (DELETE FROM \"accounts\") * New successful sign up (INSERT INTO "accounts") * New signup attempt (Started POST "/signup") [0] * Fatal Exception Error ("fatal" OR "exception" OR "error") * Warning ("warn") I do share my costs and tools publicly, in case if you're interested: https://tryhexadecimal.com/costs https://tryhexadecimal.com/costs [0] I got hit by a botnet attack couple weeks ago, so I keep a tab on their activity (https://news.ycombinator.com/item?id=21327416 https://news.ycombinator.com/item?id=21327416)

krat0sprakhar 7y ago

Offtopic: Really love how clean and readable the website is! Great work!

apankrat 7y ago
Indeedly! Very clean and simple, but not trivial or bare. Really good job, jmstfv.
- vmurthy 7y ago
  From this link [0], it appears that the site uses Netlify. I'd love for the OP to point us in the direction of the theme and/or setup details as I am fairly new to Netlify [0]https://tryhexadecimal.com/costs https://tryhexadecimal.com/costs
  - jmstfv 7y ago
    It is just static HTML and CSS files, with sprinkles of JS when necessary. I don't like using third-party frameworks or themes because they tend to get in my way. I designed it all from scratch. I've been experimenting with functional (atomic) CSS lately and built a small framework that ended up using in Hexadecimal (if you peek at the source code, you can see weird classes like m-0, p-0, etc..). I liked it, but one should be careful in not overusing it, otherwise, it is an unmaintainable mess, yet again. Below, I linked my Netlify config [0]. My static files live in the src/ folder, so you should probably change that if you have them in a different folder. If you don't know what Content-Security-Policy (CSP) or Strict-Transport-Security (HSTS) headers do, leave them out. It is easy to shoot yourself in the foot if you don't know what you're doing. [0] https://privatebin.net/?c794ba90771f6e10#GSnJJ3dnd5r3ZDGgi9RtEYegZwhFYchbPXsKJcQCrhwC https://privatebin.net/?c794ba90771f6e10#GSnJJ3dnd5r3ZDGgi9R...

randlet 7y ago

With regards to AWS RDS & EC2 multi-az deploys and cost...buy reserved instances! It is much cheaper if you can afford to pay up front for 1yr or 3 years at a time.

jmstfv 7y ago
I do consider that but I'm still early in this game. My AWS spending is $1.38/day on EC2 and $1.05/day on RDS, all covered by AWS credit: https://tryhexadecimal.com/costs https://tryhexadecimal.com/costs
threeseed 7y ago
For EC2 you really should be using spot fleets and making sure your architecture can support instances coming and going.
- randlet 7y ago
  I need to keep exactly two instances running (in separate AZs) and will never need to scale up beyond that...is it still worth investigating spot fleets in this case?

daniel_iversen 7y ago

> Monitor your infrastructure and set up alerts [..] "fatal" OR "exception" OR "error" I almost have the regex "fatal|invalid|unknown|error|except|critical|cannot" in muscle memory many years after having last had to type it - must have typed it thousands of times tailing and grepping logs :- )

dTal 7y ago
That's a smell. If you have to type a mouthful like that more than twice, put it in a script.
unoti 7y ago
Instead of having a Regex that searches for error|critical|except it’s a good thing to have a log level in your logging infrastructure, so that you can query for example log level=2 and get all the bad things. It takes a bit to work this into the code and infrastructure everywhere but it’s worth it.
cutler 7y ago
In Ruby: %w[fatal invalid unknown error except critical cannot].any? {|word| 'My critical error message'.include? word} ... may be better than a regex if it's just a list of literal strings. Javascript also has something similar.
xxpor 7y ago
watch out for someone writing [FATAL] in a form somewhere and it getting logged :)

vmurthy 7y ago

I really wish you all the best jmstfv ! The website is obviously great but I love the fact that you have shared such great tips at https://tryhexadecimal.com/costs https://tryhexadecimal.com/costs for solo founders/ small business owners.

lettergram 7y ago

This covered most of my thoughts (also a solo founder). I also have a fallback server running that is an entire duplicate app running on my local machine (in house PC). When an error occurs it fails over to my local machine. At fail-over my local machine will also switch from the dev database to prod (connecting to the RDS instance). It'll only has read-only access to relevant data, but it'll keep the site 90% functional. Is it perfect? Hell no. Pretty hacky, but it's kept my application running twice after a failure in the past two years.

cagmz 7y ago

Just curious; what do you use Redis for? Thanks for your post!

jmstfv 7y ago
Jobs (e.g. uptime checks, payment processing, sending emails) are added to the job queue (Redis) and background workers pick it up from there, and churn through them asynchronously. Most of the work happens in the background, so in my case, Redis is a critical piece of infrastructure. https://devcenter.heroku.com/articles/background-jobs-queueing https://devcenter.heroku.com/articles/background-jobs-queuei...
- cagmz 7y ago
  Ah, of course! I saw background jobs and Redis but didn't make the connection at the time.