4 ms·

+1 on good design and programming. My go to language of choice is PHP/MySQL. I've never not been able to scale a project. At one point while working at Comcast

+1 on good design and programming. My go to language of choice is PHP/MySQL. I've never not been able to scale a project. At one point while working at Comcast I prototyped a system for every error code from every cable set top box (150 million) in the country on a MySQL server instance running on a MacBook Pro pulling data from Splunk in real time. All so I could generate some png charts and embed them in Slack. I know there are limits to MySQL but so far in my career I haven't had any projects hit those limits because hardware has been getting better faster.

BonoboIO 5y ago
But my Kubernetes Cluster on AWS can handles this for only 30.000$ a month That Comcast prototype sounds amazing. PHP and MySQL can really bring you very far, i use it for nearly everything and it is so incredible easy to develop and “deploy” new versions. I use propel as ORM and it is very close to the DB and I can change schemes fast and just let the autoupdate handle the rest.
- cbg0 5y ago
  > But my Kubernetes Cluster on AWS can handles this for only 30.000$ a month A k8s cluster for something like that wouldn't cost anywhere that amount, plus you would get the advantage of being able to deploy your app in multiple availability zones. If you don't need k8s features, there are definitely simpler options that will give you redundancy and won't break the bank.
  - hparadiz 5y ago
    I only used a MacBook cause it was faster to ask IT for one than to have my ticket processed and approved by the cloud and security team which both required a write up on why I need a vm and what justification I have for needing it. Meanwhile IT is like "here you go".
ransom1538 5y ago
"know there are limits to MySQL" I have gone past mysql limits a few times. Will you go beyond its capability is a simple measure. Can your hardware write fast enough to keep up? If your hardware maxes out writing to disk 200 megs per second, welp mysql maxes out at 200 megs per second. If you need 400 megs per second of writing, well you need two instances that can write independently. That is when you are fucked. At that point you get into sharding* and doing whacky things mysql isn't meant todo. Reading things from disk, mysql's replication scales just fine.
- hparadiz 5y ago
  That's what I mean though. I've never hit those limits. These days it's easy to put together a raid 5 with NVMe disks and you'll quickly reach gigabyte per second write speeds.
  - ransom1538 5y ago
    What cloud service can you do that on? Seriously asking.
oogali 5y ago
This comment makes me unreasonably giddy. The replies are focused on MySQL (and to a lesser extent, PHP). None of that matters. The hero here is the logging system serving as seekable stream of collected events. (I’m trying hard here to not say K-fka). By consuming events from that source and being able to perform time-based seeking, you’ve offloaded (or dare I say, eliminated) your scaling pressures. Even if you commit the “sins” of no indexes on heavy queries, using MyISAM, or building a giant, memory-intensive associative array… none of it matters. You can choose to wipe your local dataset, start again in a different language, switch to another data store model, and none of your downstream consumers will know nor care (unless you blog about it). *This* type of design, that prioritizes consumer-facing resiliency and continuity, is what I look for in systems that scale.

BonoboIO 5y ago

But my Kubernetes Cluster on AWS can handles this for only 30.000$ a month That Comcast prototype sounds amazing. PHP and MySQL can really bring you very far, i use it for nearly everything and it is so incredible easy to develop and “deploy” new versions. I use propel as ORM and it is very close to the DB and I can change schemes fast and just let the autoupdate handle the rest.

cbg0 5y ago
> But my Kubernetes Cluster on AWS can handles this for only 30.000$ a month A k8s cluster for something like that wouldn't cost anywhere that amount, plus you would get the advantage of being able to deploy your app in multiple availability zones. If you don't need k8s features, there are definitely simpler options that will give you redundancy and won't break the bank.
- hparadiz 5y ago
  I only used a MacBook cause it was faster to ask IT for one than to have my ticket processed and approved by the cloud and security team which both required a write up on why I need a vm and what justification I have for needing it. Meanwhile IT is like "here you go".

ransom1538 5y ago

"know there are limits to MySQL" I have gone past mysql limits a few times. Will you go beyond its capability is a simple measure. Can your hardware write fast enough to keep up? If your hardware maxes out writing to disk 200 megs per second, welp mysql maxes out at 200 megs per second. If you need 400 megs per second of writing, well you need two instances that can write independently. That is when you are fucked. At that point you get into sharding* and doing whacky things mysql isn't meant todo. Reading things from disk, mysql's replication scales just fine.

hparadiz 5y ago
That's what I mean though. I've never hit those limits. These days it's easy to put together a raid 5 with NVMe disks and you'll quickly reach gigabyte per second write speeds.
- ransom1538 5y ago
  What cloud service can you do that on? Seriously asking.

oogali 5y ago

This comment makes me unreasonably giddy. The replies are focused on MySQL (and to a lesser extent, PHP). None of that matters. The hero here is the logging system serving as seekable stream of collected events. (I’m trying hard here to not say K-fka). By consuming events from that source and being able to perform time-based seeking, you’ve offloaded (or dare I say, eliminated) your scaling pressures. Even if you commit the “sins” of no indexes on heavy queries, using MyISAM, or building a giant, memory-intensive associative array… none of it matters. You can choose to wipe your local dataset, start again in a different language, switch to another data store model, and none of your downstream consumers will know nor care (unless you blog about it). *This* type of design, that prioritizes consumer-facing resiliency and continuity, is what I look for in systems that scale.