5 ms·

Anybody using Amazon Web Services (S3, EC2) in their startup?

14 points by mattculbreth 20y ago

mattculbreth 20y ago
This seems like the coolest thing since sliced bread. We're experimenting with S3 but I'm really interested in EC2 also. Anybody else using AWS?
- zach 20y ago
  Sure, if you're using Rails it's really easy to use attachment_fu (the new generation of the image/file storing acts_as_attachment plugin) to keep uploaded photos on S3. Also for Rails, Jamglue's technique of putting your entire static file hierarchy on S3 is great for maximimizing your app servers' bandwidth. BTW, Justin.tv runs off EC2 servers.
pg 20y ago
Practically all the current batch of YC startups are using S3.
binarybana 20y ago
I know that YouOS uses it.
nickb 20y ago
I love S3! It's so easy to get started. I have less use for EC2 since I have less computations to do. I wish I could run a database on EC2...
- zach 20y ago
  I assume they're working on a database solution to go along with S3 and EC2. Of course I have no proof they are, though.
- mattculbreth 20y ago
  Arg, I didn't notice that you can't run a database. I'm not in that program yet. Maybe I'll just keep everything on S3 as key/value pairs and then use the Queue service to make it all happen. It'd be fun anyway.
- inklesspen 20y ago
  You can't run a database? What's the point of this, then: http://developer.amazonwebservices.com/connect/entry!default.jspa?categoryID=101&externalID=519&fromSearchPage=true http://developer.amazonwebservices.com/connect/entry!default.jspa?categoryID=101&externalID=519&fromSearchPage=true
  - nickb 20y ago
    This past week I listened in on Werner Vogels' ETech 2007 speech and he said, explicitly, that EC2 is not suitable for high performance, large scale, databases. My database is more than 10Gb in size. You can't have that large image on EC2. I guess you could get away by hosting some tables there that: a) don't take up much space, b) don't change much and c) don't have to have the latest data.
    - inklesspen 20y ago
      "Each instance predictably provides the equivalent of a system with a 1.7Ghz x86 processor, 1.75GB of RAM, 160GB of local disk, and 250Mb/s of network bandwidth." I'm not seeing where you can't host that DB on EC2. It might not be the optimal platform, but my guess is that EC2 will do a pretty good job for the money.
      - igrigorik 20y ago
        1) The data is not persistent -- meaning, if your server dies, you loose your database with it. Rebooting the server gives you a clean slate. This theoretically can be addressed with a complex backup/redundancy model, but frankly, it's not worth the trouble. Alternatively, you can use a distributed filesystem based on S3, but the performance will be poor. Long story short, your database is your bread and butter, and EC2 simply can't provide the basic services required to run a DB efficiently. 2) Dynamic DNS. Extra headaches, simply not worth it. EC2 was not designed for persistent data storage, it's a compute cloud.
        gojomo 20y ago
        Re: "rebooting the server gives you a clean state" That's not quite true: the HD contents survive crashes and reboots. Only if you release the node (or lose it otherwise -- it's not guaranteed to stay with you), the HD contents are lost. I suspect lots of startups are finding it "stable enough", and the backup/redundancy planning needed manageable.
brett 20y ago
I've used S3 on a few projects and would definitely endorse it. I just got accepted to the EC2 beta last week and haven't started messing around with it.
mattculbreth 20y ago
This is just another cool thing happening to reduce the cost of new software apps. Salesforce.com has the same thing with Apex. The only bad thing there is that they tie you a bit too much to their technologies (Java/Oracle). They have adapters for other platforms but it still runs on Java/Oracle.
comforteagle 20y ago
Sure are. We're melding ec2 instances to s3 as an actual filesystem. -Steve fooworks.com
onetimeliner 20y ago
Using it on http://onetimeline.com http://onetimeline.com - EC2 for indexing nodes and web front end, s3 for data storage and awsp for webcrawler. Couldn't have got up this fast without it - 1.5 years in the making (well fast to develop a search engine anyways!)
mattjaynes 20y ago
AWS is truly amazing. I'm currently developing an open-source framework that allows theoretically infinite scalability utilizing round-robin dns, S3, EC2, S3DFS (allows mounting an S3 bucket as a local file system on multiple EC2 instances), SQLite (serverless embeddable database where each db is just a flat file). S3DFS works at the block level and has read/write caching so is really fast. However, it requires a commercial license for non-personal use. SQLite is amazing in it's power and simplicity. It will start to have issues on a high-traffic website, but I'm breaking each user into their own db file. That sounds really bad - but SQLite has a great feature that allows you to attach multiple databases together and run queries across them as if it's just one database (handy for site-wide stats, search indexing, etc). Also, since the db's are just flat-files - backing up is super-easy with tar gzip! I'm using PHP for the coding, but anticipate other languages libraries to be built to use the system as well. Help is welcome, just let me know if you're interested! I was thinking of calling it infinizon - but that sounds kinda dorky - thoughts? http://www.sqlite.org/ http://www.sqlite.org/ http://www.openfount.com/blog/s3dfs-for-ec2 http://www.openfount.com/blog/s3dfs-for-ec2 Update: I just came across http://rightscale.com http://rightscale.com which is an amazing AWS console that allows you to control almost all aspects of EC2, S3, and SQS. Too many features to list here, but definitely a must see. I've been playing around with it and it works great.
jganetsk 20y ago
Just so everyone is aware, S3 guarantees only 4 nines of relability (99.99% uptime), not 5 nines. 5 nines means the service is down, on average, 5 minutes out of a year. 4 nines would be almost an hour. That's still pretty damn good.

14 points by mattculbreth 20y ago

This seems like the coolest thing since sliced bread. We're experimenting with S3 but I'm really interested in EC2 also. Anybody else using AWS?

zach 20y ago
Sure, if you're using Rails it's really easy to use attachment_fu (the new generation of the image/file storing acts_as_attachment plugin) to keep uploaded photos on S3. Also for Rails, Jamglue's technique of putting your entire static file hierarchy on S3 is great for maximimizing your app servers' bandwidth. BTW, Justin.tv runs off EC2 servers.

pg 20y ago

Practically all the current batch of YC startups are using S3.

binarybana 20y ago

I know that YouOS uses it.

nickb 20y ago

I love S3! It's so easy to get started. I have less use for EC2 since I have less computations to do. I wish I could run a database on EC2...

zach 20y ago
I assume they're working on a database solution to go along with S3 and EC2. Of course I have no proof they are, though.
mattculbreth 20y ago
Arg, I didn't notice that you can't run a database. I'm not in that program yet. Maybe I'll just keep everything on S3 as key/value pairs and then use the Queue service to make it all happen. It'd be fun anyway.
inklesspen 20y ago
You can't run a database? What's the point of this, then: http://developer.amazonwebservices.com/connect/entry!default.jspa?categoryID=101&externalID=519&fromSearchPage=true http://developer.amazonwebservices.com/connect/entry!default.jspa?categoryID=101&externalID=519&fromSearchPage=true
- nickb 20y ago
  This past week I listened in on Werner Vogels' ETech 2007 speech and he said, explicitly, that EC2 is not suitable for high performance, large scale, databases. My database is more than 10Gb in size. You can't have that large image on EC2. I guess you could get away by hosting some tables there that: a) don't take up much space, b) don't change much and c) don't have to have the latest data.
  - inklesspen 20y ago
    "Each instance predictably provides the equivalent of a system with a 1.7Ghz x86 processor, 1.75GB of RAM, 160GB of local disk, and 250Mb/s of network bandwidth." I'm not seeing where you can't host that DB on EC2. It might not be the optimal platform, but my guess is that EC2 will do a pretty good job for the money.
    - igrigorik 20y ago
      1) The data is not persistent -- meaning, if your server dies, you loose your database with it. Rebooting the server gives you a clean slate. This theoretically can be addressed with a complex backup/redundancy model, but frankly, it's not worth the trouble. Alternatively, you can use a distributed filesystem based on S3, but the performance will be poor. Long story short, your database is your bread and butter, and EC2 simply can't provide the basic services required to run a DB efficiently. 2) Dynamic DNS. Extra headaches, simply not worth it. EC2 was not designed for persistent data storage, it's a compute cloud.
      - gojomo 20y ago
        Re: "rebooting the server gives you a clean state" That's not quite true: the HD contents survive crashes and reboots. Only if you release the node (or lose it otherwise -- it's not guaranteed to stay with you), the HD contents are lost. I suspect lots of startups are finding it "stable enough", and the backup/redundancy planning needed manageable.

brett 20y ago

I've used S3 on a few projects and would definitely endorse it. I just got accepted to the EC2 beta last week and haven't started messing around with it.

mattculbreth 20y ago

This is just another cool thing happening to reduce the cost of new software apps. Salesforce.com has the same thing with Apex. The only bad thing there is that they tie you a bit too much to their technologies (Java/Oracle). They have adapters for other platforms but it still runs on Java/Oracle.

comforteagle 20y ago

Sure are. We're melding ec2 instances to s3 as an actual filesystem. -Steve fooworks.com

onetimeliner 20y ago

Using it on http://onetimeline.com http://onetimeline.com - EC2 for indexing nodes and web front end, s3 for data storage and awsp for webcrawler. Couldn't have got up this fast without it - 1.5 years in the making (well fast to develop a search engine anyways!)

mattjaynes 20y ago

AWS is truly amazing. I'm currently developing an open-source framework that allows theoretically infinite scalability utilizing round-robin dns, S3, EC2, S3DFS (allows mounting an S3 bucket as a local file system on multiple EC2 instances), SQLite (serverless embeddable database where each db is just a flat file). S3DFS works at the block level and has read/write caching so is really fast. However, it requires a commercial license for non-personal use. SQLite is amazing in it's power and simplicity. It will start to have issues on a high-traffic website, but I'm breaking each user into their own db file. That sounds really bad - but SQLite has a great feature that allows you to attach multiple databases together and run queries across them as if it's just one database (handy for site-wide stats, search indexing, etc). Also, since the db's are just flat-files - backing up is super-easy with tar gzip! I'm using PHP for the coding, but anticipate other languages libraries to be built to use the system as well. Help is welcome, just let me know if you're interested! I was thinking of calling it infinizon - but that sounds kinda dorky - thoughts? http://www.sqlite.org/ http://www.sqlite.org/ http://www.openfount.com/blog/s3dfs-for-ec2 http://www.openfount.com/blog/s3dfs-for-ec2 Update: I just came across http://rightscale.com http://rightscale.com which is an amazing AWS console that allows you to control almost all aspects of EC2, S3, and SQS. Too many features to list here, but definitely a must see. I've been playing around with it and it works great.

jganetsk 20y ago

Just so everyone is aware, S3 guarantees only 4 nines of relability (99.99% uptime), not 5 nines. 5 nines means the service is down, on average, 5 minutes out of a year. 4 nines would be almost an hour. That's still pretty damn good.