4 ms·

Anyone know why Amazon didn't adopt existing standards like SCP / SFTP / WebDAV? I've always found the S3 APIs to be difficult to work with, especially for aut

Anyone know why Amazon didn't adopt existing standards like SCP / SFTP / WebDAV? I've always found the S3 APIs to be difficult to work with, especially for authorization and large uploads.

blasdel 8y ago
S3 buckets do not have filesystem semantics so most of that functionality would be inoperable, and also require heavyweight session-based proxies to support (which is something you can easily operate yourself)
yzmtf2008 8y ago
Because S3 is a K/V Store, not a file system.
- bcheung 8y ago
  File systems can be used as key value as well. The Key is the path, and the value is the contents of the file. Most modern filesystems also have metadata. Granted it's not hierarchical, so listing a bucket lists everything in every folder, but I don't see why that's too big of a concern. Especially since there are pseudo directories in many of the S3 tools. Many people use S3 and have a folder-like hierarchical naming convention anyways. With WebDAV, the URL is the key, and the value is the contents of the upload / download.
halbritt 8y ago
SCP, SFTP, and WebDAV are all woefully inefficient for large data transfers as a result of bandwidth delay product.* They typically use a single TCP session and transmit files serially. WebDAV does support an mput, however. S3 does a much better job of utilizing available bandwidth. It supports multipart parallel uploads and resumes nicely. In my experience, it will usually fully saturate any available upload capacity. There are other protocols that may be more efficient for high bandwidth, high latency connections, i.e. WDT from facebook or UDT, but S3 does okay. *https://en.wikipedia.org/wiki/Bandwidth-delay_product https://en.wikipedia.org/wiki/Bandwidth-delay_product
user5994461 8y ago
SCP and SFTP (SSH derivatives) are designed to run shell commands on a remote host. It's not what you want to give to customers as a file transfer method, especially not in a shared environment.

Anyone know why Amazon didn't adopt existing standards like SCP / SFTP / WebDAV? I've always found the S3 APIs to be difficult to work with, especially for authorization and large uploads.

blasdel 8y ago

S3 buckets do not have filesystem semantics so most of that functionality would be inoperable, and also require heavyweight session-based proxies to support (which is something you can easily operate yourself)

yzmtf2008 8y ago

Because S3 is a K/V Store, not a file system.

bcheung 8y ago
File systems can be used as key value as well. The Key is the path, and the value is the contents of the file. Most modern filesystems also have metadata. Granted it's not hierarchical, so listing a bucket lists everything in every folder, but I don't see why that's too big of a concern. Especially since there are pseudo directories in many of the S3 tools. Many people use S3 and have a folder-like hierarchical naming convention anyways. With WebDAV, the URL is the key, and the value is the contents of the upload / download.

halbritt 8y ago

SCP, SFTP, and WebDAV are all woefully inefficient for large data transfers as a result of bandwidth delay product.* They typically use a single TCP session and transmit files serially. WebDAV does support an mput, however. S3 does a much better job of utilizing available bandwidth. It supports multipart parallel uploads and resumes nicely. In my experience, it will usually fully saturate any available upload capacity. There are other protocols that may be more efficient for high bandwidth, high latency connections, i.e. WDT from facebook or UDT, but S3 does okay. *https://en.wikipedia.org/wiki/Bandwidth-delay_product https://en.wikipedia.org/wiki/Bandwidth-delay_product

user5994461 8y ago

SCP and SFTP (SSH derivatives) are designed to run shell commands on a remote host. It's not what you want to give to customers as a file transfer method, especially not in a shared environment.