4 ms·
Amazon Kinesis Available For All Customers
- conflagration 13y agoThis looks interesting at first glance, but to be honest, I don't fully get it from the description. Is it some kind of Storm/CEP/Graphite combination? Where does the actual "processing" happen? Do you have rent additional EC2 resources for your workers? It looks like a potentially powerful and interesting service, but the documentation could be a little more clear and maybe include an example of a DAG layout for a typical application.
- Kequc 13y agoWhen I think of streaming my first thought is that this is a cloud service that processes audio and video in real-time. But none of the example use cases mention that is a possibility. I'd love the ability to stream and convert audio then forward converted files to S3 in a simple way. Currently it involves running an EC2 instance which only converts after the audio is received in full. Then when it's done, transferring it to S3 is... slow.
- n3thin 13y agoyou might need to look into http://aws.amazon.com/elastictranscoder/ http://aws.amazon.com/elastictranscoder/ if it can satisfy your use-case
- Kequc 13y agoThe audio file must exist somewhere, it offers no live audio support. So the bottleneck would be at the beginning rather than the end.
- theg2 13y agoElastic Transcoder doesn't support this natively, we ran into this issue so ended up rolling our own solution as the commercial versions were incredibly expensive for a non-profit.
- warptera1 13y agoTake a look at the Zencoder (YC W10) API: http://zencoder.com/en/ http://zencoder.com/en/ It's since been acquired by Brightcove. [Disclaimer: I work there.]
- bgruber 13y agoamazon says real-time, but i'm 99% sure they mean (comparatively) low latency.
- dzhiurgis 13y agoHow many visitors you'll need to have to make some sense out of their behaviour?
- nnx 13y agoIt sounds exciting but I don't fully get what this is good for. It sounds like a stream (of "records") on this would be the equivalent of a queue and messages in Amazon SQS? only Kinesis is lower latency and billing is both capacity-based and quantity-based rather than only quantity-based in SQS?
- socialist_coder 13y agoI can't wait to rip out my "client => server => SQS => queue processor => S3 => redshift" flow and turn it into a "client => kinesis => redshift" flow. Web scale analytics capturing just got way easier.
- itsmeduncan 13y agoWe just wrote something very similar to this using Kafka which replaced a very similar system you have. Do it. We would have used Kinesis if it was around.
- metabrew 13y agoAs I understand it, this is very similar to Linkedin's Kafka. To understand how you would use it, read some of the kafa docs: https://kafka.apache.org/documentation.html#introduction https://kafka.apache.org/documentation.html#introduction
- dkersten 13y agoFrom my reading of it (and my use of Kafka in the past, as an input to Storm), Kafka is a message queuing system, where Kinesis is a message processing system. That is, Kafka is more similar to SQS or RabbitMQ and Kinesis is more similar to Storm or S4. Of course, I haven't used kinesis and have only glanced at the docs, so I could be wrong - it could very well be that Kinesis is the glue between data input and data processor rather than the data processor itself, in which case, yeah, it looks a lot like Kafka.
- lhaussknecht 13y agoLooks similar to my logstash -> elastic search -> kibana workflow.
- 6cxs2hd6 13y agoI wish Amazon hadn't given up even trying to do RESTful APIs. Nowadays they just shrug and bang it out RPC style (payload is JSON instead of XML, but the song remains the same). e.g. How do you delete a stream? Is it a DELETE request for, say, /streams/<stream-id> ? No, it's a POST to "/" with an entity: { "StreamName": "string" } Sigh.
- dalore 13y agoNot all proxies support DELETE and PUT. But you're right, it should support both.
- mfenniak 13y agoAmazon success is a strong indicator: RESTful APIs don't really matter. You can build a very successful business without focusing on the format of your APIs, as long as you build something valuable.
- atwebb 13y agoIt helps if you're already one of the biggest players. A smaller company may be able to make a more attractive offering with REST.
- chollida1 13y agoDoes anyone have any experience using this technology? I'd assume it works best for post processing or pseudo real time data feeds( ie within a couple of seconds as opposed to sub 20 millisecond reactions). Where do people usually store the processed data from this system? I can see something like this being useful for things like post trade analytics or post day analyzing the efficiency of an given algo. We use an inhouse CEP to do this sort of work currently but I'd rather rent than buy servers for this offline processing. By the way, if anyone is in the Toronto area and interested in working for a hedge fund my contact info is in my profile