23 ms·
Announcing Google Cloud Bigtable
- amelius 11y agoNice, but you still can't use this for your privacy-aware customers.
- ImJasonH 11y agoCare to elaborate on why not?
- amelius 11y agoStoring data in a database that is managed by a third-party is something that some customers explicitly forbid.
- hoddez 11y agoWouldn't that mean you can't use any cloud data services with any company? Or even cloud hosting? What kind of customers forbid this?
- amelius 11y ago> What kind of customers forbid this? Government entities, for instance.
- icebraining 11y agohttps://www.google.com/work/apps/government/ https://www.google.com/work/apps/government/ Google has a dedicated cloud environment for governmental agencies: http://googleforwork.blogspot.pt/2009/09/google-apps-and-government.html http://googleforwork.blogspot.pt/2009/09/google-apps-and-gov...
- amelius 11y ago> We look forward to working with governments across the country on these exciting initiatives in the months ahead. So what about foreign governments?
- SEJeff 11y agoMany in the financial services or healthcare industry. Some of those industries have requirements far in excess of both SOX and HIPAA combined due to SEC scrutiny.
- nemo44x 11y agoAre you kidding me? This is a very common requirement and one of the reasons larger institutions are building out their own clouds. It's not a requirement for everyone of course, but it's certainly a common theme.
- bluecmd 11y agoHeard of encryption?
- andybak 11y agoDepends whether he means "You can't use any SaaS IaaS datastore for privacy-aware customers" or "Google itself is less trustworthy than (AWS or [insert competitor here]). The former is a valid position - if a fairly harsh one in the current tech landscape. If the latter however - I'd like to see a more fleshed out explanation as I don't see Google being much of an outlier on this front.
- mataug 11y agoI've been quite frustrated with the google cloud platform, Just take a look at their APIs for AppEngine, CloudSQL, GCE and the sort, Its pathetic compared to their direct competitor AWS. Lets compare trying to do create an RDS instance on AWS vs Creating one on CloudSQL AWS: 1. Get AccessToken and AccessSecret from IAM 2. pip install boto 3. conn = boto.rds.connect_to_region("us-west-2") 4. db = conn.create_dbinstance("db-master-1", 10, 'db.m1.small', 'root', 'hunter2') Done ! Google 1. Get a Client Id and Client Secret 2. pip install google-api-python-client 3. Go through the OAuth Flow and run a server locally to capture the access token 4. Use the discovery api to generate a service object. Good Luck finding this in the documentation 5. use the uninspectable service object to create a cloudsql instance. The reason I don't have code for steps 3,4 and 5 is because I gave up after wasting time trying to figure this out. My point is that they've gotten into the habit of doing half assed work so I have no hopes that the've improved this time. Practically no way to automate this The only way to use this would be from the horribly slow GUIs that google provides. EDIT: I ended up using google cloud sdk cli and running the automations with subprocess.check_output(['gcloud', 'sql', 'instance' ... ])
- erlichmen 11y agoYou are doing it wrong: 1. curl https://sdk.cloud.google.com | bash 2. gcloud auth login (one time thing) 3. gcloud sql instances create db-master-1
- dantheman 11y agoThis is a bad practice: curl * | bash and should be discouraged. It's easy but it's like saying, hey I'm logged in as root all the time - it's easier.
- erlichmen 11y agoYou don't need root to install gcloud
- icebraining 11y ago
- wiradikusuma 11y agoHow is it different than Datastore? https://cloud.google.com/datastore/ https://cloud.google.com/datastore/
- Goranek 11y agodatastore is a copy of Google Megastore service. It has indexes, sql like queries, transactions.. and you don't need to run servers like with BigTable (you pay for documents and api calls only)
- SjuulJanssen 11y agoWhy would I want to manage instances?
- berns 11y agoFrom the docs, under "Cloud Bigtable and other storage options" If you need to store highly structured objects, or if you require support for ACID transactions and SQL-like queries, consider Cloud Datastore.
- deleted 11y ago[deleted]
- bduerst 11y agoI thought the same, and found this while digging through the documentation: https://cloud.google.com/bigtable/docs/#title_short_and_other_storage_options https://cloud.google.com/bigtable/docs/#title_short_and_othe... > If you need to store highly structured objects, or if you require support for ACID transactions and SQL-like queries, consider Cloud Datastore.
- eva1984 11y agoIs this like a direct competitor to DynamoDB? How about open-source solutions, like Cassandra/HBase?
- Goranek 11y agoDynamoDB competitor is more likely Datastore(because of indexes, and way of pricing)
- Blackthorn 11y agoBigtable is the original Cassandra/HBase.
- forgottenacc56 11y agoNo Python 3? Not interested. All that BigTable development, pointless without drivers. Silly Google.
- estefan 11y agoDo you actually know what hadoop & HDFS are?
- saurik 11y agoI am going to read your comment as "you should be able to use the off-the-shelf drivers for HBase for Python" (I have elided the "3" as no one uses Python 3: that must have been a typo for "2" ;P). The "APIs" that Google describes as being compatible with are for Java, not the network: "To access Cloud Bigtable, you use a customized version of the Apache HBase 1.0.1 Java client.". So, no: it seems like if you are not using Java you will need to pull apart their customized Java SDK and build your own driver.
- pdevr 11y ago"To help get you started quickly, we have assembled a service partner ecosystem to enable a diverse and expanding set of Cloud Bigtable use cases for our customers. " Any idea how the service partners were chosen?
- nivertech 11y agoInteresting if there will be Cloud Bigtable to BigQuery connector, possible using Cloud Dataflow.
- michaelwsherman 11y agoCurrently you can use the BigQuery Hadoop connector and write a MapReduce job to scan Bigtable and write everything to BigQuery. Works quickly. I'm sure dataflow support is in the works, since Google internally doesn't really use MR and therefore likely has this on the back end already. Source--I wrote one of the whitepapers on the BigTable homepage.
- rplnt 11y agoAnyone willing to be dependant on this is honestly stupid when you take into account Google's history in this area: unreliability, changes in offerings, changes in pricing, discontinuations of services, hard lock-in, bad customer service, ...
- Goranek 11y agoYou can always move to Hbase ... Google BigTable uses Hbase api.
- nemo44x 11y agoUntil the API's begin to diverge - which will happen.
- ddispaltro 11y agoRight now it's a subset of HBase, but the basic operations are unlikely to change. HBase adheres pretty well to the BigTable paper. You are intentionally forgoing things like coprocessors, and other advanced stuff listed here[1] [1] https://cloud.google.com/bigtable/docs/hbase-differences https://cloud.google.com/bigtable/docs/hbase-differences
- jpatokal 11y agoAs the blog post says, Bigtable internally runs virtually all of Google's big services. This means it's rock solid, and it's not about to get discontinued anytime soon.
- 3lux 11y agodoes anyone know of a Go client?
- Goranek 11y agoI suppose you should be able to use any Go Hbase client.
- andrewstuart 11y agoNo, but there's a Go client for AWS.
- skj 11y agohttps://github.com/google/google-api-go-client https://github.com/google/google-api-go-client is the code-gen Go client for Google APIs. It probably does not (yet) have the generated client for cloud bigtable checked in (but I'm sure it will), but you can always use it to generate a client. You pass it the API to use on the command line, it will go fetch the docs it needs to make your client, and put its source where you tell it to.
- kevinschumacher 11y agoThis uses the HBase API. You just connect to it like you would any other HBase cluster, using an HBase client. It's not like e.g., BigQuery or Datastore where you need the API client. You include a JAR and then connect to HBase like normal.
- saurik 11y agoThe website claims that you must use their customized version of the Java HBase client library: it does not claim it is network compatible, and seems to state it is API compatible with the Java API (but then describes numerous subtle differences). > To access Cloud Bigtable, you use a customized version of the Apache HBase 1.0.1 Java client.
- Goranek 11y agoBelieve it or not, it appears gcloud-golang package already contains go bigtable library :) https://github.com/GoogleCloudPlatform/gcloud-golang/commit/df65e0af2d15158fecd80eed6333e2927bf2ed54 https://github.com/GoogleCloudPlatform/gcloud-golang/commit/...
- justinsb 11y agoI think using the HBase API is a very clever move. This means that the HBase API is now supported on AWS (EMR), GCE, VMWare (Serengeti), OpenStack (Sahara), and everywhere (Hadoop, if you're willing to run it yourself). In comparing against DynamoDB (for example), you'll have to weigh a proprietary single-vendor API against an API with a good open-source implementation (that will get even better with hydrabase), yet that is also available in managed-form on all major clouds. Edit: although - ouch - the $1500 per month entry price-point does not compare well to DynamoDB's $5 per month minimum.
- turingbook 11y agoWhere is $1500 per month from? I can not find it in the pricing page.
- dudus 11y agoCost per node per hour - $0.65; Minimum number of nodes per cluster - 3 0.65 x 3 x 24 x 30 = $1404 / month And that's before any storage costs.
- turingbook 11y agoThanks!
- thesandlord 11y agoDynamoDB does not directly correlate to Cloud Bigtable, it would be more akin to Cloud Datastore (which I don't think has a minimum and has a free tier)
- cmelbye 11y agoThey're pretty explicitly saying that this isn't an entry-level database—which is perfectly OK. For example, in the docs it says that Cloud BigTable is not a good solution for less than 1TB of data. If you're storing more than a terabyte of data and need the benefits that BigTable provides, you're probably already spending at least $1,500/month on your current database.
- StevePerkins 11y agoAny information on pricing? I doubt they'd have specific prices ready to announce yet, but it would be good to at least know the DIMENSIONS by which it will be priced (e.g. per read and write, storage, etc?). Will it be accessible to "classic" App Engine front-end instances, or only meant for Compute Engine VM's and "App Engine 2.0" Managed VM's? The biggest pain point with the current Datastore is how difficult it can be to predict your costs. Also, there are weird quirks in the pricing model (e.g. "writes" used to cost more than "reads", it's more expensive to delete rows than it is to flag them as tombstoned and continue storing them indefinitely, etc). These quirks have left people with a lot of technical debt from having designed around them. If this is another database option (alongside the Datastore and CloudSQL) for "classic" App Engine apps, which aren't likely to be re-written for Managed VM's, then it might be interesting. However, if it's only for Compute Engine or Managed VM contexts, where you're not locked-in and are free to choose any technologies you want, then at this point I would need to hear some pretty amazing information on the pricing model before I could be bothered to even test it out. Google lock-in is painful... once you've gone through the trouble of breaking free from the App Engine jail, it's really difficult to even consider adding new lock-in dependencies. EDIT: Doh. You have to click through a couple of links from the original post to find it, but they have indeed posted pricing specifics already. https://cloud.google.com/bigtable/#pricing https://cloud.google.com/bigtable/#pricing Looks like it's priced by the number of VM nodes you want in your cluster, storage, and network I/O if you're using it from outside Google's datacenters. No metered pricing on "read ops" and "write ops". This model IS a significant improvement over classic Datastore pricing. Unfortunately, it doesn't look like you can use it as a Datastore-replacement on classic App Engine front-end instances... and I'm not sure that I wouldn't just use Cassandra in other contexts where I have complete control.
- B-Scan 11y agoCost per node per hour - $0.65; Minimum number of nodes per cluster - 3; SSD storage (GB/mo) - $0.17; HDD storage (GB/mo) (coming soon) - $0.026; Source: https://cloud.google.com/bigtable/ https://cloud.google.com/bigtable/
- obulpathi 11y agoPretty impressed with the performance metrics: Reads/Writes 6ms@99% compared to Cassandra 300ms for read and 10 ms for write.
- sagivo 11y agofunny they call it "open source" just because it supports other open source API.
- bbromhead 11y agoSo their benchmark of Cassandra against BigTable doesn't even match their previous benchmark of Cassandra. http://googlecloudplatform.blogspot.com/2014/03/cassandra-hits-one-million-writes-per-second-on-google-compute-engine.html http://googlecloudplatform.blogspot.com/2014/03/cassandra-hi... How did the latency for Cassandra on their cloud platform increase by 200ms from a year ago?
- ivansmf 11y agoI wrote last year's benchmark. The clusters are completely different, and so is the workload. Last year's cluster had 300 VMs, which was a much higher price point, and the workload was write only. This benchmark uses YCSB workloads A and B, which we though matches the usage we'll have on BigTable. The cluster is much smaller as well. I shared my scripts from last year, it is pretty easy (although a bit expensive) to repro the numbers. Let me check if we can share this year's benchmark scripts as well.
- bbromhead 11y agoI'm pretty surprised about the difference in latency though, throughput as you say will be different due to number of nodes. For any given replication factor in Cassandra, overhead remains the pretty much the same irrespective of whether you have 300 or 3 nodes. So should the latency. On top of that both BigTable and Cassandra use SSTables to store the data on disk (with all the compactiony goodness that goes with them), so I'm even more surprised that the difference in latency is so huge. Would love to see the scripts for the benchmarks! I don't want to take away from a great product launch and I'm sure BigTable kicks arse in certain areas that Cassandra doesn't... I'm just surprised at the differences in latency.
- gtaylor 11y agoWithout knowing a lot more about their benchmark environment this go around, these bold statements are just about useless. Let's hope further details follow. Worst case, people are going to benchmark this independently and hopefully do a better job being transparent.
- EugeneOZ 11y agoAll is cool except one thing - it's vendor lock and vendor is known for absence of customers support, often API deprecations and products shutdowns.