8 ms·
Google’s revamped Cloud Source Repositories in beta
- etaioinshrdlu 8y agoGoogle code is back? They really do indicate they can host your repo in the post!
- skj 8y agoGCP has been able to host your repo since before the turndown of Google Code. In fact, the original GCP Source Repo team and the Google Code team were the same team (though the current product has a different backend and team).
- zaphar 8y agoYep, I used to work on the Google code specific features but there was a lot of collaboration between the devs.
- prepend 8y agoThis looks interesting but impractical for me to mirror hundreds of repos here just to get better search. Has anyone played with this or similar tools that can provide search by indexing at source. It’s a little surprising that Google doesn’t just index everything public in Github, gitlab, etc. I would like a good code search solution especially because I have projects across lots of different repo servers but want developers to be able to find code regardless of repo home. A colleague indexed it with solr and that was ok, but had limited semantic ability and definitely nothing like what Google describes where your recent search history and activity is displayed to each user.
- 013a 8y agohttps://about.sourcegraph.com https://about.sourcegraph.com
- codetrotter 8y ago> It’s a little surprising that Google doesn’t just index everything public in Github, gitlab, etc. GitHub excludes a few patterns in robots.txt that means a lot of the public data is never seen by crawlers that respect robots.txt, for example: Disallow: /*/*/tree/* Disallow: /*/*/blob/* and Disallow: /*/*/commits/*/* Disallow: /*/*/commits/*?author Disallow: /*/*/commits/*?path Disallow: /*/*/branches Disallow: /*/*/tags https://github.com/robots.txt https://github.com/robots.txt They probably want people landing mainly on - user profiles - repo roots - individual issues - individual pull requests And I am glad they do, because when I search on Google, the above mentioned kinds of results are exactly what I am looking for, not individual source files etc.
- russellwolf 8y agoAuthor of the blog post here. Thanks for taking a look. I'm glad you find Cloud Source Repositories interesting. We'd love to Cloud Source Repositories more practical to use. Can you explain what is impractical for you? Is the initial process of setting up multiple mirrors? Or the routine of having to go to another site for search? (Disclaimer: I work at Google and am the PM on this product)
- prepend 8y agoThanks for responding. I work in an organization (happy to PM) with a few Github orgs and about 300 repos. Dozens of project teams work in them and use various GitHub features. I have no easy insight or overview into code across these repos without scripting. The impractical part is in cloning mirroring all of these repos to get insight from your new function because user behavior is baked into all the workflow around the Github repos. The go to another site isn’t so bad as this is a new function and not available in Github, so programmers will likely check it out when they need. But manually forming everything and syncing. It would be neat if I can import an org in one fell swoop. That being said, I do have scripting forking everything over on my list of things to do when I have cycles on the weekend.
- bryanlarsen 8y agoOne nice thing is how trivial it is to mirror a repository from Bitbucket or GitHub. I wish it was as easy to mirror a repository from Bitbucket to GitHub or GitHub to Bitbucket.
- briffle 8y agoIt would be nice if it was so easy to mirror bitbucket server, or gitlab, etc.
- mark_l_watson 8y agoVery nice. Reading that makes me miss Google’s internal dev environment. I may not much use anymore Google Search and gmail, but Google’s paid products like GCP (and Google Play Music, Movies, etc.) are awesome. Also, amazing permanent free tier levels. I need to use github for the repos for my books because they are already linked, but for my personal private repos I think I may switch.
- bogomipz 8y agoCould you share why as a former Google dev you no longer use Google search and gmail? I'm curious why you mentioned that in response to a post about cloud source repositories. Thanks.
- random_kris 8y agoI am interested in this aswell :)
- paulgb 8y agoI'm not OP, but I'm also a former Googler. I switched to DDG and Firefox after I left because I think it's important that alternatives exist. Google may be a better actor in monopoly situations than some other companies, but they're still a company trying to make profit for shareholders. (As a bonus, I find ddg+Firefox snappier than Google+Chrome)
- pjmlp 8y ago> For Java, JavaScript, Go, C++, Python, TypeScript and Proto files, you’ll see result suggestions indicating whether the match is an entity such as a class, method, enum, or field. Apparently they forgot about Flutter users.
- deleted 8y ago[deleted]
- themihai 8y agoWhy would someone move from bitbucket?
- russellwolf 8y agoAuthor of the blog post here. We've generally heard that developers move from Bitbucket to Cloud Source Repositories because they want to use other Google Cloud services and it's simpler for them to manage their source in the same place where they debug, build and deploy that code. Developers often mention that they appreciate the unified identity, IAM permissions, and integrations that Cloud Source Repositories has with Cloud Shell, Cloud Build, Cloud Debugger and others Google Cloud services. However, developers don't need to move from Bitbucket to take advantage of the utility provided by Cloud Source Repositories. You can mirror any number of your repositories into Cloud Source Repositories and take advantage of the code browser, code search, and various integrations with Google Cloud Platform. (Disclaimer: I work at Google and am the PM on this product)
- kngspook 8y agoFWIW, it'd also be great to have mirroring with Gitlab (especially since I think Google Cloud has a deal for users with them?).
- devoply 8y agoThey should list expected months until closure on the product page so people can plan accordingly. I kid, but not really.
- apoorvgarg 8y agohow many gcp products has google abandoned in total?
- theDoug 8y agoI know we deprecated the prediction API with a year's warning while bringing out Cloud ML Engine. And Cloud SQL changed a bit when the second generation came in and supported MySQL, PostgreSQL and more. Can't think of others off the top of my head, so that is two if we want to consider major upgrades "abandoned." Perception often doesn't match objective fact. (Disclosure: I work at GCP)
- jrockway 8y agoNice. Is the code search the same as what Google has internally? I do miss that. (Though honestly, the value from the internal code search was that everything was in there. So if you got an error message from a service you were a client of, you could just search for the error message in their code and see what it actually meant. "Aha, this 'optional' field that's marked 'deprecated' isn't actually optional or deprecated," most of the time ;)
- duality 8y agoYou might also want to check out https://kythe.io https://kythe.io and https://github.com/google/kythe https://github.com/google/kythe.
- romed 8y agoIt’s great that they opened Kythe but as someone who deployed it from scratch I say it’s like ordering a DIY aircraft carrier kit. Many steps before you get something useful.
- romed 8y agoHaving glanced at it a bit I don’t see how this can be based on internal code search / kythe. Kythe works by building your codes. That’s how it can find callers and specializations and whatnot. This doesn’t seem to.
- lallysingh 8y ago"It looks for matches by file names, classes, functions and other symbols, and matches the context and namespace of the symbols" Correct. Looks like it just parses and indexes the source.
- russellwolf 8y agoAuthor of the blog post here. The code search functionality is the same as what Google engineers use internally. Cloud Source Repositories uses the same indexing and retrieval technologies on the same type of infrastructure. As you mention, the more code you have, the more benefit you get from having fast search tools across your entire code base that can perform complex semantic and regular expression queries. Even with smaller codebases, I find it the fastest way to find the code I need. (Disclaimer: I work at Google and am the PM on this product)
- hjuutilainen 8y agoQuickly glanced through the blog post and the docs but didn’t find any mention of Git LFS. Does Cloud Source Repositories support mirroring LFS content?
- russellwolf 8y agoAuthor of the blog post here. Cloud Source Repositories currently supports very large repositories. The same backend scales to the needs of the Android open source project, which regularly checks in massive binary files. There are many APKs and VM images in the Git repositories we host. However, Cloud Source Repositories does not support LFS or the mirroring of LFS content. LFS is not deeply integrated into Git, creating usability problems. Because the content is not part of the object graph of the repository, you have to decide in advance to use Git LFS. You don't get the benefit in existing repositories with large files. You also can't back out --- once you're using Git LFS on a medium-sized file, you can't change your mind and instruct Git to send it inline in fetches, without rewriting history, which breaks existing clients that have cloned the repository. For the same reason, history mining commands like "git blame" and "git log -S" don't have access to the object. In addition, it complicates migrating to another host. Usually in Git, you can take out your content by running "git clone --mirror one-url and then "git -C directory.git push --mirror another-url". With Git LFS, this copies over the pointer files but not the underlying large file content and you must remember to take extra steps to instruct the new host about where the blobs are stored. Google's Cloud Source Repositories team believes that Git itself needs to deal better with large files. The first step of this work within the Git project has been partial clone, which is supported in Cloud Source Repositories and in the public Git 2.17 release (for best results, please use Git 2.19, released September, 2018). If you run "git clone --filter=blob:limit=512M <url>", files larger than 512M will be omitted from the initial clone and fetched automatically on demand when needed (for example during checkout operations). See https://crbug.com/git/2 https://crbug.com/git/2 for more details about this feature. We are continuing to work with the community on adding other features related to large file support into Git. (Disclaimer: I work at Google and am the PM on this product)
- lunch 8y agoSounds great but at this point I've been burned by Google EOLing products that I use too many times.
- client4 8y agoI was going to make this same joke. It's hard to put faith into a product when Google keeps killing them off. Wave and Google Code are two top of mind products that I missed when they were gone. Conversely, I think it's important for Google engineers to be able to try new things and publish products without having to worry about supporting them forever. Is there some sort of happy medium? Like a Google labs of products that are in various states of whimsical testing? Cloud source looks nice, and for GCP projects I'm sure it will have a nice tie in, but for now I'm 100% more likely to push code to Gitlab until it makes sense to do otherwise.
- theptip 8y agoI know that is the meme, but are there any examples of Google dropping enterprise (i.e. paid) offerings? I can think of a few free services that they have dropped, but that's a different story than dropping a commercial service that you're selling to enterprise users.
- client4 8y agoI don't know. A quick perusal of the graveyard[1] shows what I would consider mostly free services. [1] https://en.wikipedia.org/wiki/List_of_Google_products#Discontinued_products_and_services https://en.wikipedia.org/wiki/List_of_Google_products#Discon...
- toomuchtodo 8y agoMaps wasn't dropped, but the pricing shifts are similar in impact. https://news.ycombinator.com/item?id=16980166 https://news.ycombinator.com/item?id=16980166
- sorenjan 8y ago> Like a Google labs of products that are in various states of whimsical testing? They actually had precisely that, but then they cancelled it. > Google described Google Labs as "a playground where our more adventurous users can play around with prototypes of some of our wild and crazy ideas and offer feedback directly to the engineers who developed them." https://en.wikipedia.org/wiki/Google_Labs https://en.wikipedia.org/wiki/Google_Labs
- wasd 8y agoThis is pretty cool. What can we expect in terms of additional language support?
- russellwolf 8y agoAuthor of the blog post here. I'm glad you like it. Which languages would you like to see support added for? Note that search works today across all languages but the semantic understanding of source which enhances search is limited to Java, JavaScript, Go, C++, Python, TypeScript and Proto files. (Disclaimer: I work at Google and am the PM on this product)
- sharms 8y agoClojure would be a nice feature
- wasd 8y agoThanks for the post Russell. Our SaaS application is running on Rails / Typpescript frontend. We're patiently waiting for App Engine to add Standard support for Ruby before we go all in on GCP. I would be most excited to see Cloud Source support Ruby. Hope your team will consider it.
- kngspook 8y agoIn priority order, I would suggest: 1. Swift 2. Obj-C 3. Bash 4. SQL 5. Ruby 6. C#
- dabei 8y agoBeing able to search code so efficiently across the whole company’s code base is a huge productivity gain. But I found people who hasn’t worked in Google or other companies that have this capability generally don’t believe it’s a big deal.
- kngspook 8y agoWell, you need to have a fairly large codebase with a fair amount of in-house infrastructure and interdependencies before you have both the need to find code that you're not working on and have a dense enough source tree where search makes more sense than just learning the organization and navigating to where you need to go.
- joatmon-snoo 8y agoHonestly, it's not even the monorepo and tooling. Simply the ability to navigate our code IDE-style in the browser, and move back and forth between different revisions, makes a huge difference. I've been mucking around with some code in javac recently and was trying to show someone something I'd found, and browsing the openjdk mercurial repo was just so painful compared to hunting around google3.
- kyrra 8y agoFor those that haven't used Google's code search before, I believe this will be very similar to Chromium's code search[0] (which is based on Google's internal one). [0] https://cs.chromium.org/ https://cs.chromium.org/
- jdlyga 8y agoWhy would I want to host code on Google's Cloud when they shut down Google Code? What's to stop them from shutting down Google Cloud in a few years?
- tomassre 8y agoI believe the difference is this is part of GCP and their terms for GA products are that they will support for at least one year on deprecated versions / products and not just instantly remove something.
- foobaw 8y agoSimilar to OpenGrok, I'm assuming?