8 ms·

Database of all Marvel characters

96 points by uberneo 12y ago

Brashman 12y ago
How are the results sorted? When I search "Spider-man", the first result is Boomerang, and none of the results on the first page are for the Peter Parker Spider-man (result #2 is Ben Reilly Spider-man).
- VonGuard 12y ago
  Yeah, I searched for "Nimrod" and got nothing but a few other characters who reference encounters with him. He's in Marvel Universe, so clearly they're just picking and choosing who gets in here. This database is sorely lacking, and the search function is meh... Not a knock on the Node.js code, more of a knock on the Marvel API. I know we're not supposed to be negative on here, but this is a cut and dry Comic Book Guy situation.... clears throat "Worst..... database..... ever...."
  - asamil 12y ago
    This was a little project I did when Marvel's API first came out. I scraped all the data and put it in a database that allowed better searching/browsing. I probably need to periodically scrape the API again to pull in changes and additions as they add them. Hopefully Marvel will update the data and make it more complete.
    - VonGuard 12y ago
      That explains it! I really do like the presentation and the whole site. I just couldn't resist the comic book guy opportunity.
    - caseysoftware 12y ago
      How in the world is this repo compatible with their TOS - http://developer.marvel.com/terms http://developer.marvel.com/terms ?
uberneo 12y ago
Here is the Blog about it --https://orchestrate.io/blog/2014/04/08/explore-the-marvel-universe-with-orchestrate/ https://orchestrate.io/blog/2014/04/08/explore-the-marvel-un... And here is the Source Code -- https://github.com/asamiller/marvelousdb https://github.com/asamiller/marvelousdb
- Raphmedia 12y ago
  Very pertinent information. Thanks!
uberneo 12y ago
Here is the actual JSON for each comics and characters --https://github.com/asamiller/marvelousdb/tree/master/data/ https://github.com/asamiller/marvelousdb/tree/master/data/
- deleted 12y ago
  [deleted]
nutjob123 12y ago
I appreciate this being open source but why on earth would you store this information in JSON rather than a relational DB?
- simcop2387 12y ago
  Probably because the marvel comics API that this is likely pulled from gives it out that way.
- hardwaresofton 12y ago
  Generally, if you have a relatively large chunks of non-relational data, you can use a document store. I could imagine the data model for superheroes being a fit for documents - information on one super hero is not tightly coupled to another super hero. Sure you could normalize some of the abilities and what not, but the main chunk that you'll be searching by/storing is a Super Hero (or whatever you want to call that object). Also, note that document stores support somethings that you would consider relational (ex. RethinkDB supports joins), so the lines are becoming blurred these days.
  - jawngee 12y ago
    But what happens when you want to start associating them on when they appeared together or fought each other? What about the different iterations of an individual character in different continuities? (Captain America was like six different dudes right?) I think a document store is the worst way to store this kind of stuff. imho, ymmv.
    - hardwaresofton 12y ago
      What you've brought up is precisely where relational databases excel -- it's a tradeoff (as anything else, you have to decide when to use one or the other). How many relationships do you have/expect? How intermingled/related is your data? If the answer is "not much" , then maybe you should go with documents, if it's high maybe you should go with RDBMS. Also, the lines are blurred, like I previously stated: - RethinkDB does joins - Postgres has good support for storing JSON documents
nutjob123 12y ago
I appreciate this being open source but why on earth would you store this information in JSON rather than a relational DB?
runn1ng 12y ago
This looks cool and sexy, but I have to say, I still prefer the wikia database http://marvel.wikia.com/Main_Page http://marvel.wikia.com/Main_Page It's much uglier, but much more useful for any actual, ahem ahem, "research".
mhd 12y ago
Apparently they can't cope with two characters having the same name (not a rare thing in the Marvel universe): http://marvelousdb.com/character/1011256 http://marvelousdb.com/character/1011256
- zyxley 12y ago
  Yeah, comics have enough duplicates of everything (including, in some cases, multiple identical clones of the same character) that the only sane way way to handle it is to assign an arbitrary unique ID to each character.
- IndianAstronaut 12y ago
  It's like Django, can't deal with two column primary keys.
ImJasonH 12y ago
Does this adhere to the Marvel API Terms of Service?[0] Specifically, "Storage. You may not indefinitely cache Content and further agree to follow the content retention rules, as set forth here: http://developer.marvel.com/documentation/attribution." http://developer.marvel.com/documentation/attribution." This looks cool, but it seems to have just scraped the API, and I'd hate to see it taken down for violating their rules. edit: You also seem to be distributing results of the API in the GitHub repo, which might violate this: "You may not...redistribute the Content or Tools except within your Apps." [0] http://developer.marvel.com/terms http://developer.marvel.com/terms
grimmdude 12y ago
I didn't realize so many of the Marvel characters had UK citizenship.
helyka 12y ago
I'm disappointed http://marvelousdb.com/character/1017316 http://marvelousdb.com/character/1017316
coldcode 12y ago
There is one character so super he's in the database but you can't find him.
kieranajp 12y ago
First tried searching for 'Quake', then 'Daisy Johnson' [0]. Nada :( [0]: http://marvel.wikia.com/Daisy_Johnson_%28Earth-616%29 http://marvel.wikia.com/Daisy_Johnson_%28Earth-616%29

96 points by uberneo 12y ago

How are the results sorted? When I search "Spider-man", the first result is Boomerang, and none of the results on the first page are for the Peter Parker Spider-man (result #2 is Ben Reilly Spider-man).

VonGuard 12y ago
Yeah, I searched for "Nimrod" and got nothing but a few other characters who reference encounters with him. He's in Marvel Universe, so clearly they're just picking and choosing who gets in here. This database is sorely lacking, and the search function is meh... Not a knock on the Node.js code, more of a knock on the Marvel API. I know we're not supposed to be negative on here, but this is a cut and dry Comic Book Guy situation.... clears throat "Worst..... database..... ever...."
- asamil 12y ago
  This was a little project I did when Marvel's API first came out. I scraped all the data and put it in a database that allowed better searching/browsing. I probably need to periodically scrape the API again to pull in changes and additions as they add them. Hopefully Marvel will update the data and make it more complete.
  - VonGuard 12y ago
    That explains it! I really do like the presentation and the whole site. I just couldn't resist the comic book guy opportunity.
  - caseysoftware 12y ago
    How in the world is this repo compatible with their TOS - http://developer.marvel.com/terms http://developer.marvel.com/terms ?

uberneo 12y ago

Here is the Blog about it --https://orchestrate.io/blog/2014/04/08/explore-the-marvel-universe-with-orchestrate/ https://orchestrate.io/blog/2014/04/08/explore-the-marvel-un... And here is the Source Code -- https://github.com/asamiller/marvelousdb https://github.com/asamiller/marvelousdb

Raphmedia 12y ago
Very pertinent information. Thanks!

uberneo 12y ago

Here is the actual JSON for each comics and characters --https://github.com/asamiller/marvelousdb/tree/master/data/ https://github.com/asamiller/marvelousdb/tree/master/data/

deleted 12y ago
[deleted]

nutjob123 12y ago

I appreciate this being open source but why on earth would you store this information in JSON rather than a relational DB?

simcop2387 12y ago
Probably because the marvel comics API that this is likely pulled from gives it out that way.
hardwaresofton 12y ago
Generally, if you have a relatively large chunks of non-relational data, you can use a document store. I could imagine the data model for superheroes being a fit for documents - information on one super hero is not tightly coupled to another super hero. Sure you could normalize some of the abilities and what not, but the main chunk that you'll be searching by/storing is a Super Hero (or whatever you want to call that object). Also, note that document stores support somethings that you would consider relational (ex. RethinkDB supports joins), so the lines are becoming blurred these days.
- jawngee 12y ago
  But what happens when you want to start associating them on when they appeared together or fought each other? What about the different iterations of an individual character in different continuities? (Captain America was like six different dudes right?) I think a document store is the worst way to store this kind of stuff. imho, ymmv.
  - hardwaresofton 12y ago
    What you've brought up is precisely where relational databases excel -- it's a tradeoff (as anything else, you have to decide when to use one or the other). How many relationships do you have/expect? How intermingled/related is your data? If the answer is "not much" , then maybe you should go with documents, if it's high maybe you should go with RDBMS. Also, the lines are blurred, like I previously stated: - RethinkDB does joins - Postgres has good support for storing JSON documents

nutjob123 12y ago

I appreciate this being open source but why on earth would you store this information in JSON rather than a relational DB?

runn1ng 12y ago

This looks cool and sexy, but I have to say, I still prefer the wikia database http://marvel.wikia.com/Main_Page http://marvel.wikia.com/Main_Page It's much uglier, but much more useful for any actual, ahem ahem, "research".

mhd 12y ago

Apparently they can't cope with two characters having the same name (not a rare thing in the Marvel universe): http://marvelousdb.com/character/1011256 http://marvelousdb.com/character/1011256

zyxley 12y ago
Yeah, comics have enough duplicates of everything (including, in some cases, multiple identical clones of the same character) that the only sane way way to handle it is to assign an arbitrary unique ID to each character.
IndianAstronaut 12y ago
It's like Django, can't deal with two column primary keys.

ImJasonH 12y ago

Does this adhere to the Marvel API Terms of Service?[0] Specifically, "Storage. You may not indefinitely cache Content and further agree to follow the content retention rules, as set forth here: http://developer.marvel.com/documentation/attribution." http://developer.marvel.com/documentation/attribution." This looks cool, but it seems to have just scraped the API, and I'd hate to see it taken down for violating their rules. edit: You also seem to be distributing results of the API in the GitHub repo, which might violate this: "You may not...redistribute the Content or Tools except within your Apps." [0] http://developer.marvel.com/terms http://developer.marvel.com/terms

grimmdude 12y ago

I didn't realize so many of the Marvel characters had UK citizenship.

helyka 12y ago

I'm disappointed http://marvelousdb.com/character/1017316 http://marvelousdb.com/character/1017316

coldcode 12y ago

There is one character so super he's in the database but you can't find him.

kieranajp 12y ago

First tried searching for 'Quake', then 'Daisy Johnson' [0]. Nada :( [0]: http://marvel.wikia.com/Daisy_Johnson_%28Earth-616%29 http://marvel.wikia.com/Daisy_Johnson_%28Earth-616%29