3 ms·

> then we will email you once the page is ready. Ah, I completely missed this part. Long day. > Ya sorry, brand new and working on the profile to improve it a

> then we will email you once the page is ready. Ah, I completely missed this part. Long day. > Ya sorry, brand new and working on the profile to improve it and the image bit. Don't worry, so far I like the site quite a bit, especially since there is much more about ancient mesopotamia than I had thought!

bwb 2y ago
Sweet thanks! I am going to work a ton this winter to improve topic/genre accuracy, as I think we have more on Mesopotamia but the system is struggling to tag it correctly. We are using an older NLP/ML system that isn't working as well as newer ones. Update coming soon.
- dunefox 2y ago
  Great to hear. Also, a very interesting topic for me since this is close to my masters degree and current job. For a niche topic like ancient mesopotamia I was surprised. Is there a blog post or something about how the site works behind the scenes?
  - bwb 2y ago
    Very cool! What is your job? I blog every 2 to 3 weeks here about building it: https://build.shepherd.com/ https://build.shepherd.com/ But I am not a developer, so it only goes into things lightly, like our topic system with NLP/ML and other topics. (hoping to be a dev in a few years enough to work on it as well) Here are a few, are these interesting or too mundane? https://build.shepherd.com/p/a-big-focus-for-2024-improving-genre?utm_source=publication-search https://build.shepherd.com/p/a-big-focus-for-2024-improving-... https://build.shepherd.com/p/building-shepherd-updated-topic-system-is-live-1217414?utm_source=publication-search https://build.shepherd.com/p/building-shepherd-updated-topic... https://build.shepherd.com/p/building-shepherd-topic-pages-nlp-and-computer-magic-699369?utm_source=publication-search https://build.shepherd.com/p/building-shepherd-topic-pages-n... https://build.shepherd.com/p/sneak-peek-at-genre-and-age-group https://build.shepherd.com/p/sneak-peek-at-genre-and-age-gro... It is python / Django on the backend, nothing crazy I think.
    - dunefox 2y ago
      Currently, I'm working as a Data Scientist on document retrieval/text matching, and I have a masters degree in a topic close to computer science and linguistics. > Here are a few, are these interesting or too mundane? Thanks, I'll have a look. It's quite interesting, even if it's not very technical, since recently I started thinking about building a book trade or selling network for close friends and their friends (invitation based). > It is python / Django on the backend, nothing crazy I think. A solid choice, I think. Do you use something like openlibrary.org as well?
      - bwb 2y ago
        Nice! That is a very cool idea! I've been thinking about something like that to help fund the website, which is a network of book trades, and you get credit for a new used book for every trade. Although I was going to charge $10 a year or $1 a book to go 80% toward authors and 20% toward the website. > Do you use something like openlibrary.org as well? When I last looked in 2020, the data quality on open library was really bad, so I didn't use them. We created every book recommended in our system manually, as we needed to find high-quality cover images, and it was an easy way to start. Eventually, we licensed data from the Nielsen API. It has not been a great experience, but it works "ok." I also looked at Ingram and Bokwer. Now things are 90% automated, but we still have to source a high quality cover for each book as the book APIs have such small images. I am hoping to expand our book DB in 2025 to all books for a lot of the new features. I am going to look at Open Library again then, as it might work better for that limited functionality. I also could have been too pedantic back in 2020 when i looked at them :)
        dunefox 2y ago
        > That is a very cool idea! I've been thinking about something like that to help fund the website, which is a network of book trades, and you get credit for a new used book for every trade. Although I was going to charge $10 a year or $1 a book to go 80% toward authors and 20% toward the website. Sounds like a nice way to generate some funding, especially as you already have a number of users on your site to start with. I have no idea how much bureaucracy it takes to be able to take 5% of each transaction or so, so I'll probably leave mine completely free - if I ever get the time to implement it, that is. > When I last looked in 2020, the data quality on open library was really bad, so I didn't use them. > Now things are 90% automated, but we still have to source a high quality cover for each book as the book APIs have such small images. Not great, since that would have been a starting point for me. ;) Cool though, that you can invest into a custom database, since you can tailor it to your task and aren't relying on potentially bad data. > I am hoping to expand our book DB in 2025 to all books for a lot of the new features. Is there a public roadmap somewhere for the features? > I also could have been too pedantic back in 2020 when i looked at them :) Maybe, maybe not. It takes huge effort to go back and improve a system relying on bad data, especially if it's already a certain size. I know from experience, as my company just did that with a large internal catalogue. Not fun.

bwb 2y ago

Sweet thanks! I am going to work a ton this winter to improve topic/genre accuracy, as I think we have more on Mesopotamia but the system is struggling to tag it correctly. We are using an older NLP/ML system that isn't working as well as newer ones. Update coming soon.

dunefox 2y ago
Great to hear. Also, a very interesting topic for me since this is close to my masters degree and current job. For a niche topic like ancient mesopotamia I was surprised. Is there a blog post or something about how the site works behind the scenes?
- bwb 2y ago
  Very cool! What is your job? I blog every 2 to 3 weeks here about building it: https://build.shepherd.com/ https://build.shepherd.com/ But I am not a developer, so it only goes into things lightly, like our topic system with NLP/ML and other topics. (hoping to be a dev in a few years enough to work on it as well) Here are a few, are these interesting or too mundane? https://build.shepherd.com/p/a-big-focus-for-2024-improving-genre?utm_source=publication-search https://build.shepherd.com/p/a-big-focus-for-2024-improving-... https://build.shepherd.com/p/building-shepherd-updated-topic-system-is-live-1217414?utm_source=publication-search https://build.shepherd.com/p/building-shepherd-updated-topic... https://build.shepherd.com/p/building-shepherd-topic-pages-nlp-and-computer-magic-699369?utm_source=publication-search https://build.shepherd.com/p/building-shepherd-topic-pages-n... https://build.shepherd.com/p/sneak-peek-at-genre-and-age-group https://build.shepherd.com/p/sneak-peek-at-genre-and-age-gro... It is python / Django on the backend, nothing crazy I think.
  - dunefox 2y ago
    Currently, I'm working as a Data Scientist on document retrieval/text matching, and I have a masters degree in a topic close to computer science and linguistics. > Here are a few, are these interesting or too mundane? Thanks, I'll have a look. It's quite interesting, even if it's not very technical, since recently I started thinking about building a book trade or selling network for close friends and their friends (invitation based). > It is python / Django on the backend, nothing crazy I think. A solid choice, I think. Do you use something like openlibrary.org as well?
    - bwb 2y ago
      Nice! That is a very cool idea! I've been thinking about something like that to help fund the website, which is a network of book trades, and you get credit for a new used book for every trade. Although I was going to charge $10 a year or $1 a book to go 80% toward authors and 20% toward the website. > Do you use something like openlibrary.org as well? When I last looked in 2020, the data quality on open library was really bad, so I didn't use them. We created every book recommended in our system manually, as we needed to find high-quality cover images, and it was an easy way to start. Eventually, we licensed data from the Nielsen API. It has not been a great experience, but it works "ok." I also looked at Ingram and Bokwer. Now things are 90% automated, but we still have to source a high quality cover for each book as the book APIs have such small images. I am hoping to expand our book DB in 2025 to all books for a lot of the new features. I am going to look at Open Library again then, as it might work better for that limited functionality. I also could have been too pedantic back in 2020 when i looked at them :)
      - dunefox 2y ago
        > That is a very cool idea! I've been thinking about something like that to help fund the website, which is a network of book trades, and you get credit for a new used book for every trade. Although I was going to charge $10 a year or $1 a book to go 80% toward authors and 20% toward the website. Sounds like a nice way to generate some funding, especially as you already have a number of users on your site to start with. I have no idea how much bureaucracy it takes to be able to take 5% of each transaction or so, so I'll probably leave mine completely free - if I ever get the time to implement it, that is. > When I last looked in 2020, the data quality on open library was really bad, so I didn't use them. > Now things are 90% automated, but we still have to source a high quality cover for each book as the book APIs have such small images. Not great, since that would have been a starting point for me. ;) Cool though, that you can invest into a custom database, since you can tailor it to your task and aren't relying on potentially bad data. > I am hoping to expand our book DB in 2025 to all books for a lot of the new features. Is there a public roadmap somewhere for the features? > I also could have been too pedantic back in 2020 when i looked at them :) Maybe, maybe not. It takes huge effort to go back and improve a system relying on bad data, especially if it's already a certain size. I know from experience, as my company just did that with a large internal catalogue. Not fun.