9 ms·

How many lines of code is Candy Japan?

237 points by bemmu 10y ago

eriknstr 10y ago
My guess when confronted with the number of lines of Python code for the front page was 5000 SLoC. I lowballed it intentionally because I was under the suspicion that it might be lower than one might normally guess. (Don't we all take pride in keeping our SLoC count as low as we can for any given problem?) The video had some closing points, including "NoSQL sucks for reports", which was explained briefly. OP, if you could, I would like for you to elaborate a bit more on this point. I am not doubting that it's true for Google App Engine, just interested to hear about the particulars.
- stickfigure 10y ago
  I can speak to this, being a heavy GAE user and formerly a heavy MongoDB user. For a reporting system you want an ad-hoc query language, fast in-database aggregation, and joins. The GAE datastore has none of the three, and MongoDB lacks joins. These are not fatal flaws - the GAE datastore has other advantages like infinite scalability, built-in synchronous geographic replication, failover, zero maintenance, etc. But for analytics, we replicate a subset of data to Postgres. It's still cheaper to have developers write occasional replication code than to hire a DBA to maintain the database, and I never have to worry that an ill-conceived sql statement will create an incident. I've come to the conclusion that using the GAE datastore as primary datastore and replicating to specialized databases is a pretty good architecture for systems that require reliability and low-maintenance. MongoDB's aggregation framework is really painful because the query language is weird and very low-level - you have to do most query planning yourself. And without joins you hit the limits of the kinds of questions you can ask the system very quickly. FWIW, my guess was 10k lines of code.
- wodenokoto 10y ago
  In the article he writes that it sucks because you end up writing queries in python, which would have been better suited to be written in SQL. I didn't listen to the talk, so I don't know if he mentions it there as well, and you are looking for a more in-depth explanation.
unwind 10y ago
Very interesting reading, as almost always from CJ. I was impressed by the precision in the data, i.e. being able to tell exactly how many lines of code deal with "fraud detection". Perhaps there's a "frauddetection.py" file, but then there has to be someone importing it and using it, and those lines are harder to count, I'd expect. But perhaps the integrations are small enough to deal with manually.
- bemmu 10y ago
  The code is short enough that what I did was actually just go through every line of code and try to assign it to a category. It did have some which were hard, such as the import statements you mention that multiple parts might use. But I figured being off by a few dozen lines wouldn't be that important for seeing the bigger picture.
asimuvPR 10y ago
@OP Why not dump the data into postgres rather than code the schema and transactions? I had a similar issue and postgres was a godsend.
- brianwawok 10y ago
  Well he is on google cloud, which when he wrote it he used the best option. Google cloud now has cloud sql, which would likely be much better. But then he needs to code, test, and migrate.. perhaps not worth it?
  - asimuvPR 10y ago
    Yes, good points.
  - bemmu 10y ago
    It wouldn't be impossible, but probably less work overall just to continue to write little Python scripts for my reporting purposes.
  - maxmcd 10y ago
    The parent is likely referring to dumping the GAE Datastore data into a Postgres instance for reporting and other analysis. Not swapping out the primary datastore for the app. This comment elsewhere in this thread explains the strategy nicely: https://news.ycombinator.com/item?id=12613223 https://news.ycombinator.com/item?id=12613223
    - brianwawok 10y ago
      I don't think he has the scale to require NoSQL. Cloud SQL could host his data and his reporting in a single database.
wwer444453 10y ago
I work in a supermarket (in Japan). Those EAN13 codes you are printing are US only and probably reserved. I recommend that you change to any one of the in-store codes like 2*. It specifically carries no restrictions.
- jylam 10y ago
  EAN means "European Article Numbering".
  - wwer444453 10y ago
    I know. IAN and EAN are interchangeably used.
- Faaak 10y ago
  What could be the problem ?
- raverbashing 10y ago
  What problems may this cause? It's not like he's scanning this at a supermarket till
  - wwer444453 10y ago
    Nothing. It is like using reserved world IPs in your internal network. There is perfectly valid range you can use so why not shift to that?
gcatalfamo 10y ago
Thanks to this very informative blog post I also discovered https://embedd.io https://embedd.io that allows to embed HN and Reddit comments to any website. Thanks again! EDIT: I don't know why I am getting downvoted. I am NOT from embedd.io (why else would you downvote otherwise?) and just genuinely discovered the service. HN is truly toxic lately.
- jon-wood 10y ago
  You're probably being down voted because your comment doesn't really add anything to the discussion, but try not to take it too personally, people are just pushing your comment to the bottom of the page so that comments that are about the article can bubble up.
  - jnbiche 10y ago
    > You're probably being down voted because your comment doesn't really add anything to the discussion Clearly, you're incorrect, because several people have explicitly stated that they appreciated the mention of this site (and probably there are more who were glad to find out but didn't comment). `bemmu`, Candy Japan's founder, also positively responded to the comment in question. In my opinion, downvotes on HN should be reserved for malicious comments like trolls, ad hominems, off-topic spam (real spam, rare on HN thanks to the mods. not just "I don't like this comment and think it's off-topic"), factually incorrect comments, and "reddit style" humor. Users should not downvote comments they simply disagree with, on-topic promotional comments, and tangentially related comments. Tangents are a great way to discover new ideas. However, there is a small cadre of HN downvoters who disagree with me, and who rabidly downvote any comment with a whiff of being off-topic or controversial. These downvoters must spend lots of time on HN, since their downvotes always come very quickly after a comment is submitted. Luckily, they do not represent the majority of HNers who can downvote, since comments like this one typically crawl back up into positive territory after the first 30-60 minutes. Some of my comments that were initially downvoted for being "off-topic" even crawled back up to over 10 upvotes. To the GP commenter, please don't get too frustrated. It's a relatively small number of people who downvote useful comments like yours. It's true that they are among the first to vote on comments, but almost always their downvotes are reversed after some time.
Retric 10y ago
This may just be me, but I find well maintained projects tend to get sticky around 10k lines of code. Much smaller than that and every new thing is just that a new thing. Much larger and it's easer for new things to kind of get tacked on at the edges. But, around 10k the project is small enough to understand and most new code tends to result in some refactoring for shared functionality. 10k lines is also fairly close to novel length. Anyone else notice this?
- lmm 10y ago
  When working in Python I noticed that 20kloc was the point at which I stopped being able to hold a project in my head, at which point I became unable to safely refactor, which in turn meant refactoring basically stopped being worth it. Any codebase larger than that, the core ossified, and new functionality was only neally possible to add at the edges. I think I can do a lot better than that now that I'm using Scala. Certainly I don't feel like I've hit the limit yet.
  - anewhnaccount 10y ago
    This is where you need tests. Leaning on the type system can take you further, but at some point you're going to need tests with Scala too.
    - lmm 10y ago
      Disagree. Effective use of types has a much better cost/benefit than tests, and it's possible to go all the way down to 0% defect rate using just the type system (by encoding proof of correctness into the type system) if you really need to.
      - vertex-four 10y ago
        Only insofar as your types essentially become the bones of the program... which you'll then want to refactor at some point. Types are awesome. Languages like Rust give them to me without having to learn a ton of theory. However, eventually, I realise that my types have boxed me into a certain design - and I need to refactor heavily, which involves rewriting a large chunk of my types which I was relying on to show me that my code was correct.
yojex 10y ago
Great read! My guess would have been greater than 10,000. Couple of typos towards the end you might want to fix: "I don't know how I could have expect it, but somehow from the start I should have prepared for fraud. You want to at least keep an on any suspicious activity and react quickly if you start getting many chargebacks." 'expected' and 'at least keep an eye on'
- bemmu 10y ago
  Thanks!
dzdt 10y ago
Coming from an enterprise background, that number is astonishingly low.
- tobltobs 10y ago
  The Active Directory integration alone would take 5k lines.
ec109685 10y ago
It is supposing Recurly doesn't have a better fraud story. Did you look at using one one "anti fraud" as a service companies or a different vendor that does subscriptions?
- bhelx 10y ago
  At the time candyjapan had their fraud problems, Recurly didn't have very good support for these services. Since then, we've formed a partnership with Kount: https://recurly.com/press/recurly-teams-up-with-kount-to-help-subscription-based-merchants-combat-fraud/ https://recurly.com/press/recurly-teams-up-with-kount-to-hel... https://docs.recurly.com/docs/kount https://docs.recurly.com/docs/kount You get some basic protections for free but you can also integrate with your Kount account if you want to customize.
tmaly 10y ago
I totally get the point on NoSQL, I started out with Redis and some custom micro services that wrote to Boltdb for my food site. This started getting out of hand, now I am converting everything to Postgresql. I will still use Redis for some fast access/caching, but for reporting, sql is still the best way to go once you know what your data looks like.
late2part 10y ago
Yes, but how many hundreds of microservices do you have? :-)
deleted 10y ago
[deleted]
Ro93 10y ago
does he live in japan and ship from there? Or live in the uS and somehow gets the candy?
- icebraining 10y ago
  He does live in Japan: "Bemmu started Candy Japan with wife Nachi and lives in Tokushima, Japan."
deleted 10y ago
[deleted]
wkoszek 10y ago
Do you guys know similar "show offs", where a author/business presents their internals with real world requirements (shipping, freud protection) tied to the software component? I've found it especially interesting.

237 points by bemmu 10y ago

My guess when confronted with the number of lines of Python code for the front page was 5000 SLoC. I lowballed it intentionally because I was under the suspicion that it might be lower than one might normally guess. (Don't we all take pride in keeping our SLoC count as low as we can for any given problem?) The video had some closing points, including "NoSQL sucks for reports", which was explained briefly. OP, if you could, I would like for you to elaborate a bit more on this point. I am not doubting that it's true for Google App Engine, just interested to hear about the particulars.

stickfigure 10y ago
I can speak to this, being a heavy GAE user and formerly a heavy MongoDB user. For a reporting system you want an ad-hoc query language, fast in-database aggregation, and joins. The GAE datastore has none of the three, and MongoDB lacks joins. These are not fatal flaws - the GAE datastore has other advantages like infinite scalability, built-in synchronous geographic replication, failover, zero maintenance, etc. But for analytics, we replicate a subset of data to Postgres. It's still cheaper to have developers write occasional replication code than to hire a DBA to maintain the database, and I never have to worry that an ill-conceived sql statement will create an incident. I've come to the conclusion that using the GAE datastore as primary datastore and replicating to specialized databases is a pretty good architecture for systems that require reliability and low-maintenance. MongoDB's aggregation framework is really painful because the query language is weird and very low-level - you have to do most query planning yourself. And without joins you hit the limits of the kinds of questions you can ask the system very quickly. FWIW, my guess was 10k lines of code.
wodenokoto 10y ago
In the article he writes that it sucks because you end up writing queries in python, which would have been better suited to be written in SQL. I didn't listen to the talk, so I don't know if he mentions it there as well, and you are looking for a more in-depth explanation.

unwind 10y ago

Very interesting reading, as almost always from CJ. I was impressed by the precision in the data, i.e. being able to tell exactly how many lines of code deal with "fraud detection". Perhaps there's a "frauddetection.py" file, but then there has to be someone importing it and using it, and those lines are harder to count, I'd expect. But perhaps the integrations are small enough to deal with manually.

bemmu 10y ago
The code is short enough that what I did was actually just go through every line of code and try to assign it to a category. It did have some which were hard, such as the import statements you mention that multiple parts might use. But I figured being off by a few dozen lines wouldn't be that important for seeing the bigger picture.

asimuvPR 10y ago

@OP Why not dump the data into postgres rather than code the schema and transactions? I had a similar issue and postgres was a godsend.

brianwawok 10y ago
Well he is on google cloud, which when he wrote it he used the best option. Google cloud now has cloud sql, which would likely be much better. But then he needs to code, test, and migrate.. perhaps not worth it?
- asimuvPR 10y ago
  Yes, good points.
- bemmu 10y ago
  It wouldn't be impossible, but probably less work overall just to continue to write little Python scripts for my reporting purposes.
- maxmcd 10y ago
  The parent is likely referring to dumping the GAE Datastore data into a Postgres instance for reporting and other analysis. Not swapping out the primary datastore for the app. This comment elsewhere in this thread explains the strategy nicely: https://news.ycombinator.com/item?id=12613223 https://news.ycombinator.com/item?id=12613223
  - brianwawok 10y ago
    I don't think he has the scale to require NoSQL. Cloud SQL could host his data and his reporting in a single database.

wwer444453 10y ago

I work in a supermarket (in Japan). Those EAN13 codes you are printing are US only and probably reserved. I recommend that you change to any one of the in-store codes like 2*. It specifically carries no restrictions.

jylam 10y ago
EAN means "European Article Numbering".
- wwer444453 10y ago
  I know. IAN and EAN are interchangeably used.
Faaak 10y ago
What could be the problem ?
raverbashing 10y ago
What problems may this cause? It's not like he's scanning this at a supermarket till
- wwer444453 10y ago
  Nothing. It is like using reserved world IPs in your internal network. There is perfectly valid range you can use so why not shift to that?

gcatalfamo 10y ago

Thanks to this very informative blog post I also discovered https://embedd.io https://embedd.io that allows to embed HN and Reddit comments to any website. Thanks again! EDIT: I don't know why I am getting downvoted. I am NOT from embedd.io (why else would you downvote otherwise?) and just genuinely discovered the service. HN is truly toxic lately.

jon-wood 10y ago
You're probably being down voted because your comment doesn't really add anything to the discussion, but try not to take it too personally, people are just pushing your comment to the bottom of the page so that comments that are about the article can bubble up.
- jnbiche 10y ago
  > You're probably being down voted because your comment doesn't really add anything to the discussion Clearly, you're incorrect, because several people have explicitly stated that they appreciated the mention of this site (and probably there are more who were glad to find out but didn't comment). `bemmu`, Candy Japan's founder, also positively responded to the comment in question. In my opinion, downvotes on HN should be reserved for malicious comments like trolls, ad hominems, off-topic spam (real spam, rare on HN thanks to the mods. not just "I don't like this comment and think it's off-topic"), factually incorrect comments, and "reddit style" humor. Users should not downvote comments they simply disagree with, on-topic promotional comments, and tangentially related comments. Tangents are a great way to discover new ideas. However, there is a small cadre of HN downvoters who disagree with me, and who rabidly downvote any comment with a whiff of being off-topic or controversial. These downvoters must spend lots of time on HN, since their downvotes always come very quickly after a comment is submitted. Luckily, they do not represent the majority of HNers who can downvote, since comments like this one typically crawl back up into positive territory after the first 30-60 minutes. Some of my comments that were initially downvoted for being "off-topic" even crawled back up to over 10 upvotes. To the GP commenter, please don't get too frustrated. It's a relatively small number of people who downvote useful comments like yours. It's true that they are among the first to vote on comments, but almost always their downvotes are reversed after some time.

Retric 10y ago

This may just be me, but I find well maintained projects tend to get sticky around 10k lines of code. Much smaller than that and every new thing is just that a new thing. Much larger and it's easer for new things to kind of get tacked on at the edges. But, around 10k the project is small enough to understand and most new code tends to result in some refactoring for shared functionality. 10k lines is also fairly close to novel length. Anyone else notice this?

lmm 10y ago
When working in Python I noticed that 20kloc was the point at which I stopped being able to hold a project in my head, at which point I became unable to safely refactor, which in turn meant refactoring basically stopped being worth it. Any codebase larger than that, the core ossified, and new functionality was only neally possible to add at the edges. I think I can do a lot better than that now that I'm using Scala. Certainly I don't feel like I've hit the limit yet.
- anewhnaccount 10y ago
  This is where you need tests. Leaning on the type system can take you further, but at some point you're going to need tests with Scala too.
  - lmm 10y ago
    Disagree. Effective use of types has a much better cost/benefit than tests, and it's possible to go all the way down to 0% defect rate using just the type system (by encoding proof of correctness into the type system) if you really need to.
    - vertex-four 10y ago
      Only insofar as your types essentially become the bones of the program... which you'll then want to refactor at some point. Types are awesome. Languages like Rust give them to me without having to learn a ton of theory. However, eventually, I realise that my types have boxed me into a certain design - and I need to refactor heavily, which involves rewriting a large chunk of my types which I was relying on to show me that my code was correct.

yojex 10y ago

Great read! My guess would have been greater than 10,000. Couple of typos towards the end you might want to fix: "I don't know how I could have expect it, but somehow from the start I should have prepared for fraud. You want to at least keep an on any suspicious activity and react quickly if you start getting many chargebacks." 'expected' and 'at least keep an eye on'

bemmu 10y ago
Thanks!

dzdt 10y ago

Coming from an enterprise background, that number is astonishingly low.

tobltobs 10y ago
The Active Directory integration alone would take 5k lines.

ec109685 10y ago

It is supposing Recurly doesn't have a better fraud story. Did you look at using one one "anti fraud" as a service companies or a different vendor that does subscriptions?

bhelx 10y ago
At the time candyjapan had their fraud problems, Recurly didn't have very good support for these services. Since then, we've formed a partnership with Kount: https://recurly.com/press/recurly-teams-up-with-kount-to-help-subscription-based-merchants-combat-fraud/ https://recurly.com/press/recurly-teams-up-with-kount-to-hel... https://docs.recurly.com/docs/kount https://docs.recurly.com/docs/kount You get some basic protections for free but you can also integrate with your Kount account if you want to customize.

tmaly 10y ago

I totally get the point on NoSQL, I started out with Redis and some custom micro services that wrote to Boltdb for my food site. This started getting out of hand, now I am converting everything to Postgresql. I will still use Redis for some fast access/caching, but for reporting, sql is still the best way to go once you know what your data looks like.

late2part 10y ago

Yes, but how many hundreds of microservices do you have? :-)

deleted 10y ago

[deleted]

Ro93 10y ago

does he live in japan and ship from there? Or live in the uS and somehow gets the candy?

icebraining 10y ago
He does live in Japan: "Bemmu started Candy Japan with wife Nachi and lives in Tokushima, Japan."

deleted 10y ago

[deleted]

wkoszek 10y ago

Do you guys know similar "show offs", where a author/business presents their internals with real world requirements (shipping, freud protection) tied to the software component? I've found it especially interesting.