3 ms·

Recently I re-engineered an ecommerce site that takes many things from the "flat-file" montra. It's amazing how little database interaction you actually need fo

Recently I re-engineered an ecommerce site that takes many things from the "flat-file" montra. It's amazing how little database interaction you actually need for many sites. I think the database-driven content comes primarily from kitchen sink packages like Wordpress. More often than not, it's better to get an overall look at your structure and decide what needs to be dynamic and what can be "static". When talking about it internally, we refer to "flat-file" as "generated". Simply meaning that it's dynamically created by a task or user interaction.

subpixel 14y ago
FYI I'd be totally interested in more info about how you achieve this in an e-commerce environment. Blog post?
wting 14y ago
The file system is the original "database". I solved an interview question recently using a few Linux utilities, pipes, and flat files. It was a bash one liner that impressed the interviewer, but then he asked me to solve it the "real" way. Both databases and file systems use B-trees to implement fast read / writes, it's just that SQL databases enforce structure. OTOH, using flat files moves data checking into application space and losing certain ACID properties. Similar trade offs are made between NoSQL and SQL, dynamic and static languages, but I digress...
- ewang1 14y ago
  I would imagine "writes" are still being written to the database, which then triggers the re-generation of the appropriate flat-file(s).
  - wting 14y ago
    No, given a bunch of Apache logs I needed to find the top 10 queries that met conditions A and B. 1. grepping for conditions and extracting the query with sed 2. appending a char to a flat file (i.e. increasing query count by 1) 3. sort by largest files and get the top 10 Parsing log files[0]: for ./*.log -type f -exec echo 1 >> $(grep cond_A {} | grep cond_B | sed -E "s:.*(query).*:./results/\1.txt:") \; Finding top 10 results: ls ./results/*.txt -Sl | head [0]: Untested code, and it only grabs one line per log file instead of grepping all matching lines in the log file. I'd have to move the grep to the outside and loop the `echo 1 >>` command, but you get the point.
    - mbreese 14y ago
      You probably could have done the same thing in one line: cat *.log | grep condA | grep condB | sed 'some regex to get rid of dates, etc' | sort | uniq -c | sort -nr | head -n 5 Or something like that... Not very efficient, but it would work in a pinch. I actually do something like this all the time for large datasets. In the time it would take me to write something better, this set of commands is already done.
    - lotsofcows 14y ago
      Those that don't know awk are doomed to complex pipe lines.
- deleted 14y ago
  [deleted]
deleted 14y ago
[deleted]
legutierr 14y ago
Maybe I shouldn't be, but I am kind of surprised that you can do an ecommerce site using mostly generated files. The two things that I wonder about are (1) displaying cart/user information on every page (is it all javascript?) and (2) csrf in the forms. Also, I would assume that javascript becomes a requirement for the functioning of the site; I always considered ecommerce to be the one part of the web were you wanted to be able to operate without javascript, so as not to turn anyone away. I found this blog post that says that in 2010, 2% in the US had JS disabled: http://developer.yahoo.com/blogs/ydn/posts/2010/10/how-many-users-have-javascript-disabled/ http://developer.yahoo.com/blogs/ydn/posts/2010/10/how-many-...
- pjungwir 14y ago
  You can also integrate with a third-party shopping cart then load a small iframe to show "2 Items in the Cart." Not my favorite way to build a site, but it is very inexpensive to build, and for small vendors the site becomes entirely static. Even if you build it in Rails, you can use page-level caching so after the first hit it's served as a static html file.
  - jasonwatkinspdx 14y ago
    Page level caching is one of the most underused features of rails. Most applications show anonymous users identical content, and can tolerate that content being cached briefly. Setting this up gives you slashdot protection as well as access to a last chance degraded mode if there's a chain of catastrophic failures further into the app stack.
    - ericcholis 14y ago
      Exactly, this is the general idea of what I do in the ecommerce app I mentioned.
- ericcholis 14y ago
  I guess I should have been more specific about my strategy for "generated" content in an ecommerce environment. The traditional ecommerce elements that should be real-time are; such as carts, pricing, availability, etc. Everything else is generated on update via our [custom] inventory management software. The closest analogy I can give is Varnish's Edge Side Includes. It's not exactly the same, but it's very similar.

subpixel 14y ago

FYI I'd be totally interested in more info about how you achieve this in an e-commerce environment. Blog post?

wting 14y ago

The file system is the original "database". I solved an interview question recently using a few Linux utilities, pipes, and flat files. It was a bash one liner that impressed the interviewer, but then he asked me to solve it the "real" way. Both databases and file systems use B-trees to implement fast read / writes, it's just that SQL databases enforce structure. OTOH, using flat files moves data checking into application space and losing certain ACID properties. Similar trade offs are made between NoSQL and SQL, dynamic and static languages, but I digress...

ewang1 14y ago
I would imagine "writes" are still being written to the database, which then triggers the re-generation of the appropriate flat-file(s).
- wting 14y ago
  No, given a bunch of Apache logs I needed to find the top 10 queries that met conditions A and B. 1. grepping for conditions and extracting the query with sed 2. appending a char to a flat file (i.e. increasing query count by 1) 3. sort by largest files and get the top 10 Parsing log files[0]: for ./*.log -type f -exec echo 1 >> $(grep cond_A {} | grep cond_B | sed -E "s:.*(query).*:./results/\1.txt:") \; Finding top 10 results: ls ./results/*.txt -Sl | head [0]: Untested code, and it only grabs one line per log file instead of grepping all matching lines in the log file. I'd have to move the grep to the outside and loop the `echo 1 >>` command, but you get the point.
  - mbreese 14y ago
    You probably could have done the same thing in one line: cat *.log | grep condA | grep condB | sed 'some regex to get rid of dates, etc' | sort | uniq -c | sort -nr | head -n 5 Or something like that... Not very efficient, but it would work in a pinch. I actually do something like this all the time for large datasets. In the time it would take me to write something better, this set of commands is already done.
  - lotsofcows 14y ago
    Those that don't know awk are doomed to complex pipe lines.
deleted 14y ago
[deleted]

deleted 14y ago

[deleted]

legutierr 14y ago

Maybe I shouldn't be, but I am kind of surprised that you can do an ecommerce site using mostly generated files. The two things that I wonder about are (1) displaying cart/user information on every page (is it all javascript?) and (2) csrf in the forms. Also, I would assume that javascript becomes a requirement for the functioning of the site; I always considered ecommerce to be the one part of the web were you wanted to be able to operate without javascript, so as not to turn anyone away. I found this blog post that says that in 2010, 2% in the US had JS disabled: http://developer.yahoo.com/blogs/ydn/posts/2010/10/how-many-users-have-javascript-disabled/ http://developer.yahoo.com/blogs/ydn/posts/2010/10/how-many-...

pjungwir 14y ago
You can also integrate with a third-party shopping cart then load a small iframe to show "2 Items in the Cart." Not my favorite way to build a site, but it is very inexpensive to build, and for small vendors the site becomes entirely static. Even if you build it in Rails, you can use page-level caching so after the first hit it's served as a static html file.
- jasonwatkinspdx 14y ago
  Page level caching is one of the most underused features of rails. Most applications show anonymous users identical content, and can tolerate that content being cached briefly. Setting this up gives you slashdot protection as well as access to a last chance degraded mode if there's a chain of catastrophic failures further into the app stack.
  - ericcholis 14y ago
    Exactly, this is the general idea of what I do in the ecommerce app I mentioned.
ericcholis 14y ago
I guess I should have been more specific about my strategy for "generated" content in an ecommerce environment. The traditional ecommerce elements that should be real-time are; such as carts, pricing, availability, etc. Everything else is generated on update via our [custom] inventory management software. The closest analogy I can give is Varnish's Edge Side Includes. It's not exactly the same, but it's very similar.