5 ms·

you should figure out what you plan on doing with the data first. That will help inform what you use and how you store it.

by thisone 10y ago

maxxxxx 10y ago
Nobody really knows. It's all new. In my view we need something we can play with without having to write lengthy requirement docs to IT.
- sgt101 10y ago
  what skills do you have in the team? also, budget & timescales.
  - maxxxxx 10y ago
    SQL and general software development are the main skill. We are not data experts. The timeline should be as short as possible. I feel IT is making this into a big project with big budget to to justify their existence and charge a lot of money to our cost center and not acting in our interest. But I don't have hard data because I don't know that world well.
    - sgt101 10y ago
      1TB sounds big, but if you were to get a server with a couple of SSDs and put a supported RDBMS on it you could do a lot. I think that you have a cast iron case with IT because you don't know what the outcomes are for the business yet - frankly who's going to sign off for a big implementation until you do? We couldn't justify Hadoop until we were +6TB... that was some years ago, pre ssd and pre big disks - our old rdbs server had got horribly complicated because it hit that scale - when it got to 10TB its raid controller corrupted the control files for the db. We had support and the consultants who came did a bunch of mind blowing work to restore it. But by then we had everything on Hadoop and we were only maintaining one legacy app on the old box - which we had told the users was dead and buried in any case (but kept to maintain friendships and so on). Back to my main point - some people think that analytics starts with a question, I don't, I think it starts with the data and instruments to inspect and understand the data. I think that you should get the data on some sensible box, put a good database on there that your guys can use (as they are SQL folks get an SQL one) and put R, R-studio and R-shiny there and start inspecting and understanding it. If no questions or insight arises then little is lost and it can go on a cheap fileserver until someone needs or understands it. If nuggets of gold start appearing then you can invest further in both skills and kit. I would recommend Hadoop either if other datasets of the 10gb+ scale start appearing or if this one gets >4TB. The other database recommendation would be because of mega joins - which Hadoop does well, and the general need to build a cheap EDW - if you have lots of cash you can build an expensive EDW instead! 4TB because currently disks are ~6TB and there are nasty scaling bumps in not Hadoop world that start kicking in as per my experience. IT are right to charge once this is established to be valuable; if it's work £250k a year to your business then it makes sense to be spending £50k a year to put SLA's and resiliance on it. If it's worth £2k... well...
      - sgt101 10y ago
        I forgot, what kind of data is this? Is it images? rows in a table?
- thisone 10y ago
  I've no idea of what your doc requirements are, but if you don't have any real plans for the data, stick it in a cost effective format that requires little to no maintenance but that can be read by multiple systems. Say parquet or avro for example. I assume you don't have someone/some system 24/7 pouring over the data so you could look into hosted notebooks like databricks, qubole (and gleaning from people I met at a recent conference any of the bazillion that are about to launch), or host zeppelin yourself or use AWS EMR. And don't lose your source data for that moment when you do figure out what you want to do with the data and you realise you need to reformat it.
- helpfile123 10y ago
  Remember you don't need all of the data for estimating models.
- eveningcoffee 10y ago
  Do you expect to solve any known business problem with it, or you are just toying around in hopes that something comes out of it?
  - maxxxxx 10y ago
    Right now there is no data at all so a lot of people hope they will find something useful. But nobody knows for sure.

you should figure out what you plan on doing with the data first. That will help inform what you use and how you store it.

by thisone 10y ago

you should figure out what you plan on doing with the data first. That will help inform what you use and how you store it.

maxxxxx 10y ago

Nobody really knows. It's all new. In my view we need something we can play with without having to write lengthy requirement docs to IT.

sgt101 10y ago
what skills do you have in the team? also, budget & timescales.
- maxxxxx 10y ago
  SQL and general software development are the main skill. We are not data experts. The timeline should be as short as possible. I feel IT is making this into a big project with big budget to to justify their existence and charge a lot of money to our cost center and not acting in our interest. But I don't have hard data because I don't know that world well.
  - sgt101 10y ago
    1TB sounds big, but if you were to get a server with a couple of SSDs and put a supported RDBMS on it you could do a lot. I think that you have a cast iron case with IT because you don't know what the outcomes are for the business yet - frankly who's going to sign off for a big implementation until you do? We couldn't justify Hadoop until we were +6TB... that was some years ago, pre ssd and pre big disks - our old rdbs server had got horribly complicated because it hit that scale - when it got to 10TB its raid controller corrupted the control files for the db. We had support and the consultants who came did a bunch of mind blowing work to restore it. But by then we had everything on Hadoop and we were only maintaining one legacy app on the old box - which we had told the users was dead and buried in any case (but kept to maintain friendships and so on). Back to my main point - some people think that analytics starts with a question, I don't, I think it starts with the data and instruments to inspect and understand the data. I think that you should get the data on some sensible box, put a good database on there that your guys can use (as they are SQL folks get an SQL one) and put R, R-studio and R-shiny there and start inspecting and understanding it. If no questions or insight arises then little is lost and it can go on a cheap fileserver until someone needs or understands it. If nuggets of gold start appearing then you can invest further in both skills and kit. I would recommend Hadoop either if other datasets of the 10gb+ scale start appearing or if this one gets >4TB. The other database recommendation would be because of mega joins - which Hadoop does well, and the general need to build a cheap EDW - if you have lots of cash you can build an expensive EDW instead! 4TB because currently disks are ~6TB and there are nasty scaling bumps in not Hadoop world that start kicking in as per my experience. IT are right to charge once this is established to be valuable; if it's work £250k a year to your business then it makes sense to be spending £50k a year to put SLA's and resiliance on it. If it's worth £2k... well...
    - sgt101 10y ago
      I forgot, what kind of data is this? Is it images? rows in a table?
thisone 10y ago
I've no idea of what your doc requirements are, but if you don't have any real plans for the data, stick it in a cost effective format that requires little to no maintenance but that can be read by multiple systems. Say parquet or avro for example. I assume you don't have someone/some system 24/7 pouring over the data so you could look into hosted notebooks like databricks, qubole (and gleaning from people I met at a recent conference any of the bazillion that are about to launch), or host zeppelin yourself or use AWS EMR. And don't lose your source data for that moment when you do figure out what you want to do with the data and you realise you need to reformat it.
helpfile123 10y ago
Remember you don't need all of the data for estimating models.
eveningcoffee 10y ago
Do you expect to solve any known business problem with it, or you are just toying around in hopes that something comes out of it?
- maxxxxx 10y ago
  Right now there is no data at all so a lot of people hope they will find something useful. But nobody knows for sure.