8 ms·

Data management. 1) Cleaning the data as it comes in rather than in batches so we can use it sooner, invalid data is discarded, outlier detection, normalizing

Data management. 1) Cleaning the data as it comes in rather than in batches so we can use it sooner, invalid data is discarded, outlier detection, normalizing inputs etc.... 2) Warehousing of the data with proper indexes so you can perform some advanced queries on unstructured data 3) Some data is sent in bulk at the end of day, some of the data is streamed in fire hose style. How can we preprocess the fire hose data so that we don't have to wait until the end of the day to parse it all. 4) Oh and all of this data is unstructured and comes from 75 different sources. Soon the average hedge fund will have more people just cleaning and managing data than they do in quantitative research, dev ops, software development and trading. Oh and lots of the data is considered proprietary so while AWS/Azure, etc is fine, sending it to a third party to process is not. TL/DR Help me, I'm drowning in data. How do I get the time from when I acquire data to when I trade based on it down to a reasonable time frame, where reasonable is closer to hours rather than days/weeks.

deleted 10y ago
[deleted]
juskrey 10y ago
Will hedge funds take such a huge risk of using 3rd party centralized (or centrally developed) data management system?
- otoburb 10y ago
  Provided they can deploy the 3rd party solution onsite (aka "on premise"), this wouldn't be seen as a "huge risk".
  - juskrey 10y ago
    And what about the risk of 3rd party dev error which will throw several funds at once out of the business?
    - otoburb 10y ago
      Great question, but I don't see this much different from other industries or companies with millions or billions of dollars at stake. They don't all roll their own software inhouse, and most companies (hedgefunds or not, small or even large) simply cannot afford to for financial and commercial risk-management reasons unless they can justify the software truly being a core competitive competency. The same line of inquiry has been evaluated for most 3rd party software that companies rely on. For this specific instance of data collection and cleaning, I'm imagining it's not going to be a much different calculus, although perhaps you'll see a higher percentage of firms choosing to roll their own if they have the chops and pockets (e.g. Two Sigma, Bridgewater, Goldman Sachs, etc.). I will note that there are commercial mechanisms firms could try to implement to try to limit the downsides in case something like this happens: warranty & damages provisions, and insurance are two come that spring to mind. I'm sure there are numerous other considerations in the age-old "build or buy" cost-benefit analysis.
- wintermute42 10y ago
  Yes.
flamesnare3 10y ago
I think Palantir provides solutions to this problem
- seppin 10y ago
  For $$$
- dbuxton 10y ago
  The Palantir solution, from what I understand, boils down to "lots of consultants (FDEs) writing glue code".
- luckydata 10y ago
  Palantir doesn't really have a product, they write tons of code to put everything together and make it look seamless from the UI but they don't have anything drop in as far as I know.
wslh 10y ago
In my limited experience this is science, art, and craft. I don't think there is a specific method to do this in an automated way.
danielvinson 10y ago
There are good open source options for each step here - is the solution you are looking for just a UI and easy install process? Or would your ideal solution make all of the decisions for you - data structure and format, which data is/isn't valid, what output options are possible, managing server resources, etc.?
- frakkingcylons 10y ago
  I'm new to this sort of thing - can you elaborate on some of the open-source options for those steps?
  - danielvinson 10y ago
    I'm not very familiar with the open source options since after many years of coding this by hand, I work with what I know. I am a developer that works with data, not a Data Scientist, so I don't really know the lingo and whatever hipstery terms people are using these days. I will answer to the best of my ability, though (mostly for my sake, who knows if this will be useful): Cleaning: Open Refine seems to be the best product in this category. I haven't used anything but my own tools to do this before, so I can't really offer any advice. Warehousing: My understanding is that this is just a fancy way to talk about a database with a schema designed for analytics. There are many open source databases which do this very well, the one I use being Cassandra (and/or KairosDB), though it is also likely the one that is hardest to use. For a beginner, you might want to refer to this SO answer: http://stackoverflow.com/questions/8816429/is-there-a-powerful-database-system-for-time-series-data http://stackoverflow.com/questions/8816429/is-there-a-powerf... Data processing/collection: This is something that is incredibly dependent on the data sources, so I likely can't tell you anything that will help. Most of my data sources I've worked with have been internally sourced log files, messages from ZMQ, or CSV data - you might be working with something far different though, since there are lots of public data sets and such which are common. Ideally, this would be integrated into the tools that you are using to clean the data, but I don't know if that exists. Handling input from many different sources at different rates is not a very hard problem to solve if your system is build correctly - you could for example run a daemon for each data source which will populate the database when there is new data available, then send a message off to the processing engine, which will integrate the data into whatever reports you are running. Specifically for a use case of a hedge fund, the reports could be triggered by a message which is sent when the new data is available, and processing could be done in parallel in Lambda or similar dependent on need to get a nearly instant return, enabling nearly real-time reporting.
robinhowlett 10y ago
Integration platform as a service (iPaas) that can work with streaming, batch, and Big Data using a drag-n-drop UI e.g. https://www.youtube.com/watch?v=SfEuG7Dg_O8 https://www.youtube.com/watch?v=SfEuG7Dg_O8 Check out my company, SnapLogic: https://www.snaplogic.com/ https://www.snaplogic.com/
rpedela 10y ago
Have you tried Quandl? They solve some of those problems for financial data.
deleted 10y ago
[deleted]
mamcx 10y ago
This is a area I like to work. I have seen local companies working for months/years to finally use his BI package but the trouble at the step 1 is big (and also, to put the data in a "nice" schema). The problem is that enter in this space is hard. Years ago I was at a company that have a niche product (in foxpro) for this kind of task, and I have dreamed about build something like this based in my experience, but get the funding for this kind of "boring" task is hard (more in my country, Colombia). P.D: If wanna help, we can talk. I can't give a magical solution but at least I find this kind of "boring" jobs compelling ;)
asragab 10y ago
What I envision a solution to be like, would be something like an configurable/codeable OpenRefine (was Google Refine) with streaming ingestion/extraction, with a validation engine/parsing engine (something more elegant than regex, but you can drop into that if necessary) and maybe a pluggable event processor (i.e. a Spark or Flink). I would love to work on such a problem, and solve it.
mmsmatt 10y ago
Great questions. I worked on similar problems in the weather/ag space for a few years, trying to minimize the time between data was acquired and data is ready to inform a decision. We threw every rule out the window in the name of performance _when fetching raw data from external sources_. So we had weather station networks, NOAA forecast runs and NASA satellite data in a workable schema in our shop way faster than average. Mix of C, PowerShell, Perl, and the nonstandard parts of T-SQL, highly parallelized, tricky but fast. After the "workable schema" was established, the rules came back and we acted more responsibly. Smart instead of clever. Ran this stuff all day long, getting every piece of data asap. Things that can only be calculated with a full day of data we poked and prodded the meteorologists to express in "partial aggregates", which to me were just like the map steps before an EOD reduce. Took a lot of mutual understanding and iterating but worth it in the end. When the ultimate data source (satellite or radar site for us) posted its last hour of data, we were 95% done with the day's computation work. We do our last step, publish our numbers, and bam, Our ag clients have this stuff a day earlier than they are used to.
- everdaniel 10y ago
  hi mmsmatt, you mentioned your worked on the weather/ag space, is it ok to ask you a few questions? thank you!
adladl2012 10y ago
Splunk provides this kind of solution and it's pretty amazing :)
capkutay 10y ago
Check out Striim http://www.striim.com/ http://www.striim.com/ It's a native streaming platform so your data will be cleansed, processed, scanned for outliers event-by-event rather than in batches. We have dozens of streaming connectors IT/Enterprise/Web data sources. We also support initial load for your firehose data. For unstructured data, we have support for RegEx based parsers. Shoot me a message if you have any more questions. We have many big name users in Aerospace, Banking, Device manufacturing, and Logistics industries.
- jsingleton18 10y ago
  We solved RegEx and can guarantee worst-case linear time matching for any number of expressions. We should talk! www.watchful.io
  - jefftk 10y ago
    guarantee worst-case linear time matching for any number of expressions Or you could just use RE2: https://github.com/google/re2/wiki/WhyRE2 https://github.com/google/re2/wiki/WhyRE2 It's open source, and match time is linear in the length of the input string. (Disclosure: I work at Google on a different open source project)
    - alixaxel 10y ago
      RE2 is great! I first discovered it while building namegrep.com and after benchmarking it against PCRE I never looked back. :)
  - glangdale 10y ago
    Interesting. We worked in this area for many years, did the 'startup' in this area, now acquired by Intel, open sourced the product (ob plug: https://github.com/01org/hyperscan https://github.com/01org/hyperscan ) and I would never claim that we are anything near 'solving' regex. More like "mitigating" or "occasionally getting regex slightly more right than some other solutions". There are many different approaches to regex and all seem to focus on different parts of functionality (RE2 focuses on quick compiles and simplicity, libpcre has 'all the functionality', we're about streaming + large scale + high performance if you can tolerate long compiles and lots of complexity). A number of new projects are trying very interesting approaches, like icgrep and the Rust regex guys. I would be curious to hear about your approach.
jsingleton18 10y ago
Hey - we did this. We should talk. Well not #2 exactly...but we solve most of your problems. www.watchful.io
anthonyaragues 10y ago
I've been working on similar issues, and developing towards solving them. I work with mixed schemas, handle user defined processes inline and allow you to gather stats and show everything as it streams + accommodate a workflow. I'm only 1 month from launching my beta and want to give away licenses to people who can use it and give me feedback. here is the landing page for it: http://ohm.ai http://ohm.ai OH and it uploads nothing... it all runs in your browser anthony dot aragues at gmail dot com if interesed
nwatson 10y ago
I think startups Trifacta and Paxata are doing this or some aspects of this ... see a comparison (from 2015): https://www.quora.com/How-does-Paxata-compare-to-Trifacta https://www.quora.com/How-does-Paxata-compare-to-Trifacta I don't know much about Paxata but I think Trifacta are well-regarded in industry and academia. Trifacta founders worked on research / open-source-? project Data Wrangler http://vis.stanford.edu/wrangler/ http://vis.stanford.edu/wrangler/ and turned it into Trifacta. I don't know much about either product in truth.
zilchers 10y ago
One thing you don't really mention here, but it's mentioned a lot in the comments, is the data extraction piece. Is data extraction a pretty solved problem at this point, and it's really the intelligent cleaning, transforming, then warehousing / analysis that's the unsolved issue?
davidjairala 10y ago
I've actually been working on something like this for a while now, and found your comment about proprietary data interesting. Would this mean that hosting this data in a third party server is out of the question for you? OK with NDA?
muraiki 10y ago
In my current job I use Informatica Cloud, which either by itself or in combination with other Informatica products can do these things. I have two main complaints about it: 1. The UX is subpar. It insists on running in only a single tab at a time, and attempts to open multiple tabs will instead override whatever it considers to be the master tab. This is a huge pain, because I often need to have a mapping workflow open in one window and some other relevant part of the application open in another. Instead I have to save, go find the thing I want, and go back. Another problem is that when working with data sources containing tons of fields, there's no easy way to search. 2. It offers an expression language to perform some computational tasks, similar to what you'd find in Excel, but it's hamstrung by a poor UI and a limited amount of functions. The built-in editor for expressions is really poor (see Tableau for an example of a great editor for a simple Excel-like language; it even has type linting) and, unless I've misunderstood something, you can't declare any variables so you end up with huge nested expressions. There aren't many functions available, so something as simple as removing whitespace ends up as lstrip(rstrip(foo)). In combination with no support for statements (or at least a let expression like in lisp) this makes any nontrivial data munging completely indecipherable. I've looked around in this space and it seems like there are a variety of products, but the supplier of our main CRM will only support Informatica Cloud. I think that a company that can offer a product that does what you've said but makes a serious effort at UX could cause users to revolt and demand to use it! I know the joke is that Slack is just a pretty IRC with better UX... but that's exactly why it has become so successful. In terms of data munging, take a look at Microsoft's Power BI. It's visualization software but it has a nice data munging mode that, crucially, keeps track of all the changes you make and displays them in a linear format. This is great for getting a quick idea as to what was done with the data and is essential for doing reproducible data analyses. Unfortunately, Power BI also suffers from poor UX in insisting on tiny fonts and gray-on-gray palettes that are totally unreadable to anyone over 30.
maxlamb 10y ago
Have you looked into Snowflake? Seems like their solution satisfies most of your requirements, including native ingestion of unstructured data. The one caveat is that all source data must first be loaded to S3. https://www.snowflake.net/ https://www.snowflake.net/
huy 10y ago
Check out Holistics.io (disclaimer: I'm a cofounder). While I can't say we can solve all your listed problems above, we provide enough tooling on top of your DW to help you pre-process (clean, aggregate) them. https://www.holistics.io/data-preparation/ https://www.holistics.io/data-preparation/ And on the part about firehose data, you might already know this, but Kafka and their line of work should be aligned with what you're after.
Godel_unicode 10y ago
> TL/DR Help me, I'm drowning in data Omg, so many sales pitches. You should figure out which of those were automatically generated by someone who's bot is crawling HN and using NLP to find posts like this, and then hire them. There's basically 0 chance that isn't happening...
RickChen 10y ago
Chollida. I have been working with a startup in Seattle tackling these very issues. Super great team and great software, please get in touch and try it out! datablade.io
rednerrus 10y ago
My last company did this. You don't know how hard it is to get people to turn their data over to you.
reitanqild 10y ago
> 4) Oh and all of this data is unstructured and comes from 75 different sources. Sounds like a job for Apache Camel?
fs111 10y ago
This may be something for you: https://www.snaplogic.com/ https://www.snaplogic.com/
dgudkov 10y ago
On a smaller scale my EasyMorph might be of help. It's a lightweight ETL and you can do with it way more than with data preparation tools http://easymorph.com http://easymorph.com
luckydata 10y ago
I tried to evangelize THIS very problem with my previous company (a somewhat successful managed data infra service) for a year, it was AMAZINGLY difficult to make my executives even understand the problem and its magnitude.
gesman 10y ago
splunk.com +scaleable Problem solved on all points. 100% fully functional, fully featured, 100% free download right from the website. Disclaimer: I work for splunk.
roundthecorner 10y ago
Take a look at Apache Merton's architecture. They seem to have dealt with a number of inherent issues in the problem putting together data processing pipelines of open source components and ensuring they work together. The project is active but incubating right now.

deleted 10y ago

[deleted]

juskrey 10y ago

Will hedge funds take such a huge risk of using 3rd party centralized (or centrally developed) data management system?

otoburb 10y ago
Provided they can deploy the 3rd party solution onsite (aka "on premise"), this wouldn't be seen as a "huge risk".
- juskrey 10y ago
  And what about the risk of 3rd party dev error which will throw several funds at once out of the business?
  - otoburb 10y ago
    Great question, but I don't see this much different from other industries or companies with millions or billions of dollars at stake. They don't all roll their own software inhouse, and most companies (hedgefunds or not, small or even large) simply cannot afford to for financial and commercial risk-management reasons unless they can justify the software truly being a core competitive competency. The same line of inquiry has been evaluated for most 3rd party software that companies rely on. For this specific instance of data collection and cleaning, I'm imagining it's not going to be a much different calculus, although perhaps you'll see a higher percentage of firms choosing to roll their own if they have the chops and pockets (e.g. Two Sigma, Bridgewater, Goldman Sachs, etc.). I will note that there are commercial mechanisms firms could try to implement to try to limit the downsides in case something like this happens: warranty & damages provisions, and insurance are two come that spring to mind. I'm sure there are numerous other considerations in the age-old "build or buy" cost-benefit analysis.
wintermute42 10y ago
Yes.

flamesnare3 10y ago

I think Palantir provides solutions to this problem

seppin 10y ago
For $$$
dbuxton 10y ago
The Palantir solution, from what I understand, boils down to "lots of consultants (FDEs) writing glue code".
luckydata 10y ago
Palantir doesn't really have a product, they write tons of code to put everything together and make it look seamless from the UI but they don't have anything drop in as far as I know.

wslh 10y ago

In my limited experience this is science, art, and craft. I don't think there is a specific method to do this in an automated way.

danielvinson 10y ago

There are good open source options for each step here - is the solution you are looking for just a UI and easy install process? Or would your ideal solution make all of the decisions for you - data structure and format, which data is/isn't valid, what output options are possible, managing server resources, etc.?

frakkingcylons 10y ago
I'm new to this sort of thing - can you elaborate on some of the open-source options for those steps?
- danielvinson 10y ago
  I'm not very familiar with the open source options since after many years of coding this by hand, I work with what I know. I am a developer that works with data, not a Data Scientist, so I don't really know the lingo and whatever hipstery terms people are using these days. I will answer to the best of my ability, though (mostly for my sake, who knows if this will be useful): Cleaning: Open Refine seems to be the best product in this category. I haven't used anything but my own tools to do this before, so I can't really offer any advice. Warehousing: My understanding is that this is just a fancy way to talk about a database with a schema designed for analytics. There are many open source databases which do this very well, the one I use being Cassandra (and/or KairosDB), though it is also likely the one that is hardest to use. For a beginner, you might want to refer to this SO answer: http://stackoverflow.com/questions/8816429/is-there-a-powerful-database-system-for-time-series-data http://stackoverflow.com/questions/8816429/is-there-a-powerf... Data processing/collection: This is something that is incredibly dependent on the data sources, so I likely can't tell you anything that will help. Most of my data sources I've worked with have been internally sourced log files, messages from ZMQ, or CSV data - you might be working with something far different though, since there are lots of public data sets and such which are common. Ideally, this would be integrated into the tools that you are using to clean the data, but I don't know if that exists. Handling input from many different sources at different rates is not a very hard problem to solve if your system is build correctly - you could for example run a daemon for each data source which will populate the database when there is new data available, then send a message off to the processing engine, which will integrate the data into whatever reports you are running. Specifically for a use case of a hedge fund, the reports could be triggered by a message which is sent when the new data is available, and processing could be done in parallel in Lambda or similar dependent on need to get a nearly instant return, enabling nearly real-time reporting.

robinhowlett 10y ago

Integration platform as a service (iPaas) that can work with streaming, batch, and Big Data using a drag-n-drop UI e.g. https://www.youtube.com/watch?v=SfEuG7Dg_O8 https://www.youtube.com/watch?v=SfEuG7Dg_O8 Check out my company, SnapLogic: https://www.snaplogic.com/ https://www.snaplogic.com/

rpedela 10y ago

Have you tried Quandl? They solve some of those problems for financial data.

deleted 10y ago

[deleted]

mamcx 10y ago

This is a area I like to work. I have seen local companies working for months/years to finally use his BI package but the trouble at the step 1 is big (and also, to put the data in a "nice" schema). The problem is that enter in this space is hard. Years ago I was at a company that have a niche product (in foxpro) for this kind of task, and I have dreamed about build something like this based in my experience, but get the funding for this kind of "boring" task is hard (more in my country, Colombia). P.D: If wanna help, we can talk. I can't give a magical solution but at least I find this kind of "boring" jobs compelling ;)

asragab 10y ago

What I envision a solution to be like, would be something like an configurable/codeable OpenRefine (was Google Refine) with streaming ingestion/extraction, with a validation engine/parsing engine (something more elegant than regex, but you can drop into that if necessary) and maybe a pluggable event processor (i.e. a Spark or Flink). I would love to work on such a problem, and solve it.

mmsmatt 10y ago

Great questions. I worked on similar problems in the weather/ag space for a few years, trying to minimize the time between data was acquired and data is ready to inform a decision. We threw every rule out the window in the name of performance _when fetching raw data from external sources_. So we had weather station networks, NOAA forecast runs and NASA satellite data in a workable schema in our shop way faster than average. Mix of C, PowerShell, Perl, and the nonstandard parts of T-SQL, highly parallelized, tricky but fast. After the "workable schema" was established, the rules came back and we acted more responsibly. Smart instead of clever. Ran this stuff all day long, getting every piece of data asap. Things that can only be calculated with a full day of data we poked and prodded the meteorologists to express in "partial aggregates", which to me were just like the map steps before an EOD reduce. Took a lot of mutual understanding and iterating but worth it in the end. When the ultimate data source (satellite or radar site for us) posted its last hour of data, we were 95% done with the day's computation work. We do our last step, publish our numbers, and bam, Our ag clients have this stuff a day earlier than they are used to.

everdaniel 10y ago
hi mmsmatt, you mentioned your worked on the weather/ag space, is it ok to ask you a few questions? thank you!

adladl2012 10y ago

Splunk provides this kind of solution and it's pretty amazing :)

capkutay 10y ago

Check out Striim http://www.striim.com/ http://www.striim.com/ It's a native streaming platform so your data will be cleansed, processed, scanned for outliers event-by-event rather than in batches. We have dozens of streaming connectors IT/Enterprise/Web data sources. We also support initial load for your firehose data. For unstructured data, we have support for RegEx based parsers. Shoot me a message if you have any more questions. We have many big name users in Aerospace, Banking, Device manufacturing, and Logistics industries.

jsingleton18 10y ago
We solved RegEx and can guarantee worst-case linear time matching for any number of expressions. We should talk! www.watchful.io
- jefftk 10y ago
  guarantee worst-case linear time matching for any number of expressions Or you could just use RE2: https://github.com/google/re2/wiki/WhyRE2 https://github.com/google/re2/wiki/WhyRE2 It's open source, and match time is linear in the length of the input string. (Disclosure: I work at Google on a different open source project)
  - alixaxel 10y ago
    RE2 is great! I first discovered it while building namegrep.com and after benchmarking it against PCRE I never looked back. :)
- glangdale 10y ago
  Interesting. We worked in this area for many years, did the 'startup' in this area, now acquired by Intel, open sourced the product (ob plug: https://github.com/01org/hyperscan https://github.com/01org/hyperscan ) and I would never claim that we are anything near 'solving' regex. More like "mitigating" or "occasionally getting regex slightly more right than some other solutions". There are many different approaches to regex and all seem to focus on different parts of functionality (RE2 focuses on quick compiles and simplicity, libpcre has 'all the functionality', we're about streaming + large scale + high performance if you can tolerate long compiles and lots of complexity). A number of new projects are trying very interesting approaches, like icgrep and the Rust regex guys. I would be curious to hear about your approach.

jsingleton18 10y ago

Hey - we did this. We should talk. Well not #2 exactly...but we solve most of your problems. www.watchful.io

anthonyaragues 10y ago

I've been working on similar issues, and developing towards solving them. I work with mixed schemas, handle user defined processes inline and allow you to gather stats and show everything as it streams + accommodate a workflow. I'm only 1 month from launching my beta and want to give away licenses to people who can use it and give me feedback. here is the landing page for it: http://ohm.ai http://ohm.ai OH and it uploads nothing... it all runs in your browser anthony dot aragues at gmail dot com if interesed

nwatson 10y ago

I think startups Trifacta and Paxata are doing this or some aspects of this ... see a comparison (from 2015): https://www.quora.com/How-does-Paxata-compare-to-Trifacta https://www.quora.com/How-does-Paxata-compare-to-Trifacta I don't know much about Paxata but I think Trifacta are well-regarded in industry and academia. Trifacta founders worked on research / open-source-? project Data Wrangler http://vis.stanford.edu/wrangler/ http://vis.stanford.edu/wrangler/ and turned it into Trifacta. I don't know much about either product in truth.

zilchers 10y ago

One thing you don't really mention here, but it's mentioned a lot in the comments, is the data extraction piece. Is data extraction a pretty solved problem at this point, and it's really the intelligent cleaning, transforming, then warehousing / analysis that's the unsolved issue?

davidjairala 10y ago

I've actually been working on something like this for a while now, and found your comment about proprietary data interesting. Would this mean that hosting this data in a third party server is out of the question for you? OK with NDA?

muraiki 10y ago

In my current job I use Informatica Cloud, which either by itself or in combination with other Informatica products can do these things. I have two main complaints about it: 1. The UX is subpar. It insists on running in only a single tab at a time, and attempts to open multiple tabs will instead override whatever it considers to be the master tab. This is a huge pain, because I often need to have a mapping workflow open in one window and some other relevant part of the application open in another. Instead I have to save, go find the thing I want, and go back. Another problem is that when working with data sources containing tons of fields, there's no easy way to search. 2. It offers an expression language to perform some computational tasks, similar to what you'd find in Excel, but it's hamstrung by a poor UI and a limited amount of functions. The built-in editor for expressions is really poor (see Tableau for an example of a great editor for a simple Excel-like language; it even has type linting) and, unless I've misunderstood something, you can't declare any variables so you end up with huge nested expressions. There aren't many functions available, so something as simple as removing whitespace ends up as lstrip(rstrip(foo)). In combination with no support for statements (or at least a let expression like in lisp) this makes any nontrivial data munging completely indecipherable. I've looked around in this space and it seems like there are a variety of products, but the supplier of our main CRM will only support Informatica Cloud. I think that a company that can offer a product that does what you've said but makes a serious effort at UX could cause users to revolt and demand to use it! I know the joke is that Slack is just a pretty IRC with better UX... but that's exactly why it has become so successful. In terms of data munging, take a look at Microsoft's Power BI. It's visualization software but it has a nice data munging mode that, crucially, keeps track of all the changes you make and displays them in a linear format. This is great for getting a quick idea as to what was done with the data and is essential for doing reproducible data analyses. Unfortunately, Power BI also suffers from poor UX in insisting on tiny fonts and gray-on-gray palettes that are totally unreadable to anyone over 30.

maxlamb 10y ago

Have you looked into Snowflake? Seems like their solution satisfies most of your requirements, including native ingestion of unstructured data. The one caveat is that all source data must first be loaded to S3. https://www.snowflake.net/ https://www.snowflake.net/

huy 10y ago

Check out Holistics.io (disclaimer: I'm a cofounder). While I can't say we can solve all your listed problems above, we provide enough tooling on top of your DW to help you pre-process (clean, aggregate) them. https://www.holistics.io/data-preparation/ https://www.holistics.io/data-preparation/ And on the part about firehose data, you might already know this, but Kafka and their line of work should be aligned with what you're after.

Godel_unicode 10y ago

> TL/DR Help me, I'm drowning in data Omg, so many sales pitches. You should figure out which of those were automatically generated by someone who's bot is crawling HN and using NLP to find posts like this, and then hire them. There's basically 0 chance that isn't happening...

RickChen 10y ago

Chollida. I have been working with a startup in Seattle tackling these very issues. Super great team and great software, please get in touch and try it out! datablade.io

rednerrus 10y ago

My last company did this. You don't know how hard it is to get people to turn their data over to you.

reitanqild 10y ago

> 4) Oh and all of this data is unstructured and comes from 75 different sources. Sounds like a job for Apache Camel?

fs111 10y ago

This may be something for you: https://www.snaplogic.com/ https://www.snaplogic.com/

dgudkov 10y ago

On a smaller scale my EasyMorph might be of help. It's a lightweight ETL and you can do with it way more than with data preparation tools http://easymorph.com http://easymorph.com

luckydata 10y ago

I tried to evangelize THIS very problem with my previous company (a somewhat successful managed data infra service) for a year, it was AMAZINGLY difficult to make my executives even understand the problem and its magnitude.

gesman 10y ago

splunk.com +scaleable Problem solved on all points. 100% fully functional, fully featured, 100% free download right from the website. Disclaimer: I work for splunk.

roundthecorner 10y ago

Take a look at Apache Merton's architecture. They seem to have dealt with a number of inherent issues in the problem putting together data processing pipelines of open source components and ensuring they work together. The project is active but incubating right now.