3 ms·
You don't want to build your own data store. You want a better solution than the legacy system you're dealing with now. I work contract jobs, which has given m
by oneJob 11y ago
You don't want to build your own data store. You want a better solution than the legacy system you're dealing with now.
I work contract jobs, which has given me the opportunity to see many different approaches to IT infrastructure, team organization, workflow management, IT solutions, etc, in companies from some of the largest in the world to mom and pop businesses. I've only ever seen companies in the large to enormous range engaged in the type of project you're proposing.
It wouldn't just require software engineers and software licenses, you'd also need prob two business/data analysts with appropriate experience in data warehouse design and software migration to draw up requirements (if you don't want it to crap it's pants 10 months into production use, thus making it the new "Legacy System") and end user / operations documentation / training material. Then you need the software engineers (prob one application engineer and one database engineer) to turn those requirements into a working product. Oh, and these people mentioned thus far will be too busy with this work for 3-9 months to be doing anything else, and then generally not stay on after because there won't be any appropriate work for them once it's up and running and they've trained the end users. Next you'll need at least one person who's primary role is maintaining the system, even if 50% of their time is free to do other stuff. They need to be able to drop what they're doing at all times to address production issues, which will inevitably occur from time to time. This is assuming your organizations needs are static and you won't be keeping a developer on to iterate new releases in the future with new / different functionality, to keep your shiny new system relevant and from becoming "legacy".
Or, you can do what most of the business world does and get a SaaS solution. Many SaaS (Software as a Service) vendors provide free / cheap licensing options for educational institutions. Even if you can't find a solution which offers a free license, for your use case it'll be cheaper in the long run to go SaaS rather than attempting to bootstrap your own Dev shop, and paying for all the mistakes that come along with that enterprise.
What you're looking for is an end user SaaS solution from a vendor that will continue to innovate and has open-source at its core (these type of companies generally don't leverage closed source code to lock-in customers, and if they do they are at least easier to migrate away from). You're also looking for a vendor who's mission is in line with your organization's mission. This requirement is an intangible, but one that will pay massive dividends down the road. Finally, steer clear of any solution that includes the first step of: "Pay our consultant to come to your office to migrate your legacy system to Solution XYZ (which we didn't actually design and build, but trust us, we're certified)." I could make a living on just the contract jobs where I'm paid to come in after these companies and clean up their mess or finish implementing functionality they weren't able to implement before going over budget. Additionally, this type of company almost never cares to understand the domain specific issues or instance specific issues relevant to the customer. They may "specialize in and only do educational software / institutions", but more often then not, the result is boiler plate code from their last three jobs dropped into your servers, totally ignoring the specifics of your system, historical data, and historical work arounds.
To prevent this issue, and to do it right, you should considering either hiring or training one employee to be full time on this software, and have that employee be someone that has intimate knowledge of both your old system and your organization's operations. This employee must also show a willingness to adopt best practices and work from a place of consensus (both as relates to technical AND organizational decisions), rather than someone who will stubbornly implement what they are already comfortable with and refuse to listen to anyone that is not as technologically savvy as they consider themselves to be. This person should consider themselves a steward of the school's system and its many stakeholders (and maintain buy in and seek the input of the other stakeholders, who should include a senior administrative "champion" of the project who also fosters consensus rather than deliver decries), and not consider the school a user of their system. This person should aim to learn the solution inside and out, and develop an effective workflow / ticketing system to see that the system is responsive to it's end users' needs. One of the first orders of business for this person should be to document all aspects of their job. This documentation should be updated whenever any aspect of the role changes. This is the so called "bus manual". If that person is hit by a bus on the way into work any particular morning, the mysteries of operation your system do not go with them. Morbid, but if you only take two things from this post, let this be the first.
Insist on ease of use of the product, clarity of terms of software licensing (so that you aren't surprised by unforeseen costs if you require scaling up, adding new users, or making the system available via a web site, all things which will enviably happen). To avoid vendor lock in and avoid paying shoddy, expensive consults to implement the system, insist, I say again, insist on the following two points. If you take away only two things from this post, let this two part point be the second thing:
2a) And open data model and unfettered / unlimited api access to your data
2b) Excellent end user documentation, both for the api and the application
I'd suggest starting with checking out Tableau, Socrata (http://www.socrata.com http://www.socrata.com), and Informatica. You want something that is as dead easy to use and as powerful as Tableau. And you want something that is as awesome with your data as Socrata. I've worked with a Dev from Socrata at a hackathon, I was really impressed. They've really got their heart in the right place and they have an awesome product. Informatica would be a more powerful/expensive alternative to Socrata, but likely overkill for your needs. I'm sure there are other products out there that will work for your organization with which I'm unfamiliar. Sit with and look over the shoulder, for a whole day, of people that already implemented any solution you're seriously considering. Ask them about their experience of implementing it and utilizing it. Also speak with people that may have gone with another solution, but started out with the same problem you're trying to solve. Ask them what they learned, would have done differently, biggest headaches / opportunities, etc. don't be afraid to solicit guidance from someone in a meetup or university for guidance through the process. I'm certain someone would be happy to serve as a sounding board. Just be wary of anyone eager to take an active role in the project. They are not a long term stakeholder, so they should only be fielding questions and offering wisdom, not driving the process, at all.
Best of luck! :) I have several family members who are educators; I wish you success!!!
- antigen8 11y agooneJob, I can't thank you enough for the depth and thoughtfulness of your response- you're 100% right about looking for a SaaS solution, about finding one that is aligned with our mission. I've been experimenting with Tableau for the last two days and it strikes the perfect balance of incredible power and usability. From what I can tell, Socrata is focused on open data and sharing data with outside organizations. Their product is what we're looking for, their mission is spot on with ours, but our data has to remain secure and private. We need a socrata for internal use. I've shared your thoughts with my administrators, thanks again
- thoumasd 11y agoMay I suggest you to have a look at OpenDataSoft? This is a SaaS solution which lets you publish data both externally AND internally with fine grained access control. It has advanced data processing and data visualization capabilities. Feel free to get in touch If you would like to give it a try. http://www.opendatasoft.com http://www.opendatasoft.com Disclaimer: I work for OpenDataSoft.
- antigen8 11y agoThanks for mentioning this, I'll definitely check it out
- oneJob 11y agoHello antigen8, I wanted to clarify my reasoning for including Socrata in that list of software suggestionss. All-in-one solutions often don't address and sometimes actually create a coupling of your data and the applications you utilize to access it or create value from it. Your initial post mentioned a data store. And we've talked about Tableau. A data store, data warehouse, or database will usually be separate from the applications making use of the data. Analytics software, such as Tableau, makes use of the data source, but is not necessarily meant to be your data source. (http://www.tableau.com/solutions/data-sources http://www.tableau.com/solutions/data-sources and http://www.tableau.com/solutions/environments http://www.tableau.com/solutions/environments) One solution to this problem is to go with an "API-First" architecture pattern. (https://www.leaseweblabs.com/wp-content/uploads/2013/10/api_first.pdf https://www.leaseweblabs.com/wp-content/uploads/2013/10/api_...) The API-First pattern creates an abstraction layer between your application and data source. You can then make changes to your data source (say, switch from MS Access to MS SQL Server, or go from using a collection of Excel files to a single Oracle database) without requiring any changes to your applications. Or, conversely, you could switch out, add to, or remove some of your applications (say, switch from Tableau to Pentaho) without having to touch your data sources. http://onlinehelp.tableau.com/v8.0/pro/online/en-us/extracting_TDE_API.html http://onlinehelp.tableau.com/v8.0/pro/online/en-us/extracti... http://www.tableau.com/new-features/data-engine-api-0 http://www.tableau.com/new-features/data-engine-api-0 http://open-source.socrata.com/architecture/ http://open-source.socrata.com/architecture/ So really Socrata is a super user friendly API solution, which your school can use completely privately and securely in-house, in conjunction with your data sources and applications. The CEO of Socrata actually speaks directly to this use-case here: https://www.quora.com/Are-there-any-products-like-Socrata-but-for-private-data-sets https://www.quora.com/Are-there-any-products-like-Socrata-bu... One scenario would be using PostgreSQL (to store your data) + Socrata (API layer) + Pentaho (analytics, visuals, and reporting). Then, if you don't like Pentaho, you can switch it out for Tableau without having to touch PostgreSQL. Or, if you decide you don't enjoy managing your PostgreSQL instance, you can switch to Google Cloud SQL (https://cloud.google.com/sql/docs/introduction https://cloud.google.com/sql/docs/introduction). Just want to say that I have zero affiliation with any of these companies, and that I'm sure there are other products out there that would accomplish the same solution, architecture wise. Another link that may help decide if this solution might be worth looking into more: http://customer-summit.socrata.com/sessions/added-value-open-datas-internal-use-case/ http://customer-summit.socrata.com/sessions/added-value-open... And, since I mentioned it above, you might want to look at http://www.pentaho.com/ http://www.pentaho.com/. Regardless, I'd spend some time on this site before making any decisions.http://apievangelist.com/ http://apievangelist.com/. And glance through this book: http://htchttp.s3.amazonaws.com/books/apis_a_strategy_guide.pdf http://htchttp.s3.amazonaws.com/books/apis_a_strategy_guide....