4 ms·
Possibly not the right place for this, but has anyone noticed how unreasonably complicated it is to code seemingly simple ML algorithms? For instance text docum
by jakear 7y ago
Possibly not the right place for this, but has anyone noticed how unreasonably complicated it is to code seemingly simple ML algorithms? For instance text document classification. I'd expect there to be a package everyone uses that exposes (rough sketch, but you get the idea):
interface TextClassifier {
train(documents: {contents: string, label: string}[], options: TextClassifierOptions): TrainingStats
classify(document: string): string
}
But instead everyone seems to want me to implement tokenization and stop word skipping and TF-IDF and etc etc etc. Which sure, I can do, but why should I? I thought that was bullshit busywork college classes made me do, not the industry standard. Perhaps this is revealing my JS background, where I expect there to be an NPM package for every conceivable oft-repeated menial task, is that just not the vibe in PyPI? I've been checking out classification through SciKit-learn and Keras, is there some wrapper package I've overlooked?
- sansnomme 7y agoHow else do you think AMZN/GOOG/MSFT are able to get you to use their turnkey ML-as-as-service?
- WoodenChair 7y agoApple has created this and even gave it a GUI and it's called Create ML: https://developer.apple.com/machine-learning/create-ml/ https://developer.apple.com/machine-learning/create-ml/
- onion2k 7y agoTo use an analogy from mathematics, it seems like you should be looking for "applied machine learning" rather then "pure machine learning". In my experience if a course or article doesn't explicitly state it's "applied" then you'll have to wade through a lot of background and theory.
- zby 7y agoTokenization and stop word skipping is something that in most cases needs adjustments. These adjustments are too complex to be config settings - they need to be in code. The solution could be default implementations that you could replace by Dependency Injection or Inheritance with Overriding. Unfortunately DI is not yet popular enough.
- etaioinshrdlu 7y agoI think SciKit learn and Keras represent a decent level of abstraction for most real world use cases. You end up wanting a decent amount of control for most tasks. More abstract would give you tools called things like AutoML and they do work, but you may have a hard time improving on what they do. They are an area of active research.
- pedrosorio 7y agoFor a production grade library with good default implementations of the things you mentioned, look into spacy.