liuliu

81 ms·

871.
by liuliu 8y ago
Assuming you mean Fourier domain. It is not very useful to keep the activation in that domain. Winograd transform operates at 4x4 tiles (it is effective because 16 FMA of that 4x4 tiles can generate 2x2 activations (roughly speaking), while
872.
by liuliu 8y ago
It would actually be interesting to quantize Winograd transformed weights directly and then see whether we can squeeze some accuracy out (since Winograd is more accurate numerically than direct convolution). Maybe just too much work for pot
873.
by liuliu 8y ago
The problem is that if you build the graph as-you-go, dataflow graph optimizations cannot be done efficiently (some high level optimizations such as data layout optimization, automatic data / model parallelism etc.). Swift can do all t
874.
by liuliu 8y ago
It is challenging. Most likely your data at rest are compressed. Go with the highest-end SSD money can buy, RAID0 them, and layout your compressed data as sequentially as possible. You still need to trip to the main memory and decode though
875.
by liuliu 9y ago
And very few people can know 100% 100% times. We build software on top of abstractions. Abstraction leaks, abstraction breaks and abstraction isolates you from "know what you are doing". If you build software in C, you will build
876.
by liuliu 9y ago
There is XLA though: https://www.tensorflow.org/performance/xla/ But I guess unlike TC / Halide / TVM, that is less generic and much less customization.
877.
by liuliu 9y ago
When optimizing start up time for apps.
878.
by liuliu 9y ago
MIT license doesn't contain any patent clause. So it is the same as BSD + Patents with patents part revoked. Thus, in theory, FB can still have a case on patent over React even if you use the source with copyright grant. Apache 2.0 has
879.
by liuliu 9y ago
To be fair, CNTK (BrainScript) has quite impressive list of features to support dynamic control structure (in a symbolic fashion, comparing to PyTorch which delegated much of the dynamic control structure to underlying language Python). I t
880.
by liuliu 9y ago
Doesn't seem to have support for control structure (If, While etc.).
881.
by liuliu 9y ago
That assumes distributed environment (for example, server or microservices). CPU / memory / bandwidth is finite on client-side applications (maybe it is more Swift heavy actually) and you cannot simply stop waiting and not waste r
882.
by liuliu 9y ago
Async / await also need a idiom to accompany on how to support cancellation. In practice, cancellation happens a lot because the unbounded latency for async operations. Without a throughout support, async / await syntax is probabl
883.
by liuliu 9y ago
It doesn't matter. Most of the time, especially with Tensorflow, you will have a static computation graph somewhere. Nothing (well, implementation-defined) prevent you compile that into CoreML / Caffe / Caffe2 or just Tensorf
884.
by liuliu 9y ago
For 1)., if you use a LLVM backed language, XRay supposedly should be it: https://llvm.org/docs/XRay.html
885.
by liuliu 9y ago
Lanczos or Gaussian are separable filters. That reduces computation considerably.
886.
by liuliu 10y ago
Thanks. Yeah, my understanding is that you cannot have eager execution all the way down, some eager executions inside the function is fine, but you shouldn't have these outside the function blocks. Without this function block abstracti
887.
by liuliu 10y ago
Saying it is around the same time is to ignore the history. Metal is available since Jun 2014. Vulkan announced on 2015 and the initial release is on 2016.
888.
by liuliu 10y ago
From my understanding you cannot do the similar dynamic batching with PyTorch since it gets eagerly executed. Not sure about Chainer. Can you explain a bit more about how this can be applied to other frameworks?
889.
by liuliu 10y ago
Let's assume Tensorflow has basic symbolic computation graph expressiveness. What you would do is to build a symbolic representation while executing your graph inline, your symbolic representation doesn't need to have any control
890.
by liuliu 10y ago
You can build both the symbolic computation graph and do the computation at the time when defining the network architecture, thus, gaining the ability to be "dynamic" and also supporting advanced features with the symbolic represe
891.
by liuliu 10y ago
Someone has to pay for that. This is an open-source project.
892.
by liuliu 10y ago
The tools and stdlib improved ton too. I had address sanitizer and undefined sanitizer default on for a while and that helped a lot on memory issues. Things such as int32_t, intptr_t, ptrdiff_t made it is easier to be confident about your p
893.
by liuliu 10y ago
More about data hazard. You can arrange your access pattern pretty well, but fast-math helps to avoid data dependency therefore better instruction level parallelism and pipelining.
894.
by liuliu 10y ago
The problem is data. I am not sure how to collect enough data for this easily (on the order of 50k or so, we don't need to train from scratch).
895.
by liuliu 10y ago
LINQ: https://msdn.microsoft.com/en-us/library/bb397926.aspx Most of the time, the problem is to "embed" a foreign language (SQL) into an existing one (Python, Ruby, etc.), but you can do it in a proper
896.
by liuliu 10y ago
Both of the implementations you mentioned are not best known for its speed, rather, its their versatility (or bloatedness, some may argue).
897.
by liuliu 11y ago
Or flashcache: https://github.com/facebook/flashcache Or bcache: https://bcache.evilpiepirate.org/
898.
by liuliu 11y ago
Cannot emphasize enough how important it is nowadays for any serious work to have bigger RAM. For a machine with 128GiB memory, you can reside a lot of pre-computed results comfortably in RAM, and substantially speed up your computations.
899.
by liuliu 11y ago
Exactly. It is very hard to make credible predictions for longer period of time, but linear extrapolation normally works for 2 to 5 year scope http://liuliu.me/eye/predicting-future-2016/
900.
by liuliu 11y ago
Exactly. See The Paris Review's interview with Ursula K. Le Guin for a comparison: http://www.theparisreview.org/interviews/6253/the-art-of-fic...

More ›