kmike84

11 ms·

61.
by kmike84 10y ago
There are many methods for different tasks. What do you mean by 'data extraction', do you have some specific examples?
62.
by kmike84 10y ago
If one doesn't want to dive deep in RL, and just want to start with recent methods then https://www.nervanasys.com/demystifying-deep-reinforcement-l... blog posts provide a very good starting point.
63.
by kmike84 10y ago
+1 to Sutton & Barto. There is a second edition of their book available; see a link at https://webdocs.cs.ualberta.ca/~sutton/book/the-book.html . https://www.udacity.com/course/reinforceme
64.
by kmike84 10y ago
IDEA-based IDEs has a nice support for reStructuredText, I use this PyCharm feature almost daily.
65.
by kmike84 10y ago
* Write tests and run then on CI. Try to figure out how to write tests with a least amount of pain - you'll need lots of them. Use py.test or a similar framework. Doctests are great, but it takes some time to learn how to use them effi
66.
by kmike84 10y ago
I think the main difference is that Scrapy is async - it downloads pages in parallel by default, so it is more efficient. But async APIs can be harder to use - you need callbacks or generators everywhere, so sync packages (like RoboBrowser)
67.
by kmike84 10y ago
It looks like Developer tools also use these JS utilities, so you can't trust Network tab when checking what URL is sent to a remote server; to check what's being sent one have to either inspect network traffic or check it server
68.
by kmike84 10y ago
Yeah, it is a recipe for confusion. Browsers use a workaround: when you copy an URL of a link (using "Copy Link Address" in Chrome or "Copy Link Location" in Firefox) it copies escaped URL, not original URL.
69.
by kmike84 10y ago
nice, this explains it!
70.
by kmike84 10y ago
A good question; I'm not sure UTF-8 is always used because systems we tested were all using UTF-8 system encoding. This question was not relevant for Scrapy because it makes sense to use UTF-8 as default regardless of what browsers do
71.
by kmike84 10y ago
There are 3 ways to get an URL: a) URL is entered to the address bar. Default encoding (it seems UTF-8) is used for query and fragment. b) User clicks on a link. Page URL is used; page encoding is already known. c) Redirect to a new URL hap
72.
by kmike84 10y ago
Thanks for the references!
73.
by kmike84 10y ago
It uses encoding of a page URL is extracted from, not of a page URL leads to.
74.
by kmike84 10y ago
It seems nobody really understands how to handle non-ascii URLs. To get it right for Scrapy (i.e. do same as browsers) we checked how several browsers behave; the results was consistent but surprising, if not crazy. Before sending an URL to
75.
by kmike84 10y ago
Hey, ASRock EP2C602-2T/D16 has EEB 12" x 13" form factor, it may be a bit harder to find a case for it (but that's possible, at least form factor is standard). Also, it has 2 PCIe slots which means you can insert only a
76.
by kmike84 10y ago
Thanks! Yeah, adding a spinning drive is a good idea. It seems part list should be saved to enable comments ( http://pcpartpicker.com/user/kmike/saved/QnGp99 ). I used pcpartspicker for the first time to well,
77.
by kmike84 10y ago
I built a rig a couple days ago; it works nicely both for deep learning and for RAM and CPU-heavy tasks: http://pcpartpicker.com/p/GvTGsY - two 8 core xeons, 128GB RAM, slots for 2 GPUs for ~2000$. Currently it has gtx
78.
by kmike84 11y ago
The first problem is not specific to Scrapy; it is the same if you're using inlineCallbacks from twisted, tornado.gen or async def: there is no way to serialize tasks inside coroutine and save them to disk, e.g. to be able to stop the
79.
by kmike84 11y ago
I was investigating how to add asyncio / async def support to Scrapy (see https://github.com/scrapy/scrapy/issues/1144#issuecomment-14... ). Small examples like the one at the link look neat, but it is no
80.
by kmike84 11y ago
at least on OS X installing numpy and scipy is very easy - pip installs prebuilt wheels by default, no need to compile anything
81.
by kmike84 11y ago
Wrapper induction library is separated from Scrapy: https://github.com/scrapy/scrapely . It is used in Portia under the hood. Portia can be seen as a tool to annotate scrapely templates and define crawling rules and pos
82.
by kmike84 11y ago
Yeah, totally agree; I did it for io.BytesIO vs cStringIO performance issue ( https://news.ycombinator.com/item?id=11037213 ), and Python core devs fixed it very fast even though the fix was far from trivial. You have to wait
83.
by kmike84 11y ago
> The docs where pretty good, but it was unclear sometimes how to proceed; There was a lot of structure to understand in order to get started. Yeah, docs used to be a problem; they improved a lot in 1.0 and 1.1 releases though. > When
84.
by kmike84 11y ago
that's an example of https://en.wikipedia.org/wiki/Galilean_invariance
85.
by kmike84 11y ago
For scientific code I've seen slowdowns, but in the end they turn out to be unrelated to Python 3 - e.g. numpy compiled against different BLAS libraries in Python 2.x and 3.x. There also was a gotcha with replacing cStringIO with io.By
86.
by kmike84 11y ago
A data point: `scrapy bench` is about 1.5x - 2x slower in Python 3.5 than in Python 2.7. This command starts a simple HTTP server which serves HTML pages linked in arbitrary depth, and crawls it using Scrapy. Profiler shows URL parsing is m
87.
by kmike84 11y ago
We're seriously considering this option (maybe for scrapy 2.0?), but no concrete plans yet. It can't be just asyncio - we'll need e.g. aiohttp and other packages. Some links: * POC for aiohttp as a http handler: https:/
88.
by kmike84 11y ago
Thanks for the details! A really nice article, by the way.
89.
by kmike84 11y ago
DAWG name is ambiguous, there are 2 different structures called DAWG - sometimes it is used as a synonym to DAFSA and sometimes it is used as a synonym to DAFSA which has all key substrings in it, not only keys. The linked DAWG wikipedia ar
90.
by kmike84 11y ago
I'd like to try compressing DOI urls; what was the file used? https://archive.org/details/doi-urls has 11 files; on my machine they are 4.3GB combined (after uncompressing), and there is no a file which is 2.8 GB

More ›