3 ms·
Something I love about pandas is that often you can pass a URL in place of a file name. The other day I needed to scrape data from a table on a webpage. Thinki
by jpxw 7y ago
Something I love about pandas is that often you can pass a URL in place of a file name.
The other day I needed to scrape data from a table on a webpage. Thinking about traversing the DOM and building up an array was already giving me a headache. Thankfully pandas has the “read_html” function. Getting a list of dataframes for each table on the page was as easy as:
dfs = pd.read_html(url)
- reportgunner 7y ago1. pandas can do date ranges date_range = pd.date_range(date_from, date_to, freq="D") 2. merge with indicator if you set `indicator` parameter of merge() to True pandas adds a column that tells you which dataset the row came from 3. merge with approximate match - the tolerance parameter of merge_asof() pd.merge_asof(trades, quotes, on="timestamp", by='ticker', tolerance=pd.Timedelta('10ms'), direction='backward') 4. Create an Excel report and add some charts 5. Save the dataframe in gzipped form edit: formatting edit2: I managed to get an outline link https://outline.com/fY26Aw https://outline.com/fY26Aw
- dr_zoidberg 7y agoThis is basically a summary of the link posted by OP. Edit: not that it's a bad thing, I just mention it so people who read this know for sure they can avoid the link if they already know how to use the tips listed here. The article has code examples and more details.
- HIP_HOP 7y agoThat is a useful one, yes.
- akx 7y agoOn the flipside, now you need to validate your input in one more way if you don't want to accidentally be connecting to somewhere on the internet. EDIT: for instance, it looks like on a suitably configured environment, passing in `s3://` and `gs://` URLs works fine too...