portInit

5 ms·

1.
by portInit 4y ago
Appreciate you taking the time to play around with crul and share your thoughts! They're incredibly valuable. 1. Although it has limitations, were you able to try the normalize command? https://www.crul.com/docs/qu
2.
by portInit 4y ago
We certainly have, it is admittedly quite new to both of us, so we have been exploring the best way to introduce something like this, as well as tying up technical bits. The open core model with commercial features is certainly appealing. O
3.
by portInit 4y ago
Thank you! Will make this more clear.
4.
by portInit 4y ago
Thank you! The query below would get you the full page text, although it likely wont be too legible. I'll read up on readability JS some, it looks quite magic and could possibly be added in. open https://news.ycombinator.com
5.
by portInit 4y ago
I'm having trouble replicating. Although I did get a prompt from Safari to allow downloads from crul.com. The screenshot looks like it is redirecting from the button, is that what you are seeing - rather than the button triggering a do
6.
by portInit 4y ago
Yeah exactly. Living up to your username :) nice find! Note this is a global default, unlike domain policies which are associated with a domain.
7.
by portInit 4y ago
Thanks for the heads up. Could you share what browser you are using? And are you downloading from https://www.crul.com/account ?
8.
by portInit 4y ago
You would need to do a bit filtering to get the exact text you need. For example, to get the text for this post you could run: open https://news.ycombinator.com/item?id=34970917 || filter "(attributes.class == 't
9.
by portInit 4y ago
We hand wrote a number of integrations, sometimes it was a simple as reusing a schema with slightly different values, we are also using the awesome https://www.benthos.dev/ !
10.
by portInit 4y ago
That's unfortunate. A quick look suggests ISP but maybe not with the hotspot. Will do a little more digging.
11.
by portInit 4y ago
Thank you! Docker image has everything for functionality, there's just an update check that pings our end to check for updates.
12.
by portInit 4y ago
Sorry to hear that - we do need to think about this. It's our first pass at product tiers and features and we may need to adjust. Scheduling and Domain Policies were the main features we chose to gate initially as they don't affec
13.
by portInit 4y ago
We'll have to see how close our current postgres integration is! Would like to understand more and will reach out.
14.
by portInit 4y ago
Thanks for this, we're still trying to figure out these details ourselves. Related to the quote, we've seen interest for API data wrangling, where prepackaged data feeds can be cumbersome to edit, or other implementation details b
15.
by portInit 4y ago
Ah! We didn't quite get an html table command in this release but it will be in the next one. Here's a query that shows an option, but the table command will be far more straightforward. open https://www.w3schools.com&#
16.
by portInit 4y ago
Thank you! Really means a lot to us
17.
by portInit 4y ago
Agreed, caching does come with its own set of quirks and mind-numbing bugs, crul does have a caching override flag at the command/stage level which alleviates some of this: https://www.crul.com/docs/queryconcepts&#
18.
by portInit 4y ago
Yes that is possible, although we currently have the scheduler set as an enterprise feature. We should look into a free trial, I enjoyed the flow of the EasyDataTransform installation with the free trial option.
19.
by portInit 4y ago
Would love to chat and try a few use cases together! At first glance I see you can drag in csv files, which could be generated by crul and either manually downloaded or scheduled and written to the filesystem. Crul is a really easy way of p
20.
by portInit 4y ago
Yeah the default domain throttle policy is 1 req per second per domain. Configurable through domain policies https://www.crul.com/docs/features/domain-policies - although currently an enterprise feature. We found
21.
by portInit 4y ago
Thank you and your comment really warms our hearts. It's been fun building in the "cave" but comes with self doubt. We've built using a microservice architecture to allow us to scale out the parts that need to scale, mai
22.
by portInit 4y ago
This makes sense and I am curious about this. Was there consistency between those 1k client sites or were they all rather different? Mind if I reach out?
23.
by portInit 4y ago
lol - yeah. https://www.imdb.com/title/tt0085811/?ref_=tt_urv We've been thinking of ways to make the pronunciation a bit clearer, maybe a mascot or something. Open to ideas!
24.
by portInit 4y ago
It's a tricky question. Part of it is looking at APIs as the main source of data for scheduled queries/data feeds. Crul sort of operates as a text only browser when interacting with a single page at a time, but when you expand and
25.
by portInit 4y ago
At this point we're considering it a foundational concept to build around - web content changes, so our best option currently is to make the query as easy as possible to change, and alert when things break. We have done some preliminar
26.
by portInit 4y ago
Will absolutely reach out! Our experience has been that just getting data is often really challenging, so we've really focused on that piece, and being able to easily share with destinations that are purpose built for analytics, viz, e
27.
by portInit 4y ago
Thanks! The find ( https://www.crul.com/docs/commands/find ) command works really well if you are trying to construct a filter expression and just want to quickly look for results containing a particular string so y
28.
Show HN: Crul – Query Any Webpage or API(crul.com)
241 points by portInit 4y ago|79 comments
29.
Post Manhattan Project FOMO – George Gamow and “Maladia Biologica”(apicaster.wordpress.com)
2 points by portInit 6y ago|0 comments
30.
by portInit 6y ago
Much appreciated! I'm only scratching the surface of what is possible but it was so surprisingly easy for someone with minimal starting knowledge to get something going. I can definitely start to see that the nutrient, ph, light detail

More ›