4 ms·

Foursquare Open Source Places: A new foundational dataset

154 points by po 2y ago

unmoeboiner2011 2y ago
[dead]
zeristor 2y ago
A nice exploration, this is a link to the announcement itself from the article: https://location.foursquare.com/resources/blog/products/foursquare-open-source-places-a-new-foundational-dataset-for-the-geospatial-community/ https://location.foursquare.com/resources/blog/products/four...
deleted 2y ago
[deleted]
TekMol 2y ago
What is the motivation of Foursquare to open source their places data?
- big_jimmer 2y ago
  Incentivise integrations to gain market share, I assume? Maybe also grow developer knowledge and trust? I honestly haven't thought about 4S in a long time, although I am outside the US so maybe that's why?
- nness 2y ago
  Largely, improve the data quality from contributions.
  - hk__2 2y ago
    Why should one contribute to this dataset instead of OpenStreetMap?
- Pikamander2 2y ago
  Cynically, a desperate attempt at restoring its own relevance, which it hasn't had since 2011. Not that I'm complaining; it's always nice to have more options.
- chippy 2y ago
  There are a few reasons. Firstly, the published reason is they intend to let people contribute and maintain the dataset. This is a good open data mindset to allow a dataset to maintain its health and grow. It would make the dataset sustainable. An unpublished reason is that it could be a sign that things within Foursquare are winding down. Making the dataset sustainable reduces internal costs and increases the value to themselves and others. (like the reasons to open source software) Foursquare City Guide is shutting down, and recently laid off 105 (25%) employees. I would imagine releasing data before things stop is a good ethical move. https://foursquare.com/city-guide-sunset/ https://foursquare.com/city-guide-sunset/ https://techcrunch.com/2024/05/23/foursquare-just-laid-off-105-employees/ https://techcrunch.com/2024/05/23/foursquare-just-laid-off-1... "The letter does, however, address which units were impacted and, relatedly, the divisions that Foursquare plans to ditch, including Visits, OCF, and Foursquare City Guide. According to Little’s letter, Foursquare is also pausing work on a number of other initiatives, including "Mobile Developers Tools, Geode, and the current version of FSQ Insights."
- snthd 2y ago
  "Places Pro" and "Premium Data" are not open source.
somishere 2y ago
Looking forward the taking a peek. Both at the foursquare data and datasette .. which I feel I've seen spruiked so much I want the tshirt. As an aside a place in my (small, regional) city is listed in the sample gist. Feel like I should win something.
jpnc 2y ago
[flagged]
- JosephRedfern 2y ago
  Because they're interesting?
- graemep 2y ago
  Because his articles are really popular with people here. he does lots of interesting stuff, and most of it is open source. You will notice only a few of his posts are posted here by him - lots of different people.
- dewey 2y ago
  Everyone can share their blog posts here, and it will only show up on the front page if enough people think it's interesting enough to be there. Apparently more people (Including myself) think it's interesting vs. uninteresting.
- deleted 2y ago
  [deleted]
jillesvangurp 2y ago
I used to work for Nokia Maps (before it was renamed Here Maps) working on their places registry, which was a proprietary data set compiled from various sources of data with a similar amount of places. It was really messy data. Lots of duplicates, lots of stale data, lots of misleading data (e.g. hotels claiming to be right on the beach that in reality were a few kilometers away), etc. Part of my job was cleaning that data up and making sense of it. The registry was used to power Nokia's address and places search and became hugely popular when Nokia started bundling free maps and navigation with all their phones. Up until then you had to pay for in car navigation. Foursquare at the time was an app for checking into places and becoming the mayor. Which was hugely popular with several of my colleagues that would obsessively checking to places to claim mayorship of places they liked. To ensure Foursquare had good content, they came up with a clever solution: just ask users that are there. This community driven curation was pretty smart. They got users to categorize and correct information. Later when doing my first startup, I tried to work with open place data. It was a mess. There were bits and pieces available but nothing great. I actually crawled Foursquare and a few other websites in HTML form at some point to get at their ~30M places (at the time). In the end, we never launched anything and there were obvious IP issues with that data set that would have prevented us from using that commercially. Anyway, Foursquare got all corporate and the whole checking in thing was spun off into a separate app and the main app became a glorified restaurant recommendation thingy which never really managed to stand out from the field of other recommendation thingies. And a data set + API you could license. Because that was still pretty good. More recently, other companies partnered up to form Overture, which is a competing open data set that Bing, Amazon, and others are contributing to. Overture is kind of solving the same problem. I'm guessing that's a big contributing reason for Foursquare to be open sourcing their data. It would be interesting to see if their data is going to be contributed to Overture or whether they are somehow against that. Last I checked, Overture still had lots of data issues. Combining the two data sets would probably help.
- nunodonato 2y ago
  I remember the good old time of Foursquare. What caused its downfall? The checkin thing was quite nice, I even remember some places offering discounts or having special offers to checked in visitors or mayors. Its one of those apps that could still have a vibrant community 10 years later.
jarpineh 2y ago
There’s also Overture Maps with its 3.7B features: https://overturemaps.org/ https://overturemaps.org/ Foursquare could conceivably become a member and donate their data.
- twelvechairs 2y ago
  Both are open source. However, Foursquare open source places seems to be Apache 2 licensed where overture is CDLA Permissive 2.0. So not sure if you can port from one to another. It does my head in trying to work out the kind of licensing issues. I'd imagine foursquare is much better on 'places' like this - overture has other things like building footprints etc.
hu3 2y ago
Nice. Works with clickhouse local cli too. ./clickhouse local -q "select count(*) from s3('s3://fsq-os-places-us-east-1/release/dt=2024-11-19/places/parquet/places-00000.snappy.parquet');" prints: 4180424
davidavidavid 2y ago
[flagged]
davidavidavid 2y ago
[flagged]
notpushkin 2y ago
Neat! Wondering if it's possible to reuse this is OSM?
deleted 2y ago
[deleted]
Qem 2y ago
It's a bit bothersome they nudge you to install and use aws tools to download the data, instead of providing a plain CSV or JSON downloadable dump instead.
- morgenkaffee 2y ago
  I also found it hard to deal with the AWS CLI and Parquet. I built this site so you can download it in CSV and Geopackage. https://repromptai.com/data/foursquare https://repromptai.com/data/foursquare
- simonw 2y ago
  It's S3 so this works to list the bucket contents: curl -s "https://fsq-os-places-us-east-1.s3.amazonaws.com/?list-type=2&prefix=release/dt=2024-11-19/places/parquet/&delimiter=/" And you can download a file called places-00000.snappy.parquet like this: curl -O "https://fsq-os-places-us-east-1.s3.amazonaws.com/release/dt=2024-11-19/places/parquet/places-00000.snappy.parquet"
- dbbk 2y ago
  I mean not really, why would you want to direct download a CSV or JSON dump this big

154 points by po 2y ago

unmoeboiner2011 2y ago

[dead]

A nice exploration, this is a link to the announcement itself from the article: https://location.foursquare.com/resources/blog/products/foursquare-open-source-places-a-new-foundational-dataset-for-the-geospatial-community/ https://location.foursquare.com/resources/blog/products/four...

deleted 2y ago

[deleted]

TekMol 2y ago

What is the motivation of Foursquare to open source their places data?

big_jimmer 2y ago
Incentivise integrations to gain market share, I assume? Maybe also grow developer knowledge and trust? I honestly haven't thought about 4S in a long time, although I am outside the US so maybe that's why?
nness 2y ago
Largely, improve the data quality from contributions.
- hk__2 2y ago
  Why should one contribute to this dataset instead of OpenStreetMap?
Pikamander2 2y ago
Cynically, a desperate attempt at restoring its own relevance, which it hasn't had since 2011. Not that I'm complaining; it's always nice to have more options.
chippy 2y ago
There are a few reasons. Firstly, the published reason is they intend to let people contribute and maintain the dataset. This is a good open data mindset to allow a dataset to maintain its health and grow. It would make the dataset sustainable. An unpublished reason is that it could be a sign that things within Foursquare are winding down. Making the dataset sustainable reduces internal costs and increases the value to themselves and others. (like the reasons to open source software) Foursquare City Guide is shutting down, and recently laid off 105 (25%) employees. I would imagine releasing data before things stop is a good ethical move. https://foursquare.com/city-guide-sunset/ https://foursquare.com/city-guide-sunset/ https://techcrunch.com/2024/05/23/foursquare-just-laid-off-105-employees/ https://techcrunch.com/2024/05/23/foursquare-just-laid-off-1... "The letter does, however, address which units were impacted and, relatedly, the divisions that Foursquare plans to ditch, including Visits, OCF, and Foursquare City Guide. According to Little’s letter, Foursquare is also pausing work on a number of other initiatives, including "Mobile Developers Tools, Geode, and the current version of FSQ Insights."
snthd 2y ago
"Places Pro" and "Premium Data" are not open source.

somishere 2y ago

Looking forward the taking a peek. Both at the foursquare data and datasette .. which I feel I've seen spruiked so much I want the tshirt. As an aside a place in my (small, regional) city is listed in the sample gist. Feel like I should win something.

jpnc 2y ago

[flagged]

JosephRedfern 2y ago
Because they're interesting?
graemep 2y ago
Because his articles are really popular with people here. he does lots of interesting stuff, and most of it is open source. You will notice only a few of his posts are posted here by him - lots of different people.
dewey 2y ago
Everyone can share their blog posts here, and it will only show up on the front page if enough people think it's interesting enough to be there. Apparently more people (Including myself) think it's interesting vs. uninteresting.
deleted 2y ago
[deleted]

jillesvangurp 2y ago

I used to work for Nokia Maps (before it was renamed Here Maps) working on their places registry, which was a proprietary data set compiled from various sources of data with a similar amount of places. It was really messy data. Lots of duplicates, lots of stale data, lots of misleading data (e.g. hotels claiming to be right on the beach that in reality were a few kilometers away), etc. Part of my job was cleaning that data up and making sense of it. The registry was used to power Nokia's address and places search and became hugely popular when Nokia started bundling free maps and navigation with all their phones. Up until then you had to pay for in car navigation. Foursquare at the time was an app for checking into places and becoming the mayor. Which was hugely popular with several of my colleagues that would obsessively checking to places to claim mayorship of places they liked. To ensure Foursquare had good content, they came up with a clever solution: just ask users that are there. This community driven curation was pretty smart. They got users to categorize and correct information. Later when doing my first startup, I tried to work with open place data. It was a mess. There were bits and pieces available but nothing great. I actually crawled Foursquare and a few other websites in HTML form at some point to get at their ~30M places (at the time). In the end, we never launched anything and there were obvious IP issues with that data set that would have prevented us from using that commercially. Anyway, Foursquare got all corporate and the whole checking in thing was spun off into a separate app and the main app became a glorified restaurant recommendation thingy which never really managed to stand out from the field of other recommendation thingies. And a data set + API you could license. Because that was still pretty good. More recently, other companies partnered up to form Overture, which is a competing open data set that Bing, Amazon, and others are contributing to. Overture is kind of solving the same problem. I'm guessing that's a big contributing reason for Foursquare to be open sourcing their data. It would be interesting to see if their data is going to be contributed to Overture or whether they are somehow against that. Last I checked, Overture still had lots of data issues. Combining the two data sets would probably help.

nunodonato 2y ago
I remember the good old time of Foursquare. What caused its downfall? The checkin thing was quite nice, I even remember some places offering discounts or having special offers to checked in visitors or mayors. Its one of those apps that could still have a vibrant community 10 years later.

jarpineh 2y ago

There’s also Overture Maps with its 3.7B features: https://overturemaps.org/ https://overturemaps.org/ Foursquare could conceivably become a member and donate their data.

twelvechairs 2y ago
Both are open source. However, Foursquare open source places seems to be Apache 2 licensed where overture is CDLA Permissive 2.0. So not sure if you can port from one to another. It does my head in trying to work out the kind of licensing issues. I'd imagine foursquare is much better on 'places' like this - overture has other things like building footprints etc.

hu3 2y ago

Nice. Works with clickhouse local cli too. ./clickhouse local -q "select count(*) from s3('s3://fsq-os-places-us-east-1/release/dt=2024-11-19/places/parquet/places-00000.snappy.parquet');" prints: 4180424

davidavidavid 2y ago

[flagged]

davidavidavid 2y ago

[flagged]

notpushkin 2y ago

Neat! Wondering if it's possible to reuse this is OSM?

deleted 2y ago

[deleted]

Qem 2y ago

It's a bit bothersome they nudge you to install and use aws tools to download the data, instead of providing a plain CSV or JSON downloadable dump instead.

morgenkaffee 2y ago
I also found it hard to deal with the AWS CLI and Parquet. I built this site so you can download it in CSV and Geopackage. https://repromptai.com/data/foursquare https://repromptai.com/data/foursquare
simonw 2y ago
It's S3 so this works to list the bucket contents: curl -s "https://fsq-os-places-us-east-1.s3.amazonaws.com/?list-type=2&prefix=release/dt=2024-11-19/places/parquet/&delimiter=/" And you can download a file called places-00000.snappy.parquet like this: curl -O "https://fsq-os-places-us-east-1.s3.amazonaws.com/release/dt=2024-11-19/places/parquet/places-00000.snappy.parquet"
dbbk 2y ago
I mean not really, why would you want to direct download a CSV or JSON dump this big