4 ms·

General Transit Feed Specification and Clojure

78 points by simonpure 7y ago

markc 7y ago
Fun project! I also used Clojure for a transit status project - calling the REST API of my local bike share system to display how many bikes were in the nearest station. Clojure is great for exploring and transforming that kind of data. One small tip for the author: clojure.data.csv would simplify csv handling. Its readme includes a simple routine to transform the data to maps.
- mycall 7y ago
  Can you track which bikes move to which station?
  - markc 7y ago
    No, because that data isn’t available in the public API.
- flaie 7y ago
  Author here, thank you for the comment, you're right in the latest code I replaced it with clojure.data.csv but for the article I kept it that way to show that you don't need dependencies for such little utilities, split, map and zipmap and we're done.
BrianHenryIE 7y ago
You reminded me to work on my Jump Bikes Siri Shortcut which uses General Bike Share Feed specification. https://github.com/BrianHenryIE/Bikeshare-Siri-Shortcuts https://github.com/BrianHenryIE/Bikeshare-Siri-Shortcuts I made it so I could ask "where's the nearest Jump bike?" in Sacramento. At some point the API stopped working, but, after checking today, it was just the URLs of the updated feeds had changed (the old ones still 200 with no actual data) so I think I've managed to fix it. Unfortunately, Jump shut down in Sacramento because of Corona virus! Overall, it only works OK. When Apple Shortcuts requests GPS location, it doesn't just return the GPS, it makes a HTTP call to get the street info, which isn't needed here and slows things down. Then the gbfs.uber.com API is very slow... as though they don't cache at all, whereas I thought they only refreshed once/minute: https://gbfs.uber.com/v1/sacb/station_information.json https://gbfs.uber.com/v1/sacb/station_information.json https://github.com/NABSA/gbfs/pull/190 https://github.com/NABSA/gbfs/pull/190 https://www.sacbee.com/news/business/article241326756.html https://www.sacbee.com/news/business/article241326756.html
simongray 7y ago
From the title, my mind was primed to think this was related to Transit, which is (primarily) a Clojure standard for sending Clojure data structures back and forth between frontend and backend: https://github.com/cognitect/transit-format https://github.com/cognitect/transit-format It's cool to see some practical Clojure examples on hacker news.
jlarocco 7y ago
Seems like they could cut out a lot of that code by using macros. Here's a quick and dirty Common Lisp script that reads the CSVs from the .zip file and generates classes at runtime using the headers in each CSV for the slot names. It also defines a 'select' function to make queries easy to write. It's a little too long to post here: https://gist.github.com/jl2/fd324fa3faa919803152465fade14b6b https://gist.github.com/jl2/fd324fa3faa919803152465fade14b6b And here's what it looks like to use it: CL-USER> (defparameter *transit-data* (read-transit-data "~/Downloads/gtfs_current.zip")) CL-USER> (select (table *transit-data* :stops) 'stop-name "PIERNE") ((STOPS "PIERNE01" "" "PIERNE" "" "49.102273" "6.179863" "" "http://lemet.fr/screen/index2.php?stop=257" "0" "" "1") (STOPS "PIERNE02" "" "PIERNE" "" "49.102558" "6.179825" "" "http://lemet.fr/screen/index2.php?stop=234" "0" "" "1")) CL-USER> (select (select (table *transit-data* :trips) 'trip-headsign "L5a - MAISON NEUVE") 'route-id "5-77") ((TRIPS "5-77" "SMIN_H1920-CLaS50V2-Lun-Sam-98" "409137-SMIN_H1920-CLaS50V2-Lun-Sam-98" "L5a - MAISON NEUVE" "1" "60112" "50057") (TRIPS "5-77" "SMIN_H1920-CLaS50V2-Lun-Sam-98" "409138-SMIN_H1920-CLaS50V2-Lun-Sam-98" "L5a - MAISON NEUVE" "1" "60111" "50057") ;; ... )
- markc 7y ago
  >reads the CSVs from the .zip file Neat! I can't find anything in Clojure that does that, so I guess I'd have to use java.util.zip via Clojure ineterop. But making records and doing queries is easy: > generate classes at runtime using the headers in each CSV In Clojure: (ns gtfs.core (:require [clojure.data.csv :as csv])) (defn load-records [f] (let [csv-data (csv/read-csv (slurp f))] (map zipmap (->> (first csv-data) ;; First row is the header (map keyword) repeat) (rest csv-data)))) (def agency-maps (load-records "data/agency.txt")) (def routes-maps (load-records "data/routes.txt")) ;; etc. (count routes-maps) ;; 30 (first routes-maps) ;; {: route_id "1-100", ;; :route_short_name "1", ;; :route_long_name "Ligne 1", ;; :route_desc "", ;; :route_type "3", ;; :route_url "", ;; :route_color "#B22E75"} > defines a 'select' function to make queries easy to write ;; Query for long_name and route_color (def route_ln_color (map #(select-keys % [:route_long_name :route_color]) routes-maps)) (take 2 route_ln_color) ;; ({:route_long_name "Ligne 1", :route_color "#B22E75"} ;; {:route_long_name "Ligne 2", :route_color "#B22E75"})

78 points by simonpure 7y ago

Fun project! I also used Clojure for a transit status project - calling the REST API of my local bike share system to display how many bikes were in the nearest station. Clojure is great for exploring and transforming that kind of data. One small tip for the author: clojure.data.csv would simplify csv handling. Its readme includes a simple routine to transform the data to maps.

mycall 7y ago
Can you track which bikes move to which station?
- markc 7y ago
  No, because that data isn’t available in the public API.
flaie 7y ago
Author here, thank you for the comment, you're right in the latest code I replaced it with clojure.data.csv but for the article I kept it that way to show that you don't need dependencies for such little utilities, split, map and zipmap and we're done.

BrianHenryIE 7y ago

You reminded me to work on my Jump Bikes Siri Shortcut which uses General Bike Share Feed specification. https://github.com/BrianHenryIE/Bikeshare-Siri-Shortcuts https://github.com/BrianHenryIE/Bikeshare-Siri-Shortcuts I made it so I could ask "where's the nearest Jump bike?" in Sacramento. At some point the API stopped working, but, after checking today, it was just the URLs of the updated feeds had changed (the old ones still 200 with no actual data) so I think I've managed to fix it. Unfortunately, Jump shut down in Sacramento because of Corona virus! Overall, it only works OK. When Apple Shortcuts requests GPS location, it doesn't just return the GPS, it makes a HTTP call to get the street info, which isn't needed here and slows things down. Then the gbfs.uber.com API is very slow... as though they don't cache at all, whereas I thought they only refreshed once/minute: https://gbfs.uber.com/v1/sacb/station_information.json https://gbfs.uber.com/v1/sacb/station_information.json https://github.com/NABSA/gbfs/pull/190 https://github.com/NABSA/gbfs/pull/190 https://www.sacbee.com/news/business/article241326756.html https://www.sacbee.com/news/business/article241326756.html

simongray 7y ago

From the title, my mind was primed to think this was related to Transit, which is (primarily) a Clojure standard for sending Clojure data structures back and forth between frontend and backend: https://github.com/cognitect/transit-format https://github.com/cognitect/transit-format It's cool to see some practical Clojure examples on hacker news.

jlarocco 7y ago

Seems like they could cut out a lot of that code by using macros. Here's a quick and dirty Common Lisp script that reads the CSVs from the .zip file and generates classes at runtime using the headers in each CSV for the slot names. It also defines a 'select' function to make queries easy to write. It's a little too long to post here: https://gist.github.com/jl2/fd324fa3faa919803152465fade14b6b https://gist.github.com/jl2/fd324fa3faa919803152465fade14b6b And here's what it looks like to use it: CL-USER> (defparameter *transit-data* (read-transit-data "~/Downloads/gtfs_current.zip")) CL-USER> (select (table *transit-data* :stops) 'stop-name "PIERNE") ((STOPS "PIERNE01" "" "PIERNE" "" "49.102273" "6.179863" "" "http://lemet.fr/screen/index2.php?stop=257" "0" "" "1") (STOPS "PIERNE02" "" "PIERNE" "" "49.102558" "6.179825" "" "http://lemet.fr/screen/index2.php?stop=234" "0" "" "1")) CL-USER> (select (select (table *transit-data* :trips) 'trip-headsign "L5a - MAISON NEUVE") 'route-id "5-77") ((TRIPS "5-77" "SMIN_H1920-CLaS50V2-Lun-Sam-98" "409137-SMIN_H1920-CLaS50V2-Lun-Sam-98" "L5a - MAISON NEUVE" "1" "60112" "50057") (TRIPS "5-77" "SMIN_H1920-CLaS50V2-Lun-Sam-98" "409138-SMIN_H1920-CLaS50V2-Lun-Sam-98" "L5a - MAISON NEUVE" "1" "60111" "50057") ;; ... )

markc 7y ago
>reads the CSVs from the .zip file Neat! I can't find anything in Clojure that does that, so I guess I'd have to use java.util.zip via Clojure ineterop. But making records and doing queries is easy: > generate classes at runtime using the headers in each CSV In Clojure: (ns gtfs.core (:require [clojure.data.csv :as csv])) (defn load-records [f] (let [csv-data (csv/read-csv (slurp f))] (map zipmap (->> (first csv-data) ;; First row is the header (map keyword) repeat) (rest csv-data)))) (def agency-maps (load-records "data/agency.txt")) (def routes-maps (load-records "data/routes.txt")) ;; etc. (count routes-maps) ;; 30 (first routes-maps) ;; {: route_id "1-100", ;; :route_short_name "1", ;; :route_long_name "Ligne 1", ;; :route_desc "", ;; :route_type "3", ;; :route_url "", ;; :route_color "#B22E75"} > defines a 'select' function to make queries easy to write ;; Query for long_name and route_color (def route_ln_color (map #(select-keys % [:route_long_name :route_color]) routes-maps)) (take 2 route_ln_color) ;; ({:route_long_name "Ligne 1", :route_color "#B22E75"} ;; {:route_long_name "Ligne 2", :route_color "#B22E75"})