10 ms·

How We Found New Patterns in LA’s Homeless Arrest Data

100 points by akras14 8y ago

AdmiralAsshat 8y ago
"An XLSX file is just a zip of xml files" is definitely a useful tidbit I will keep in my toolbox. I have clients at my day-job who send us Excel files all the time, even after explicitly cautioning them that the software only works with CSV.
- tdb7893 8y ago
  I know that at least .docx is a zip of xml also. As far as I know all of the .*x microsoft office formats are
  - LeoPanthera 8y ago
    Apple Numbers/Pages/Keynote documents are also zip files. They're not XML inside, but there is always a PDF preview in there.
    - derefr 8y ago
      More technically, those documents (and other Apple-spawned formats such as TextEdit's .rtfd) are "document bundles"—a directory hierarchy with a file extension or xattr that makes Launch Services treat it as a file. (The most well-known document bundles being macOS applications.) The storage frameworks used for accessing the contents of document bundles treat zip archives as transparent, so a document bundle is still a document bundle if you zip its contents and then give it the name of the original directory. And the reverse is also true: take, for example, an .epub (which is a zipped hierarchy of HTML files) and unzip its contents into a new directory. Name that directory "foo.epub". iBooks will happily open this "file." And it stores the .epubs you've imported into your iCloud Library this way on disk!
  - subculture 8y ago
    Came to say something similar. There was a discussion a couple of weeks back on Metafilter about using this technique to modify comments within Word files. Handy knowledge to have: https://ask.metafilter.com/320606/Removing-timestamps-from-comments-in-Word-2016 https://ask.metafilter.com/320606/Removing-timestamps-from-c...
    - TeMPOraL 8y ago
      At my previous job we were working on a project that had a "propertiary" format that was just an XML file with a different file extension. At some point they asked to GZIP that XML before saving, though. So one day, I'm sitting at customer's office, working on something different, when they come and tell me they have this problem when the application is fed this particular file. I say to one of their devs, show me the file in text editor. They open it, and seeing garbage, say "it's no use". I immediately recognized the ZIP header, so I tell them, unzip it and open the output. Lo and behold, an XML content that let me debug their problem without running the application. Their technical people were impressed by my "wizardy". (Probably one of the reasons why they later asked my colleague to "encrypt" the file by running some binary transformation on the zipped XML before saving it.)
  - murftown 8y ago
    Ableton Live .als files are also gzipped .xml files.
- mieseratte 8y ago
  I believe one can also embed images and possibly other content that end up in the embedded files. However, if you ever need to perform basic manipulation then all you need to care about is ZIP + XML.
  - TeMPOraL 8y ago
    AFAIR the embedded files simply end up in the ZIP structure and are referenced by XML documents.
- mattmanser 8y ago
  It's bloody complicated, weird XML though. So don't expect it to be easy. It's been like that since they switched over to xslx in 2007 or whatever.
  - speps 8y ago
    It is an ECMA spec though: https://msdn.microsoft.com/en-us/library/office/gg607163%28v=office.14%29.aspx https://msdn.microsoft.com/en-us/library/office/gg607163%28v...
    - bkor 8y ago
      The format used by Microsoft Office differs from the ECMA spec (how much I'm not sure). By default it saves it in a different format. You have to choose a Strict filetype for that format to be used. Meaning, most files out there don't follow the spec. Further, some bits of the spec are quite strange ("follow the behaviour of Word/WordPerfect"). See also https://en.wikipedia.org/wiki/Office_Open_XML https://en.wikipedia.org/wiki/Office_Open_XML.
- sfnrm 8y ago
  This was a result of microsoft's settlement with the DOJ over the anti-trust case. Since then all microsoft docs end in .*x and are simply zipped xml files with an open format.
- TallGuyShort 8y ago
  You'd be surprised how many proprietary-seeming formats are similar. I frequently work with one that's just a gzipped tarball of JSON files, but people freak out when I'm able to see what's inside it without the proprietary tooling it's used by.
deleted 8y ago
[deleted]
robotkdick 8y ago
A link in the OPs article that supposedly supports the conclusions derived from the data says: ...from the Los Angeles Homeless Services Authority. Homeless population in the city of LA is up 48% just since 2013. The reporters data shows a 31% increase in the amount of arrests of homeless people. So, per-capita arrests have gone down. That's basic statistics. That would indicate that the headline of the article supported by the data... Huge increase in arrests of homeless in L.A. — but mostly for minor offenses ..is misleading. I lived in Santa Monica during the same time period. The homeless population has skyrocketed along with homeless crime. We moved further south because of the problem. The LA Times is a joke. The LAPD are not the best police department in the country, but I do believe they're doing their best with a complex problem the rest of our society has chosen to ignore since the 1980s. The homeless problem is not due to the police, but they are the ones we unfairly expect to deal with it.
- pokstad 8y ago
  Reminds me of the strong opposition in Irvine against emergency homeless shelters: https://www.ocregister.com/2018/03/27/o-c-supervisors-rescind-vote-to-place-homeless-shelters-in-irvine-huntington-beach-and-laguna-niguel/ https://www.ocregister.com/2018/03/27/o-c-supervisors-rescin...
- wjossey 8y ago
  I was having dinner with a buddy last week, and we were discussing the issue, as he lives in Mar Vista and I live in Beverly Grove (just recently having moved from Santa Monica). It's become a remarkable problem for the city, and it's hard to convey the impact if you don't live here. One thing I wondered was how we properly incentivize the city to take steps to manage the situation. Right now, the city brings in $6B in annual revenue (almost 25% of its budget) in property taxes. Despite the remarkable climb in homelessness, property tax revenue continues to climb, because property values are seemingly skyrocketing despite the issue. As an example, I live on a street where one half of a 100 year old duplex would run > $1M, with dozens of homeless people living within a hundred yards of my house. It's not uncommon for one to walk up and down the street, ringing doorbells asking for assistance, or harassing you as you walk down the street. Never mind the piles of trash & excrement left on the street. I've learned not to drive down the alleyway that runs perpendicular to my street, as it's not uncommon to have people sleep in the middle of the road. If property values were sinking because of the issue (which in a normal housing market would happen because of safety concerns), the city would be more well incentivized to come up with solutions. But, outside of just citizen complaints, city hall doesn't really have any skin in the game. Now, you could say, "You're part of the problem. If people like you are willing to move there, you cause the property values to rise." Well, yep, you're right. However, I WANT the city to solve this problem responsibly, and moving away doesn't help me be a part of this solution. I'm hopeful that the sales tax we self-imposed to raise funds to address the issue begin to help, but I'm mostly unimpressed with the changes thus far.
icantdrive55 8y ago
Local police departments are using that Broken Window theory to harass the Homeless so they leave, or end up in jail. In my county, we don't have much crime, but Homlessness was inching up there. Hell, I live in Marin County. The enclave to the successful Founders. They make their wad, and many end up here. They like to "act" liberal. Well, the cops basically harass the homeless. I have an aqqaintance who lives basically in a Scotch Broom thicket. He told me the how the Coos are ticketing for everything. If you claim you didn't do one of these alleged crimes, they just come down on you harder. He said, I live in fear. I used to worry about dying on the streets with pneumonia; now I worry I will end up in San Quentin. 'Hell--maybe, I'd be better off?' Marin County dosen's have a big crime problem, but Homlessness was inching up. I recently saw a middle aged women commit the crime of sitting on a sidewalk. She just sat down. I thought she must be tired. Well five cops came raining down on her. A big female cop, picked her up, threw her against a wall, and frisked her aggressively. The cop appeared to enjoy treating this women terribly? Yes--the cop seemingly enjoyed her job too much? Cops, "I don't recognize her? She must be homeless?" They then emptied her purse on the sidewalk. A tampon, and a stick of Chapstick rolled into the wet gutter. Her well worn journal/phone book was face down getting wet. It was a depressing scene. It seemed like they were all asking her questions at the same time. They warned her to, "just get on that bus!" They took digital pictures of her. The whole ugly incident was out of a Spielberg picture. I couldn't believe the way she was being treated. Most of us would be taliking to a lawyer after an incident like that? This is America's new way of dealing with Homlessness. My county has basically one homeless shelter. It's usually filled up, and has a arms list of rules. Again, my county has very little crime. We just have people who were inched out of the game. Some had mental breakdowns. Some were screwed over by family members--"loved ones". Sone just lost their job. Many were former Programmers, or at least the one's I'm familiar with. Yes--it seems like a lot of Programmers end up homless? It makes sence in a way. You turn forty/fifty, and you're expendable. No unions. Usually--far from family. No options? And no real skills. You spent your best days thinking everything will be o.k., until that day. You show up to interviews, but no call backs. Your skills get rusty. The bank account dwindles. I'm really appalled at how cites are taking care of the Homeless problem. All I can do is write about what I see. And yes--I'm very close to being harassed by some cop too. Great feeling? Let me repeat, my county has very little crime. So little, newspapers use police logs as humor. For instance,"Woman reported ugly man picking lemons off the ground. We couldn't find perpetrator after a lengthy search of the neighborhood." I guess Rudy's way of cleaning up New York is the new norm? Harass, and use the system to get them to move? The problem is my county has virtually no crime? We just can't afford another rate increase, and many if us have no social network to fall back upon.
sgt101 8y ago
Deep domain knowledge required to extract value from data.... Not such a surprise!
- mcherm 8y ago
  Not a surprise, but the story is a good small example of the problem, written out with enough detail for non-experts to follow. Answering a very simple question necessitated the use of a mix of skills like interviewing the homeless, persuading a government agency to release data and hacking the structure of Excel documents.
  - sgt101 8y ago
    Agree. I guess that I should amend to : "getting the data, ingesting the data, using deep domain knowledge". My point is/was that the super duper ML and visualisation is secondary (initially). What I've seen is that the above list is table stakes, after which (quite a long time after) someone realises that they can do something interesting with analysis. The next step is that we realise that. everything. we. thought. about. this. is. wrong.. Then much interesting thing!
nmeofthestate 8y ago
Reading this, I wondered whether this database maintained by the LA Times would be illegal anywhere that fell under the new EU GDPR regulations. It doesn't look anonymised.

100 points by akras14 8y ago

"An XLSX file is just a zip of xml files" is definitely a useful tidbit I will keep in my toolbox. I have clients at my day-job who send us Excel files all the time, even after explicitly cautioning them that the software only works with CSV.

tdb7893 8y ago
I know that at least .docx is a zip of xml also. As far as I know all of the .*x microsoft office formats are
- LeoPanthera 8y ago
  Apple Numbers/Pages/Keynote documents are also zip files. They're not XML inside, but there is always a PDF preview in there.
  - derefr 8y ago
    More technically, those documents (and other Apple-spawned formats such as TextEdit's .rtfd) are "document bundles"—a directory hierarchy with a file extension or xattr that makes Launch Services treat it as a file. (The most well-known document bundles being macOS applications.) The storage frameworks used for accessing the contents of document bundles treat zip archives as transparent, so a document bundle is still a document bundle if you zip its contents and then give it the name of the original directory. And the reverse is also true: take, for example, an .epub (which is a zipped hierarchy of HTML files) and unzip its contents into a new directory. Name that directory "foo.epub". iBooks will happily open this "file." And it stores the .epubs you've imported into your iCloud Library this way on disk!
- subculture 8y ago
  Came to say something similar. There was a discussion a couple of weeks back on Metafilter about using this technique to modify comments within Word files. Handy knowledge to have: https://ask.metafilter.com/320606/Removing-timestamps-from-comments-in-Word-2016 https://ask.metafilter.com/320606/Removing-timestamps-from-c...
  - TeMPOraL 8y ago
    At my previous job we were working on a project that had a "propertiary" format that was just an XML file with a different file extension. At some point they asked to GZIP that XML before saving, though. So one day, I'm sitting at customer's office, working on something different, when they come and tell me they have this problem when the application is fed this particular file. I say to one of their devs, show me the file in text editor. They open it, and seeing garbage, say "it's no use". I immediately recognized the ZIP header, so I tell them, unzip it and open the output. Lo and behold, an XML content that let me debug their problem without running the application. Their technical people were impressed by my "wizardy". (Probably one of the reasons why they later asked my colleague to "encrypt" the file by running some binary transformation on the zipped XML before saving it.)
- murftown 8y ago
  Ableton Live .als files are also gzipped .xml files.
mieseratte 8y ago
I believe one can also embed images and possibly other content that end up in the embedded files. However, if you ever need to perform basic manipulation then all you need to care about is ZIP + XML.
- TeMPOraL 8y ago
  AFAIR the embedded files simply end up in the ZIP structure and are referenced by XML documents.
mattmanser 8y ago
It's bloody complicated, weird XML though. So don't expect it to be easy. It's been like that since they switched over to xslx in 2007 or whatever.
- speps 8y ago
  It is an ECMA spec though: https://msdn.microsoft.com/en-us/library/office/gg607163%28v=office.14%29.aspx https://msdn.microsoft.com/en-us/library/office/gg607163%28v...
  - bkor 8y ago
    The format used by Microsoft Office differs from the ECMA spec (how much I'm not sure). By default it saves it in a different format. You have to choose a Strict filetype for that format to be used. Meaning, most files out there don't follow the spec. Further, some bits of the spec are quite strange ("follow the behaviour of Word/WordPerfect"). See also https://en.wikipedia.org/wiki/Office_Open_XML https://en.wikipedia.org/wiki/Office_Open_XML.
sfnrm 8y ago
This was a result of microsoft's settlement with the DOJ over the anti-trust case. Since then all microsoft docs end in .*x and are simply zipped xml files with an open format.
TallGuyShort 8y ago
You'd be surprised how many proprietary-seeming formats are similar. I frequently work with one that's just a gzipped tarball of JSON files, but people freak out when I'm able to see what's inside it without the proprietary tooling it's used by.

deleted 8y ago

[deleted]

robotkdick 8y ago

A link in the OPs article that supposedly supports the conclusions derived from the data says: ...from the Los Angeles Homeless Services Authority. Homeless population in the city of LA is up 48% just since 2013. The reporters data shows a 31% increase in the amount of arrests of homeless people. So, per-capita arrests have gone down. That's basic statistics. That would indicate that the headline of the article supported by the data... Huge increase in arrests of homeless in L.A. — but mostly for minor offenses ..is misleading. I lived in Santa Monica during the same time period. The homeless population has skyrocketed along with homeless crime. We moved further south because of the problem. The LA Times is a joke. The LAPD are not the best police department in the country, but I do believe they're doing their best with a complex problem the rest of our society has chosen to ignore since the 1980s. The homeless problem is not due to the police, but they are the ones we unfairly expect to deal with it.

pokstad 8y ago
Reminds me of the strong opposition in Irvine against emergency homeless shelters: https://www.ocregister.com/2018/03/27/o-c-supervisors-rescind-vote-to-place-homeless-shelters-in-irvine-huntington-beach-and-laguna-niguel/ https://www.ocregister.com/2018/03/27/o-c-supervisors-rescin...
wjossey 8y ago
I was having dinner with a buddy last week, and we were discussing the issue, as he lives in Mar Vista and I live in Beverly Grove (just recently having moved from Santa Monica). It's become a remarkable problem for the city, and it's hard to convey the impact if you don't live here. One thing I wondered was how we properly incentivize the city to take steps to manage the situation. Right now, the city brings in $6B in annual revenue (almost 25% of its budget) in property taxes. Despite the remarkable climb in homelessness, property tax revenue continues to climb, because property values are seemingly skyrocketing despite the issue. As an example, I live on a street where one half of a 100 year old duplex would run > $1M, with dozens of homeless people living within a hundred yards of my house. It's not uncommon for one to walk up and down the street, ringing doorbells asking for assistance, or harassing you as you walk down the street. Never mind the piles of trash & excrement left on the street. I've learned not to drive down the alleyway that runs perpendicular to my street, as it's not uncommon to have people sleep in the middle of the road. If property values were sinking because of the issue (which in a normal housing market would happen because of safety concerns), the city would be more well incentivized to come up with solutions. But, outside of just citizen complaints, city hall doesn't really have any skin in the game. Now, you could say, "You're part of the problem. If people like you are willing to move there, you cause the property values to rise." Well, yep, you're right. However, I WANT the city to solve this problem responsibly, and moving away doesn't help me be a part of this solution. I'm hopeful that the sales tax we self-imposed to raise funds to address the issue begin to help, but I'm mostly unimpressed with the changes thus far.

icantdrive55 8y ago

Local police departments are using that Broken Window theory to harass the Homeless so they leave, or end up in jail. In my county, we don't have much crime, but Homlessness was inching up there. Hell, I live in Marin County. The enclave to the successful Founders. They make their wad, and many end up here. They like to "act" liberal. Well, the cops basically harass the homeless. I have an aqqaintance who lives basically in a Scotch Broom thicket. He told me the how the Coos are ticketing for everything. If you claim you didn't do one of these alleged crimes, they just come down on you harder. He said, I live in fear. I used to worry about dying on the streets with pneumonia; now I worry I will end up in San Quentin. 'Hell--maybe, I'd be better off?' Marin County dosen's have a big crime problem, but Homlessness was inching up. I recently saw a middle aged women commit the crime of sitting on a sidewalk. She just sat down. I thought she must be tired. Well five cops came raining down on her. A big female cop, picked her up, threw her against a wall, and frisked her aggressively. The cop appeared to enjoy treating this women terribly? Yes--the cop seemingly enjoyed her job too much? Cops, "I don't recognize her? She must be homeless?" They then emptied her purse on the sidewalk. A tampon, and a stick of Chapstick rolled into the wet gutter. Her well worn journal/phone book was face down getting wet. It was a depressing scene. It seemed like they were all asking her questions at the same time. They warned her to, "just get on that bus!" They took digital pictures of her. The whole ugly incident was out of a Spielberg picture. I couldn't believe the way she was being treated. Most of us would be taliking to a lawyer after an incident like that? This is America's new way of dealing with Homlessness. My county has basically one homeless shelter. It's usually filled up, and has a arms list of rules. Again, my county has very little crime. We just have people who were inched out of the game. Some had mental breakdowns. Some were screwed over by family members--"loved ones". Sone just lost their job. Many were former Programmers, or at least the one's I'm familiar with. Yes--it seems like a lot of Programmers end up homless? It makes sence in a way. You turn forty/fifty, and you're expendable. No unions. Usually--far from family. No options? And no real skills. You spent your best days thinking everything will be o.k., until that day. You show up to interviews, but no call backs. Your skills get rusty. The bank account dwindles. I'm really appalled at how cites are taking care of the Homeless problem. All I can do is write about what I see. And yes--I'm very close to being harassed by some cop too. Great feeling? Let me repeat, my county has very little crime. So little, newspapers use police logs as humor. For instance,"Woman reported ugly man picking lemons off the ground. We couldn't find perpetrator after a lengthy search of the neighborhood." I guess Rudy's way of cleaning up New York is the new norm? Harass, and use the system to get them to move? The problem is my county has virtually no crime? We just can't afford another rate increase, and many if us have no social network to fall back upon.

sgt101 8y ago

Deep domain knowledge required to extract value from data.... Not such a surprise!

mcherm 8y ago
Not a surprise, but the story is a good small example of the problem, written out with enough detail for non-experts to follow. Answering a very simple question necessitated the use of a mix of skills like interviewing the homeless, persuading a government agency to release data and hacking the structure of Excel documents.
- sgt101 8y ago
  Agree. I guess that I should amend to : "getting the data, ingesting the data, using deep domain knowledge". My point is/was that the super duper ML and visualisation is secondary (initially). What I've seen is that the above list is table stakes, after which (quite a long time after) someone realises that they can do something interesting with analysis. The next step is that we realise that. everything. we. thought. about. this. is. wrong.. Then much interesting thing!

nmeofthestate 8y ago

Reading this, I wondered whether this database maintained by the LA Times would be illegal anywhere that fell under the new EU GDPR regulations. It doesn't look anonymised.