4 ms·

In this case the author doesn't want to use Storage Access Framework APIs to access the file system which were mandated a few years ago as the way to access dat

by izacus 2y ago

In this case the author doesn't want to use Storage Access Framework APIs to access the file system which were mandated a few years ago as the way to access data outside the app sandbox. They're Java-only APIs and since Syncthings core isn't written in Java, the author would have to write JNI glue to call Android when writing/reading files (and honestly this would be quite tedious, because the way SAF works is to assign Uris to each file and you need to keep querying it to get folder structure since you can't just concat paths like with files). The author didn't want to do that and tried to persuade Google to continue letting them access all files, all photos and all documents directly and Google said no. That was the "difficulty" - turns out it's really hard to publish on Play if you refuse to follow the guidelines. Just like on AppStore.

pjmlp 2y ago
Many folks keep thinking that just because Android uses the Linux kernel, it is supposed to behave like GNU/Linux. Likewise they will be rather surprised to insist in targeting iOS/iPadOS/watchOS as if they are UNIX clones.
- bmicraft 2y ago
  Android being linux has exactly nothing to do with the fact that apps can't access resources outside their sandbox if they aren't mapped in somehow. Just like with lxc or docker containers on linux, jails on *bsd, or any other sandboxes.
  - pjmlp 2y ago
    The only Linux thing on Android is the kernel, use anything else at your peril regarding portability between updates and OEM custom forks. https://developer.android.com/ndk/guides/stable_apis https://developer.android.com/ndk/guides/stable_apis
    - bmicraft 2y ago
      The only linux thing is always the kernel. Userspace isn't linux by definition on any* system. That might seem like a pedantic point to make, but I would like to point out that gnu utils aren't required to make a system "Linux", and whether you're using a drop-in replacement like uutils or go a more different route doesn't make it any less "Linux". Android Apps are and mostly always have been restricted to the Java virtual machine (or its modern equivalent) exactly because that makes them portable between sometimes quite different base systems from different vendors. If you insist that makes it less of a Linux system I'd like to know what you think of flatpak apps on top of immutable distros. I hope you agree that, conceptually, they're quite close actually.
      - pjmlp 2y ago
        It certainly makes it less Linux, as most people mean Linux kernel plus GNU userspace with glibc, or possibly muslc as alternative to glibc. Nothing of that is expected to be supported on NDK, at least officially, Google and partners have the freedom to break those expectations as much as they feel like it.
      - __turbobrew__ 2y ago
        I'd just like to interject for a moment. What you're refering to as Linux, is in fact, GNU/Linux, or as I've recently taken to calling it, GNU plus Linux. Linux is not an operating system unto itself, but rather another free component of a fully functioning GNU system made useful by the GNU corelibs, shell utilities and vital system components comprising a full OS as defined by POSIX. Many computer users run a modified version of the GNU system every day, without realizing it. Through a peculiar turn of events, the version of GNU which is widely used today is often called Linux, and many of its users are not aware that it is basically the GNU system, developed by the GNU Project. There really is a Linux, and these people are using it, but it is just a part of the system they use. Linux is the kernel: the program in the system that allocates the machine's resources to the other programs that you run. The kernel is an essential part of an operating system, but useless by itself; it can only function in the context of a complete operating system. Linux is normally used in combination with the GNU operating system: the whole system is basically GNU with Linux added, or GNU/Linux. All the so-called Linux distributions are really distributions of GNU/Linux!
    - deleted 2y ago
      [deleted]
  - beeboobaa3 2y ago
    The point is you can't use the regular filesystem syscalls on android, it has to go through a weird java layer
    - scottbez1 2y ago
      And perhaps more to the point - you USED to be able to use normal Java file apis and syscalls outside of Java, but that functionality has been gradually whittled away (in the name of legitimate security improvement) over the years, meaning "basic" IO functionality your apps relied upon could be taken away at any point and replaced by less ergonomic Java-only APIs with less functionality. Fun fact: the official Dropbox Android app used to use inotify to watch for changes to the publicly writeable synced files in order to sync back to the cloud! Had to be replaced by java Storage Access Framework APIs later. Another fun fact is that the Android sdk came with a JNI wrapper around inotify but it buggily stored inotify handles in a static (global, within an app vm) dictionary, meaning you'd silently lose tracking if you created more than one Java file watcher object that happened to be for the same path, so I had to rewrite that JNI wrapper back in the day to avoid that global state bug.
- josephcsible 2y ago
  But it does behave like GNU/Linux where it matters, other than Google's arbitrary restrictions.
treyd 2y ago
To be fair, it's really messy to do Go on Android calling back into Java because of how its runtime works. I'm not surprised they don't want to do it if it's a hobby project and it'd require making substantial changes to Syncthing's core logic.
- izacus 2y ago
  It is - the way I always structured our architecture in this case was to write as much as possible of file handling in Java side and keep JNI surface minimal (it's also better for performance). But that's really hard to do if you didn't begin with cross platform architecture that doesn't take into account having to modularize the filesystem layer for Android/iOS :/
  - beeboobaa3 2y ago
    Since you have such strong opinions on the matter, and experience, why don't you contribute to the SyncThing android app and implement this? Alternatively you could grab your time machine, travel back several years and let them know to anticipate this arbitrary change google would pull in the future.
    - refulgentis 2y ago
      > (aggro question) I think it's because he doesn't have a time machine and doesn't have time to donate to rewrite someone else's project that the owner expressly doesn't want rewritten. n.b. it wasn't arbitrary
      - beeboobaa3 2y ago
        > (snarky response) This is what we call "Put up or shut up". It's easy to bash someone for not wanting to spend many hours of their time to work they have no interest in, just because some third party is now demanding it. The change is absolutely arbitrary, also. There used to be no way to grant apps access to specific folders. This is when the app was written. This still works. Google's own apps work that way. But now Google has also implemented additional ways to access the filesystem, and they are demanding people who don't even work for them to rewrite their projects. It would be understandable if they demanded new apps to adhere to these new policies. But blocking older apps, that were written when there literally wasn't an alternative available, to do a full rewrite or be banned from updating? Absurd.
codethief 2y ago
Wait, I seem to remember this discussion from years ago and thought it was resolved. Back then, Google wanted to drop the "access all files" permission from Android's permission system entirely but IIRC Syncthing & file manager devs then convinced them to keep it. But now Google comes back at them with a Google Play policy that prevents them from using that permission in practice?
- IshKebab 2y ago
  You have to ask permission via a form from Google and your app must closely fit a white list of use cases (file manager, etc.) Obviously there's a high chance they just say "nah". Definitely sucks.
  - juliangoldsmith 2y ago
    It probably doesn't help that Syncthing is a direct competitor to Google Drive.
- izacus 2y ago
  Yeah, but the legitimate use-case for that permission is VERY narrow - as in, if there's ANY way to implement your use-case using SAF, you MUST use SAF. And SyncThing can be implemented using SAF - it's "just" a lot of work and that's usually not enough for Play Store to grant exception.
  - codethief 2y ago
    I see, thanks so much for elaborating! > it's "just" a lot of work What about I/O performance? I would expect SAF to come with some overhead that might not make it a good solution for a file syncing application.
fph 2y ago
To be fair, you're making the most used mobile operating system in the world and can't be bothered to make API bindings for more than one language? Or at least make the process easy so that someone else creates them? I am not an Android developer, but that seems also part of the problem.
- creshal 2y ago
  Yeah, Android has an immense churn of very low quality APIs with spotty support outside JVM languages and worse documentation. Combine that with the fact that Android users are magnitudes less willing to fund apps (either by buying them or donating), and the result is an abysmal ecosystem that does not reward continued participation in it.
- pjmlp 2y ago
  Android has two official languages for userspace Java and Kotlin. NDK is only for writing native methods, reuse C and C++ libraries, and better performance for 3D and real time audio. Anything else, is not officially supported by the Android team.
  - deleted 2y ago
    [deleted]
  - NotPractical 2y ago
    fopen(), fwrite(), etc. are a part of libc and as such are officially supported [1], but they become somewhat useless with SAF. (It makes sense that you'd at least have to use some Java to open a file picker and request access, but why not provide native access as well when a file is picked?) [1] https://developer.android.com/ndk/guides/stable_apis https://developer.android.com/ndk/guides/stable_apis
    - pjmlp 2y ago
      And they can be used the same as java.io.File, inside the application sandbox. Other than that, it boils down to: " - Squeeze extra performance out of a device to achieve low latency or run computationally intensive applications, such as games or physics simulations. - Reuse your own or other developers' C or C++ libraries. " From https://developer.android.com/ndk/guides https://developer.android.com/ndk/guides And most likely safety, not having C and C++ dealing directly with random bytes from untrusted files.
ihuman 2y ago
> They're Java-only APIs and since Syncthings core isn't written in Java, the author would have to write JNI glue to call Android when writing/reading files Syncthing-android is already written in Java, so shouldn't there already be JNI glue code? https://github.com/syncthing/syncthing-android https://github.com/syncthing/syncthing-android
- carstenhag 2y ago
  It's an app that ships with/downloads the synching (go-based) binary. (I am pretty sure this is still the case, I looked at it some years ago)
- izacus 2y ago
  The actual file monitoring, access nad upload is handled by syncthing cross-platform Go library wrapped with the Java app.

by izacus 2y ago

pjmlp 2y ago

Many folks keep thinking that just because Android uses the Linux kernel, it is supposed to behave like GNU/Linux. Likewise they will be rather surprised to insist in targeting iOS/iPadOS/watchOS as if they are UNIX clones.

bmicraft 2y ago
Android being linux has exactly nothing to do with the fact that apps can't access resources outside their sandbox if they aren't mapped in somehow. Just like with lxc or docker containers on linux, jails on *bsd, or any other sandboxes.
- pjmlp 2y ago
  The only Linux thing on Android is the kernel, use anything else at your peril regarding portability between updates and OEM custom forks. https://developer.android.com/ndk/guides/stable_apis https://developer.android.com/ndk/guides/stable_apis
  - bmicraft 2y ago
    The only linux thing is always the kernel. Userspace isn't linux by definition on any* system. That might seem like a pedantic point to make, but I would like to point out that gnu utils aren't required to make a system "Linux", and whether you're using a drop-in replacement like uutils or go a more different route doesn't make it any less "Linux". Android Apps are and mostly always have been restricted to the Java virtual machine (or its modern equivalent) exactly because that makes them portable between sometimes quite different base systems from different vendors. If you insist that makes it less of a Linux system I'd like to know what you think of flatpak apps on top of immutable distros. I hope you agree that, conceptually, they're quite close actually.
    - pjmlp 2y ago
      It certainly makes it less Linux, as most people mean Linux kernel plus GNU userspace with glibc, or possibly muslc as alternative to glibc. Nothing of that is expected to be supported on NDK, at least officially, Google and partners have the freedom to break those expectations as much as they feel like it.
    - __turbobrew__ 2y ago
      I'd just like to interject for a moment. What you're refering to as Linux, is in fact, GNU/Linux, or as I've recently taken to calling it, GNU plus Linux. Linux is not an operating system unto itself, but rather another free component of a fully functioning GNU system made useful by the GNU corelibs, shell utilities and vital system components comprising a full OS as defined by POSIX. Many computer users run a modified version of the GNU system every day, without realizing it. Through a peculiar turn of events, the version of GNU which is widely used today is often called Linux, and many of its users are not aware that it is basically the GNU system, developed by the GNU Project. There really is a Linux, and these people are using it, but it is just a part of the system they use. Linux is the kernel: the program in the system that allocates the machine's resources to the other programs that you run. The kernel is an essential part of an operating system, but useless by itself; it can only function in the context of a complete operating system. Linux is normally used in combination with the GNU operating system: the whole system is basically GNU with Linux added, or GNU/Linux. All the so-called Linux distributions are really distributions of GNU/Linux!
  - deleted 2y ago
    [deleted]
- beeboobaa3 2y ago
  The point is you can't use the regular filesystem syscalls on android, it has to go through a weird java layer
  - scottbez1 2y ago
    And perhaps more to the point - you USED to be able to use normal Java file apis and syscalls outside of Java, but that functionality has been gradually whittled away (in the name of legitimate security improvement) over the years, meaning "basic" IO functionality your apps relied upon could be taken away at any point and replaced by less ergonomic Java-only APIs with less functionality. Fun fact: the official Dropbox Android app used to use inotify to watch for changes to the publicly writeable synced files in order to sync back to the cloud! Had to be replaced by java Storage Access Framework APIs later. Another fun fact is that the Android sdk came with a JNI wrapper around inotify but it buggily stored inotify handles in a static (global, within an app vm) dictionary, meaning you'd silently lose tracking if you created more than one Java file watcher object that happened to be for the same path, so I had to rewrite that JNI wrapper back in the day to avoid that global state bug.
josephcsible 2y ago
But it does behave like GNU/Linux where it matters, other than Google's arbitrary restrictions.

treyd 2y ago

To be fair, it's really messy to do Go on Android calling back into Java because of how its runtime works. I'm not surprised they don't want to do it if it's a hobby project and it'd require making substantial changes to Syncthing's core logic.

izacus 2y ago
It is - the way I always structured our architecture in this case was to write as much as possible of file handling in Java side and keep JNI surface minimal (it's also better for performance). But that's really hard to do if you didn't begin with cross platform architecture that doesn't take into account having to modularize the filesystem layer for Android/iOS :/
- beeboobaa3 2y ago
  Since you have such strong opinions on the matter, and experience, why don't you contribute to the SyncThing android app and implement this? Alternatively you could grab your time machine, travel back several years and let them know to anticipate this arbitrary change google would pull in the future.
  - refulgentis 2y ago
    > (aggro question) I think it's because he doesn't have a time machine and doesn't have time to donate to rewrite someone else's project that the owner expressly doesn't want rewritten. n.b. it wasn't arbitrary
    - beeboobaa3 2y ago
      > (snarky response) This is what we call "Put up or shut up". It's easy to bash someone for not wanting to spend many hours of their time to work they have no interest in, just because some third party is now demanding it. The change is absolutely arbitrary, also. There used to be no way to grant apps access to specific folders. This is when the app was written. This still works. Google's own apps work that way. But now Google has also implemented additional ways to access the filesystem, and they are demanding people who don't even work for them to rewrite their projects. It would be understandable if they demanded new apps to adhere to these new policies. But blocking older apps, that were written when there literally wasn't an alternative available, to do a full rewrite or be banned from updating? Absurd.

codethief 2y ago

Wait, I seem to remember this discussion from years ago and thought it was resolved. Back then, Google wanted to drop the "access all files" permission from Android's permission system entirely but IIRC Syncthing & file manager devs then convinced them to keep it. But now Google comes back at them with a Google Play policy that prevents them from using that permission in practice?

IshKebab 2y ago
You have to ask permission via a form from Google and your app must closely fit a white list of use cases (file manager, etc.) Obviously there's a high chance they just say "nah". Definitely sucks.
- juliangoldsmith 2y ago
  It probably doesn't help that Syncthing is a direct competitor to Google Drive.
izacus 2y ago
Yeah, but the legitimate use-case for that permission is VERY narrow - as in, if there's ANY way to implement your use-case using SAF, you MUST use SAF. And SyncThing can be implemented using SAF - it's "just" a lot of work and that's usually not enough for Play Store to grant exception.
- codethief 2y ago
  I see, thanks so much for elaborating! > it's "just" a lot of work What about I/O performance? I would expect SAF to come with some overhead that might not make it a good solution for a file syncing application.

fph 2y ago

To be fair, you're making the most used mobile operating system in the world and can't be bothered to make API bindings for more than one language? Or at least make the process easy so that someone else creates them? I am not an Android developer, but that seems also part of the problem.

creshal 2y ago
Yeah, Android has an immense churn of very low quality APIs with spotty support outside JVM languages and worse documentation. Combine that with the fact that Android users are magnitudes less willing to fund apps (either by buying them or donating), and the result is an abysmal ecosystem that does not reward continued participation in it.
pjmlp 2y ago
Android has two official languages for userspace Java and Kotlin. NDK is only for writing native methods, reuse C and C++ libraries, and better performance for 3D and real time audio. Anything else, is not officially supported by the Android team.
- deleted 2y ago
  [deleted]
- NotPractical 2y ago
  fopen(), fwrite(), etc. are a part of libc and as such are officially supported [1], but they become somewhat useless with SAF. (It makes sense that you'd at least have to use some Java to open a file picker and request access, but why not provide native access as well when a file is picked?) [1] https://developer.android.com/ndk/guides/stable_apis https://developer.android.com/ndk/guides/stable_apis
  - pjmlp 2y ago
    And they can be used the same as java.io.File, inside the application sandbox. Other than that, it boils down to: " - Squeeze extra performance out of a device to achieve low latency or run computationally intensive applications, such as games or physics simulations. - Reuse your own or other developers' C or C++ libraries. " From https://developer.android.com/ndk/guides https://developer.android.com/ndk/guides And most likely safety, not having C and C++ dealing directly with random bytes from untrusted files.

ihuman 2y ago

> They're Java-only APIs and since Syncthings core isn't written in Java, the author would have to write JNI glue to call Android when writing/reading files Syncthing-android is already written in Java, so shouldn't there already be JNI glue code? https://github.com/syncthing/syncthing-android https://github.com/syncthing/syncthing-android

carstenhag 2y ago
It's an app that ships with/downloads the synching (go-based) binary. (I am pretty sure this is still the case, I looked at it some years ago)
izacus 2y ago
The actual file monitoring, access nad upload is handled by syncthing cross-platform Go library wrapped with the Java app.