5 ms·

Veles: Open-source tool for binary data analysis

128 points by LorenDB 2y ago

brudgers 2y ago
Not sure how new the tool is because it was new in 2017...https://news.ycombinator.com/item?id=15966021 https://news.ycombinator.com/item?id=15966021
- p_l 2y ago
  There was a major push for development back in 2015-2016, back before it was open sourced, but AFAIK it went dead pretty soon as internal funding dried up.. Looking at github commit log, it seems it died by mid 2018. First time I had to setup CI on a Mac, and what a horrible experience it was (Windows was comparatively easy, though finding out that starting with Windows Core wasn't best option took some time to learn)
  - LorenDB 2y ago
    Did you work for the parent company?
    - p_l 2y ago
      There was no "parent company" per se - codisec was a team inside CodiLime, starting out as an initiative to build more cybersecurity presence, including being the name we participated in various CTF contests. (This is all public info, you can find codisec at various CTF participation trackers too). And yes, I was there (worked at CodiLime late 2014 - early 2017), including in the early days of Veles (those who know my full name can easily find me in credits in the github repo, direct link with my account was lost when the repo was migrated into current one). With regards to Veles, I mostly did infrastructure and tooling for CI/CD - so including the CMake stuff and making Jenkins setup for the builds. I think it was actually the first project on which I used Jenkinsfile, which was then very fresh thing. I was never much of a C++ fan so I didn't participate much in the editor code itself, especially with another project taking lion share of my attention later on.
  - self_awareness 2y ago
    > First time I had to setup CI on a Mac, and what a horrible experience it was Could you share some details what was horrible? I've created pipelines for Linux, macOS and Windows, and think that Windows is by far the worst of all three to setup anything related to CI.
    - p_l 2y ago
      One of the issues was making builds that worked on OS versions older than the build host, general annoyances of admining macOS, fragility of the environment (this was before image based upgrades)
MontagFTB 2y ago
The GitHub repo [1] was made a read-only archive on Sep 30, 2020. Are there forks picking up where the original author left off? [1]: https://github.com/codilime/veles https://github.com/codilime/veles
- rurban 2y ago
  See by yourself. There is only one fork wich fixed the msgpack encoding breakage: https://github.com/0xBattleSong/veles https://github.com/0xBattleSong/veles
- an-unknown 2y ago
  Since this is an old tool and it fails to deal with large files, I made a completely new version from scratch a while ago, using a completely different rendering approach that's more like how you'd render a volumetric data set like an MRI scan. It loads the file and processes it into a 256x256x256 volumetric data set, which is then rendered using shaders. As a result, the file size doesn't matter for rendering, only the loading time depends on the file size. Unlike the original Veles, it also doesn't need any subsampling for huge files, but you need a powerful enough graphics card. The source code is on github and unlike the original Veles, it doesn't have countless dependencies and build problems on modern systems: https://github.com/hackyourlife/veles https://github.com/hackyourlife/veles
  - tatref 2y ago
    You should add some screenshots!
    - an-unknown 2y ago
      I added a few screenshots of different files right now.
p_l 2y ago
I am honestly surprised this still shows up... learnt so much CMakeFile pain...
butz 2y ago
ImHex has some similar data pattern analysis tools. And as it is actively maintained, might be worthy to take a look at.
ta988 2y ago
Their linux build is in ubuntu 16.04 ...
- p_l 2y ago
  Sorry, that was what was current LTS when I setup CI builds
xxmarkuski 2y ago
The main page of codisec [0] lists that Veles has been retired. They since have a new project, CodiLime [1], which seems like is a general technology consulting/outsourcing company. [0] https://codisec.com/ https://codisec.com/ [1] https://codilime.com/ https://codilime.com/
- p_l 2y ago
  CodiLime was the actual company, CodiSec was "brand" for various security related activities, starting with CTF competitions where we used the name first, then it was used as brand for the team that worked on Veles (we continued to take part in CTFs) None of the CodiSec members were among the management/founders/etc of CodiLime - we were AFAIK plain employees, though one of us was in fact mber of DragonSector which is partly how CodiSec started.
turtledragonfly 2y ago
When these tools come up, I like to post my little list of similar tools, for those interested. Apparently it's a hobby of mine: * fq - like jq for binary data: https://github.com/wader/fq https://github.com/wader/fq * Kaitai Struct - https://kaitai.io/ https://kaitai.io/ * visualizer, for the above: https://github.com/kaitai-io/kaitai_struct_visualizer/ https://github.com/kaitai-io/kaitai_struct_visualizer/ * HexFiend - a hex editor, but with "binary templates" feature : https://github.com/HexFiend/HexFiend https://github.com/HexFiend/HexFiend * binary templates, for the above: https://github.com/HexFiend/HexFiend/blob/master/templates/T https://github.com/HexFiend/HexFiend/blob/master/templates/T... * binspector - https://github.com/binspector/binspector https://github.com/binspector/binspector * binary-parsing - a collection of links to similar such tools : https://github.com/dloss/binary-parsing https://github.com/dloss/binary-parsing * unblob - https://github.com/onekey-sec/unblob https://github.com/onekey-sec/unblob * ImHex - https://github.com/WerWolv/ImHex https://github.com/WerWolv/ImHex * GNU Poke - https://www.gnu.org/software/poke/ https://www.gnu.org/software/poke/ * dtrx - https://github.com/dtrx-py/dtrx/ https://github.com/dtrx-py/dtrx/ dtrx * feng - https://github.com/martinlindhe/feng https://github.com/martinlindhe/feng ** they also have a similar list of prior art: https://github.com/martinlindhe/feng/blob/main/RESOURCES.md https://github.com/martinlindhe/feng/blob/main/RESOURCES.md And some previous HN threads for similar such tools: * https://news.ycombinator.com/item?id=34434249 https://news.ycombinator.com/item?id=34434249 (unblob) * https://news.ycombinator.com/item?id=34986042 https://news.ycombinator.com/item?id=34986042 (GNU Poke #1) * https://news.ycombinator.com/item?id=36182873 https://news.ycombinator.com/item?id=36182873 (GNU Poke #2)
- ranger_danger 2y ago
  don't forget hachoir-wx
- deleted 2y ago
  [deleted]
golebiewsky 2y ago
Oh, I know something about this. Some people that made this are part of DragonSector [1] polish hacking group that broke DRMs in trains quite recently [2]. Sadly I joined company after there were long gone. Really surprised seeing this posted here as it was pointed out, the repo is pretty old and archived. [1]: https://dragonsector.pl/ https://dragonsector.pl/ [2]: https://youtu.be/XrlrbfGZo2k https://youtu.be/XrlrbfGZo2k
- p_l 2y ago
  One person, IIRC, don't think anyone other than q3k was involved in CodiSec, and I think he left the company before proper development on Veles started (I think he was present when first talks about were happening and gave a lot of inspiration though, but isn't in credits for first open source version)
DaoVeles 2y ago
I was summoned? ... oh sorry!
thot_experiment 2y ago
Neat, I was looking for such a tool a few days back and ended up not having the requisite google skills to find this (embarrassingly, it seems easy it find now) I ended up writing my own script for visualizing my particular problem (trying to work out what's going on in the "mett" data frames in videos from my pixel phone, I was able to make some headway but I've gotten distracted by other projects, if anyone has info I'd appreciate it)

128 points by LorenDB 2y ago

Not sure how new the tool is because it was new in 2017...https://news.ycombinator.com/item?id=15966021 https://news.ycombinator.com/item?id=15966021

p_l 2y ago
There was a major push for development back in 2015-2016, back before it was open sourced, but AFAIK it went dead pretty soon as internal funding dried up.. Looking at github commit log, it seems it died by mid 2018. First time I had to setup CI on a Mac, and what a horrible experience it was (Windows was comparatively easy, though finding out that starting with Windows Core wasn't best option took some time to learn)
- LorenDB 2y ago
  Did you work for the parent company?
  - p_l 2y ago
    There was no "parent company" per se - codisec was a team inside CodiLime, starting out as an initiative to build more cybersecurity presence, including being the name we participated in various CTF contests. (This is all public info, you can find codisec at various CTF participation trackers too). And yes, I was there (worked at CodiLime late 2014 - early 2017), including in the early days of Veles (those who know my full name can easily find me in credits in the github repo, direct link with my account was lost when the repo was migrated into current one). With regards to Veles, I mostly did infrastructure and tooling for CI/CD - so including the CMake stuff and making Jenkins setup for the builds. I think it was actually the first project on which I used Jenkinsfile, which was then very fresh thing. I was never much of a C++ fan so I didn't participate much in the editor code itself, especially with another project taking lion share of my attention later on.
- self_awareness 2y ago
  > First time I had to setup CI on a Mac, and what a horrible experience it was Could you share some details what was horrible? I've created pipelines for Linux, macOS and Windows, and think that Windows is by far the worst of all three to setup anything related to CI.
  - p_l 2y ago
    One of the issues was making builds that worked on OS versions older than the build host, general annoyances of admining macOS, fragility of the environment (this was before image based upgrades)

MontagFTB 2y ago

The GitHub repo [1] was made a read-only archive on Sep 30, 2020. Are there forks picking up where the original author left off? [1]: https://github.com/codilime/veles https://github.com/codilime/veles

rurban 2y ago
See by yourself. There is only one fork wich fixed the msgpack encoding breakage: https://github.com/0xBattleSong/veles https://github.com/0xBattleSong/veles
an-unknown 2y ago
Since this is an old tool and it fails to deal with large files, I made a completely new version from scratch a while ago, using a completely different rendering approach that's more like how you'd render a volumetric data set like an MRI scan. It loads the file and processes it into a 256x256x256 volumetric data set, which is then rendered using shaders. As a result, the file size doesn't matter for rendering, only the loading time depends on the file size. Unlike the original Veles, it also doesn't need any subsampling for huge files, but you need a powerful enough graphics card. The source code is on github and unlike the original Veles, it doesn't have countless dependencies and build problems on modern systems: https://github.com/hackyourlife/veles https://github.com/hackyourlife/veles
- tatref 2y ago
  You should add some screenshots!
  - an-unknown 2y ago
    I added a few screenshots of different files right now.

p_l 2y ago

I am honestly surprised this still shows up... learnt so much CMakeFile pain...

butz 2y ago

ImHex has some similar data pattern analysis tools. And as it is actively maintained, might be worthy to take a look at.

ta988 2y ago

Their linux build is in ubuntu 16.04 ...

p_l 2y ago
Sorry, that was what was current LTS when I setup CI builds

xxmarkuski 2y ago

The main page of codisec [0] lists that Veles has been retired. They since have a new project, CodiLime [1], which seems like is a general technology consulting/outsourcing company. [0] https://codisec.com/ https://codisec.com/ [1] https://codilime.com/ https://codilime.com/

p_l 2y ago
CodiLime was the actual company, CodiSec was "brand" for various security related activities, starting with CTF competitions where we used the name first, then it was used as brand for the team that worked on Veles (we continued to take part in CTFs) None of the CodiSec members were among the management/founders/etc of CodiLime - we were AFAIK plain employees, though one of us was in fact mber of DragonSector which is partly how CodiSec started.

turtledragonfly 2y ago

When these tools come up, I like to post my little list of similar tools, for those interested. Apparently it's a hobby of mine: * fq - like jq for binary data: https://github.com/wader/fq https://github.com/wader/fq * Kaitai Struct - https://kaitai.io/ https://kaitai.io/ * visualizer, for the above: https://github.com/kaitai-io/kaitai_struct_visualizer/ https://github.com/kaitai-io/kaitai_struct_visualizer/ * HexFiend - a hex editor, but with "binary templates" feature : https://github.com/HexFiend/HexFiend https://github.com/HexFiend/HexFiend * binary templates, for the above: https://github.com/HexFiend/HexFiend/blob/master/templates/T https://github.com/HexFiend/HexFiend/blob/master/templates/T... * binspector - https://github.com/binspector/binspector https://github.com/binspector/binspector * binary-parsing - a collection of links to similar such tools : https://github.com/dloss/binary-parsing https://github.com/dloss/binary-parsing * unblob - https://github.com/onekey-sec/unblob https://github.com/onekey-sec/unblob * ImHex - https://github.com/WerWolv/ImHex https://github.com/WerWolv/ImHex * GNU Poke - https://www.gnu.org/software/poke/ https://www.gnu.org/software/poke/ * dtrx - https://github.com/dtrx-py/dtrx/ https://github.com/dtrx-py/dtrx/ dtrx * feng - https://github.com/martinlindhe/feng https://github.com/martinlindhe/feng ** they also have a similar list of prior art: https://github.com/martinlindhe/feng/blob/main/RESOURCES.md https://github.com/martinlindhe/feng/blob/main/RESOURCES.md And some previous HN threads for similar such tools: * https://news.ycombinator.com/item?id=34434249 https://news.ycombinator.com/item?id=34434249 (unblob) * https://news.ycombinator.com/item?id=34986042 https://news.ycombinator.com/item?id=34986042 (GNU Poke #1) * https://news.ycombinator.com/item?id=36182873 https://news.ycombinator.com/item?id=36182873 (GNU Poke #2)

ranger_danger 2y ago
don't forget hachoir-wx
deleted 2y ago
[deleted]

golebiewsky 2y ago

Oh, I know something about this. Some people that made this are part of DragonSector [1] polish hacking group that broke DRMs in trains quite recently [2]. Sadly I joined company after there were long gone. Really surprised seeing this posted here as it was pointed out, the repo is pretty old and archived. [1]: https://dragonsector.pl/ https://dragonsector.pl/ [2]: https://youtu.be/XrlrbfGZo2k https://youtu.be/XrlrbfGZo2k

p_l 2y ago
One person, IIRC, don't think anyone other than q3k was involved in CodiSec, and I think he left the company before proper development on Veles started (I think he was present when first talks about were happening and gave a lot of inspiration though, but isn't in credits for first open source version)

DaoVeles 2y ago

I was summoned? ... oh sorry!

thot_experiment 2y ago

Neat, I was looking for such a tool a few days back and ended up not having the requisite google skills to find this (embarrassingly, it seems easy it find now) I ended up writing my own script for visualizing my particular problem (trying to work out what's going on in the "mett" data frames in videos from my pixel phone, I was able to make some headway but I've gotten distracted by other projects, if anyone has info I'd appreciate it)