3 ms·

People in this thread might be able to offer me advice on a project I have open. I know C but am not experienced working on large C codebases. Below I will des

People in this thread might be able to offer me advice on a project I have open. I know C but am not experienced working on large C codebases. Below I will describe its rough build structure. I hope for strong tool or re-framing advice. I have an existing python codebase. I want to make it easy to create native-code plugins. The method I have worked out: I compile shared-libraries. Python then calls into these via python-ctypes. Each plugin is a directory project/ext/plugin, * Python code: __init__.py and ext.py * C code, api.c * A rough build script, build.py. This accesses global config (e.g. path information) held outside this directory. It compiles api.c into a .so or .dll. It does this by generating a shell script or batch file, and then shelling out to it. A python script in the root directory walks through this structure and forks a process for each plugin build.py. It is multiplatform (currently linux gcc and windows mingw32). So I have if statements saying "if windows, generate this big string, otherwise generate that big string." The shared-object is compiled to a separate directory, res/ext/plugin. I did this to create separation between my source and build artifacts separate. (Is this sensible?) This naive build structure was easy to create, and allowed me to spend my focus elsewhere while I was bootstrapping. Now that is done, I am interested in how I could improve the build so that it could scale to a large number of plugins. Would I be well-served to use Make or Cmake or something else? Can they access configuration from outside of their own directory? Maybe I should communicate configuration via environment variables?

falsedan 9y ago
> how I could improve the build so that it could scale to a large number of plugins Is your existing parallel builder too slow? What's the problem with it? Make can do parallel builds as well, but it won't walk a directory so you'd have to write a million targets. I think you should post this on the SO site for programming design/advice though.
bjourne 9y ago
It sounds like your build process is not well designed. Why would a build script need to generate a shell script to call? And why would a python script fork a process for each plugin? I think waf, which I've plugged before, would be a good choice for your project. Without knowing the details, your build script would look something like: def configure(ctx): # Load C compiler support ctx.load('compiler_c compiler_cxx') if ctx.env.DEST_OS == 'win32': # Special thing for Windows def build(ctx): for plugin in ['a', 'b', ...]: ctx.shlib(source=ctx.path.ant_glob('ext/'+plugin+'/*.c'), target=plugin) You get cross-platform support for free. The code will create an 'a.dll' on Windows, 'a.so' on Linux and something else on OS X. Note that while the build function above looks like imperative code, it actually is not. Instead it is Python code to create a dependency graph which waf rebuilds for you when it is needed.

falsedan 9y ago

> how I could improve the build so that it could scale to a large number of plugins Is your existing parallel builder too slow? What's the problem with it? Make can do parallel builds as well, but it won't walk a directory so you'd have to write a million targets. I think you should post this on the SO site for programming design/advice though.

bjourne 9y ago

It sounds like your build process is not well designed. Why would a build script need to generate a shell script to call? And why would a python script fork a process for each plugin? I think waf, which I've plugged before, would be a good choice for your project. Without knowing the details, your build script would look something like: def configure(ctx): # Load C compiler support ctx.load('compiler_c compiler_cxx') if ctx.env.DEST_OS == 'win32': # Special thing for Windows def build(ctx): for plugin in ['a', 'b', ...]: ctx.shlib(source=ctx.path.ant_glob('ext/'+plugin+'/*.c'), target=plugin) You get cross-platform support for free. The code will create an 'a.dll' on Windows, 'a.so' on Linux and something else on OS X. Note that while the build function above looks like imperative code, it actually is not. Instead it is Python code to create a dependency graph which waf rebuilds for you when it is needed.