3 ms·
You caught me trying to escape some embarrassment. Here's an example: To set some context: The code handles isolated execution of machine learning code. Origi
by imaginaryspaces 2y ago
You caught me trying to escape some embarrassment. Here's an example:
To set some context: The code handles isolated execution of machine learning code.
Original version:
def run_code(code, timeout=3600):
try:
p = Process(target=exec, args=(code,))
p.start()
p.join(timeout)
if p.is_alive():
p.terminate()
return "Timeout"
return "Success"
except Exception as e:
return f"Failed: {str(e)}"
New version:
class ProcessExecutor(Executor):
def __init__(self, execution_id: str, code: str,
working_dir: Path, dataset: pd.DataFrame,
timeout: int = 3600):
super().__init__(code, timeout)
self.working_dir = Path(working_dir).resolve() / execution_id
self.working_dir.mkdir(parents=True, exist_ok=True)
self.dataset = dataset
def run(self) -> ExecutionResult:
start_time = time.time()
code_file: Path = self.working_dir / self.code_file_name
with open(code_file, "w") as f:
f.write(self.code)
try:
process = subprocess.Popen(
[sys.executable, str(code_file)],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
cwd=str(self.working_dir),
text=True,
)
stdout, stderr = process.communicate(timeout=self.timeout)
return ExecutionResult(
term_out=[stdout],
exec_time=time.time() - start_time,
model_artifacts=self._collect_artifacts(),
performance=extract_performance(stdout),
)
except subprocess.TimeoutExpired:
process.kill()
return ExecutionResult(
term_out=[],
exec_time=self.timeout,
exception=TimeoutError(
f"Execution exceeded {self.timeout}s timeout"
),
)
- dakiol 2y agoThe second snippet of code seems to come from a template. Perhaps your colleague had already worked on on a similar problem before (e.g., python code to runs processes)?
- imaginaryspaces 2y agoPerhaps, shall check with him! But this is only one example, there’s a lot of other places as well. In case you’re interested to see more the project we’re working on is https://github.com/plexe-ai/smolmodels https://github.com/plexe-ai/smolmodels
- sophiebits 2y agoIt may be that his does something important yours doesn’t, but yours is much easier to understand! Short and clear is good.
- waonderer 2y ago100%. I code like OP and find it much easy to understand. The latter could be well structured but a novice coder like me needs comments to understand it.
- imaginaryspaces 2y agoFair enough! Thats a good point :)
- android521 2y agowithout further context, i think the original version is the better code.
- imaginaryspaces 2y agoThanks!
- zerr 2y agoDoes your co-founder come from 90s/early 2000s Java background?
- imaginaryspaces 2y agoNo, he’s from a Python background :)