CodeScientist: Automated scientific discovery system for code-based experiments

Here are links to the main prompts used in CodeScientist: 11. Frequently Asked Questions Q: What is and is not included in the cost estimates? A: Generally, the cost of running/debugging the experiment is what's calculated -- which is centrally the LLM cost for the Experiment Builder reflection loop, and the cost of any LLM calls made by the experiment code itself. The ideation costs, initial experiment code generation costs, and modal container costs are not included (as these are generally small compared to the debugging costs). Q: Why not use local/free Docker containers instead of Modal containers? A: You could do this, and create a version of src/modules/ModuleRunPythonInModal.py intended for local Docker use intead of Modal use. The initial version of CodeScientist did this, but the containers were very slow to spin up, and it limited the number of simultaneous experments that could be spooled up to what was available on the local machine. The Modal containers are generally ... Read full article.