Here is a short post about conda environments and a conda script that adds a bit of extra functionality to the “conda env export” tool.
One minor annoyance I have with Conda is the limitation of its environment exporting tool (conda env export) for my particular use. I decided to look for a solution that would solve the following pain-points:
- Not including pip installations when using flag “
--from-history“ - When using flag “
--from-history“, package versions are often not included (depending on whether the user manually typed them) - When using “–from-history” flag, third-party channels are not included in the file
Since I did not find something that addressed all the points above, I had to write a script for it, conda_export.py.
The quick version of this post is that if you want an output like the one described above, just run the script like this (make sure PyYAML is installed in the environment, with “pip install pyyaml“):
#run this script inside the conda environment you want to export
python conda_export.py --from-history --use-versions -o environment.py
Using The Conda Environment Export Script
As I mentioned above, ideally, I would like to run the following conda command:
conda env export --from-history > environment.yml
and generate a file that contains strictly the packages I installed (including those installed with pip). Additionally, those packages should indicate the specific version number that I installed.
The script should be run inside the conda environment you wish to export (you will need to install the PyYAML Python library: “pip install pyyaml“). Here are the flags to use if you want an output that includes only manually installed packages, including pip packages, as well as their versions:
python conda_export.py --from-history --use-versions -o environment.yml
The “--from-history” flag will simply get the same output as using conda directly. However, in this case, I’m also using "--use-versions“, which extracts version numbers by doing another call to “conda env export” and then joining the two results together. The “-o” flag is used to indicate the output file of the export. If “-o” flag is not provided, the output will be dumped to the terminal.
Example Exporting a Conda Environment
To showcase the expected output from this script, we will create a simple conda environment with some packages:
conda create -n test_env python=3.9 numpy matplotlib
Once the install is complete and the environment is activated, I want to also install a package from the “conda-forge” channel:
#this is inside test_env environment
conda install -c conda-forge manim
In case you are curious, manim is a mathematical animation engine originally developed by the creator of the YouTube channel 3Blue1Brown. I wrote a blog post about how to use it last week. Finally, we will also install a package with pip, such as the one I created a while ago: netbuilder.
pip install netbuilder
With all that completed, here is the output of conda:
conda env export --from-history
name: test_env
channels:
- defaults
dependencies:
- matplotlib
- python=3.9
- numpy
- manim
prefix: /home/andres/miniconda3/envs/test_env
As you can see, we lost information about the specific version numbers for all the packages unless we manually typed them. Also, we lost information about the channel “conda-forge” used to install the manim library. And, there is no information about the packages installed with pip.
Now, if we run the conda_export.py script, we get the following:
python conda_export.py --from-history --use-versions
channels:
- conda-forge
- defaults
dependencies:
- manim=0.17.3
- matplotlib=3.7.1
- numpy=1.25.0
- python=3.9.17
- pip:
- netbuilder==0.2.3
name: test_env
This output contains all the information needed to recreate the environment.
How The Script Works
You can check the script in the repository. Since I wrote it mostly for myself and very quickly, I think maybe some functions are a bit convoluted and messy, but they work.
Basically, if the “--from-history” flag is used, the script will execute two commands:
conda env export
conda env export --from-history
The output of each command is stored in a separate dictionary and a separate function merges them into a single dictionary only containing the common items.
Here is the part of the script that handles this condition:
...
elif from_history and use_versions:
#-- Create merged list of dependencies
full_env_output:dict = export_env(from_history=False, no_builds=False)
hist_env_output:dict = export_env(from_history=True)
_merged_dependencies:list = merge_dependencies(full_env_output, hist_env_output, use_versions)
#-- setup final env dictionary
final_env_dict = dict()
final_env_dict['name'] = full_env_output['name']
final_env_dict['channels'] = full_env_output['channels']
final_env_dict['dependencies'] = _merged_dependencies
_pip_section = get_pip_section(full_env_output['dependencies'])
if len(_pip_section.values()) > 0: #this adds the pip part
final_env_dict['dependencies'].append(_pip_section)
final_env_dict['prefix'] = full_env_output['prefix']
In the snippet above, “export_env” is the function that runs a sub-process with the two conda commands. I borrowed that function from this GitHub gist.
Future Work
Hopefully, conda implements some of these features as options natively in the future. But, in the mean time the script is useful to me. One thing I would like to do is rewrite it to be less verbose and to use only standard Python libraries. Currently, PyYAML needs to be installed in the environment the script is used in. If I want to export my environment, I need to install that package, polluting the environment with unwanted libraries.
Let me know if you have any questions or suggestions about this or anything Python related. You can also check my other Python posts here, and let me know what you think.