Executing external programs or running a system or shell command from our Python script can be useful in a lot of situations. Instead of navigating back and forth between our terminal and IDE, we can directly run commands from our Python code.
In this tutorial, we will learn about the os module and the subprocess module and see how to run system commands using them.
The os module provides us with the os.system() function, which can be used to directly run system commands. We just need to pass the command as a string to os.system().
However, this is not the recommended way of running system commands. We have to handle a lot of things manually, like escaping the shell characters, and os.system() will not even return the output to an object variable. It is used mostly to run simple commands.
The os.system() just returns the exit code after executing the system command. For example, if the command is executed successfully, then the exit code will be 0.
If you are using an IDE, the output of the system command will get printed to the console because it is used as the output stream.
Consider the following example where we first import the os module as it contains the system() function. We then run a simple system command. If we print its output, we will see that it returns 0, indicating that the command was executed successfully.
import os
# Executing dir command​ using system() function
ext_code = os.system('dir')
print("The Exit Code is: ", ext_code)
The output of the above code is shown below.
Output:
Volume in drive C is Windows
Volume Serial Number is 9096-B75F
​
Directory of C:\Users\Lenovo\PycharmProjects\untitled11
​
07/08/2021 09:09 AM <DIR> .
07/08/2021 09:09 AM <DIR> ..
07/08/2021 09:05 AM <DIR> .idea
03/04/2021 06:24 PM 2,320 asdadsa.py
03/04/2021 09:41 AM 2,639 Netcom.py
03/04/2021 10:34 AM 2,027 netcom2.py
07/08/2021 09:09 AM 79 netcom3.py
03/04/2021 05:26 PM 1,269 rfcs.py
03/02/2021 09:50 AM <DIR> venv
5 File(s) 8,334 bytes
4 Dir(s) 790,547,144,704 bytes free
The Exit Code is: 0
The os module also provides an os.popen() function. This will open a pipe to the command line, and we can access the output from Python. It returns an object which we can use to view the standard input and output of the system command.
Once the stream is created using this method, we can run the read() function to get the entire output as a string.
import os
# Executing dir command using popen() function
pipe = os.popen('dir')
print(pipe.read())
Output:
Volume in drive C is Windows
Volume Serial Number is 9096-B75F
​
Directory of C:\Users\Lenovo\PycharmProjects\untitled11
​
07/08/2021 09:09 AM <DIR> .
07/08/2021 09:09 AM <DIR> ..
07/08/2021 09:05 AM <DIR> .idea
03/04/2021 06:24 PM 2,320 asdadsa.py
03/04/2021 09:41 AM 2,639 Netcom.py
03/04/2021 10:34 AM 2,027 netcom2.py
07/08/2021 09:09 AM 79 netcom3.py
03/04/2021 05:26 PM 1,269 rfcs.py
03/02/2021 09:50 AM <DIR> venv
5 File(s) 8,334 bytes
4 Dir(s) 790,547,144,704 bytes free
The subprocess module provides lots of flexibility and additional options which were absent in the os module. This is the recommended module to use when you wish to run programs or commands from Python.
The subprocess module can create or spawn new processes, and we can also obtain the stdout, stdin, and stderr using this module.
The Popen class of subprocess can be seen as a replacement for os.popen() function, but it is slightly more complicated to use.
We need to pass the command and its arguments in the form of a list to its constructor. For example, if we wish to run the ls -a -l command to list the files and directories(including the ones that are hidden), we will pass ["ls", "-a", "-l"] to the Popen class.
We can also tell Python to get the output and the errors after running the command to an object, or we can write it to a file. We need to set the stdout and the stderr arguments to do so.
Pass shell=True to run a shell or terminal commands, and text=True to get the output as strings instead of bytes.
There is a communicate() method of the Popen class that can be used to pass input to the system command. It also lets us read the output and errors returned after executing the command.
Another notable thing about Popen class is that it won't install the entire Python program, and it does not wait for the system command to finish executing. We can check whether the command is still executing by using the poll() function. It returns the exit code if the command was completed; else it returns None.
import subprocess
​
system_call = subprocess.Popen(["dir"], stdout=subprocess.PIPE, text=True, shell=True)
out, err = system_call.communicate()
print(out)
It will return the same output as shown above.
To write the output to a file, run the following code.
import subprocess
with open("output.txt", 'w') as file:
  system_call = subprocess.Popen(["dir"], stdout=file, text=True, shell=True)
The run() function of the subprocess module can also be used to execute system commands from our Python programs. Just like the Popen class constructor, we need to pass the command and its arguments as a list.
The stdout parameter can be used to direct the output to an object or even to a file.
Make sure to use the shell=True, when running a shell or terminal command.
Setting the text=True will return the standard input and output as strings instead of bytes.
import subprocess
​# Execute command using run() function
system_call = subprocess.run(["dir"], stdout=subprocess.PIPE, shell=True, text=True)
print(system_call.stdout)
print("The Exit Code is: ", system_call.returncode)
Output:
Volume in drive C is Windows
Volume Serial Number is 9096-B75F
​
Directory of C:\Users\Lenovo\PycharmProjects\untitled11
​
07/08/2021 09:29 AM <DIR> .
07/08/2021 09:29 AM <DIR> ..
07/08/2021 09:05 AM <DIR> .idea
03/04/2021 06:24 PM 2,320 asdadsa.py
03/04/2021 09:41 AM 2,639 Netcom.py
03/04/2021 10:34 AM 2,027 netcom2.py
07/08/2021 09:29 AM 185 netcom3.py
03/04/2021 05:26 PM 1,269 rfcs.py
03/02/2021 09:50 AM <DIR> venv
5 File(s) 8,440 bytes
4 Dir(s) 790,454,452,224 bytes free
​
The Exit Code is: 0
To store the output of the command in a file.
import subprocess
​
with open("stdout.txt", 'w') as file:
  system_call = subprocess.run(["dir"], stdout=file, shell=True, text=True)
Running system commands and executing other programs from our Python code can help us a lot with automation tasks. We can run them by using the os module or the subprocess module of Python. The os module is deprecated and replaced by the subprocess module, and the subprocess module is the recommended way of running system commands from Python. It provides a lot of additional features and functionalities when compared to the os module. The os.system() function just returns the exit code after running the command, but the subprocess returns the object, and we can extract the output and errors from this object.