Why Are Scripting Languages (E.G. Perl, Python, and Ruby) Not Suitable as Shell Languages

Why are scripting languages (e.g. Perl, Python, and Ruby) not suitable as shell languages?

There are a couple of differences that I can think of; just thoughtstreaming here, in no particular order:

1. Python & Co. are designed to be good at scripting. Bash & Co. are designed to be only good at scripting, with absolutely no compromise. IOW: Python is designed to be good both at scripting and non-scripting, Bash cares only about scripting.

2. Bash & Co. are untyped, Python & Co. are strongly typed, which means that the number 123, the string 123 and the file 123 are quite different. They are, however, not statically typed, which means they need to have different literals for those, in order to keep them apart.
   
   Example:

                   | Ruby             | Bash    
   -----------------------------------------
   number          | 123              | 123
   string          | '123'            | 123
   regexp          | /123/            | 123
   file            | File.open('123') | 123
   file descriptor | IO.open('123')   | 123
   URI             | URI.parse('123') | 123
   command         | `123`            | 123
   

3. Python & Co. are designed to scale up to 10000, 100000, maybe even 1000000 line programs, Bash & Co. are designed to scale down to 10 character programs.

4. In Bash & Co., files, directories, file descriptors, processes are all first-class objects, in Python, only Python objects are first-class, if you want to manipulate files, directories etc., you have to wrap them in a Python object first.

5. Shell programming is basically dataflow programming. Nobody realizes that, not even the people who write shells, but it turns out that shells are quite good at that, and general-purpose languages not so much. In the general-purpose programming world, dataflow seems to be mostly viewed as a concurrency model, not so much as a programming paradigm.

I have the feeling that trying to address these points by bolting features or DSLs onto a general-purpose programming language doesn't work. At least, I have yet to see a convincing implementation of it. There is RuSH (Ruby shell), which tries to implement a shell in Ruby, there is rush, which is an internal DSL for shell programming in Ruby, there is Hotwire, which is a Python shell, but IMO none of those come even close to competing with Bash, Zsh, fish and friends.

Actually, IMHO, the best current shell is Microsoft PowerShell, which is very surprising considering that for several decades now, Microsoft has continually had the worst shells evar. I mean, COMMAND.COM? Really? (Unfortunately, they still have a crappy terminal. It's still the "command prompt" that has been around since, what? Windows 3.0?)

PowerShell was basically created by ignoring everything Microsoft has ever done (COMMAND.COM, CMD.EXE, VBScript, JScript) and instead starting from the Unix shell, then removing all backwards-compatibility cruft (like backticks for command substitution) and massaging it a bit to make it more Windows-friendly (like using the now unused backtick as an escape character instead of the backslash which is the path component separator character in Windows). After that, is when the magic happens.

They address problem 1 and 3 from above, by basically making the opposite choice compared to Python. Python cares about large programs first, scripting second. Bash cares only about scripting. PowerShell cares about scripting first, large programs second. A defining moment for me was watching a video of an interview with Jeffrey Snover (PowerShell's lead designer), when the interviewer asked him how big of a program one could write with PowerShell and Snover answered without missing a beat: "80 characters." At that moment I realized that this is finally a guy at Microsoft who "gets" shell programming (probably related to the fact that PowerShell was neither developed by Microsoft's programming language group (i.e. lambda-calculus math nerds) nor the OS group (kernel nerds) but rather the server group (i.e. sysadmins who actually use shells)), and that I should probably take a serious look at PowerShell.

Number 2 is solved by having arguments be statically typed. So, you can write just 123 and PowerShell knows whether it is a string or a number or a file, because the cmdlet (which is what shell commands are called in PowerShell) declares the types of its arguments to the shell. This has pretty deep ramifications: unlike Unix, where each command is responsible for parsing its own arguments (the shell basically passes the arguments as an array of strings), argument parsing in PowerShell is done by the shell. The cmdlets specify all their options and flags and arguments, as well as their types and names and documentation(!) to the shell, which then can perform argument parsing, tab completion, IntelliSense, inline documentation popups etc. in one centralized place. (This is not revolutionary, and the PowerShell designers acknowledge shells like the DIGITAL Command Language (DCL) and the IBM OS/400 Command Language (CL) as prior art. For anyone who has ever used an AS/400, this should sound familiar. In OS/400, you can write a shell command and if you don't know the syntax of certain arguments, you can simply leave them out and hit F4, which will bring a menu (similar to an HTML form) with labelled fields, dropdown, help texts etc. This is only possible because the OS knows about all the possible arguments and their types.) In the Unix shell, this information is often duplicated three times: in the argument parsing code in the command itself, in the bash-completion script for tab-completion and in the manpage.

Number 4 is solved by the fact that PowerShell operates on strongly typed objects, which includes stuff like files, processes, folders and so on.

Number 5 is particularly interesting, because PowerShell is the only shell I know of, where the people who wrote it were actually aware of the fact that shells are essentially dataflow engines and deliberately implemented it as a dataflow engine.

Another nice thing about PowerShell are the naming conventions: all cmdlets are named Action-Object and moreover, there are also standardized names for specific actions and specific objects. (Again, this should sound familar to OS/400 users.) For example, everything which is related to receiving some information is called Get-Foo. And everything operating on (sub-)objects is called Bar-ChildItem. So, the equivalent to ls is Get-ChildItem (although PowerShell also provides builtin aliases ls and dir – in fact, whenever it makes sense, they provide both Unix and CMD.EXE aliases as well as abbreviations (gci in this case)).

But the killer feature IMO is the strongly typed object pipelines. While PowerShell is derived from the Unix shell, there is one very important distinction: in Unix, all communication (both via pipes and redirections as well as via command arguments) is done with untyped, unstructured strings. In PowerShell, it's all strongly typed, structured objects. This is so incredibly powerful that I seriously wonder why noone else has thought of it. (Well, they have, but they never became popular.) In my shell scripts, I estimate that up to one third of the commands is only there to act as an adapter between two other commands that don't agree on a common textual format. Many of those adapters go away in PowerShell, because the cmdlets exchange structured objects instead of unstructured text. And if you look inside the commands, then they pretty much consist of three stages: parse the textual input into an internal object representation, manipulate the objects, convert them back into text. Again, the first and third stage basically go away, because the data already comes in as objects.

However, the designers have taken great care to preserve the dynamicity and flexibility of shell scripting through what they call an Adaptive Type System.

Anyway, I don't want to turn this into a PowerShell commercial. There are plenty of things that are not so great about PowerShell, although most of those have to do either with Windows or with the specific implementation, and not so much with the concepts. (E.g. the fact that it is implemented in .NET means that the very first time you start up the shell can take up to several seconds if the .NET framework is not already in the filesystem cache due to some other application that needs it. Considering that you often use the shell for well under a second, that is completely unacceptable.)

The most important point I want to make is that if you want to look at existing work in scripting languages and shells, you shouldn't stop at Unix and the Ruby/Python/Perl/PHP family. For example, Tcl was already mentioned. Rexx would be another scripting language. Emacs Lisp would be yet another. And in the shell realm there are some of the already mentioned mainframe/midrange shells such as the OS/400 command line and DCL. Also, Plan9's rc.

Why is bash language often slower than python or ruby?

bash loads a large number of commands from disk. Most other scripting languages have many more instructions that they run internally.

For example, to do a simple computation in bash, you'd use a=`expr 1 + 2` and bash will first load /usr/bin/expr, run that command which writes the result in the output, bash collects the output (the ` quotes) and saves the result in the variable 'a'. That's definitively slow.

The advantage of bash is the incredible flexibility though. Each person may have a different set of powerful "instructions". For example, I have a small tool called hex to print out numbers in octal, hexadecimal and decimal all at once. Other languages would not integrate in the way bash does...

How does Python stack up to other scripting languages?

First off, an advice - look for POSITIVE things said from opposite sides. Meaning, you should trust positive things said about Perl by Python-related sources (or positive things said about Python by Perl related sources) much more than the opposite.

The two reasons are that:

1) People who have reason to like Python would NOT be inclined to say good things about Perl unless they were truly trying to be objective (which is what you ideally want). And vice versa - it's a problem with human biases and motivations, NOT with either Perl or Python.

2) People writing such comparisons favoring one language are VERY often completely mis-informed and not very practiced about the one they are unfavorable about. As evidence see this article: http://python.about.com/od/gettingstarted/ss/whatispython_5.htm - pretty much anything it has to say about Perl is, to put it gently and mildly, a comlpete and utter bunk. I'm sure there are equally moronic Python put-downs from Perl fans, I just never read enough about Python to be able to think one up off the top of my head.

---

Second, please be aware that, at least for Perl - and I strongly suspect, for Ruby and Python - the moniker of "scripting" language is not really applicable anymore.

Yes, their (especially Perl's) origins are somewhat connected to shell scripting, and yes, a small subset of the languages capabilities can be - and are - used to write shell scripts, and make that small task very easy and extremely productive.

However, at this point in its long history, Perl is in no way even remotely restricted to those capabilities, and is used for developing anything from scripts to web frameworks to servers to large scale enterprise software to bio-informatics software.

---

Third, please look at this - it has a lot of links:

http://wiki.python.org/moin/LanguageComparisons

Real and non-embedded use of Ruby, Python and their friends

Python (combined with PyQt) is a very solid combination for GUI desktop applications (note that while QT is LGPL, PyQt (the Python bindings) is dual licensed: GPL or commercial).

It offers the same (GUI library-wise) as Qt on C++ but with Python's specific strenghts. I'll list some of the more obvious ones:

• rapid prototyping
• extremely readable (hence maintainable) code

Should I stick with C(++) for desktop apps?

In general: no, unless you want to / need to (for a specific reason).

What does Ruby have that Python doesn't, and vice versa?

You can have code in the class definition in both Ruby and Python. However, in Ruby you have a reference to the class (self). In Python you don't have a reference to the class, as the class isn't defined yet.

An example:

class Kaka
  puts self
end

self in this case is the class, and this code would print out "Kaka". There is no way to print out the class name or in other ways access the class from the class definition body in Python.

What among Bash/Python/Perl/Ruby/Sed/Awk for System administration , coding accessories

I would use Perl over a bash/sed/awk combination. Why ?

1. You only have the one executable, rather than spawning off multiple executables to do work.
2. You can make use of a wide range of Perl modules to do most anything (see CPAN for the modules available)

In fact I would recommend any scripting language over the shell/awk/sed combination, for the same reasons. I don't have a problem with sed/awk per se, but as your required solutions become more complex/lengthy, I find the more powerful scripting languages more scalable, and (to some degree) refactorable for re-use.

Which of these scripting languages is more appropriate for pen-testing?

You probably want Ruby, because it's the native language for Metasploit, which is the de facto standard open source penetration testing framework. Ruby's going to give you:

• Metasploit's framework, opcode and shellcode databases
• Metasploit's Ruby lorcon bindings for raw 802.11 work.
• Metasploit's KARMA bindings for 802.11 clientside redirection.
• Libcurl and net/http for web tool writing.
• EventMachine for web proxy and fuzzing work (or RFuzz, which extends the well-known Mongrel webserver).
• Metasm for shellcode generation.
• Distorm for x86 disassembly.
• BinData for binary file format fuzzing.

Second place here goes to Python. There are more pentesting libraries available in Python than in Ruby (but not enough to offset Metasploit). Commercial tools tend to support Python as well --- if you're an Immunity CANVAS or CORE Impact customer, you want Python. Python gives you:

• Twisted for network access.
• PaiMei for program tracing and programmable debugging.
• CANVAS and Impact support.
• Dornseif's firewire libraries for remote debugging.
• Ready integration with WinDbg for remote Windows kernel debugging (there's still no good answer in Ruby for kernel debugging, which is why I still occasionally use Python).
• Peach Fuzzer and Sully for fuzzing.
• SpikeProxy for web penetration testing (also, OWASP Pantera).

Unsurprisingly, a lot of web work uses Java tools. The de facto standard web pentest tool is Burp Suite, which is a Java swing app. Both Ruby and Python have Java variants you can use to get access to tools like that. Also, both Ruby and Python offer:

• Direct integration with libpcap for raw packet work.
• OpenSSL bindings for crypto.
• IDA Pro extensions.
• Mature (or at least reasonable) C foreign function interfaces for API access.
• WxWindows for UI work, and decent web stacks for web UIs.

You're not going to go wrong with either language, though for mainstream pentest work, Metasploit probably edges out all the Python benefits, and at present, for x86 reversing work, Python's superior debugging interfaces edge out all the Ruby benefits.

Also: it's 2008. They're not "scripting languages". They're programming languages. ;)

Shell Scripting and other languages

Strengths of shells:

• pipelines, filters and redirection to easily utilise the combined power of other programs/utilities
• easy display and change of environment variables exported to - and OS limits affecting - programs you invoke
• powerful and programmable command completion that's not only aware of built-in commands, but also directories, files and other programs and their command-line options (at least in good shells - zsh, bash...)
• job control functions to run multiple jobs in the background, wait on their completion
• filename globbing allows easy operations on sets of matching files
• very terse/concise and reasonably powerful ad-hoc adjustments to environment variables (splitting them into words, removing/substituting substrings, taking the head or tail of directory paths...) that are unfortunately often hard to remember if you spend most of your time on things other than shell scripting...

What shells don't tend to be so great at is:

• calling library routines written in other languages (though many can be compiled into utility programs)
• user-defined structured data types and containers (hash tables, linked lists, trees)
• type safety
• performance

Why learn Perl, Python, Ruby if the company is using C++, C# or Java as the application language?

A lot of times some quick task comes up that isn't part of the main software you are developing. Sometimes the task is one off ie compare this file to the database and let me know the differences. It is a lot easier to do text parsing in Perl/Ruby/Python than it is in Java or C# (partially because it is a lot easier to use regular expressions). It will probably take a lot less time to parse the text file using Perl/Ruby/Python (or maybe even vbscript cringe and then load it into the database than it would to create a Java/C# program to do it or to do it by hand.

Also, due to the ease at which most of the dynamic languages parse text, they are great for code generation. Sure your final project must be in C#/Java/Transact SQL but instead of cutting and pasting 100 times, finding errors, and cutting and pasting another 100 times it is often (but not always) easier just to use a code generator.

A recent example at work is we needed to get data from one accounting system into our accounting system. The system has an import format, but the old system had a completely different format (fixed width although some things had to be matched). The task is not to create a program to migrate the data over and over again. It is to shove the data into our system and then maintain it there going forward. So even though we are a C# and SQL Server shop, I used Python to convert the data into the format that could be imported by our application. Ultimately it doesn't matter that I used python, it matters that the data is in the system. My boss was pretty impressed.

Where I often see the dynamic languages used for is testing. It is much easier to create a Python/Perl/Ruby program to link to a web service and throw some data against it than it is to create the equivalent Java program. You can also use python to hit against command line programs, generate a ton of garbage (but still valid) test data, etc.. quite easily.

The other thing that dynamic languages are big on is code generation. Creating the C#/C++/Java code. Some examples follow:

The first code generation task I often see is people using dynamic languages to maintain constants in the system. Instead of hand coding a bunch of enums, a dynamic language can be used to fairly easily parse a text file and create the Java/C# code with the enums.

SQL is a whole other ball game but often you get better performance by cut and pasting 100 times instead of trying to do a function (due to caching of execution plans or putting complicated logic in a function causing you to go row by row instead of in a set). In fact it is quite useful to use the table definition to create certain stored procedures automatically.

It is always better to get buy in for a code generator. But even if you don't, is it more fun to spend time cutting/pasting or is it more fun to create a Perl/Python/Ruby script once and then have that generate the code? If it takes you hours to hand code something but less time to create a code generator, then even if you use it once you have saved time and hence money. If it takes you longer to create a code generator than it takes to hand code once but you know you will have to update the code more than once, it may still make sense. If it takes you 2 hours to hand code, 4 hours to do the generator but you know you'll have to hand code equivalent work another 5 or 6 times than it is obviously better to create the generator.

Also some things are easier with dynamic languages than Java/C#/C/C++. In particular regular expressions come to mind. If you start using regular expressions in Perl and realize their value, you may suddenly start making use of the Java regular expression library if you haven't before. If you have then there may be something else.

I will leave you with one last example of a task that would have been great for a dynamic language. My work mate had to take a directory full of files and burn them to various cd's for various customers. There were a few customers but a lot of files and you had to look in them to see what they were. He did this task by hand....A Java/C# program would have saved time, but for one time and with all the development overhead it isn't worth it. However slapping something together in Perl/Python/Ruby probably would have been worth it. He spent several hours doing it. It would have taken less than one to create the Python script to inspect each file, match which customer it goes to, and then move the file to the appropriate place.....Again, not part of the standard job. But the task came up as a one off. Is it better to do it yourself, spend the larger amount of time to make Java/C# do the task, or spend a much smaller amount of time doing it in Python/Perl/Ruby. If you are using C or C++ the point is even more dramatic due to the extra concerns of programming in C or C++ (pointers, no array bounds checking, etc.).

When is a language considered a scripting language?

A scripting language is a language that "scripts" other things to do stuff. The primary focus isn't primarily building your own apps so much as getting an existing app to act the way you want, e.g. JavaScript for browsers, VBA for MS Office.

---

Related Topics