- The Powershell idea to push structured data through the pipe, which this project fundamentally replicates with a relational structure, has its merits; but it will always displease someone, and carries the burden of having to rewrite millions of utilities in order to unlock the real potential.
What if, instead, we pushed the approach “upstream”, asking systems to have an additional stream that all shells can access? We have stdout and stderr, we could add “stddata” - a pipe that expects and produces structured data. Then it would become trivial to add support around the userland, without displeasing anyone - because it’s just an additional interface, not a replacement. The pipe should support two or three formats to start with (csv, json, and maybe xml, so you can handle both tabular and child-parent relations with a varying degree of precision) and shells could have a special character for piping stddata (I like ¥) so everything else about them would stay the same.
- I am at the stage that I am more than happy to rewrite whatever it takes to get better experience. This includes the OS. I think we have sacrificed too much on the altar of backward compatibility. I guess I am just tired of breakage because a file has a space in its name.
+ Lorem ipsum foo bar baz.
- Yeah, some places I have worked we had a research kernel because we could not measure HW performance accurately with Linux.
+ Lorem ipsum foo bar baz.
- Hey, no cheating -- one that has a substantial baggage of applications on top of it. A research kernel is something you can do in one year working with a few dedicated undergrads. But then so is pretty much every single thing running on top of it. It looks easy until you figure out how much work gets into properly writing a PNG decoder or a PDF viewer. It's not fun, even in a memory-safe language.
I admire your optimism and I, too, wish we'd leave this POSIX hellhole behind us (and I'm fond of this POSIX hellhole!) but I think at this stage in the story of computers, it's no longer realistic.
- I think the "POSIX hellhole" is really just the Linux hellhole; there's no reason the underlying implementation or the desktop on top of it needs to be that way (and Haiku is more or less succeeding at being POSIX and not a hellhole :)
+ Lorem ipsum foo bar baz.
- Oh yeah, don't get me started on the Linux hellhole :). I like Haiku (in fact I keep putting off that weekend when I'll try my hand at fixing a bug or writing a new app for it, because I really liked BeOS back in the day, too). A lot of it isn't even POSIX' fault, it was just abused by things built on top of it, and now you can't use those things without making a mess.
I know a lot of it like the back of my hand, I've been writing POSIX code (or, at one point, POSIX implementations) for much, if not most of my career. That doesn't mean I don't want to see something better, or that I'm blind at the fact that Plan 9 is so fun to code for precisely because they skipped POSIX (although, that being said, I'm also not blind to the fact that it's on of the big reasons why it never really took off, either).
- Totally agree with you. Now the real question, do you need to implement PNG decoder and PDF viewer? Or you need to design alternatives that are far more safe to implement?
+ Lorem ipsum foo bar baz.
- Everyone has different criteria for what a new and improved OS needs. Some say make it safer. Some say it must be much simpler. Others say why is it so inefficient, make it faster and focus on mixed level languages. Nobody can agree and nobody needs any of these badly enough to effect a real change. So we have these brainstorms every year and the work that comes from them never gets far.
+ Lorem ipsum foo bar baz.
- > Now the real question, do you need to implement PNG decoder and PDF viewer?
If you ask me, no, but that's because I remember (and not very fondly) an age when it seemed like we'd have a new arbitrary code execution bug in a PNG or JPEG or PDF decoder every week. I don't really wanna go through it either.
Edit: Lots of people don't remember (or don't know about) that period and think writing an OS and all the boring stuff that goes with it is easy. Four or five years and it should be all done, eh?
That's also why we got things like the security community, which is full of enthusiastic people and big companies that know exactly what OpenSSL needs and how it should be written. But when it comes to sitting down and doing it, or committing the money, the crowd gets awfully small. Foundational software isn't too glamorous, and writing things like a PNG decoder is pretty boring, too.
- Well, not exactly. He merely suggested using another one. But I thought it would be good to talk something a little bit more specific. And I am not sure he knows there are already others available (for shell scripts at least fd[0-9]).
After all, /dev/std{in,out,err} are more or less just convenience links (allowed extensions of POSIX). But using /dev/stderr might fail on some POSIX compliant systems.
- Not breaking stdin and stdout is key. Looking at the output from that shell, text parsing tools like awk are not going to integrate if you dont have a compatible tool yet. Everything in a base Linux install has to be rewritten to work.
Personally i prefer the keep it simple approach. Id much rather /etc/hosts stayed the way it is rather than becoming xml, json or binary.
The whole idea of "Power" shell is broken if you ask me. Shell should be simple. If it isnt, write some proper code. And do some code review, and testing first!
- Shell stopped being simple long long ago, at the latest when bash and ksh were introduced. (Re)writing the base utilities has happened many times, and is quite doable.
+ Lorem ipsum foo bar baz.
- Yes, backward compatibility is important; that’s why I think adding a new named stream could be the way to go, since it wouldn’t break almost anything (I guess someone might have to reroute something, if it happens to be on the fd that is chosen for the link).
I honestly don’t know how much work it would entail from OS devs, but once you have it in Linux and BSDs/Mac that’s basically it. Microsoft could take the chance to be an early adopter, considering they already have all the machinery to output structured data in PS; at that point, whether it’s pure posix or not would matter little in practice.
I don’t particularly like PS for a number of reasons, but I do admit that its output facilities can be quite nice. Having something like that in Bash for the few times one really needs it, with standard interfaces and without having to learn a completely new paradigm, would be a nice win imho.
This doesn’t mean we should touch /etc/hosts or anything, btw.
- I know quite a few places where shell scripts would probably blow up if some arbitrary FD number >2 would be used... Opening a file in bash (and I suppose other shells too) is done by specifying the FD number explicitly:
exec 3<> somefile.txt
-
-That might trigger some very weird behaviour when piping data from a 'stddata' aware tool into shell scripts - something which is guaranteed to happen somewhere.
+ Lorem ipsum foo bar baz.
- Why not add a new parameter, --nushell, to any too that supports the 2-dimensional format? These tools would just be stored in nushell's config file and the shell would silently add --nutshell to any command it ran that was registered to support it.
+ Lorem ipsum foo bar baz.
- The problem with adding a new standard fd is that lots and lots of code relies on first three fds being stdin out err. It could of course have a high fd number. Would be a great thing!
+ I am not sure how long "ls" will take with 1 million entries in directory, given this paradigm. This has been a problem so I use "ls -1" to list files in bash shell, otherwise it takes forever.
Probably it might benefit some. Good to see competition in this area with sh, bash, zsh, fish, conch and many others.
- I am not sure how long "ls" will take with 1 million entries in directory, given this paradigm. This has been a problem so I use "ls -1" to list files in bash shell, otherwise it takes forever.
Probably it might benefit some. Good to see competition in this area with sh, bash, zsh, fish, conch and many others.
- A csv output with one field is basically the same amount of data, but you would be guaranteed that the result can be ingested by a csv parser following the “stddata” dialect. It wouldn’t hurt your usecase, and it would make life easier for other ones.
+ Lorem ipsum foo bar baz.
- The compelling idea here is that they convert the output of common shell commands into tabular data that can be manipulated using common operators, so that you don't have the remember sorting/filtering/grouping flags that may be different for every different shell command. So, imagine being able to use the same sorting/filtering logic on the output of `ls` as you might on the output of `ps`, and without relying on hacky solutions like pipelines to `grep`, `cut`, and `sort`.
It also means shell command output can be easily transposed into JSON and CSV. Actually pretty clever!
- I've been delving into bash scripting a bit more than I'd like as of late and the lack of universally available consistent structured output for the CLI really got to me. Most of the script contents end up being these obfuscating and brittle "hacky solutions" that never should have been necessary. When I thought about pursuing fixing this I felt like the task was a bit overwhelming. I'm delighted that these developers are working on this!
+ Lorem ipsum foo bar baz.
- The key flaw of UNIX philosophy is destructuring deserialization and reserialization based on lines, necessitating all manner of argument escaping and field delimiters, when pipelines should be streams of typed messages that encapsulate data in typed fields. Logs especially (logging to files is a terrible idea, because it creates log rotation headaches and each program requires a log parser because of the loss of structured information) also. Line-oriented pipeline processing is fundamentally too simple. Settling on a common data simple/universal format robust enough for all purposes, including very large data sets, to exchange between programs and is just complicated enough to eliminate escapement and delimiter headaches without throwing away flexibility (by a new/refined set of processing tools/commands) is key.
+ Lorem ipsum foo bar baz.
- It's not so much a flaw as a tradeoff. Because it's so easy to produce lines of text, virtually every program supports it, even programs written by hackers who aren't interested in following OS communication standards.
+ Lorem ipsum foo bar baz.
- No, that's your opinion, not mine. It's a flaw because it creates escaping magic, delimiters that are in-band (spaces) and every program has to know how to parse the output of every other program, wasting human time and processing time. Structured communication, say like an open standard for a schema-less, self-describing, flexible de/serialization like protobufs/msgpack, would be far superior, more usable, more efficient and simple but no too simple to process streams of data with structure and programmability already there.
Being able to dump structured information out of an exception directly into a log, and then from a log into a database, without any loss of information or extraneous log parsing, is a clear win. Or from a command as simple as listing files ("ls") into a database or into any other tool or program. Outputing only line-oriented strings is just throwing away type information, and creates more work for everyone else, even more so than continuing to stay with lines processing tools.
- So, if you're not a fan of the UNIX philosophy, maybe check out Powershell. Or take a look at WMI and DCOM in Windows. Eschew shell scripts in favor of strongly-typed programs that process strongly-typed binary formats, or XML, or whatever. The alternatives are out there.
- Nushell does not seem like a violation of the unix philosophy, or at least the version of it that I like best.
"Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface."
Perhaps I'm wrong, isn't nushell simply adding one more dimension?
Instead of a one-dimensional array of lines, the standard is now... a two-dimensional array of tabular data. Perhaps it is not strictly a "text stream" but this does not seem to me to be a violation of the spirit of the message.
Simple line-based text streams are clearly inadequate IMO. 99.99% of all Unix programs output multiple data fields on one line, and to do anything useful at all with them in a programmatic fashion you wind up needing to hack together some way of parsing out those multiple bits of data on every line.
maybe check out Powershell
-
-I left Windows right around the time PS became popular, so I never really worked with it.
It seems like overkill for most/all things I'd ever want to do. Powershell objects seem very very powerful, but they seemed like too much.
Nushell seems like a nice compromise. Avoids the overkill functionality of Powershell.
- Semantically, a one-dimensional list/array of “things” can be trivially transposed into a series of lines with a linebreak separator, but I don’t think the same holds true for a “list of lists” (2-dimensional data) or a “list of lists of lists” (3D data) etc. At least without a standard begin-end list delimiter that allows nesting.
Just thinking about a way that perhaps an old tool can be “wrapped” to be tricked into working with 2+-dimensional data by somehow divvying up the 2+ dimension input data into concurrent 1-dimensional streams, but this seems to require a way to represent more than 1 dimension of data without breaking existing utilities (unless there was, like, a wrapper/unwrapper layer that handled this...)
- When perl came along it killed the 'Unix one tool philosophy' dead as a doornail. And since then and people have just kinda ignored the smell coming off the rotting corpse.
I don't write complex scripts in shell anymore because it's insanity. But ad-hoc loops and crap like that... hell yeah. At least a few a day. Sometimes dozens.
People need to be reminded, I think, that shell isn't a programming language first. It's a user interface. And when I look at Powershell scripts and other things of that nature and think about living in Powershell day in and day out I don't see the big pay-off over something like fish.
'Is this going to make it so I can get my work done faster?'
'Is this going to be more pleasant to use as a primary interface for a OS?'
When I go into fish or zsh and use curl to grab json and 'jq' to get a idea of how to use the API in python or node...
versus running 'curl' in powershell against some random API I have never used before..
I get the distinct impression that 'This is fucking garbage' in that It would take me a lot longer to figure out how to use powershell in a useful way then the time I would save by doing so in the long run.
- Yeah, I tried using Powershell as my shell and that's when I found out Powershell is more about the scripting language than an optimized shell used for everyday use. I was confronted with this almost immediately because one of the things I rely on most in Bash is 'dirs -v', pushd and popd. I have dirs aliased to dirs -v and I usually have 15-20 paths on my stack. I'll leave implementing the same functionality in Powershell as a user exercise.
+ Lorem ipsum foo bar baz.
- Im confused... Why doesn't Push-Location (aliased to pushd by default), Pop-Location (aliased to popd by default), and Get-Location -Stack (trivially aliasable to dirs) not work? You can even have multiple location stacks and, if you care about the registry, you can also use the same commands to handle registry navigation as well.
+ Lorem ipsum foo bar baz.
- ‘dirs -v’ shows an enumerated list of directories. If I follow that up with pushd +11 I will be taken to the 11th path listed in the prior dirs command. As far as I know this isn’t implemented out of the box in PS
+ Lorem ipsum foo bar baz.
- The irony is that the very attempt to be one tool for everything caused Perl's own destruction. Perl 5 is still used by some veterans for small scripts but who wants to use Perl 6?
Unix follows the KISS principle, and that is key for success. Albert Einstein said: "Keep things as simple as possible but not too simple". In that sense Unix and Posix are well done. However, that doesn't mean that good ideas like Nushell are not welcome.
- I think the failure of Perl 6 was caused by a lack of community building and implementation leadership, not by trying to be too many things at once.
+ Lorem ipsum foo bar baz.
- Sounds like the CLI tools, GNU and otherwise, could benefit from some kind of "--format" switch to define the output in detail. I mean something like ls --format "{ \"filename\": \"%{filename}\", \"size\": %{size} }" for a JSON-style output (or whatever the user wanted).
Or even something like --format-rfc4627 filename,size,permissions-numeric to get only those fields out in a valid JSON format.
This wouldn't remove the "every program has to know how to parse the output of every other program", but I am not convinced it is needed. For instance, how would e.g. grep know what field you want to process? And does e.g. "size" carry universally the same meaning and semantics in all programs there is and can ever be? Ditto for "permissions". And what about "blacklist".
-The fictious "json-shunt" (for lack of better name) would pass only that input-parameter to its --cmd as an input, in this case grep command, the | part would be done for things for which the grep matched, but with all the other parameters intact. So it'd print the filenames and sizes of filenames with case-sensitive "MegaCorp", and output it in JSON.
Yes, I know there are more concise and elegant ways to express the same thing of printing out file sizes and filenames of matching filenames... Also, when scripting pipelines the verbosity doesn't matter, IMO it'd actually be a benefit to be able to read the script.
Edit: fix pre-caffeine wonky redirection and brain fart
- If GNU decided tomorrow that all utilities need to have a --json output argument then that would make me a very happy person.
+ Lorem ipsum foo bar baz.
- Better than nothing, but the problem with that is that you can't construct a deep, lazily evaluated pipeline. JSON can't be outputted until all the data is collected.
+ Lorem ipsum foo bar baz.
- That still limits us too much due to the rather severely limited available data types (although arbitrary types could be serialized into binary/string data, I guess...)
+ Lorem ipsum foo bar baz.
- Funny that you should mention this. I just hit that problem yesterday. The lack of a binary type is a problem. This is the same thing that hit XML. Unfortunately (or fortunately), the best solution is still the MIME stuff.
+ Lorem ipsum foo bar baz.
I'm trying to elicit more of a response from the comment author. I think they make some good points. I would like to learn more about their ideal system.
- Yes, I agree that it's a tradeoff. Although that tradeoff was made when memory was sparse and expensive, a modern shell should certainly be more structured - e.g. like PowerShell or Nushell here.
+ Lorem ipsum foo bar baz.
However, the unstructuring of data and keeping a single form of communication is what has kept UNIX around for so long and why Linux hasn't collapsed under its own weight long ago.
I was voicing your opinions exactly when I was new to the professional computing world. Over time I saw a lot of structured data schemes come and go and they all fall down with this: inflexibility and improper implementations.
Where do you think does the structure inside the data come from? You need to come up with a standard. Now you need everyone to stick to that standard. You need to parse the entire data or at least most of it to get to the information you need, finding new security footguns on the way. Soon you will realise you need to extend the standard. Now you have not only conflicting implementations (because noone ever gets them completely right) but conflicting versions.
And this needs to be done for every single fucking little filter and every single language they are written in.
Take a look at the various implementations of DER and ASN.1. The standard seems simple at first glance, but I haven't seen a single implementation that wasn't wrong, incomplete or buggy. Most of them are all of that. And DER is a very old standard that people should have understood in the meantime.
In order to get at least a bit of sanity in all of this, you need a central source of The Truth wrt. your standard. Who is that going to be for Linux and the BSDs and for macOS? Linus? Microsoft? Apple? The ISO standards board? You?
And all of this is equally true for log data.
I'm okay with tossing backwards compatibility over board. But not in favour of the horrible nightmare some people here seem to propose.
- It was a decision that lead to success. Before Unix most filesystems were record oriented. Unix's philosophy of treating everything as a stream of char was different and very successful.
+ Lorem ipsum foo bar baz.
- > The key flaw of UNIX philosophy is destructuring deserialization and reserialization based on lines
not so sure about your assertion there eugene :o)
interfacing with funk formatted lines of text is waay more easier because (imho) of the 'wysiayg' principle, and encourages, for lack of a better term, 'tinkering'
> Settling on a common data simple/universal format robust enough for all purposes...
output from 'ls, gcc, yacc/bison, lex/flex, grep, ...' should all follow the same fmt, sure...the good thing about standards is that we don't have enough of them to choose from, f.e. xml, sgml, s-expressions, json, csv, yaml, ...
having said that, when dealing with copious volumes of data, structure doesn't hurt, but in such cases, my guess is, there are only a handful of data-producers, and data-consumers for automatic consumption, and are all tightly controlled.
- I used to think this, and am still sympathetic to the motivation behind this. But I've since changed my mind when I saw the challenges faced in Big Data environments. Structured data is nice at first when you get into it. But the need for data prep stubbornly will not go away. If only because how your stakeholders want the data structured only sometimes aligns with the pre-defined structure. And when you get those numerous requests for a different structure, you're right back to data munging, and wrangling lines of text.
In my limited experience, you're either taking the cognitive hit to learn how to grapple with line-oriented text wrangling, or taking the cognitive hit to learn how to grapple with line-oriented text wrangling AND on top of that a query language to fish out what you want from the structure first before passing it to the text wrangling. I'd sure like a general solution though, if you have a way out of this thicket.
- I don't think I'd go as far as typed messages as that doesn't strike me as a simple or light-weight format. Something like JSON is almost good enough, but perhaps something even simpler/lighter would be ideal here.
+ Lorem ipsum foo bar baz.
- is "destructuring deserialization and reserialization based on lines" really an aspect of the UNIX philosophy or just an implementation detail? I thought it was more about doing one thing and doing it well [1]. It could be argued that nushell follows the UNIX philosophy.
- “This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.”
Doug McIlroy (2003). The Art of Unix Programming: Basics of the Unix Philosophy
That said, I’m looking forward to testing our nushell!
- You can also just use PowerShell. Its chief crime is its verbosity in naming, but it has the "universally available consistent structured output for the CLI" you're asking for.
+ Lorem ipsum foo bar baz.
- I spent a lot of time in Powershell on Windows. Almost 3-4 years. I simply could not use it as a shell.
I enjoyed converting our many build scripts into Powershell, but using it as a shell was just uncomfortable.
It's not clear to me why. If I had to guess, besides the verbosity, it was the additional "indirection". Nothing was what it appeared on the screen. It was something else.
Bash felt far more comfortable as a daily shell because what you see is what you get.
I want to try Nu because it appears to combine that WYSIWYG simplicity of bash with the Powershell quality of passing data rather than text. Maybe tables are indeed a better data structure to use than objects.
What if, instead, we pushed the approach “upstream”, asking systems to have an additional stream that all shells can access? We have stdout and stderr, we could add “stddata” - a pipe that expects and produces structured data. Then it would become trivial to add support around the userland, without displeasing anyone - because it’s just an additional interface, not a replacement. The pipe should support two or three formats to start with (csv, json, and maybe xml, so you can handle both tabular and child-parent relations with a varying degree of precision) and shells could have a special character for piping stddata (I like ¥) so everything else about them would stay the same.
+ Lorem ipsum foo bar baz.- reply