distro automation: design by documentation

[jamesbond]

This is what I like. Frank, lively, yet civil discussions. Agreements are good, but even disagreements brings interesting points of view for others to consider. I hope it stays this way for future posts too.

There are many good ideas discussed. I just want to add a few things.

---

First, some information about Woof-CE. Woof-CE is actually a two-in-one package.

1. It is the "Puppy builder". It contains build scripts Puppy from individual packages from another compat distro.

2. It is, also, the "Puppy essence". All the scripts, tools, configurations, etc that makes a puppy a "puppy", is contained in Woof-CE.

These two things are actually separate and (to a certain extent) independent from each other. It is theoretically possible to take (1) component only and build a non-Puppy with Woof-CE. It is also possible to take (2) only and build a Puppy using alternative build scripts. (For those of you who have been here long enough, you may have heard of woof-next as an example of the second case).

@wiak, if you're looking for "these core/independent Puppy components", then look no further. They are in Woof-CE. Isolating them may require quite some work but it is possible. But remember, these "core" components aren't usable directly. They need to be tweaked depending on which the compat distro is used, and this is done by the build scripts. Hence my qualified statement of them "being independent to a certain extent".

---

Secondly, I want to offer a counter-argument of saying that "official" Puppy is not important. Like I posted elsewhere, Puppy is both an idea, a concept (=I use the term "puppy-like" to denote this), and a distro proper. The word "official" is tagged to the distro which can carry this flag as being "Puppy Linux the distro". It is important because when people look for Puppy Linux, they usually think in terms of "Puppy the distro" and not "Puppy the idea" with tons of (potentially confusing) distros behind it.

It's just like everyone can use debootstrap and make an identical Debian ISO using Debian's own tools, but none of them can call their ISO/distro as "Debian". Debian-derivative perhaps, but not Debian. There is only one Debian, that's the one that is listed and gets distributed from debian.org.

So, "official" Puppy is important. That makes Woof-CE important too, because that's where official Puppy comes from.

---

Thirdly, I'm perfectly fine with the idea of diversity. It is perhaps not obvious, but my point of mentioning Woof-CE isn't about Woof-CE itself, but the value of Woof-CE of being able to assemble a distro from nothing. (s243a has pointed out some issues with Woof-CE that I'm happy to address but I think that should be discussed elsewhere, not on this topic)

My argument is, if you want to make a distro proper, it is better to use a build tool that will assemble the distro from nothing, instead of taking an existing ISO and pick it apart. Or start from a "base" ("barebones" whatever it is called) and "add" to it. Because every existing distro, no matter how small ("barebones" included), have some sort of baggage. It is best to start from "zero-base" or "no-bones" which has no baggage at all, and build from it.

I don't mean that Woof-CE should only be the tool to do it. Woof-CE is useful because that's what is used to build "official" Puppy, but since in this topic we're not talking about official Puppy (or perhaps we're not even talking about Puppy at all - the idea is applicable to __any__ distro), then lets bring forth all other build tools or build systems that can build a distro. Forks of Woof-CE is welcome too, if that's what it takes.

Again, remaster works great for adding packages, and making a custom for-myself ISOs, but when you start removing stuff, you need to know why certain stuff are included in the first place. If not, you run the risk of removing stuff that you think is unnecessary but only later your user will find out that certain apps don't work. If you go the other way, __you__ know exactly why certain packages goes into the ISO.

But that's just my opinion

---

Finally, I will happily skip discussions about "what constitutes a Puppy" here. Because, firstly, I don't think its appropriate for this topic, and secondly it will open a can of worms that will invite a lot of heated off-topic posts. We can start (yet) another thread on it but unless the thread has a clear objective why we should ever discuss this thing again, I'd probably not participate.

I would rather people upload an ISO and says "see this ISO, __this__ is how a puppy should behave/look like" rather than spending days spewing mouthful posts of "this is what I think how a puppy __should__ behave/look like". We have been there far too often than necessary and all of them without exception doesn't lead to anything.

Other than that, please carry on!

[nosystemdthanks]

i think your explanation of woof-ce is good and very on-topic. it is, after all, the definitive (official) spec on what constitutes puppy linux.

when we talk about the importance (or relative importance) of being official, its only in some contexts. in other words, append "sometimes" to every line that questions it. because i certainly wasnt saying "never." if woof-ce makes up 50% or more of the puppy distros, thats alright by me.

as for building from "no-bones" as you put it, woof-ce counts as a spec in that regard as well. you even figured out (as in very accurately, more than would be expected of anybody) what the deal is with not using it. in that its not without a certain amount of trouble to separate the two parts. still, i feel like i know a bit more about it than when this started.

for the sake of argument, treating woof-ce as "a" tool rather than "the" tool (just for the sake of argument) lets us create new specs that may or may not rely on woof-ce. for example, it may be a lot of trouble (not worth it) to isolate the essence from the parts of the script that apply those parts in a distro-dependent fashion.

it could be worth it to isolate that functionality (with the essence) from the gui, so that other tools can use the scripts to apply the essence (so to speak.) and thats not necessarily a task for the woof team, as enterprising tinkerers like myself (or someone who likes bash more, which includes most people here.) wiaks scripts once again, are what im thinking of.

so its not about negating the value of woof, so much as putting a spotlight on alternatives and broader (less puppy specific) options.

above all, its very possible (not certain though) that adapting woof to this would take the longest, which means that for those less patient we would want to mess with alternatives first.

but regardless, the "whatever" spec looks different when you make a version of it that covers usage of woof-ce.

and while the fast, fun way to get an automated distro IS to remaster, (that kernel step takes a while, you know?) having the specs to build a "truepup" could be very handy.

so once again i want to point out that none of this precludes specs that are centred around woof. it just doesnt require woof in every specification.

diversity for the sake of diversity isnt even it-- its the recognition that if everybody does exactly what they want, diversity is inevitable.

thats why its interesting to conceive a system that depending on the spec, may-- or may not-- rely on woof-ce.

but such a spec could (in theory) make it easier for people to understand what theyre doing with it.

for example, a spec that isnt itself automated, but that walks you through woof-ce.

we arent talking about a single unified spec, but the act of creating specs from docs.

so even a guided tour through each step of using woof-ce would qualify as a spec in this regard, or at the very least documentation.

the fact that it could also skip woof-ce is a given. but i figure there will be both people who do and dont want to use it, and thats why i have outlined both (and suggest that this process, when desired, can cater to both.)

as usual, the information is welcome (and frankly, interesting.)

i would also add, that to follow the idea to the letter, would probably require a fork of woof-ce (or, a woof-ce team very keen on the idea, which seems just as unlikely. i never put anything on the woof-ce team just for assuming) that actually follows the documentation.

in other words, if someone creates an acceptable spec that walks through using woof-ce, then woof-ce breaks that-- it is categorically a "bug" that needs to be fixed in either the docs or the scripts. and preferably the scripts, unless there is great justification for breaking it.

just because its broken doesnt mean the team has to fix it. but it means the bug is open until something is fixed.

its not really design by documentation, if the docs are always tweaked to fix the breakage in the scripts. if its design by documentation, then the scripts ought to stay as close to the documentation as possible.

of course this is hypothetical, as nobody including myself ever said that woof-ce would be held to such standards. woof-ce is only an example, in this context. sure, its official. but this idea isnt. it could be, but thats not necessarily where we were going with this.

it would actually presume too much to suggest that woof-ce would have anything to with this idea. the idea is free, and they (or anybody) can take the idea up with their favourite tools if they wish. woof-ce is no exception to that.

something to consider: being official, for better or worse, may prevent woof-ce from moving forward.

heres what i mean. everyone is trying to clean up woof-ce and make it more flexible and modular and useful for a wider variety of tasks. and thats a good thing.

suppose that someone wanted to really innovate it, clean up the whole thing, like was done with reddit when it was ported entirely to python. not that python is always a cleaner language, but it has its strengths.

now, thats a huge project. but the moment youre done, its no longer woof-ce. so no matter what you do to overhaul it, it becomes officially moot as its truly no longer what it was.

im not saying this has any real effect on whether woof-ce moves forward. and what you honestly want as an individual changes what "forward" even means (it could easily be a bad thing, and just as well if we dont move "forward" with it.)

im only suggesting that maybe we dont know. but i do think the woof-ce team probably knows the answer to that better than we do. probably. im not going to saddle them with infallibility, but i figure the odds are on their side.

[jamesbond]

nosystemd, agree with all of your above points.

As for this:

something to consider: being official, for better or worse, may prevent woof-ce from moving forward.

heres what i mean. everyone is trying to clean up woof-ce and make it more flexible and modular and useful for a wider variety of tasks. and thats a good thing.

suppose that someone wanted to really innovate it, clean up the whole thing, like was done with reddit when it was ported entirely to python. not that python is always a cleaner language, but it has its strengths.

now, thats a huge project. but the moment youre done, its no longer woof-ce. so no matter what you do to overhaul it, it becomes officially moot as its truly no longer what it was.

I'd like to say this: "Official" Puppy is just that, official. The distro that is known as Puppy Linux, and distributed from puppylinux.com.

Is the "official" Puppy the best puppy out of the oceans of non-official (and to a greater extent, puppy-like) distros? That is open to interpretation, and you'll probably get as many different answers as the number of people you ask, depending on experiences that they seek after.

The point is, "official" doesn't always mean "better". Un-official and non-official puppies and puppy-like distros can indeed be "better". And that's good, that's healthy.

In your scenario above, you're right that the "cleaned out" version of Woof-CE would be "not Woof-CE" anymore, but I would not say it's officially moot, because a different scenario could happen. If the changes are good, it is plausible and possible that this "cleaned-out Woof-CE" would be adopted as the "next-gen Woof-CE"; and in the best case scenario, the original Woof-CE will then be slowly deprecated where all new development will go inside the newly "officially" adopted baby ... You know how this played out in glibc/eglibc fork.

The reason I say this is to encourage people to innovate, either within Woof-CE, or outside Woof-CE. Either by improving Woof-CE, or coming with with a better alternative. Doing it within Woof-CE context is great, but I fully agree that it is not neither the be-all nor the end-all of all things Puppy. And while it's better (in terms of resource pooling) to contribute back to Woof-CE if possible, if you cannot see eye-to-eye with Woof-CE maintainers, why not fork Woof-CE and make it better. Eventually, if your changes/innovation is truly better, it will either be adopted into Woof-CE/Puppy, or it will __become__ the new Woof-CE/Puppy, moving forward

PS: I saw your PM when I was about to post this. Will reply to that later, cheers!

[mavrothal]

why not fork Woof-CE and make it better.

I do not really care if the next puppy, official or not will be built with woof or even if it will be built at all! But whatever is going to have access to my personal info I would like to be able to READ it, even if I do not fully understand it.
I think that for a distro that runs as root the most important part is to be able to be fully reproduced from human readable files. I believe that the most important contribution of woof-CE or any build system, is exactly that.

I actually still find it very surprising that people that worry if you ping ibiblio can happily run an OS that cannot find out how was made and what is in there! That they can run a binary package the build recipe of which is unknown and thus its content unverifiable. But that’s a different story...

Bottom line, fork/make new or whatever but have it out in the open for everyone to see. The exact process rather than the product.

[nosystemdthanks]

I actually still find it very surprising that people that worry if you ping ibiblio

i find it surprising that you can even keep track of that concern as the core of your critique without understanding the problem with it.

but not as surprising as the idea that woof-ce is being called "human readable." the purpose of woof-ce is to create binaries. either everyone runs woof-ce, or some people (far more) trust the binaries that come out of it. it is good to have the audit, but your defense is silly when woof also lets you fold in any binary you please.

given that woof-ce provides no receipt of what it has done, and that anybody can modify it before running it, being able to examine woof actually tells you less about some things than the binary output does.

it isnt by any fantasy the (complete) source code for puppy. it is certainly a substantial part of it, but just to show how disingenuous your argument really is--

the pings people have complained about are all in HUMAN READABLE scripts in the iso itself!

so youre basically saying that youre surprised people who can find issues with text-based scripts in their puppy iso arent encouraged to go look for issues in a much larger bash script too?

what are they going to find? the part that they take issue with is already as human readable as woof-ce is... if not more!

so far, i have encountered nothing but reason, logic and good points from woof-ce proponents. you lowered the bar with your cheap shot, and you made a point that practically refutes itself. or are the bash scripts in the puppy iso somehow easier for a human to read when theyre nestled in several thousands of lines of woof-ce?

and once again, this is not just about puppy-based distros (so i fail to understand how forking woof-ce would necessarily cover everything) but ive made it very clear that for some purposes, woof-ce would possibly be suitable eventually. you want more than that? okay-- why?

please leave this to james, he was doing just fine. all your post did was leave me with the feeling of "james was making this sound like a good idea, but mavrothal just makes me want to avoid woof like a plague, and recommend others do as well."

please do not work too hard just to give me that impression. im fairly certain woof is worth the trouble for a lot of people. dont add to the trouble that comes with it. if people dont want to watch ghostbusters, they dont want to watch ghostbusters! you dont need to insult them for it.

Bottom line, fork/make new or whatever but have it out in the open for everyone to see. The exact process rather than the product.

you can actually go a step farther than that.

this is not just about making it out in the open for people to see, but about making it out in the open to make it easier for people to change what they dont like when they do see it.

thats the idea here.

now-- just to keep you honest--

if someone has a problem (this is a tender subject, since security concerns about puppy were already mentioned as being censored) with the pings that are distributed throughout various features of puppy-- (and historically, across more than just ibiblio) what do you propose someone do to fix them?

you make a big deal about visibility, but once the problem is found, what can they do about it?

because 1. if people want to bring back or update an older puppy, pizzapup being one example i like (because its relatively ancient and i tried it back then-- also it had some problems) how is woof-ce going to help?

2. if i want to remove the pings from debiandog, are you suggesting i rebuild the entire thing, or use a tool to open it, clean up the mess, and close it back up?

3. how can i get the woof team to recognise the ping problem, or do anything?

if they wont, and im not able to do that, it doesnt do much good that woof lets you see the problem. it just waves it in front of your face and says "good luck!"

if not for your attitude, i could find a different way to describe it than that. let its better advocates advocate, please.

[mavrothal]

let its better advocates advocate, please.

But I don't!
As I said do not really care how the next or an older puppy, or a puppy like will be build, woof or not, as long as it is open and reproducible from flat files.
Regarding the rest... hey, that would be advocating wouldn't it?

[nosystemdthanks]

goals include automation from spec and spec syntax that is beginner friendly / easy to edit.

it is obviously not very sophisticated yet-- ive already automated most of the things in the spec, but not the handling of the schema yet.

future versions of distrolibre should use this.

the spec should develop new abilities as well. it is already non-distro specific.

it may appear simple, but the idea is to be able to perform these changes on multiple distros.

the design is heavy on user freedom and organisational autonomy.

the spec does not preclude building (as opposed to remastering) and would probably incorporate or modify existing build tools (such as woof-next) for that purpose.

remastering is easier and faster to implement, the spec begins with that.

i am also not interested at this time in supporting architectures other than 32-bit x86. that has more to do with the implementation than the spec-- it does not yet specify architecture.

the way you would use this for 64-bit architectures would be to modify mkdlibre.fig 0.5 or replace it with a different tool in the spec.

Code: Select all

#### license: creative commons cc0 1.0 (public domain)
#### http://creativecommons.org/publicdomain/zero/1.0/

= spec

spec name
    distrolibre
spec family
    distrolibre
spec version
    0.5
stage
    experimental


= automation

automated
    yes
automation tools
    fig
    python
    bash
default automation engine
    mkdlibre.fig 0.5


= base

default base distro
    debian
alternative base distros
    puppy linux
    ubuntu
    trisquel
    void linux
    refracta
    devuan
    lfs


= boot

default image bootloader
    isolinux
default image type
    hybrid iso
default installed bootloader
    grub 2
default init system
    sysvinit
alternative init systems
    puppy scripts
    upstart
    runit


= mode

default environment
    x11
gui inclusion optional
    yes
default basic shell
    dash
default user shell
    bash
default root shell
    bash
default vt editor
    nano


= network

enable dhcp
    optional
        default
            yes
sync with time server
    optional
        default
            no
hosts prevent mozilla updates
    optional
        default
            yes
disable pings from puppy scripts
    yes
        = not fully automated


= code support

include perl
    yes
include python2
    yes
include pip for python2
    optional
include pypy
    optional
include pip for pypy
    optional
alternative python versions
    optional
include fig
    yes
        include versions
            3.1
            4.6
include figplus
    optional
include alex shell
    yes
        version
            2.9


= gui environment

default desktop
    icewm
default taskbar location
    top
show clock
    optional
        default
            no
default text editor
    leafpad
include graphics editor
    optional
default graphics editor
    mtpaint
include pygame for python2
    optional
default term window
    xterm
default pdf viewer
    xpdf


= free licenses

remove non-free programs and libraries
    yes
remove files related to non-free firmware and drivers
    yes
replace kernels containing non-free firmware
    when technically possible
remove non-free documentation
    as found
classify gnu fdl as non-free
    yes
remove non-free artwork
    as found

[wiak]

@nosystemdthanks: I'm discussing your specification idea (with some expansions to the idea) in other thread regarding poll of Puppy future direction:

http://www.murga-linux.com/puppy/viewto ... 88#1024288

http://www.murga-linux.com/puppy/viewto ... 00#1024300

[nosystemdthanks]

@nosystemdthanks: I'm discussing your specification idea (with some expansions to the idea)

that is super cool. with the utmost respect, i figure theyre going to say its not worth it. and thats alright-- if (probably) and when i implement it, they may decide its worth it then. if they use the idea, thats great, that would be cool and help a lot of people i think.

of course ill be implementing it in something a little easier to work with than bash, im sure they will use bash (its not a bad language, im just not likely to get any better with it than i already am-- and im not as good with it as those guys.) the advantage of bash its its small and ubiquitous. an obvious and understandable choice for most of these guys. it would be very cool if puppy had this feature and i wish them the best of luck. its doable.