perl to publish : perl as a first language

in this maiden programming post, i test my mettle imparting the experience of gaining new knowledge without completely awashing the reader in jargon or in unearned programming confidence.

i am very, very new to programming. the idea of "learning to program" without knowing what you want to do and why is the first and most important barrier for anyone willing to dive in to breach. i think a lot of would-be learners really struggle with finding an initial jumping-off point more than anything else. there's also no kidding that learning to program by yourself is very, very hard, and that that part of you that asks "but why" whenever you're told something that helps you stay curious can actually do more to hurt you than help.

enter: Perl, and a problem to solve.

i'm lined up for my second year of college. i'm a confident guy, but something in my gut tells me that things are about to get a hell of a lot more difficult quicker than i might expect. sounds scary, sure. but when you are a nerd, this is the type of thing that excites you. i'd been using Obsidian, the zettelkasten¹ Markdown program, for around a year and a bit prior, but i'd been using it as if the "zettelkasten" part didn't matter. the whole of my notes for all of last year were fit inside of 20 files and three folders total. and you know i don't care to go back and read a single one of them. they're a mess. while playing around and learning with my new homelab, i'd been making a lot of use out of zk, a program that creates and manages database files that link other notes and folders together. it's still 'an Obsidian' in concept, but it pulls me out of typing inside of a Program like it's microsoft word. as of right now, i do all of my writing inside of hx (Helix). if "it's like vim" doesn't mean anything to you then i won't bother explaining it here; rest assured that it doesn't mean much to me either.

talking to objects

i don't care to strut my stuff here, but i switched to linux a while ago. that's right, folks. just god damn do it already. you can ignore all the putzy nerdy shit. it's just a computer. we all know how to use one.

i think this is a larger computer-semantics-reshuffling operation than people give it credit for. linux can do what it can to paper over the programming by being Fedora or Ubuntu but it's still a machine meant to be worked on. you're not supposed to not know how your computer works. it's uncouth and rude. computers spent all this time in their history being touched and communicated with only by certified computer touchers and now look what we did. i got an ad on my house's smart TV the other day trying to sell me a bastardized search engine as a "business solution", whatever that's supposed to god damn mean to me, and it made super sure how to tell me how smart it was because it could read "winter" and "boots" and serve you results for "winter boots", as if this was something new, as if this was a something that i was expected to be in awe of. i could have died laughing.

/^---$/

this little guy is a regular expression, also known as a regex. it encloses whatever it wants to match inside of two forward slashes, prepended by a carat ^ and appended by a dollar sign $. what's important about these two symbols is that they mark the beginning and end of an exact sequence of characters. and when i say exact, i mean exact -- these three dashes are what an entire line must consist of in order for the regex to match. what happens when a regex is matched is entirely up to the program. two three-sets of dashes --- flank our desired transformation area.

i make mention on my index page (at time of publishing) of two human readable data languages called YAML (yammel) and TOML (tomal). they're the same aside from two subtle differences. YAML looks like this:

---
title: group ownership vs. ACLs: how they work together
showMiniToc: true
intro: i'm having a tough time understanding both a) what is different between set/getfacl and chgrp b) how they can coexist and not have it be completely redundant
taxonomies:
  perch-topics:
    - access control lists
    - security
    - user groups
---

and TOML looks like this:

+++
title = "group ownership vs. ACLs: how they work together"
showMiniToc = true
intro = "i'm having a tough time understanding both a) what is different between set/getfacl and chgrp b) how they can coexist and not have it be completely redundant"
[taxonomies]
perch-topics = ["access control lists", "security", "user groups"]
+++

• --- delimiters vs. +++ delimiters
• title = vs. title:
• YAML's cascading list-style variable trees vs. TOML's array one-liners

the site that you are reading this on is generated by a program called Zola. Zola can parse YAML if it had to but it would really prefer TOML instead. thus, i had found it: my very first problem.

my very first problem

i now had to figure out how computers read. did you ever get an assignment in grade school where you had to write a perfect set of instructions to assemble a peanut butter and jelly sandwich from scratch?

my current interest in coding comes in no small part from how scary code can be to look at, and how scary it remains to read even after you've written it. when i initially planned out this blog post in my head i'd figured that i'd be able to turn this into a learning exercise for the viewer. please don't expect that of me here. i've got to bop the head of the part of myself that expects to be an immediate genius about any subject they touch. this post is more humiliation ritual for them than it is a tutorial for you.

Perl is largely a database language, or what's called a glue language. that is: Perl does data administration jobs like transformation (turning this into that) who's output is then typically used for something else. Perl is for turning a balloon into a dog. Perl's development philosophy is informed by the creator's background as a professional linguist and a devout Nazarine Methodist; the hermeneutics² of programming languages will be a theme on this blog once i have more things of substance to say.

computers are ones and zeros. the computer at any given time is taking on anywhere from a billion or a trillion math calculations per second. this would not be possible if we had not developed standardized communication methods for telling individual transistors and electrons what to do and where to go. the above code snippet tells Perl: from this file you are reading --- which, in this case, is an array @ (a list) of one file, for testing purposes --- turn what you see into a variable: $content. it does this by doing the following things:

1. opening the file with the "<" operator. This symbol expects to be flanked on it's left and it's right by a variable to connect this action to and what file to source from, respectively.

• in programming, "opening a file" and "reading a file" are two different things. think of it like opening a book, where the very first page is more often than not the title, completely empty, or otherwise just confirms that you have opened a book by virtue of now looking at paper. you cannot say that you have read the book.
• or die is the (funny) way Perl handles errors. $! symbolizes the returned error message for the purposes of logging and development.

2. stay with me: open --- per Perl's documentation --- "associates an internal FILEHANDLE $fh with the external file specified by EXPR", the variable to the right of <.

3. for my $content --- a new variable we are declaring --- do this: read it all in one go. when you write what you're reading down, do it all on one line: on a lined piece of paper infinitely wide.

• "we are ignoring newline control characters³, but we are not discarding them". typically, Perl would stop reading after a newline, because that's how the program is written.

4. now, close the connection we have with this file through $fh.

i'd like to pause here and really appreciate how much energy it took for me to translate only the first ten lines of this script into plain english. not including any breaks i had to take, this took me over an hour to write. that tells me what i need to know about setting expectations for myself here. don't expect explanations to remain so thorough and consistent...

we can now begin to use the knife

Perl has been my toe-dipping language before i wade into C. i won't flex programming language knowledge that i do not yet have; it's my understanding that Perl and C are alike in that they can each absolutely tear through anything you throw at them. if you don't write something, it doesn't happen. if you write something, it's guaranteed to happen too much in a way that completely tears through anything you give it like a chainsaw. this is not 'volatile' behaviour. Perl and C do exactly what you tell them to do.

we materialize the space for which various bits of information will be stored. the most important variable of all is the first variable declared after the one long $content string is created:

my @lines = split /\n/, $content, -1;

chop $content at every newline character --- \n, the character we made sure to ignore but to keep. place these lines into an array. i don't remember what -1 does, sorry. it had something to do with control characters.

the foreach my $line (@lines) { [...] is where essentially everything else contained within the script happens. if you were able to follow along with the first explanation, you may understand why. semantically: "for each line in the lines array ..." and then a collection of sequential instructions. this loop begins on line 23, and this script is, quite pleasingly, an exact 100 lines long. if you were able to understand everything you have read so far, you understand at least 23% of the program. i hope that for both our sakes that this number continues increasing.

if this, do this ... if there, do this ... if you see this, you are now here ...

keeping track of this stuff is so, so hard. it's around at this point where i had to come to terms with and confront the immense friction that will be caused by a lack of any implicit understanding of what i "meant to do" from the computer's perspective. no AI quips here. on the computer's behalf there is no "and this goes without saying; this is what i meant".

do you remember what a regular expression is? if you don't, that's ok. you can scroll back up to give yourself a refresher. we are going to make use of regular expressions in order to write exact instructions. correlate the example regex i gave earlier with what you notice about the YAML structure. this pattern is going to be very important for writing the logic required to tell Perl when to stop modifying anything.

    perch-topics:
    - access control lists
    - security
    - user groups
---
^^^ stop modifying stuff *here*

# chgrp and webdeploy
we want to create a group called `webdeploy`. we then want to `chgrp` the `cgi-bin` files to this new group. what this does it choose which "keyring" sits in that one group slot, and controlling what the keyring can do ...
[...]

frontmatter can sit at the top of any Markdown file and is used in this context to create page metadata: meaning 'data that describes this page'. i made brief mention of 'delimiters' before when introducing YAML and TOML to you. a delimiter is anything that marks the boundary between separate parts of data. the first and second set of three dashes mark the start and end of the data block. everything below is page body text that we do not want to modify. this is what has to be definitively ironed out before any transformation logic can commence.

• you are already aware of delimiters: [a,b,c], (a,b,c), and a:b:c all use them in varying combinations to create boundaries between different pieces of information.
• you will see a smaller example of a delimiter in the next chapter below.

we start picking up steam pretty quickly here, so this is where i remind myself and inform you the reader that this is by no means meant to be a tutorial or an authoritative source of information by any measure. believe me that i doubt you are more overwhelmed than i have been trying to walk back and describe something that i wrote myself. learning is difficult. learning anything can be, but there's something about programming that implicitly overwhelms me. i barely passed any maths class past grade 6. it and gym are the only classes i ever constantly cried in.

the symbols

here, catch:

/^(\w+): (.+)$/
^~~~~~~~~~~~~~^
      |
      |
      V
 $key = $1;
 $value = $2;

ignore what the symbols inside the parentheses mean for a second, and notice that there are two sets of them. Perl will collect data from capture groups and assign them numbers in sequence automatically. in this instance, i choose to store these captures in two variables named $key and $value. i do this because, in the next if statement, i evalute whether $value matches one of two exact text strings: true or false. if $value is either of these words, i have determined that this current line of info text is a boolean: a data type that's either "yes" or "no".

the transformed boolean becomes:

showMiniToc: true --> showMiniToc = true

it's important that the colon part of showMiniToc: true is not enclosed within anything. we don't capture it and we don't assign it a value. this colon is a type of delimiter, and it's how we know that $1 will always be a key, and $2 will always be the value of that key. the colon is not captured because it's the only part of the line/regex that will never change. by virtue of being $in_info, colon means something very specific.

syntactically speaking⁴, we do not need to give $1 or $2 names unless we want to use them to evaluate against another capture group. this has the secondary semantic value⁵ of making the code much easier to read.

anything that isn't a boolean undergoes this transformation

push @output, "$key = \"$value\"";

which is the same transformation except $value gets wrapped in quotes.

another "checkpoint" function and accumulators

we are now in taxonomies. we know we are now in taxonomies, because we have hit a line --- /^taxonomies:$/ --- that defines the start of the taxonomies block. Zola uses taxonomies to collect tags into groups that link to pages. we must now neatly, and consistently, push these tags under this key block into a new accumulator within @output called '[taxonomies]'.

i will now introduce what the capture group symbols within two example regexes represent, starting with the one i told you to ignore. ignoring the forward slashes, parentheses, and ^$, which we have already covered. remember that the first parentheses group is the $key, and the second parentheses group is the $value. i'll place the two examples one after the other so that you may compare them.

/^(\w+): (.+)$/
(intro):( this is the introduction to my post.)

symbol	function
\w	"match any word character: letters, digits, or underscores"
+	"one or more of these"
.	"any character except for newline"
+	"one or more of these until a newline stops it"

/^\s+([\w-]+):$/
(perch-topics):

symbol	function
\s+	"one or more whitespaces"
([\w-]+)	the capture group; within the square brackets is a set of conditions: "match any one character in this set", the set being "either word characters or a literal hyphen" (the group being captured in this example is named perch-topics). as established, a plus will always mean "one or more of these".
:	a literal colon that, like last time, is itself not captured.

• we do not use '(.+)' because 'any character' includes a space. a key line will never contain a space.
• capture groups are placed within these regexes with parentheses. these capture groups turn into numbered variables starting from '$1'. if you see '$(number)' referenced in a line, it is referring to the contents of the capture group(s).

in this second regex here, we want to capture only a value on a line with some amount of leading whitespace. the line underneath taxonomies will always be indented; any values nested within another value always will be, as YAML is an indentation language. anything indented underneath something else is nested information within it.

taxonomies:
  learn-perch: <--
    - access control lists

coding loops are linear

if ($in_taxonomies && $line =~ /^\s+([\w-]+):$/) {

	if (@items) {
		my $list = join ', ', map { qq("$_") } @items;
		push @output, "$active_key = [$list]";
	}

	$active_key = $1;
	@items = ();
  next;
}

there's a couple of things about this part of the script onwards that are and will remain confusing for me, and it's likely that as time goes on understanding this fundamental programming concept will remain my tallest hurdle. who knows, though. syntax is very difficult for me to intuit.

this if has a nested if inside of it that references an @items array. this is confusing, because if we read this script as one linear set of instructions, nothing concerning @items has happened yet. this is because if (@items) only executes when it's value is true, and in Perl, anything that isn't 0 is true. this array called @items must evaluate as true; else, all that this function would print is an empty key line, and all keys have tags in them.

@items is true the moment it contains any items. this if below, when true, is what allows for @items to become true.

if ($in_taxonomies && $line =~ /^\s+-\s+(.+)$/) {
  push @items, $1;
  next;
}

• "... if we see an indentation, a dash, a space, and then a value, this is an item, part of the active key array."
  
• "it is the job of each if (@items) loop to determine when it is finished --- 'if (@items)' is finished the moment i'm no longer being called."
  
• "i'm no longer being called as soon as a new key or a dedent is matched."

the biggest difference between YAML and TOML here is that as opposed to indented and dashed lists, TOML expects the key to be on the same line as @items. this requires that the items not be treated as lines to be re-written. instead, each time the script sees something that looks like an item, it pushes the result of regex capture group to an @items array that is written out with each line. once @items is no longer true --- which is the moment it stops being called --- @items is flushed = ();, and it now contains nothing. this allows for the capture and transformation of another key and item pair if one exists.

this is an important concept: stopping is the absence of something else being told what to do. @items is no longer pushed when there is no new value to push to it. no separate function to 'declare' if (@items) finished is necessary.

the finish line

if ($in_taxonomies && $line =~ /^\S/) {
	if (@items)
		{
			my $list = join ', ', map { qq("$_") } @items;
			push @output, "$active_key = [$list]";
		}
	$in_taxonomies = 0;
	$active_key = '';
	@items = ();
}

with the arrival of /^\S/ or "Completely nothing", no whitespace, no letters, no anything, we have officially exited taxonomies. like the item flush before it, we also reset $active_key to nothing in case it's needed for another one. everything above hopefully helps explain something that i didn't mention earlier: the very first if statement had a nested if (@items) and matching $active_key and item flush logic. i'm hoping if i did well enough that you now understand why.

push @output, $line;

if a $line does not evaluate to any regex in the foreach loop,it is simply appended to the end of the @output array and nothing else, leaving it untouched. if we wanted to be nefarious we could intentionally write post body content that breaks our sacred oath. for our script to function properly, we mentally declare that a body line should never evaluate to ---. when Perl was used to collect user input through web forms in the early world-wide-web age, this kind of behaviour was easy to exploit if input guardrails were insufficient.

if you'd like to learn more about Perl and the history of early web applications, here's a good post on Cybercultural about the subject. i lack the ability to convey here how important Perl was to how any early website you could interact with worked.

$content = join "\n", @output;

print $content;

print all lines to the screen.

it's about the journey

i ended up not actually needing this script at all

i imagine this happens a lot during the learning process. i had initially started this project with the intent of using this for some kind of draft-to-publish pipeline (hence the name, perl to publish) but i'd discovered long afterwards, much to my chagrin, that "intro" and "showMiniToc" are not native frontmatter variables that Zola cares to use. each of their use cases are deprecated by native Tera⁶ functions. not only that, but during the programming and note-taking process, i never touched Obsidian once, rendering the whole needing YAML thing moot to begin with. it should be clear though that this bothers me a minimal amount. you may recall, if your working memory hasn't shooed it away, that i said i've been doing all of my note taking with hx and zk anyway. i even mentioned not being fond of feeling like i'm "in a program". it didn't become clear to me how little i needed it until the script was already finished.

syntax and hard rules

i hate it. i hate it i hate it i hate it, i rake my clenched hands through the dirt and i kick my feet in agony. i, hate, rules. everything happens for a reason to my dismay. every particle, every electron, every flake of ash that falls from the sky: all of their positions are pre-determined whether or not they've been figured out. to me that means little and it means little to most people. but those are the rules of how everything works.

i undertook this project because of a false dichotomy lodged in my brain. i'd made a potentially fatal assumption while choosing sociology as my desired university major. that classic assumption that the humanities were somehow "the opposite" of "hard science". you know, that cultural assumption that the "soft sciences" are the less important ones by virtue of being all squishy and pathological. not my words, just something that's very culturally ingrained here in the West that can be hard to notice. this project and all projects hereafter i'm sure was a nightmare of steering away from a constant subconscious shorthanding of information. i felt here that i really better came to understand why rules can be so hard for me to follow. as i take information in, it's like it has to go through this filter first that compresses everything before i'm able to interpret it; the compression always comes before reading the data, not afterwards like it should. i've never done something like this before. this 100-line script took me over a month to write.

neurodivergency

i'm proud of myself. i developed the patience and discipline to guide myself through a difficult project. i have a better understanding of how and why so many "learn to code" learning resources and bootcamps online have failed me. i have a long way to go in being able to better articulate 'why' beyond the usual criticisms.

why is it seemingly such a common method for coding bootcamps to make up a pretend video game with impossibly simple 'ifHit = doDamage 3'-type logic that doesn't translate to anything i'd actually want to read and write? 'programmers are geeks. geeks like video games. what if school work were like a video game?' it's some real Little Boy logic.

things are hard for me for reasons i can't explain to other people, and for reasons i don't need to explain to people who understand. i am blessed every day that i exist within a sizable group of comrades whom have passively transferred the gift of Taking it Easy to me. the only way i got through this project was by taking it extraordinarily easy on myself, a level of easy than what even the most beginner-friendly courses or tutorials may've ben able to accomodate. i regret that i wasn't writing this post while learning but having to go back and assess "okay, what happened, what did i do" definitely helped me reinforce some very fundamental programming concepts that i'll have to get a handle on going forward. because this was post-hoc though i did not have any access to the process of live decision making, and the flow of it all may have suffered for it, but i did my best.