rationale for a regular expression package

[none) (albert]

There is an effort underway to collect valuable snippets of
Forth, https://github.com/project-forth-works/

I have regular expression package that I used (i.a.) to transform
the fifth chapter of "thinking forth" from xml to tex and the
manx music score that went through half a dozen different notations.
I have recently used it to convert ARM assembler
language to risc-V language. Regular expression without the
facility to transform texts is a joke.

The program doesn't comply to ISO, so I thought an elaborate
rationale is in order. If you consider the rationale unconvincing,
I will not bother to upload the package.

"
Why ciforthisms?
In short, I could never manage to write the regular expression
package with ciforthism.
Note that to have recently used it to convert ARM assembler
language to risc-V language. Regular expression without the
facility to transform texts is a joke.

****************** STRINGS **********************

A regular expression package handles strings. Any attempt to handle
strings portably in Forth sucks.

1. String denotations.
Strings defined by S" are unacceptable:
the S leads to confusion
Much less visible is the problem that S" AAP" is 3 characters
and S" AAP" is 4, I guess?. In ciforth there is a normal string
denotation:
"AAP" is three chars
" AAP" is four chars
" AAP" is five chars
2.
Strings with escapes are unacceptable:
They conflict with the escapes needed in this package.
There is one exception, the double quote convention.
In order to have a double quote &" in a string you have
to double it like so `` "" ''.
In ciforth you can have strings that span multiple lines,
albeit only in a program.

3. Character denotations, function denotations.
We use &A and for a good reason.
CHAR [CHAR] are verbose and cumbersome.
Moreover the notation 'x' for the character x overloads the tick
too much, because in ciforth ' is reserved to denote dea's.
Functions in a regular expression are passed around as executions tokens
like so 'ADVANCE+ , using a single tick, without square brackets
around it.
POSTPONE 'ADVANCE+
fails because "'ADVANCE+" is not in the dictionary, which is just
as well. ( "ADVANCE+" is present, of course).

4. Low level abstraction fails.
COUNT -TRAILING S" C" WORD PLACE form a non-unified conglomerate
of handling strings.

You need : $! $@ $+! C+$ and $/
Get familiar with these. They are easy, because they use
the underlying buffers explicitly. Any half-assed attempt to
use string-stacks and more poorly conceived abstractions
adds to the problem not the solution.

************************** The library *************************

I use WANT. All words following are supposed to be available.
You can load them, or give an error are they not present.

The library contains tools, notably the SET and BAG tools.
It is not good to have them as part of the package.
First get familiar to the tools, then attempt to read the
package. You have lifted yourself to a higher plane.
That is needed for regular expressions, they are not easy.
Anytime you encounter WANT you have to get familiar to the
tools wanted. In ciforth they are mostly Forth words that
are present in fat Forth's, so there in lies no problem.

****************** USABILITY ******************************

Using Forth programs sucks. Mostly you must have the particular
Forth installed in your computer. Then the invocations leave
much to be desired.
The best one come up with is
<forth> <somesource>
but the question is "where do the parameters go
-- in particular filenames -- ? "

ciforth solves this in four ways, each of those are
preferable in some situations.
1. interactive, you start forth then you load your program
2. scripting, the forth is in the first line of the script
Parameters are passed naturally to the script.
The forth must load the regular expression module first
3. scripting, the regular expressions library is compiled into
an enhanced forth. In the script you specify the the particular
expressions you want changed.
Parameters are passed naturally to the script.
Note that you don't need a Forth installed. The scripting
program is enough. You can send it as an attachment and
it will run on the same Operating System.
4. Compiled. Now the transformations you want to do are
burned into the executable.
Parameters are passed naturally to the script.
Note that you don't need a Forth installed. The scripting
program is enough. You can send it as an attachment and
it will run on the same Operating System.

This sounds a lot of hype. Therefore I have supplied a
simple practical examples, in several circumstances and operating
systems.
"

Groetjes Albert

[NN]

Post by none) (albert
There is an effort underway to collect valuable snippets of
Forth, https://github.com/project-forth-works/
I have regular expression package that I used (i.a.) to transform
the fifth chapter of "thinking forth" from xml to tex and the
manx music score that went through half a dozen different notations.
I have recently used it to convert ARM assembler
language to risc-V language. Regular expression without the
facility to transform texts is a joke.
The program doesn't comply to ISO, so I thought an elaborate
rationale is in order. If you consider the rationale unconvincing,
I will not bother to upload the package.
"
Why ciforthisms?
In short, I could never manage to write the regular expression
package with ciforthism.
Note that to have recently used it to convert ARM assembler
language to risc-V language. Regular expression without the
facility to transform texts is a joke.
****************** STRINGS **********************
A regular expression package handles strings. Any attempt to handle
strings portably in Forth sucks.
1. String denotations.
the S leads to confusion
Much less visible is the problem that S" AAP" is 3 characters
and S" AAP" is 4, I guess?. In ciforth there is a normal string
"AAP" is three chars
" AAP" is four chars
" AAP" is five chars
2.
They conflict with the escapes needed in this package.
There is one exception, the double quote convention.
In order to have a double quote &" in a string you have
to double it like so `` "" ''.
In ciforth you can have strings that span multiple lines,
albeit only in a program.
3. Character denotations, function denotations.
We use &A and for a good reason.
CHAR [CHAR] are verbose and cumbersome.
Moreover the notation 'x' for the character x overloads the tick
too much, because in ciforth ' is reserved to denote dea's.
Functions in a regular expression are passed around as executions tokens
like so 'ADVANCE+ , using a single tick, without square brackets
around it.
POSTPONE 'ADVANCE+
fails because "'ADVANCE+" is not in the dictionary, which is just
as well. ( "ADVANCE+" is present, of course).
4. Low level abstraction fails.
COUNT -TRAILING S" C" WORD PLACE form a non-unified conglomerate
of handling strings.
Get familiar with these. They are easy, because they use
the underlying buffers explicitly. Any half-assed attempt to
use string-stacks and more poorly conceived abstractions
adds to the problem not the solution.
************************** The library *************************
I use WANT. All words following are supposed to be available.
You can load them, or give an error are they not present.
The library contains tools, notably the SET and BAG tools.
It is not good to have them as part of the package.
First get familiar to the tools, then attempt to read the
package. You have lifted yourself to a higher plane.
That is needed for regular expressions, they are not easy.
Anytime you encounter WANT you have to get familiar to the
tools wanted. In ciforth they are mostly Forth words that
are present in fat Forth's, so there in lies no problem.
****************** USABILITY ******************************
Using Forth programs sucks. Mostly you must have the particular
Forth installed in your computer. Then the invocations leave
much to be desired.
The best one come up with is
<forth> <somesource>
but the question is "where do the parameters go
-- in particular filenames -- ? "
ciforth solves this in four ways, each of those are
preferable in some situations.
1. interactive, you start forth then you load your program
2. scripting, the forth is in the first line of the script
Parameters are passed naturally to the script.
The forth must load the regular expression module first
3. scripting, the regular expressions library is compiled into
an enhanced forth. In the script you specify the the particular
expressions you want changed.
Parameters are passed naturally to the script.
Note that you don't need a Forth installed. The scripting
program is enough. You can send it as an attachment and
it will run on the same Operating System.
4. Compiled. Now the transformations you want to do are
burned into the executable.
Parameters are passed naturally to the script.
Note that you don't need a Forth installed. The scripting
program is enough. You can send it as an attachment and
it will run on the same Operating System.
This sounds a lot of hype. Therefore I have supplied a
simple practical examples, in several circumstances and operating
systems.
"
Groetjes Albert
--
Don't praise the day before the evening. One swallow doesn't make
spring. You must not say "hey" before you have crossed the bridge.
Don't sell the hide of the bear until you shot it. Better one bird in
the hand than ten in the air. - the Wise from Antrim -

1. string denotations

I dont follow why S" is unacceptable. Its just another forth word. You write
your own forth so its just as easy for you to define and use "

" AAP" is four chars
" AAP" is five chars

I looked these up on a fixed font and they both appear the same. So not sure
why you would get 5 chars

PS : regarding "AAP" is three chars -
Once you stray from the principle that forth words are separated by space(s),
every string will become an exception because you will need to test if its a string
you are reading or a word.


2. strings with escapes

Are you using non-standard escapes?
S\" is now in ext-core so you can use some escapes codes.

3. characater denotations

char /[char] are verbose but they stand out
'x' is easy to read


4. Low level abstractions

(a) C", COUNT, PLACE, WORD
(b) S"
(c) -TRAILING

(a) assumes a byte counted string
you can always use LCOUNT, LPLACE, PARSE and/or these are easy to write.

(b) is supposed to be what we should be using unless we are saving space.

[Gerry Jackson]

Post by none) (albert
There is an effort underway to collect valuable snippets of
Forth, https://github.com/project-forth-works/
I have regular expression package that I used (i.a.) to transform
the fifth chapter of "thinking forth" from xml to tex and the
manx music score that went through half a dozen different notations.
I have recently used it to convert ARM assembler
language to risc-V language. Regular expression without the
facility to transform texts is a joke.
The program doesn't comply to ISO, so I thought an elaborate
rationale is in order. If you consider the rationale unconvincing,
I will not bother to upload the package.

In my experience I don't think the Forth world is interested in Regular
Expression packages. About 10 years ago I wrote a comprehensive Regex
package in ANS Forth (yes it can be done) that works on several Forths
including GForth, VFX Forth, SwiftForth, WF32 and my own system and made
it available on a website and there was virtually no interest.

I agree using regular expressions is extremely useful for text
processing and wonder why more use isn't made of it.

I didn't bother with a rationale for writing it, because I wanted to was
sufficient.

I hope you have a better response than me but if it only works on
ciforth ...

Post by none) (albert
Why ciforthisms?
In short, I could never manage to write the regular expression
package with ciforthism.

^^^^
Don't you mean without ciforthisms

Post by none) (albert
Note that to have recently used it to convert ARM assembler
language to risc-V language. Regular expression without the
facility to transform texts is a joke.
****************** STRINGS **********************
A regular expression package handles strings. Any attempt to handle
strings portably in Forth sucks.
1. String denotations.

Why, I used them without any problems

Post by none) (albert
the S leads to confusion

No it doesn't

Post by none) (albert
2.
They conflict with the escapes needed in this package.

I had no problems with conflicts.

Post by none) (albert
There is one exception, the double quote convention.
In order to have a double quote &" in a string you have
to double it like so `` "" ''.

Yuk

Post by none) (albert
In ciforth you can have strings that span multiple lines,

Regular expressions in my package could span several lines for readability

Post by none) (albert
3. Character denotations, function denotations.
We use &A and for a good reason.
CHAR [CHAR] are verbose and cumbersome.

No need for CHAR or a prefix was needed

Post by none) (albert
4. Low level abstraction fails.
COUNT -TRAILING S" C" WORD PLACE form a non-unified conglomerate
of handling strings.

I didn't, but of those I only used S" (I think)

[Paul Rubin]

Regular Expression packages. About 10 years ago I wrote a
comprehensive Regex package in ANS Forth (yes it can be done) that
works on several Forths including GForth, VFX Forth, SwiftForth, WF32
and my own system and made it available on a website and there was
virtually no interest.

Well I didn't know about it. I think it is good to have such a thing,
for those who do any type of text processing in Forth. I can't think of
any times where I'd have used it myself though.

[[e-mail]]

Regular Expression packages. About 10 years ago I wrote a
comprehensive Regex package in ANS Forth (yes it can be done) that
works on several Forths including GForth, VFX Forth, SwiftForth, WF32
and my own system and made it available on a website and there was
virtually no interest.

Well I didn't know about it. I think it is good to have such a thing,
for those who do any type of text processing in Forth. I can't think of
any times where I'd have used it myself though.

For some alarm log filtering long time ago I just called awk from Forth.
But we had to upload the logs to a server to do it and send the result back.
Had we some regex in the remote machines it could perhaps be done
locally - theoretically.

[Hans Bezemer]

Post by ***@arcor.de
For some alarm log filtering long time ago I just called awk from Forth.
But we had to upload the logs to a server to do it and send the result back.
Had we some regex in the remote machines it could perhaps be done
locally - theoretically.

There are several things to the issue. First, getting the individual elements.
I can use both a sscanf() like function (if I'm lazy) or simply use several "PARSE"
variants to break those up (like a PARSE-CSV).
Second, if you really got messed up elements there are several ways to handle
those. I got a small lib which enables you to encode REGEX like expressions -
but in Forth.
(https://sourceforge.net/p/forth-4th/code/HEAD/tree/trunk/4th.src/lib/chmatch.4th)

For some purposes, I got specialized libs to handle those. Like dates:
(https://sourceforge.net/p/forth-4th/code/HEAD/tree/trunk/4th.src/lib/getdate.4th)

Or time:
https://sourceforge.net/p/forth-4th/code/HEAD/tree/trunk/4th.src/lib/hms2secs.4th

Once you got their numerical equivalents, it's not too hard to reformat them accordingly.

Therefore - I hardly ever need regex to get the job done. In the 30 odd years I've been
using 4tH, I built a vast array of specialized functions to tackle that. Every time I
encountered something I didn't have a function for - I created it. Note I use (and used)
it ad hoc for my professional work too - with significant success.

Once I wrote a tiny program to do some conversion. The guy asked if he could get the
source. So I printed it for him. To my surprise he opened the tray. "Why you're doing that"
I asked him. He said "Only a single page came out. I wondered if it were empty".

Another time was when I was confronted with a user list - which had almost any known
variant of combining and formatting first names, last names and initials. They wanted
to hire 3 guys for 3 months to figure that out. My program resolved about 80% of the
issues in 0.1s

So I think I'm good. In any case, I also got a tiny matching regex if I'm being lazy. Used
it once in a program I guess - just because the expression had to be entered at runtime.

Hans Bezemer

[none) (albert]

Post by Hans Bezemer

Post by ***@arcor.de
For some alarm log filtering long time ago I just called awk from Forth.
But we had to upload the logs to a server to do it and send the result back.
Had we some regex in the remote machines it could perhaps be done
locally - theoretically.

There are several things to the issue. First, getting the individual elements.
I can use both a sscanf() like function (if I'm lazy) or simply use several "PARSE"
variants to break those up (like a PARSE-CSV).
Second, if you really got messed up elements there are several ways to handle
those. I got a small lib which enables you to encode REGEX like expressions -
but in Forth.
(https://sourceforge.net/p/forth-4th/code/HEAD/tree/trunk/4th.src/lib/chmatch.4th)
(https://sourceforge.net/p/forth-4th/code/HEAD/tree/trunk/4th.src/lib/getdate.4th)
https://sourceforge.net/p/forth-4th/code/HEAD/tree/trunk/4th.src/lib/hms2secs.4th
Once you got their numerical equivalents, it's not too hard to reformat them accordingly.
Therefore - I hardly ever need regex to get the job done. In the 30 odd years I've been
using 4tH, I built a vast array of specialized functions to tackle that. Every time I
encountered something I didn't have a function for - I created it. Note I use (and used)
it ad hoc for my professional work too - with significant success.

<SNIP>

Post by Hans Bezemer
So I think I'm good. In any case, I also got a tiny matching regex if I'm being lazy. Used
it once in a program I guess - just because the expression had to be entered at runtime.

How do you go about the following situation, just curious.
No need to go in detail, just outline.

Suppose you are confronted with a number of files like
so
"

SCORE Vader-Jacob
DEFAULT: 112 /4 MM

INFO: Copyright AH & CH

PART one
\ PDEFAULT: GOLD
PDEFAULT: ORGAN 3 TRANSPOSE-UP L7
||: 4 /4 MEASURE
L: m1
/4 C3 D3 E3 C3 | duplicate
/4 E3 F3 /2 G3 | duplicate
/8 G3 A3 G3 F3 /4 E3 C3 | duplicate
/4 C3 G2 /2 C3 |
L: m2 duplicate
m1 .. m2 .S bis .S
||

"

These files should be converted to one octave lower.
That means that everything should remain but
C3 must be converted to C2 and G2 to G1 etc.

I build in refilter into an enhanced forth,
the -s is inherited to forth itself.

"
#!/bin/refilter -s

1 ARG[] GET-FILE

"(<[A-Ga-g][eisEIS]*)2>" "\11" GLOBAL
"(<[A-Ga-g][eisEIS]*)3>" "\12" GLOBAL
"(<[A-Ga-g][eisEIS]*)4>" "\13" GLOBAL
"(<[A-Ga-g][eisEIS]*)5>" "\14" GLOBAL
"(<[A-Ga-g][eisEIS]*)6>" "\15" GLOBAL

2 ARG[] PUT-FILE
"

I can now proceed with

basso.fs vjaco.sco vjaco.sco.new
basso.fs landmann.sco landmann.sco.new
basso.fs landmann3.sco landmann3.sco.new
..
..
..

Post by Hans Bezemer
Hans Bezemer

Groetjes Albert

[Hans Bezemer]

Post by none) (albert
Suppose you are confronted with a number of files like
so
"
SCORE Vader-Jacob
DEFAULT: 112 /4 MM
INFO: Copyright AH & CH
PART one
\ PDEFAULT: GOLD
PDEFAULT: ORGAN 3 TRANSPOSE-UP L7
||: 4 /4 MEASURE
L: m1
/4 C3 D3 E3 C3 | duplicate
/4 E3 F3 /2 G3 | duplicate
/8 G3 A3 G3 F3 /4 E3 C3 | duplicate
/4 C3 G2 /2 C3 |
L: m2 duplicate
m1 .. m2 .S bis .S
||

I frankly have ZERO idea what I'm looking at - or what is expected of me.
I'm only a (light) consumer of music. Sports examples aren't much
good for me as well.

Got another one? Or at least show me what is expected to come out?

Hans Bezemer

[[e-mail]]

Post by Hans Bezemer

Post by none) (albert
Suppose you are confronted with a number of files like
so
"
SCORE Vader-Jacob
DEFAULT: 112 /4 MM
INFO: Copyright AH & CH
PART one
\ PDEFAULT: GOLD
PDEFAULT: ORGAN 3 TRANSPOSE-UP L7
||: 4 /4 MEASURE
L: m1
/4 C3 D3 E3 C3 | duplicate
/4 E3 F3 /2 G3 | duplicate
/8 G3 A3 G3 F3 /4 E3 C3 | duplicate
/4 C3 G2 /2 C3 |
L: m2 duplicate
m1 .. m2 .S bis .S
||

I frankly have ZERO idea what I'm looking at - or what is expected of me.
I'm only a (light) consumer of music. Sports examples aren't much
good for me as well.
Got another one? Or at least show me what is expected to come out?

[none) (albert]

Post by Hans Bezemer

Post by none) (albert
Suppose you are confronted with a number of files like
so
"
SCORE Vader-Jacob
DEFAULT: 112 /4 MM
INFO: Copyright AH & CH
PART one
\ PDEFAULT: GOLD
PDEFAULT: ORGAN 3 TRANSPOSE-UP L7
||: 4 /4 MEASURE
L: m1
/4 C3 D3 E3 C3 | duplicate
/4 E3 F3 /2 G3 | duplicate
/8 G3 A3 G3 F3 /4 E3 C3 | duplicate
/4 C3 G2 /2 C3 |
L: m2 duplicate
m1 .. m2 .S bis .S
||

I frankly have ZERO idea what I'm looking at - or what is expected of me.
I'm only a (light) consumer of music. Sports examples aren't much
good for me as well.

This is the score used for controlling instruments. The melody is
"frere jacques". C2 plays the C note of scale 2 , C3 play one octave
higher. In musical scores "basso" with a line indicating which notes
it refers to indicate that the note should be played on octave lower.
(I thought everybody knows that :-) )

Post by Hans Bezemer
Got another one? Or at least show me what is expected to come out?

This is what has to come out:
"
SCORE Vader-Jacob
DEFAULT: 112 /4 MM

INFO: Copyright AH & CH

PART one
\ PDEFAULT: GOLD
PDEFAULT: ORGAN 3 TRANSPOSE-UP L7
||: 4 /4 MEASURE
L: m1
/4 C2 D2 E2 C2 | duplicate
/4 E2 F2 /2 G2 | duplicate
/8 G2 A2 G2 F2 /4 E2 C2 | duplicate
/4 C2 G1 /2 C2 |
L: m2 duplicate
m1 .. m2 .S bis .S
||
"

There are dozen of score, and they have notes like
SCORE Vader-Jacob
DEFAULT: 112 /4 MM

INFO: Copyright AH & CH

PART one
\ PDEFAULT: GOLD
PDEFAULT: ORGAN 3 TRANSPOSE-UP L7
||: 4 /4 MEASURE
L: m1
/4 C2 D2 E2 C2 | duplicate
/4 E2 F2 /2 G2 | duplicate
/8 G2 A2 G2 F2 /4 E2 C2 | duplicate
/4 C2 G1 /2 C2 |
L: m2 duplicate
m1 .. m2 .S bis .S
||
"
Dozens of files have to be processed, so preferably
it must also recognize flat and sharp notes.
A2 ... F2 AIS2 .. GIS2 AS2 BES2 .. DES2 ES2 FES2 GES2
a2 ... f2 ais2 .. gis2 as2 bes2 .. des2 es2 fes2 ges2

Groetjes Albert

Post by Hans Bezemer
Hans Bezemer

Groetjes Albert

[Hans Bezemer]

On Sunday, January 15, 2023 at 11:51:45 AM UTC+1, none albert wrote:
- First of all, I'd use my "convert" template, which simply requires you to define several words - like how to read the file, how to process it, any preprocessing or post processing;
- I'd make it configurable by adding command line options. In 4tH, that's simply making an entry in a table;
- Third, I'd add a word that recognizes notes - starts with A-G followed by zero or more combinations of EIS and ends in a number;
- The "Process" word could be easy or hard. Hard way - punch out lines verbatim till L: is encountered, then switch to "notes" mode. Toggle the flag when L: is encountered again. Easy way: test each token for a "notes" notation. If so, process it - if not send it out verbatim;
- The "notes" word - same story: the hard way or the easy way. The hard way: split it up at the alphabet / number boundary (I got a word for that). Send out the alphabetic part verbatim, in- or decrement the number thing and push it out. The easy way: assume that only the last character is numerical.

I assume your script is doing it the easy way all the way.

E.g. this is a simple DOS -> UNIX textconverter in template form:

: Usage abort" Usage: duc infile outfile" ;
: Read-file pad 1 accept ;
: Process pad c@ dup 13 - if emit ;then drop ;
[needs lib/convert.4th]

Hans Bezemer

[Hans Bezemer]

On Sunday, January 15, 2023 at 3:47:26 PM UTC+1, Hans Bezemer wrote:
So - how hard would it be to recognize a note:

include lib/chmatch.4th

s" AaBbCcDdEeFfGg" sconstant 'notes'
s" 0123456789" sconstant 'digits'
s" EeIiSs" sconstant 'sharpflat'

: isnote?
'notes' char-match >r
'sharpflat' skip-while
'digits' char-match >r
2drop r> r> and
;

Exactly this hard. Let's test a few tokens:

: report 2dup type space isnote? . cr ;

/4 0
C3 1
D3 1
E3 1
F3 1
/2 0
G3 1
/8 0
A3 1
G2 1
GIS2 1
AS2 1
BES2 1

Now let's change it:

: fixit >r 2dup isnote? if 2dup 1- chars + dup c@ r> + swap c! ;then rdrop ;
s" BES2" 2dup type ." => " 2 fixit type cr
s" BLA5" 2dup type ." => " 3 fixit type cr

Result:
BES2 => BES4
BLA5 => BLA5

Oh boy - this really *was* rocket science..

Hans Bezemer

[Hans Bezemer]

On Sunday, January 15, 2023 at 4:22:49 PM UTC+1, Hans Bezemer wrote:
Given all "easy options" it was just a shame not to make a complete program of it:

include lib/chmatch.4th
include lib/parsname.4th
include lib/getopts.4th

0 value octave

s" AaBbCcDdEeFfGg" sconstant 'notes'
s" 0123456789" sconstant 'digits'
s" EeIiSs" sconstant 'sharpflat'

aka refill read-file

: isnote?
'notes' char-match >r
'sharpflat' skip-while
'digits' char-match >r
2drop r> r> and
;

: fixit >r 2dup isnote? if 2dup 1- chars + dup c@ r> + swap c! ;then rdrop ;
: Usage abort" Usage: notes -o<octave> infile outfile" ;
: Process begin parse-name dup while octave fixit type space repeat 2drop cr ;

: get-octave
get-argument number error? abort" Not a number" -9 max 9 min to octave
;

create options \ required by CONVERT.4TH
char o , ' get-octave ,
NULL ,

include lib/convert.4th

$ cat test.not
SCORE Vader-Jacob
DEFAULT: 112 /4 MM

INFO: Copyright AH & CH

PART one
\ PDEFAULT: GOLD
PDEFAULT: ORGAN 3 TRANSPOSE-UP L7
||: 4 /4 MEASURE
L: m1
/4 C2 D2 E2 C2 | duplicate
/4 E2 F2 /2 G2 | duplicate
/8 G2 A2 G2 F2 /4 E2 C2 | duplicate
/4 C2 G1 /2 C2 |
L: m2 duplicate
m1 .. m2 .S bis .S
||
$ cat res.not
SCORE Vader-Jacob
DEFAULT: 112 /4 MM

INFO: Copyright AH & CH

PART one
\ PDEFAULT: GOLD
PDEFAULT: ORGAN 3 TRANSPOSE-UP L7
||: 4 /4 MEASURE
L: m1
/4 C5 D5 E5 C5 | duplicate
/4 E5 F5 /2 G5 | duplicate
/8 G5 A5 G5 F5 /4 E5 C5 | duplicate
/4 C5 G4 /2 C5 |
L: m2 duplicate
m1 .. m2 .S bis .S
||

QED.

Hans Bezemer

[none) (albert]

<SNIP>

You have me convinced that you could do it easily with the tools you
have.

I'm equally convinced that I'm using less tools, and do it more easily,
on the premise that you need to have familiarity with the tools.
The decision to implement regular expression, on behalf of the score
conversion, was the right one, I think.

<SNIP>
That is -- date I say this -- a bit ugly. But in the original solution
that was more ugly, separating out the cases by hand.

<SNIP>

Post by Hans Bezemer
: Usage abort" Usage: notes -o<octave> infile outfile" ;

At least you generate a practical facility, probably usable by other persons.
Proof that you actually use Forth.

Post by Hans Bezemer
QED.
Hans Bezemer

Groetjes Albert

[Hans Bezemer]

Post by none) (albert
You have me convinced that you could do it easily with the tools you
have.

Thank you! From your mouth that actually means something. :)

Post by none) (albert
I'm equally convinced that I'm using less tools, and do it more easily,
on the premise that you need to have familiarity with the tools.
The decision to implement regular expression, on behalf of the score
conversion, was the right one, I think.

Well - look at it this way. All include files are actually inlined. So, when compiled
all code is there. And this is the report:

4tH message: No errors at word 494
Object size: 494 words
String size: 145 chars
Variables : 4 cells
Strings : 0 chars
Symbols : 0 names
Reliable : Yes

But I didn't need a huge regex machine to achieve it. Now - I'll be the first person
to agree your solution is actually shorter - with much less overhead. And maybe
even easier to read and maintain. Most likely faster to build. But it *does* require
a large blob to do it.

In effect, I think it boils down on what you find important (or handy) - and those
things can vary greatly from person to person. I think we can agree on that.

Post by none) (albert
That is -- date I say this -- a bit ugly. But in the original solution
that was more ugly, separating out the cases by hand.

I know it is. But - as I said - I've taken the simple route. All the way ;-)

Hans Bezemer

[Hans Bezemer]

Post by none) (albert
SCORE Vader-Jacob
DEFAULT: 112 /4 MM
INFO: Copyright AH & CH
PART one
\ PDEFAULT: GOLD
PDEFAULT: ORGAN 3 TRANSPOSE-UP L7
||: 4 /4 MEASURE
L: m1
/4 C3 D3 E3 C3 | duplicate
/4 E3 F3 /2 G3 | duplicate
/8 G3 A3 G3 F3 /4 E3 C3 | duplicate
/4 C3 G2 /2 C3 |
L: m2 duplicate
m1 .. m2 .S bis .S
||

Just to satisfy my curiosity, Albert - what IS this musical ASCII format? Is it a standard? Doesn't resemble
Lillypond or ABC..

Hans Bezemer

[none) (albert]

Post by Hans Bezemer

Post by none) (albert
SCORE Vader-Jacob
DEFAULT: 112 /4 MM
INFO: Copyright AH & CH
PART one
\ PDEFAULT: GOLD
PDEFAULT: ORGAN 3 TRANSPOSE-UP L7
||: 4 /4 MEASURE
L: m1
/4 C3 D3 E3 C3 | duplicate
/4 E3 F3 /2 G3 | duplicate
/8 G3 A3 G3 F3 /4 E3 C3 | duplicate
/4 C3 G2 /2 C3 |
L: m2 duplicate
m1 .. m2 .S bis .S
||

Just to satisfy my curiosity, Albert - what IS this musical ASCII
format? Is it a standard? Doesn't resemble
Lillypond or ABC..

That is a format invented by Dutch Forth to play the tingle tangle and
the organ. It went through numerous formats, and I collected them in 2002
and invent a new format, hence the conversions.
There is a demonstration on youtube that has been published here a
number of times. (hcc tingel tangel).

Post by Hans Bezemer
Hans Bezemer

[Hans Bezemer]

Post by none) (albert
That is a format invented by Dutch Forth to play the tingle tangle and
the organ. It went through numerous formats, and I collected them in 2002
and invent a new format, hence the conversions.
There is a demonstration on youtube that has been published here a
number of times. (hcc tingel tangel).

Thanks for that! I already wondered for which format I'd written a converter ;-)

Hans Bezemer

[S Jack]

"(<[A-Ga-g][eisEIS]*)2>" "\11" GLOBAL
"(<[A-Ga-g][eisEIS]*)3>" "\12" GLOBAL
"(<[A-Ga-g][eisEIS]*)4>" "\13" GLOBAL
"(<[A-Ga-g][eisEIS]*)5>" "\14" GLOBAL
"(<[A-Ga-g][eisEIS]*)6>" "\15" GLOBAL

Just a FYI should someone missed what's going on:

:) cat job

SCORE Vader-Jacob
DEFAULT: 112 /4 MM

INFO: Copyright AH & CH

PART one
\ PDEFAULT: GOLD
PDEFAULT: ORGAN 3 TRANSPOSE-UP L7
||: 4 /4 MEASURE
L: m1
/4 C3 D3 E3 C3 | duplicate
/4 E3 F3 /2 G3 | duplicate
/8 G3 A3 G3 F3 /4 E3 C3 | duplicate
/4 C3 G2 /2 C3 |
L: m2 duplicate
m1 .. m2 .S bis .S
||
:) cat koo
sed -e '
s/\([A-Ga-g][eisEIS]*\)2/\11/g
s/\([A-Ga-g][eisEIS]*\)3/\12/g
s/\([A-Ga-g][eisEIS]*\)4/\13/g
s/\([A-Ga-g][eisEIS]*\)5/\14/g
s/\([A-Ga-g][eisEIS]*\)6/\15/g' job

:) ./koo

SCORE Vader-Jacob
DEFAULT: 112 /4 MM

INFO: Copyright AH & CH

PART one
\ PDEFAULT: GOLD
PDEFAULT: ORGAN 3 TRANSPOSE-UP L7
||: 4 /4 MEASURE
L: m1
/4 C2 D2 E2 C2 | duplicate
/4 E2 F2 /2 G2 | duplicate
/8 G2 A2 G2 F2 /4 E2 C2 | duplicate
/4 C2 G1 /2 C2 |
L: m2 duplicate
m1 .. m2 .S bis .S
||
:)
--
me

[S Jack]

"(<[A-Ga-g][eisEIS]*)2>" "\11" GLOBAL
"(<[A-Ga-g][eisEIS]*)3>" "\12" GLOBAL
"(<[A-Ga-g][eisEIS]*)4>" "\13" GLOBAL
"(<[A-Ga-g][eisEIS]*)5>" "\14" GLOBAL
"(<[A-Ga-g][eisEIS]*)6>" "\15" GLOBAL

This would have been clearer:
:) cat job

SCORE Vader-Jacob
DEFAULT: 112 /4 MM

INFO: Copyright AH & CH

PART one
\ PDEFAULT: GOLD
PDEFAULT: ORGAN 3 TRANSPOSE-UP L7
||: 4 /4 MEASURE
L: m1
/4 C3 D3 E3 C3 | duplicate
/4 E3 F3 /2 G3 | duplicate
/8 G3 A3 G3 F3 /4 E3 C3 | duplicate
/4 C3 G2 /2 C3 |
L: m2 duplicate
m1 .. m2 .S bis .S
||
:) cat koo.sed
s/\([A-Ga-g][eisEIS]*\)2/\11/g
s/\([A-Ga-g][eisEIS]*\)3/\12/g
s/\([A-Ga-g][eisEIS]*\)4/\13/g
s/\([A-Ga-g][eisEIS]*\)5/\14/g
s/\([A-Ga-g][eisEIS]*\)6/\15/g

:) sed -f koo.sed job

SCORE Vader-Jacob
DEFAULT: 112 /4 MM

INFO: Copyright AH & CH

PART one
\ PDEFAULT: GOLD
PDEFAULT: ORGAN 3 TRANSPOSE-UP L7
||: 4 /4 MEASURE
L: m1
/4 C2 D2 E2 C2 | duplicate
/4 E2 F2 /2 G2 | duplicate
/8 G2 A2 G2 F2 /4 E2 C2 | duplicate
/4 C2 G1 /2 C2 |
L: m2 duplicate
m1 .. m2 .S bis .S
||
:)
--
me

[none) (albert]

Post by Paul Rubin

Regular Expression packages. About 10 years ago I wrote a
comprehensive Regex package in ANS Forth (yes it can be done) that
works on several Forths including GForth, VFX Forth, SwiftForth, WF32
and my own system and made it available on a website and there was
virtually no interest.

Well I didn't know about it. I think it is good to have such a thing,
for those who do any type of text processing in Forth. I can't think of
any times where I'd have used it myself though.

Have you never undertaken such conversion before?
Do you think the example of converting a ARM 64 assembler source to
Risc-V assembler source is not convincing?
The package has been applied to conversions chapter 5 of "thinking
Forth" from xml to tech.
I have exterminate idiosyncrasies from Marcel Hendrix source.
An further example could be transforming old K&R c-code to ANSI c.

Examples abound. I think you merely missed the opportunity to
use such a package.

Groetjes

[Marcel Hendrix]

Regular Expression packages. About 10 years ago I wrote a
comprehensive Regex package in ANS Forth (yes it can be done) that
works on several Forths including GForth, VFX Forth, SwiftForth, WF32
and my own system and made it available on a website and there was
virtually no interest.

Well I didn't know about it. I think it is good to have such a thing,
for those who do any type of text processing in Forth. I can't think of
any times where I'd have used it myself though.

I think I will have a look, for inspiration. I still have a problem in the to-do
queue that is a little bit too complicated for a quick hack. It involves a
comma separated list where the items themselves can be yet other
comma separated lists (like functional calls of a conventional language).

-marcel

[S Jack]

Post by Gerry Jackson

Post by none) (albert
1. String denotations.

Why, I used them without any problems

Post by none) (albert
the S leads to confusion

I guessing he means applying regex to Forth words (not building regex)
is a pain because Forth words contain pu
nctuation which require much
escaping when doing regex on them. One of the main reason I built a
symbolic Forth, to look like assembly, to have short words with no
punctuation. Easy as text to deal with by Regex and don't effect
communication controls. (But I did compromise and used math symbols
e.g. "+" and not "plus").

--
me

[Hans Bezemer]

Post by none) (albert
There is an effort underway to collect valuable snippets of
Forth, https://github.com/project-forth-works/
I have regular expression package that I used (i.a.) to transform
the fifth chapter of "thinking forth" from xml to tex and the
manx music score that went through half a dozen different notations.
I have recently used it to convert ARM assembler
language to risc-V language. Regular expression without the
facility to transform texts is a joke.
The program doesn't comply to ISO, so I thought an elaborate
rationale is in order. If you consider the rationale unconvincing,
I will not bother to upload the package.
"
Why ciforthisms?
In short, I could never manage to write the regular expression
package with ciforthism.
Note that to have recently used it to convert ARM assembler
language to risc-V language. Regular expression without the
facility to transform texts is a joke.
****************** STRINGS **********************
A regular expression package handles strings. Any attempt to handle
strings portably in Forth sucks.
1. String denotations.
the S leads to confusion
Much less visible is the problem that S" AAP" is 3 characters
and S" AAP" is 4, I guess?. In ciforth there is a normal string
"AAP" is three chars
" AAP" is four chars
" AAP" is five chars
2.
They conflict with the escapes needed in this package.
There is one exception, the double quote convention.
In order to have a double quote &" in a string you have
to double it like so `` "" ''.
In ciforth you can have strings that span multiple lines,
albeit only in a program.
3. Character denotations, function denotations.
We use &A and for a good reason.
CHAR [CHAR] are verbose and cumbersome.
Moreover the notation 'x' for the character x overloads the tick
too much, because in ciforth ' is reserved to denote dea's.
Functions in a regular expression are passed around as executions tokens
like so 'ADVANCE+ , using a single tick, without square brackets
around it.
POSTPONE 'ADVANCE+
fails because "'ADVANCE+" is not in the dictionary, which is just
as well. ( "ADVANCE+" is present, of course).
4. Low level abstraction fails.
COUNT -TRAILING S" C" WORD PLACE form a non-unified conglomerate
of handling strings.
Get familiar with these. They are easy, because they use
the underlying buffers explicitly. Any half-assed attempt to
use string-stacks and more poorly conceived abstractions
adds to the problem not the solution.
************************** The library *************************
I use WANT. All words following are supposed to be available.
You can load them, or give an error are they not present.
The library contains tools, notably the SET and BAG tools.
It is not good to have them as part of the package.
First get familiar to the tools, then attempt to read the
package. You have lifted yourself to a higher plane.
That is needed for regular expressions, they are not easy.
Anytime you encounter WANT you have to get familiar to the
tools wanted. In ciforth they are mostly Forth words that
are present in fat Forth's, so there in lies no problem.
****************** USABILITY ******************************
Using Forth programs sucks. Mostly you must have the particular
Forth installed in your computer. Then the invocations leave
much to be desired.
The best one come up with is
<forth> <somesource>
but the question is "where do the parameters go
-- in particular filenames -- ? "
ciforth solves this in four ways, each of those are
preferable in some situations.
1. interactive, you start forth then you load your program
2. scripting, the forth is in the first line of the script
Parameters are passed naturally to the script.
The forth must load the regular expression module first
3. scripting, the regular expressions library is compiled into
an enhanced forth. In the script you specify the the particular
expressions you want changed.
Parameters are passed naturally to the script.
Note that you don't need a Forth installed. The scripting
program is enough. You can send it as an attachment and
it will run on the same Operating System.
4. Compiled. Now the transformations you want to do are
burned into the executable.
Parameters are passed naturally to the script.
Note that you don't need a Forth installed. The scripting
program is enough. You can send it as an attachment and
it will run on the same Operating System.
This sounds a lot of hype. Therefore I have supplied a
simple practical examples, in several circumstances and operating
systems.
"
Groetjes Albert
--
Don't praise the day before the evening. One swallow doesn't make
spring. You must not say "hey" before you have crossed the bridge.
Don't sell the hide of the bear until you shot it. Better one bird in
the hand than ten in the air. - the Wise from Antrim -

First, you know me to be quite frank - but that doesn' mean disrespect.
I'm just being straight forward.
(A) I once had a look at your package, even began converting it, but the
CIforthisms kept me from it. It was just too much hassle.
(B) Yours can do replacement - mine just matching. But I found I even
used my own regex package very seldom. I tend to use a package that does
complex matching as well - but the Forth way - not needing a regex
interpreter.
(C) Some forths claim turnkey functionality - but I never got it to work.
Hence 4tH - just link in the runtime and you're good. Works as if written
in C.
(D) S" doesn't bother me anymore. I'm quite used to it. Doesn't mean I
find it a thing of beauty.
(E) I have no problem with [CHAR] - still, not a thing of beauty. If you want 'a'
[CHAR] a CONSTANT 'a'. Sorry - no uppercase. UPPER helps.
(F) S\" is a thing I think is ugly for a change. Can do it, but only when porting.
(G) I find string stacks extremely useful. I have at least three different
implementations - from heavy duty to very light weight. Essential part of my
preprocessor.

In short - I don't have much of a taste for regexes. Certainly not a heavy one.
I won't deny in the hands of some people, it can be useful - but it's not my way
of approaching such problems (and I've done some conversions). It all feels
bloated and bulky.

My two cents.

Hans Bezemer

[none) (albert]

Post by none) (albert
There is an effort underway to collect valuable snippets of
Forth, https://github.com/project-forth-works/
I have regular expression package that I used (i.a.) to transform
the fifth chapter of "thinking forth" from xml to tex and the
manx music score that went through half a dozen different notations.
I have recently used it to convert ARM assembler
language to risc-V language. Regular expression without the
facility to transform texts is a joke.

Searching
"in forth" regular expression
in google I found
https://home.hccnet.nl/a.w.m.van.der.horst/forthlectureA.html

Apparently I had it published on my web site 2003 oct.

Groetjes Albert
I want to add "eerste gewin is kattegespin" to the words of
the Wise of Antrim, but then the signature is too long.