Multiple Readfile Oddities, Newlines Etc

Home » Asterisk Users » Multiple Readfile Oddities, Newlines Etc
Asterisk Users 5 Comments

I have a plain text file, ASCII, unix line breaks. 1 single line, and all that is in it is the word “radio”.

Here’s some test dialplan:

exten => 5,1,Verbose(Context: ${CONTEXT} Exten:${EXTEN})

same => n,Set(feature=${FILE(/home/test/feature-1.txt,0,1,l,u)})
same => n,Verbose(${feature})

same => n,Set(featurefile=/home/test/feature-1.txt)
same => n,Set(feature2=${FILE(${featurefile},0,1,l,u)})
same => n,Verbose(${feature2})

Both should output “radio”, right? Here’s the output:

— Executing [5@fromvoipfone201:2] Set(“PJSIP/6001-00000052”,
“feature=radio”) in new stack
— Executing [5@fromvoipfone201:3] Verbose(“PJSIP/6001-00000052”,
“radio”) in new stack radio

— Executing [5@fromvoipfone201:4] Set(“PJSIP/6001-00000052”,
“featurefile=/home/test/feature-1.txt”) in new stack
— Executing [5@fromvoipfone201:5] Set(“PJSIP/6001-00000052”, “feature2=
*radi*”) in new stack
*GAAAAARRRGGGHHHH!
^^^^^*

— Executing [5@fromvoipfone201:6] Verbose(“PJSIP/6001-00000052”,
“radi”) in new stack

And this is what’s just at the top of the script. If I put it way down in another context:

— Executing [s@track-handler:3] Verbose(“Local/s@root-00000026;2”,
“/home/test/feature-1”) in new stack
/home/test/feature-1
— Executing [s@track-handler:4] Set(“Local/s@root-00000026;2”,
“feature=*radio▒▒*”) in new stack
*EVEN MORE**
GAAAAARRRGGGHHHH!
^^^^^*

[Oct 17 13:29:33] ERROR[5093][C-0000003c]: json.c:704 ast_json_vpack: Error building JSON from ‘{s: s, s: s}’: Invalid UTF-8 string.
[Oct 17 13:29:33] ERROR[5093][C-0000003c]: stasis_channels.c:773
ast_channel_publish_varset: Error creating message
— Executing [s@track-handler:5] Verbose(“Local/s@root-00000026;2”,
“radio▒▒”) in new stack radio▒▒
— Executing [s@track-handler:6] GotoIf(“Local/s@root-00000026;2”,
“1?radio▒▒,s,1”) in new stack

So, at the top of the file, when it’s just a straight file as the filename, it gives the full word. If the filename is a variable, it strips and character. And further down, same thing, but it adds two weird blocks, which appear side by side in the console, but weirdly, stacked up in this email.

(Oh, by the way, if there is more than one line in the file, even if I used
“u” for line breaks, it adds a newline to the variable, Is that correct?)

What am I missing? I’ve opened it in both nano and notepad++, I’ve used iconv and all the tools I can think of to check that file, and all the asterisk conf files, too. It all looks as it should here:

$ cat feature-1.txt radio$ file feature-1.txt feature-1.txt: ASCII text
$ wc -l feature-1.txt
0 feature-1.txt wc -c feature-1.txt
5 feature-1.txt

After 6 hours struggling with this, I think I’m starting to lose the plot. Can anyone tell me where I’m going wrong? Thanks.

5 thoughts on - Multiple Readfile Oddities, Newlines Etc

  • This is a multipart message in MIME format.

    ——=_NextPart_000_001D_01D2288E.3FB59EA0
    Content-Type: text/plain;
    charset=”UTF-8″
    Content-Transfer-Encoding: quoted-printable

    Theres always garbage in the end of the files.

    I do this when I want to read a file:

    same => n,Set(featurefile=/home/test/feature-1.txt)
    same => n,Set(unfilteredfeat2=${FILE(${featurefile},0,1,l,u)})

    same => n,Set(feature2=${SHIFT(unfilteredfeat2)})

    After that, add a , inside end of the file, so

    Cat feature-1.txt

    Reads:

    radio,

    Thus if there is garbage in the file, it will happen after the ,. Same if garbage happens to find its way into the end of the variable for some reason.

    Från: asterisk-users-bounces@lists.digium.com [mailto:asterisk-users-bounces@lists.digium.com] För Jonathan H
    Skickat: den 17 oktober 2016 15:32
    Till: Asterisk Users Mailing List – Non-Commercial Discussion
    Ämne: [asterisk-users] Multiple readfile oddities, newlines etc

    I have a plain text file, ASCII, unix line breaks. 1 single line, and all that is in it is the word “radio”.

    Here’s some test dialplan:

    exten => 5,1,Verbose(Context: ${CONTEXT} Exten:${EXTEN})

    same => n,Set(feature=${FILE(/home/test/feature-1.txt,0,1,l,u)})
    same => n,Verbose(${feature})

    same => n,Set(featurefile=/home/test/feature-1.txt)
    same => n,Set(feature2=${FILE(${featurefile},0,1,l,u)})
    same => n,Verbose(${feature2})

    Both should output “radio”, right? Here’s the output:

    — Executing [5@fromvoipfone201:2] Set(“PJSIP/6001-00000052”, “feature=radio”) in new stack
    — Executing [5@fromvoipfone201:3] Verbose(“PJSIP/6001-00000052”, “radio”) in new stack radio

    — Executing [5@fromvoipfone201:4] Set(“PJSIP/6001-00000052”, “featurefile=/home/test/feature-1.txt”) in new stack
    — Executing [5@fromvoipfone201:5] Set(“PJSIP/6001-00000052”, “feature2=radi”) in new stack

    GAAAAARRRGGGHHHH! ^^^^^

    — Executing [5@fromvoipfone201:6] Verbose(“PJSIP/6001-00000052”, “radi”) in new stack

    And this is what’s just at the top of the script. If I put it way down in another context:

    — Executing [s@track-handler:3] Verbose(“Local/s@root-00000026;2”, “/home/test/feature-1”) in new stack
    /home/test/feature-1
    — Executing [s@track-handler:4] Set(“Local/s@root-00000026;2”, “feature=radio▒▒”) in new stack

    EVEN MORE GAAAAARRRGGGHHHH! ^^^^^

    [Oct 17 13:29:33] ERROR[5093][C-0000003c]: json.c:704 ast_json_vpack: Error building JSON from ‘{s: s, s: s}’: Invalid UTF-8 string.
    [Oct 17 13:29:33] ERROR[5093][C-0000003c]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message
    — Executing [s@track-handler:5] Verbose(“Local/s@root-00000026;2”, “radio▒▒”) in new stack radio▒▒
    — Executing [s@track-handler:6] GotoIf(“Local/s@root-00000026;2”, “1?radio▒▒,s,1”) in new stack

    So, at the top of the file, when it’s just a straight file as the filename, it gives the full word. If the filename is a variable, it strips and character. And further down, same thing, but it adds two weird blocks, which appear side by side in the console, but weirdly, stacked up in this email.

    (Oh, by the way, if there is more than one line in the file, even if I used “u” for line breaks, it adds a newline to the variable, Is that correct?)

    What am I missing? I’ve opened it in both nano and notepad++, I’ve used iconv and all the tools I can think of to check that file, and all the asterisk conf files, too. It all looks as it should here:

    $ cat feature-1.txt radio$ file feature-1.txt feature-1.txt: ASCII text
    $ wc -l feature-1.txt
    0 feature-1.txt wc -c feature-1.txt
    5 feature-1.txt

    After 6 hours struggling with this, I think I’m starting to lose the plot. Can anyone tell me where I’m going wrong? Thanks.

    ——=_NextPart_000_001D_01D2288E.3FB59EA0
    Content-Type: text/html;
    charset=”UTF-8″
    Content-Transfer-Encoding: quoted-printable

  • Just by chance, I was browsing the mailing list and it looks like Sebastian’s reply made it to the list, but not to me because it’s apparently blacklisted by UCEPROTECTL3.

    This is what he wrote:

    Well, OK, that sort of works, but it revealed even more odd things. And, if tools like wc are saying that a file is a certain length, how can there be junk hiding in it?

    So, then I found that having different lengths in the file messed things up, too. Can anyone tell me this is normal?!?

    Have a look, this is for changing the following words in the features.txt. In all of the examples, the LEN function gets it wrong, too.

    radio < gets cut a letter short even with SHIFT trimmer radio, < works, but only by chance because , makes it 6 characters radui12345 < garbage at end radio12345, < shift trimmer removes garbage rad < junk at end Context: fromvoipfone201 Exten:5 Using a string, feature is set to ---radio--- and is 7 long Using a variable, feature2 is set to ---radi--- and is 8 long Using a string with shift method, feature3 is set to ---radi--- and is 8 long Context: fromvoipfone201 Exten:5 Using a string, feature is set to ---radio,--- and is 7 long Using a variable, feature2 is set to ---radio,--- and is 8 long Using a string with shift method, feature3 is set to ---radio--- and is 8 long Context: fromvoipfone201 Exten:5 Using a string, feature is set to ---radio12345--- and is 7 long [Oct 17 15:30:29] ERROR[5813][C-00000073]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:29] ERROR[5813][C-00000073]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message Using a variable, feature2 is set to ---radio12345G▒--- and is 8 long [Oct 17 15:30:29] ERROR[5813][C-00000073]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:29] ERROR[5813][C-00000073]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message [Oct 17 15:30:29] ERROR[5813][C-00000073]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:29] ERROR[5813][C-00000073]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message Using a string with shift method, feature3 is set to ---radio12345G▒--- and is 8 long Context: fromvoipfone201 Exten:5 Using a string, feature is set to ---radio12345,--- and is 7 long [Oct 17 15:30:35] ERROR[5814][C-00000074]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:35] ERROR[5814][C-00000074]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message Using a variable, feature2 is set to ---radio12345,▒--- and is 8 long [Oct 17 15:30:35] ERROR[5814][C-00000074]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:35] ERROR[5814][C-00000074]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message [Oct 17 15:30:35] ERROR[5814][C-00000074]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:35] ERROR[5814][C-00000074]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message Using a string with shift method, feature3 is set to ---radio12345--- and is 8 long Context: fromvoipfone201 Exten:5 Using a string, feature is set to ---rad--- and is 7 long [Oct 17 15:30:43] ERROR[5815][C-00000075]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:43] ERROR[5815][C-00000075]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message Using a variable, feature2 is set to ---rad▒--- and is 8 long [Oct 17 15:30:43] ERROR[5815][C-00000075]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:43] ERROR[5815][C-00000075]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message [Oct 17 15:30:43] ERROR[5815][C-00000075]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:43] ERROR[5815][C-00000075]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message Using a string with shift method, feature3 is set to ---rad▒--- and is 8 long So, using the SHIFT trimmer cuts the junk, but still loses the last character in some cases. Here's the dialplan same => n,Set(feature=${FILE(/home/test/feature-1.txt,0,1,l,u)})
    same => n,Verbose(1,Using a string, feature is set to
    —${feature}— and is ${LEN(feature)} long)

    same => n,Set(featurefile=/home/test/feature-1.txt)
    same => n,Set(feature2=${FILE(${featurefile},0,1,l,u)})
    same => n,Verbose(1,Using a variable, feature2 is set to
    —${feature2}— and is ${LEN(feature2)} long)

    same => n,Set(unfilteredfeat=${FILE(${featurefile},0,1,l,u)})
    same => n,Set(feature3=${SHIFT(unfilteredfeat)})
    same => n,Verbose(1,Using a string with shift method, feature3 is set to —${feature3}— and is ${LEN(feature3)} long)

    Bug or… “feature”?!

  • –Apple-Mail=_2FE027E4-8D7D-4175-92CB-D1F3F106FB1D
    Content-Transfer-Encoding: quoted-printable Content-Type: text/plain;
    charset=us-ascii

    Heya Jonathan

    Interesting problem!

    Unfortunately I can’t help with suitable dialplan code to resolve your issue, but I thought I’d at least throw this snippet out to you:

    If you want to know what is _really_ in that file (including all invisible characters and anything else that wc etc might not count), pipe it through ‘hexdump’. Then you get to see the raw contents of the file in hex and can compare it to other files or the output you get from other tools (and you can look up the ASCII char codes for invisible chars to explore what they are etc).

    Eg:

    cat /home/test/feature-1.txt | hexdump

    Hope this tool helps you in your quest 🙂

    Pete

    –Apple-Mail=_2FE027E4-8D7D-4175-92CB-D1F3F106FB1D
    Content-Transfer-Encoding: quoted-printable Content-Type: text/html;
    charset=us-ascii

    On 18/10/2016, at 2:31 am, Jonathan H <lardconcepts@gmail.com> wrote:


    I have a plain text file, ASCII, unix line breaks. 1 single line, and all that is in it is the word “radio”.


    <snip>


    Heya Jonathan


    Interesting problem!

    Unfortunately I can’t help with suitable dialplan code to resolve your issue, but I thought I’d at least throw this snippet out to you:

    If you want to know what is _really_ in that file (including all invisible characters and anything else that wc etc might not count), pipe it through ‘hexdump’. Then you get to see the raw contents of the file in hex and can compare it to other files or the output you get from other tools (and you can look up the ASCII char codes for invisible chars to explore what they are etc).

    Eg:

    cat /home/test/feature-1.txt | hexdump

    Hope this tool helps you in your quest 🙂

    Pete




    –Apple-Mail=_2FE027E4-8D7D-4175-92CB-D1F3F106FB1D

  • –Apple-Mail=_DBF23CE7-2ABB-46DB-88D2-08FFC94FCBB4
    Content-Transfer-Encoding: quoted-printable Content-Type: text/plain;
    charset=us-ascii

    Heh.. yes, fair call 😉

    Pete

    –Apple-Mail=_DBF23CE7-2ABB-46DB-88D2-08FFC94FCBB4
    Content-Transfer-Encoding: quoted-printable Content-Type: text/html;
    charset=us-ascii

    On 18/10/2016, at 10:38 am, Steve Edwards <asterisk.org@sedwards.com> wrote:

    cat /home/test/feature-1.txt | hexdump


    Or just:

    hexdump /home/test/feature-1.txt


    Heh.. yes, fair call 😉


    Pete


    –Apple-Mail=_DBF23CE7-2ABB-46DB-88D2-08FFC94FCBB4

  • I’m going to go ahead and file a bug report, ‘cos something definitely ain’t right here! Bug filed:
    https://issues.asterisk.org/jira/browse/ASTERISK-26481

    This bit of dialplan…..

    exten => 5,1,Verbose(Context: ${CONTEXT} Exten:${EXTEN})
    same => n,Set(featurefile=/home/test/feature-1.txt)
    same => n,set(myList=1,12,123,1234,12345,123456,A,AB,ABC,ABCD,ABCDE,ABC123)
    same => n,While($[“${SET(rawVar=${SHIFT(myList)})}” != “”])
    same => n,SET(FILE(${featurefile})=${rawVar})
    same => n,SET(fileVar=${FILE(${featurefile},0,1,l,u)})
    same => n,SET(hexDump=${SHELL(hd ${featurefile})})
    same => n,Verbose(Before file: —${rawVar}— len ${LEN(rawVar)}
    after file: —${fileVar}— len ${LEN(fileVar)} hexdump:${hexDump})
    same => n,EndWhile()
    same => n,Hangup()

    …produces this output: (tidied only for posting) – this is messed up in so many ways.

    LEN is showing the wrong length.
    1 character gets a weird character added.
    23, and 4 characters OK
    5 characters loses a character – BUT LOOK! Even though hexdump prints the correct contents and length, have a close look at this: 35|12345|
    whereas normally it would have a space before the pipe.
    6 characters is OK

    Before file: —1— len 6 after file: —1▒— len 7 hexdump:00000000
    31|1| 00000001
    Before file: —12— len 6 after file: —12— len 7
    hexdump:00000000 31 32 |12| 00000002
    Before file: —123— len 6 after file: —123— len 7
    hexdump:00000000 31 32 33 |123| 00000003

    [Oct 18 12:23:53] ERROR[8584][C-0000001a]: json.c:704 ast_json_vpack:
    Error building JSON from ‘{s: s, s: s}’: Invalid UTF-8 string.
    [Oct 18 12:23:53] ERROR[8584][C-0000001a]: stasis_channels.c:773
    ast_channel_publish_varset: Error creating message

    Before file: —1234— len 6 after file: —1234— len 7
    hexdump:00000000 31 32 33 34 |1234| 00000004
    Before file: —12345— len 6 after file: —1234— len 7
    hexdump:00000000 31 32 33 34 35|12345| 00000005
    Before file: —123456— len 6 after file: —123456— len 7
    hexdump:00000000 31 32 33 34 35 36 |123456| 00000006

    [Oct 18 12:23:53] ERROR[8584][C-0000001a]: json.c:704 ast_json_vpack:
    Error building JSON from ‘{s: s, s: s}’: Invalid UTF-8 string.
    [Oct 18 12:23:53] ERROR[8584][C-0000001a]: stasis_channels.c:773
    ast_channel_publish_varset: Error creating message

    Before file: —A— len 6 after file: —A▒— len 7 hexdump:00000000
    41|A| 00000001
    Before file: —AB— len 6 after file: —AB— len 7
    hexdump:00000000 41 42 |AB| 00000002
    Before file: —ABC— len 6 after file: —ABC— len 7
    hexdump:00000000 41 42 43 |ABC| 00000003

    [Oct 18 12:23:53] ERROR[8584][C-0000001a]: json.c:704 ast_json_vpack:
    Error building JSON from ‘{s: s, s: s}’: Invalid UTF-8 string.
    [Oct 18 12:23:53] ERROR[8584][C-0000001a]: stasis_channels.c:773
    ast_channel_publish_varset: Error creating message

    Before file: —ABCD— len 6 after file: —ABCD— len 7
    hexdump:00000000 41 42 43 44 |ABCD| 00000004
    Before file: —ABCDE— len 6 after file: —ABCD— len 7
    hexdump:00000000 41 42 43 44 45|ABCDE| 00000005
    Before file: —ABC123— len 6 after file: —ABC123— len 7
    hexdump:00000000 41 42 43 31 32 33 |ABC123|