6 thoughts on - Multiple Readfile Oddities, Newlines Etc

• This is a multipart message in MIME format.

  ——=_NextPart_000_001D_01D2288E.3FB59EA0
  Content-Type: text/plain;
  charset=”UTF-8″
  Content-Transfer-Encoding: quoted-printable

  Theres always garbage in the end of the files.

  I do this when I want to read a file:

  same => n,Set(featurefile=/home/test/feature-1.txt)
  same => n,Set(unfilteredfeat2=${FILE(${featurefile},0,1,l,u)})

  same => n,Set(feature2=${SHIFT(unfilteredfeat2)})

  After that, add a , inside end of the file, so

  Cat feature-1.txt

  Reads:

  radio,

  Thus if there is garbage in the file, it will happen after the ,. Same if garbage happens to find its way into the end of the variable for some reason.

  Från: asterisk-users-bounces@lists.digium.com [mailto:asterisk-users-bounces@lists.digium.com] För Jonathan H
  Skickat: den 17 oktober 2016 15:32
  Till: Asterisk Users Mailing List – Non-Commercial Discussion
  Ämne: [asterisk-users] Multiple readfile oddities, newlines etc

  I have a plain text file, ASCII, unix line breaks. 1 single line, and all that is in it is the word “radio”.

  Here’s some test dialplan:

  exten => 5,1,Verbose(Context: ${CONTEXT} Exten:${EXTEN})

  same => n,Set(feature=${FILE(/home/test/feature-1.txt,0,1,l,u)})
  same => n,Verbose(${feature})

  same => n,Set(featurefile=/home/test/feature-1.txt)
  same => n,Set(feature2=${FILE(${featurefile},0,1,l,u)})
  same => n,Verbose(${feature2})

  Both should output “radio”, right? Here’s the output:

  — Executing [5@fromvoipfone201:2] Set(“PJSIP/6001-00000052”, “feature=radio”) in new stack
  — Executing [5@fromvoipfone201:3] Verbose(“PJSIP/6001-00000052”, “radio”) in new stack radio

  — Executing [5@fromvoipfone201:4] Set(“PJSIP/6001-00000052”, “featurefile=/home/test/feature-1.txt”) in new stack
  — Executing [5@fromvoipfone201:5] Set(“PJSIP/6001-00000052”, “feature2=radi”) in new stack

  GAAAAARRRGGGHHHH! ^^^^^

  — Executing [5@fromvoipfone201:6] Verbose(“PJSIP/6001-00000052”, “radi”) in new stack

  And this is what’s just at the top of the script. If I put it way down in another context:

  — Executing [s@track-handler:3] Verbose(“Local/s@root-00000026;2”, “/home/test/feature-1”) in new stack
  /home/test/feature-1
  — Executing [s@track-handler:4] Set(“Local/s@root-00000026;2”, “feature=radio▒▒”) in new stack

  EVEN MORE GAAAAARRRGGGHHHH! ^^^^^

  [Oct 17 13:29:33] ERROR[5093][C-0000003c]: json.c:704 ast_json_vpack: Error building JSON from ‘{s: s, s: s}’: Invalid UTF-8 string.
  [Oct 17 13:29:33] ERROR[5093][C-0000003c]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message
  — Executing [s@track-handler:5] Verbose(“Local/s@root-00000026;2”, “radio▒▒”) in new stack radio▒▒
  — Executing [s@track-handler:6] GotoIf(“Local/s@root-00000026;2”, “1?radio▒▒,s,1”) in new stack

  So, at the top of the file, when it’s just a straight file as the filename, it gives the full word. If the filename is a variable, it strips and character. And further down, same thing, but it adds two weird blocks, which appear side by side in the console, but weirdly, stacked up in this email.

  (Oh, by the way, if there is more than one line in the file, even if I used “u” for line breaks, it adds a newline to the variable, Is that correct?)

  What am I missing? I’ve opened it in both nano and notepad++, I’ve used iconv and all the tools I can think of to check that file, and all the asterisk conf files, too. It all looks as it should here:

  $ cat feature-1.txt radio$ file feature-1.txt feature-1.txt: ASCII text
  $ wc -l feature-1.txt
  0 feature-1.txt wc -c feature-1.txt
  5 feature-1.txt

  After 6 hours struggling with this, I think I’m starting to lose the plot. Can anyone tell me where I’m going wrong? Thanks.

  ——=_NextPart_000_001D_01D2288E.3FB59EA0
  Content-Type: text/html;
  charset=”UTF-8″
  Content-Transfer-Encoding: quoted-printable

• Just by chance, I was browsing the mailing list and it looks like Sebastian’s reply made it to the list, but not to me because it’s apparently blacklisted by UCEPROTECTL3.

  This is what he wrote:

  Well, OK, that sort of works, but it revealed even more odd things. And, if tools like wc are saying that a file is a certain length, how can there be junk hiding in it?

  So, then I found that having different lengths in the file messed things up, too. Can anyone tell me this is normal?!?

  Have a look, this is for changing the following words in the features.txt. In all of the examples, the LEN function gets it wrong, too.

  radio < gets cut a letter short even with SHIFT trimmer radio, < works, but only by chance because , makes it 6 characters radui12345 < garbage at end radio12345, < shift trimmer removes garbage rad < junk at end Context: fromvoipfone201 Exten:5 Using a string, feature is set to ---radio--- and is 7 long Using a variable, feature2 is set to ---radi--- and is 8 long Using a string with shift method, feature3 is set to ---radi--- and is 8 long Context: fromvoipfone201 Exten:5 Using a string, feature is set to ---radio,--- and is 7 long Using a variable, feature2 is set to ---radio,--- and is 8 long Using a string with shift method, feature3 is set to ---radio--- and is 8 long Context: fromvoipfone201 Exten:5 Using a string, feature is set to ---radio12345--- and is 7 long [Oct 17 15:30:29] ERROR[5813][C-00000073]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:29] ERROR[5813][C-00000073]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message Using a variable, feature2 is set to ---radio12345G▒--- and is 8 long [Oct 17 15:30:29] ERROR[5813][C-00000073]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:29] ERROR[5813][C-00000073]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message [Oct 17 15:30:29] ERROR[5813][C-00000073]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:29] ERROR[5813][C-00000073]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message Using a string with shift method, feature3 is set to ---radio12345G▒--- and is 8 long Context: fromvoipfone201 Exten:5 Using a string, feature is set to ---radio12345,--- and is 7 long [Oct 17 15:30:35] ERROR[5814][C-00000074]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:35] ERROR[5814][C-00000074]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message Using a variable, feature2 is set to ---radio12345,▒--- and is 8 long [Oct 17 15:30:35] ERROR[5814][C-00000074]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:35] ERROR[5814][C-00000074]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message [Oct 17 15:30:35] ERROR[5814][C-00000074]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:35] ERROR[5814][C-00000074]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message Using a string with shift method, feature3 is set to ---radio12345--- and is 8 long Context: fromvoipfone201 Exten:5 Using a string, feature is set to ---rad--- and is 7 long [Oct 17 15:30:43] ERROR[5815][C-00000075]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:43] ERROR[5815][C-00000075]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message Using a variable, feature2 is set to ---rad▒--- and is 8 long [Oct 17 15:30:43] ERROR[5815][C-00000075]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:43] ERROR[5815][C-00000075]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message [Oct 17 15:30:43] ERROR[5815][C-00000075]: json.c:704 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string. [Oct 17 15:30:43] ERROR[5815][C-00000075]: stasis_channels.c:773 ast_channel_publish_varset: Error creating message Using a string with shift method, feature3 is set to ---rad▒--- and is 8 long So, using the SHIFT trimmer cuts the junk, but still loses the last character in some cases. Here's the dialplan same => n,Set(feature=${FILE(/home/test/feature-1.txt,0,1,l,u)})
  same => n,Verbose(1,Using a string, feature is set to
  —${feature}— and is ${LEN(feature)} long)

  same => n,Set(featurefile=/home/test/feature-1.txt)
  same => n,Set(feature2=${FILE(${featurefile},0,1,l,u)})
  same => n,Verbose(1,Using a variable, feature2 is set to
  —${feature2}— and is ${LEN(feature2)} long)

  same => n,Set(unfilteredfeat=${FILE(${featurefile},0,1,l,u)})
  same => n,Set(feature3=${SHIFT(unfilteredfeat)})
  same => n,Verbose(1,Using a string with shift method, feature3 is set to —${feature3}— and is ${LEN(feature3)} long)

  Bug or… “feature”?!

  —

• –Apple-Mail=_2FE027E4-8D7D-4175-92CB-D1F3F106FB1D
  Content-Transfer-Encoding: quoted-printable Content-Type: text/plain;
  charset=us-ascii

  Heya Jonathan

  Interesting problem!

  Unfortunately I can’t help with suitable dialplan code to resolve your issue, but I thought I’d at least throw this snippet out to you:

  If you want to know what is _really_ in that file (including all invisible characters and anything else that wc etc might not count), pipe it through ‘hexdump’. Then you get to see the raw contents of the file in hex and can compare it to other files or the output you get from other tools (and you can look up the ASCII char codes for invisible chars to explore what they are etc).

  Eg:

  cat /home/test/feature-1.txt | hexdump

  Hope this tool helps you in your quest 🙂

  Pete

  –Apple-Mail=_2FE027E4-8D7D-4175-92CB-D1F3F106FB1D
  Content-Transfer-Encoding: quoted-printable Content-Type: text/html;
  charset=us-ascii

  On 18/10/2016, at 2:31 am, Jonathan H <lardconcepts@gmail.com> wrote:

  
  

  I have a plain text file, ASCII, unix line breaks. 1 single line, and all that is in it is the word “radio”.

  
  

  <snip>

  
  

  Heya Jonathan

  
  

  Interesting problem!

  
  

  Unfortunately I can’t help with suitable dialplan code to resolve your issue, but I thought I’d at least throw this snippet out to you:

  
  

  If you want to know what is _really_ in that file (including all invisible characters and anything else that wc etc might not count), pipe it through ‘hexdump’. Then you get to see the raw contents of the file in hex and can compare it to other files or the output you get from other tools (and you can look up the ASCII char codes for invisible chars to explore what they are etc).

  
  

  Eg:

  
  

  cat /home/test/feature-1.txt | hexdump

  
  

  Hope this tool helps you in your quest 🙂

  
  

  Pete

  
  

  
  

  
  

  
  –Apple-Mail=_2FE027E4-8D7D-4175-92CB-D1F3F106FB1D

• –Apple-Mail=_DBF23CE7-2ABB-46DB-88D2-08FFC94FCBB4
  Content-Transfer-Encoding: quoted-printable Content-Type: text/plain;
  charset=us-ascii

  Heh.. yes, fair call 😉

  Pete

  –Apple-Mail=_DBF23CE7-2ABB-46DB-88D2-08FFC94FCBB4
  Content-Transfer-Encoding: quoted-printable Content-Type: text/html;
  charset=us-ascii

  On 18/10/2016, at 10:38 am, Steve Edwards <asterisk.org@sedwards.com> wrote:

  
  

  cat /home/test/feature-1.txt | hexdump
  

  
  Or just:
  
  hexdump /home/test/feature-1.txt

  
  

  Heh.. yes, fair call 😉

  
  

  Pete

  
  

  
  –Apple-Mail=_DBF23CE7-2ABB-46DB-88D2-08FFC94FCBB4

• I’m going to go ahead and file a bug report, ‘cos something definitely ain’t right here! Bug filed:
  https://issues.asterisk.org/jira/browse/ASTERISK-26481

  This bit of dialplan…..

  exten => 5,1,Verbose(Context: ${CONTEXT} Exten:${EXTEN})
  same => n,Set(featurefile=/home/test/feature-1.txt)
  same => n,set(myList=1,12,123,1234,12345,123456,A,AB,ABC,ABCD,ABCDE,ABC123)
  same => n,While($[“${SET(rawVar=${SHIFT(myList)})}” != “”])
  same => n,SET(FILE(${featurefile})=${rawVar})
  same => n,SET(fileVar=${FILE(${featurefile},0,1,l,u)})
  same => n,SET(hexDump=${SHELL(hd ${featurefile})})
  same => n,Verbose(Before file: —${rawVar}— len ${LEN(rawVar)}
  after file: —${fileVar}— len ${LEN(fileVar)} hexdump:${hexDump})
  same => n,EndWhile()
  same => n,Hangup()

  …produces this output: (tidied only for posting) – this is messed up in so many ways.

  LEN is showing the wrong length.
  1 character gets a weird character added.
  23, and 4 characters OK
  5 characters loses a character – BUT LOOK! Even though hexdump prints the correct contents and length, have a close look at this: 35|12345|
  whereas normally it would have a space before the pipe.
  6 characters is OK

  Before file: —1— len 6 after file: —1▒— len 7 hexdump:00000000
  31|1| 00000001
  Before file: —12— len 6 after file: —12— len 7
  hexdump:00000000 31 32 |12| 00000002
  Before file: —123— len 6 after file: —123— len 7
  hexdump:00000000 31 32 33 |123| 00000003

  [Oct 18 12:23:53] ERROR[8584][C-0000001a]: json.c:704 ast_json_vpack:
  Error building JSON from ‘{s: s, s: s}’: Invalid UTF-8 string.
  [Oct 18 12:23:53] ERROR[8584][C-0000001a]: stasis_channels.c:773
  ast_channel_publish_varset: Error creating message

  Before file: —1234— len 6 after file: —1234— len 7
  hexdump:00000000 31 32 33 34 |1234| 00000004
  Before file: —12345— len 6 after file: —1234— len 7
  hexdump:00000000 31 32 33 34 35|12345| 00000005
  Before file: —123456— len 6 after file: —123456— len 7
  hexdump:00000000 31 32 33 34 35 36 |123456| 00000006

  [Oct 18 12:23:53] ERROR[8584][C-0000001a]: json.c:704 ast_json_vpack:
  Error building JSON from ‘{s: s, s: s}’: Invalid UTF-8 string.
  [Oct 18 12:23:53] ERROR[8584][C-0000001a]: stasis_channels.c:773
  ast_channel_publish_varset: Error creating message

  Before file: —A— len 6 after file: —A▒— len 7 hexdump:00000000
  41|A| 00000001
  Before file: —AB— len 6 after file: —AB— len 7
  hexdump:00000000 41 42 |AB| 00000002
  Before file: —ABC— len 6 after file: —ABC— len 7
  hexdump:00000000 41 42 43 |ABC| 00000003

  [Oct 18 12:23:53] ERROR[8584][C-0000001a]: json.c:704 ast_json_vpack:
  Error building JSON from ‘{s: s, s: s}’: Invalid UTF-8 string.
  [Oct 18 12:23:53] ERROR[8584][C-0000001a]: stasis_channels.c:773
  ast_channel_publish_varset: Error creating message

  Before file: —ABCD— len 6 after file: —ABCD— len 7
  hexdump:00000000 41 42 43 44 |ABCD| 00000004
  Before file: —ABCDE— len 6 after file: —ABCD— len 7
  hexdump:00000000 41 42 43 44 45|ABCDE| 00000005
  Before file: —ABC123— len 6 after file: —ABC123— len 7
  hexdump:00000000 41 42 43 31 32 33 |ABC123|

  —

• October 2019, Asterisk 17.0, this is still happening

  Bug updated: https://issues.asterisk.org/jira/browse/ASTERISK-26481

  File is a single line unix-line UTF-8 file, with “1,” (no quote marks)

  hexdump output is as follows

  0000000 2c31
  0000002

  dialplan is

  same =>
  n(skipto),Set(unfilteredfeat=${FILE(${scanPath}/skipto.txt,0,1,l,u)})
  same => n,Set(feature=${SHIFT(unfilteredfeat)})
  same => n,Verbose(1,feature is ${feature} unfilteredfeat is
  ${unfilteredfeat})

  Console log output is

  — Executing [s@root:40] Set(“Local/s@root-00000011;2”,
  “unfilteredfeat=1,▒=”) in new stack
  [Oct 29 09:36:15] ERROR[2134][C-00000012]: json.c:607 ast_json_vpack: Error building JSON from ‘{s: s, s: s}’: Invalid UTF-8 string.
  [Oct 29 09:36:15] ERROR[2134][C-00000012]: Got 11 backtrace records
  # 0: /usr/sbin/asterisk(ast_json_pack+0xa5) [0x5627588a7f45]
  # 1: /usr/sbin/asterisk(ast_channel_publish_varset+0x30) [0x562758937820]
  # 2: /usr/sbin/asterisk(pbx_builtin_setvar_helper+0x148) [0x5627588e8e78]
  # 3: /usr/sbin/asterisk(pbx_builtin_setvar+0x119) [0x5627588e9209]
  # 4: /usr/sbin/asterisk(pbx_exec+0x158) [0x5627588dedf8]
  # 5: /usr/sbin/asterisk(+0x137479) [0x5627588d3479]
  # 6: /usr/sbin/asterisk(+0x139134) [0x5627588d5134]
  # 7: /usr/sbin/asterisk(+0x13a7a1) [0x5627588d67a1]
  # 8: /usr/sbin/asterisk(+0x1bd598) [0x562758959598]
  # 9: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9669) [0x7fa95fd1b669]
  #10: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fa95fa9f323]

  [Oct 29 09:36:15] ERROR[2134][C-00000012]: stasis_channels.c:1161
  ast_channel_publish_varset: Error creating message
  [Oct 29 09:36:15] ERROR[2134][C-00000012]: json.c:607 ast_json_vpack: Error building JSON from ‘{s: s, s: s}’: Invalid UTF-8 string.
  [Oct 29 09:36:15] ERROR[2134][C-00000012]: Got 12 backtrace records
  # 0: /usr/sbin/asterisk(ast_json_pack+0xa5) [0x5627588a7f45]
  # 1: /usr/sbin/asterisk(ast_channel_publish_varset+0x30) [0x562758937820]
  # 2: /usr/sbin/asterisk(pbx_builtin_setvar_helper+0x148) [0x5627588e8e78]
  # 3: /usr/lib/asterisk/modules/func_strings.so(+0x64c0) [0x7fa93fd054c0]
  # 4: /usr/sbin/asterisk(ast_func_read+0x326) [0x5627588e35b6]
  # 5: /usr/sbin/asterisk(pbx_substitute_variables_helper_full+0x282)
  [0x5627588e7cc2]
  # 6: /usr/sbin/asterisk(+0x137445) [0x5627588d3445]
  # 7: /usr/sbin/asterisk(+0x139134) [0x5627588d5134]
  # 8: /usr/sbin/asterisk(+0x13a7a1) [0x5627588d67a1]
  # 9: /usr/sbin/asterisk(+0x1bd598) [0x562758959598]
  #10: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9669) [0x7fa95fd1b669]
  #11: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fa95fa9f323]

  [Oct 29 09:36:15] ERROR[2134][C-00000012]: stasis_channels.c:1161
  ast_channel_publish_varset: Error creating message
  — Executing [s@root:42] Set(“Local/s@root-00000011;2”, “feature=1”) in new stack
  — Executing [s@root:43] Verbose(“Local/s@root-00000011;2”, “1,feature is 1 unfilteredfeat is ▒=”) in new stack feature is 1 unfilteredfeat is ▒=