nifi convert text file to json

0

I'm trying to load log text files from a ftp server to elastic .

The log files look like this :

0:0:21: Processing events from events
0:0:21: Processing croned build types from q_type
0:0:21: Process croned releases from trls
0:0:22: Processing croned regression list from regression
0:0:22: Processing commit loop

in data provenance (hex view , because other views not showing anything)i see the data like this :

0x00000090  66  69  65  6C  64  3A  20  52  4E  20  53  74  61  74  75  73  field: RN Status
0x000000A0  2E  20  4F  62  6A  65  63  74  20  72  65  66  65  72  65  6E  . Object referen
0x000000B0  63  65  20  6E  6F  74  20  73  65  74  20  74  6F  20  61  6E  ce not set to an
0x000000C0  20  69  6E  73  74  61  6E  63  65  20  6F  66  20  61  6E  20  instance of an
0x000000D0  6F  62  6A  65  63  74  2E  0D  0A  30  3A  30  3A  31  34  3A  object...0:0:14:
0x000000E0  20  43  61  6E  27  74  20  72  65  61  64  20  69  73  73  75  Can't read issu
0x000000F0  65  3A  20  41  49  2D  32  34  37  20  63  75  73  74  6F  6D  e: AI-247 custom
0x00000100  20  66  69  65  6C  64  3A  20  52  4E  20  53  65  63  74  69  field: RN Secti
0x00000110  6F  6E  2E  20  4F  62  6A  65  63  74  20  72  65  66  65  72  on. Object refer
0x00000120  65  6E  63  65  20  6E  6F  74  20  73  65  74  20  74  6F  20  ence not set to
0x00000130  61  6E  20  69  6E  73  74  61  6E  63  65  20  6F  66  20  61  an instance of a
0x00000140  6E  20  6F  62  6A  65  63  74  2E  0D  0A  30  3A  30  3A  31  n object...0:0:1
0x00000150  34  3A  20  43  61  6E  27  74  20  72  65  61  64  20  69  73  4: Can't read is
0x00000160  73  75  65  3A  20  41  49  2D  32  34  37  20  63  75  73  74  sue: AI-247 cust
0x00000170  6F  6D  20  66  69  65  6C  64  3A  20  52  4E  20  44  6F  63  om field: RN Doc
0x00000180  20  69  6E  20  56  65  72  2E  20  4F  62  6A  65  63  74  20  in Ver. Object
0x00000190  72  65  66  65  72  65  6E  63  65  20  6E  6F  74  20  73  65  reference not se
0x000001A0  74  20  74  6F  20  61  6E  20  69  6E  73  74  61  6E  63  65  t to an instance
0x000001B0  20  6F  66  20  61  6E  20  6F  62  6A  65  63  74  2E  0D  0A  of an object...
0x000001C0  30  3A  30  3A  31  34  3A  20  43  61  6E  27  74  20  72  65  0:0:14: Can't re
0x000001D0  61  64  20  69  73  73  75  65  3A  20  41  49  2D  32  34  37  ad issue: AI-247

I can get the file with "getftp" processor, but how do I convert it to json so I can send it to Elastic ?

I am new to nifi hope im not missing something basic, any help will be appreciated.

Thanks

elasticsearch
apache-nifi
elastic-stack
asked on Stack Overflow Jul 2, 2019 by alex • edited Jul 4, 2019 by marc_s

1 Answer

0

You can use the ConvertRecord processor with a CSVReader for the input (configure to use : as the delimiter) and a JsonRecordSetWriter for the output.

NiFi can automatically infer the schema, but as it doesn't appear you have a header line for the incoming data, this will probably not be helpful. In that case, you can use the Schema Registry to hold two schemas -- one for the incoming log lines, indicating what each field should be called and the data type, and one for the JSON output. Bryan Bende has written a great article about this process.

answered on Stack Overflow Jul 2, 2019 by Andy

User contributions licensed under CC BY-SA 3.0