Forum Discussion

richie's avatar
richie
Community Hero
7 years ago

Unsure if I can test in ReadyAPI! - ESB polls a directory to pick up a flat file??

Hi,

 

A previous project I worked on had the following process flow:

 

 

1. 3rd party system, ETL generates a flat file, FTP moves file to a directory

2. ESB polls the directory and picks up the flat file, transforms the file to .json

3. .json is published to the MS Dynamics CRM.

 

I have 2 lots of tests exercising the above:

 

1st set of tests to verify the files are transformed by the ESB's adaptor to .json correctly

 

2nd set of files is that I publish the .json files directly to the CRM .

 

I was just wondering if there's anyway to alter my test approach to include creating a file and moving it to the directory the ESB polls?  At the moment my testing completely skips point 1 and I was wondering if I could actually include point 1 in my tests?

to be honest - I'm unsure if this really adds much value as once I've proved it worked once, I wouldn't need to test it again anyway as I'd just be testing the FTP config I suppose. Also - I should highlight that I've tested this end to end approach manually rather than in SoapUI - i.e. create a file, move it to the directory, allow the FTPclient to move the file to the directory the ESB polls and then query the CRM's SQL Server database to make sure the records contained within the file were updated within the CRM correctly.

 

I was just wondering if anyone had done something like the above before - and whether it would actually be worth it?

 

I have been reading a little and can see there's an FTP step and a File Wait step - but I'm unsure if these actually provide what I need - the File Wait step appears to just 'pass' if a file appears in the relevant directory and the FTP step allows you to upload files to an FTP server and 'passes' if the file is moved successfully.

 

Any help/guidance/tips/hints would be most appreciated!

 

Cheers guys,

 

richie

 

 

 

 

My testing of this previously is that I create test data .json files and publish to the CRM 

  • PM's avatar
    PM
    Contributor
    7 years ago

    Hey Richi, 

     

    Any updates on this please.....how you are testing it ........did u get any solution of it ???? Please share - we seems to hv such in ours as well.

     

    Flat file ---> ETL (talend) ---> JSON --> Elastic DB ---> Schema free JSON 

     

    Testing scope :- 

    Content Validation : Comparing data of JOSN from ETL with Flatfile

    Content Validation : Comparing Schema free JSON with JSON frm ETL 

     

    Regards, 

    PM

    • richie's avatar
      richie
      Community Hero
      7 years ago
      hi,

      well what i wanted really was an extension or an improvement to my tests. is your messaging synchronous or asynch? cos thats going to make a difference....if ita asynch (and if your message broker paradigm is publish and subscribe then this is by definition asynch. you can also have a combination of synch and asynch messaging with your ESB too.
      the problem with asynch messaging is you wont have any responses to assert against. occasionally the synch responses wont give you what you need also. for instances like this its necessary for your dev teams to develop additional apis that allow you to do your testing.

      if you can give me more info about the message exchsnge patterns etc., i can define an integration testing strategy for you(essentially thats my job) but someone there....maybe an integration practice lead or an architect should be defininf your integration testing strategy.

      i can make suggestions i.e. like the pros and cons of testing e2e apis (so in example where an etl generates a flat file, gets pulled onto esb, is transformed (filetype, field attributes and field values) and the resultant message is consumed by a webservice) so you could have a POST api created that pushes your file to a flow on your esb and then subsequently onto the webservice followed by a GET to prove the transformations correctly or inversely 2 point to point apis, 1 that allows you to post to the adaptor to verify your transformations and another POST to the webservice followed by either a jdbc query or a GET request to verify the record was inserted/updated/upserted into your db, but someone at your place of work should be defining this for you.

      as i say, i'll help in anyway i can but 1. im going to need more info and 2. youre asking for an integration testing strategy which someone else should be ok'ing.
      the approach you are using should be defined within an integration test plan by someone doing my job at your place.

      cheers,

      richie
      • richie's avatar
        richie
        Community Hero
        7 years ago
        in the short term i wouldnt worry about the ftp bit anyway....really youd just be testing the ftp co fig and so this doesnt really provide enough value to warrant the effort involved so as a start id just injected manually created flat files.
  • nmrao's avatar
    nmrao
    Champion Level 3
    7 years ago
    It is worth to automate, if possible, any thing which can help to avoid manual work.


    It is not clear when you say "create file". What sort of file is this? If you have data, and have certain template, then file can be created automatically.

    FTP, yes, it is very much possible. You can find example code snippets online.