Second MuleSoft Guadalajara Meetup!
Welcome again to the Integration World! On Thursday April 11, we had our second MuleSoft Meetup in Guadalajara, this time using Mule 4. Our partner, MuleSoft, sent us some gifts and souvenirs for the occasion. We are glad to tell you what happened.
The Meetup took place in the offices of IO Connect Services. Starting at 7:00 pm by receiving and welcoming our guests and giving them at the registration a Max the Mule figure. Mireya Negrete, our HR girl, opened by giving an introduction of the company to the assistants and introduced the speakers. Main presentation started at 7:45 pm, being the main subject “Integrations Patterns and Best Practices”, with Salvador Zalapa and Larisa Arenas as speakers.
Salvador presented the rest of the integration team and stated the agenda:
- Introduction of the use case: Big Data Pipeline
- The solution as a hybrid Java-Mule app
- The solution with Mule Enterprise elements
- Integrations Patterns
- Best Practices to follow
- Networking time (pizza and beers to everybody, yeah!)
Salvador started presenting the big data pipeline scenario. The main idea is to implement a system that imports CSV files, with millions of records, transform them into JSON objects and send them to an AWS S3 bucket that serves as a data lake. Also, the system must separate failed records and send notifications if needed. The non-functionals requirements of this scenario are: the app must be reliable, secure and have good performance in time and machine resources.
He continued by presenting the proposed architecture, you can see it at the next image.
Then, he explained to the group the Java-Mule solution. The app used a File connector to pick the CSV from a specific folder, later Java classes were in place to handle the manual streaming and pagination of the file. When the app executed, one million records were split into pages of 2500 records, creating a page to be inserted to S3 bucket. In case a record has an invalid value, it’s sent to a dead-letter channel implemented in a queue in AWS SQS, so it could be processed later and not lose the value. In case of any other error, the On Propagate scope handles the error and sends a notification email with an SMTP connector. He explained each of the elements in the application while it was running and processing the million records. After the process was finished, he proved that all the records were saved correctly in the S3 bucket, also they tested the invalid record scenario to demonstrate that the entire row is saved in SQS. This part was interesting because many of the guests had questions and comments, all of them were focused at the presentation all the time. To finish Salvador’s topic, he commented on the level of accomplishment non-functionals requirements.
15-minute break time! Yes! All the participants ate chips and drank soda, chose stickers, pencils and sticky notes provided by MuleSoft.
After the break, Larisa continued with the Mule Enterprise solution, she exposed how the record processing can be done using the fascinating batch scope, that handled the split and processing of the records by itself. The batch inside of its steps, sent the valid records to S3 and the failed ones to SQS. In case of any errors, the app would send a notification email with the error in it. Once Larisa finished the Mule solution, she explained the accomplished non-functionals requirements and how it was easier to achieve them.
She continued by explaining the most common integration patterns implemented in this solution, which were: polling consumer, service activator, point-to-point channel, document message, guaranteed delivery, content enricher, dead-letter channel and poisoned message.
The last topic was best practices, which were also used in both solutions: retry strategy (until-successful scope), reconnection strategies in each external system connector, error handling, properties externalization, secure configuration properties, asynchronous processes, transaction management, streaming and usage of Mule language (DataWeave).
Once she finalized, they got some MuleSoft presents because of their assistance, enthusiasm and participation, the gift package included a shirt, a Mule sticker, an IO Connect sticker and an IO Connect pen.
Finally, we thanked everybody for their time and attention and then it was networking time! The time had come to meet new people, eat pizza and drink beer, cheers!
If you want to have a look at the presentation, see it in our MuleSoft Meetup page (in Spanish)
A short video